This week, I spoke with Jon Widdicombe, DevOps Lead at FXC, to learn about a typical day in his life.

Tell us about your journey at FXC so far

I joined in May 2021, so it’s been almost four years! I was initially hired as a DevOps engineer, and during my time here, I have been promoted first to Senior DevOps Engineer and then, last year, I became the DevOps Lead. Mostly, I’ve been building the platform that our applications run on, first on bare metal, and now we're moving back into AWS, so building up a better infrastructure. 

There has been a big transition in the way that we run applications. Before, everything was very cobbled together to some extent. Using the analogy of “cattle, not pets”, the idea is that if something is running and has a problem, you don’t spend time trying to fix it — you just kill it off and replace it. You treat your services like cattle, not pets: if a cow gets sick, you put it down rather than spend a fortune trying to save it. We used to have a very “pets” mentality, where we’d invest a lot of time and care into individual systems, but now we’ve shifted to a much more “cattle” mindset, which makes our infrastructure far more scalable and resilient. 

Describe your typical workday?

We have the morning DevOps team stand-up, where we review the in-progress work and flag any issues team members are encountering. 

I spend the rest of my mornings in meetings with management, scheduling, planning or picking up new things to fix later on. Then my afternoon is spent fixing tickets, responding to incidents or working on building a new platform in AWS.

Which of your responsibilities has had the biggest impact at FXC as a company?

So a lot of the stuff we do is in the background, and if there are problems, then we're not doing it right. I think a lot of what we do, since we're providing the infrastructure, is best when you don't hear about it. I look after our estate servers, our databases, our hard disk backends and make sure we have backups, that the backups work and that they're not failing.

The biggest impact we have really is ensuring we have platform stability, and that comes from multiple places of making sure the new infrastructure we're building is not going to interfere with old infrastructure, making sure that the existing infrastructure is kept up to date as much as possible and ensuring stability across the platform.

Could you tell me a bit about where we are with the AWS migration?

We started talking about this migration in May 2024 and broke ground late October 2024. We aim to have it fully completed by January 2026, but the sooner the better! 

Where we are now with the AWS migration, I'm hoping we're over the first set of hurdles. There have been a few teething problems, moving platforms from bare metal to AWS. There are lots of changes that have been made, with a lot more infrastructure as code and tools we weren’t using previously – for example, we were using Ansible for maintaining physical servers and we're now using Terraform to deploy AWS infrastructure. We're moving away from using Helm file, which is the Helm management tool for managing the Kubernetes manifests, to using Argo CD. 

So there's a more UI for people to actually see what's going on, but also it allows more people to have easier access to manipulating those files. It’s been a great learning opportunity and, hopefully, we’re getting to the point where we have a platform sufficiently stable for us to actually deploy stuff on. At this deployment stage, we know what to do, as it's just a case of adapting what we've already done onto those new platforms.  

What's the most rewarding part of your job at FXC?

I think the most rewarding part of the job for me is when you get to the point where you've had to learn a very complicated system, implement it and get it running, and then it does what it's meant to do. For example, we have a remote backend for persistent disk storage called Ceph. It’s a very complex system but there are tools out there that let you kind of blackbox, just throw it out there, but it's very risky because you end up not knowing how any of it works and being unable to dig around in that and figure out how it works and then being confident that it is going to work. 

What is the biggest challenge you face in your role?

For me, it's prioritisation. It can be difficult. There are a lot of things that are high priority at the same time, and once everything's high priority, nothing's high priority. But it's picking out which one of those things is actually urgent as well as priority, rather than things that aren't urgent but are high priority. It's checking the immediacy of the issue – is it a problem that is actually affecting something, or is it something that someone just wants to change?

What would you say is your proudest moment at FXC?

A lot of our work is supporting other teams and providing infrastructure, so the thing to be most proud of is the reliability of our platform, where we've been supporting pretty much all the components from the bare metal up. I suppose one of the best examples is when we've upgraded or migrated Ceph, which backs all our data storage, between servers with almost no visibility to the applications using that infrastructure.