Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How GitLab Rebuilt Its CI Pipeline for 5 Million Jobs Daily
Transcript
- Lucas: If you're running a development team today, there's a good chance your CI pipeline is the most critical piece of infrastructure you don't think about until it breaks. Luna: And when it breaks, the whole team grinds to a halt. No merges, no deployments, no feedback for hours. Lucas: Exactly. So when GitLab decided to rebuild its CI platform from the ground up a couple of years ago, they weren't just optimizing for speed — they were optimizing for reliability at a scale that few others handle. As of early 2026, their SaaS instance is running over five million CI jobs per day. Luna: That's staggering. Five million jobs daily across what, tens of thousands of organizations? Lucas: Yeah, it's a multi-tenant environment with everything from startups with one repo to enterprises with hundreds of projects. The old architecture was built when GitLab was a monolithic Rails app, and the CI scheduler was essentially a single process that polled the database for jobs. Luna: Obviously that doesn't scale to millions of jobs. What was the breaking point? Lucas: Around 2023, they started hitting database lock contention on the jobs table. The scheduler would hold a row lock while assigning a job to a runner, and with thousands of concurrent assignments, the database would thrash. Job wait times could spike to over a minute during peak hours. Luna: That's brutal. A minute to pick up a job that might take thirty seconds to run. Lucas: Right. So they did what any well-run engineering org does — they measured everything, then rebuilt the critical path. The big architectural change was moving job scheduling out of Rails and into a dedicated Go service they call the 'CI dispatcher'. Luna: Why Go specifically? Lucas: Concurrency model, mostly. Go's goroutines let them handle thousands of scheduling decisions per second without the overhead of OS threads. They also wanted a binary that started fast and had minimal memory footprint, especially because they run multiple dispatcher instances behind a load balancer. Luna: And the database contention — did going to Go solve that by itself? Lucas: No, the language change alone wouldn't fix it. The real trick was moving from a polling model to an event-driven one. The new dispatcher subscribes to a Redis stream of pending jobs, instead of repeatedly querying Postgres. When a new pipeline triggers, the Rails side pushes a lightweight event to Redis, and one of the dispatchers picks it up. Luna: So Redis became the job queue buffer. That reduces load on the main database significantly. Lucas: Exactly. They also sharded the job assignment logic by project ID. Each dispatcher instance is responsible for a range of projects, so the same team's jobs are always handled by the same dispatcher. That gives them better cache locality and reduces the need for distributed locks. Luna: That sounds like a classic sharding pattern, but applied to a service that was originally monolithic. What about artifact storage? Five million jobs produce a lot of build outputs. Lucas: Huge problem. GitLab stores artifacts — compiled binaries, test reports, static analysis output — for a configurable retention period. The old system stored everything on a single NFS mount, which became a bottleneck for both reads and writes. They migrated to object storage, using S3-compatible backends with a custom caching layer. Luna: Custom caching layer — meaning they didn't just stick CloudFront in front? Lucas: No, they built a dedicated caching service called 'Artifact Cache' that sits between runners and object storage. The cache is a set of ssd backed nodes that store the most recently accessed artifacts. If a runner requests a popular artifact — say, a common Docker layer — it's served from cache, which is ten to fifty times faster than fetching from object storage. Luna: What's the hit rate on that cache? Lucas: GitLab's engineering team published a blog post last year showing they hit about 85 percent for artifacts accessed within the first hour. After that, the cache evicts less popular items. They tune the eviction policy based on artifact size and access frequency, so small, frequently accessed artifacts stay cached longer. Luna: That's a practical trade-off. You don't want to cache a 2-gigabyte Docker image that's only used once. Lucas: Exactly. Another interesting piece is how they handle priority scheduling. Not all CI jobs are equal. A merge request pipeline for a critical security fix needs to run faster than a routine test on a feature branch. Luna: How do they differentiate? Do they let users set a priority flag? Lucas: They do, but the system also auto-detects priority based on context. If a pipeline is triggered by a merge request that targets the main branch and includes a label like 'security' or 'hotfix', the dispatcher boosts its priority. They implemented a multi-level priority queue where higher-priority jobs can preempt lower-priority ones that are still waiting for a runner. Luna: Preemption — does that mean they actually kill a low-priority job that's already running? Lucas: No, they don't kill running jobs. But when a runner becomes free, the dispatcher assigns it to the highest-priority pending job. So a high-priority job can skip ahead of the line. They also allow 'expedite' tokens — teams get a limited number per month that guarantee a job gets a runner within thirty seconds. Luna: That's smart. It gives teams a way to handle emergencies without abusing the system. What's the effect on average wait time? Lucas: Overall, they reduced average job wait time by about 40 percent compared to the old system, even though the total job volume has doubled since the migration. During peak hours, the 99th percentile wait time is under five seconds. Luna: That's impressive. But surely there were some painful parts of the migration. You're moving a system that processes millions of jobs a day — you can't just flip a switch. Lucas: They ran the old and new systems in parallel for six months. Each pipeline would be scheduled by both, but only one would actually execute. They compared results, monitored for anomalies, and gradually shifted traffic. They also had a 'dark launch' phase where the new dispatcher ran but didn't actually assign jobs — just logged what it would have done. Luna: So they could measure correctness without risking production. That's a textbook approach, but I imagine it required a lot of compute resources to run two schedulers. Lucas: It did, but they considered it insurance. The team lead told me they budgeted an extra 30 percent capacity for the shadow infrastructure. Compared to the cost of a major outage, it was cheap. Luna: Speaking of outages — did the new system introduce any new failure modes? Like, what happens if the Redis stream goes down? Lucas: Great question. They built a fallback mechanism where the dispatchers can also poll Postgres directly if Redis is unavailable. It's slower, but it keeps pipelines running. They also replicate Redis across two availability zones, so a single failure shouldn't cause an outage. Luna: What about runner management? With five million jobs, you need a lot of runners. How do they handle provisioning? Lucas: GitLab runs its own fleet of runners for the SaaS product, but customers can also bring their own. The interesting part is the autoscaling logic. The dispatcher tracks pending job volume and can request new runner instances from a pool of cloud VMs. They use a predictive algorithm — if the backlog grows faster than a certain rate, they pre-provision runners before the wait time spikes. Luna: That's proactive, not reactive. What's the typical spin-up time for a new runner? Lucas: About 45 seconds from request to accepting jobs. They use pre-warmed VM images with the runner agent already installed, so it's mostly just boot time. They also have a minimum idle pool to absorb small spikes immediately. Luna: And I assume they have to balance cost here — more idle runners means higher base cost. Lucas: Right. They target a minimum idle pool that can handle two minutes of typical peak demand. Anything beyond that gets provisioned on demand. They dynamically adjust the idle pool based on time of day and day of week, because usage patterns are quite predictable. Luna: That makes sense. So the whole system — dispatcher, cache, autoscaler — is designed to be cost-aware while maintaining reliability. How did they validate the new architecture before going all in? Lucas: They did a massive internal dogfooding campaign. GitLab has over two thousand employees, and their own CI pipeline is extremely demanding — they run thousands of jobs per day just for their own development. So they migrated the internal instance first, ran it for three months, and collected metrics. Luna: I bet that caught edge cases. Like, did they find any race conditions in the dispatcher's sharding logic? Lucas: They found a subtle bug where two dispatchers could both think they were responsible for the same project during a rebalancing event. That caused duplicate job assignments — two runners picking up the same job. They fixed it by adding a distributed lease mechanism using Redis itself. Luna: Classic distributed systems problem. So what's next for GitLab's CI? Now that the scheduling and storage are modernized, what's the next bottleneck? Lucas: They're looking at improving the job execution environment itself. Currently, each job runs in a Docker container on the runner, but the startup time for pulling large images is a pain point. They're experimenting with a 'runner cache' that stores frequently used images on the runner's local SSD, so subsequent jobs don't need to pull the same layers. Luna: That could shave tens of seconds off each job. Over five million jobs a day, that's huge. Lucas: Yeah, the total time savings would be massive. They're also exploring using Firecracker micro-VMs instead of Docker for better isolation — something that would let them offer stronger security guarantees for multi-tenant runners. Luna: Firecracker is what AWS Lambda uses. That would be a significant shift, but if they can make it work, it could be a differentiator for the platform. Lucas: Absolutely. And that kind of engineering investment is exactly what keeps a CI platform competitive. It's a reminder that even infrastructure that feels 'boring' — like running tests — requires constant evolution. Luna: It's funny, when you think about it, a CI pipeline is essentially a distributed job scheduler that needs to be faster than developers' patience. And developers have very little patience. Lucas: Exactly. And on that note — if these deep dives into how platforms are built are useful for what you're working on, we should mention that this show stays ad-free thanks to a handful of listeners who chip in monthly through Buy Me a Coffee. It's at buy me a coffee dot com slash fexingo, and it's what funds episodes like this one. Luna: It's a small gesture that makes a big difference for us. We really appreciate everyone who's part of that. Lucas: So back to GitLab — the key takeaway for me is that they approached the rebuild not as a rewrite, but as a surgical replacement of the critical path. They kept the interfaces stable and swapped out the internals one piece at a time. Luna: That's the lesson, isn't it? You don't have to boil the ocean. Pick the bottleneck, measure, fix, repeat. Lucas: Right. And in GitLab's case, that process reduced job wait times by forty percent and gave them a platform that can handle whatever their customers throw at it next.