Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / Why Your Engineering Team Needs a Congestion Budget
Transcript
- Lucas: So here's a question I've been sitting with: Why do most engineering teams treat their CI/CD pipeline like it has infinite capacity? Luna: I'm guessing the answer is not 'because it does.' Lucas: Exactly. It doesn't. And when teams ignore that, they get congestion. Deployments slow down, rollbacks spike, and engineers start blaming each other instead of the system. Luna: So what's the fix? More runners? Bigger clusters? Lucas: Sometimes. But more often, the fix is something I'm calling a 'congestion budget.' It's a concept borrowed from traffic engineering — specifically, from how urban planners manage road networks. Luna: Okay, I'm interested. How does a traffic model apply to code deployments? Lucas: Think of your CI/CD pipeline as a network of queues. Every commit enters a queue. Tests are queues. Builds are queues. Deployments are queues. When too many items enter the system at once, the queues grow, latency increases, and eventually, the whole thing gridlocks. Luna: Yeah, I've seen that. A team merges ten PRs in five minutes and suddenly the pipeline takes an hour. Lucas: Right. A congestion budget sets hard limits on how much work can be in each queue at any time. You define a maximum WIP — work in progress — for each stage. Once that limit's hit, new work doesn't enter until something exits. Luna: That sounds like a pull system instead of a push system. Lucas: Exactly. It's kanban for your pipeline. I saw this work brilliantly at a Series A startup — about forty engineers, shipping to production maybe ten times a day. Their average deployment time was forty-five minutes, and their rollback rate was around eight percent. Luna: Which is pretty bad. Lucas: It was. They kept throwing hardware at the problem — more build agents, more test runners. But congestion wasn't a capacity problem. It was a flow problem. Luna: So what did they actually change? Lucas: They mapped their entire pipeline as a queue network. Identified three key choke points: the test suite, the staging deployment, and the production rollout. Then they set a WIP limit for each. Max two concurrent test runs. Max one staging deployment at a time. Max one production rollout per thirty minutes. Luna: And the developers didn't revolt? That sounds like slowing things down. Lucas: Some did at first. But within two weeks, deployment time dropped from forty-five minutes to twelve. Rollback rate fell to about three percent. The reason? By limiting WIP, they halved the queue depth. So each individual item moved through faster. Luna: Okay, so the metrics are WIP limits, queue depth, and... what's the third? Lucas: Batch size. How many changes are in each deployment. They also started batching smaller — instead of deploying five features together, they'd deploy one or two. That reduced the risk surface. Luna: But smaller batches mean more deployments. Doesn't that increase overhead? Lucas: It does if your deployment process is heavy. But if you've already reduced the cycle time, the overhead per deployment shrinks. The net effect is that you ship more value per unit time, with less risk. Luna: So a congestion budget has three knobs: WIP limits, queue depth limits, and batch size limits. Lucas: Exactly. And here's the thing — most teams only think about one of those knobs: capacity. They buy more servers, more licenses, more parallelization. But capacity isn't the bottleneck in a knowledge-work system. Variability is. Luna: Variability in arrival rate? Or in processing time? Lucas: Both. When arrivals are bursty — like after a sprint review — the system gets overwhelmed. And when processing times are unpredictable — say a test suite that sometimes takes five minutes, sometimes thirty — you can't plan. A congestion budget smooths that out by design. Luna: This reminds me of something from queueing theory: Little's Law. Average number of items in a system equals arrival rate times average time in system. Lucas: Exactly. If you want to reduce time in system, you either reduce arrival rate or reduce the number of items. Since you can't always control arrival rate — features have to ship — you control the number of items in the system. That's the WIP limit. Luna: So a congestion budget is essentially an implementation of Little's Law. Lucas: In practice, yes. But the term 'congestion budget' makes it more tangible for engineers. It's like saying: 'We have X units of congestion capacity. If we exceed it, the system breaks.' Luna: Where else could you apply this? Code reviews? On-call? Lucas: Absolutely. Code review is a classic queue. If you have ten open reviews per engineer, the latency per review skyrockets. Some teams set a WIP limit on open reviews — say, max three per person. Once you hit three, you can't submit a new one until you've reviewed someone else's. Luna: That forces a balance between submitting and reviewing. Lucas: Yeah. And for on-call, you can set a congestion budget on incidents. If there are more than, say, three active incidents, the on-call engineer can escalate or pull in backup. It prevents the single point of failure from being overwhelmed. Luna: But isn't there a risk that these limits become arbitrary? Engineers might game them — split a large batch into tiny ones just to bypass the limit. Lucas: That's a real concern. The budget has to be enforced by the system, not by policy. If your CI pipeline enforces that only two builds can run concurrently, engineers can't game it. They just have to wait their turn. Luna: So the system enforces the constraint, and the constraint creates the behavior. Lucas: Right. And the key is to measure and tune. Start with generous limits and tighten them over time. The startup I mentioned started with a WIP limit of five concurrent test runs. They tightened it to two over a month. Luna: What about teams with really unpredictable workloads? Like a platform team that sometimes gets zero requests and sometimes fifty? Lucas: That's trickier. For highly variable arrival rates, you might need a dynamic congestion budget — one that adjusts based on recent history. But even then, you need an upper bound. Otherwise, you're back to infinite capacity assumptions. Luna: So the principle is: treat your pipeline as a finite resource, and budget accordingly. Lucas: Exactly. And the beauty is, it's cheap. You don't need new tools or more infrastructure. You just need the discipline to set limits and the instrumentation to measure them. Luna: Yeah, and that reminds me — if conversations like this are useful to you, it's partly because we keep the show independent and ad-free. If you get value from episodes like this, you can support the show at buy me a coffee dot com slash fexingo. It's a small way to keep the conversations going. Lucas: Absolutely. Even a few dollars helps us invest in better research and bring on more guests. But let's get back to the congestion budget — because I want to talk about one more application: incident response. Luna: Oh, interesting. How would that work? Lucas: During an incident, you have multiple streams of work: investigating the root cause, communicating with stakeholders, applying a fix, monitoring the fix. Each of those is a queue. If one person tries to do all of them, they get overloaded. A congestion budget says: 'At most two people per queue.' The incident commander delegates. Luna: That's essentially what the Incident Command System does in emergency services. Lucas: Exactly. It's the same principle. And it works because it prevents any single node from becoming a bottleneck. So whether you're deploying code, reviewing PRs, or fighting a fire, a congestion budget gives you a simple, measurable way to keep the system flowing. Luna: And it's a lot cheaper than just buying more servers. Lucas: Right. It's a free performance improvement — if you have the discipline to implement it.