Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / Why Your Engineering Team Needs a Congestion Budget
Transcript
- Lucas: Let me ask you something, Luna. Have you ever watched an engineering team where everyone seems busy, code is being written, but somehow features just take forever to ship? Luna: Oh, I've lived it. The team is sprinting in place. What's the root cause? Lucas: The root cause is almost never bad engineers or laziness. It's almost always congestion. Too many things in flight, not enough capacity at key choke points — code review, testing, deployment. And most teams don't measure it. They just feel it. Luna: So you're talking about a congestion budget. Like a financial budget, but for work in progress. Lucas: Exactly. A congestion budget is a predetermined limit on how much work can be in a given stage of your pipeline at any time. The idea comes from queue theory — the math that governs how wait times explode when utilization passes a certain threshold. If your code review queue is consistently more than, say, five pull requests deep, your lead time doubles or triples. Luna: And most teams don't set those limits. They just pile on more work and wonder why everything slows down. Lucas: Right. I want to talk about a real example. There's a mid-stage fintech company — about 80 engineers, growing fast. Their deployment frequency had plateaued, and feature cycle time had crept from two days to nearly two weeks. They tried hiring more engineers, but that actually made things worse for a while, because the new hires needed reviews and onboarding. Luna: Classic. More people can increase congestion before they increase throughput. Lucas: So they did something different. They introduced a congestion budget for each stage of their pipeline: development, code review, staging testing, and deployment. For code review, they set a budget of six open pull requests per team. Once that queue hits six, no new pull requests can be opened until some are closed. And they made it visible — a dashboard showing queue depth across teams. Luna: Six seems low. Did the developers revolt? Lucas: At first, yes. Engineers felt constrained. But within three weeks, cycle time dropped by 60 percent. Because the bottleneck shifted from 'waiting for review' to 'actually writing code that gets reviewed quickly.' People started breaking work into smaller chunks, and reviewers had capacity to do thorough reviews without context-switching. Luna: So the budget forced better behavior. That's the mechanism — it makes the constraint visible and creates pressure to resolve it. Lucas: That's the whole philosophy. A congestion budget isn't just a cap. It's a forcing function for the team to ask: what's the simplest way to clear this queue? That might mean swarming on reviews, or pairing to finish a feature faster, or even saying 'no' to a new piece of work until capacity frees up. Luna: It's like the WIP limits in Kanban, but applied more granularly across the entire pipeline. Lucas: Exactly. Traditional Kanban limits work in progress per column. A congestion budget takes it a step further — it's a total system limit that accounts for dependencies between stages. For example, if the test environment can only handle five concurrent test suites, your congestion budget for the testing stage should be five, even if your WIP limit for 'in test' is higher. Luna: So it's based on actual capacity constraints, not just arbitrary numbers. How do you figure out the right budget numbers? Lucas: You start by measuring. For each stage, track the time work items spend waiting in queue versus being actively processed. That's your queue depth and wait time. The theory is simple — Little's Law: average number of items in a system equals average arrival rate times average time in system. If you know your desired cycle time, you can calculate the max queue depth that keeps you on track. Luna: So if you want features to ship in five days, and your team completes one feature per day on average, you can't have more than five features in the pipeline at any time. Lucas: Right. Obviously real pipelines are messier, but the principle holds. The fintech team I mentioned started with historical data: they measured that every time the code review queue exceeded eight pull requests, the average review time doubled. So they set their congestion budget at six, which gave them a buffer. Then they tuned it over time. Luna: Did they apply budgets to every stage? Development, review, test, deploy? Lucas: They started with the two biggest bottlenecks: code review and staging testing. Code review had the deepest queue, staging testing had the longest wait due to environment contention. For staging, they limited concurrent test runs to three, based on their test infrastructure capacity. That forced teams to prioritize which features got tested first, and it eliminated the logjam where five teams were all waiting for the staging environment. Luna: Environment contention is such a silent killer. Nobody budgets for it. Lucas: Exactly. And that's where the budget makes trade-offs explicit. If team A's feature is in staging test, team B knows they have to wait. But because the budget is visible, team B can either help team A finish their test faster, or they can pair with another team on something that doesn't require staging. It drives collaboration instead of frustration. Luna: I can see this being really powerful for platform teams too. They're often the biggest bottleneck — everyone needs their infrastructure changes. Lucas: Great point. Platform teams are like shared resources in a factory. If they don't have a congestion budget, they get overwhelmed by requests from every product team. One company I know — a large e-commerce player — gave their platform team a congestion budget of five active requests. Product teams had to queue up, and the platform team committed to a two-day turnaround per request. That forced product teams to batch their requests and be more thoughtful. Luna: So it's not just about limiting work, it's about creating predictability. Teams know how long they'll wait. Lucas: That's a huge benefit. When you have a congestion budget, you can give stakeholders a reliable estimate: 'Your feature will enter review within one day, because our budget ensures no more than six items ahead of you.' Without the budget, you're guessing. And the guess is almost always wrong. Luna: What about the downside? If you set the budget too tight, you starve the pipeline and reduce throughput. Lucas: That's a real risk. The fintech team actually set their initial budget too low — four for code review — and they saw a drop in throughput because reviewers had nothing to review. Engineers were waiting for reviews to clear so they could submit new code, but reviewers had idle time. They bumped it to six and that was the sweet spot. Luna: So it requires tuning. You need to monitor both queue depth and throughput, and adjust the budget iteratively. Lucas: Yes. And you need to involve the team. The budget shouldn't be imposed by a manager in a spreadsheet. The engineers who live in the pipeline should help set the thresholds. They know when they feel the pain of waiting versus the pain of being rushed. The budget is a shared agreement, not a top-down constraint. Luna: So how do you roll this out? Let's say I'm a CTO listening, and I want to try this next quarter. Lucas: First, pick one stage — usually code review or test environment. Measure current queue depth and cycle time for two weeks. Then set a budget that's slightly lower than the average queue depth you observed. The goal is to create mild pressure, not a choke. Communicate it as an experiment: 'For the next two weeks, we're limiting the review queue to six items. We'll measure the impact and adjust.' Luna: And you need a dashboard. Visible to everyone. So when the queue hits five, people can see it coming. Lucas: Absolutely. Make it real-time. The fintech team had a big screen in the office showing each team's congestion level — green, yellow, red. Green was under 60 percent of budget, yellow was 60 to 90 percent, red was over 90 percent. When a team hit red, they had two options: drop everything and clear the queue, or escalate to the engineering manager to negotiate a temporary budget increase with the teams waiting downstream. Luna: That escalation process is interesting. It forces a conversation about priority. Lucas: Exactly. If team A wants to exceed their congestion budget, they have to get buy-in from team B, who will be delayed as a result. That makes trade-offs visible and explicit. Before the budget, team A would just submit the work and team B would suffer silently. Now there's a conversation. Luna: And that conversation builds a healthier culture. Teams start thinking about the system, not just their own feature. Lucas: That's the ultimate goal. A congestion budget is a tool, but the real shift is cultural — from 'push work into the system and hope it flows' to 'pull work through the system based on capacity.' It's a lean manufacturing principle applied to software, and it works. Luna: And it costs nothing to implement. No tools to buy, just a mindset change and a dashboard. Lucas: Exactly. Which is why it's one of the highest-leverage changes an engineering leader can make. If today's conversation gave you something useful, you know, it's the kind of practical insight that listener support helps us keep producing ad-free. If you feel like tossing a coffee our way, it's at buymeacoffee dot com slash fexingo. No pressure — just helps us keep these conversations going. Luna: And we really appreciate it. Now, back to the budget — one thing I want to circle back on: what happens when a critical security fix needs to go out immediately? Doesn't the budget get in the way? Lucas: Great question. The budget is for normal flow, not for emergencies. You add an override mechanism — if there's a P0 incident or a security vulnerability, you can bypass the budget. But you track how many times you use the override. If it's more than once a quarter, your normal budget thresholds might be too tight. The override becomes a signal, not a loophole. Luna: So it's a diagnostic tool as well. Overuse of overrides tells you something about your system. Lucas: Exactly. And that's the beauty of the congestion budget. It doesn't just fix a bottleneck — it reveals where your system is under pressure, so you can invest in the right places. Maybe you need more test environments, or a faster CI pipeline, or to hire another reviewer. The budget tells you where the constraint is, and then you can address the root cause. Luna: So this is something every engineering team should try, even for a month. The worst case is you learn something about your flow. Lucas: That's exactly right. And you don't need a perfect implementation. Just pick one queue, set a limit, and measure. The data will tell you if it's working. That's the real leadership move — not just solving the problem, but installing a system that continuously reveals and resolves congestion.