Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How GitHub Automated Its Own Code Review Bottlenecks
Transcript
- Lucas: So we have been looking at how various tech giants are handling technical debt and scaling issues, but today I want to zoom in on a very specific, very painful bottleneck that plagues almost every mid-size to large engineering org: the code review queue. Luna: Right, because nothing kills momentum like waiting three days for someone senior enough to approve your merge request. Lucas: Exactly. And GitHub actually tackled this head-on last year. They didn't just hire more senior engineers to sit in review meetings. They built an internal AI system designed to catch stylistic errors, lint violations, and even some basic security patterns before a human developer ever clicked 'approve'. The result was a forty percent drop in average review time within six months. Luna: That is significant, but I have to ask: did the engineers trust it? Because if you automate the filter, you risk letting bad code slip through or blocking good code unnecessarily. Lucas: That was the exact fear. The data showed that initially, when they rolled out the beta version, about fifteen percent of the automated flags were false positives. That noise eroded trust quickly. But here is the twist: they didn't try to make the AI perfect. They changed the workflow so that the AI acted as a pre-screener, not a gatekeeper. Luna: So the AI does the boring work, but the human still makes the final call? Lucas: Precisely. The system would annotate the diff directly in the pull request, highlighting exactly where the style guide was violated or where a potential race condition existed. It gave the reviewer a one-page summary of issues instead of forcing them to read two thousand lines of code from scratch. It turned a thirty-minute manual scan into a five-minute verification pass. Luna: I see. It shifts the cognitive load from discovery to validation. That feels much more sustainable for senior staff. Lucas: It does, but there is a cost to building and maintaining that system. We are talking about dedicated ML engineers, API integrations, and continuous retraining of the models based on engineer feedback loops. Luna: And who pays for that? Is it worth the headcount investment for a company that size? Lucas: For GitHub, yes. Their release velocity was stagnating. Every day a pull request sat idle was a day a feature was delayed. The opportunity cost of slow deployments outweighed the salary of the team building the bot. But for smaller companies, this might be overkill. Luna: True. Most startups don't have a dedicated ML ops team. They probably just rely on junior devs reviewing each other's work. Lucas: Which leads me to think about the broader implication here. We often talk about AI replacing jobs, but in this case, AI is augmenting the review process. It is not removing the need for human judgment; it is elevating it. Luna: But doesn't that create a dependency? What happens when the model drifts or the codebase changes architecture entirely? Lucas: Drift is real. They found that after six months, the model started missing new types of vulnerabilities because the underlying framework had updated its APIs. They had to institute a weekly review of the flagged items to retrain the classifier. Luna: So it is not set it and forget it. It is a living system that requires constant attention. Lucas: Always. Just like any other part of the engineering org. Now, before we go deeper into the mechanics of how they handled that drift, I want to take a quick moment to talk about how this show stays ad-free and independent. Luna: If these deep dives into how companies actually solve hard problems are useful for your own team, consider supporting us directly. Lucas: You can find us at buy me a coffee dot com slash fexingo. It helps cover our research costs and keeps the signal to noise ratio high. Luna: Thanks for keeping this running. Back to the code review crisis. Lucas: So, back to the drift issue. When the model starts failing, how do you fix it without breaking the pipeline for everyone else? Luna: Do they roll it back? Or do they run both systems in parallel? Lucas: They ran them in shadow mode first. The new model would generate flags silently, while the old model continued to display results. They compared the outputs against a sample of manually reviewed PRs to measure accuracy gains. Luna: That is smart. You never want to surprise your engineers with a broken tool in production. Lucas: Agreed. Once they hit ninety-five percent precision, they flipped the switch. But here is the interesting part: they didn't just train the model on code. They trained it on conversation history. Luna: Conversation history? Like comments in the pull request? Lucas: Yes. The AI learned which types of comments usually lead to long debates versus quick approvals. It could then prioritize suggestions that were likely to be uncontroversial, saving the reviewer from having to argue over minor formatting choices. Luna: So it optimizes for peace as well as speed. That is actually quite clever. Lucas: It reduces friction. And in engineering culture, friction is the enemy of shipping. But let me challenge you on this: does relying on AI for style checks make junior engineers lazy? Luna: That is a fair point. If the bot catches all the typos and indentation errors, do they stop learning those basics? Lucas: They worried about that too. But the data showed the opposite. Junior engineers got faster feedback loops. They saw their mistakes highlighted immediately rather than days later. It accelerated their learning curve significantly. Luna: Because immediate correction is better than delayed critique. I can see that logic. Lucas: Exactly. It turns code review from a policing exercise into a teaching moment. The AI acts as the teacher, and the senior reviewer acts as the mentor. Luna: So the hierarchy changes slightly. It is less top-down approval and more collaborative refinement. Lucas: I think so. It flattens the org a bit. Everyone is focused on the code quality, not just the status of the approver. Luna: And what about security? Can an AI really catch a zero-day vulnerability or a complex injection attack? Lucas: Not reliably. That is why they kept strict boundaries. The AI handles syntax, style, and known pattern matches. Anything that looks structurally suspicious gets flagged for mandatory human review. No exceptions. Luna: Good. So the AI is a sieve, not a shield. Lucas: Perfect way to put it. A sieve removes the sand so you can focus on the rocks. The rocks are the architectural decisions and security implications. Luna: I like that metaphor. It clarifies the division of labor between machine and human. Lucas: It should. If you blur those lines, you get neither efficiency nor safety. You just get confused engineers. Luna: So, would you recommend this approach for a startup of fifty engineers? Lucas: Maybe not a custom-built AI. But the principle applies. Use existing linters, use static analysis tools, and enforce clear guidelines. Automate the easy stuff so humans can handle the hard stuff. Luna: Practical advice. Don't build the wheel until you need to drive off-road. Lucas: Exactly. Start with the tools available. Optimize the process. Then scale the technology. Luna: Makes sense. It is about matching the solution to the problem size. Lucas: Right. And as these models get better, maybe we will see a future where the AI handles eighty percent of the review. But until then, the human element is crucial for context and intent. Luna: Which reminds me, intent is something machines still struggle with. They see code, not purpose. Lucas: Indeed. For now, we need both. The speed of the machine and the wisdom of the engineer. Luna: A balanced approach seems to be the key takeaway here. Lucas: Let's see if that balance holds up as the next generation of coding assistants comes online. Are you ready to let your code review bots loose, or are you still holding onto the manual reins?