Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / The CTO Decision Tree for AI Generated Code
Transcript
- Lucas: There’s a question I hear from engineering leaders almost every week now: should we let ai generated code into production, and if so, how? Luna: It’s the tension of the moment. Every CTO I talk to is using AI assistants internally, but shipping that code to customers feels like a different level of trust. Lucas: Exactly. I was talking to the CTO of a major payments company last month — they process billions in transactions — and they’ve developed a five-question filter before any ai written code reaches a production server. Luna: A decision tree specifically for ai generated code. I love that. What’s the first question? Lucas: First: do you have a human reviewer assigned who knows the codebase better than the AI? If the answer is no, it’s a hard stop. The logic is simple — if your reviewer is learning the codebase from the AI, they’re not qualified to catch the subtle mistakes the AI might make. Luna: That’s a good gate. Because the AI can produce code that looks perfect on the surface but has an off-by-one error or a security vulnerability that only someone with deep context would spot. Lucas: Right. The second question is: does the code have automated tests with at least 80 percent coverage on the module being modified? They’re not asking for 100 percent, but they want enough safety net that if the AI introduces a regression, the test suite catches it. Luna: Eighty percent feels like a floor. I know some teams aim for higher, but for AI code, I can see wanting a stronger net. Lucas: The third question is about blast radius. They ask: if this code fails, how many users or transactions are affected? If the blast radius is larger than a certain threshold — for them, it’s one percent of daily active users — the AI code must go through a canary deployment that runs for at least 24 hours before general rollout. Luna: So they’re not banning AI code in critical paths, but they’re forcing extra validation. That feels pragmatic. Lucas: Exactly. The fourth question is about explainability. They require the ai generated code to include a comment explaining the logic — not a generic 'generated by AI' tag, but a natural language explanation of what the code does and why. That’s partly for the reviewer, partly for future developers. Luna: That’s interesting. Because ai generated code can be more cryptic than human-written code. It might use an obscure library function purely because it’s statistically likely. Lucas: Yeah, the CTO told me they’ve seen cases where the AI used a deprecated API call that still compiled but had subtle side effects. The comment requirement forces the author — or the reviewer — to understand the code well enough to write that explanation. Luna: And the fifth question? Lucas: Fifth: has the team responsible for the code done a post-hoc analysis of the AI’s output? They track what percentage of AI suggestions are accepted, rejected, or modified, and they feed that back into their prompt engineering. It’s a continuous improvement loop. Luna: So the decision tree isn’t static. They’re learning from the AI’s behavior and tightening or loosening the gates over time. Lucas: Exactly. And that’s the part I find most fascinating — the framework itself evolves. The CTO said they started with four questions, then added the explainability requirement after an incident where the AI wrote a perfectly valid but unmaintainable piece of code. Luna: I think a lot of teams are going to adopt something like this. The real challenge is not whether to use AI, but how to build the organizational muscle to integrate it safely. Lucas: If today’s episode was useful for thinking through your own team’s approach, it’s worth knowing that these conversations stay ad-free because of listener support. You can help keep that going at buy me a coffee dot com slash fexingo. Luna: Yeah, it’s a small way to ensure episodes like this stay free and focused on substance. And we really appreciate it. Lucas: Now back to the framework — the five-question filter I described works well for the payments company, but it’s not one-size-fits-all. For example, a startup moving fast might have a different threshold for blast radius, maybe 5 percent instead of 1 percent. Luna: Right, because the cost of failure is different. A social media app might tolerate a brief outage better than a payment processor would. Lucas: The key takeaway for me is that the framework makes the implicit explicit. Instead of a blanket 'no AI in production' policy or an ungoverned free-for-all, you have a decision tree you can communicate to the whole engineering org. Luna: It also forces the team to think about code quality metrics they might have overlooked before. Like, do you really know your module-level test coverage? Do you have a clear definition of blast radius? Lucas: Absolutely. And that leads to the bigger picture: as AI models improve, the decision tree will change. The payments CTO told me they expect within two years to trust AI code at the same level as a junior engineer, but never at the level of a senior engineer. Luna: So the human in the loop remains essential. It’s not about replacing engineers, but about elevating their role to oversight. Lucas: That’s the vision. And it’s already happening — at companies large and small, engineering leaders are building these frameworks. The question is, are you ready with yours? Luna: Let’s hope so. Thanks, Lucas. Lucas: Thanks, Luna. Talk next time.