Latest / MBA Training AI / DeepMind AI code review: write prompts with a one-sentence check

Transcript

DeepMind AI code review: write prompts with a one-sentence check · MBA Training AI

• 0:05
  Host: This is MBA Training. On the table: 88 percent of DeepMind's AI code changes still need human review, so write prompts that can be checked.
• 0:15
  Host: Picture a Michelin kitchen where the robot cooks three plates out of four, and the head chef still tastes every single one at the pass before it goes out.
• 0:24
  Expert: That's Google DeepMind, more or less. And the tasting is the part everyone skips when they quote the headline.
• 0:31
  Host: The headline being "AI writes most of Google's code."
• 0:35
  Expert: Roughly three quarters of production code, yes. A DeepMind research scientist, Haydn Belfield, published the internal numbers this week. People read that line and stop. Wrong place to stop.
• 0:48
  Host: Fine. I'm keeping reading. What's under it?
• 0:51
  Expert: Only about one code change in eight is fully handed to the AI. The other seven need human oversight. And even that lucky one gets a human look before it ships.
• 1:02
  Host: Wait, what? So the robot cooks, but it basically never gets the restaurant to itself.
• 1:08
  Expert: Never. And that's what I want to tear apart today, because this little report is the best prompt-engineering document of the year, and it doesn't contain a single magic phrase.
• 1:19
  Host: Oh, come on, no "you are a world-class senior engineer"?
• 1:24
  Expert: Not one. Which tells you how much that line was ever worth.
• 1:28
  Host: Okay, teardown. Piece by piece. Start with the part that works.
• 1:32
  Expert: What works is where the automation lands. They surveyed senior DeepMind researchers, and delegation runs highest on well-defined, verifiable tasks. Verifiable meaning somebody can check the answer cheaply, like a test that passes or fails. It runs lowest on open-ended scientific ideation, the "come up with a new idea" stuff.
• 1:56
  Host: Which is exactly the stuff people want to hand off.
• 1:59
  Expert: Exactly the stuff. Everyone wants the model to write the strategy. The lab with one of the strongest models on earth lets it fix the plumbing.
• 2:08
  Host: Quick aside, since we're on that model. The Neuron reported Google opened Gemini 4 Argon to paid developers through the API, the plug-in point where apps talk to the model, around the thirteenth or fourteenth. Pro and Ultra subscribers got it in the Gemini app too.
• 2:25
  Expert: With an introductory price that doubles once November ends.
• 2:28
  Host: Doubles. That's a "teaser rate" move. Very mortgage-broker of them.
• 2:33
  Expert: Which is relevant, because a sloppy prompt that needs four rewrites costs you four times. Cheap tokens make bad habits feel free. In December they won't be.
• 2:46
  Host: Back to the teardown. What's the bit that doesn't hold up?
• 2:49
  Expert: The idea that the model is the thing producing reliability. It isn't. Across their internal agent runs, an agent being the model working through multi-step tasks on its own, success without human input dropped sharply as tasks got more complex. The reliability comes from scoping and review. Belfield says outright they've seen no measurable acceleration of AI research driven by AI itself.
• 3:17
  Host: Hang on, say that again.
• 3:19
  Expert: No measurable acceleration. Their capability index, a score they built across a lot of models and benchmarks, grows at a steady pace. Argon sits slightly above trend, and they credit more compute and organisational investment. Money and people, basically.
• 3:37
  Host: Money and people. The two oldest productivity tricks.
• 3:41
  Expert: Before prompt engineering was cool.
• 3:43
  Host: Uncomfortable question. Isn't this Google lowballing itself so regulators relax?
• 3:49
  Expert: Could be motive in there, I can't see inside their heads. They're internal figures, not an audit. But Bloomberg reported some Google employees saying Argon scores well on benchmarks and does less well on real work, front-end design for one. Google called that inaccurate.
• 4:07
  Host: So staff and company disagree.
• 4:09
  Expert: And that disagreement proves my point. A benchmark is a task somebody wrote a check for. Front-end design, "make it look good", nobody wrote the check. The model's great until the finish line goes blurry.
• 4:23
  Host: Which brings us to the rule you keep waving around.
• 4:26
  Expert: Write the check before you write the prompt. If the check fits in one sentence, delegate. If it needs a paragraph, split the task. If you can't write it at all, the model is a sparring partner, not a worker.
• 4:40
  Host: Give me a one-sentence check.
• 4:42
  Expert: "Every figure in the summary matches the source table." Or "The email is under a hundred and fifty words and names the deadline." You can check those in seconds.
• 4:52
  Host: And the paragraph one?
• 4:54
  Expert: "A good quarterly memo is accurate, persuasive, on-brand, short, and flags risks." That's five jobs in a trench coat.
• 5:03
  Host: Five jobs in a trench coat, trying to get into the cinema at the child price.
• 5:09
  Expert: And the model will do two of them well and quietly fumble the others. Split it.
• 5:14
  Host: Here's where I push back. Your rule sounds like it shoves all the interesting work into "sparring partner." That feels like a cop-out. Models write decent strategy drafts now.
• 5:25
  Expert: Drafts, sure. But you're the check. If you can't say what done looks like, the model can't either, it just sounds confident.
• 5:34
  Host: People can't say what done looks like for half their own strategy decks, and they still ship them.
• 5:40
  Expert: Yes, and those decks are bad.
• 5:42
  Host: Some are bad. Some are good because someone argued with a draft. I'm not convinced the line is as clean as you think.
• 5:50
  Expert: It's not clean. It's useful. Different thing.
• 5:54
  Host: We'll leave that bleeding. Next piece of the artefact: how Google tells people to prompt Argon. Vendor guidance about its own product, so salt.
• 6:04
  Expert: Salt, but it's unusually specific. Tip one: before starting, have the model confirm it has the tools and docs and understands your intent. In their coding environment, Antigravity, there's a command called "grill-me" where the model interviews you with clarifying questions before writing a line.
• 6:24
  Host: Grill-me. Someone at Google enjoyed naming that.
• 6:28
  Expert: Somebody got a promotion. But it works because ambiguity in the brief turns into errors in the output. A minute of questions catches the bad assumption you'd otherwise find on page four.
• 6:40
  Host: And they say let it run, but pause before irreversible steps.
• 6:45
  Expert: Deleting things, sending things, paying for things. The chef lets the robot chop. Not plate the wedding cake unsupervised.
• 6:54
  Host: There's a retrospective trick too.
• 6:56
  Expert: My favourite. At the end of a session you ask the model to look back at where things stalled and propose updates to your AGENTS file, a standing instruction document the agent reads every time. Translate that to normal life: after a bad run, edit your saved instructions. People who do that get better every month. Everyone else starts from zero every Monday.
• 7:21
  Host: Now the folklore. Tear down the bit everyone still does.
• 7:25
  Expert: Wharton's Generative AI Labs, independent academics, tested it. Threatening the model or promising a tip: no meaningful effect on hard benchmarks.
• 7:36
  Host: So I can stop telling my laptop I'll unplug its family.
• 7:40
  Expert: Please do. It's not scared. It's embarrassed for you.
• 7:44
  Host: And "think step by step"? Chain-of-thought, where you ask the model to show its reasoning?
• 7:50
  Expert: Modest gains on older non-reasoning models, with answers getting more erratic. On reasoning models, the ones that already think before answering, marginal gains and a lot more waiting. You're asking a chess player to please consider his moves.
• 8:06
  Host: What actually rattled you in that research?
• 8:09
  Expert: The swing. Small wording changes moved individual answers by up to about thirty-six points on one test. So when someone tweaks a prompt, tries it once and declares victory...
• 8:21
  Host: ...they've basically flipped a coin and called it a methodology.
• 8:25
  Expert: Exactly. You need a little test set. Twenty real past inputs, old prompt versus new, before you pick a winner.
• 8:33
  Host: Honest moment. A lot of this "humans review everything" turns reviewers into the bottleneck. That lands a bit heavy for junior people whose job used to be writing the code.
• 8:45
  Expert: Yeah. It lands heavy for me too. Reviewing well is a skill nobody trained them for, and it's suddenly the whole job. Teams should teach it on purpose.
• 8:55
  Host: Okay. Practical stuff that's not in the code world. My memo has no compiler.
• 9:01
  Expert: Right, code comes with tests, a forecast doesn't. Checking costs you more, so the check-first rule matters more for you, not less. Two cheap moves. First, show, don't describe. For tagging or sorting, a handful of labelled examples, shuffled so the model doesn't just copy the last one it saw. That's called few-shot prompting, teaching by example.
• 9:28
  Host: And second?
• 9:29
  Expert: Static stuff at the top. Style guide, policy, examples first, the changing question last. Argon caches shared opening text, meaning it remembers it and charges you far less for repeats. One independent guide warns it's not automatic, you need a cache hit, but the order is what makes it possible.
• 9:50
  Host: Any company that's actually picked a measurable target?
• 9:53
  Expert: Asana, per OpenAI, who sell the models involved, so treat it as vendor material and cross-check. They say Asana got its browser agent running about seventy-six times cheaper and faster by having the model run the experiments. The human picked the target and decided what counted as better.
• 10:13
  Host: The human tasted the plate.
• 10:15
  Expert: Always the plate.
• 10:16
  Host: Give me one thing to do tomorrow. Just one.
• 10:20
  Expert: Take the AI task you repeat most and write its check in one sentence, then paste that sentence into the prompt and tell the model to grade itself against it before handing anything over. If you can't write the sentence, stop delegating that task and start arguing with it instead.
• 10:38
  Host: And stop threatening the laptop.
• 10:40
  Expert: Especially that. It's watching you at the pass.
• 10:43
  Host: Drawn from The Decoder, Alphaxiv and Alphasignal, links in the show notes.
• 10:48
  Host: We'll stop there. The whole AI track, in order, is waiting at mba-training.com.