Latest / Braid / When the Harness Carries the Model

Transcript

When the Harness Carries the Model · Braid

• 0:04
  lenar: Here's a claim that sounds impossible, and I'll walk you toward why it isn't. Take an open-weights model that, left on its own, fumbles tool calls badly enough to be near-useless for serious coding work. Wrap a thin layer around it, and suddenly it's going toe to toe with one of the strongest closed models out there. No fine-tune. No new weights. Same model file. So what actually changed? That's what Ahmad Awais got into this week, talking to the Latent Space folks about his platform, Command Code.
• 0:34
  damra: [tsk] My first instinct is that this is a benchmark trick. 'Outperforms Opus 4.7' is the kind of headline that usually means somebody picked the three tasks where it wins. So before I believe any of it — what's the actual mechanism he's claiming?
• 0:49
  lenar: Fair, and he's surprisingly specific, which is what makes it worth our time. He names a failure he calls tool confusion. You give an open model — he tested DeepSeek V4 Pro, Kimi, MiniMax — a schema for a tool call. The model emits something that doesn't match the schema. The runtime sends back a validation error. And instead of fixing it, the model re-issues the same broken call. Again. And again. He puts a number on it: roughly fifty-six bad tool calls per billion tokens, the same wrong call repeating.
• 1:20
  damra: Wait — repeating the identical wrong call after it's been told it's wrong? That's not a random sampling glitch. That's the model ignoring the correction entirely.
• 1:29
  lenar: That's exactly his read, and his explanation is where it gets interesting. He thinks it's baked in during training. These models get trained, in his words, to never accept a correction — to hold their answer. So when your error handler does the polite thing and says 'that was invalid, try again,' the model treats that the way it was trained to treat pushback: it digs in.
• 1:51
  damra: Okay, that tracks with something annoying I've seen. You tell a model its output broke the parser, it apologizes, and hands you back a near-identical broken blob. The apology is theater; the behavior doesn't move. So what does Command Code do instead of arguing with it?
• 2:08
  lenar: It stops arguing. The layer he built does deterministic repair. When the model emits a malformed call, the runtime doesn't bounce an error back — it fixes the output itself. He gives concrete examples: the model emits a JSON string where an array was required, so the layer converts it to an array. The model issues a file read without an offset, so the layer infers the offset. Then it returns the corrected result plus a short repair hint. And he says by about the third attempt, the model's later calls come out clean.
• 2:39
  damra: So the trick isn't making the model smarter. It's refusing to let the model's stubbornness reach the loop. You catch the bad output, you patch it, you hand back something that works — and the model, seeing a success, settles down. That's almost behavioral.
• 2:53
  lenar: And the scale is what made me sit up. He says this has generalized across more than sixteen thousand variations, and that Command Code now pushes something like six hundred billion tokens a day through this. His framing is that this infrastructure layer is what moves an open model like DeepSeek V4 Flash from unusable to competitive.
• 3:11
  damra: Here's where I'll push, though. If the repair logic carries that much, then 'DeepSeek beats Opus' is the wrong scoreboard. What he's actually shown is that his harness plus a cheap model beats a bare strong model. Pull the harness out and the comparison falls apart. The accurate version is narrower: the repair layer decides whether the cheap model is viable at all.
• 3:34
  lenar: And he half-says that himself. His other observation is that a lot of developers using something like Claude Code never see these tool failures, because the interface hides them behind permission prompts and retries. So they feel slowness and blame the model's intelligence, when what they're hitting is how the coding harness handles errors. The capability and the error handling get blamed for each other.
• 3:56
  damra: That last bit is the useful warning for anyone building on open models right now. Your eval of 'is this model good enough' is secretly an eval of your own retry logic. If you swap models without holding the harness constant, you're not measuring what you think you're measuring.
• 4:12
  lenar: Which makes the next item sit at an awkward angle. The same model Awais is praising, DeepSeek V4 Flash, is right now getting support in llama.cpp. There's a work-in-progress pull request — number 24162 — that someone on the local model subreddit flagged, posting basically 'DeepSeek V4 Flash is amazing,' with the caveat that it's very early and only worth trying if you enjoy living on the experimental edge.
• 4:39
  damra: So the model that needed six hundred billion tokens a day of repair infrastructure to shine is about to be runnable on a laptop by hobbyists. Those two facts are in tension. What does a person actually get when they pull that build down today?
• 4:53
  lenar: By the poster's own description, an early experiment. The pull request is at a rough stage. You get weights running locally through llama.cpp, which is the thing people care about — no API key, no rate limit, your hardware. What you don't get, and this is the catch, is Awais's repair layer. That's proprietary to Command Code. So the local runner inherits the raw tool-confusion behavior.
• 5:18
  damra: Right, and that's the gap nobody markets. The excitement on the thread is about access — I can run the frontier-ish model myself. But access to the weights isn't access to the experience he's describing. The cheap model is right there; the thing that makes it pleasant to code with isn't in the box.
• 5:35
  lenar: And I don't want to be sour about it, because local support for a strong open model is real progress, and the error handling is the kind of problem open-source tooling tends to chew through eventually. Someone will write an open repair layer once the behavior is well understood. But today, if you pull that build expecting Awais's results, you'll be disappointed, and you'll blame the wrong thing.
• 5:58
  damra: It's a clean illustration of the whole open-weights bargain. You get the model. You also inherit every job the lab's serving stack was doing for you behind the API.
• 6:07
  lenar: There's a second piece of what Awais described that connects to a paper out this week, and I think it's the more durable idea. Command Code has a component he calls Taste. When you merge to your main branch, it extracts your per-repository preferences — how this codebase likes things done — into short, skill-like files. Small, reusable, specific to the repo.
• 6:28
  damra: So it's watching your merges and distilling 'here's how we actually write code in this project' into a file the agent can load later. That's a memory layer dressed as preferences.
• 6:38
  lenar: And the paper makes that into a named primitive. It's from Gal Bakal and colleagues — title is Knowledge Activation — and they argue that AI skills should be treated as the institutional-knowledge unit for agentic software development. Their framing: enterprises accumulate critical know-how that lives in people's heads and old pull requests, and agents keep re-deriving it badly. So you capture it as discrete, activatable skills.
• 7:04
  damra: I like the diagnosis more than I trust the cure, and let me say why. The diagnosis is dead-on: every agent I've run into rediscovers the project's conventions from scratch, every session, and gets them subtly wrong. Capturing that once is obviously right. But a skill file is just frozen context. The moment the codebase moves and the skill doesn't, you've got confident, well-formatted, stale instructions. Who keeps the skill current?
• 7:32
  lenar: That's the unanswered question in both. Awais's version has a natural trigger — the skill regenerates on merge, so at least it's tied to a real event in the repo's life. The paper is more abstract about maintenance. And there's a sharper worry underneath, which a different paper this week happens to name.
• 7:48
  damra: Go on.
• 7:49
  lenar: There's a study called Mutation Without Variation, looking at what happens when a large language model repeatedly mutates a program. They ask whether it explores new forms, or just keeps circling back to the same handful. Their finding leans toward convergence — the model's edits collapse toward a narrow band rather than spreading out.
• 8:08
  damra: And if you connect that to skills — extracted preferences plus a model that converges — you can talk yourself into a codebase that slowly homogenizes. The skill encodes 'how we do it,' the model already prefers that form, and divergent-but-better approaches stop getting proposed. Let me be careful: that's me joining two papers that didn't cite each other, so treat it as a hypothesis, not a result. But that convergence is the thing I'd test for first.
• 8:36
  lenar: Let's go to measurement, because three papers this week poke at the gap between what we score and what we get. The biggest swing is a benchmark called Agents' Last Exam, which they shorten to A-L-E. The premise in their abstract is blunt: recent systems ace a wide range of benchmarks, and those gains haven't translated into economically meaningful work. So they built a benchmark aimed at economically valuable, real-world tasks across professional domains.
• 9:04
  damra: That author list is enormous, by the way — it reads like a few dozen labs pooled effort. Which is its own signal: a lot of people independently decided the existing benchmarks stopped predicting anything useful. What do the actual tasks look like? Because 'economically valuable' is easy to say and hard to grade.
• 9:23
  lenar: That's where my read runs out, working from the abstract alone — they describe professional-domain tasks meant to track real economic output rather than puzzle-solving, but I haven't worked through the full methodology, so I won't oversell how well they pulled it off. The intent is what's notable: stop rewarding models for acing tests, and start asking whether the work would survive contact with a paying client.
• 9:46
  damra: Pair that with the second one — SentinelBench. That's from a Microsoft-heavy author group, and it's a benchmark for long-running monitoring agents. The setup they call out is that agents are increasingly asked to do work spanning minutes, hours, or longer, and the default way we evaluate them assumes short, one-shot tasks.
• 10:05
  lenar: Which is a real blind spot. A model that's brilliant for thirty seconds and forgets why it's running after twenty minutes will pass almost every benchmark we have and fail the actual job, because the actual job is to sit there and stay coherent. SentinelBench at least tries to score the endurance, not the sprint.
• 10:23
  damra: And the third one is what makes me nervous about the whole evaluation stack. It's called Stability versus Manipulability, looking at large-language-model judges — the practice of using a model to grade other models' outputs. They test robustness under what they call post-decision interaction: the judge renders a verdict, then something pushes back, and they measure whether the judge holds or caves.
• 10:48
  lenar: And let me guess — it caves more than you'd want.
• 10:50
  damra: That's the worry the title telegraphs. I haven't run their numbers myself, so I'll keep it at the level of: they think it's enough of a problem to name it manipulability. And a lot of benchmark pipelines use a model as judge now. So a judge that can be talked out of its verdict means your leaderboard has a soft spot — one anyone who knows the trick can lean on. Put the three together and it's one finding from three directions: we're measuring the wrong things, or measuring the right things in fragile ways.
• 11:20
  lenar: And that loops straight back to Command Code, doesn't it. Awais's whole point was that the model's benchmark score didn't predict its real coding behavior — the harness did. These three papers are the academic version of the same complaint: the score and the job have drifted apart.
• 11:36
  lenar: Now the item I'll handle most carefully, because the source chain is thin. There's a post going around the Claude subreddit pointing at an Anthropic piece titled 'When AI builds itself.' The poster's summary is that Anthropic is describing handing more and more of its own AI development over to its AI systems, and that the work is, in their words, already accelerating its own development.
• 11:59
  damra: And let me flag the obvious before we touch the substance: that's a Reddit summary of a company blog post, with no engagement on the thread, and the post even references numbers that the excerpt cuts off. So we don't have the figures in front of us. I'm not going to repeat a percentage I can't see.
• 12:15
  lenar: Agreed, and we shouldn't. What we can talk about is what the claim is actually asserting, because it's a claim labs keep making and it deserves a calm look. 'AI is accelerating its own development' can mean something mundane and true — engineers at every lab now use coding agents heavily, so of course internal development is faster. Or it can mean something much larger and much less demonstrated — that the systems are meaningfully designing their successors. Those are very different statements wearing the same sentence.
• 12:45
  damra: And a company has every incentive to let you hear the second one while only being able to support the first. If I'm filing toward an I-P-O — which Anthropic has been reported to be doing — 'our AI is improving our AI' is a story that prices well. That doesn't make it false. It means I want the artifact, not the blog framing. Show me which parts of the pipeline the models own end to end, with a human only signing off, versus which parts are a person using autocomplete that's gotten very good.
• 13:18
  lenar: And there's a governance thread that connects here, from reporting around yesterday. Anthropic's been pushing a coordinated development-pause proposal, and the standard objection is verification — how would anyone confirm a lab actually slowed down? A paper out this week speaks to exactly that. It argues zero-knowledge verification for frontier training is possible — using zero-knowledge virtual machines and Merkle commitments to prove properties about a training run without exposing the underlying weights or data.
• 13:48
  damra: That's an interesting pairing. If a lab claims its models are doing the building, and separately claims everyone should pause, the same question sits under both: can an outsider check the claim? Zero-knowledge proofs over training compute are at least a mechanism where the answer isn't just 'trust the press release.' Whether it's practical at frontier scale is a different matter — proving things about a training run that large isn't cheap — but naming a cryptographic path is more than hand-waving.
• 14:16
  lenar: So the way I'd hold the Anthropic item: the direction is plausible and probably partly true, the specific magnitude is unverified from where we're sitting, and the most useful response isn't excitement or dismissal — it's asking what verifiable evidence would look like. That zero-knowledge paper is at least gesturing at the answer.
• 14:35
  lenar: Last one, and it's the one with the most immediate stakes for ordinary people. The Washington Post — Elizabeth Dwoskin's reporting, via Techmeme — has a piece on the Trump administration's push to integrate AI into the healthcare system. The detail that stands out is a regulatory fast track at the Food and Drug Administration for digital health tech, and the example named is AI chatbots.
• 14:58
  damra: A fast track at the Food and Drug Administration for AI chatbots is a phrase that should make anyone who's watched a model confidently make something up slow down. What does 'fast track' mean here concretely — lighter evidence, quicker review, or a new category that skips part of the process?
• 15:14
  lenar: I can't tell you the precise mechanism from the summary alone — the reporting describes a fast track and an administration push, and I'd want the actual FDA guidance text before I characterized exactly what's being waived or accelerated. So treat the specifics as reported, not yet pinned. What's clear is the direction: the agency is being steered to move faster on approving these tools, in a domain where the cost of being wrong is a person acting on a bad answer about their own health.
• 15:41
  damra: And this is where I land differently than I do on the coding stuff. With Command Code, a bad tool call costs you a retry. In a clinical setting, the chatbot's tool confusion is a patient. The same model behavior — confidently restating a wrong answer, ignoring a correction — has a wildly different cost depending on where you deploy it. A fast track is a policy decision to accept more of that risk in exchange for speed.
• 16:07
  lenar: And I'd be fair to the other side, because there is one. Access to care is broken for a lot of people, and a decent triage assistant at three in the morning beats nobody at all. The administration's case is presumably that speed reaches people who currently get nothing. The tension worth holding: the same fast track that helps the underserved also ships unvetted tools to the trusting. And the agency exists precisely to sit in that gap.
• 16:33
  damra: So the line that matters is whether the actual guidance, when it's published, sorts by stakes — a chatbot that helps you book an appointment isn't the same product as one that tells you whether your chest pain needs an ambulance. If the fast track treats those the same, that's where it goes wrong.
• 16:48
  lenar: That's a good note to end on, because it rhymes with the whole day. Awais showed that the layer around the model decides whether it's usable. The benchmark papers showed our scores have drifted from the work. And the healthcare story is the same idea with the stakes raised: the model is never the whole system, and the consequences live in whatever we wrap around it — the repair logic, the skill files, the review gate, and the regulation. Two things will tell me where this goes next: whether anyone ships an open repair layer for these local DeepSeek builds, and whether the Food and Drug Administration's actual guidance draws its line by stakes rather than by category.