Latest / Braid / The Paradox of Supervision, a Four-Line Vendor Swap, and the Chart Its Authors Don't Trust

Transcript

The Paradox of Supervision, a Four-Line Vendor Swap, and the Chart Its Authors Don't Trust · Braid

• 0:04
  Yesterday a developer named Lars Faye published an essay called Agentic Coding Is a Trap.
• 0:10
  It hit Hacker News, climbed to 309 points and 208 comments overnight, and is the thing the developer-frontier was arguing about all morning.
• 0:20
  It's not a generic anti-AI rant.
• 0:22
  It's a careful argument from a working developer about a specific contradiction inside the way most teams are now wiring agents into their workflow.
• 0:33
  Faye's frame is Spec-Driven Development.
• 0:36
  The pitch around it goes something like this: you, the senior engineer, write a careful plan.
• 0:42
  You stop writing code.
• 0:44
  The agents do the implementation.
• 0:46
  You provide good taste, you review the outputs, you steer.
• 0:50
  You become an orchestrator.
• 0:52
  Faye says the workflow takes many shapes, but the shared move is putting more and more distance between the orchestrator and the code that's being committed under their name.
• 1:05
  He lists the trade-offs you'd expect.
• 1:08
  More complexity in the surrounding systems to handle the agent's nondeterminism.
• 1:13
  Vendor lock-in — when Claude Code went down, entire teams sat idle.
• 1:18
  Token costs that move month to month, where employee costs don't.
• 1:23
  And then the part the essay actually turns on, which is skill atrophy.
• 1:28
  Not as a vibe, but as something his sources keep pinning to data.
• 1:32
  Here's the line Faye pulls from a recent Anthropic study.
• 1:36
  He calls it the paradox of supervision.
• 1:39
  Anthropic's own researchers wrote that — and I'm quoting — effectively using Claude requires supervision, and supervising Claude requires the very coding skills that may atrophy from AI overuse.
• 1:53
  The model provider is telling you, in a published study, that the workflow they're selling has a structural problem: the skill it asks you to bring to the table is the skill it's eroding while you use it.
• 2:08
  Faye stacks this with a second Anthropic finding — a 47 percent drop-off in debugging skills among aggressive AI users — and a quote from Sandor Nyako, a director of software engineering at LinkedIn who oversees fifty engineers.
• 2:24
  Nyako has told his team not to use coding agents for tasks that require critical thinking or problem-solving.
• 2:32
  His reasoning, in his own words: to grow skills, people need to go through hardship.
• 2:38
  They need to develop the muscle to think through problems.
• 2:42
  Faye's sharpest sentence is technical, not philosophical.
• 2:46
  He writes: you cannot replace a deterministic system with a probabilistic one and expect zero ambiguity.
• 2:54
  That's the whole problem with reading the orchestration workflow as a higher-level abstraction.
• 3:00
  A compiler is an abstraction.
• 3:03
  It is deterministic.
• 3:04
  An agent is not an abstraction in that sense — it's a probability cloud you're standing inside.
• 3:11
  The reason the cognitive load doesn't drop the way the boosters claim is that the ambiguity has to be absorbed somewhere, and so far, the place it gets absorbed is the supervisor.
• 3:24
  The Hacker News thread on the essay is worth reading because it isn't a pile-on in either direction.
• 3:30
  The top comment is a developer named mehagar who said something simple: he uses AI tools to brainstorm and sometimes generate code, but he types the final version himself, so he's less likely to forget the mechanics and the language over time.
• 3:47
  That is, in three lines, Faye's recommended workflow.
• 3:50
  A reply from a commenter going by turtleyacht has a useful research idea buried in it.
• 3:56
  Quote: I'd like to see a study of brain scans during flow, manual programming, compared to code review.
• 4:03
  If the conclusion is different parts of the brain are activated, then orchestration is a separate activity entirely.
• 4:11
  Reading code is not the same as writing code.
• 4:14
  This is a good question and it's underexplored.
• 4:17
  We've been talking about the orchestration workflow as if it's just like coding, only faster.
• 4:23
  The thing turtleyacht is pointing at is that it might not be.
• 4:27
  Reviewing a generated diff might exercise different cognition than writing the diff yourself, in the same way that reading a novel exercises different muscles than writing one.
• 4:39
  If that turns out to be true — and any of us who've shipped a real system have a working hunch that it is — then the atrophy story isn't a moral failing.
• 4:50
  It's a structural one.
• 4:51
  You can't keep the writing muscle by reading more.
• 4:55
  Faye's own counter-workflow is concrete enough to copy.
• 4:58
  He uses LLMs to help generate specs and plans while he facilitates the implementation.
• 5:04
  He writes pseudo-code into the model when he engages it, to close the distance between the request and the generated code.
• 5:12
  He never generates more than he can review in a sitting.
• 5:16
  And the rule that's the most loadbearing — he never asks an agent to implement something he's never done before or couldn't do on his own, except for purely educational reasons.
• 5:28
  That last one is the difference between a tool that extends your reach and a tool that defines it.
• 5:35
  I'd like to see more shops adopt that rule explicitly.
• 5:38
  Not as policy, just as a habit you build into the way you sit down to work.
• 5:43
  While the discourse was eating Faye's essay, an engineer at LangChain named Mason Daugherty posted a small thread that's a good companion piece.
• 5:53
  Reposted by Harrison Chase.
• 5:55
  Daugherty writes — and I'll quote it cleanly — the same model in a different harness can yield much different performance.
• 6:03
  We've seen this on a few different occasions now.
• 6:07
  We took GPT-5.2-codex from 52.8 percent to 66.5 percent on Terminal-Bench 2.0, top thirty to top five at the time of publishing, just by applying harness layer changes.
• 6:18
  Fourteen points of benchmark performance from changing the scaffolding around a frozen model.
• 6:24
  That's the kind of number that should reorganize how a team allocates engineering time.
• 6:30
  The model layer is increasingly something you rent.
• 6:34
  The harness — the tool surface, the verification gates, the way state is passed across turns, the recovery behavior when something goes wrong — that's the thing that compounds with effort.
• 6:47
  We've been talking about this on the show for a couple of weeks now.
• 6:51
  Yesterday we discussed the harness-architecture question — outside versus inside the sandbox.
• 6:57
  The week before, the move toward the harness as the durable artifact.
• 7:02
  Daugherty's thread is a clean datapoint on that thesis.
• 7:06
  If you can get fourteen points by re-engineering the loop around a model you don't control, you should be at least as interested in the loop as you are in the next model release.
• 7:18
  The obvious skeptical question here: Terminal-Bench is a benchmark, and benchmarks decay, and we'll come back to exactly that problem later in the episode.
• 7:29
  So I'd take fourteen points as a directional read, not a turning point.
• 7:33
  But the direction matters.
• 7:35
  The harness is doing more of the work than the headlines suggest.
• 7:40
  Right next to that on Hacker News today, climbing to 550 points and 232 comments, is a project called DeepClaude.
• 7:48
  The pitch is that it's the Claude Code agent loop pointed at DeepSeek V4.
• 7:54
  The whole switch fits in a four-line shell snippet that the top commenter pasted, and it's the most concrete demonstration of the harness-as-artifact thesis I've seen this week.
• 8:08
  It's: export ANTHROPIC_BASE_URL pointed at api.deepseek.com slash anthropic.
• 8:13
  Export ANTHROPIC_AUTH_TOKEN to your DeepSeek key.
• 8:17
  Export ANTHROPIC_MODEL to deepseek-v4-flash.
• 8:21
  Then exec claude.
• 8:22
  That's it.
• 8:23
  The Claude Code CLI does not change.
• 8:25
  The workflow does not change.
• 8:28
  The model behind the API surface changes, and the developer's harness — the tool calls, the prompt templates, the state machine — keeps working.
• 8:39
  This is what Faye is pointing at when he warns about lock-in, and it's also the answer to the lock-in.
• 8:47
  The reason DeepClaude works is that Claude Code's protocol surface — its tool format and its message shape — has become portable enough that DeepSeek can offer an Anthropic-compatible endpoint and the rest of the stack doesn't notice.
• 9:05
  The thread mentions OpenCode too, an open-source coding agent that decoupled cleanly from a single vendor.
• 9:13
  My read here is that vendor lock-in at the model layer is, if you want it to be, a workflow choice now, not a structural one.
• 9:22
  The harness — the part you wrote, or the part you adopted — is portable.
• 9:28
  If your team got stuck during the last Claude outage, the question to ask is not whether to use agents.
• 9:36
  It's whether your harness is the kind that takes four lines of shell to retarget.
• 9:42
  On a more practical note, an r/ClaudeAI post this morning from a developer named petburiraja is the kind of thing I'd send to a friend who's running up a Sonnet bill they don't want to talk about.
• 9:57
  They audited where their Claude usage was actually going.
• 10:01
  Quote: it was embarrassing.
• 10:03
  Classifying files.
• 10:05
  Reformatting JSON.
• 10:06
  Pulling fields out of text.
• 10:08
  Summarizing docs I was going to skim anyway.
• 10:12
  None of that needed Sonnet.
• 10:14
  All of it cost the same as the work that did.
• 10:17
  The fix they landed on is a small cheap model running as a side worker, called from Claude Code through an MCP tool.
• 10:26
  Three weeks of real usage: 217 mechanical calls offloaded, total spend on the side model was 41 cents, the same workload on Sonnet would have been roughly seven dollars.
• 10:39
  That's a sixty-times cost gap on the mechanical slice of their work.
• 10:44
  The side model is DeepSeek V4 Flash by default but, in their words, the endpoint is one config line and works with anything OpenAI-compatible — local Ollama, vLLM, or LM Studio.
• 10:58
  Pick your poison.
• 10:59
  The prompt-engineering detail in the post is the kind of thing you only learn from running this for three weeks.
• 11:07
  They wrote: the CLAUDE.md rule that actually works is negative framing.
• 11:13
  Not use DeepSeek for X, but do NOT use Claude for — and then the list.
• 11:18
  JSON formatting, field extraction, file classification, and summarization you will review anyway.
• 11:26
  Positive framing got ignored maybe 30 percent of the time.
• 11:30
  The deny list catches it.
• 11:32
  A reply from a developer going by ecompanda confirms the same pattern from their own audit: positive instructions got treated like suggestions, deny lists got treated like rules.
• 11:45
  Another commenter, leogodin217, makes the right second-order point — a chunk of the work being routed to the side model could be done by scripts and linters with no LLM at all, which is also true.
• 12:00
  But the deny-list detail generalizes beyond LLM-versus-LLM routing.
• 12:05
  If your CLAUDE.md guidance is getting ignored, the framing might be the wrong shape.
• 12:12
  Another r/ClaudeAI post worth pulling from today, by a developer who goes by WEEZIEDEEZIE, opens with a diagnosis every long-running Claude Code user will recognize.
• 12:24
  Quote: every long Claude Code session has the same hidden problem.
• 12:29
  The agent is always working from stale context.
• 12:33
  It re-reads the same twelve files across three sessions to remind itself of an interface you already showed it.
• 12:41
  It refactors getUserById without checking who calls it.
• 12:45
  It edits a config with no memory of why the previous version was that way.
• 12:51
  Their tool, called Memtrace, tries to fix this by giving the agent something a senior engineer takes for granted: a time-aware view of the codebase.
• 13:02
  Two pieces.
• 13:03
  First, an always-fresh state, where every edit triggers a 42-millisecond incremental snapshot, so the agent's memory is never one session old.
• 13:14
  After a refactor, it knows the blast radius — every caller, every test, every consumer — before you do.
• 13:21
  Second, the part that's the more interesting bet: bi-temporal storage.
• 13:27
  Every change becomes a recallable episode, and the agent can replay how a function got to its current state.
• 13:35
  What worked before, what changed when, and which commit introduced the regression.
• 13:41
  Here's the architectural decision underneath it.
• 13:45
  Quote: zero LLM inference during indexing.
• 13:48
  Tree-sitter parses your code into an AST, and the AST is the structural representation.
• 13:54
  You don't pay an LLM to re-derive what your compiler already knows.
• 13:59
  That's the right shape of bet.
• 14:02
  It's the same instinct behind the Memtrace retrieval stack — Tantivy BM25 for lexical recall, and Jina-code embeddings in HNSW for semantic.
• 14:12
  Use the cheap deterministic tool where it's right, use the expensive probabilistic tool where it's right.
• 14:20
  The top reply on the post is the necessary skeptical note.
• 14:24
  Quote: yeah, works well on sole hobby projects, falls apart on multimillion-LOC codebase with 50 commits per day by 50 developers.
• 14:34
  That's fair.
• 14:35
  We don't yet have a benchmark for how AST-backed agent memory scales when the AST is being torn up by half a hundred parallel branches.
• 14:45
  I'd like to see Memtrace tested on a real corporate monorepo before declaring this pattern won.
• 14:52
  But the bet is a good one, and the design instinct — let the compiler do the structural work, don't pay tokens to re-derive it — is the kind of thing the harness layer is going to converge on whether Memtrace specifically wins or not.
• 15:10
  The other field report from today is the one that, if you build for regulated industries, will feel like a punch in the gut.
• 15:19
  Posted to r/AI_Agents by a developer who goes by soul_eater0001 — and they say this is the fourth time they've seen this pattern in a year.
• 15:29
  A founder hires a developer who's good with Cursor.
• 15:33
  Six weeks later there's something that looks like a healthcare product — a login screen, a database, a dashboard, and a clean UI.
• 15:42
  Demo-ready.
• 15:43
  They go after their first real customer, a clinic or a regional health system, and procurement sends back a vendor questionnaire — encryption at rest, audit logs, BAA coverage, role-based access controls, and whether any PHI touches third-party infrastructure they haven't reviewed.
• 16:03
  Quote from the post: the developer didn't think about any of that.
• 16:08
  Not because they were careless.
• 16:10
  Cursor doesn't know what a BAA is.
• 16:13
  The prompts never asked for it.
• 16:15
  The rebuild, in the cases this developer has seen, costs more than building it right the first time.
• 16:22
  In one case, roughly three times the original cost.
• 16:26
  In a different one the founder had already done a soft launch and had to tell pilot users the product was going on pause while the architecture got fixed.
• 16:37
  A reply from a commenter going by Emerald-Bedrock44 puts the spread cleanly: the eight-thousand-dollar MVP becomes an eighty-thousand-dollar rewrite once legal gets involved.
• 16:49
  Most teams don't even know what questions to ask until they're already in the room with the customer's compliance officer.
• 16:58
  Be careful with the easy reading here — that AI coding is dangerous in regulated spaces.
• 17:05
  The mechanic is more specific.
• 17:07
  The post says it well — in regulated SaaS, compliance shapes the schema, the auth model, the logging strategy, and which third-party services you can even choose.
• 17:18
  It is not a layer you add later.
• 17:21
  So the failure isn't that an agent wrote the code.
• 17:24
  The failure is that the architectural decisions an agent helps you make fast are the exact decisions that need domain knowledge baked in from day one.
• 17:35
  A senior engineer who's shipped HIPAA-regulated software before would have asked about audit logs in week one.
• 17:43
  Cursor will not.
• 17:44
  The agent doesn't know what it doesn't know, and the speed advantage is borrowed from the compliance retrofit budget that arrives later.
• 17:54
  This is a place where Faye's argument lands hard.
• 17:58
  If your team's institutional memory of HIPAA, or PCI, or SOC 2 lives in a single senior engineer, and your agent workflow makes that engineer's job harder to apprentice into, you are eating your own seed corn.
• 18:13
  The thing to do, if you're shipping into regulated space with agents in the loop, is to write the compliance constraints into the spec — into the deny list, into the planning template, into whatever mechanism your harness uses to keep the agent inside a bounded box.
• 18:32
  Not at review time.
• 18:33
  Up front.
• 18:34
  Switching gears.
• 18:35
  The local inference world had a quietly meaningful day.
• 18:40
  First, llama.cpp now has multi-token prediction support in beta.
• 18:45
  The PR landed this morning.
• 18:47
  ilintar, the contributor announcing it, writes — and I'll quote — between this and the maturing tensor-parallel support, expect most performance gaps between llama.cpp and vLLM, at least when it comes to token generation speeds, to be erased.
• 19:08
  That's a meaningful claim.
• 19:10
  vLLM has been the obvious answer for serving open weights at production throughput.
• 19:17
  If llama.cpp closes that gap, the calculus for builders running their own inference stack changes — you can stay on the simpler runtime with the smaller dependency footprint and not pay a throughput penalty for it.
• 19:34
  The top comment from coder543 is worth tempering the news with.
• 19:40
  Quote: this seriously has the potential to be the biggest game changer llama.cpp has ever seen.
• 19:48
  I think MTP will make the biggest difference for dense models, maybe not so much for MoEs.
• 19:55
  Then we just need DFlash and EAGLE.
• 19:58
  Translation: speculative decoding wins for dense models more than for mixture-of-experts ones, and the next wishlist items are still ahead.
• 20:10
  So directional, not arrived.
• 20:12
  Currently tested on Qwen 3.6 27B — the post mistakenly says 3.5, and a commenter corrects it.
• 20:20
  In the same direction, a developer going by abhinand05 posted a deep technical writeup of running Qwen 3.6 35B-A3B on a five-year-old laptop.
• 20:31
  An Asus ROG Zephyrus G14 from 2020, RTX 2060 Max-Q with six gigabytes of VRAM, twenty-four gigabytes of system RAM.
• 20:41
  They're getting roughly twenty-three tokens per second on the 35B model.
• 20:47
  Ten-plus tokens per second even unplugged.
• 20:50
  The recipe is very specific — CPU mixture-of-experts offload, q8 KV cache, ngram speculative decoding, and NUMA isolation, the works — and the comments include other builders running similar configs on 2018-vintage notebooks.
• 21:09
  The point isn't this one rig.
• 21:12
  The point is that the floor for serious local inference is dropping, and the recipes for getting a 35B-class model usable on five-year-old hardware are now specific enough to copy and paste.
• 21:27
  And one small operational detail that bites.
• 21:31
  A LocalLLaMA post from jacek2023: it's time to update your Gemma 4 GGUFs.
• 21:37
  The chat template was fixed a few days ago, and the rebuild has propagated across bartowski's and unsloth's repos for the 31B, the 26B-A4B, the E4B, and the E2B variants.
• 21:51
  If your structured outputs from Gemma 4 got worse this week, the chat template is probably why.
• 21:59
  A commenter notes you can also pass the updated jinja template directly to existing weights with --chat-template-file, no re-download required.
• 22:11
  Machine Learning Street Talk published a long-form interview this morning that should change how a builder reads a benchmark — a conversation with Beth Barnes, who runs METR, and David Rein, who co-authored the HCAST and Time Horizons papers.
• 22:28
  Together they're the two people most responsible for the time-horizons graph that has eaten the AI-timelines discourse — the chart everyone screenshots when they want to argue that models can now do twelve-hour tasks, or whatever the latest number is.
• 22:44
  The through-line of the conversation is that the people who built the chart are the most cautious about how it's being read.
• 22:53
  Their concern is construct validity.
• 22:55
  They list the failure modes in their own work: data contamination, approximate retrieval, shortcut-taking, models terminating their own processes, and RL agents exploiting reward shaping through destructive loops.
• 23:10
  Their argument is that headline accuracy on a benchmark and underlying capability can come apart, and currently they often do.
• 23:18
  The sharpest case study they walk through is ARC.
• 23:21
  Francois Chollet's adversarial benchmark, V1 had eight hundred to a thousand tasks, V2 was designed to be harder.
• 23:29
  Performance on V2 crashed when LLMs first faced it, then saturated within eight months.
• 23:35
  Paraphrasing them: cheaply generated, adversarially selected benchmarks inevitably trigger regression to the mean.
• 23:42
  Labs quickly produce synthetic data targeting narrow failure modes, scores inflate, and the benchmark stops measuring what it was designed to measure.
• 23:53
  Their prescription, for what it's worth, is to shift toward diverse, long-horizon tasks held strictly out of training data.
• 24:01
  The generalization gap from a benchmark subset to the full suite should mirror the gap from the suite to deployment.
• 24:09
  If those gaps are wildly different, the benchmark isn't telling you what you think it is.
• 24:15
  The reason this matters for builders, not just AI-safety people, is that the same logic applies one floor down, where most of us live.
• 24:24
  When a vendor publishes that they hit X percent on a coding benchmark, the number you actually want is the gap between that benchmark and your codebase.
• 24:34
  Mason Daugherty's fourteen-point swing on Terminal-Bench from a harness change — that's the kind of number that should remind you that the benchmark is a measurement of the system, not the model, and the system you're going to deploy is yours, not theirs.
• 24:51
  I'd like to see this construct-validity framing show up in more model cards.
• 24:56
  Not as a disclaimer at the bottom.
• 24:59
  As a section on what was held out, what's known to be contaminated, and what the authors think the generalization gap looks like.
• 25:07
  The fact that Barnes and Rein are publicly arguing this about their own chart is, to me, a piece of intellectual honesty the field needs more of.
• 25:17
  One last item, from outside the developer-tooling fight.
• 25:21
  While we were doing that, IBM Research published MAMMAL, in Nature.
• 25:26
  It's a multimodal model that combines proteins, small molecules, and gene-expression data, and it hits state of the art on nine of eleven biological benchmarks.
• 25:37
  On a couple of them — notably antibody-antigen binding — it beats AlphaFold 3.
• 25:43
  The poster on r/singularity is careful to note that AlphaFold 3 and MAMMAL serve different purposes; they're complementary tools for drug discovery, not direct replacements.
• 25:55
  The benchmark spread is wide: drug-target interaction prediction, ligand binding affinity, gene expression response, molecular toxicity, and cross-domain generalization across different biological systems.
• 26:10
  I'm not a computational biologist, and I'd like a working biologist to tell me how much movement nine-of-eleven represents in the actual practice of drug discovery.
• 26:22
  My read is that this is meaningful, but I'd want a second source on what the bench-to-pipeline gap looks like before saying more.
• 26:31
  It's a useful counterweight to the loop the rest of today lives inside.
• 26:36
  The frontier is wider than the part we're loudest about.
• 26:40
  So today's beat, if there is one — and I'm saying this lightly because not every day has a thread — is about the developer's relationship to the loop.
• 26:48
  Faye's essay names the cost of standing too far outside it.
• 26:51
  Daugherty and DeepClaude show how portable the loop has become if you build it well.
• 26:55
  The HIPAA story shows what happens when the loop has no domain knowledge at all.
• 26:59
  And Barnes and Rein remind us that the numbers we use to argue about the loop are themselves a measurement of the system, not the underlying capability.
• 27:07
  I'll be watching three things tomorrow.
• 27:09
  Whether the Faye essay gets a response from inside one of the big agent shops — Anthropic, Cursor, Cognition — and whether it's a substantive one.
• 27:16
  Whether the llama.cpp MTP beta survives contact with real workloads.
• 27:20
  And whether anyone publishes a model card that actually states a generalization-gap estimate, the way Barnes and Rein argue they should.
• 27:27
  That's what I'm watching next.
• 27:28
  Lenar Kess.