Latest / Braid / Full DAG test episode

Transcript

Full DAG test episode · Braid

• 0:04
  Anthropic published an engineering post yesterday called "Building agents that reach production systems with MCP," and I want to start there.
• 0:14
  It's the most concrete piece of guidance about how to wire an agent to a real system that I've read this year, and the nearest thing we have to Anthropic telling us where the platform investment is going.
• 0:29
  The post is organized around a simple frame.
• 0:32
  There are three ways to connect an agent to an external system: direct API calls, a CLI, and MCP.
• 0:40
  Direct API calls are where every team starts.
• 0:43
  They're fine when one agent talks to one service.
• 0:47
  They fall apart at the M-by-N integration problem: once you have more than a couple of agents and more than a couple of services, every pair becomes its own bespoke integration, with its own auth handling, its own tool descriptions, its own edge cases.
• 1:05
  A CLI is a thin common layer — fast, cheap, perfect for local sandboxes, but it hits hard limits on mobile, web, and anything cloud-hosted that doesn't expose a shell.
• 1:18
  MCP is the third option.
• 1:19
  It's the protocol layer: one server exposes your system's capabilities, every compatible client can reach it, and auth and discovery are standardized.
• 1:31
  Anthropic's pitch is that as production agents move into the cloud — running continuously, at scale, reaching systems that are also in the cloud and behind auth — MCP is the layer that compounds.
• 1:45
  They offer a number.
• 1:47
  MCP SDK downloads hit 300 million a month, up from 100 million at the start of the year.
• 1:53
  Take the number with the usual vendor-stat grain of salt, but the direction is consistent with what you see on the ground: every agent harness worth using now speaks MCP.
• 2:06
  The interesting part of the post isn't the pitch.
• 2:09
  It's the specific patterns.
• 2:11
  Three of them are worth holding onto.
• 2:14
  First: group tools around intent, not endpoints.
• 2:18
  This is the advice most MCP servers I've seen in the wild do not follow.
• 2:23
  The default move when you wrap an API is to mirror the OpenAPI spec one-to-one — forty endpoints, forty tools, the agent picks one at a time and wanders around your call graph.
• 2:36
  Anthropic's counter-pattern: a single create_issue_from_thread beats get_thread plus parse_messages plus create_issue plus link_attachment.
• 2:46
  Fewer, better-described tools.
• 2:49
  The agent finishes a task in two calls instead of twelve.
• 2:53
  Second: for very large surfaces — Cloudflare, AWS, Kubernetes — stop trying to enumerate the primitives.
• 3:01
  Expose just two tools, search and execute, and let the agent write a short script that you run against your API in a sandbox.
• 3:10
  Cloudflare's MCP server is the reference example: two tools, roughly 2,500 endpoints, about 1,000 tokens of schema.
• 3:18
  That's the shape of an interface designed for code to consume rather than a human menu.
• 3:25
  Third: on the client side, the post is blunt about context efficiency.
• 3:30
  Tool search — loading definitions on demand instead of dumping the whole catalog into context — cut tool-definition tokens by 85% in their testing while keeping selection accuracy high.
• 3:44
  Programmatic tool calling, where the agent processes results in code rather than returning them raw to the model, cut multi-step token usage by about 37%.
• 3:55
  Both numbers are internal, but they match the direction you feel in your own builds: the bill is almost never the model doing work.
• 4:05
  It's the model reading a tool schema for the forty-seventh time.
• 4:10
  I'd read the post even if you don't use Claude.
• 4:13
  The patterns are protocol-independent.
• 4:16
  And they're a reasonable spec for the shape of a production agent integration circa right now.
• 4:23
  On the same day, Anthropic did something else with less noise.
• 4:28
  The Claude Code SDK is now the Claude Agent SDK.
• 4:31
  The harness that powers Claude Code — the terminal-first, gather-context-take-action-verify loop — is being repositioned as a general-purpose agent harness.
• 4:43
  The companion blog post puts it this way: at Anthropic, Claude Code has become far more than a coding tool.
• 4:50
  They're using it for deep research, video creation, note-taking.
• 4:55
  It powers, in their words, almost all of their major agent loops.
• 5:00
  The rename says it directly: the distinctive thing Claude Code did — giving the model access to a real computer, with bash and files and a working terminal — is the distinctive thing that makes any of their agents work.
• 5:16
  I think this is directionally correct and worth a minute of reflection.
• 5:21
  The design principle behind Claude Code was never "write an IDE plugin" or "wrap a codebase." It was "let the model have the same tools a programmer uses — grep, tail, a shell, a filesystem — and stop trying to predigest the world for it." When you apply that principle to a research agent, the research agent gets better.
• 5:44
  Apply it to a customer-support agent, same result.
• 5:48
  Apply it to a scheduling agent, same story.
• 5:51
  The shape of the harness matters more than the task.
• 5:55
  The SDK guide formalizes this.
• 5:57
  The agent loop is named: gather context, take action, verify work, repeat.
• 6:03
  Subagents are first-class — both for parallelism and, maybe more importantly, for context isolation, so an orchestrator doesn't drown in a hundred tool results it didn't need.
• 6:15
  Three verification patterns: rules-based feedback like linters, visual feedback from screenshots, and LLM-as-judge when the rules are fuzzy.
• 6:26
  A short list.
• 6:26
  A usable list.
• 6:27
  What I'd watch for: the distance between "Claude Code with a different prompt" and "a production agent for your actual business process" is mostly about tools, not about the harness.
• 6:41
  Anthropic has been saying this for months.
• 6:44
  Between the MCP patterns post and the Agent SDK rename, they're now saying it in code.
• 6:50
  Also from the Claude team yesterday: a new slash command in Claude Code called /ultrareview.
• 6:57
  It runs a fleet of bug-hunting agents in the cloud.
• 7:00
  Findings land in the CLI or the Desktop app.
• 7:04
  Anthropic's pitch is to run it before merging critical changes — auth, data migrations, the stuff you don't want to catch on a Friday night.
• 7:14
  Pro and Max users get three free reviews through May fifth.
• 7:18
  After that, each run is billed to extra usage at $5 to $20 depending on the size of the change.
• 7:25
  A reply from Wilhelm van der Walt surfaced the pricing before Anthropic highlighted it, pulling from the docs.
• 7:33
  I want to steelman this and be skeptical of it at the same time.
• 7:37
  On the steelman side: this is the exact pattern Shopify's CTO said, on a podcast that same day, he could not find in market.
• 7:46
  He'll be on in a minute.
• 7:48
  His claim was that the best pre-merge review setup is a pro-tier model, running a critique loop with another pro-tier model, taking its time.
• 7:58
  Not a swarm of cheap agents.
• 8:00
  Few agents, expensive models, long latency, high-quality output.
• 8:05
  $5 to $20 per review is consistent with a pro-tier model doing real work.
• 8:10
  For an auth-path change, you'd pay that and consider it cheap.
• 8:14
  On the skeptical side: the replies are already showing the two failure modes you'd expect.
• 8:21
  One user reports the command fails after a few minutes but still consumes the allowance — "could we make something that reverts back the allowance if it has failed," which, yes, you should.
• 8:34
  Another asks a question I like: if /ultrareview finds bugs that Opus wrote in the first place, do you get charged for the whole bug lifecycle?
• 8:45
  That is a real question about the business model.
• 8:48
  It's also the kind of thing that's going to get asked a lot more, and it ties directly into the next story.
• 8:56
  Uber's CTO Praveen Neppalli Naga told The Information last week — it circulated widely on Reddit yesterday — that Uber's 2026 AI budget is blown.
• 9:07
  His words: "I'm back to the drawing board, because the budget I thought I would need is blown away already." Primary cause: Claude Code uptake.
• 9:17
  A few numbers from the piece worth sitting with.
• 9:21
  About 1,800 code changes a week at Uber are now written entirely by an internal AI agent with no direct human input.
• 9:29
  Roughly 95% of Uber engineers use AI tools monthly.
• 9:33
  Close to 70% of the code that gets committed is AI-generated.
• 9:38
  Uber's internal AI agent share of code changes went from under 1% to around 8% in a few months.
• 9:45
  Two things are happening at once here.
• 9:48
  First: the fixed-tier enterprise pricing everyone grew up on is gone.
• 9:53
  Anthropic — and this is covered in the India Today write-up — moved enterprise customers to a hybrid model.
• 10:01
  Lower per-seat fees, plus mandatory token commits.
• 10:04
  Even if your team doesn't burn the tokens you committed to, you pay for them.
• 10:10
  The predictability a CFO needs — headcount times dollars — is broken.
• 10:15
  You now have to forecast token consumption on a usage curve that's doubling every few months, in a workflow that's still experimental for half the engineers using it.
• 10:28
  Second: a new phenomenon the piece calls "tokenmaxxing." At companies like Meta, internal dashboards are reportedly ranking employees by token usage.
• 10:39
  That's a measurement choice that gets you lines of code measured in 2014 all over again.
• 10:45
  We know how that movie ends.
• 10:47
  Critics in the piece flag it explicitly — high token consumption does not translate to better outcomes.
• 10:55
  And we'll hear Shopify's CTO say, directly, that the anti-pattern he's watching for is exactly this.
• 11:02
  My read, tentatively: the vendor pricing story here is the important one.
• 11:08
  A CFO can plan around fixed seats.
• 11:10
  A CFO cannot plan around a hybrid model with a token commit when usage is growing 40% a quarter, and the tools themselves change pricing every other week.
• 11:22
  We had one version of this conversation yesterday, with Anthropic's Pro-tier A/B test on Claude Code.
• 11:29
  The Uber story is the enterprise version.
• 11:32
  Same underlying instability, different tier.
• 11:36
  The piece I'd most strongly recommend from the last twenty-four hours is Mikhail Parakhin's conversation with swyx on Latent Space, titled "CI/CD Breaks at AI Speed." Parakhin is Shopify's CTO.
• 11:50
  Before that he ran Microsoft's Web Services group — Bing, Edge, ads.
• 11:55
  He is not a hype-curve guy.
• 11:57
  He brought charts.
• 11:59
  Internal, real data from Shopify.
• 12:01
  A few things stood out.
• 12:03
  The first is what he calls the December 2025 phase transition.
• 12:07
  Before December, internal AI-tool adoption at Shopify was growing steadily but not dramatically.
• 12:15
  In December, something clicked.
• 12:17
  Models crossed a threshold.
• 12:19
  The curve broke open.
• 12:21
  Daily active users of at least one AI tool now approach 100% of the company.
• 12:26
  "It's hard not to do your job now without interacting deeply at least with one tool," he said.
• 12:33
  The shape of the curve matches what a lot of people felt privately in December and couldn't quite name.
• 12:41
  The second thing: CLI-based agents are outgrowing IDE tools.
• 12:45
  Cursor, Copilot — Parakhin's word was that they're "not exactly shrinking, but they're not growing as fast." Claude Code, Codex, Shopify's internal agent (called Roast, apparently) are taking off.
• 13:00
  He's describing the same pattern DHH described this week when he posted a screenshot of Neovim in the middle with Kimi and Claude on either side.
• 13:11
  The IDE was the context-delivery mechanism when humans were doing most of the reading.
• 13:17
  When the agent does most of the reading, the IDE is a UI for an audience that isn't there.
• 13:24
  The third thing: token distribution inside Shopify is skewing.
• 13:29
  The top 10% of users are growing their consumption faster than the 75th percentile, which is growing faster than the median.
• 13:38
  Parakhin's reaction was honest: "it feels not ideal." Taken to the limit, one person ends up consuming all the tokens.
• 13:47
  Swyx asked him about Jensen Huang's line — that a $200K engineer not spending $100K a year in tokens is underutilizing.
• 13:56
  Parakhin's response was careful.
• 13:58
  He said Jensen is "directionally correct for sure," but he spent the rest of the segment naming the anti-pattern.
• 14:07
  Quote: "The anti-pattern is running multiple agents, too many agents in parallel that don't communicate with each other.
• 14:16
  That's almost useless compared to just fewer agents and burns tokens very efficiently." His actual preferred pattern is a critique loop with a pro-tier model: one agent writes, another agent — ideally a different model — critiques, the first agent revises, the second critiques again.
• 14:37
  Slow.
• 14:38
  Expensive per-turn.
• 14:39
  Fewer tokens total than a swarm, much better code.
• 14:43
  The fourth thing, and I think the most important operationally: the CI/CD pipeline is the new bottleneck.
• 14:51
  More PRs, higher probability of at least one test failing, more time spent isolating the offending PR, evicting it, retesting without it, redeploying.
• 15:02
  The global mutex of merge is becoming the cost center.
• 15:06
  Parakhin is candid that he doesn't see a replacement metaphor yet.
• 15:11
  He said he's long been an opponent of microservices and now, reluctantly, he's starting to wonder if microservices make a comeback in an agent-first world because small independently-deployable units let the merge mutex decentralize.
• 15:28
  What I took away: the story inside Shopify is not "AI makes everyone faster." It's "AI makes some people a lot faster, CI/CD strains under the volume, and the human processes that sit around the codebase — PR review, merge queue, test flakiness recovery — become the bottleneck." If you run a platform team, that's your next twelve months.
• 15:53
  Smaller piece, but I can't let it go.
• 15:56
  Mitchell Hashimoto — Terraform, Ghostty, now working on Zig — posted an observation yesterday about writing Go again with an agent in the loop.
• 16:07
  The pattern he named is one every senior engineer recognizes the moment they hear it.
• 16:13
  Here's the quote.
• 16:14
  "It's far too easy to allocate, and agents, probably people too, do it too often.
• 16:20
  For example, to undo work on error, it's enticing to keep track of the work done, but that's a mistake.
• 16:28
  If an error case is rare, and they usually are, you should pessimize the error case and optimize the success case.
• 16:37
  Don't allocate unnecessarily on the happy path if it's going to succeed 99+ percent of the time.
• 16:44
  Let the error case be slower."
• 16:48
  The Zig compiler doesn't store file, line, and column info when it parses, because parsing succeeds most of the time and that metadata is a waste of memory on the happy path.
• 17:01
  When an error happens, Zig just re-parses the whole file from scratch in a slow path that does collect the metadata.
• 17:10
  It's pessimistic about the rare case instead of dragging it through the common one.
• 17:16
  Conditional branches, Hashimoto added, are basically free when they're predictable.
• 17:22
  Memory access is not free, even cached, because cache locality dominates.
• 17:28
  The reason I'm flagging this: Hashimoto is describing the failure mode that makes agent-written Go ship, pass tests, and feel subtly wrong.
• 17:38
  Agents allocate state eagerly because the training signal rewards code that looks safe in a diff.
• 17:45
  A sharp junior wouldn't do it.
• 17:48
  A seasoned engineer wouldn't do it.
• 17:50
  An RL reward that optimizes for "appears to handle failure correctly" does do it.
• 17:56
  Hashimoto himself said he doesn't expect models to reason to this pattern reliably in the short term, and that you'd have to bake it into tools and result criteria — in a skill, in a harness rule, in a review agent.
• 18:12
  That's the operational takeaway.
• 18:15
  If you're using agents to write hot-path Go, Rust, or Zig, the taste-level critique isn't going to come from the first-pass model.
• 18:25
  It has to come from a tool call, a lint rule, or a review loop that's told to care about it.
• 18:32
  This is the thing Anthropic's Agent SDK post is gesturing at when it talks about rules-based feedback.
• 18:39
  While Anthropic was publishing patterns, OpenAI was shipping surface.
• 18:44
  Workspace Agents in ChatGPT — a research preview for Business, Enterprise, Edu, and Teachers plans.
• 18:51
  The pitch: shared agents that handle complex, long-running workflows across tools and teams.
• 18:58
  Build an agent once, share it across a team, let it pull context from docs, email, chat, code, and take approved actions in Linear, Slack, and the usual connector graph.
• 19:10
  A couple of honest reactions.
• 19:12
  First, the framing of "shared agent" is new-ish.
• 19:16
  Individual chat assistants have been around.
• 19:19
  Per-org agents that can be configured once, versioned, and pointed at a team's workflows are closer to what Anthropic pitches with Managed Agents.
• 19:29
  Two companies, same quadrant, different UX.
• 19:32
  Aaron Levie retweeted it and called it "the biggest news yet in software going headless," which — Aaron is Aaron, so factor for the floor there — but the direction he's naming is real.
• 19:46
  If the shared agent becomes the primary surface for a team's knowledge work, every SaaS vendor whose moat was their UI has a problem.
• 19:55
  Second, look at the split.
• 19:57
  Workspace Agents is Business and up.
• 20:00
  Plus users don't get it.
• 20:01
  OpenAI has spent the last year learning that consumer plans and enterprise plans want different things, and they're routing capability down the tier structure accordingly.
• 20:14
  Anthropic is doing the same thing with Claude Code.
• 20:17
  This is the normal shape of a market that's deciding who the high-margin customer is.
• 20:24
  What I'd watch: the concrete examples OpenAI gave — qualifying leads, routing feedback, reviewing vendor requests, pulling reports — are exactly the workflows a $300-a-seat enterprise plan is priced for.
• 20:38
  They are not, by and large, agentic coding.
• 20:41
  The enterprise edge is filling in with business-process agents faster than it is with coding agents.
• 20:48
  Both matter.
• 20:49
  They have different failure modes and different procurement stories.
• 20:54
  Switching modalities.
• 20:56
  Google DeepMind published a paper yesterday called "Image Generators are Generalist Vision Learners," and — because everyone at DeepMind is apparently required to name things after fruit now — the model is called Vision Banana.
• 21:11
  Radu Soricut, one of the co-authors, posted the pitch.
• 21:15
  The claim: traditional computer vision tasks — segmentation, depth estimation, surface normal prediction — can be performed at or near state of the art with a single generalist image-generation model.
• 21:28
  Segmentation becomes generating a mask.
• 21:31
  Depth becomes generating a depth map.
• 21:33
  Normals become generating a normal map.
• 21:36
  The interface is image-to-image.
• 21:38
  The model is one model.
• 21:40
  A few days ago OpenAI's GPT Image 2 topped the LM Arena leaderboard by a record 242 points.
• 21:46
  The reaction on Reddit was about a photo of a woman in a chair.
• 21:50
  The reaction in research land should be about the direction: the generative image stack is absorbing what used to be distinct computer-vision pipelines.
• 22:00
  What this means if it holds: a lot of pipeline code you currently maintain — depth maps for a robotics stack, segmentation masks for a labeling tool, normals for a 3D workflow — becomes a prompt-and-diffuse call instead of a model-zoo.
• 22:16
  A reply from James Clawn, which I thought was the sharpest on the thread, asked the right question: if image generation becomes the interface for vision tasks, what catches the failure mode when a plausible generated view is just wrong enough to steer the next action, but clean enough to pass as evidence?
• 22:37
  That's the real engineering problem.
• 22:39
  A misclassified pixel in a segmentation model you at least know was a classifier.
• 22:45
  A plausible-but-wrong depth map from an image generator looks like a photo.
• 22:50
  The failure mode is subtler, and the QA strategy is different.
• 22:54
  I don't have a clean answer.
• 22:56
  I'd watch what the first robotics teams that actually put Vision Banana in a control loop do for verification.
• 23:03
  That's the experiment worth tracking.
• 23:06
  Two local-inference notes.
• 23:08
  One: Qwen 3.6 27B dense.
• 23:10
  The Unsloth quantized GGUF is 16.8 gigabytes.
• 23:14
  Runs on a MacBook.
• 23:16
  Simon Willison ran his informal pelican-on-a-bicycle SVG test and said, quote, "definitely the best pelican riding a bicycle I've had from a 16.8GB model file." Alibaba's Qwen team claims the 27B dense surpasses the older Qwen 3.5 397B MoE on major coding benchmarks.
• 23:40
  A Reddit thread in LocalLLaMA with 871 upvotes literally asked, forgive my ignorance, but how is a 27B model better than a 397B?
• 23:52
  The answer is probably some combination of better data, better post-training, and the fact that dense models can out-punch sparse MoEs at this size when the training pipeline is right — but the headline for the builder on this podcast is simple: real coding work, on a laptop, at laptop latencies.
• 24:18
  Two: OpenAI released Privacy Filter.
• 24:22
  First open model from OpenAI in 2026.
• 24:25
  Apache 2.0.
• 24:26
  It's a bidirectional token-classification adaptation of GPT-OSS, trained to detect and mask personally identifiable information.
• 24:37
  1.5 billion parameters total, 50 million active.
• 24:42
  Small enough to run in a browser.
• 24:45
  Elie Bakouch called the 128K context window at that size "quite impressive." If you have a RAG-over-enterprise-data project and PII redaction is in the critical path — and it usually is — this is the kind of low-drama, high-utility release that just saves you a week of integration work.
• 25:11
  Grab it.
• 25:11
  One more data point worth sitting with.
• 25:14
  Anthropic published the follow-up analysis to their 81,000-user Claude survey, focused on the economics.
• 25:21
  There's one finding I want to pull out because it's sharper than the usual displacement headline.
• 25:28
  The relationship between speedup and perceived job threat is U-shaped.
• 25:33
  People who report AI slowing them down — creative workers, some writers, some lawyers — express high job-displacement worry.
• 25:41
  They don't benefit personally, but they see the technology coming for the field.
• 25:47
  That's one arm of the U.
• 25:48
  The other arm is the group reporting the largest speedups.
• 25:52
  The people AI is most dramatically helping are also the most worried about being replaced.
• 25:58
  The people in the middle — moderate speedup, neither slowed down nor transformed — worry the least.
• 26:05
  Anthropic's own read is that this makes economic sense: if your tasks are shrinking rapidly in time-to-complete, there's more uncertainty about the future viability of the role.
• 26:17
  I think that's right.
• 26:19
  The second related finding is about career stage.
• 26:22
  Only 60% of early-career respondents said they personally benefit from AI.
• 26:27
  For senior workers it was 80%.
• 26:29
  The early-career story — the one everyone's been trying to read from macro hiring data — is now visible in Anthropic's own customer survey.
• 26:39
  Treat this with the usual caveats.
• 26:41
  Claude-user respondents who opted into a survey are a biased sample.
• 26:46
  But the direction matches every other signal we have.
• 26:50
  If you're hiring, the juniors you do bring on need a different ramp than the one you had.
• 26:56
  Three quick ones before I go.
• 26:58
  Ethan Mollick flagged that OpenAI released a free healthcare build of ChatGPT-5.4 for clinicians that, per OpenAI, beats specialty-matched physicians on a benchmark of real clinical tasks, even when those physicians have unlimited time and web access.
• 27:16
  Mollick — to his credit — flagged the caveat immediately.
• 27:20
  The benchmark was designed by OpenAI.
• 27:23
  It is fully open, so third parties can rerun it.
• 27:26
  Watch for the independent reruns before updating.
• 27:29
  OpenAI-designed benchmarks clearing humans is the category of claim that demands outside replication every time.
• 27:37
  Ars Technica published its newsroom AI policy yesterday.
• 27:41
  Short, plain-English, worth reading if you build retrieval or summarization for enterprise.
• 27:48
  The key line: "AI tools must not be used to generate, extract, or summarize material that is then attributed to a named source, whether as a direct quote, a paraphrase, or a characterization of someone's views." It's the constraint most enterprise summarization products are not written to enforce.
• 28:09
  Accountability, Ken Fisher writes, is non-transferable — you can't hand it to a colleague, an editor, or a tool.
• 28:17
  That's a product spec disguised as a policy.
• 28:20
  And — Sony published a paper in Nature today on an autonomous ping-pong robot called Ace.
• 28:27
  Reinforcement learning, Sony's own vision sensors, expert-level play.
• 28:32
  Ping-pong is the canonical hard-real-time robotics benchmark — sub-100ms perception-to-action loops, adversarial opponent, noisy sensing.
• 28:41
  A Nature-level peer-reviewed public baseline on it is a quiet but real milestone.
• 28:47
  The custom-sensor story is probably the part most robotics people will want to read closely.
• 28:54
  Put today together and I see three narrower things worth holding separately.
• 28:59
  One, the agent stack is filling in around MCP and a gather-act-verify loop, and Anthropic is writing the patterns down.
• 29:08
  Two, the bill for all of this is showing up in CFO spreadsheets faster than CFOs can price it, and nobody has a stable answer.
• 29:17
  Three, the people inside companies that are actually shipping with agents — Shopify, Uber — are converging on the same taste: fewer agents in a critique loop, pro-tier models, long latencies, rigorous PR review.
• 29:32
  Not swarms.
• 29:33
  Not tokenmaxxing.
• 29:34
  I'd rather ship with Parakhin's pattern than Jensen's number.
• 29:38
  That's what I'm watching next.
• 29:40
  — Lenar Kess.