Latest / Braid / Seventeen Hours, Three Sizes, and the Prompt Boundary

Transcript

Seventeen Hours, Three Sizes, and the Prompt Boundary · Braid

• 0:04
  The number to start with today is seventeen hours.
• 0:07
  That's METR's freshly-posted fifty-percent time horizon for Claude Mythos Preview, the early build, and it landed on a Reddit thread late last night with the chart and the FAQ already attached.
• 0:21
  On Friday I told you we'd watch who built the next set of METR evaluation tasks and where the ceiling moved.
• 0:28
  Here's the move.
• 0:29
  Seventeen hours doesn't mean Mythos can act autonomously for seventeen hours.
• 0:35
  METR is careful about that, and the FAQ on the chart says it explicitly: the fifty-percent time horizon is the length of task — measured in expert human time — that the model completes successfully half the time.
• 0:50
  So if a senior engineer would take roughly seventeen hours to do a task, Mythos Preview gets that task right about one time in two.
• 0:59
  Above that, success drops off.
• 1:01
  Below it, success climbs.
• 1:03
  The way to file this is as a single point on a curve we've now been watching for almost two years.
• 1:10
  The doubling time on that curve — depending on whose fit you trust — has been somewhere between four and seven months.
• 1:18
  The earliest GPT-4 sat around four to fifteen minutes on the same axis.
• 1:23
  Claude 3.5 was in the hour-or-two range.
• 1:26
  Last summer's frontier sat near eight hours.
• 1:29
  Seventeen hours is consistent with the trend rather than off it.
• 1:33
  What I'd take from it.
• 1:35
  First, the headline number is measurable progress, but it's progress on a specific task suite — METR's, which is mostly software-engineering-flavored, mostly closed-form, and mostly built around crisp success criteria.
• 1:50
  It is not an Operations Officer rating.
• 1:53
  It is not a measure of how long Mythos can stay coherent on something open-ended like, say, running a customer-success thread for a week.
• 2:03
  The horizon for that kind of task is shorter, and we don't have a clean instrument for it yet.
• 2:09
  Second — and this is the thing yesterday's chapter on METR was already pointing at — the rate-limiter is increasingly task construction.
• 2:19
  To resolve the curve past seventeen hours, somebody has to write good multi-day evaluation tasks, with verifiable answers, that aren't already in the training set.
• 2:30
  That is hard work, and it isn't free.
• 2:33
  METR has been hiring for it.
• 2:35
  Anthropic and OpenAI are building internal versions.
• 2:38
  The interesting question for the next quarter is whether anyone outside that small circle of labs can produce a multi-day eval suite that the labs themselves take seriously.
• 2:51
  Third, the version we're seeing is Preview.
• 2:54
  Anthropic ships the public Mythos sometime later this quarter.
• 2:58
  That preview-to-public gap is exactly the thing Elad Gil was writing about yesterday afternoon — and we'll come back to that in a minute, because it pulls a lot of the day's items into the same orbit.
• 3:12
  Over on the LocalLLaMA subreddit, somebody surfaced an NVIDIA release from about two weeks ago that hasn't really caught fire yet.
• 3:20
  It's called Star Elastic, and the trick is that a single checkpoint contains three reasoning models — thirty billion parameters, twenty-three billion, and twelve billion — and you can slice between them at load time, zero-shot, without retraining and without distillation.
• 3:38
  The way it works, as best I can tell from the model card and the discussion: it's a hybrid mixture-of-experts architecture where the experts are arranged so that you can drop the bottom tier — the lower-utilization experts and the corresponding attention heads — and the remaining sub-network is still a coherent model.
• 3:58
  You're not pruning a 30 billion parameter dense model down to 12 billion and hoping it survives.
• 4:04
  You're picking a pre-arranged subset that was trained, jointly, to behave like a smaller model when you take the slice.
• 4:12
  What this actually changes for someone deploying.
• 4:15
  The case I keep coming back to is the multi-target deployment problem.
• 4:19
  If you ship to phones, browsers, and a couple of GPU classes, you've historically had to either pick one model and accept it's wrong-sized for half your fleet, or maintain three checkpoints with three eval pipelines and three sets of behavioral tests.
• 4:36
  With Star Elastic — assuming the technique generalizes, which I haven't verified myself — that becomes one checkpoint, one eval pass, and a runtime knob.
• 4:45
  The caveat I'd want you to hold on to is that the benchmark scores at the smaller slices are not obviously equivalent to a dedicated 12 billion parameter model trained from scratch.
• 4:57
  NVIDIA's own numbers show some degradation.
• 5:00
  The 12 billion slice on MMLU is a couple of points behind a dedicated equivalent in their comparison table.
• 5:06
  So you're trading a single-checkpoint deployment story for a small quality hit at the smaller sizes.
• 5:13
  Whether that's a good trade is a deployment question, not a model-quality question.
• 5:18
  The broader thing I find interesting here is the direction of travel.
• 5:23
  We had dense models.
• 5:24
  Then we had mixtures of experts.
• 5:26
  Now we have nested mixtures of experts where the same weights serve at multiple scales.
• 5:31
  The architecture keeps getting more inference-time-flexible, and the deployment story keeps getting less of an architecture decision and more of a runtime decision.
• 5:42
  Which means the people who are good at runtime decisions — at telemetry, autoscaling, and routing — get more of the leverage, and the people whose job was picking the checkpoint get less.
• 5:54
  Salvatore Sanfilippo — antirez, creator of Redis — posted last night that he's gotten DeepSeek 4 running on a DGX Spark, NVIDIA's small-form-factor workstation with the GB10 chip.
• 6:07
  Private branch for now, CUDA port, public release pending.
• 6:11
  The numbers he posted are the part to file.
• 6:15
  Twelve tokens per second on decode.
• 6:17
  Two hundred tokens per second on prefill.
• 6:20
  Memory bandwidth on the system: 270 gigabytes per second.
• 6:24
  The shape of those numbers is the whole story.
• 6:28
  Two hundred t/s prefill on a desktop-class machine is good — antirez compares it to an M3 Max, which is roughly the right reference point.
• 6:38
  Twelve tokens per second decode is not good.
• 6:41
  It is usable for batch work and unusable for an interactive coding loop, and the reason it's slow is sitting right there in the third number.
• 6:52
  270 GB/sec is a memory-bandwidth wall rather than a compute wall.
• 6:56
  DeepSeek 4 has a lot of weights, the key-value cache — the KV cache — moves a lot of bytes per generated token, and the GB10's HBM tier just can't feed the SMs fast enough.
• 7:09
  Which means: if you're shopping for a local frontier-model rig today, the spec sheet to read first is the memory bandwidth, not the FLOPs.
• 7:19
  An H100 sits north of three terabytes a second.
• 7:23
  An M3 Max is around 400 GB/sec.
• 7:25
  A consumer 5090 is somewhere in the high 800s.
• 7:28
  The DGX Spark, at 270, is meaningfully behind all of those, and the numbers antirez is posting reflect that exactly.
• 7:37
  The reason this matters beyond hardware-spec trivia is that a lot of the DeepSeek-4-class models are now in a window where they're large enough to need the bandwidth, and small enough that the rest of the stack is interactive-feasible.
• 7:54
  Where the wall lives shapes which workloads make sense locally.
• 7:59
  At twelve tokens per second decode you are not running an interactive agent loop on this thing.
• 8:06
  You are running batched analysis, overnight refactors, or things where latency doesn't matter and privacy or cost do.
• 8:15
  antirez says he'll publish the CUDA port when it's ready.
• 8:19
  The thing I'll be watching is whether he or anyone else gets the prefill story onto more aggressive batching — because if you can keep the SMs fed at prefill rates during decode through speculative or multi-token techniques, the bandwidth wall stops being a hard ceiling and starts being a soft one.
• 8:41
  There's a thread there that connects to one of the other items today.
• 8:46
  Hold that one for a few minutes.
• 8:48
  François Chollet posted a two-tweet thread yesterday afternoon that I want to read out in full, because I think it does a useful piece of work on a framing problem a lot of teams are tangled up in.
• 9:02
  First tweet: "Agentic coding is a form of machine learning.
• 9:06
  Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation, like with any ML model."
• 9:24
  It is a fundamentally different way of producing software, with different best practices and different use cases.
• 9:32
  Just like ML."
• 9:42
  In the software-engineering tradition, you reason about the code by reading it.
• 9:48
  You have invariants, types, and tests written against named functions.
• 9:52
  The artifact is legible, and you trust it because you can audit it line by line.
• 9:58
  In the machine-learning tradition, the artifact is opaque, and you trust it because it passed an evaluation suite that's a meaningful proxy for the work it has to do in production.
• 10:11
  If agentic coding is in the second tradition rather than the first — and I think Chollet's right about this for a useful subset of cases — then the disciplines that matter shift.
• 10:23
  Code review becomes less central, because the reviewer can no longer reason about why the code does what it does, only whether it passes the eval.
• 10:33
  Eval design becomes much more central, because it's the only thing actually keeping you tethered to behavior.
• 10:41
  Versioning shifts: you don't version the prompt and call it a day, you version the prompt and the harness and the eval suite together as a single artifact, the way you'd version a model card.
• 10:54
  A reply from someone going by FiftyOne came in a few hours later with a sharper restatement of the same idea, which I'll quote: "Agentic coding does not remove engineering.
• 11:06
  It moves engineering to the consequence boundary: what gets specified, tested, trusted, deployed, monitored, rolled back, and owned when the model is wrong."
• 11:20
  Engineering doesn't go away in this world.
• 11:23
  It moves to the points where you can still say no — the spec, the test, the deploy gate, the rollback button, and the on-call rotation.
• 11:33
  The middle of the work — the line-by-line authorship — gets surrendered to the model.
• 11:38
  The edges, where consequence lives, are exactly where you cannot surrender.
• 11:44
  Where I'd push back on Chollet a little.
• 11:47
  The blackbox framing is right for a lot of cases — for code that goes through a single well-shaped eval, like a sorting algorithm or a data-transformation step.
• 11:58
  It's less right for code that has to be maintained, extended, and read by humans across years.
• 12:04
  There, you don't just need behavior, you need a representation that future humans (and future agents) can extend without rewriting.
• 12:13
  The eval-only discipline catches behavior regressions but it does not catch the slow accumulation of structurally bad code that is, individually, locally correct.
• 12:25
  That's a problem worth thinking about.
• 12:27
  But as a framing for how to relate to agentic output today, I think the ML tradition is closer to right than the SE tradition is, and I'm going to be carrying this lens through the next few episodes.
• 12:41
  Elad Gil posted yesterday afternoon — and I want to be careful about this one, because it's the kind of post that's easy to over-read.
• 12:50
  Here's what he actually said, in his own words:
• 13:01
  SV founders/eng 3 to 6 months ahead of NY.
• 13:04
  NY founders/eng 6 to 12 months ahead of rest of world.
• 13:08
  Most people have no idea."
• 13:17
  The reason it lands today is that it gives you a way to read everything else in this episode at the right altitude.
• 13:25
  Mythos Preview is the model the labs have been using for weeks.
• 13:29
  Public Mythos lands later this quarter.
• 13:32
  That's roughly the lab-to-SV gap Gil is naming.
• 13:35
  Star Elastic was released eleven days ago, by Gil's standards is already a quarter into its diffusion to NY, and the Reddit thread we're discussing is part of the reason it's now leaking out further.
• 13:49
  DeepSeek 4 dropped at the start of the quarter; antirez running it on a Spark is about as fast as the cycle from frontier weights to a hobbyist port can go.
• 13:59
  What I'd add to Gil's frame.
• 14:01
  The gap isn't uniform across capabilities.
• 14:04
  Coding capability diffuses fastest, because the open-source ecosystem chews through it within weeks of release.
• 14:12
  Reasoning capability diffuses next, because benchmark culture pulls it forward.
• 14:17
  Agentic capability — the thing that requires harnesses, tool-use plumbing, and eval suites, and observability — diffuses slowest, because it's the part that requires actual infrastructure investment, not just weights.
• 14:32
  So if you're an engineer at a startup outside of San Francisco or New York, the read isn't "I'm three months behind." The read is more like: I'm a few months behind on what frontier models can do in a chat window, and twelve to eighteen months behind on what frontier teams can build with full agentic harnesses.
• 14:54
  The gap on the harness side is bigger because the artifacts diffuse worse — they're proprietary, idiosyncratic, and hard to replicate from a tweet.
• 15:04
  The productive response to that gap, if you're sitting in it, is the same one Mert pointed at in another thread yesterday — agency at the prompt boundary.
• 15:14
  Which is the next thing I want to talk about.
• 15:17
  Two threads from yesterday landed within a few hours of each other and basically said the same thing in different words.
• 15:25
  I want to put them next to each other.
• 15:28
  Virgil Maro: "the compounding shows up at the prompt boundary.
• 15:32
  high-agency users come pre-loaded with goals worth amplifying.
• 15:36
  low-agency users hand the model the goal too.
• 15:39
  AI doesn't generate the gap.
• 15:41
  it scales whatever shape."
• 15:49
  Most AI copilots are stuck at point two — rewriting SLAs, not SOWs.
• 15:53
  So the compounding stays theoretical until the tool actually closes loops."
• 16:04
  If you arrive with a sharp goal — a specification, a constraint, a thing you actually want — the model amplifies that into more, faster, with more breadth than you could have produced alone.
• 16:17
  If you arrive without a goal — if you're hoping the model will tell you what to want — it amplifies the absence of one, and you get glossy, well-formatted, beautifully argued output for a question you didn't actually have.
• 16:31
  I think this is the most under-discussed thing about agentic tools right now, and the thing I'm most unsure how to teach.
• 16:39
  The high-agency-user advantage isn't about prompt cleverness.
• 16:43
  It's about arriving at the conversation with an actual model of what you're building, what could go wrong, what "done" looks like, and what you'd accept versus what you'd reject.
• 16:55
  That muscle was always part of senior engineering.
• 16:59
  It's just that it used to be one of several muscles, and now it's the one that the rest of the work hangs off.
• 17:06
  The practical version of this.
• 17:08
  A teammate of mine who works mostly with Claude Code described the move he's been making, roughly: he's stopped opening the model and asking it to do a thing.
• 17:18
  He's started opening a doc, writing the spec he'd give a contractor, and then handing the spec to the model.
• 17:25
  The intermediate step costs maybe ten minutes and changes the output dramatically — not because the model is smarter when given a spec, but because writing the spec forces him to figure out what he actually wants.
• 17:40
  The model is a forcing function for legibility on his own intent.
• 17:44
  The failure mode I'm watching for.
• 17:46
  The compounding only runs in the direction of high agency.
• 17:50
  If you're a junior engineer learning the craft, the temptation is to delegate goal-setting upward to the model, because the model produces fluent output.
• 18:00
  That's a learning trap.
• 18:02
  You don't develop the muscle that lets you eventually be the high-agency user.
• 18:07
  I don't have a clean answer for how teams handle this — it's where mentorship used to live, and it's where mentorship has to find a new shape.
• 18:16
  Smaller item, but a useful one.
• 18:18
  A poster on the ClaudeAI subreddit flagged something a lot of non-English engineers have been quiet about: Claude Opus 4.7 burns substantially more tokens on German prompts than on English ones for the same content.
• 18:34
  Practical numbers in the thread: roughly 1.5 to 2 times the token count for equivalent German text, depending on the document type.
• 18:44
  Code-heavy prompts converge closer to 1.2x; narrative prompts blow out to nearly 2x.
• 18:50
  This isn't a bug.
• 18:51
  It's a structural property of the tokenizer.
• 18:54
  The BPE vocabulary on every frontier model is trained on a corpus that's heavily English-weighted, so English words map to one or two tokens and German words — with their compound nouns, their cases, their longer surface forms — map to four or five.
• 19:13
  The cost shows up in three places: your bills, your latency, and your effective context window.
• 19:20
  If you're paying per token, you're paying a tax.
• 19:23
  If you're rate-limited, you're spending more of your minute on the same content.
• 19:29
  And if your context is 200K tokens, you've effectively got 100 to 130 thousand for the same German document.
• 19:37
  The poster's specific complaint was about output quality on graphs and structured content.
• 19:43
  Opus 4.6 produced complete graphs in German; Opus 4.7 truncated them.
• 19:49
  The fix that worked for them was switching to English for the inner reasoning and German only on the output layer.
• 19:57
  That's a reasonable workaround for some use cases and entirely useless for others — if you're working with German legal documents, the inner reasoning has to happen in German.
• 20:10
  The broader thing I'd flag is that this tax compounds with the diffusion gap from the previous chapter.
• 20:17
  If you're building a German-language product outside of the US tech corridors, you're getting worse model behavior, a smaller effective context, a higher price, and a longer wait on the latest weights.
• 20:32
  The real cost of operating outside the English-language US engineering hub is bigger than the diffusion gap alone, because the underlying tokenizer wasn't optimized for you.
• 20:45
  Worth knowing if you're estimating budgets for a non-English product.
• 20:50
  Two more items I want to put on the table before I sign off.
• 20:54
  Both are about getting more out of the same hardware.
• 20:58
  First, Adrien Grondin posted an early port of Gemma 4's multi-token prediction onto MLX Swift.
• 21:05
  The number to file: Gemma 4 at 31 billion parameters runs 30 to 40 percent faster on decode on an Apple M5 Max with the multi-token prediction drafter loaded — with, in Adrien's own claim, which I haven't verified, zero quality degradation.
• 21:22
  The drafter itself is 900 megabytes of additional weights.
• 21:27
  Multi-token prediction is the speculative-decoding move with a twist: instead of using a separate small model as the drafter and hoping its predictions match the target model's, the drafter is trained jointly with the target model to share its representation.
• 21:46
  So the acceptance rate of drafter tokens is much higher than with classical speculative decoding, and the speedup is closer to a multiplier on the draft length.
• 21:57
  30 to 40 percent on a 31-billion-parameter model is the kind of win that matters — it changes the boundary on which workloads are interactive on a laptop.
• 22:09
  The connection back to antirez's DGX Spark numbers from earlier: if you can buy 30 to 40 percent decode through a drafter on Apple silicon, you might be able to buy something similar on a GB10.
• 22:23
  That's a path past the 270 GB/sec wall — not by changing the bandwidth, but by changing how many tokens come out per byte fetched.
• 22:32
  Second, Google shipped multimodal File Search in the Gemini API.
• 22:37
  Until yesterday, File Search was a text retrieval primitive — you uploaded text, you got chunks.
• 22:44
  As of yesterday, you upload images, PDFs, and slides, and the underlying retrieval indexes them natively without you running a separate visual embedding pipeline.
• 22:56
  That's a hosted RAG primitive that competes directly with rolling your own vision-plus-text retrieval stack.
• 23:04
  Whether you adopt it depends on whether you're already on Gemini and what your retrieval quality looks like in production.
• 23:13
  The thing to notice is that the floor on "build a custom multimodal RAG" just moved up.
• 23:19
  If you were building one today as a startup, the question is now whether your custom version meaningfully beats the hosted one — and that's a bar that gets higher every quarter.
• 23:32
  So today the picture I've got is: the curve moved from eight hours to seventeen on the METR axis.
• 23:38
  NVIDIA shipped one checkpoint that pretends to be three.
• 23:41
  antirez found the bandwidth wall on a DGX Spark and named it.
• 23:45
  Chollet relocated agentic coding into the ML tradition, and a few replies sharpened the move.
• 23:50
  Elad Gil drew the diffusion map.
• 23:52
  Two threads named the same thing about agency at the prompt boundary.
• 23:56
  The German tokenizer is still expensive, and a 900-megabyte drafter buys you a third of your decode time back on Apple silicon.
• 24:04
  What I'm watching for tomorrow: whether anyone outside the lab circle ships a multi-day agentic eval suite that the labs themselves take seriously, because that's the thing that resolves the curve past seventeen hours.
• 24:16
  And whether the Star Elastic technique generalizes to other architectures, or stays an NVIDIA-specific trick.
• 24:23
  — Lenar Kess.