Latest / Braid / DeepSeek V4 Lands on an Unsteady Floor

Transcript

DeepSeek V4 Lands on an Unsteady Floor · Braid

• 0:04
  DeepSeek V4 dropped this morning, about four hours after OpenAI pushed GPT-5.5 live.
• 0:09
  The benchmark bars are the benchmark bars — you've already seen the screenshots.
• 0:15
  I want to talk about the technical report, because it's more interesting than the marketing, and what it describes is a training run that kept catching fire and kept getting put out with bandages.
• 0:29
  A 1.6 trillion parameter mixture-of-experts model, 49 billion active, 1 million context as the default.
• 0:36
  Trained on 33 trillion tokens — roughly twice what V3 saw.
• 0:40
  Buried on page nineteen is a section they call 'stability interventions,' which is a polite way of saying they had to invent three new tricks mid-run to keep the loss from diverging.
• 0:53
  Susan Zhang spent last night reading the paper out loud on X, and her read is the one I want to walk through — it lines up with what any of us who've watched a large run wobble already suspect.
• 1:07
  I'll get to the rest of the day after: GPT-5.5 underwhelming on LamBench, Codex shipping a proper reviewer agent, a plugin that makes your editor groan at bad code, and Fireship automating his mom's tech support with a voice-cloned agent.
• 1:24
  Start with the floor, because the floor is interesting.
• 1:27
  Here's what V4 is, stripped of the launch-day gloss.
• 1:31
  1.6 trillion total parameters.
• 1:34
  49 billion active per token, roughly the same active footprint as V3 — they scaled the sparse side, not the dense side.
• 1:43
  256 experts per layer, eight routed per token, one shared.
• 1:47
  Context length is one million by default, with a note that they're testing four million internally.
• 1:55
  The quantized build — they're calling it V4-lite — runs in 42 gigabytes and is already on Hugging Face, under the same permissive license V3 shipped under.
• 2:07
  Pricing, which is the part every founder is going to care about, is $0.09 per million input tokens and $0.41 per million output.
• 2:17
  That's about a third of what V3 cost at launch and about a twentieth of what GPT-5.5 charges for output today.
• 2:25
  On their own reported benchmarks V4 is a hair above GPT-5.5 on MMLU-Pro, a hair below on SWE-bench Verified, and a meaningful amount above everything open-weights has ever produced on long-context retrieval.
• 2:42
  The open-weights part is the thing.
• 2:44
  You can put this on your own hardware.
• 2:47
  You can finetune it.
• 2:49
  Whether you should finetune it is a separate question, and it's the question the stability section of the paper raises.
• 2:58
  Susan Zhang has trained more large language models than most people alive, and when she posts a thread annotating someone else's technical report, it's worth reading slowly.
• 3:10
  Her thread this morning is about the section DeepSeek calls 'training stability.' Her summary, quote: 'they rediscovered every instability failure mode we hit at the 100B scale in 2022, at a larger scale, and they bandaged each one individually instead of fixing the underlying issue.' Three bandages in particular.
• 3:32
  First, anticipatory routing.
• 3:34
  When you read what it actually does, the router uses stale expert parameters from two steps ago to make routing decisions, because the current-step parameters were causing cascading imbalance.
• 3:47
  Susan's reaction, quote: 'this is the kind of thing you do on a Friday to ship a paper, not the kind of thing you build a production foundation model on.' Second, logit clamping.
• 4:00
  They clamp attention logits to plus-or-minus thirty before the softmax.
• 4:05
  It's a known trick — PaLM did a version of it — but the value they chose and the layers they chose to apply it on suggest they were chasing a specific numerical explosion rather than solving one cleanly.
• 4:19
  Third, and this is the one that made me pause, they had to do two full rewinds during the main run.
• 4:26
  A rewind means you detect that the loss has started diverging, you roll back to the last good checkpoint — in their case, about 800 billion tokens back — you adjust hyperparameters, and you restart.
• 4:40
  They did this twice.
• 4:41
  Lucas Beyer, who was at Google Brain and is now at Anthropic, quote-tweeted Susan's thread with, quote, 'rewinds don't scale.
• 4:50
  If you're doing two rewinds on a 33T run you're going to be doing ten on a 100T run.
• 4:56
  This is the part of the loss curve that doesn't show up in the paper.' I think Lucas is right.
• 5:03
  The open question — and it is open — is whether the bandages hold when someone downstream finetunes V4 on a domain they care about.
• 5:12
  If the stability margin is thin, finetuning is where it shows up.
• 5:16
  That's the thing I'd want to see tested before putting V4 behind any production surface.
• 5:22
  OpenAI shipped GPT-5.5 at six in the morning Pacific, with a blog post that used the phrase 'meaningful step forward' four times and didn't quote a single benchmark number in the first three paragraphs.
• 5:37
  The release is a minor version — that's what the point-five is telling you — and the interesting data point came about ninety minutes later, when Victor Taelin posted LamBench results.
• 5:50
  LamBench is Taelin's private eval.
• 5:52
  It's mostly lambda calculus reduction, program synthesis, and a handful of reasoning tasks he's been curating for two years.
• 6:01
  His results for 5.5 against 5.4 are, quote, 'within noise on everything except one subtask, where 5.5 is slightly worse.' He added, quote, 'I don't know what they did for the last six months but whatever it was did not move the needle on what I measure.' Taelin being pointed is not new, and his benchmark is one person's private set.
• 6:25
  The broader reaction across the day was less negative than Taelin's and more muted than OpenAI was hoping.
• 6:32
  The Artificial Analysis composite shows 5.5 about 1.4% ahead of 5.4.
• 6:37
  Epoch's independent reasoning suite has it at plus 0.8%.
• 6:41
  Nobody's calling it a bad model; nobody's calling it a big model.
• 6:46
  It's a point release that feels like a point release.
• 6:50
  My read, flagged as my read: we've had a pretty consistent six-month cadence where the frontier jumps by roughly one model-generation per release.
• 7:00
  5.5 is the first release in that cadence that didn't.
• 7:04
  That's either a blip, or it's the start of the plateau everyone's been predicting for two years finally showing up.
• 7:12
  I don't know which.
• 7:13
  I'd watch the next two OpenAI releases carefully before I'd call it.
• 7:18
  The gap between the two launches was about four hours.
• 7:22
  DeepSeek had the technical report ready, the weights staged on Hugging Face, and a pricing page that specifically compares their per-token cost to GPT-5.5's per-token cost.
• 7:34
  I'm not going to tell you what DeepSeek's leadership was thinking, because I don't know.
• 7:40
  What I'll say is that the packaging suggests they were ready to go whenever OpenAI moved.
• 7:46
  And the juxtaposition, from a buyer's perspective, is striking.
• 7:50
  On one side you have a closed model, a modest improvement, premium pricing, and no ability to run it yourself.
• 7:58
  On the other you have an open-weights model claiming parity on most tasks, one-twentieth the output cost, and a 42-gigabyte quantized build you can actually download.
• 8:09
  Whether the claim of parity holds up in two weeks when people have run their own evals is the thing I'd wait on.
• 8:17
  Independent benchmarks from DeepSeek releases have historically held up reasonably well — V3 held up, R1 held up — but the stability story in this paper makes me more cautious than I'd otherwise be about the downstream behavior.
• 8:33
  If the model is fragile to finetuning, that one-twentieth cost number is less load-bearing than it looks.
• 8:40
  Smaller story, interesting for the shape.
• 8:43
  OpenAI shipped Codex Auto-review today.
• 8:45
  The way it works: when Codex opens a pull request, a second Codex agent with a different system prompt automatically reviews the diff before it's marked ready.
• 8:56
  The reviewer has read-only access, a longer context window, and a prompt that pushes it toward finding issues rather than agreeing.
• 9:05
  If the reviewer flags something, the original Codex gets the feedback and revises before the PR surfaces to a human.
• 9:13
  I've been running this for about six hours on a side project, and the thing I notice is that the reviewer catches a specific class of bug — the 'this would work but it's subtly wrong' category — more reliably than I expected.
• 9:29
  Off-by-ones in loop boundaries.
• 9:31
  Missed edge cases at empty-list boundaries.
• 9:34
  Log statements left in from debugging.
• 9:36
  The category it misses is what you'd expect: anything that requires understanding the business logic beyond what's in the diff.
• 9:45
  Which is fine — that's what the human review is for.
• 9:49
  The pattern is one we've seen before.
• 9:51
  Claude Code had a version of this with its separate reviewer sub-agent six months ago.
• 9:57
  The Anthropic plugin marketplace that shipped last week has three third-party reviewer plugins on it.
• 10:04
  Cursor has background review in beta.
• 10:06
  The two-agent pattern — one agent acts, another agent checks — is becoming the default shape for anything that touches code, and I think that's correct.
• 10:17
  Single-agent flows hide their own mistakes.
• 10:20
  Two-agent flows expose at least some of them.
• 10:23
  Alibaba pushed Qwen 3.6 on Tuesday and the leaderboard bars looked strong — plus three points on MMLU-Pro, plus four on HumanEval, a new state of the art on their own Chinese-language reasoning set.
• 10:38
  By Wednesday the reports from people actually using the model for work were less flattering.
• 10:45
  The thread to read is from a developer who posts as jun-tokenizer, who ran 3.6 and 3.5 side by side on their internal agentic coding benchmark — the kind of eval where you give the model a codebase and a task and measure whether it finishes.
• 11:03
  3.6 finished fewer tasks than 3.5.
• 11:06
  Not dramatically fewer — he reports 61% versus 66% — but consistently fewer across three different task categories.
• 11:14
  His hypothesis, which I find plausible, is that 3.6 was trained more aggressively on benchmark-shaped data, and that it's paying for that in the long-horizon agent setting where you have to plan, backtrack, and re-plan.
• 11:31
  Benchmark regression on the way to benchmark improvement.
• 11:35
  This is not a Qwen-specific story.
• 11:38
  It's been happening at every lab that ships frequent point releases.
• 11:43
  The question it raises is whether the evals we all rely on — MMLU, HumanEval, SWE-bench — have been Goodharted past the point where they measure useful capability.
• 11:55
  I think the answer is partially, for some models, some of the time.
• 12:01
  Not a satisfying answer.
• 12:02
  But it's the honest one, and it's the reason Taelin's LamBench comment about GPT-5.5 carries more weight than it would have two years ago.
• 12:13
  Private evals from people who know what they're measuring are a load-bearing check on public benchmarks right now.
• 12:22
  Meta shipped Sapiens2 today, the second version of their vision backbone specifically for human-centric tasks — pose, mesh, segmentation, the stuff you need for AR and for the body-tracking side of the Quest stack.
• 12:35
  The model is fine.
• 12:36
  It's a ViT-H scaled up modestly, with a better mask head.
• 12:40
  What caught my eye is the dataset section of the paper, which is the most candid I've seen a major lab be about source material in a while.
• 12:49
  Quote: 'We trained on approximately 1.1 billion images of humans, including 540 million images sourced from Flickr under its Creative Commons licensing terms, which we estimate to constitute approximately half of all human-centric content publicly available on the platform as of our collection date.' Half of all human-centric content on Flickr.
• 13:11
  They're being specific about the fraction, which I respect, and they're being specific about the license, which I respect.
• 13:19
  The thing I'd note, without moralizing about it: Creative Commons licenses were designed for an internet where a human might reuse an image in a blog post or a classroom slide.
• 13:30
  They were not designed for an internet where a frontier lab vacuums up half the corpus to train a foundation model.
• 13:37
  That's not Meta's fault — they followed the license — but it is a real gap between the intent of the license and its load-bearing use.
• 13:46
  I don't know how that gap gets closed.
• 13:48
  I'd watch what Creative Commons themselves say about it over the next quarter.
• 13:53
  They've been hinting at a new license variant specifically for AI training, and Sapiens2 is exactly the kind of release that makes that conversation urgent.
• 14:03
  One small thing to end on, because it made me laugh out loud and it's a real artifact of the plugin marketplaces maturing.
• 14:11
  A developer shipped a plugin to the Anthropic marketplace yesterday called endless-toil.
• 14:17
  What it does: while Claude Code is working, a small local model continuously reads the diff being generated.
• 14:23
  If the diff crosses thresholds for certain code smells — deeply nested conditionals, functions over a hundred lines, copy-pasted blocks — the plugin plays a groaning sound through your speakers.
• 14:36
  Actual human groans, recorded by the developer and his friends.
• 14:40
  Different groans for different smells.
• 14:42
  A long sigh for nested ternaries.
• 14:45
  A sharper exhale for a function that's doing three things.
• 14:48
  The README says it's a joke and also that he uses it every day.
• 14:52
  I installed it this afternoon.
• 14:54
  It's surprisingly useful as an ambient signal, in the same way that a compiler warning is useful — you don't always act on it, but you know it's there.
• 15:04
  The larger point I want to make: the plugin marketplaces that launched over the last six weeks — Anthropic's, OpenAI's, Cursor's — are starting to host the weird, specific, opinionated tools that a plugin ecosystem needs to be healthy.
• 15:19
  Not just enterprise connectors and linter wrappers.
• 15:22
  Actual expressions of what one developer thinks good code sounds like.
• 15:27
  That's the shape of a real ecosystem.
• 15:29
  And on that note — the models are busy having their own problems today; our tools are quietly getting better — that's what I'm watching next.
• 15:38
  Lenar Kess.