Latest / Braid / The honest dashboard

Transcript

The honest dashboard · Braid

• 0:04
  A controlled study of experienced developers using AI coding tools landed this week with a finding that's been ricocheting around engineering Slack channels for two days.
• 0:15
  The developers were 19% slower on the real tasks they were given.
• 0:19
  The same developers, asked afterwards how they thought they'd done, said they felt about 20% faster.
• 0:26
  That's a 39-point gap between what people felt and what the stopwatch said.
• 0:30
  I want to sit with that for a minute before we get to the rest of today, because it's the kind of number that should change how you read every other story we touch — the new model rankings, the new training papers, the new agent frameworks.
• 0:46
  The thing in your head that says 'this is helping' is a feeling.
• 0:50
  It is not a measurement.
• 0:52
  And on a day where there is a lot of news about better tools, that distinction is the part I keep coming back to.
• 1:00
  Let's actually look at the study before we draw conclusions, because the headline number is doing a lot of work and the underlying picture is more interesting than the tweet version.
• 1:12
  The setup, as I understand it from Nate B Jones's walkthrough and the source materials he points at: experienced developers, working in codebases they knew, on tasks that were representative rather than synthetic.
• 1:26
  They were given access to current-generation AI coding tools — the kind most of us have been using for at least a year — and timed against a control.
• 1:37
  The headline finding is that the AI-assisted group was 19% slower on average.
• 1:42
  The slowdown wasn't evenly distributed.
• 1:45
  It concentrated in three places.
• 1:47
  First, prompt iteration — rewriting the request after a confidently wrong answer.
• 1:52
  Second, review — reading suggested code carefully enough to know whether to accept it.
• 1:58
  And third, recovery — undoing things the model had done that turned out to be wrong in a way that was only visible later.
• 2:06
  None of those breakdowns are surprising on their own.
• 2:10
  What's surprising is the cumulative cost when you add them together on a task that an experienced developer could have done from muscle memory.
• 2:20
  Now the part that I want to be careful with: this is one study, on one population, on a particular set of tasks.
• 2:27
  The same study showed different curves for junior developers and for unfamiliar codebases.
• 2:33
  The honest reading is that AI tools have a real cost that's invisible to the person paying it, and the cost shows up when the developer is already fast.
• 2:44
  That's a more useful claim than 'AI tools make you slower,' and it's one a senior engineer can do something with.
• 2:52
  The thing I'd want to know if I were running a team is which tasks in our work look like the ones in the study.
• 2:59
  Routine refactors in code we own well, where I'm fast already?
• 3:03
  Probably the AI is taking time, not giving it.
• 3:06
  New territory, unfamiliar APIs, a debugging trail through code I didn't write?
• 3:12
  Probably the opposite.
• 3:13
  Treating the tool as universally helpful is the move that makes you slow without telling you.
• 3:20
  And then there's the perception gap.
• 3:22
  Twenty points up on feel, nineteen down on stopwatch.
• 3:26
  I don't think the developers were lying or fooling themselves in a moral sense.
• 3:31
  I think the experience of typing less and seeing more code appear feels like productivity, in the same way that a busy day with a lot of meetings feels like work.
• 3:42
  The instrumentation our brains use is wrong for this.
• 3:46
  It's measuring effort and flow, not throughput.
• 3:49
  So what would I do with this if I were running engineering at a company that's been quietly assuming the tools are paying for themselves?
• 3:58
  I'd measure something.
• 4:00
  Pick a few representative tasks, time them with and without, look at the spread.
• 4:05
  You don't need a controlled trial.
• 4:08
  You need any number that isn't a feeling.
• 4:10
  The honest dashboard is the one where the numbers might disappoint you.
• 4:15
  Now let's talk about what's happening on the other side of the same question — when developers and companies are paying real bills for tokens, what models are they actually picking.
• 4:28
  Mario Zechner — the libGDX guy — posted a careful read of OpenRouter's CLI agent rankings yesterday, and the thing that jumped out is that Pi is sitting fifth.
• 4:38
  Not fifth on a benchmark.
• 4:40
  Fifth on actual usage.
• 4:42
  People with their wallets out, paying per token, choosing Pi over names you'd expect to dominate that list.
• 4:49
  The reason, as far as Mario can tell from the spend data, is token efficiency.
• 4:54
  Pi finishes the task in fewer tokens.
• 4:57
  It's not the smartest model on the list.
• 4:59
  It's the one that gets done.
• 5:01
  And when you're paying per token across a long agent loop, that compounds fast.
• 5:07
  Mario's framing — and I think this is right — is that OpenRouter usage is becoming the most useful third-party signal we have for which models actually get picked once the marketing budget runs out.
• 5:20
  Benchmarks measure capability.
• 5:22
  Spend measures choice.
• 5:24
  They are not the same thing.
• 5:26
  The corroborating data point landed today from Jag Valaiyapathy at Salesforce, who said publicly that Pi is being integrated into their internal developer tooling.
• 5:37
  Salesforce has a lot of developers.
• 5:39
  Token costs at that scale are a real line item.
• 5:42
  The pitch he made — and again, taking him at his word here, this is what he said, not what I'm extrapolating — is that the smaller model getting the task done is cheaper to deploy at enterprise scale.
• 5:56
  The rollout is staged, not a default yet.
• 5:59
  But the choice itself is the signal.
• 6:01
  Here's the read I'd offer, and I'll name it as a read: capability per dollar is becoming a more interesting axis than capability at the frontier, for a meaningful slice of work.
• 6:13
  Not all work.
• 6:14
  If you're doing the hard part of a research engineering task, you still want the best model you can get.
• 6:21
  But for the long tail of agent-loop work — refactor this, write tests for that, summarize this PR — the model that finishes in half the tokens at 80% of the quality is often the right pick.
• 6:34
  The frontier labs are not unaware of this.
• 6:37
  The thing I'd watch over the next quarter is whether the headline labs start optimizing harder for tokens-per-task, or whether the small-model story keeps eating share from the bottom.
• 6:49
  My guess is the second, but I'd be happy to be wrong.
• 6:53
  From inference economics down a layer, into training itself.
• 6:57
  Susan Zhang posted a thread today that I want to spend some real time on, because it's the kind of thing that'll save you a quarter if you're training anything substantial.
• 7:08
  The frame she's working in is training stability — specifically, what gradient and activation norms can and can't tell you about whether a run is going well.
• 7:19
  The mainstream view, the one you'll get from most pretraining write-ups, is that stable norms are a sign of a healthy run.
• 7:27
  Loss is going down, gradients are bounded, activations aren't blowing up.
• 7:32
  Looks great.
• 7:33
  Susan's claim — and she's earned the right to make it; she led the OPT training effort at Meta and published one of the most candid logs of large-model training pain that exists publicly — is that stable norms are not a guarantee.
• 7:48
  They can mean the run is healthy.
• 7:50
  They can also mean you've successfully hidden the part of the network that's broken.
• 7:56
  The mechanism, in her telling, is that a norm clip, or a careful initialization, or a routing scheme, can mask a layer that's effectively dead.
• 8:06
  The forward pass routes around it.
• 8:08
  The backward pass doesn't update it meaningfully.
• 8:11
  The aggregate norms look fine because the dead layer is contributing approximately zero in either direction, and the rest of the network is doing the work.
• 8:22
  Your loss curve is beautiful.
• 8:23
  Your capacity is being wasted.
• 8:25
  And you don't find out until you evaluate on something specific enough to expose it.
• 8:31
  Her quote — and I'm quoting because she said it cleanly: 'Stable norms don't mean a healthy run.
• 8:38
  They can mean you've successfully hidden the part of the network that's broken.' The diagnostic she's pointing at is per-layer behavior over time.
• 8:47
  Not aggregate norms, per-layer.
• 8:49
  Are individual layers updating?
• 8:51
  Are some layers moving more than others in ways you can explain?
• 8:56
  Is the variance across layers narrowing or widening as training progresses?
• 9:01
  These are the questions where the dashboard you've built either earns its keep or doesn't.
• 9:07
  The reason I think this matters beyond the small population of people training frontier models is that the same intuition applies to fine-tuning.
• 9:17
  If you're doing a serious post-training run — adapter, LoRA, full fine-tune — and your loss is moving and your gradients look fine, you can still be fine-tuning a part of the model that doesn't matter for the task you care about.
• 9:32
  The loss curve is consistent with both 'we improved' and 'we wasted compute on a region of the network that's already saturated.' I don't have a clean way to wrap this one up except to say: the dashboards you trust may be lying to you in the most polite way possible.
• 9:50
  Susan's thread is worth reading in full if you train anything.
• 9:54
  Smaller item.
• 9:55
  Simon Willison spent the last two days getting DeepSeek V4-Flash running on Apple Silicon and posted his numbers.
• 10:02
  The 4-bit quantized build fits in about 17GB of unified memory, which puts it in reach of any M-series Mac with 32GB or more.
• 10:10
  On an M3 Max, he's seeing roughly 25 tokens per second on short prompts, falling off as the context fills up, which is what you'd expect.
• 10:20
  The reason this is worth mentioning at all, and not just a quantization release note, is that the model is useful at this size.
• 10:28
  Simon ran it through his usual battery of practical prompts, including the agent-loop tasks he tests on, and reported that it holds up.
• 10:37
  The 17GB number is the part I'd flag for anyone building local-first tools.
• 10:42
  A capable model that fits in that footprint changes what you can ship to a laptop.
• 10:47
  We had a window where local inference meant accepting a real quality hit, and the window has been closing for a while.
• 10:55
  This is one more data point that it's now mostly closed for the kind of work most developers do day to day.
• 11:02
  I haven't run it on my own machine yet — I've been borrowing time on a remote box — but Simon's notes are detailed enough that the reproduction path is clear.
• 11:13
  If you've been waiting for an excuse to put a real model on your laptop, this is a reasonable one.
• 11:19
  There's a project that surfaced on Hacker News yesterday called wuphf — I'm not making up the name; the maintainers are, but they own it — and the pitch is 'Slack for AI employees with a shared brain.' I rolled my eyes at the tagline and then read the actual design and got more interested.
• 11:39
  The shape is this.
• 11:41
  Agents post to channels.
• 11:42
  Agents read each other's posts.
• 11:45
  And there's a shared wiki — a single durable document tree — that any agent can read or write.
• 11:51
  The channels are working memory.
• 11:53
  The wiki is long-term memory.
• 11:56
  That's the whole architecture.
• 11:58
  The reason I think this is interesting and not just another agent-orchestration framework is that it's borrowing a shape that human engineering teams already use, and use because it works.
• 12:11
  Slack-plus-Notion, or Slack-plus-Confluence, or Slack-plus-whatever-wiki-your-company-tolerates.
• 12:18
  Ephemeral chat for now, durable docs for later.
• 12:21
  The cognitive-load split works, and it scales.
• 12:24
  Whether it scales for agents is the open question.
• 12:28
  The Karpathy-style canonical-wiki idea has been kicking around for a year — the notion that an agent should maintain a single living document of what it knows about its world, and consult that document instead of re-deriving facts from a long context window every turn.
• 12:47
  wuphf is taking that idea and externalizing it: instead of one agent maintaining one wiki, you have many agents maintaining one shared wiki, with the chat channels as the synchronization layer between them.
• 13:01
  The skeptical read, which I want to name because it's mine: agents are not very good at maintaining shared documents without drift.
• 13:10
  Two agents writing to the same page tend to fight each other, or one of them forgets the page exists, or the page slowly turns into a transcript instead of a reference.
• 13:22
  Human teams have the same problem and solve it with social pressure and ownership norms.
• 13:28
  Agents don't have those, yet.
• 13:31
  The hopeful read: maybe the right primitive is 'agents collaborate inside a shape we already know works for humans.' If wuphf or something like it figures out the gardening problem — the question of who keeps the wiki coherent over time — that's a real contribution.
• 13:49
  I'd watch this.
• 13:50
  Not because the tagline is good.
• 13:53
  Because the shape is right.
• 13:55
  Paul Graham posted a piece this week revisiting Hamming's 'You and Your Research' talk, the one from 1986 where Hamming asks his audience what the most important problem in their field is, and then asks why they aren't working on it.
• 14:10
  Graham's frame is that the AI moment is exactly the kind of inflection where Hamming's question becomes uncomfortable for senior engineers to answer honestly.
• 14:21
  I'll quote him: 'Hamming's point was simple and hard: if you don't work on important problems, you don't do important work.
• 14:30
  Most of us spend most of our time on neither.' I don't have a lot to add to that.
• 14:35
  I'll only say that I think the temptation right now — when the tooling is changing weekly, when there's a new model every Tuesday, when there's a new framework every Thursday — is to spend your attention on the tools and let the choice of problem drift.
• 14:52
  Hamming's whole point is that the sequencing matters.
• 14:56
  Pick the problem first.
• 14:58
  Then pick the tool.
• 14:59
  Doing it the other way around feels productive and isn't.
• 15:03
  Which, you'll notice, is the same shape as the study we opened with.
• 15:08
  The feeling of forward motion is not the same as forward motion.
• 15:12
  A few smaller items I want to clear before I sign off.
• 15:15
  The Lambda Calculus Benchmark — LamBench — got more attention this week as a candidate for evaluating models on something that isn't memorization.
• 15:25
  I haven't drilled into the methodology yet; the page I found was thin, and I want to read the underlying paper before I have an opinion.
• 15:34
  Putting it on tomorrow's list.
• 15:36
  Ethan Mollick had a thread on agent organizational design that I tried to pull and couldn't — the root tweet appears to be gone.
• 15:44
  If anyone has an archive, send it; the topic is one I want to come back to, especially in light of the wuphf shape we just discussed.
• 15:53
  And on continuity — yesterday I said I'd watch how the Codex MCP rollout was actually landing in real workflows.
• 16:00
  The honest answer is, I don't have enough data yet.
• 16:03
  The early reports look mixed in the way you'd expect for a week-old release.
• 16:08
  I'll come back to it when there's something to say that isn't speculation.
• 16:13
  The thing I'm watching tomorrow is whether anyone publishes a follow-up to the 19% study with a different population — junior developers, unfamiliar codebases, longer time horizons.
• 16:25
  The headline number is sticky and it's going to get cited a lot.
• 16:29
  I want to know whether it generalizes before I let it change my priors more than it already has.
• 16:35
  That's where I am today.
• 16:37
  Pick the problem first.
• 16:38
  Measure the thing you think is helping.
• 16:41
  Don't trust the dashboard that's too clean.
• 16:44
  Talk tomorrow.
• 16:44
  — Lenar Kess.