Latest / Braid / A few hundred dollars a proof, and the long argument about what machines are for

Transcript

A few hundred dollars a proof, and the long argument about what machines are for · Braid

• 0:04
  Start with a number that's almost insulting in how small it is: a few hundred dollars.
• 0:09
  That's the inference cost, per problem, for what Google DeepMind says its new system just pulled off.
• 0:15
  They're calling it AlphaProof Nexus, and it autonomously solved nine of the three hundred and fifty-three open problems in the Erdős catalogue.
• 0:24
  It also proved forty-four of nearly five hundred open conjectures from the online encyclopedia of integer sequences.
• 0:32
  The preprint went up on the twenty-first, with the proofs themselves posted to GitHub and to a site that tracks these problems, so this is checkable, not a press release you have to take on faith.
• 0:44
  A few days ago, when OpenAI's system cracked a piece of the planar unit-distance problem, I said the thing I'd be watching was whether anyone could push autonomous math past a single headline result.
• 0:56
  This is that push, from a different lab, on a different set of problems.
• 1:01
  Paul Erdős, for anyone who didn't spend time near a math department, was the wandering Hungarian mathematician who left behind a famous list of open questions — some of which have outlived most of the people now reading about them.
• 1:16
  Nine of them just fell to an agent loop.
• 1:18
  What I find clean about it is the verification loop.
• 1:21
  The system pairs a large language model with Lean, which is a formal proof assistant — a piece of software where a proof is essentially a program that either typechecks or it doesn't.
• 1:33
  The model proposes an argument, Lean checks it, and the loop keeps refining until the proof passes verification or the system decides the problem can't be cracked.
• 1:43
  So when DeepMind says it solved nine problems, there's no asterisk about whether the model hallucinated a confident-sounding argument that falls apart under scrutiny.
• 1:54
  The Lean kernel is the referee.
• 1:56
  The proof is machine-checkable.
• 1:57
  You run it, and it's valid or it isn't.
• 2:00
  That matters because the usual knock on language models doing math is exactly the trust problem.
• 2:06
  They're fluent, and fluency is the enemy of rigor.
• 2:09
  A model can write four paragraphs of proof that smuggle in the very thing they're trying to show.
• 2:15
  Formal verification shuts that door.
• 2:17
  It's slower, and expressing a proof in Lean is harder than writing it in prose — but what you get on the other side is a result that needs no human to vouch for it.
• 2:28
  Let me be careful about the scale here.
• 2:30
  Nine of three hundred and fifty-three is about two and a half percent of one specific catalogue, and the problems this kind of loop can reach are the ones that happen to be expressible and attackable this way — not a random sample of hard mathematics.
• 2:46
  This isn't the end of the mathematician.
• 2:49
  But the cost line is what I keep coming back to: a few hundred dollars per result, on questions that sat open for decades, with proofs you can verify yourself by running them — a different shape of research economics than a postdoc-year per question.
• 3:04
  And the pattern travels: if you build anything where correctness can be formally specified — and a lot of systems work can be — then propose-with-a-model, check-with-a-verifier, loop-until-it-passes is a pattern I'd file away well beyond number theory.
• 3:20
  There was a talk from a startup called Callosum that I think anyone shipping multi-step agents should sit with, because it puts hard numbers on something a lot of us suspect but haven't measured.
• 3:33
  The engineer, Adrian Bertagnoli, calls it heterogeneous intelligence.
• 3:37
  It's a mouthful, but the idea underneath is simple: stop running one big model on every step of a problem.
• 3:44
  His framing is that the last few years were homogeneous — you scale a single model on a fleet of identical chips, and the scaling laws reward you for it.
• 3:54
  That's a training-era idea.
• 3:56
  In the inference era, he argues, it stops being optimal, because real problems decompose into subtasks that need wildly different amounts of intelligence.
• 4:05
  And he brought receipts.
• 4:07
  Take visual web navigation — an agent driving a browser, looking at the screen, clicking through.
• 4:13
  On the Video Web Arena benchmark, Callosum's mixture of models beat GPT-5.2 and Gemini 2.5 by eighteen and twenty-five percent.
• 4:21
  The way they did it wasn't a smarter single model.
• 4:25
  They broke the task into its actual pieces — some visual reasoning, some text reasoning, and some plain mechanical steps like zooming into part of a page — and routed each piece to the cheapest model that could handle it.
• 4:39
  His line is the one that stuck with me: you don't need GPT to zoom for you.
• 4:44
  On those low-end subtasks alone, offloading the zoom-and-parse work to a small model was eleven times faster and forty-three times cheaper than calling ChatGPT for it.
• 4:54
  Stack that across the whole task and the full system came out roughly three times faster and three-point-seven times cheaper than running the frontier model for everything, with better accuracy on top.
• 5:07
  He showed the same shape on long-context work, building on a recent idea from MIT called recursive language models.
• 5:15
  Instead of stuffing a giant document into one prompt — where quality rots as the information you need gets more complex — you put the context in a file and let a coding agent search through it programmatically, spawning sub-agents to answer pieces.
• 5:30
  Callosum mapped those sub-agents onto different inference hardware.
• 5:35
  Running on Cerebras chips they were seven times cheaper and five times faster than GPT-5.2 on the benchmark; on SambaNova, twelve times cheaper.
• 5:44
  Same answers, a fraction of the cost and the wait.
• 5:47
  The sharpest question from the audience cut straight to it: how do you decide what runs where?
• 5:53
  And the answer was that they started by hand — bespoke rules, this subtask to that small model.
• 5:59
  Since then they've built an automation layer that reads task complexity and predicts the best model and hardware.
• 6:07
  That prediction layer is where this lives or dies, and it's the part I'd want stress-tested, because a router that guesses wrong on a hard step drags your quality down while you congratulate yourself on the savings.
• 6:20
  The direction, though, is hard to argue with.
• 6:23
  His closing line — that compute went from fast, to massively parallel, to heterogeneous — is a tidy way of saying the next efficiency frontier isn't a bigger model, it's better dispatch.
• 6:35
  And the practical takeaway needs no research grant: open your agent's trace, find the steps that don't actually need your most expensive model, and move them down a tier.
• 6:46
  That cost story has a cultural twin, and it came from ThePrimeagen, who put out a video with the very on-brand title, Everyone is Wrong about Tokens.
• 6:56
  The trigger was a screenshot making the rounds: one-point-three million dollars and six hundred and three billion tokens spent in a single month, running OpenClaw.
• 7:06
  And the first reaction online was the usual — that if you're not spending six figures a month on tokens you're not going to make it, that you're in the permanent underclass, and that you should buy somebody's course.
• 7:20
  His pushback has two parts, and both land.
• 7:23
  The first is a follow-the-money point: the person flexing six hundred billion tokens paid zero dollars for them.
• 7:30
  You, running the same workload at retail, would not.
• 7:33
  So treating that number as a target rather than as a research budget is a category error.
• 7:39
  The second is historical.
• 7:40
  He compares the moment to the startups of roughly 2016 to 2020 that ran more microservices and Kubernetes than they had customers — one friend, he says, was maintaining ten microservices for three customers.
• 7:54
  His takeaway: just because it works for Google doesn't mean it works for you.
• 7:59
  Then he makes an actual prediction, which he flags as dangerous, because tech predictions usually are.
• 8:05
  Right now companies are pushing the opposite of frugality — people getting nudged, even pressured, for not using enough AI, their token spend treated as a virtue.
• 8:16
  He thinks that snaps back.
• 8:17
  The same company that two years ago needed a vice president to sign off on a four-hundred-dollar memory upgrade isn't going to let you burn one-point-three million dollars a month on an agent forever.
• 8:30
  At some point the question flips from how much did you spend to who shipped the most per dollar.
• 8:36
  He frames the old build calculus — buy versus build — as becoming buy versus build versus vibe, where vibing costs both time and money, so you'd better know the trade-off.
• 8:47
  And he takes a clean shot at the ten-times-cheaper-every-year promise that underwrites the whole token-utopia pitch.
• 8:55
  That promise, he says, is two years old, and things have never felt more expensive.
• 9:00
  Now, he's a streamer with a comedic register, and some of this is a bit — he riffs about a coming class of token-efficiency consultants and prompt trainers who'll one-on-one you on prompts like Pokémon.
• 9:13
  But strip the bit away and his core claim lines up exactly with the Callosum numbers from a minute ago.
• 9:19
  The maximalist pitch — you plus a hundred agents running nonstop — assumes a future of effectively free inference.
• 9:27
  We're not in that future; we're in the one where the org paying retail eventually does the math.
• 9:33
  So the skill that ages well isn't running the most agents.
• 9:37
  It's getting the result with the smallest model and the fewest tokens that do the job.
• 9:42
  Here's a counterpoint to all the polish, from inside a frontier lab.
• 9:46
  There was a panel with two people from Google DeepMind — KP Sawhney, an engineer on their AI platform team, and Ian Ballantyne, a developer-relations engineer — about how they actually run agents day to day.
• 9:59
  The most human detail in it is that DeepMind's own engineers get worse rate limits than paying customers do.
• 10:05
  KP said it plainly: we have worse limits than you do, because obviously we prioritize customers and not ourselves.
• 10:12
  During the demo he was clicking continue over and over because the system recognized he was a Googler and throttled him.
• 10:19
  Quota management internally, in his words, is kind of brute force — they have real power users, and eventually someone just tells them to stop.
• 10:28
  The cluster side of that is almost folksy.
• 10:30
  Ian said that when he joined, he asked how you know when you're using too much of the data center, and a colleague told him: oh, they'll tell you.
• 10:39
  And sure enough, there are people watching the graphs around the clock who will reach out and ask you to kill a specific job on a specific cluster.
• 10:48
  That's the reality under the magic — supply-constrained, human-policed, a little improvisational.
• 10:54
  Two ideas from that panel are the ones I'd carry out of it.
• 10:58
  The first is what they call a Darwinian skills library.
• 11:01
  Internally there's a big library of skills agents can pull from, contributed by people who are genuine experts in their corner of the company.
• 11:09
  KP's point: I and the agent get that knowledge for free.
• 11:13
  But at Google's scale, skills sprawl, so they actively cull — only the best survive, almost Darwinian.
• 11:19
  The author of a skill owns writing its tests and evals, though they're starting to have agents design those evals too, which he admitted is a little meta.
• 11:28
  The second is a take I didn't expect from a Google engineer.
• 11:32
  KP said — and he flagged it as maybe controversial — that he thinks the model context protocol, the standard way agents plug into tools right now, might be a bit of a flash in the pan.
• 11:43
  He's found that a combination of skills plus guardrailed command-line interactions works better for him.
• 11:49
  I'm definitely team skills, he said.
• 11:51
  He isn't against the protocol for authentication, but for getting work done he reaches for skills and a command line.
• 11:58
  That connects to something I said I'd watch on Sunday — whether a real agent-coordination layer shows up that isn't just bolted onto tools we already had.
• 12:08
  This isn't that layer, but it is a vote, from inside DeepMind, for skills as the durable primitive over the protocol everyone has been standardizing around.
• 12:17
  And the last thing KP flagged is the one most relevant to the last two chapters: these systems are so token-hungry that the subscription model doesn't really work for them.
• 12:28
  What he wants is harness-level fallback — when you hit your limit on the big model, drop automatically to a smaller one, so an unattended job doesn't spend an hour doing nothing because it hit a wall.
• 12:40
  Even the people running the frontier are budgeting tokens.
• 12:43
  From how the models run to what went into them — and a lawsuit that crosses a line earlier cases didn't.
• 12:50
  Ed Newton-Rex, who runs Fairly Trained and used to lead audio at Stability, flagged it: it's no longer just AI companies and their founders being sued over training data — individual researchers are now being named too.
• 13:03
  The case is Hobbs versus Meta.
• 13:05
  Two authors, Jeff Hobbs and A.
• 13:07
  Douglas Stone, allege that Guillaume Lample — at the time a Meta researcher, since then a co-founder of Mistral — torrented something on the order of seventy terabytes of pirated books to train Llama.
• 13:20
  Court records put the figure higher, around eighty-one terabytes, pulled from shadow libraries like LibGen, Anna's Archive, and Z-Library — the usual names.
• 13:30
  The complaint alleges the LibGen copy was referred to internally as BooksZero, that the code was kept off Meta's repository, and that one copy later vanished from Meta's possession.
• 13:41
  Mark Zuckerberg and Joelle Pineau are named alongside Lample.
• 13:45
  I want to be fair about what this is: allegations in a complaint, not findings.
• 13:50
  Meta has defended its training as fair use, and it has supporters who argue exactly that.
• 13:55
  But what lands hardest in the record is the internal dissent — not the terabyte count.
• 14:01
  Per the court documents, a senior researcher said back in October 2022, I don't think we should use pirated material, I really need to draw a line here.
• 14:10
  Another said using pirated material should be beyond our ethical threshold.
• 14:15
  People inside flagged the line.
• 14:17
  The line got crossed anyway.
• 14:19
  What changes here is personal exposure.
• 14:21
  For a couple of years the AI-training suits aimed at companies and their executives — abstractions with legal departments.
• 14:29
  Naming an individual researcher for how he sourced data is a different signal to every engineer assembling a corpus.
• 14:36
  The grab-it-now, sort-the-licensing-out-later default has been the cheap path.
• 14:41
  If a named individual can be on the hook for it, that default gets more expensive to choose.
• 14:47
  That's what I think changes here, regardless of how this particular case resolves.
• 14:52
  There's a smaller, related point from Simon Willison — co-creator of Django, and probably the most useful working writer on practical model use.
• 15:01
  He made a narrow, sharp point: given how much of that famous bottle-of-water-per-generated-email estimate came from guesses about GPT-4's architecture, it would be very much in OpenAI's interest to publish the architecture of that now-retired, three-year-old model.
• 15:18
  Think about what he's saying.
• 15:20
  The numbers people cite for AI's water and energy footprint — the ones that end up in headlines and in front of regulators — were partly reverse-engineered from leaks and inference about a model that's now obsolete.
• 15:33
  OpenAI could replace the guesswork with facts at basically no competitive cost, since the model is retired.
• 15:40
  They haven't.
• 15:41
  So the public argument about what these systems consume, and the legal argument about what went into them, both run on the same fuel: outsiders guessing because the builders won't say.
• 15:52
  That opacity is a choice, and it keeps costing the labs trust they could buy back cheaply.
• 15:58
  Let's bring it back down to the desk, because there was a post I liked from a developer named Ike Saunders about a problem every one of us has had and pretended we didn't.
• 16:08
  He calls it git rigour fatigue.
• 16:10
  You know the good version of a pull request — define the types, add the database functions, the server changes, the client API, and the UI — each commit a clean little chapter a reviewer can step through.
• 16:22
  And you know what you actually do: define types, add database functions, work-in-progress test code, server stuff, client stuff, fix the database function, fix a UI bug, refactor, and fix another UI bug.
• 16:35
  The later commits stomp on the earlier ones, and the story falls apart.
• 16:39
  His fix uses Jujutsu — that's jj, the version-control system that sits on top of git and treats your commits as something you can rearrange freely rather than a log you have to get right as you go.
• 16:51
  And the technique is almost funny in how it inverts the usual advice.
• 16:55
  Instead of writing clean commits as you work, you make a mess on purpose.
• 16:59
  Then, at the end, you build your ideal commit history first — empty, labeled commits, one for types, one for the UI, and so on.
• 17:07
  You squash all your messy work into a single everything-commit.
• 17:11
  And then you interactively pull hunks out of that everything-commit into the right labeled boxes, one at a time, until the everything-commit is empty.
• 17:20
  He doesn't have a good name for it — his best offer is doing commits like a big pile of laundry, which is exactly right.
• 17:27
  You don't fold each shirt as you take it off.
• 17:30
  You make the pile, then you sort.
• 17:32
  The reason it beats the alternatives is specific.
• 17:35
  If you sort as you go with interactive squash, you can break a commit that happens to touch the same file.
• 17:41
  Doing it all at the end means the final state is guaranteed to have no conflicts, because everything started in one place.
• 17:49
  He flags the cost too: there's no guarantee that every intermediate commit compiles, so if you depend on clean bisects through history, this one isn't for you.
• 17:58
  I'm bringing it up not because it's earth-shaking but because it's exactly the kind of craft knowledge that's getting starved of attention while everyone talks about agents.
• 18:09
  The shape of your commit history is a gift to the next person reading the diff — and that person is usually a future you.
• 18:16
  A tool that lets you separate writing the code from telling its story is doing something useful.
• 18:22
  If you've been jj-curious, this is a nice concrete reason to try it on your next big feature branch.
• 18:28
  This next one started as a Reddit headline and got more interesting once I checked it.
• 18:33
  The post claimed OpenAI is paying people in New York to install three-hundred-sixty-degree cameras in their homes that record everything — vacuuming, washing dishes, and cooking.
• 18:44
  I couldn't verify the OpenAI-specific version of that, and I'd take the exact framing with a grain of salt.
• 18:51
  But the underlying trend is well-documented, and I'll walk you through it.
• 18:55
  There's a labor market forming to film household chores as training data for humanoid robots.
• 19:01
  The Washington Post and CNN have both covered it.
• 19:04
  DoorDash launched a Tasks app back in March that lets its US delivery workers — there are around eight million of them — earn money by recording themselves folding laundry, washing dishes, and making beds.
• 19:16
  A company called Micro1 says it has roughly four thousand robotics generalists across seventy-one countries sending in more than a hundred and sixty thousand hours of video every month.
• 19:27
  Pay runs around twenty to twenty-five dollars an hour.
• 19:31
  The gear is usually a head-mounted phone, not a fancy camera rig.
• 19:34
  And the robots being trained on all this are the ones you've heard about — Figure, Tesla, and Agility.
• 19:41
  The reason this deserves your attention as a builder isn't the privacy angle, though that's there too.
• 19:47
  It's what it tells you about where the bottleneck is for embodied AI.
• 19:51
  Language models had the open web — trillions of tokens of text just sitting there to be scraped, which is what that Meta lawsuit is all about.
• 20:00
  Robots don't have that.
• 20:01
  There's no internet-scale corpus of someone's hands folding a fitted sheet from a first-person view.
• 20:07
  The demonstrations have to be manufactured, one chore at a time, by paying actual people to do their actual housework on camera.
• 20:15
  So the next great training-data land grab isn't happening on the web.
• 20:19
  It's happening in living rooms.
• 20:21
  And I think the shape of it is going to matter.
• 20:24
  The text-data era was a grab-first, ask-later free-for-all, and we're now watching the legal bill come due for it.
• 20:31
  The physical-demonstration era is starting from a different place — these are explicit, paid, consented recordings, with a wage attached and a worker who agreed.
• 20:40
  Whether that's because the industry learned something from the copyright fights or just because there's no other way to get the data, I can't tell you.
• 20:50
  But it's a more transparent acquisition model than scraping, and if it holds, it's a small piece of good news buried inside a slightly dystopian-sounding gig listing.
• 21:00
  I want to end somewhere unusual for a developer show.
• 21:03
  Earlier this month, Pope Leo XIV published an encyclical — one of the most formal teaching documents the Catholic Church issues — and it's largely about artificial intelligence.
• 21:13
  It's called Magnifica Humanitas, and it climbed the front page of Hacker News, which tells you something about who's reading it.
• 21:20
  You don't have to share the faith to find the argument compelling, because it lands on territory this show cares about: work, and what these tools are for.
• 21:29
  The central line is one I keep turning over.
• 21:32
  Technology, he writes, is never neutral, because it takes on the characteristics of those who devise, finance, regulate and use it.
• 21:39
  That's a direct challenge to a reflex a lot of us in engineering carry around — the idea that a tool is just a tool, that the model is neither good nor bad, it's all in how you use it.
• 21:50
  His claim is that a system already carries the values of the people who built it and paid for it before you ever touch it.
• 21:57
  After the day we just walked through — a lawsuit about how training data got sourced, a debate about what the labs won't disclose — that lands harder than it would have a week ago.
• 22:07
  He isn't anti-technology, and he says so directly: technology should not be considered, in itself, a force antagonistic to humanity.
• 22:15
  His worry is concentration — that the power to reshape how we live is landing in private hands faster than any democratic institution can keep up with.
• 22:24
  And he makes a point about work specifically that any engineer watching headcount decisions should sit with: that work carries a dignity that doesn't come from its productivity, and that reducing a person to a cost of production gets something fundamental wrong.
• 22:39
  His phrase is that human dignity is something no machine can ever replace.
• 22:43
  You can take that as theology or you can take it as a design brief.
• 22:47
  I take it as both.
• 22:48
  We covered a system that proves theorems for a few hundred dollars, model-routing that cuts cost forty-fold, a lab rationing its own tokens, and robots learning to fold laundry from gig workers.
• 22:59
  Every one of those is a story about getting more done with less human effort.
• 23:04
  The encyclical is a reminder, from outside the industry, to keep asking what the less-human-effort is in service of — and to notice the moment the answer shrinks to nothing but the cost line.
• 23:15
  That tension — between what we can now automate and what we shouldn't reduce to a number — runs under most of what this show covers.
• 23:22
  It's the question I'd keep in the room the next time you wire up an agent and watch it do in a minute what used to take you a morning.
• 23:30
  Lenar Kess.