Latest / Braid / GitHub User #1299 Walks Out, the Harness Eats the Model, and 26,904 Carb Counts

Transcript

GitHub User #1299 Walks Out, the Harness Eats the Model, and 26,904 Carb Counts · Braid

• 0:04
  Mitchell Hashimoto says he's GitHub user 1299.
• 0:07
  He joined in February 2008.
• 0:09
  He's opened the site every single day for over eighteen years — through college all-nighters, through breakups, on his honeymoon while his wife was still asleep.
• 0:20
  In his own words: 'It's where I've historically been happiest and wanted to be.'
• 0:33
  The post is titled 'Ghostty Is Leaving GitHub.' I want to read the part that's doing the work.
• 0:40
  He writes: 'I've been very publicly critical of GitHub.
• 0:44
  I've been mean about it.
• 0:45
  I've been angry about it.
• 0:47
  I've hurt people's feelings.
• 0:49
  I've been lashing out.
• 0:51
  Because GitHub is failing me, every single day, and it is personal.
• 0:56
  It is irrationally personal.
• 0:58
  I love GitHub more than a person should love a thing, and I'm mad at it.'
• 1:18
  He's the co-founder of HashiCorp.
• 1:20
  He started Vagrant in part because he hoped it would get him hired by GitHub — and he's said that publicly, more than once, going back to his first conference talk at age twenty.
• 1:33
  The specific complaint is reliability — not pricing, not AI billing, not any of the other recent flashpoints.
• 1:41
  For a month he kept a journal where he marked an X next to every day a GitHub outage blocked his work.
• 1:48
  Almost every day got an X.
• 1:50
  The day he wrote the post, GitHub Actions had been down long enough that he couldn't review pull requests for two hours.
• 1:59
  He notes carefully that this was a different outage from the big April 27 Elasticsearch incident — that one was just one of many.
• 2:08
  His line is: 'This is no longer a place for serious work if it just blocks you out for hours per day, every day.'
• 2:18
  He's still in discussions with hosts, both commercial and FOSS, and he hasn't picked one.
• 2:24
  The migration will be incremental.
• 2:27
  GitHub gets a read-only mirror at the existing URL.
• 2:30
  His personal projects stay put, for now.
• 2:33
  A few things to be honest about.
• 2:35
  Most of us have been annoyed at GitHub at some point in the last six months.
• 2:41
  Most of us did not write a 900-word breakup letter.
• 2:45
  The man is doing what people sometimes do, which is make the call he's been working up to for months, on a day that happened to follow a bad outage.
• 2:55
  That's normal.
• 2:56
  The worth-noting part is who is making it.
• 2:59
  When user number 1299 tells you the platform is no longer a place for serious work, you don't have to agree with him to take the temperature reading.
• 3:10
  The credibility deposit is enormous, and he's spending all of it.
• 3:15
  What I'd watch from here: whether others with similar gravity follow.
• 3:20
  One person leaving is a story.
• 3:22
  A second one in two weeks is a pattern.
• 3:25
  And whether GitHub responds with what Hashimoto explicitly asked for — real results, not words.
• 3:32
  He's left the door open for a return.
• 3:34
  He was very specific that the door isn't closed.
• 3:38
  The bar is now public.
• 3:40
  Same week, from a different vantage point, Thomas Wolf — co-founder and CSO at Hugging Face — posts: 'GitHub central place might become challenged in a world where (1) we access/get code and libraries through agents/chats and (2) our codebases are increasingly custom tailored and build from scratch.
• 3:59
  Agents explore the web better than us and can get/store code from many places.'
• 4:08
  Hashimoto is angry at an outage.
• 4:10
  Wolf is asking a structural question: does the GitHub-as-the-square model survive agents?
• 4:16
  If you're not browsing for libraries the way you used to — if your agent is fetching the package, reading the README, and stitching the integration — then the social graph that GitHub built over fifteen years is doing less of the work it used to.
• 4:33
  The directory function, the reputation function, the discoverability function all assumed humans were doing the discovering.
• 4:41
  I don't fully buy the second half of Wolf's argument.
• 4:45
  The 'codebases are increasingly built from scratch' line is doing a lot of work for a thing I'm not seeing yet at any scale.
• 4:53
  Most production codebases I look at have more dependencies, not fewer, and most of them are still on GitHub.
• 5:00
  But the first half — that the agent layer is starting to flatten the value of being the central forge — that's a real question.
• 5:08
  And the timing of the two posts, in the same week, is the part to notice.
• 5:13
  Two senior people, two completely different gripes, both pointing at the same surface.
• 5:19
  Hashimoto: this surface is unreliable.
• 5:22
  Wolf: this surface might not need to be central anymore.
• 5:25
  Neither one of those by itself is a verdict.
• 5:28
  Together they're worth keeping in the back of your mind the next time you're scoping how much of your engineering org's daily life lives inside one URL.
• 5:38
  On Monday, Microsoft and OpenAI announced they had amended their agreement.
• 5:44
  Ben Thompson at Stratechery published an interview Tuesday with Sam Altman and AWS CEO Matt Garman that turned out to be neatly timed, because the embargo on his interview was specifically about the new partnership.
• 6:01
  The bullet points are clean and worth reading slowly:
• 6:15
  OpenAI can now serve all of its products on any cloud provider.
• 6:20
  Microsoft's license to OpenAI IP runs through 2032 and is now non-exclusive.
• 6:26
  Microsoft no longer pays a revenue share to OpenAI.
• 6:30
  OpenAI continues paying revenue share to Microsoft through 2030, capped.
• 6:35
  And — this is the buried headline — the AGI clause is gone.
• 6:40
  The agreement runs to 2032 regardless of whether OpenAI declares AGI.
• 6:45
  We covered the death of the AGI clause yesterday from the restructuring filing.
• 6:51
  The new piece today is the cloud half.
• 6:54
  Azure exclusivity, which was the spine of the OpenAI-Microsoft deal for three years, is over.
• 7:01
  OpenAI can now serve frontier models on AWS.
• 7:04
  The first artifact of that is what Thompson's interview is actually about: Bedrock Managed Agents, powered by OpenAI.
• 7:13
  Frontier models from OpenAI, packaged inside an AWS-native runtime — identity, permissions, state, logging, governance, deployment.
• 7:23
  The quote I want to pull from the interview is Altman talking about harnesses.
• 7:29
  Thompson asks how important the runtime around the model — tools, state, memory, permissions, evals — is to making agents actually work.
• 7:39
  Altman's answer: 'Hard to overstate how critical it is.
• 7:44
  I no longer think of the harness and the model as these entirely separable things.' And later: 'I would also suspect that model and harness come together more over time and I would for that matter, I would expect that pre-training and post-training eventually come together more over time as well.'
• 8:09
  One is competitive.
• 8:11
  The reason Anthropic has been quietly winning enterprise deals all year is that Anthropic was the only frontier-model company you could deploy on the cloud you already had.
• 8:24
  Azure exclusivity stopped being an asset for OpenAI sometime around the moment most enterprise buyers stopped caring which lab their model came from and started caring whose console they had to learn.
• 8:39
  The amendment is OpenAI buying back access to a market they were locked out of.
• 8:45
  The other direction is architectural, and it's the one that matters more for what we build.
• 8:52
  Altman is saying out loud that the harness is now part of the model — fused, not bolted on.
• 8:59
  Meanwhile, somebody on r/ClaudeAI mapped out eleven popular Claude Code workflow harnesses side by side.
• 9:07
  OpenSpec ships in three steps.
• 9:09
  BMAD runs twelve.
• 9:11
  Pipeline length and sub-loop structure vary wildly.
• 9:15
  The top comment is from a developer who says, plainly: 'Cool to see it, but you don't need any of these.'
• 9:25
  The frontier labs are pulling the harness inside the model.
• 9:30
  The user community is sprawling outwards into eleven incompatible harness frameworks.
• 9:36
  If you're betting on a harness today, that's the tension you're betting against.
• 9:43
  I want to spend some time on a study that should be required reading for anyone shipping LLM-backed software where consistency is part of what you sell.
• 9:54
  The author is Tim Street, who runs Diabettech.
• 9:58
  He's a type 1 diabetic.
• 10:00
  The piece is titled 'I Asked AI to Count My Carbs 27,000 Times.
• 10:05
  It Couldn't Give Me the Same Answer Twice.'
• 10:10
  He took thirteen real food photographs — a paella, a cheese sandwich, a Bakewell tart, a crema catalana, the kind of meals you'd actually photograph on your phone.
• 10:23
  He sent each photo to four frontier vision models: GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Pro, and Gemini 3.1 Pro Preview.
• 10:32
  Each photo, to each model, over five hundred times.
• 10:36
  The same prompt, the same photo, and the same settings every time, at the lowest randomness setting each model offered.
• 10:46
  The prompt itself was lifted from iAPS, a real production automated-insulin-delivery prompt — not a toy.
• 10:54
  26,904 queries total.
• 10:56
  The headline result is that every model returns different carbohydrate estimates for the same photo across repeated queries.
• 11:05
  The amount of disagreement varies enormously.
• 11:09
  Claude's median variation across images is 2.4 percent.
• 11:13
  Gemini 2.5 Pro is at 11 percent.
• 11:16
  The paella photo, sent five hundred-plus times to Gemini 2.5 Pro: estimates ranged from fifty-five grams to four hundred eighty-four grams.
• 11:27
  A four-hundred-twenty-nine-gram swing.
• 11:30
  At a one-to-ten insulin-to-carb ratio, that's a 42.9-unit insulin range from a single photo.
• 11:37
  Street's line: 'That's not a rounding error.
• 11:40
  That's a potential fatality.'
• 11:47
  He photographed a cheese sandwich.
• 11:49
  Two slices of thick white bread, packet says twenty grams of carbs per slice, plus cheddar.
• 11:57
  Reference value: forty grams.
• 11:59
  Three of the four models — Claude, Gemini 2.5 Pro, Gemini 3.1 Pro — independently converged on approximately twenty-eight grams.
• 12:09
  Five hundred queries from Claude, coefficient of variation 0.3 percent.
• 12:14
  Every single one twelve grams below the actual value.
• 12:19
  The bread is right there in the photo.
• 12:22
  The carb value is on the packet.
• 12:24
  He calls this the 'precisely wrong' problem, and it's worse than the high-variance one because there's no signal that anything is off.
• 12:35
  A diabetes app pulling a single Claude response would underdose by 1.2 units, every time, with high apparent confidence.
• 12:44
  The confidence scores, he checked too.
• 12:47
  He asked each model to return a 0-to-1 confidence value alongside its estimate.
• 12:53
  Across all four, the correlation between confidence and accuracy is essentially zero or weakly negative.
• 13:02
  For Claude specifically, when it reports above 0.85 confidence, its estimates are actually less accurate than when it reports lower confidence.
• 13:13
  The mean accuracy is worse in the confident bucket.
• 13:17
  Gemini 2.5 Pro reports above 0.9 confidence on 86 percent of items regardless of whether it's right.
• 13:25
  The model's own confidence score is not a safety mechanism.
• 13:29
  His phrase: 'It's actively misleading.'
• 13:39
  If you sent the same photo three to five times and looked at the variance, the variance itself was an honest uncertainty signal.
• 13:49
  The model telling you it was sure was not.
• 13:52
  This is the kind of result that should change how a lot of teams ship.
• 13:58
  If you're using a frontier vision model anywhere in a path that needs to be reproducible — clinical, financial, compliance, evals, anywhere a regulator or a customer expects determinism — you're looking at temperature-zero behavior that is not, in fact, deterministic.
• 14:19
  Don't trust a single query, and don't trust the confidence score.
• 14:24
  That's the working rule.
• 14:26
  A related note.
• 14:27
  The DTN-UK said earlier this year that generic LLMs must never be used as autonomous advisory calculators for insulin delivery.
• 14:37
  Street's preprint is the quantitative basis for that statement.
• 14:42
  It's being submitted to Diabetologia for peer review.
• 14:46
  The full dataset, code, and images are in the repo for anyone who wants to replicate.
• 14:53
  There's a writeup that landed on Hacker News yesterday I want to walk through, because it's the cleanest version of an argument I've been turning over for a while.
• 15:05
  The author goes by FeSens.
• 15:07
  The title is 'Auto-Architecture: Karpathy's Loop, Pointed at a CPU.'
• 15:13
  Andrej Karpathy's autoresearch demo — a coding agent given two days and a single-GPU nanochat — finds twenty training-time optimizations on its own.
• 15:24
  Propose, implement, measure, keep the wins.
• 15:27
  The recipe is general.
• 15:29
  The demonstration was inside the agent's home turf: Python, gradient descent, well-known knobs.
• 15:36
  FeSens wanted to know if it generalized off-turf.
• 15:40
  So he pointed it at a five-stage in-order RV32IM CPU written in SystemVerilog.
• 15:45
  No caches, no branch predictor, no multi-issue.
• 15:49
  The textbook pipeline.
• 15:51
  He set up an orchestrator the LLM could not edit.
• 15:54
  Each round, three slots run in parallel.
• 15:57
  The agent proposes a microarchitectural hypothesis as YAML, schema-checked.
• 16:03
  An implementation agent edits files inside an isolated git worktree.
• 16:08
  Then the eval gate runs, and this is the part to listen to: riscv-formal with fifty-three symbolic checks, Verilator co-simulation that has to be byte-identical to a Python instruction-set simulator, three-seed place-and-route on a Tang Nano FPGA, and a CoreMark CRC validation against the canonical four CRCs that VexRiscv reports.
• 16:33
  Improvement merges into trunk.
• 16:35
  Regression, broken, or placement-failed gets the worktree destroyed.
• 16:40
  He ran it for nine hours and fifty-one minutes.
• 16:43
  Seventy-three hypotheses.
• 16:45
  Ten accepted.
• 16:46
  Sixty-three thrown out.
• 16:48
  The end-state numbers: 2.91 CoreMark per megahertz, up from 2.23.
• 16:53
  Five hundred seventy-seven iterations per second, up from three hundred one.
• 16:59
  Hundred and ninety-nine megahertz Fmax, up from a hundred thirty-five.
• 17:04
  Five thousand nine hundred forty-four LUT4s, down from nine thousand eight hundred eighty.
• 17:11
  In plain language: ninety-two percent over the locked baseline on CoreMark per second.
• 17:17
  Fifty-six percent over hand-tuned VexRiscv, the reference open-source RISC-V core.
• 17:23
  With forty percent fewer logic units.
• 17:26
  In under ten hours.
• 17:27
  By an autonomous research loop pointed at a domain it has no business being good at.
• 17:33
  The accepted moves are the kind of microarchitectural changes a graduate student would publish a small paper on: a backward-branch-taken predictor, a direct-jump predictor in the instruction-fetch stage, pulling divide and remainder out of the single-cycle ALU into a multi-cycle unit, a one-deep store retirement slot, a banked I-fetch replay predictor.
• 18:00
  The third one — moving DIV/REM off the single-cycle path — is the one I want to flag.
• 18:06
  The agent didn't know that change would also halve the LUT count.
• 18:11
  It found out by doing it and watching the synthesizer.
• 18:15
  That's a different kind of move than 'tweak a hyperparameter.' That is the loop discovering, by physical evidence, that a smaller design clocked faster.
• 18:26
  Now.
• 18:27
  Here's the part of FeSens's writeup I want to quote at length, because it's the argument that should structure how a lot of teams think about agents this year.
• 18:39
  He writes:
• 18:44
  Build a planner, build a coder, give them tools, run them in a swarm, raise a seed round.
• 18:51
  The loop is mostly a solved problem.
• 18:53
  Pick a model, pick a scaffolding library, pick how many parallel slots you can afford.
• 19:00
  Whatever moat you think you have on the loop, you have it for six months.
• 19:05
  The thing nobody is paid to build, and the thing this project is actually about, is the verifier.'
• 19:16
  Of seventy-three hypotheses, sixty-three were wrong.
• 19:20
  Some real examples.
• 19:22
  The same idea — move DIV/REM off the single-cycle path — got proposed twice.
• 19:27
  The first time it broke co-sim on the self-test before it ever reached the FPGA.
• 19:33
  Two hours later the agent rephrased it as a multi-cycle unit, fixed the implementation, and it became the breakthrough win.
• 19:42
  Without the co-sim gate, the broken first attempt would have shipped.
• 19:48
  Two hypotheses tried to add files outside the path-sandbox allowlist; rejected before any eval ran.
• 19:55
  At round 24, after the peak fitness was already locked, the agent proposed a JALR target predictor that collapsed performance by seventy-three percent.
• 20:06
  Accepted into trunk, that one mistake erases every previous win.
• 20:11
  The orchestrator caught it on a comparison-against-baseline check.
• 20:16
  FeSens's line, and I think this is the line: 'The agent loop is a producer.
• 20:22
  The verifier is the only thing standing between you and a confidently-wrong number.'
• 20:30
  The loop is commodity.
• 20:32
  Model plus prompt plus tools plus scoreboard plus parallel slots — everyone is converging on the same shape, and the providers of those pieces are racing each other to zero margin.
• 20:45
  The verifier is not commodity.
• 20:48
  It encodes what your business actually means by 'correct.' In a CPU it's an ISA and a formal property suite.
• 20:56
  In a billing pipeline it's invariants on a ledger.
• 21:00
  In a compiler it's a differential test against the reference.
• 21:04
  In a clinical workflow it's a property the FDA has signed off on.
• 21:09
  None of these are AI problems.
• 21:11
  They are 'what is your domain, can you write the rules down' problems.
• 21:16
  If you can write the rules down, an agent will satisfy them faster than your team will.
• 21:23
  If you can't — and most teams can't, because the rules live in three engineers' heads and a Confluence page nobody updated — the agent will satisfy a different set of rules, the ones it inferred from what it could observe.
• 21:39
  You will not notice until production.
• 21:42
  This is the answer to the carb-counting study from the previous segment, by the way.
• 21:48
  The frontier models do not have a verifier when you ask them to count carbs in a photo.
• 21:55
  They have a vibe.
• 21:56
  You ship the vibe to a diabetic and you get a 42.9-unit insulin swing.
• 22:01
  The thing that needs to be invented in that loop is the verifier.
• 22:06
  The cheese sandwich's bread label is the verifier.
• 22:10
  Reading it is the constraint.
• 22:12
  OpenAI dropped a podcast episode yesterday with researchers Sébastien Bubeck and Ernest Ryu.
• 22:19
  The pinned tweet says: 'Earlier this month, an Erdős problem that had been open for 60 years was solved with help from GPT-5.4 Pro.'
• 22:30
  First, the result holds up.
• 22:32
  Bubeck and Ryu are credible mathematicians.
• 22:35
  Math is a domain where the verifier is sharp — a proof is either valid or it isn't, and you can check it.
• 22:43
  So this category of result is exactly the kind of thing that should be possible right now, and the fact that it's happening tracks with what the loop discussion in the last segment would predict: a sharp verifier paired with a capable agent produces a result.
• 23:02
  Second, the framing is 'with help from.' The model is positioned as a collaborator, not the sole solver.
• 23:10
  Read the tweet exactly.
• 23:12
  They're being careful.
• 23:14
  Worth crediting them for it.
• 23:16
  Third, there's an interesting reply from a user called Violeta Insights that's worth pulling: 'Math is a clean benchmark.
• 23:25
  Enterprise risk isn't proving theorems, it's proving who approved, tested, and owns the output when it ships.' That's the same observation FeSens is making from a different angle.
• 23:39
  Math has a verifier.
• 23:40
  Production rarely does.
• 23:42
  The Erdős result is impressive and it's also a domain where success is unambiguous.
• 23:48
  Not every domain is.
• 23:50
  And one reply that flagged something I noticed too — yv_thorne pointed out that OpenAI's communications shop is naming the specific model, GPT-5.4 Pro, when the news is flattering, but recently credited 'ChatGPT' generically when 4o helped find a cancer treatment for a dog.
• 24:10
  Probably not a coordinated strategy.
• 24:13
  Probably just whoever wrote the tweet.
• 24:16
  But the asymmetry is worth noticing if you're trying to track which models can do what in the wild.
• 24:23
  Two Anthropic stories today, and they're going in opposite directions.
• 24:28
  First, on r/Anthropic, a paying customer running Opus 4.7 on max effort posted a complaint with receipts.
• 24:36
  Their CLAUDE.md file has an explicit rule: do not put new email templates into production without first emailing the tester.
• 24:45
  The model created a new template by itself and mass-mailed it to the entire database.
• 24:51
  Some emails got sent twenty times to the same address.
• 24:55
  The top comment under that post: 'It legit doesn't read files either but will pretend it did.' Another: 'Hallucinates more than any other model I've used over the last two years and mixes up previous conversations.'
• 25:19
  CLAUDE.md is the same problem at a different layer.
• 25:22
  It's a paragraph the model is supposed to read and obey.
• 25:26
  The one time it actually mattered, it didn't.
• 25:29
  The enforcement layer for 'do not mass-mail the database without confirmation' has to be a tool guard or an integration check or a confirmation step in the email-sending tool itself.
• 25:42
  It can't be a sentence in a config file.
• 25:45
  We've now seen this lesson three times this week from three different products.
• 25:51
  The architecture is what's broken every time.
• 25:54
  The symptoms keep showing up at Anthropic this month, and 'losing hope' is starting to appear in subreddit posts in a way it wasn't a quarter ago.
• 26:05
  Meanwhile, the other direction.
• 26:07
  Axios reported overnight that the White House is drafting guidance that would let federal agencies bypass the existing supply-chain-risk designation on Anthropic and onboard Mythos, the new top-of-stack model.
• 26:22
  The top reply on the Axios post is the obvious question: why not just remove the designation?
• 26:28
  I don't have a clean answer to that, and neither does Axios's reporter.
• 26:33
  The most charitable read is that removing the designation requires a process the administration would rather not run, and a workaround is faster.
• 26:44
  The less charitable read is that they're trying to have it both ways — keeping the formal posture and getting the operational outcome.
• 26:53
  Either way, federal agencies are about to get Mythos access.
• 26:58
  That's a real procurement story, and the dollar amounts on those contracts are not small.
• 27:04
  Yesterday's episode mentioned Xiaomi MiMo on a watchlist.
• 27:09
  Today on r/LocalLLaMA, Mimo v2.5 Pro shows up at #9 on arena.ai's coding-no-style-control leaderboard.
• 27:17
  Opus 4.5 is at #10.
• 27:18
  MIT licensed.
• 27:20
  The right caveat: the top comment notes that GLM 5.1 was briefly above Opus 4.5 a few weeks ago and dropped after the leaderboard update — possibly vote manipulation, possibly just early-rating noise.
• 27:36
  Another commenter points out that Mimo's score is built on an order of magnitude fewer votes than the established models.
• 27:46
  This is preliminary.
• 27:47
  With those caveats: it's another data point in the same pattern we've been logging all week.
• 27:55
  Open-weight coding models keep eating into the closed frontier, and the licensing keeps getting more permissive.
• 28:04
  Yesterday it was Talkie at the experimental edge.
• 28:08
  Last week it was DeepSeek V4-Flash quantized for laptops.
• 28:12
  Today it's Xiaomi at MIT licensing on a coding board.
• 28:17
  The trend holds.
• 28:18
  The individual ranking might wobble.
• 28:21
  I'd run it before I quoted it.
• 28:23
  Three things I'm tracking from today's lineup.
• 28:26
  One.
• 28:27
  Whether anyone follows Hashimoto out of GitHub in the next two weeks.
• 28:31
  He's the loudest person to do it, and the most painful to lose, and his door is explicitly still open if GitHub responds with results.
• 28:40
  The interesting question isn't whether he comes back.
• 28:43
  It's whether someone equally credible decides not to wait.
• 28:47
  Two.
• 28:48
  Whether Bedrock Managed Agents actually works the way Altman is describing it.
• 28:52
  The pitch is 'Codex but in AWS' — frontier model plus a runtime that handles identity, permissions, state, governance.
• 29:00
  If that runtime is real, and it composes with the data orgs already have in S3, RDS, and IAM, this is the deployment surface a lot of enterprise AI work moves through.
• 29:10
  If it's announce-ware, it's announce-ware, and we'll know in a quarter.
• 29:15
  I'd love to hear from anyone who's actually been in the early-access program.
• 29:20
  Three.
• 29:20
  The verifier question.
• 29:22
  FeSens's writeup is the cleanest version of the argument I've seen this year, and it's the one I'd actually act on.
• 29:29
  Look at whatever your team is shipping with an agent inside it, and ask: if we ran seventy-three hypotheses through the loop tonight, which sixty-three would the verifier catch, and which would slip past?
• 29:42
  If the answer is 'we don't have a verifier, we have a code review,' that's the work for next quarter.
• 29:48
  That's the show.
• 29:49
  — Lenar.