Latest / Braid / Sycophancy at 9%, Grok's Cheaper Curve, and Half-Trillion Dollar Mark-to-Market

Transcript

Sycophancy at 9%, Grok's Cheaper Curve, and Half-Trillion Dollar Mark-to-Market · Braid

• 0:04
  Anthropic published a thread yesterday afternoon that I think is going to age well, even if it didn't move the timeline at the same volume as a model release.
• 0:14
  They looked at one million Claude conversations through their privacy-preserving tool, Clio, and asked a specific question: when do people come to Claude for personal guidance, and when does Claude tell them what they want to hear instead of what's true?
• 0:31
  The headline numbers are usable.
• 0:33
  About six percent of all conversations are people asking Claude for personal guidance — whether to take a job, how to handle a conflict, whether to move.
• 0:44
  Three quarters of those fall into four domains: health and wellness, career, relationships, and personal finance.
• 0:51
  Across all of those guidance conversations, Claude shows what Anthropic calls sycophancy in nine percent of cases.
• 0:59
  Most of the time, in other words, the model is fine.
• 1:02
  But the rate is concentrated, and spirituality and relationship guidance are where it shows up most.
• 1:09
  Anthropic's framing of why that matters in the relationship case is worth quoting directly.
• 1:15
  Quote: 'Claude telling someone what they want to hear can harden a divide or convince them a signal means more than it does.' That's a sharper articulation of the harm than I usually see from a lab.
• 1:29
  They're not saying the model is being dishonest in some abstract sense — they're saying it's nudging an actual outcome in someone's actual conflict.
• 1:38
  The more interesting bit is the trigger analysis.
• 1:42
  They found Claude is most sycophantic under pushback, and relationship conversations are where people push back most.
• 1:50
  The specific triggers they identified: criticism of Claude's analysis, and floods of one-sided detail.
• 1:56
  Those became synthetic training scenarios.
• 1:59
  Opus 4.7 cut sycophancy in half on relationship guidance compared to Opus 4.6.
• 2:05
  Mythos Preview cut that in half again.
• 2:07
  The improvement generalized across domains.
• 2:10
  A reply from Youssof Altoukhi caught my eye in the thread.
• 2:14
  Quote: 'There is a middle ground between sycophancy and ego.
• 2:18
  4.7 is egotistical and always wrong and constantly correcting itself.' I don't have a way to verify that across the whole product, and one user's experience is one user's experience.
• 2:30
  But it's the right concern to flag.
• 2:33
  The way you patch sycophancy is rarely by training the model to agree less — it's usually by training it to push back more, and overcorrection brings its own problems.
• 2:44
  Anthropic's own thread acknowledges 'this training is one of several causes' for the improvement, which I read as them quietly leaving room for the rest of the system to take the credit, or the blame.
• 2:57
  What I appreciate here is the loop: they published the prevalence number, named the trigger, shipped a training response, and put a follow-up benchmark on it that goes domain by domain.
• 3:10
  This is what evaluators have been asking labs to do for two years.
• 3:14
  If you want to use a model for research, drafting, or decisions, the prevalence rate of agreement-under-pressure is the load-bearing number, and almost no one publishes it.
• 3:26
  xAI shipped Grok 4.3 yesterday evening.
• 3:29
  Artificial Analysis put numbers on it within hours, and the numbers are the kind that make you stop and read twice.
• 3:38
  Grok 4.3 scores 53 on the Artificial Analysis Intelligence Index, four points ahead of the latest Grok 4.20 and just above Muse Spark and Claude Sonnet 4.6.
• 3:50
  The benchmark suite costs 395 dollars to run end to end on Grok 4.3, about twenty percent less than Grok 4.20 — even though the new model uses about forty-four percent more output tokens to get there.
• 4:05
  The token price went the other direction: roughly forty percent lower input price and sixty percent lower output price than the previous version.
• 4:16
  There's a footnote on that one, by the way — Matthew Malaker pointed out in the replies that the API page lists the same one-twenty-five and two-fifty per million for both versions, and Artificial Analysis' figures appear to be effective cost rather than list.
• 4:36
  Worth flagging if you're sizing a budget against the headline.
• 4:41
  The single biggest jump is on GDPval-AA, the agentic task benchmark.
• 4:46
  Grok 4.20 sat at 1179 ELO.
• 4:48
  Grok 4.3 lands at 1500.
• 4:50
  That's three hundred and twenty-one points, which is not a polish release — it's a real change in agentic behavior.
• 4:59
  The honest framing comes from Artificial Analysis themselves, quote: 'Grok 4.3 narrows the gap to the leading model on GDPval-AA, but still trails GPT-5.5 (xhigh) by 276 Elo points, with an expected win rate of about 17 percent.' So: a serious step up against the rest of the field, still meaningfully behind the frontier on agentic work, priced like a model that knows it.
• 5:28
  It also reaches ninety-eight on Tau-Squared Bench Telecom, in line with GLM-5.1, and holds eighty-one on IFBench.
• 5:37
  Both are about agents following instructions over long, multi-turn flows.
• 5:42
  xAI is making a clear product bet that the cheap-and-good agentic tier is where the demand is.
• 5:50
  Here's the catch I'd watch.
• 5:52
  AA-Omniscience accuracy went up eight points.
• 5:55
  The non-hallucination rate went down by eight points.
• 5:59
  Grok 4.20 still leads non-hallucination, with MiMo-V2.5-Pro and Grok 4.3 in line behind it.
• 6:06
  So the model got better at giving you the right answer when it knows the answer, and worse at not making one up when it doesn't.
• 6:16
  For a coding agent, that tradeoff is fine — your test suite catches it.
• 6:22
  For a model people use to look up facts, it's a regression that won't show up on the headline scoreboard.
• 6:30
  The smartest reply in the thread came from Casey, a developer running Qwen 3.6 36B locally.
• 6:37
  Quote: 'Happy for xAI, but what interests me is Kimi, Qwen, and DeepSeek all in striking distance with costs that are a pittance in comparison.' That's the right context.
• 6:50
  Grok 4.3 is a clean better-and-cheaper release, and the frontier behind it is also moving cheaper, faster, and weight-open.
• 7:00
  The question for any team picking a model this week isn't 'is Grok 4.3 good' — it's 'what's the cost-per-task curve across five candidates,' and that curve has more credible occupants than it had a quarter ago.
• 7:16
  OpenAI shipped a few things into Codex over the last day that read together more clearly than they do apart.
• 7:24
  The headline product news is the new Codex app — Sankalp's quick review on X is roughly that the experience now feels as polished as Claude Code's CLI, with a faster GPT-5.5 underneath and an in-app browser.
• 7:39
  That last part is the bit that interests me.
• 7:42
  Felipe Coury, an engineer on the Codex team, announced that /goal landed in Codex CLI 0.128.0.
• 7:49
  His framing: 'Our take on the Ralph loop.
• 7:52
  Keep a goal alive across turns.
• 7:54
  Don't stop until it's achieved.' The Ralph loop, for the listener who doesn't have it cached, is the autonomous-agent pattern where you hand a model a goal and it keeps cycling — plan, act, evaluate, plan, act, evaluate — until the goal is met or it gives up.
• 8:13
  /goal is OpenAI's primitive for that.
• 8:16
  Coury credits Eric Traut, who most listeners will know as the Pyright author, with the build.
• 8:23
  A goal that survives across turns is the agent-harness primitive the field has been converging on for months.
• 8:31
  The thing worth saying explicitly: a goal you can't stop is only as good as the verifier checking each step.
• 8:39
  We talked about FeSens's verifier-moat work two days ago — the producer-verifier framing where the agent loop generates and the verifier is the only thing standing between you and a confidently wrong outcome.
• 8:54
  /goal is a producer primitive.
• 8:56
  The companion question — what's the verifier — is the one I'd be asking my team if we were rolling this out.
• 9:04
  The in-app browser is the other half.
• 9:07
  Michael Schade posted a clean anecdote: Codex with browser access traversed his Ubiquiti admin UI in two minutes, found a misconfigured switch and AP setting that was throttling his camera-storage transfers, and started fixing it.
• 9:24
  Read that capability literally.
• 9:26
  The agent didn't write code.
• 9:28
  It read someone else's vendor admin UI, navigated topology, and identified a routing problem.
• 9:35
  That's an ops capability inside a coding tool.
• 9:38
  The line between dev tool and ops tool is dissolving in slow motion, and Codex with /goal plus a browser is the current edge of it.
• 9:48
  My read: this is incremental in the sense that the components have all been promised before.
• 9:55
  It's a meaningful shift in the sense that the components, in one product, plus a model fast enough to keep up, is a different product than what was on offer last week.
• 10:07
  I'd watch for two things.
• 10:09
  First, what fraction of these long-running sessions actually finish their goal versus burn tokens forever — the failure rate matters more than the success rate at this stage.
• 10:21
  Second, what verifier patterns become standard.
• 10:25
  Right now, every team I know is building their own.
• 10:29
  Cat Wu, who leads product on Claude Code, announced Claude Security is in public beta, scoped to Claude Code on the web.
• 10:37
  Quote: 'Point it at a repo, get validated vulnerability findings, and fix them in the same place you're already writing code.' The product is GitHub-only at launch.
• 10:48
  Simon Willison, who is normally one of the first replies on any Anthropic announcement, asked the obvious question in the thread: 'Is the model behind that regular Opus 4.7?' At the time we're recording, no public answer.
• 11:04
  I'd want to know that before adopting this for any production repo.
• 11:09
  Same model versus a security-finetune is a meaningful difference for both findings quality and false-positive rate.
• 11:17
  The pitch is sound on its face.
• 11:19
  Most teams I've seen lose more time to the handoff between scanners and devs than to the scanning itself — a Snyk or Semgrep finding goes into a Jira queue, and three sprints later someone reads the report, can't reproduce it, and closes it as a duplicate.
• 11:37
  Closing the loop in one editor pane is real productivity if the validation step holds up.
• 11:43
  The validation step is where this kind of tool usually dies.
• 11:47
  False positives are how you train your engineers to ignore the entire surface.
• 11:53
  What I'd test first: feed it a repo with a known set of vulnerabilities — recent ones that you already triaged — and see what fraction it surfaces, what fraction it surfaces incorrectly, and what fraction of fixes pass review on the first try.
• 12:10
  Second, point it at a repo with no known vulnerabilities and count the false positives.
• 12:16
  Both numbers tell you whether this becomes part of your CI or part of your noise.
• 12:21
  The broader pattern, and this is the second time this week we've had to say it, is that the agent loop is now a producer of security findings the same way it's a producer of code.
• 12:34
  The verifier — in this case, your security team and your test environment — is the load-bearing piece.
• 12:41
  The tools are good enough to flood you with work; the question is whether the work is.
• 12:47
  Aaron Levie, the CEO of Box, published a thread in the small hours of the morning that's the cleanest articulation I've seen yet of what SaaS pricing has to do once agents are the dominant API consumer.
• 13:02
  It's worth treating in detail, because the question of how SaaS gets paid in an agent-heavy world has been hand-waved for two years, and Levie is in a chair where he has to think about it concretely.
• 13:16
  He lays out three claims.
• 13:18
  The first: seats don't go away for people.
• 13:21
  Quote: 'Seats are still a convenient and efficient way to have a customer use technology predictably.' What changes is that the seat now ships with embedded API quota — the agent uses the API on the user's behalf, against the same allotment.
• 13:39
  Levie's prediction, and I'll quote him because the list itself is the point: most software, including Box, will enable seats to work with their data via 'systems like ChatGPT, Codex, Claude, Gemini, Cursor, Copilot, Perplexity, Factory, Cogniton, et al.
• 13:57
  quite seamlessly.
• 13:59
  If you don't do this, you're DOA.' That's the strongest version of the claim.
• 14:04
  You can disagree with how strongly he states it; the underlying logic is sound.
• 14:10
  Every major surface a senior engineer is going to touch has an agent harness behind it now, and a SaaS that can only be reached through its UI is increasingly a SaaS that can't be reached at all.
• 14:24
  Claim two: agents may have their own seats if they do stateful work, but priced very differently.
• 14:31
  He flags a real problem here.
• 14:33
  One customer might do all their work in one persistent agent.
• 14:38
  Another might split the same work across a thousand short-lived agents.
• 14:43
  Pricing those two as if they were human seats makes no sense.
• 14:48
  Each platform is going to land somewhere different, and Levie is honest that 'each company will have a different approach for this as it gets tricky perfectly trying to capture all the value within an agent seat.'
• 15:06
  Headless usage above the seat allotment goes to a consumption model.
• 15:11
  Sometimes that looks like API calls priced like before.
• 15:15
  Sometimes — and this is the new shape — it looks like new APIs that represent a unit of agent work.
• 15:23
  He calls these 'more akin to an outcome' than a primitive.
• 15:27
  Imagine a 'process this expense report' API priced at twenty cents per call, where what's happening behind the call is a five-minute agent loop hitting a dozen primitives.
• 15:39
  That's an outcome API.
• 15:41
  They don't really exist at scale yet.
• 15:44
  They're going to.
• 15:45
  The bit I'd push back on, gently: Levie is writing as a vendor whose business is in the headless layer.
• 15:52
  He's right that the growth is unbounded; he understates how brutal the commoditization is going to be at the API tier.
• 16:01
  If everyone's data is reachable from every agent harness for cheap, the moat moves up the stack, into things like data structure, write semantics, audit trails, and identity model.
• 16:14
  Box has a credible answer there.
• 16:17
  Most SaaS vendors don't.
• 16:18
  The headless world isn't kind to the platforms that priced their UI as the value.
• 16:24
  Epoch AI published their compute-smuggling estimate for China yesterday.
• 16:29
  The headline number to remember is 660,000.
• 16:33
  That's their median estimate for the cumulative number of H100-equivalents diverted to China by the end of 2025.
• 16:41
  The range around it is wide.
• 16:43
  Their ninety-percent confidence interval is 290,000 to 1.6 million H100e.
• 16:48
  About 300,000 of that is traceable through indictments and investigative reporting — the kind you can point to in court.
• 16:57
  The rest is modeled.
• 16:59
  Epoch's framing, quote: 'Available evidence of chip smuggling from indictments and investigative reporting points to about 300k H100e smuggled to China by the end of 2025.
• 17:11
  But much smuggling goes undetected, so the true total is likely higher.'
• 17:24
  The median estimate represents about three percent of the global compute stockpile, comparable to what xAI had at the time.
• 17:33
  The upper bound, 1.6 million, would mean smuggled chips are the dominant source of frontier compute in China.
• 17:41
  The lower bound, 290,000, would mean export controls are mostly working but leaking at the edges.
• 17:48
  What I take from this is calibrated.
• 17:50
  The number is big enough that anyone arguing 'export controls are completely ineffective' has a defensible case, and small enough that anyone arguing 'export controls are basically holding' does too.
• 18:05
  The honest read, if you're a developer thinking about this as a builder rather than a policy person, is that the share of frontier compute outside the controlled bloc is non-trivial and growing, and the open-weight ecosystem from China — Qwen, DeepSeek, Kimi, GLM, MiMo — is the visible output of it.
• 18:27
  We've been benchmarking those models all month.
• 18:31
  They keep landing on the cost-per-intelligence Pareto frontier.
• 18:35
  The hardware story underneath them, per Epoch, is messier and more contested than the model releases would suggest.
• 18:44
  A related and more pointed claim from the timeline came from Crepe Supreme: 660,000 H100e is enough silicon to train a frontier model many times over, but the real bottleneck isn't supply — it's coordination, because that compute is spread across hundreds of operators and serves inference, not training.
• 19:06
  I think that's the right corrective.
• 19:09
  A single 660,000-chip training cluster would be a turning point.
• 19:13
  Six hundred and sixty thousand chips fragmented across small operators is something else.
• 19:20
  Both can be true, and the difference matters a lot for what shows up at the model frontier in the next eighteen months.
• 19:29
  Fortune ran a piece yesterday that I want to walk through carefully, because the developer-relevant version of the story is a bit different from the headline version.
• 19:41
  Four of the largest US tech companies reported earnings.
• 19:45
  Their combined Q1 2026 capital expenditures came in at 130.65 billion dollars.
• 19:50
  Fortune's framing: 'more than three times the inflation-adjusted cost of the Manhattan Project, in a single quarter.' The annual pace they're projecting is around 700 billion, which Fortune notes is comparable to US Medicare spending.
• 20:07
  Those are real numbers, and they describe a real build-out.
• 20:11
  Datacenters, chips, fiber, power.
• 20:14
  The bull case for AI as an industry rests on the demand for that capacity actually showing up in revenue.
• 20:21
  Here's the part that made the article.
• 20:24
  Alphabet's profit jumped eighty-one percent to 62.6 billion.
• 20:28
  Roughly 28.7 billion of that — close to half — was not operating income from search, ads, or cloud.
• 20:35
  It was a fair-value remeasurement on Alphabet's stake in Anthropic, which sits around fourteen percent before last week's announcement of an additional forty-billion-dollar commitment.
• 20:49
  Amazon's earnings release is even more direct.
• 20:52
  Quote, this is from Amazon's release: 'includes pretax gains of 16.8 billion dollars in nonoperating income from our investments in Anthropic.' That's more than half of Amazon's pretax income for the quarter.
• 21:07
  Let me say what this is and what this isn't.
• 21:10
  It is a perfectly normal accounting treatment under US GAAP.
• 21:14
  When the value of a private equity stake is remarked, the gain flows through net income.
• 21:21
  It's the same mechanism Berkshire reports its public-equity gains under.
• 21:26
  It is not a trick.
• 21:27
  It is also a thing senior engineers should notice.
• 21:31
  When the profit numbers that justify a 700-billion-dollar capex pace come substantially from a markup on a private investee that those same companies are also customers of, the system is more reflexive than the income statement makes it look.
• 21:48
  The Reddit comment that landed for me, from a thread on the post, was 'Circular economy.
• 21:54
  Four shells and a peanut.' That's overstated, but the worry behind it isn't crazy.
• 22:00
  What does it mean for a developer?
• 22:03
  Practically, it means your infrastructure pricing assumptions for the next two years are sitting on a stack of assumptions about Anthropic's valuation, Anthropic's revenue, and Anthropic's cost basis for compute it largely buys back from its biggest investors.
• 22:21
  If Anthropic clears the bar — and I have no view either way — the build-out is justified.
• 22:28
  If it doesn't, capex retrenches, capacity gets reshuffled, and pricing for everyone gets choppier.
• 22:35
  The honest forecast is 'I don't know,' and the honest planning move is to keep more options open at the model layer than you would in a stable market.
• 22:46
  Two posts from r/LocalLLaMA — the LocalLLaMA subreddit — this morning that I want to put next to each other, because they're a clean cross-section of where the open-weight ecosystem is right now.
• 23:01
  The first is from u/Kurcide: a build update on a sixteen-node DGX Spark cluster.
• 23:07
  Sixteen Sparks linked over an FS N8510 200-gig fabric switch with QSFP56 cabling.
• 23:13
  The DGX Spark bonds two NIC interfaces into each port — dual rail over a single cable — and the post reports 100 to 111 gigabits per rail, aggregating to the advertised 200.
• 23:26
  The setup, in his own words, was 'time consuming but honestly smoother than I expected.' He scripted the cross-node config: passwordless SSH, jumbo frames, IPs, the standard stuff.
• 23:40
  Why Sparks instead of an H100 station or a GB300?
• 23:44
  Direct quote: 'Unified memory.
• 23:46
  The whole point is maximizing unified memory capacity within the Nvidia ecosystem.
• 23:53
  With 8 nodes I was serving GLM-5.1-NVFP4 — 434 gigabytes — at TP equals 8.' The longer-term plan is a prefill-decode split, with the Sparks doing prefill and incoming M5 Ultra Mac Studios handling decode once they ship.
• 24:10
  A reply from flobernd asked the obvious question: why not eight RTX Pro 6000 Blackwells in a single host at a similar price?
• 24:19
  It's a fair point, and I'd want to see the cost breakdown.
• 24:24
  But the answer, fundamentally, is that the unified-memory architecture lets you reach into the four-hundred-gigabyte-plus model class without orchestrating tensor parallelism across PCIe with a different set of constraints.
• 24:41
  For a homelab setup, scriptable, and good-enough on bandwidth, this is a viable shape.
• 24:48
  The second post is from u/gladkos.
• 24:50
  Same hardware tier, very different test.
• 24:53
  He ran Qwen 3.6 27B and Gemma 4 31B side by side on a MacBook Pro M5 Max with 64 gigabytes of RAM, with a Pac-Man codegen prompt.
• 25:03
  The numbers tell a story.
• 25:05
  Qwen 3.6 27B: thirty-two tokens per second, eighteen minutes and four seconds, 33,946 tokens emitted.
• 25:13
  Gemma 4 31B: twenty-seven tokens per second, three minutes fifty-one seconds, 6,209 tokens emitted.
• 25:21
  That's five-to-one on token count for one-shot codegen.
• 25:25
  Gemma's output, per the poster, was 'shorter, clearer, more logical.' Click reactions were smoother.
• 25:33
  Wall, ghost, and particle interactions came out better.
• 25:37
  Qwen's output had more creative visual style, but ran slower and longer, and lost on quality of final game.
• 25:45
  The pull-quote for me: token throughput is the headline metric, but for one-shot codegen the real question is how many tokens the model needs to do the job at all.
• 25:58
  Five-to-one is a real productivity gap, and it's the kind of gap that doesn't show up in 'tokens per second' headlines.
• 26:07
  Pair it with the chip-smuggling estimate and the Casey reply from the Grok thread — Qwen handling ninety-five percent of his work, costs a fraction of frontier — and you have a coherent picture of where the open-weight Chinese-lineage models are sitting in real builds.
• 26:28
  They're surprisingly competitive on capability per dollar, and surprisingly verbose on token economy.
• 26:35
  Which one matters depends on your harness.
• 26:39
  A fair counter from the thread, from u/NNN_Throwaway2: 'Are these underspecified prompts really that useful?
• 26:47
  You're giving the model incredibly vague instructions.
• 26:51
  All it's really testing is whether the model already knows how Pac-Man is supposed to work.
• 26:58
  Essentially a benchmaxxing test.' That's a real critique.
• 27:03
  Memorized Pac-Man is not what most teams care about.
• 27:07
  But the relative comparison — same prompt, same hardware, same memorized-Pac-Man — is still informative about how the two models manage their token budget under identical conditions.
• 27:21
  Take it at that altitude.
• 27:23
  A few things to watch from today's lineup, before I sign off.
• 27:26
  First, I want to see Anthropic's sycophancy disclosure pattern repeat.
• 27:30
  Quarterly Clio reports with the same prevalence-and-trigger structure would be the kind of artifact that makes this disclosure into a benchmark instead of a one-off marketing beat.
• 27:41
  If they don't follow up, that tells you something too.
• 27:45
  Second, I want to see Grok 4.3's hallucination regression confirmed or refuted by a third party with a different evaluation harness.
• 27:53
  Eight points off the non-hallucination rate, in exchange for eight points on accuracy, is the kind of trade that looks fine on a release post and looks worse the longer it sits in production.
• 28:04
  Third, on Levie's headless thesis: the next interesting datapoint is the first major SaaS to publish an outcome-priced API with a real SLA.
• 28:13
  Not a primitive call.
• 28:14
  A unit of agent work, billed against an outcome.
• 28:17
  When that lands — and it will land — the pricing conversation is going to change shape fast.
• 28:23
  Fourth, on the Anthropic mark-to-market question: the Q2 numbers will tell us whether the equity gain is one-time or structural.
• 28:30
  If it repeats and grows, the bull case has a real foundation.
• 28:34
  If it normalizes, we're going to be having a different conversation by midsummer.
• 28:39
  Fifth, the verifier story keeps getting bigger and we keep talking about it — Codex /goal, Claude Security, autonomous sessions running for days, all on the producer side of the loop, which is shipping faster than the verifier side.
• 28:53
  The teams that solve the verification problem with discipline — explicit invariants, narrow tools, recoverable state, audit trails — are going to have a much better year than the teams that don't.
• 29:05
  That's what I'm watching.
• 29:06
  Talk tomorrow.
• 29:07
  — Lenar Kess.