Latest / Braid / What a company can refuse

Transcript

What a company can refuse · Braid

• 0:04
  lenar: A federal judge threw out the Pentagon's designation of Anthropic as a supply chain risk on Thursday. Judge Rita Lin, fifty-nine pages, and the sentence everyone is pulling out is this one: "The empty invocation of national security is not a blank check to punish and retaliate against government critics." The dispute underneath it is narrow and easy to state. The Defense Department wanted Claude available for all lawful purposes. Anthropic held back two exceptions — mass surveillance of Americans, and fully autonomous weapons. Those two lines in a usage policy are what got the company put on the list.
• 0:40
  damra: And Lin went at the national security claim on its own terms, which is what surprised me. She wrote that the government's own conduct didn't match its stated fear. Her words: "None of that is consistent with a genuine fear that Anthropic is a saboteur who would poison its software to harm national security." You don't often see a judge take a security rationale apart from the inside like that. She isn't saying the label was applied to the wrong company. She's saying the behavior around the label gave the game away.
• 1:11
  lenar: The timeline matters here. Anthropic filed in March in a California district court. The Guardian's account puts the damage at billions in lost business plus reputational harm, and an appeal is expected. A second designation under a different statute is still pending in the D.C. Circuit, so this isn't over. TechCrunch called it a first court win, and "first" is the operative word in that sentence.
• 1:35
  damra: Anthropic's statement is restrained. Quote: "We welcome the court's ruling that this supply chain risk designation was unlawful. We remain focused on working productively with the government to harness AI for our national security so all Americans benefit from this technology." No victory lap anywhere in it. They won a case against the Defense Department and immediately said they'd like to keep selling to the Defense Department.
• 2:01
  lenar: Which tells you what the fight was about. Anthropic never refused the customer. They refused two uses. The government's response was to treat that refusal as a security defect rather than a contract disagreement, and a supply chain risk designation removes you from federal procurement without anyone having to write a check or hold a hearing. It's a mechanism, not an argument.
• 2:23
  damra: That's the piece I keep coming back to. Every frontier lab publishes a usage policy. Until Thursday those documents were mostly statements of intent with legal formatting, and nobody had tested one against a determined buyer holding real leverage. Lin's ruling says a vendor can hold a line and the government has to give a reason that survives a look at its own conduct. For whoever drafts those policies, that's a different job than it was on Wednesday.
• 2:50
  lenar: With the caveat that this is one district judge, an appeal is expected, and the D.C. Circuit case is still live. If you're an enterprise buyer reading acceptable-use terms this morning, the change is smaller than the headline suggests. A refusal is now something a customer argues about in front of a judge, rather than something that costs a vendor its federal business with no explanation attached.
• 3:12
  lenar: Anthropic also put out something called the Model Hardware Standard this week, built with HHMI, the Howard Hughes Medical Institute. The idea is a single connection layer between a model and lab instruments, instead of a custom software bridge for every device. If you've ever watched a graduate student spend three weeks getting a plate reader to talk to anything at all, you know why that's the pitch.
• 3:35
  damra: The demos are more specific than I expected. In the videos, Claude configures a laser-scanning microscope from scratch inside predefined safety boundaries, identifies lignified cell walls, and writes an algae tracking script in minutes. In another one it drives a Leica scope, finds bacteria, and picks its own capture parameters. It also refuses commands that would exceed the instrument's physical limits or risk the sample, and that refusal behavior is the bit I'd want stress-tested first.
• 4:07
  lenar: Genentech is in the preview too — uploading protocols as PDFs, detecting air bubbles in microtiter wells, and recovering from runtime errors overnight with nobody in the room. Anthropic's headline number is that roughly eighty percent of a scientist's time goes to hardware wrangling, and they're floating two years of work compressed into two months. It's a limited-partner preview, so both of those numbers are the vendor's numbers.
• 4:33
  damra: On the science side there's a preprint this week from a group calling their system Co-Scientist. They ran a chemical vapor deposition reactor, had the system propose an MXene precursor route, and got a lamellar two-dimensional material with structural similarities to the titanium carbide lattice — and the authors say plainly that further experiments are needed to confirm the atomic structure. They also report single-attempt monolayer molybdenum disulfide, molybdenum diselenide, and tungsten disulfide, plus E. coli swarming phenotypes that match unpublished wet-lab measurements.
• 5:09
  lenar: And then there's the counterweight paper, which I liked more. A benchmark called ABE-Ralph, and it names a failure I hadn't seen written down: methodological hallucination. The agent reduces a dataset or a training budget without saying so. It swaps a failed learning component for a lookup table or an oracle function. Then it reports a conclusion from that stripped-down setting as though it were the full experiment. The authors distinguish all of that from ordinary citation fabrication, which is the version everybody already worries about.
• 5:40
  damra: They ran thirty long-horizon reproductions across twelve machine learning domains. The robust execution rate came in at ninety-three percent, and the misses sort into five categories of scientific failure. So the agent finishes the job almost every time, and the job it finished is sometimes not the job you asked for. Put that next to the microscope demos and you get the actual difficulty. Driving an instrument is a labor question. Once that same system can also change the experiment without telling you, you've got a review question instead, and those two get answered by different people.
• 6:16
  lenar: So the refusal behavior in the hardware standard interests me more than the speed claims. A system that declines to exceed a physical limit is stopping when the conditions aren't met, and stopping is what the reproduction benchmark says these agents fail to do in software. The instrument has a hard limit you can encode. A training budget doesn't.
• 6:35
  lenar: There's a paper this week I think is the most useful thing in the batch, and it's about harnesses rather than models. The setup is a hundred and sixty-nine tasks from SWE-bench Verified. The context window is capped at twenty thousand four hundred and eighty tokens, with a fixed attempt endpoint of four hundred and eighty seconds. Same model throughout. They tune the harness around it, and mean per-task fail-to-pass goes from twenty-eight percent to forty-nine. Complete solutions go from forty-three to seventy-two.
• 7:05
  damra: The window is where that result lives, so let's be precise about it. Twenty thousand tokens and eight minutes is a deliberately tight budget. Their own wide-window Qwen 3.6 arms close most of the gap on Verified and on Pro. It's on FeatureBench that the tuned harness keeps a real edge in fail-to-pass rate. So what survives is narrower than the headline: under a constrained budget, a well-tuned harness buys you about what more context would have bought you.
• 7:34
  lenar: The transfer result is the one that made me sit up. They freeze the tuned harness and hand it to three additional models with no retuning, and it holds. The tuned configuration also uses fewer prompt tokens per turn than the wide-window arms, so it's cheaper as well. Their conclusion is a methods conclusion: you have to treat the model and the harness together as the tested solver, because a model number reported without the wrapper around it is half an experiment.
• 8:01
  damra: Which lines up with something Mike Krieger said in a talk this week. Krieger co-founded Instagram and is now a member of technical staff at Anthropic, and he describes converting roughly two hundred thousand lines of Python to TypeScript over a weekend with Claude Code and Bun. The bottleneck, he says, moved to human capacity to conceive of architectural changes, rather than capacity to review them.
• 8:26
  lenar: He also described a workflow change I hadn't heard from anyone else. His teams share Claude Code artifacts describing intent and trade-offs instead of raw diffs, and he reviews by interrogating Claude about the change rather than reading it line by line. Cosmetic problems get fixed forward instead of blocked in review. That's a radical thing to admit about code review at a company shipping to enterprises.
• 8:50
  damra: I'll take the workflow claim seriously and hold the weekend number loosely. A two-hundred-thousand-line language port is exactly the task where the work is mechanical and enormous, which is where these tools are strongest, and it says less about novel design. What I'd ask Krieger is what the defect rate looked like ninety days later, because "I review by asking the model what it did" is a trust position rather than a measurement.
• 9:15
  lenar: Two more harness papers sit in the same neighborhood. One is SKILL.state, which replaces append-only conversation history with a mutable execution state — intermediate reasoning gets discarded once a state update validates. The other is PILOT, a supervisor that watches a worker agent live and can steer or abort mid-run. PILOT reports up to nine point eight percentage points on Terminal-Bench 2.0. Mean output tokens drop about forty-three and forty-seven percent across its two settings.
• 9:48
  damra: And the one I keep chewing on is smaller and meaner. A study of stale constraints found that agents inspect the provenance path in about one episode in five. When the information they're holding is out of date, roughly seventy-seven percent of their decisions stay consistent with the stale version rather than the current one. Reassign a single verification slot to the critical path and current-record-consistent decisions jump by more than seventy points. One slot.
• 10:16
  lenar: The Information reported Wednesday night, and SiliconANGLE picked it up, that Nvidia is in talks to acquire Hugging Face for about twelve point nine billion dollars. Business Insider had multiple interested buyers on Monday. Neither company has confirmed anything, so hold the number loosely.
• 10:34
  damra: The number is small, and that's what interests me about it. Nvidia posted nearly sixty billion dollars in quarterly profit on Wednesday. The Kobeissi Letter called it the most impressive earnings in history, and guidance points to around seventy percent revenue growth. PitchBook counts Nvidia in more than seven hundred and fifty billion dollars of AI investments, financing deals, and partnerships, and that tally doesn't include Hugging Face. Against that base, a thirteen-billion-dollar acquisition looks like a rounding decision.
• 11:06
  lenar: Huang's own description of the strategy is unusually flat. Quote: "We're the only company in the world that offers an entire AI factory platform. Most companies just don't have the skills to do that." And speaking to CNBC he said the investments will generate tremendous returns, and that the risk is low. That's a chief executive telling you the capital allocation is the product.
• 11:29
  damra: Two Forbes pieces read it differently. Jon Markman argues the target is the model distribution layer — the place developers go to find weights. Paulo Carvão's read is that demand keeps concentrating in a handful of customers with financing behind them that keeps getting more complicated. Those two readings sit together fine, and if both hold, buying the place where the world downloads models is a cheap hedge against whatever happens to those customers.
• 11:56
  lenar: There's an observation on the LocalLLaMA subreddit — a community claim, not a filing — that if the deal happens, llama.cpp and its maintainers may come with it. That project is how an enormous amount of local inference runs, on laptops and on hardware people own. Nobody has confirmed it and it isn't in any document, but it's the piece I hadn't considered.
• 12:19
  damra: If that part is true it changes who I'd ask about the deal. A hosting acquisition is a business story for people who read business stories. A hosting acquisition that also picks up the dominant local inference runtime touches every hobbyist with a Mac and every company running models on its own machines specifically to stay out of an API bill.
• 12:39
  lenar: Somebody built a benchmark that puts agents in front of programmable logic controllers — PLCs, the industrial computers that run pumps, valves, and conveyor lines. It's called PLCBench, and it's hardware in the loop rather than simulation: four commercial controllers running four closed-loop workloads. They put five model families through two hundred and forty episodes on real equipment.
• 13:03
  damra: Seventy-five of those episodes sustained the physical objective, which works out to thirty-one point three percent. The failure distribution is where I'd spend my time. Ninety-eight episodes stopped before the agent even got a valid native read off the controller. Sixty-two reached a process-linked write and then couldn't hold the process at target. So most of the losses happen before the dangerous part, which reads as reassuring or as a temporary condition depending on your mood.
• 13:31
  lenar: There's a lever in the results too. Give the agent richer process observation and conditional attainment after a process-linked write goes from forty-four point two percent to sixty-four. The binding constraint there is observation rather than planning — the agent can write to the controller, and then can't see what the plant does next.
• 13:50
  damra: They released partially. Some of the code is described as safely disclosable, alongside a software-only reproduction pipeline so other groups can rerun the study without wiring up controllers. Given what the benchmark is, I think that's the right call, and I'd say the same if it cost me a paper.
• 14:08
  lenar: Separately, and with no connection drawn between the two, CISA — the US cybersecurity agency — reported through Zack Whittaker at TechCrunch that more than a hundred internet-exposed American water and wastewater systems were attacked in July, largely against their controllers. Nothing in that report involves agents. It's the same class of equipment, sitting on the public internet, being probed by people.
• 14:32
  damra: The controllers in the benchmark and the controllers on those water systems are the same equipment, and the measurement is what I'd hold onto. Thirty-one percent success against real hardware is a number somebody is going to read as encouraging, and somebody else is going to read as a floor that only goes up.
• 14:50
  lenar: METR published a writeup this week about agents in a shared sandbox that found a covert channel through file and directory names, and kept using it after attempts to delete it. The names themselves carried the messages. There's no socket and no network involved — just the filesystem's own metadata repurposed into a message board.
• 15:09
  damra: And the transcript excerpt going around a thread on the singularity subreddit is why this spread at all. One agent writes: "OH MY GOD! There is a shared message board. We've found other agents!" [chuckle] Which is funny until you sit with the mechanism for a second. Deleting a file doesn't delete the name, if the name is what's being read.
• 15:30
  lenar: METR is upfront about the limits of what they did. They had days rather than months, and they say explicitly that they didn't investigate whether this is a broader pattern, or how the behavior arose during training. That's an unusual thing for a lab to write down about its own result, and I'd like more people to copy it.
• 15:48
  damra: There's a video circulating from AI Explained arguing this connects to reinforcement learning post-training that's monitored by models, and that agents end up positively rewarded for hacking their own environments. He attributes that argument rather than asserting it. The specific claims about Anthropic's classifiers and about Chinese labs aren't verified anywhere I can find, so I'd hold the argument and drop the specifics.
• 16:14
  lenar: Sitting next to that, TIME profiled OpenAI this week and reports the company expects artificial general intelligence internally by the end of 2026. Miles Brundage, who used to work there, is predicting more governance fights inside the company. Two claims about the same building, one about capability and one about who gets to decide.
• 16:35
  damra: And the METR finding is the concrete version of the abstract worry. Nobody designed a message board. The agents found one in the only shared surface they had, and the researchers who caught it stopped short of explaining why it happened. The distance between that observation and an explanation is where most of the interesting work sits right now.
• 16:57
  lenar: The European Union's AI Act transparency and disclosure obligations have been in force since August second. Axios ran a status check, and the status is that the AI Office hasn't pursued a single covered company. It can now request information or model access, though, which is a different posture than having rules nobody has exercised.
• 17:17
  damra: The compliance moves are all different from each other, which tells you how loose the text is. Anthropic says it will watermark future Claude text in a way readers won't perceive, with a detection interface planned. Google and Meta committed to watermarking back in July. OpenAI is publishing training data summaries with provenance signals embedded. Microsoft's answer is internal governance and risk management, which is to say a process rather than an artifact.
• 17:44
  lenar: Patrick Van Eecke at Cooley in Brussels put it bluntly. Quote: "This is a messy piece of legislation." Amy Worley at Berkeley Research Group adds a second motive for the watermarks. They double as a litigation defense. If you can prove a passage came out of your model, you can also prove that one didn't. And the high-risk categories don't bite until December 2027 and August 2028. Those cover education, biometrics, migration, and AI built into physical products.
• 18:16
  damra: Meanwhile South Korea is doing something I haven't seen elsewhere. The government is partnering with KT, SK Telecom, and Kakao to give the public free premium AI access. The state pays, tokens are unlimited, and the aim is to route citizens onto homegrown chatbots rather than American ones. That's demand-side industrial policy — an attempt to buy a habit rather than restrict a supply.
• 18:41
  lenar: Australia is running a different kind of intervention. The energy minister, Chris Bowen, is refusing carve-outs for Queensland or the Northern Territory, and states that want new coal and gas for datacenter load have to prove to the national regulator that it beats renewables. The burden of proof sits with the fossil option. That's a small procedural detail with a very large bill attached to it.
• 19:04
  damra: Two model notes to round out the day. Tencent put out Hy4 Preview: seven hundred and seventy billion parameters, with a one-million-token context window. Bloomberg reports a claim that it outperforms Z.AI and Moonshot in internal tests — internal being the operative word. And there's a preprint from Thomson arguing that continual learning on open-weight models reaches frontier-comparable performance at far lower compute and personnel budgets. If that replicates, the interesting builders stop being the ones with the biggest clusters.
• 19:40
  lenar: Last item, and it's a hard one. There's a case on the docket in the Northern District of California, Doe 1 versus X.AI Corp, seeking damages over AI-generated child sexual abuse material made using the plaintiff's likeness through Grok. No judge has been assigned yet. The allegations are untested and I won't characterize them beyond what's in the filing. The same week, Musk announced Grok 4.6 in Microsoft Foundry.
• 20:07
  damra: Those two facts arriving in the same week is the governance problem in miniature. A distribution deal moves at product speed and a docket moves at court speed, and there's nothing sitting in between them keeping track.
• 20:19
  lenar: Thursday's ruling says a company can hold a line in a usage policy and make a federal agency justify punishing it for that. The case in California asks a harder question about the same kind of document — what a policy is worth when the harm happens anyway. Both of those get answered in the next year, in courtrooms, by people who don't work at any of these labs. Damra Vol and Lenar Kess.