Latest / 80,000 Hours Podcast / What everyone is missing about Anthropic vs the Pentagon. And: The Meta leaks are worse than you think.
Transcript
- Today we're releasing two essays by me back to back. First we've got one about whether Anthropic is being hypocritical, naive, or undemocratic in its dispute with the Pentagon. And then there’s a second short and sweet one covering some extraordinary leaks from Meta regarding their treatment of ads for frauds and scams, and what I think that demonstrates about regulating big tech, especially AI companies. Enjoy! I’ve spent years calling for more government oversight of frontier AI development. So am I a hypocrite for opposing the Pentagon’s attempt to commit “corporate murder” against Anthropic? That’s what I’ve heard. As venture capitalist Marc Andreessen put it on Twitter: “Every single person who was in favor of government control of AI, is now opposed to government control of AI.” It’s a natural way to think. But it’s also completely wrong — and I want to explain why it’s wrong, because you see the same underlying confusion all over the place. It’s not just hypocrisy though. I count at least three distinct charges critics have levelled at Anthropic and people who support them in their dispute with the Pentagon. I mentioned the hypocrisy charge, but there’s a separate accusation of naivety: that when you’re building something as powerful as AI, the state will inevitably crush you if you try to set conditions on their use of it, and so Anthropic was in the wrong to pick this fight. And third, there’s an accusation of being undemocratic: that a private company has no business telling the elected government what it can and can’t do with military technology. I’m going to take each of these charges seriously and explain where I think they go wrong and why. Let’s dive in. Let’s start with hypocrisy, because it’s the charge I hear most often and the most straightforward of the three. Just to quickly jog your memory about the dispute we’re talking about: Anthropic had a contract with the Pentagon that included two restrictions on the use of their AI: no mass domestic surveillance, and no decisions to kill people made by artificial intelligence alone. The Trump administration demanded those restrictions be removed, Anthropic refused, and rather than merely ending the contract, which people mostly agree would have been fine, Secretary of War and Defense Pete Hegseth declared Anthropic a “supply chain risk” — a designation previously used only for foreign adversaries — in so doing, threatening the company’s ability to do much business in the United States at all. The case to overturn that designation has attracted support from expected allies and unexpected ones alike — including Anthropic’s competitors OpenAI and Microsoft, as well as conservative technology experts who for the most part otherwise agree with the Trump administration’s approach to AI. For years, people in the AI safety world — absolutely including me — have argued that frontier AI is too important and too dangerous to leave entirely in the hands of private companies. We’ve called for government oversight, safety standards, a wide range of different things. So the hypocrisy logic runs: you wanted government control of AI. Well, now you’ve got government control of AI. So why complain? But to state the obvious: supporting public oversight of frontier AI training doesn’t require you to support the government strong-arming a company into allowing its product to be used for domestic mass surveillance. Those views just on its face aren’t even in tension. Of course, people don’t only care who decides, who has control over AI — they also care what they decide, and what they decide to do. I might support legislation that allows the leader of a city’s fire service to close streets for public safety, but if that fire chief starts closing arbitrary streets and demanding bribes to pass through them, I wouldn’t be a hypocrite for opposing that as well. The reason people get confused here is because they’re mentally compressing something extremely multidimensional down onto a single axis of variation. In this case, the axis being ‘more government control of AI’ versus ‘less government control of AI.’ But that’s an absurdly crude way to think about things. Everyone has always known full well that it doesn’t just matter whether the government gets involved; it matters how it gets involved, on what terms, and what it’s trying to accomplish when it does. You won’t be shocked to hear that supporters of AI regulation in general bicker endlessly among themselves about exactly what details here would be helpful versus harmful. Nothing this complex and delicate can be boiled down to ‘more government good, less government bad.’ I think the debate about the Anthropic/Pentagon dispute in particular got so abstract so fast in part because, for AI commentators, “Who should control AI?” is a much more interesting and important and generative topic than a narrow military contract dispute — and in part because for people who want to defend the Pentagon, it’s a hell of a lot easier to defend the general principle that the government should have some influence over AI than to defend what the Pentagon was actually doing in this instance. A phenomenon at play here is that there’s often a genuine tradeoff between what decision-making process seems best in the abstract, and our opinions about what outcome is best in a particular case. Say I think zoning decisions should be made at the city level — that that’s the right process. But the current mayor happens to be blocking all new housing construction, while I support doing the opposite. I now have a real tension: adopting the process I think is best as a general rule would produce an outcome I think is genuinely harmful, at least in the immediate term. Reasonable people can disagree about which consideration ought to win out in any given instance. But there’s no rule that says you’re only allowed to evaluate the process, never the outcome that the process would actually produce in real life. Noticing the tension between these two things, and weighing them against one another, isn’t hypocritical — it’s clear thinking. The second line of criticism is that Anthropic was naive or foolish to resist the government’s demands in this case. The most influential version of this line of argument comes from Ben Thompson at the blog Stratechery. Thompson’s reasoning runs roughly like this: Dario Amodei, the CEO of Anthropic, has very publicly compared advances in AI to nuclear weapons. Well, if AI really is that powerful, then any company building it is constructing a power base that could rival the US military. Realistically, no government is going to tolerate that. And Thompson also observes that international law is ultimately a function of power — that “might still makes right.” And he applies that same logic domestically, reaching a stark conclusion: Anthropic either had to accept a fully subservient position relative to the US government, or the US government would invariably try to destroy it. I’ll evaluate the two underlying arguments in turn: That the government’s actions here are so natural it’s absurd and counterproductive to object to them. Secondly, that the government was motivated by fears of a private company threatening their sovereignty. First up: Thompson’s piece opens with a long passage arguing that international law is essentially fake, that power dynamics determine behaviour at the international level. He then applies that same realist framework to Anthropic: the company is “fundamentally misaligned with reality” for resisting demands from the executive branch — which is, after all, so, so much more powerful than it as a private company. As a prediction of how powerful actors tend to behave, that may be correct, at least to a large extent. But the post then slips from attempting to describe reality into a prescription of what we should accept and how we ought to react to it — literally arguing that possessing overwhelming might can make your actions right. It’s actually pretty easy to accidentally equivocate between something being predictable on the one hand and it being acceptable on the other. And I think Thompson here was being very sincere and saw himself as pointing out some hard truths. And he did have useful things to say in that post. But it really is very dangerous to start seeing harmful or unlawful actions as unobjectionable, just because they’re not surprising or because they’re being done by very powerful actors. As to whether it’s naive and pointless to object in this case, that remains to be seen. Anthropic’s resistance has galvanised something like 90% of the tech industry to oppose the use of the supply chain risk designation for this purpose. Companies that have previously stayed quiet have been alarmed enough by the potential precedent that they’re joining the effort to push back. Past guest of the show Dean Ball, who wrote AI policy for the Trump administration, called the “attempted corporate murder” of Anthropic perhaps “the most [damaging] policy move” he’d ever seen the US government try to take — a high bar, surely. And using AI for surveillance or fully autonomous kill chains is even more controversial in Silicon Valley now than it was before — and it was already pretty controversial. The US is still a nation of laws for the most part, and legal analysts give Anthropic a good chance of prevailing in court, likely even securing a preliminary injunction to block the order before this video goes out. If that happens, the “naive” company will have established a legal precedent that helps protect the entire AI industry from economic coercion — a risky path to choose, perhaps, but not necessarily a stupid one. Second, in the rest of the post, Thompson gives interesting arguments for what views might sensibly motivate the kind of extreme action the Pentagon has taken here. But when you look at the specifics of this actual case, those motivations just don’t seem to be what’s driving things. If the US government had genuinely just become convinced that AI companies were building something as dangerous as private nuclear weapons, what might we expect them to do? Well, they’d presumably focus on the whole AI industry, not single out the one company that was most proactive about working with them and alerting them to exactly this risk. That OpenAI or xAI, that they’re offering them looser contractual terms on business contracts, that would hardly make them safe if the concern were really that a private company could very quickly accumulate military power to rival the entire US government. They’d presumably be thinking about rules to prevent something this explosive being built by incompetent idiots, or built in such a way that the designs get leaked to China. They would likely propose some legislation to Congress that would enable them to handle the many other issues that this is going to create down the road. And they would presumably try to keep AI researchers roughly on their side, not alienate them on an unprecedented scale over a relatively minor issue. But none of that is happening. Indeed, it’s generally antithetical to current US government policy. What is happening is that one company is getting punished for rejecting the government’s proposed terms in a contract dispute. Thompson’s high-level abstract arguments about AI’s transformative power justifying government intervention, they probably make sense. I kind of agree with them. But it’s hard to find evidence that that argument is what’s motivating the government’s particular actions in this dispute with Anthropic. Let’s turn to the third charge: that Anthropic’s position is an undemocratic one — that by setting conditions on how the military uses its technology, Anthropic is in effect usurping a role that properly belongs to elected leaders. The strongest version of that argument came from Palmer Luckey, the founder of military contractor Anduril. Luckey thinks the core two questions are: “Do we believe in democracy?” and “Should our military be regulated by our elected leaders, or corporate executives?” He goes on to argue that even seemingly innocuous terms and conditions — like “you cannot target innocent civilians” — involve difficult judgement calls about what counts as a civilian, what counts as targeting, and so on. And under Anthropic’s proposed framework, Anthropic would get some say on those questions — questions which really feel like government policy decisions. Luckey sees that setup as fundamentally at odds with democratic self-government. Luckey is pointing to a legitimate issue here. From the government’s perspective, it’s most straightforward for your military operations to be unconstrained by the opinions and moral qualms of your suppliers. Even to me, that practical worry is a potentially reasonable argument for the government to end their contract with Anthropic and look for other AI providers who are more straightforward to work with. The story is a little bit more complicated than sometimes portrayed though. In the current contract, Anthropic couldn’t cut the military off from Claude suddenly, even if they really objected to how it was being used. And the government has now accepted the same conditions for contract termination from OpenAI that were supposedly completely intolerable from Anthropic just a month ago. Plus, a far-sighted secretary of defence might welcome contractual barriers against certain uses of AI, not because they think they’d abuse the technology — presumably they’d trust themselves — but because they see value in guardrails that could limit a less scrupulous secretary of defence down the road. And if we’re making appeals to democratic self-government, it’s worth noting how the public feels about this issue. A YouGov/Economist poll found that Americans are nearly twice as likely to support AI companies restricting military use of their tools as to say the military should be able to use them however it wants. The public actually wants these restrictions placed on their own government. Is it really so democratic to deny the people what they want? But the fundamental issue here is a different one. Luckey is using the vague expression “believe in democracy” to equivocate between two entirely different things. Yes, democracy requires that the public gets to choose its leaders and that those leaders get to make decisions about national security. But no, it does not require that any private individual must supply their labour and their products for any purpose the government demands, on terms entirely set by the government, on pain of destruction. As one Twitter joker put it: “remember—you should do whatever the government wants, even things you think are immoral, because otherwise you’re deciding what you do instead of the government, which is undemocratic” Of course, the reverse is closer to the truth: being compelled to personally work on projects you oppose or face crushing government retaliation is clearly required for democracy to exist. And it’s undemocratic countries, like China and North Korea, where the state demands a right to make you offers you can’t refuse on any topic of their choice. There’s a common thread across all three of these charges. In each case, an abstract principle that sounds kind of reasonable gets invoked in a way that diverts attention from both common sense, and what’s actually going on: “You wanted government involvement in AI” becomes a reason you can’t object to any government action on AI. “Powerful states will inevitably assert control over powerful technology” becomes a reason we just have to lie down and accept whatever form that assertion takes. “Democratic leaders should make decisions about national security” becomes a reason no person or company can ever set terms when they sell things to the government. These questions are going to come up more and more, because AI really is becoming more powerful and governments really will need to be involved in governing it in some form or other. And as the debate goes mainstream, it could easily become a dumbed-down culture war where you’re either for government control or against it. But caring about precisely what the government is doing and whether it’s justified isn’t hypocritical, naive, or undemocratic. People should be proud to say that they care about the specifics and are actively pushing for the ones they think would be best. And they certainly shouldn’t allow themselves to be bullied into silence by the kind of mediocre arguments we’ve seen above. And with that, I’ll speak with you again soon. An overarching question that matters for governance of artificial intelligence is how much we can trust technology companies to do the right thing in cases where it becomes seriously costly for them to do so. Well, some internal documents leaked from Meta (the company behind Facebook, Instagram, and WhatsApp) are a really important piece of evidence in this debate — and one that, in my opinion, far too few people have heard anything about. These documents, which were leaked to Reuters by a frustrated staff member, show that Meta itself estimated that 10% of all its revenue was coming from running ads for scams and goods that they themselves had banned: around $16 billion a year. That’s 10% of all revenue coming quite often from, fundamentally, enabling crimes. Meta also estimated that its platforms were involved in initiating a third of all successful scams taking place in the entire United States. How much might we expect that to actually be costing ordinary people, on average? Well, the FTC estimates that Americans lose about $158 billion a year to fraud. If Meta is really leading to a third of that, we’re talking roughly $50 billion a year in losses connected to Meta’s platforms — or about $160 per person per year. Now, not everyone inside Meta loved this state of affairs, as you can imagine. And the documents also show that a Meta anti-fraud team came up with a screening method that, when tested, successfully halved the prevalence of scams being operated from China — at the time, one of their worst fraud hotspots. The only trouble? Meta was making $3 billion a year from these fraudulent Chinese ads. And according to the documents, after Zuckerberg was briefed on the team’s work, he instructed them to pause. Meta’s spokesperson disputes this characterisation. He says that Zuckerberg’s instruction “was to redouble efforts to reduce them all across the globe, including in China.” But whatever he said, what actually happened is that the China-focused anti-fraud team was disbanded entirely, and the freeze on giving new Chinese ad agencies access to the platform was lifted. Naturally then, within months, fraud from China bounced back to nearly its previous level. More broadly, the documents show that managers overseeing anti-fraud efforts were told that they couldn’t take any action that would cost Meta more than 0.15% of total revenue. Meta says this wasn’t a hard limit, but the maths kind of speaks for itself. If fraud accounts for roughly 10% of your revenue, and your anti-fraud team is capped at affecting 0.15%, their hands are really pretty much tied. There’s not a lot that they’re going to be able to do. A particularly grim detail about this is that Meta’s ad-targeting algorithm very naturally identifies people vulnerable to fraud, like the elderly or the desperate. And if they click one scam ad, it learns to feed them more and more until their feed is basically completely stuffed with them. Now, this all sounds pretty outrageous. And Meta’s own documents suggest the company realised that it was legally exposed here. They anticipated fines of up to a billion dollars for what they were doing, but the documents show them doing a cold calculation: “A billion dollars in fines. But wait, we’re making $3.5 billion every six months from high-risk scam ads in the United States alone.” That figure, one document noted, almost certainly exceeds the cost of any regulatory settlement involving scam ads. So those fines to Meta just became another cost of doing business. Rather than voluntarily cracking down, the same document finds Meta’s leadership deciding to act only if faced with impending regulatory action. So what are the takeaways for policymakers? First, and most obviously, penalties have to scale with expected profits and the probability of catching a lawbreaker in the first place. Meta could look at $1 billion in expected fines and shrug, because they were just making so much more money from running the fraudulent ads. If someone were to steal $1,000, we don’t fine them $100 and let them keep the other $900 — but that’s effectively how the law was written to operate here. Second, voluntary commitments should be taken with an enormous bucket of salt. Meta repeatedly made them very strategically just to buy time and stall binding regulation with no honest intention of following through in the end. The documents are quite explicit about that being a deliberate strategy. Another Reuters investigation revealed that Meta developed a global playbook for neutralising regulators, including manipulating their own ad library so that when regulators searched it, scam ads were quietly scrubbed from the results and would give a misleading impression about how common they were. So third, I think regulators need genuine access to internal information. If the only window regulators have into a company is one the company completely controls, they’re going to see whatever the company wants them to see. Thinking about all this has brought me back to an idea that I found myself repeatedly mulling over with respect to artificial intelligence governance. Compared to social media, AI technology changes faster, the potential consequences are larger, and the gaps between what companies know and what outsiders can verify is wider still. With AI systems, model outputs often have to be secret for good reason, much more secret than social media posts. And regulators can’t usually evaluate what a model is doing and why, and often even the company itself only vaguely understands. These systems are literally the first invention humans have ever made that can independently determine when they’re being tested, and have been demonstrated to strategically shift their behaviour to try to trick the very company that made them about their personality and likely behaviour once released. It’s honestly bonkers, and truly something new under the sun that we’ve never had to deal with before. Just in general, it’s a really complex situation to handle clearly, and we don’t yet know everything that we’re going to want companies to do to address it sensibly at a reasonable cost. That’s why some of the smartest people working in AI governance from across the political spectrum are converging on the idea that AI regulation needs to draw in some ways on how the Federal Reserve and other central banks watch over massive financial institutions. We’ll link to a major report on this that I can definitely recommend to you. It’s worth reading. But in brief, the Fed does not wait around for banks to collapse and then issue them a fine. It embeds supervisors directly inside systemically important banks — people who have deep technical expertise who remain there full time, watching how decisions get made, reviewing internal risk models, having constant conversations with leadership about what they perceive as emerging risks long before they can become crises. Dean Ball, who previously served in the Trump White House Office of Science and Technology Policy, and is nobody’s idea of a heavy handed AI regulator, he told me on this podcast that bank supervision is actually quite analogous to what we ultimately might need for AI. Now, this model, as you are probably thinking, it’s not perfect. The 2008 financial crisis happened despite embedded Fed supervision existing back then. But the status quo is a combination of no oversight, oversight by people who don’t understand what they’re dealing with, and oversight by technical experts who do understand but can’t access the information that they need to make the best decisions in time for it to actually matter to determine outcomes. The leaks from Meta are “fun” in a perverse way — because it’s always interesting to look inside a company and have our very worst suspicions about human nature verified. But waiting around for powerful tech companies to cause disasters before building an understanding of what they’re up to is a choice that we as a society can make or choose not to make. We’ll be trusting AI companies with something much more important than targeted advertising — and in my view, this time around, we should aim to be sophisticated actors, not total muppets who the companies can just easily run rings around. And with that, I’ll speak to you again soon.