Latest / 80,000 Hours Podcast / #227 – Helen Toner on the geopolitics of AGI in China and the Middle East

Transcript

#227 – Helen Toner on the geopolitics of AGI in China and the Middle East · 80,000 Hours Podcast

• Helen Toner: One thing that was reported was the UAE getting hundreds of thousands of next-generation Nvidia chips. And that is just an unbelievable amount of computing power. A big thing that the companies will say is if the US doesn’t sell it, then China will sell it — so that China is waiting outside the door: if the deal doesn’t go through with the US, they’ll just offer the exact same deal. Rob Wiblin: But China can’t even make the chips! Helen Toner: Exactly, exactly. So it’s totally disconnected from the reality of what they can do. The UAE is an autocratic country. Political parties are banned, they do mass trials of dissidents, they persecute the families of dissidents. The economy runs on immigrant labour; in the worst cases, they are essentially forced labour. There’s quite strict rules around what the media can say about the royal family. It’s a hereditary autocracy with a royal family that is going to stay in power. Rob Wiblin: So when one of these deals was announced for the data centre in the UAE, it was a collaboration, OpenAI was involved. They put out a press release saying this is a huge win for democracy. Is this just kind of boundless cynicism, or is there something else going on? Helen Toner: It certainly looks cynical to me. Rob Wiblin: Today I’m speaking with Helen Toner, the interim executive director of the Center for Security and Emerging Technology, or CSET, based here in Washington, DC. CSET is one of the most, possibly the most, influential think tanks doing analysis of the security implications of advances in artificial intelligence and suggesting potential policy responses. Helen is also well known for having been one of the board members of the OpenAI nonprofit back in 2023, and having been one of the four board members who voted to remove Sam Altman from his role as CEO. Thanks so much for coming back on the show, Helen. Helen Toner: It’s great to be back. Rob Wiblin: So you tried to remove Sam Altman from his role as CEO of OpenAI back in 2023. I think people are less confused now about why you and three other board members were interested in doing that than they were back at that time, when I think people didn’t necessarily understand what the motivation was. But I hear from people, and I imagine you hear from people as well, who say that even if the goal was a noble one, it was handled perhaps somewhat naively or amateurishly — and maybe there wasn’t enough forethought given to how staff will react, how investors will react, how the public is going to perceive and understand what’s going on. And as a result, you were perhaps caught flat-footed and didn’t actually manage to remove Altman in the end. I know there’s a lot of confidentiality issues around here which mean that you maybe can’t say everything that’s on your mind, everything you’d like to say, but what’s your reaction to that? Helen Toner: Yeah. I mean, I don’t think I can give a fully satisfactory account of all the decisions we made and why we made them. But two things I will say. One is I totally understand why people come to that conclusion. I really understand why it looks that way from the outside. I think at this point I’ve basically made my peace with the fact that, for people who see it that way, I don’t think I’m going to be able to convey why it looks differently from the inside. But what I will say is that there have been, at the time — and to some extent since, but especially at the time — a lot of reactions of, “Oh my god, why didn’t the board do X? Why didn’t the board think about X?” And for pretty much every single X, it was something that we were thinking about, we considered carefully, we had good reasons not to do. And in most cases I still stand behind those reasons. You know, it was a very complicated situation, it was a fast-moving situation. So I think it’s just probably not something that I will ever be able to fully convey my perspective to outsiders. Not just because of confidentiality, but also just because there’s so many little details of specific things that happened, specific people involved, best guesses and judgements about those, and lots of different tradeoffs. I will say for anyone who is interested in this, and who, since it happened, hasn’t seen some of the more information that came out about how it actually went down, I think that can also help explain some of the decisions that we made. So one place to look is an interview I did in May of last year on The TED AI Show, where I wasn’t able to share everything, but really tried my best to give a clear description of the basics of what happened. There’s also been these two books that have come out actually this year — one called *The Optimist by Keach Hagey, and one called Empire of AI by Karen Hao — and they both include reporting about what happened on the board. I don’t want to vouch for every single detail of their reporting, but I think both of them in broad strokes have it just about right. And in particular, Karen’s book gets into quite a lot of detail. So for people who are interested and confused and haven’t seen that material, that might also help make sense of it. Rob Wiblin: So the bottom line is maybe you made some mistakes, but if you think it was so obvious what to do, if you think it would have been so easy if only you were in that position, then perhaps you don’t fully understand how difficult the position was. Helen Toner: Yeah. I mean, we made hundreds of small decisions over the course of that weekend. I think the big decisions I basically stand behind. And there’s lots of ways that it could have gone worse as well. Rob Wiblin: I guess back in 2023, you and the other board members came in for some pretty harsh criticism, particularly from the tech industry, which I guess was really blindsided and confused and a bit dismayed by the decision, by and large. But my impression was that you got a much more sympathetic hearing in DC among policy folks, and also in the press and the media, because they were inclined to understand the situation pretty differently, as maybe a public power struggle within a company. Although it isn’t entirely a company; it’s actually a nonprofit. How has that played out for your career over the last couple of years? Do people kind of respect what you tried to do? Helen Toner: Yeah, I definitely see really wide-ranging reactions about what we did. Another big difference I see is between people who have experience in high-profile boardroom decisions and people who don’t. But yeah, what you said about tech industry versus more policy folks is definitely true. To speculate a little bit, I think on this coast, in this city, for one thing there’s much less of the cult of the founder — which is really incredibly pronounced in Silicon Valley — of the founder or the CEO is a genius and always correct, and the board’s job is to sort of stand behind and cheer. And that’s just not a cultural expectation here or an organisational expectation. I think maybe a little bit more inclination to not assume that companies are always doing the right thing, that probably contributes. I also think something I really noticed is just different expectations around how much information is going to be shared out of a sensitive and complicated situation. I think as well, maybe the most important piece about this, from my perspective, was that I was the only person on that board who really had strong professional roots outside of the West Coast. And I think that it’s not a coincidence that I’ve been the person who’s been able to talk about this a little more than the others, and I think it contributed a lot to how we were able to make the decision in the first place — basically not being vulnerable to having my reputation destroyed in Silicon Valley. At this point, I think my reputation has recovered somewhat, as you said, because of the changes in perceptions of our decision. But I think for many others in and around the company, they are just very vulnerable to perceptions in that one specific community. So the fact that I had networks and respect and people who know and understand and value my work over here that weren’t going to be subject to that was really important. I mean, I still meet people who know CSET and don’t know my name. So they know our organisation, they know that we do really high-quality work on AI and national security, and that’s what they know — which is very different from how on the West Coast I’m much more likely to meet people who only know about me from the board, and have no idea what I spend most of my time doing. Rob Wiblin: So while more people probably have heard of you because of OpenAI, you were on the board for two years, and it’s an unpaid role and a part-time role, definitely. And by far the thing that you’ve been working the most on since 2019 is the Center for Security and Emerging Technology. What should people know for the purposes of this conversation about what CSET is and what actually you do? Helen Toner: Yeah. So we were set up, like you said, in 2019. Actually the last time I came on the show was right as we had gotten set up. And I went back and looked at that interview, and it was interesting because at the time, to describe CSET was basically saying, AI and national security: these are big topics. Emerging tech and national security. We decided there should be a centre, that it’s really important — which at the time was kind of a novel idea. These days I think the way to understand CSET is that that is very much still what we do, but to explain a little bit more what makes us special: we’re based within Georgetown University, but we’re not academics. We don’t do the academic journal thing and don’t have tenure-track academics. We’re very policy-focused, very focused on what is going to be relevant and useful for policymakers and other decision makers. And in our work, we really strive to be independent, evidence-driven, and technically informed. So what do I mean by that? Independent: nonpartisan, not coming in with an agenda, not primarily there to do advocacy, but really to just tell it like it is. Evidence-driven: data-driven where we can be. We have a data science team who do incredible work. We have large data holdings of different kinds — data on papers and publications, financial flows, job postings. We have an in-house translation team, so we’re really focused on using data and evidence to show what is actually going on out there in the world. And then technically informed: meaning our work is really specialised on a small number of technologies — primarily AI, semiconductors, biotechnology. And that means that we can have people on staff who really understand those technologies really well. So an example, just to give a sense of our work, a piece that came out fairly recently was combining our data and China translation capabilities, looking at several thousand contracts from the PLA, the Chinese military, on AI and analysing it. We have more coming out from that dataset, but in this case it was looking at what are the organisations that are working with the PLA, and what does that tell us about China’s military-civil fusion strategy — which is a strategy they have to try and sort of bring together their public sector defence ecosystem and their private sector. And really interesting findings when you look at this big dataset of almost 3,000 tenders around the kinds of new and non-traditional vendors: universities that haven’t traditionally been linked with the defence ecosystem; or smaller, newer companies coming up that aren’t just the gigantic state-owned enterprises that have traditionally done Chinese defence. So that is the kind of work that we like to do to bring more clarity, shed more light on these issues, to help policymakers make more informed decisions. Rob Wiblin: One of the big influences that CSET has had is I think you were one of the groups that provided a lot of information and analysis data and suggestions that culminated in the imposition of export controls on semiconductors and semiconductor manufacturing equipment to China. I think that started around 2019 at the very beginning, and I guess it’s gotten more intense and now it’s been very topical this year. How do you feel about that influence with the benefit of hindsight? Helen Toner: There’s some technical nuances to unpack here. I guess we have time, so I’ll unpack them. So in CSET’s work, one of the first things that we looked at in 2019 was inputs into AI advancements. So we did a lot of work on talent, we did some work on data, and then we also looked at the compute, the semiconductors as an important component, and wanted to understand how does access to semiconductors affect AI progress? And it’s really important to distinguish between the two kinds of export controls you mentioned, namely semiconductors, the chips themselves, versus semiconductor manufacturing equipment — which is the gigantic pieces of machinery that get put in the fabs, the factories that make the chips. Our research really focused on the semiconductor manufacturing equipment, that supply chain. The reason that matters is I think there’s actually just a really straightforward, really solid case for why we should be controlling certain kinds of semiconductor manufacturing equipment. The most famous one for people who follow this space is EUV lithography, extreme ultraviolet lithography. These are these machines that are made by this one company in the Netherlands and flown over to Taiwan in multiple jumbo jets, because they’re these very complicated, large machines and no one else in the world can make them. And there’s other examples. Another sort of chokepoint is EDA software — “electronic design automation” is the acronym — which is primarily made by US providers. And the key things about these are: if you’re going to control something, if you’re trying to prevent someone from accessing something, and you could supply it to them and you want to put a control on it, the question is, what will they do instead? If you want to prevent a country from having light bulbs, you’re going to have a really hard time, because if you prevent them from buying your light bulbs, they’re going to be able to buy them anywhere else, so it’s kind of pointless. You’re just sort of shooting yourself in the foot. So the key thing about these pieces of semiconductor manufacturing equipment, like lithography machines and EDA software and so on, is if they really are a chokepoint — meaning there really is only one or a very small number of providers, and it’s not going to be easy to start a new company that can replace that — then you can potentially really slow down China’s efforts to build up its own domestic supply chain, which means that they stay dependent on US chips, which I think is just strategically very solid. It doesn’t actually really antagonise them very much at all or cause problems for their economy, but it does give the US and allies a significant strategic advantage. In 2019, that was what we identified at CSET, based on our research, and that has been controlled. It’s kind of a shame that semiconductor manufacturing equipment, or “SME,” as people in the field will say, those SME controls have kind of fallen out of the limelight or not been a major focus as there’s been so much debate over chips. We can talk about chips in a second if you like, but the stuff around chips is a little more complicated, a little more tradeoffs, a little bit different theories of change. And ideally, if you were going to dabble in that, you would keep the SME controls as this very obvious baseline that you’re really focusing on. But instead, because of limited resources, limited focus, I think those controls have actually not been implemented particularly well or particularly rigorously, while there’s been a lot of discussion about the chips instead, which I think is kind of a mistake. Rob Wiblin: You think machines are being smuggled into China? Helen Toner: I would have to go look at the details. There’s licencing, and there have been lots of licences granted that colleagues of mine who are deeper in the space are sort of saying, “Why are we granting these licences?” Things like that. So it’s not necessarily just smuggling, but also a question of focus on which actors get access to which things. Rob Wiblin: I’m surprised you said that you thought the export controls weren’t antagonising China. I would have thought they would feel like they’re being put in a pretty vulnerable position, security wise, and that maybe their economic ambitions are also being somewhat constrained by this. It’s also just a reasonably hostile move, you would think, to deny them access to these machines, to the chips as well. Am I understanding this wrong? Helen Toner: So again, we should separate out the SME from the chips. I was saying the SME is not necessarily very antagonising. I think the chips question is really interesting. It’s important to look at the backdrop here. I do think that the export controls on semiconductors themselves, China did not like that, obviously. The backdrop, though, was they already were saying in at least 2015, if not before, that they were assuming that the US was going to do this kind of thing, that they were assuming that they needed to indigenise, that the US was a hostile actor that you couldn’t trust, and that was kind of against them. And then in the first Trump administration, there were a bunch of actions around semiconductors and the tech industry more broadly — targeting Huawei as the well known case, but there was also ZTE, another company which was targeted beforehand. So already then, in 2018 I want to say, that was sort of confirmation for China of what they had already believed — which was that the US was going to behave in this way, and that they had to indigenise, and they had to be looking out for themselves. So I think it’s a real open question, what was the marginal effect of the different controls? I would say that the marginal effect of the SME controls was probably not very large. The marginal effect of the chip controls may be a little bit more debatable, certainly some marginal effect. But I think sometimes I hear people in this space who are very focused on AI and not focused on the US-China relationship acting as though that was like a bolt from the blue, gigantic hostile action, and often nothing was going wrong in the relationship. I think that’s not right. I think it’s a little bit trickier to say what was the effect. And notably, China’s reaction — I mean, there’s now been multiple years of a little bit of back and forth — but certainly at the time the reaction from China was actually not very retaliatory, was not very escalatory. Which I know was another thing people were concerned about, was will this create a huge backlash? So I don’t want to say it was all kumbaya and no problem whatsoever, but I also think it’s possible to overstate. Rob Wiblin: A strange thing that’s happened this year is there was this big debate in the US about whether the US should be selling or allowing more Nvidia chips to be sold to China. People would argue it’s bad to do that just for the obvious national security reasons, and other people would argue back that no, actually it’s good — because what we want to do is maintain China’s dependence on these chips, rather than basically force them, and indeed encourage them, to develop their own local semiconductor industry and give that industry a big boost. And then recently China made the decision to not allow the Nvidia chips that the US was debating whether or not to allow into China itself. So in a sense, China has imposed export controls on itself, at least on these particular Nvidia chips. Can it be the case that it’s both a smart move for the US strategically to deny China the chips and a good move for China to deny itself the chips? It seems like it’s a reasonably zero-sum situation, so how can it be in both of their interests to do it? Helen Toner: Yeah, lots going on here, I think. So one immediate question is, does China really mean it or is it an ambit claim? Is it a bargaining position? I think at this point, most countries around the world know that President Trump loves a deal, and loves making gigantic opening bids on deals and then retreating to something more reasonable seeming. So I think it’s very possible that this is just sort of mostly signalling from the Chinese government, saying, “You’re going to have to beg us to buy your chips, and we don’t care anyway.” And so far, I think the effect of that rhetoric on the US debate does seem to have been, “We should just let them buy them.” If it is a move by China to be allowed to buy the chips, it seems like so far it’s working. So that’s one possibility. Though I think this broader thing, and also what you said about how the US debate has shifted, really shows how there’s been very little clarity and very little agreement on what the goal of these export controls is, and what theory of change is. I think partly that comes from the fact that this is a complicated technology, and it relates to AI, and AI is general purpose technology with lots of different components. I think partly it comes from this situation in US policy over the last few years, where one of the few things people could agree on was essentially “China bad.” So if your policy was like, “China bad, and therefore we’ll do something that will hurt China,” then people could get behind that without necessarily needing to have a very clear position on the nuances of the policy goals, and how you would tell if they were succeeding. You know, the initial messaging around this was really about China’s use of chips for military applications and for human rights abuses. And astute observers pretty quickly were like, hang on, military applications: a lot of what you need for that is going to be much smaller chips onboard. If you’re having something on a drone or on any kind of equipment, you don’t use these gigantic data-centre-style chips, you use something much smaller. So if we’re primarily targeting their military, this is kind of a weird way to do it. Likewise, human rights abuses, that’s often going to be like image recognition, speech recognition, other things that you don’t need these gigantic clusters for. So then the rhetoric has shifted around to how it’s actually about China’s ability to build very advanced AI systems. I think that’s the most sensible, or that one actually makes sense — if that’s what you’re trying to prevent, then preventing large accumulations of the most advanced chips is a good way to do that, or a reasonable thing to try at least. But then now the rhetoric has shifted again to what “winning” means. And David Sacks, the White House AI and crypto czar, has said this very explicitly: what winning against China means is having more market share in chips. So as long as we’re selling them chips, we’re winning. And that’s just a very different theory of change from “winning” means that we are using the chips domestically to build something that gives us a strategic advantage that they’re not able to build, or that helps our economy because we have this broad base of computing power that they don’t have, so we have some kind of strategic advantage. So there’s really a lot of confusion and a lot of almost rug pulling over what exactly are we doing here, and how will we know if it’s working? Rob Wiblin: Yeah, I’ve heard this a lot this year: that it’s really important for the US strategically to get the rest of the world to start using its AI models and to be part of its AI stack. To me, it’s not intuitively obvious that that is a national security priority, or that it really does make a big difference whether someone in Australia or South Africa or Brazil is using ChatGPT or whatever the Chinese model might be. A cynic might say that this is in the AI industry’s interest to convince the government that it’s really important that they provide support so they can sell their products more and compete for market share. Why wouldn’t they try to persuade people of that if they can get away with it? Does this argument make sense to some extent? How strong is the argument, and how should I think about it? Helen Toner: Yeah, I think the version of it that makes sense to me and something that colleagues of mine at CSET wrote about is basically as a soft power play, meaning that if you look back at the history of technologies and who’s providing what, there’s just a lot of cases where you get some kind of broad, diffuse soft power benefits from being the provider of a key technology. A different example would be Hollywood and American music. Why exactly is it good for America that everyone around the world enjoys Hollywood movies and listens to American music? It’s sort of hard to put your finger on it, but I think it is clearly good. That’s one of the canonical soft power examples. Likewise, it does seem right to me that it’s going to be better for a wide range of reasons in terms of influence and standard settings and expectations and relationships if American AI is used around the world. But a big caveat there is: I think if that is your focus, then it doesn’t necessarily make sense to focus on these frontier systems — meaning computing clusters that have hundreds of thousands of the most advanced chips and are being used to train or run the very most advanced models. I think if you actually talk to people who are going around and trying to sell an AI stack or get government partnerships with AI stacks, often what people want is more of a turnkey solution of like, “What is this going to do for me?” And there, you often don’t necessarily need huge amounts of computing power; you don’t necessarily need the very most advanced models. Instead, maybe you need some kind of support to figure out how to use a model for some particular use case that’s interesting to you. Or if we’re talking government usage or large enterprise use, the support you maybe need is, “How does this fit in with our existing IT procurement policies? And what is the UX? How should we build it into our existing offerings for our staff?” Or if it’s a government, “How should we fit this into our online digital government initiatives?” And that’s much more the kind of thing where you need embedded software engineers to go in and understand the problem and help make progress, as opposed to the kind of thing where you really need hundreds of thousands of chips. So I think the argument is basically reasonable, and I understand why people coming out of the tech world want to say, “Let’s do this full stack, American AI stack approach,” but I think it’s sometimes used for this like, “…and therefore we should let them build their own world class supercomputers” — which I don’t think follows. Rob Wiblin: Yeah, something that’s been even more confusing about this argument to me is I think people will make that argument and then say, “…and so it’s very important that we have leading open source AI developed in the US.” But I think if Meta open sources a really good model, so anyone anywhere in the world can use it and run it on their own equipment, then they can just fine-tune it to have whatever kind of properties, whatever personality, whatever values they like, and then just run it locally on some equipment that’s not in the United States. How does this really help the US’s national security goals? Helen Toner: Again, I think from a soft power perspective it maybe makes more sense. I also think that often people in an open source ecosystem will kind of engage and go back and forth. They don’t just grab the open model and then run away and hide in a cave and do their own thing with it. Instead, there’s often more of a rich back and forth and engagement. So I tend to think I am pretty in favour of there being more of an effort to produce really high quality open source models. I signed onto this project called the ATOM Project — the American Truly Open Models project — by Nathan Lambert, who’s a researcher at AI2 in Seattle. The key thing in my mind is actually separating out two different questions: should we be trying to have leading open models, versus should we be trying to open source or open weight the most frontier models? And I think those should be separate questions. So I think that it does make sense for US companies to be trying to offer models that are at least roughly as good as the best open models, openly. But I don’t think that necessarily means they should be racing to open source their very best models. Rob Wiblin: Just because it creates security risks? Helen Toner: Yeah, yeah. I think that my overall stance on open source is to think that, starting point: open source is great; sharing everything as widely as possible is great, and has lots of diffuse benefits that are hard to count up but really meaningful in terms of accountability and access and broad benefit. But the main exception in my mind is these frontier models, the most advanced models, because we just understand them least well, and we’re least well positioned to mitigate any new risks that they pose. So just having some amount of, the phrase that’s gotten a little bit established is “precautionary friction,” meaning having a little bit of lag — having six months, 12 months, 18 months to test and understand those systems, to figure out what can they do if you really push them to their limits, what kind of unexpected effects emerge? And having that happen in a time where you can still pull them back from the API, do some fine-tuning, put some new safeguards on, rather than having it be fully open. Rob Wiblin: The rhetoric in DC is really around the US being in a race to develop AI and AGI against China. Is China actually racing to build AGI? Helen Toner: It’s a great question. We actually recently put out a short post on this, because there have been these two recent op-eds — one by Eric Schmidt, one by Jack Shanahan, both very smart and well-informed people — saying the US is doing it wrong because we’re focusing on AGI, and China is doing it right because they’re focusing on applications and diffusion. And our researchers, William Hannas and Huey-Meei Chang, who are two of our most senior China researchers at CSET, put out a post saying that’s actually not right. Actually China is doing all of the above. So it’s definitely true that China’s big push right now, or one of their big government pushes, is called the “AI Plus” plan. And it’s “AI Plus” because it’s AI plus some other sector — so AI plus manufacturing, AI plus healthcare — looking for those intersections and those useful applications. So that is a big focus, but they can walk and chew gum at the same time, and they are continuing to also emphasise AGI, general purpose AI. So some wrinkles here. One is that in Chinese, the word for AGI is the same as the word for general purpose AI. So it’s just 通用人工智能. It means general use. It’s very unhelpful. In the US, or in the West, or in Anglophone discussions, we have for a long time talked about AGI, and then after ChatGPT came out, this idea of general purpose AI kind of got established — which is this more mundane thing that can be used for many things, but it’s not the full-on human level or building towards superintelligence. In Chinese, it’s the same word, so it’s a little bit unclear. That being said, they do sometimes just use the English letters “AGI” and they do sometimes talk about superintelligence. I think it’s very clear that they are right now pushing for both. Something that’s less clear is the question of how I don’t know of a better term than “AGI pilled” Chinese leadership is — meaning they’re really all-in on AGI is going to be an enormous deal, and it’s going to happen soon, and the way to do it is to scale up. The best thing that I’ve seen on this is actually from Jordan Schneider at ChinaTalk, who has a debate posted on his Substack. I think if you say “ChinaTalk AGI debate” or something, it will probably come up on Google. It’s basically formulated as a debate between a believer and a sceptic — the believer saying that obviously China is very AGI-pilled and here’s all the evidence, and the sceptic saying obviously they’re not. I tend to come down with the sceptic on that debate, meaning I don’t see the evidence that they are really gunning for AGI or superintelligence in the same way that the leading US companies are, or the leading Chinese companies — I think DeepSeek is certainly very AGI-pilled as a company. So there’s degrees of what you could mean for, “Is China racing towards AGI?” I also don’t think the US government right now seems very AGI-pilled, and I think that could be for good reasons. I think there’s a lot of assumptions bundled into what that means that may actually not be correct. So I don’t want to say they’re not AGI-pilled and that’s a mistake. I think there’s reasons to have lots of space for different views on that. Rob Wiblin: So DC has a pretty hawkish attitude towards China, I would say. Do you think this is kind of appropriate, given how China has behaved? I think my worry is it’s become such an easy thing to say, “We’ve got to do X or Y things that I liked anyway… because of China.” And it’s kind of uncomfortable for people to push back and say, “Is it really necessary to be this antagonistic towards China?” When something becomes that degree of conventional wisdom, and so easy to say and so awkward to oppose, it basically never gets challenged and just gets accepted by everyone, despite the arguments perhaps being weaker than they seem. What do you make of that? Helen Toner: Yeah. I mean, I think groupthink is bad. I think people feeling that they need to self-censor is bad. I have good news for you: I think over even just the past three to six months, this has been softening a bit. I think President Trump’s stance on China is pretty unclear, shall we say, and/or maybe somewhat fluid, and that has created some space. I think also certainly in the AI side, the top folks advising him on AI seem much more interested in commercial engagement with China, selling US chips to China in a way that is not as compatible with the traditionally China hawk stances. So I think it’s really valuable if there can actually be many voices in the debate, and many perspectives. And I think there’s been a little bit of a lack over the last few years, maybe, of creative thinking about what are all the different ways that we could relate to China. You know, it all gets collapsed into hawk versus dove, which is maybe sort of a callback to 20, 30 years ago of, do we assume that China is on this path to being a responsible stakeholder, and a more politically liberal, nice, friendly country; or do we assume that they are on track for war with the US, and then we have to be maximally hawkish? And of course there’s a very wide range of possibilities in between, so I think it’s valuable to be able to consider a wider range of options. Rob Wiblin: Something I’ve been a bit confused by is it doesn’t seem like there’s any direct diplomacy going on, trying to move us any closer towards having some sort of treaty or agreement with China on governance of AGI, preventing AI being integrated into military prematurely before either country feels comfortable that they actually have a grip on this technology and understand its pros and cons — or let alone a treaty to govern superintelligence and say that maybe we should be cautious about that one, even if we’re barreling ahead on all the applications that we want in the economy right now. Am I right that there is very little inter-country discussion of that? And if so, do you think that’s a mistake? Helen Toner: I think you’re right. Especially at what gets called the track one level, meaning official government-to-government talks. I think there’s a few reasons for it. One basic reason, one sort of AI-specific reason is I don’t think that the groundwork is there in terms of concern about this as a problem, or concern about this as a major problem to prioritise. What is AGI? What is superintelligence? Are they things we could build ever? Are they things we might build soon? Would that be good or bad, and for whom? I think there’s, especially at the government level, very little agreement on those questions — in contrast to perspectives in the AI safety community, for example. Rob Wiblin: Or the companies. Helen Toner: Or many of the companies. That’s right. Well, in the companies, it’s, “We can build this; we will build it soon.” And then in the AI safety community, it’s, “…and it’s going to be really bad, so we should be talking about it.” But I think that is a relatively insular set of beliefs still. That’s kind of the AI-specific reason that the groundwork is not there. I think it’s also really important to look at the broader relationship as well, and understand that from the perspective of a diplomat — or certainly of a president or a chairman — this has to slot in with everything else that is going on in the relationship. So what else is going on in the relationship? Well, one thing is the US and Chinese governments are barely talking at all. Over the last couple of years there have been some direct country-to-country talks on some small number of issues, but they’ve also often been completely suspended. So for a period, I think it was after Nancy Pelosi went to Taiwan, I think just basically all talks were suspended. Rob Wiblin: Isn’t that just crazy? It’s mind-blowing to me. This is the two most powerful countries in the world. They’ve got so many things to talk about! Helen Toner: Yeah. My understanding of this is that usually the US is the one that has more appetite to talk. And China knows that, so that becomes a bargaining chip for China to say, “We don’t want to talk to you. We’re not going to do these military-to-military talks about extremely sensitive, important issues, because we’re mad. And if you want us to do them, then you have to give us something in return to come back and join these talks.” Rob Wiblin: And I guess you don’t want to give into that. Helen Toner: Well, yeah. I mean, if the whole point is that it’s mutually beneficial, then we don’t want to be acting as though we’re making a concession, or that they’re making a concession by letting us talk. There’s a lot of context here, and a lot of baggage. A couple of other pieces of context and baggage I would give: one is I think a fair amount of scepticism among US diplomats. Diplomats in general are usually pro-engagement, pro-negotiation, pro-conversation: that’s why they become diplomats. I hear among US diplomats a lot of scepticism about the value of that with China, based on their track record of what happens when we do that. An important example is in 2015, President Obama and Chairman Xi had a long discussion about this problem of China spying on US companies and stealing trade secrets from US companies, which is different from the long established “countries spy on each other for strategic reasons.” China was doing something different, namely corporate espionage — where they take trade secrets, benefit economically, hand things to their companies: very much not OK, very much out of the norm internationally. In 2015, at the end of a long series of discussions, there was a big deal signed between Obama and Xi saying that China was going to stop doing that. And the consensus is that they basically started a few months later again. They stopped very briefly and then restarted. That’s one emblematic example that’s in cyberspace. A lot of the people who do AI stuff have previously worked in cyberspace, so that’s a very salient example for them. There’s also scepticism about are they a good negotiating partner. Another example I would add there is the big example of track one government-to-government talks on AI in Geneva last year, this sort of initial foray into this conversation. My sense is that that didn’t go terribly, but it also didn’t go great. And part of the reason — which again is sort of emblematic of US-China negotiations more generally — was the US sent over some really technical, well informed, some of their best AI policy people to have quite in-depth conversations. And the Chinese sent their America specialists — so their US diplomats who knew almost nothing about AI, who were just specialists in trying to handle the Americans. The term that sometimes gets used is they sent their “barbarian handlers” — meaning they specialise in going out there and playing nice with the foreigners. So again, I don’t think that dialogue went terribly, but it wasn’t a good start. And again, it suggests that the groundwork is not there in terms of this actually being a priority that the Chinese government actually wants. In contrast to the default explanation for why would China agree to talks is because it makes them look like a great power, and they’re up there with the US having bilateral talks on AI. Doesn’t that show that they are doing so well at AI? So that is also a motivation for them to talk, and if that’s why they’re there, then it’s really not going to be very productive. Rob Wiblin: I mean, they are a great power, and they are doing pretty well. Helen Toner: Sure, yeah. But if the reason they want to be there is to be able to brag about it, then you’re not going to make much progress on reducing risks. Rob Wiblin: So you would say it’s mostly on China that there aren’t more negotiations along these lines? Or some shared responsibility, but more on China than the US? Helen Toner: I think there’s plenty of fault on China, yes. Rob Wiblin: Let the record show that I’m disappointed in China over that. You can tell them I’m frustrated. Is the US on track to stop its best AI models being snatched by China at the last minute, to have the weights exfiltrated and then just used by the Chinese military? Helen Toner: No. Rob Wiblin: What should it do that it’s not doing? Helen Toner: I think the best hope right now is that, to the extent that China perceives the gap between the best open source models and the best closed models to be small, they might just not invest in trying to steal weights or steal models. And I think that is a pretty widespread perspective right now. So maybe they just won’t bother, basically. I think that’s the best hope. But I don’t know, man, cybersecurity is really hard. Rob Wiblin: But isn’t there a bit of a contradiction if you’re saying we’re kind of racing with China, we’re trying to get to AGI first — but also it’s all going to be so close together they’re not even going to bother to steal the thing that we’re making, that we’re putting so much effort into? How can these things coexist? Helen Toner: I don’t think it’s that clear that we’re racing with China towards AGI. I think there’s sometimes a set of background assumptions here around even the language of a race. I’m perfectly happy to say that we’re competing with China on AI. The thing with a race is there’s a finish line, and whoever crosses the finish line first wins for real. And if you cross the finish line second, then that’s useless and doesn’t matter. And I think sometimes, in some AI circles, there’s an assumption that AI or AGI really is a race with a clear finish line — and the finish line is whoever builds self-improving AI first. Because then, if it’s true that once you get to a certain level of AI, then your AI can improve itself or can improve the next generation and you have this kind of compounding improvement, then that could genuinely be a situation where whoever gets to a certain point first then ultimately wins. I don’t think it’s at all clear that is actually what it’s going to look like, versus the systems get more and more advanced, they’re used in more and more ways, they’re sort of diffused through multiple different applications. And in that case, I think we’re in this state of ongoing competition with China, but not necessarily a heated race, where whoever is a hair ahead at the very end at the finish line ultimately wins the future or something. I think the shape of the competition is actually pretty unclear, and when people treat it as though it is very obviously just a winner-take-all race, that is a pretty risky proposition — because it implies that certain kinds of tradeoffs and certain kinds of decisions are obviously a good idea, when in fact I think it’s not at all clear. Rob Wiblin: So two years ago, my memory is that the attitude was we’ve got to tighten up security, we’ve got to make sure that they can’t steal the weights. It sounds like you’re saying there’s been a bit of a shift in attitude, that people are more fatalistic now. Helen Toner: I mean, I think even at the time, the trouble is that stopping a sophisticated state actor from stealing anything online is really hard. Or not even online: any kind of digital infiltrating, any kind of digital system: really, really hard. The best known work on this is by RAND; they have their security levels that they came up with. I do think that it’s very clear that we should be trying to make it harder for a wider range of actors to steal advanced AI models or algorithmic secrets or other proprietary information. Because if you are just nihilistic about it, then maybe it’s not just China, but it’s also North Korea, and it’s also a terrorist group or it’s also a disaffected young person who uses, not this generation of ChatGPT, but two generations from now ChatGPT to carry off a somewhat sophisticated cyberattack. So I do think we should be very much focusing on improving security. But I also think it is going to be a really big lift to try and actually make these systems resistant to sophisticated state-based hacking attempts. Rob Wiblin: If that’s the case, it does seem like it really undermines the idea that we have to train the best model in the US. Because if it actually does provide a big strategic advantage, we can fully expect that China will have it pretty soon after it’s trained, or we can think that that’s pretty likely. The thing that would remain is you want to ensure that there’s more compute in the US, that China is denied the ability to do massive amounts of inference. Even if they can steal the weights, that could still provide a strategic advantage — but if they need to train the thing urgently to get ahead, that argument feels a lot weaker. Helen Toner: Again, I think it’s this question of what is the shape of the competition, or what are the parameters here. Maybe another way to say it is: if it is purely about who has the most advanced model, then certainly the fact that China will probably be able to steal the most advanced model makes it less important where it is developed, or makes it less beneficial to be racing to develop it first. But as you say, there’s also a question of deployment. There’s also maybe a question of, if part of why you want the most advanced model is to be able to build more advanced models, then maybe you need the computing base for that. I think this is reflected in some of the compute modelling in the AI 2027 scenario, for example. In that scenario, China steals a model, but then they can’t actually use it to make as much progress because they don’t have as much compute to make the progress. Again, that’s baking in a bunch of assumptions about what progress will look like and what kind of advantage will accrue to which actors with what resources — so I don’t think it’s like a knockdown argument to say that because China can steal the models, we shouldn’t be trying to have the best models. But I do think it should at least temper that claim, or should be always in the foreground when we’re thinking about to what extent does it make sense to move fast — especially if we’re saying, well, we have to move fast, even if it’s risky, because we have to get there first. Then it’s like, well, which risks, and how are you weighing them against the possibility that your model will just get exfiltrated? Rob Wiblin: A few years ago you wrote that you thought China was two to three years behind the frontier on AI. That feels like a lot now. Now I imagine that the number is lower. What would you say it is now? Helen Toner: Yeah. So this was two years ago at this point, in a piece in Foreign Affairs. The point of that piece was actually to say that we shouldn’t use the concern that China will race ahead as a reason not to regulate our AI sector. I still very much stand behind that. And there’s a bunch of other things in the piece that kind of back that up. But yes, the estimate we used there, which was actually coming from some Chinese investors and folks over there, was two to three years. I think that’s clearly not right anymore. At the same time, I mean, DeepSeek had its big moment in January, it obviously made a huge splash. And in the wake of that, I saw some people pointing back at this piece and being like, “Oh my god, two to three years. What a joke. They’re right exactly on our heels. There’s no gap whatsoever.” I think that is also overstating the case. I think the smallest gap that we have observed is three months — which is the time between when OpenAI put out o1, its first reasoning model, and when DeepSeek released R1, its first reasoning model. That was very impressive. I was surprised. That was really well done by DeepSeek. But Miles Brundage, for example, who’s a former OpenAI researcher, at the time said that the way that this works is this is a new paradigm, this reasoning paradigm, and right now they’re at the very low-compute, earliest stages of it. This is going to be the easiest instance to recreate this o1 level. And since then we’ve seen OpenAI scale that approach to create o3 and o4-mini, and I think also o4, but I’m not sure: is o4 now GPT-5 thinking? Maybe. Unclear. And China has DeepSeek, which is essentially the leader in the reasoning space. Maybe Alibaba’s Qwen series has something that could compete. To the best of my knowledge, they have not previewed anything as good as o3, which confusingly is the second generation of the OpenAI models — because o2 is a telecommunications company, they didn’t want a trademark dispute, so OpenAI put out their second iteration. They previewed it in December and they released it in January. We’re now in September when we’re recording this. And as far as I know, no Chinese company has previewed a similarly sophisticated model, which means we’re now at something like a nine-month gap, and that might be longer depending on how long it takes them. And also it’s not clear how to account for o4, which is the third-generation OpenAI model, which certainly they don’t seem to be competing with. So it’s all very murky, and it’s hard to put an exact timeline on it. I’ve been saying something more like six to 12 months now. I could imagine pushing that out longer, depending on how long it takes to have that competitive with o3 performance. It’s also hard to compare because OpenAI is releasing this as a product, as a full system. So o3, for example, is well known for being really good at tool use. And DeepSeek or Alibaba would put out an open source model which is a little bit different, so then how do you think about the tool use? So it’s very hard to compare. But the short answer would be right now I would say something like six to 12 months, probably. Rob Wiblin: CSET was years ahead of the curve, I would say, in recognising the national security implications of AI advances, and probably years ahead of the curve in realising that export controls were going to be a huge deal. Is there anything you’re doing now that could be the next thing where CSET is ahead of the curve, where other people haven’t appreciated just how important some issue is? Helen Toner: We haven’t actually spun up work on this yet, but the one that comes to mind for me is robotics and advanced manufacturing, which is sometimes treated as two separate topics, but I think is very closely related — basically producing really sophisticated industrial machinery at scale, which could include robots or could include other things. This is something I want to learn more about, I want to look into more. I think sometimes the conversation about manufacturing in the US can become too much about manufacturing jobs, and that neglects the strategic importance of having really sophisticated, high-volume manufacturing capabilities here from a strategic perspective of what we can actually produce domestically, or among our allies and partners. So a thing that I’ve been thinking about is — I forget where I got this term; it’s not mine — this idea of an “industrial explosion,” meaning if AI is actually succeeding, will we see a huge buildout of energy, a huge buildout of data centres, a huge buildout of increasingly advanced robotics and the manufacturing you need to produce them? And I think right now China is really well poised to make the most of that if things do go in that direction. So there are some interesting questions around what does the trajectory of that technology and that buildout look like? And what would it look like for the US to position itself better? Rob Wiblin: Despite being pretty bullish about AI, the Trump administration has done a bunch of stuff that I think is kind of inconsistent with wanting to win any sort of AI race. It’s made it more difficult for people to immigrate to the US, including skilled workers, and generally just seems kind of hostile to scientists and students coming over. It’s been negative on rollout of energy generation, at least renewable energy generation. The tariffs have made it more difficult for the manufacturing sector in some ways because it increases the cost of industrial inputs, basically, of the machinery that you might import from overseas. I guess they had that raid on the Hyundai battery factory. I think Trump then said that he actually didn’t support that, so maybe they kind of rolled that one back. But it feels like there’s a bit of a contradiction here. There’s a bunch of stuff that you probably would be doing, like encouraging high-skilled immigration, if you wanted to be as far ahead of China as you possibly could be. Those things aren’t happening; indeed, it’s kind of going the other way. How can you make sense of that? Helen Toner: I make sense of it by there being sort of different factions inside the Trump administration — different specific officials with different agendas and different priorities — and them not necessarily coming together into a coherent policy vision. So for example, on immigration, that was one of the topics CSET looked at earliest. And when we were in the space in 2019, there was sort of this common wisdom that high-skilled immigration was good economically because it benefited US companies and helped the economy grow, but it was only a downside from a national security perspective because people could leak information, steal information, or at a minimum just get educated here and then go back to their home countries and benefit their home countries. And some of our earliest work was on understanding actually the national security benefits of having high-skilled immigrants here, including just the fact that the US’s high-tech ecosystem is so driven by immigrants. Depending on which numbers you’re looking at, they’re at a minimum something like 30% or 40%, sometimes up to well over half of any given pool. I think it’s something like half of the founders of top startups are foreign born, things like that. And that makes sense if you look at if the US is going to compete internationally: we just have a much smaller pool of domestic workers than somewhere like China or somewhere like India. So the fact that we can import them and that we can draw the best talent to this country is a huge asymmetric advantage. But that is just obviously in contrast with the Trump administration and the Trump MAGA movement’s perspective on immigration more generally, so I think that policy wonky set of considerations around pros and cons just gets lost in the broader push to be anti-immigrant. I do think that among the policy actions that the Trump administration has taken, the many things that are going to deter high-skilled immigrants from coming here are up there with the cuts to science funding as some of the most damaging to US competitiveness, just in terms of being technologically as sophisticated as this country has been in the past. And my understanding of it is not that there is a clear, well-thought-through strategy that explains why that is. It’s just these different components of the coalition sort of doing their own thing. Rob Wiblin: The Center for a New American Security and Lennart Heim (the compute governance expert who was actually on the show two years ago) have recently been pushing pretty hard this line that the US should be renting compute to other countries rather than selling the chips itself — basically just because there’s a clear security win that if you’re just renting access to them, if people are just basically buying access to the compute on the cloud, then you could always cut them off if they’re using it for some nefarious purpose or against US interests for any reason. Do you think that’s a good argument? Helen Toner: I think it’s pretty good. I think it comes back to this question of what are your objectives and how do you know if you’ve achieved them? So I think this is quite good if you are trying to be able to continue to profit from Chinese companies using AI, and you think you’re going to be able to monitor and shut off concerning uses, then I think that’s great. If instead you’re trying to maximally profit from the Chinese market, then probably you do want to sell instead. And if instead you’re trying to maximally limit China’s access, then you don’t want to rent. So I tend to think that it’s a good way of sort of meeting the tradeoffs involved, but again, I think this debate is so lacking in agreement on objectives and how you know if your policy is working that it’s a little bit hard to know if this will hit the spot for different people involved. Rob Wiblin: What’s the best argument against the “rent, don’t sell” approach? I think one thing is, if the UK just couldn’t buy any Nvidia chips — you know, if even allied countries of the US couldn’t buy any chips — then they would feel in a very vulnerable position basically, because they could always just get cut off. It’s not really a security situation that any country would gladly accept. So I suppose that would create a lot of pressure or a lot of enthusiasm perhaps for just sourcing AI chips from anywhere else — in particular, I guess China would be the main alternative source. That could be a negative side effect that it might have. Helen Toner: Yeah. I think with a lot of the chip controls, a huge question is how much does it help the Chinese ecosystem? And that’s just an empirical question that I haven’t seen very good answers to. So this is true for the original October 2022 controls: the big downside, the big thing you want to avoid, is in preventing them from buying US chips, you supercharge Huawei, you supercharge SMIC, which is their best chip manufacturing company. And I think people who are against the chip controls often assume that preventing them from buying US chips will be a boon to their domestic industry, and people who are for the chip controls often assume that it won’t help them that much because, for instance — and this is true — it was already a huge priority for the Chinese government to be supporting their domestic semiconductor ecosystem and pouring just hundreds of billions of dollars into trying to subsidise it. And they were already struggling. I think similarly with “rent, don’t sell,” it’s just an unanswered question of how much does this actually help them indigenise their supply chains? How happy are different countries to just rent instead and say that’s totally fine, no problem, versus do they then go looking elsewhere? And if they do go looking elsewhere, does that help? Or is it just so technologically difficult to reproduce this supply chain that it doesn’t make much of a difference? Rob Wiblin: The US has approved the construction of big data centres with Nvidia chips in several Gulf countries — I think Saudi and UAE, possibly Qatar as well. What are the pros and cons of that, in your mind? Helen Toner: I think the deals that you’re talking about are, as far as I understand, provisional — sort of early-stage announced, without the details being hammered out — so it will really depend on what the specifics of those deals turn out to be. The big pro is that, if you look at the different inputs to AI, if you think computing power is an important input, then what do you need for that? You need chips, you need land — like permitting, the permission to build — and you need energy. So there’s sort of a natural trade that you could do and say, in the US, we have lots of chips, permitting is a nightmare, and our grid is struggling. But in the Gulf, they have plenty of land, they have plenty of sunlight, they have plenty of oil — so we bring the chips over there, they let us build and they give us lots of energy, great deal. So I think that’s the main pro, is being able to build out more computing capacity than you otherwise could. I think the cons depend a lot on the specifics of the deals. There might not be that many cons. I’ve heard a comparison between US data centres in the Gulf and US military bases in the Gulf: it’s like this is a US asset, it is US soil, it is fully under US control. Maybe there’s not that many downsides if that’s the case. But the more that the countries themselves have ownership rights over the chips or usage rights or ability to access the facilities, then there’s sort of two big potential downsides. One is the connections with China that these countries have. This includes doing joint military exercises or having very tight personal and commercial relationships with Chinese political leaders and business leaders. Meaning, does this help China reverse-engineer chips, have more access to advanced ships in a way we wouldn’t want? The other big set of downsides is just specifically around the fact that these countries are autocracies. They are not nice governments. Rob Wiblin: It might be worth elaborating a little bit on that. I read a blog post you wrote about this and I was shocked by the degree of repression that there is in the UAE. Helen Toner: Yeah, yeah. I think people tend to know that Saudi Arabia is not a democracy: famous for not letting women drive; famous for hacking apart a journalist, Jamal Khashoggi, in the Saudi embassy, assassinating him in cold blood — which is sort of emblematic of how they think about free speech and free press. The UAE though, I think people just have a bit of a sense of like, Dubai and Abu Dhabi are kind of nice places to visit. It’s a bit too hot but they have big skyscrapers and fun indoor skiing or whatever. But the UAE is an autocratic country. I think the score on the Freedom House democracy index is something like 18 out of 100, which is really, really low. Political parties are banned. There’s one body that’s half elected and half appointed by the royal family, but it doesn’t actually have formal power anyway. So there’s kind of elections, but they’re sort of fake elections. They do mass trials of dissidents that are clearly not due process and actual real rule of law. They persecute the families of dissidents. The economy runs on immigrant labour, and those people have very few rights — at a minimum they’re non-citizen workers who have very little ability to participate politically; in the worst cases they are essentially forced labour. So this is really not a country that respects rule of law or that is interested in empowerment of its population. There’s quite strict rules around what the media can say about the royal family. It’s a hereditary autocracy with a royal family that is going to stay in power. The shape of these deals is still up in the air; it’s not clear exactly who gets what. But if you believe, as some of the leading companies making these deals have said, that access to compute is going to be a huge determinant of national power in the future, and the deals are structured in such a way that the royal families, the autocratic governments here get access to essentially world class supercomputers, then that’s pretty concerning — because you’re handing over a large amount of power to actors whose interests are not in line with the public generally, or even with the US in terms of a long-term strategic outlook. Their priority is staying in power, essentially. Rob Wiblin: Yeah. So when one of these deals was announced, I think for a data centre in the UAE, it was a collaboration with OpenAI. And I think they put out a press release saying this is a huge win for democracy and democratic AI or something along those lines. How should we interpret that? Is this just kind of boundless cynicism or is there something else going on? Helen Toner: It certainly looks cynical to me. I think it’s really playing fast and loose with what democracy means, what democratic AI means. If you read line by line, they never imply that the UAE is a democracy, but they talk about building on democratic rails or promoting democratic AI internationally. I think there’s two ways you could interpret this, and both could be reasonable in principle, but aren’t really present in this deal. One is that access to AI can be a democratic force for good in general. For example, if OpenAI were to make ChatGPT available freely in a country that previously had restrictions on speech or restrictions on access to information, I do think that having an AI system that you can ask questions of, that you can get information from, that you can learn things from, that you can use to further your own goals, I think that is obviously empowering of the individual in a way that is compatible with democracy. But it doesn’t seem to be part of the deal with the UAE. Shortly after that deal was announced, one of their executives was on stage and was asked by a US journalist who had lived in the UAE, “I’ve lived in the UAE. There’s these red lines of what media can and can’t say. Are you going to build those red lines into ChatGPT and the UAE?” And he was sort of like, “Well, we’ll see” — which is not very encouraging. So this first theory of change — of giving more people access to AI, and that being this democratic, grassroots-empowering force — doesn’t seem like the case. The other one, which I think they are trying to sell more, is this idea that American AI is inherently more democratic than Chinese AI — so anything that furthers American AI or helps America win against China is good for democracy. And again, that one in principle could make sense, coming back to this story of the big pro here is being able to build more data centres in a way that you couldn’t do domestically. The problem is, on the one hand, again, if we think that the UAE government is going to be empowered by this deal, that they’re going to have access to really sophisticated AI, then why exactly is that better than China having access, given that they are so autocratic? But secondly, what exactly are the parameters of the deal, and what did the US have to give up in order to get this strategic benefit? Again, if it’s a data centre that is essentially like a US military base, and fully US-owned and controlled and everything accruing to the US, great. But a big thing that the companies will say is if the US doesn’t sell it, then China will sell it. So China is waiting outside the door: if the deal doesn’t go through with the US, they’ll just offer the exact same deal. Rob Wiblin: China can’t even make the chips! Helen Toner: Exactly, exactly. So it’s totally disconnected from the reality of what they can do. And actually, I think this has been a recurring theme in the debates about chip controls and US versus China capacity: really neglecting the difference between what are the specs that are announced versus what is their actual production capacity and what are their real specs. Just to close the thought on the UAE deal: one thing that was reported was the UAE getting hundreds of thousands of next-generation Nvidia chips. That is just an unbelievable amount of computing power that China absolutely cannot match. They are really struggling to meet their own domestic demand. There is no way that if that deal didn’t go through, they could step through the door and offer the UAE the same thing. So I think that’s just a really bad motivation. I think in the discussions about chip controls more generally, it’s been really neglected what number of chips are we talking about? What is China’s capacity to manufacture them? And when they demonstrate manufacturing, how have they done that? For example, it’s now pretty well demonstrated, pretty clear, that one or two, I forget, generations of Huawei’s most advanced chips, they’ve demonstrated this pretty impressive performance. And do you know the way they’ve done it? Have you heard this story? Rob Wiblin: No, I haven’t heard this one. Helen Toner: Basically, they fooled TSMC into making chips for them. So this is this Taiwanese semiconductor manufacturing company that makes the most advanced chips. So this was after the controls were in place; TSMC was not allowed to manufacture chips for Huawei. Huawei set up a front company, sent in some chip designs that were clearly for Huawei chips. But TSMC just either turned a blind eye, or didn’t have the processes in place, or something else and went ahead and manufactured a large number of chips — or gave them dies, I forget the exact specifics — but basically handed them this giant stockpile of very advanced inputs for their chip-making process. So they are now kind of marketing and selling those chips and acting as though it’s their own sophistication. But it’s like, well, no. Rob Wiblin: If anything it shows the opposite. Helen Toner: Exactly. So there’s often these sort of headlines of, “oh my goodness, Huawei announced this chip. Huawei demonstrated this performance. Huawei manufactured this small run of phones with advanced chips in them” that really isn’t looking at like, wait, how do they make that? And how many are they going to be able to make in the future? Which is what you need to do to try and understand the actual effects and tradeoffs with these controls. Rob Wiblin: So we almost always assume that frontier models are going to be developed and first deployed in the US or China, that those are the two leading countries. Is it crazy to think that however you want to define AGI or superintelligence, it could first be trained in Saudi or the UAE? I think they have a lot of disadvantages, but if they have access to an enormous amount of compute, and the benefit they have is they have just enormous stockpiles of cash basically that they could throw at this problem. They have a lot more disposable income than a country like the US has. Maybe not as much as the US, but being autocratic, they can direct money much more with far fewer constraints. Is it possible that superintelligence could be developed in Saudi first? Helen Toner: I think it’s possible. I would say the UAE more than Saudi Arabia is my impression. Certainly by all accounts the UAE government is very so-called AGI-pilled: very into this idea that AGI is going to be a huge deal and that the way there is scaling up compute and things. So if that’s true, then they’re at an advantage, because they already believe that. I think, as with all questions AI, it depends on the timelines, depends on how long we’re talking. I think in the next couple of years they would struggle to stand up a really competitive effort. But if we’re talking a little bit longer scale, then potentially. I think also they’re very well positioned to benefit from the US rejecting high-skilled engineers and scientists. They can really offer great compensation packages, great quality of life, and I’m sure that they are trying to do that, if not already doing that. So it’s not what I would bet will happen, that the most advanced models will be developed in the Gulf, but I definitely think it’s possible. And certainly if we continue to make these giant deals and really build up a huge amount of computing power there, then it becomes even more possible. Rob Wiblin: Do you have any proposals for how we could go through the transition to AGI, potentially superintelligence, without ending up with an absurd amount of power being concentrated in the hands of some companies, possibly some governments, possibly some individuals? Maybe you could explain to people why that feels like it could be a natural outcome. Helen Toner: I think there’s a couple of reasons why AI might be very power concentrating. The most obvious one is if it continues to be as capital intensive as it has been recently. So I think the clearest trend over the last 10, 15 years of AI development is this compute scaling idea: that the way to make progress is to have the biggest computing clusters and to spend increasingly gobsmacking amounts of money training individual AI systems. If that continues to be true, then you’re naturally going to have actors that have access to large amounts of money — whether that’s very well capitalised companies or whether it’s rich governments, they’re going to be the ones developing the AI. Another way that it might come to be would be if you have a regulatory ecosystem that protects the top players, so that there’s some very stringent requirements on how to develop AI, and that’s very hard to break into so you have a regulatory capture and there’s a small number of players. Maybe those regulations were introduced with good intentions, but if they end up concentrating power, then that’s not great. And I think there’s a natural tension here as well among some people who are very concerned about existential risk from AI, really bad outcomes, and AI safety: there’s this sense that it’s actually helpful if there’s only a smaller number of players. Because, one, they can coordinate better — so maybe if racing leads to riskier outcomes, if you just have two top players, they can coordinate more directly than if you have three or four or 10 — and also a smaller number of players is going to be easier for an outside body to regulate, so if you just have a small number of companies, that’s going to be easier to regulate. So I think there’s often a sense of actually that concentration is valuable. I see the logic there. But the problem is then the “Then what?” question of, if you do manage to avoid some of those worst-case outcomes, and then you have this incredibly powerful technology in the hands of a very small number of people, I think just historically that’s been really bad. It’s really bad when you have small groups that are very powerful, and typically it doesn’t result in good outcomes for the rest of the world and the rest of humanity. I guess that’s all preface to your actual question of how do we avoid this? I don’t really know. I would love for there to be more work on this. A lot of the thinking that has happened about this has been between people who say the risks are really large, so we have to try for concentration because otherwise we all die; and other people saying that’s stupid, we’re obviously not going to all die, and so therefore we should just diffuse power maximally. And I think trying to get those people to actually engage with each other and say maybe there is — Rob Wiblin: What if both risks are medium? Then we’re in a real tough spot. Helen Toner: Or both risks are high, right? Yeah. Sometimes when I talk about this, people think that I’m optimistic, and I’m like, “It’s all good, it’ll be fine, just let there be less power concentration.” But I actually think my take is a more pessimistic take: I don’t think concentrating is the solution, and I don’t think maximum diffusion is probably the solution — so how do we navigate that middle ground? I think it’s really hard. One answer might be we might get lucky in terms of how the technology develops. It might be the case that actually things develop relatively gradually. There’s time for fast followers to catch up, and there can be relatively broad access to capabilities, and there aren’t these really decisive, huge civilisational downside risks that you need to manage in a concentrated way. So I think we might just get lucky. That’s sort of my best hope. There’s also tools that I would love to see explored more that would target this. At a very basic level there’s things like AI literacy: how do we get a larger range of people to understand what is going on with this technology, to be able to engage with it, to be able to think about tradeoffs, pros and cons, risks and benefits? How do we think about worker empowerment or the role of workers, including workers at frontier companies, in shaping the development of the technology? There’s also very basic things like taxing companies. If the only problem is just that it’s a naturally capital-intensive technology, so large actors are going to build it, and we don’t have to worry about the downside risks, then you have very traditional tools like tax and antitrust that can come in and help try to diffuse that concentration of power as well. But that doesn’t solve the safety challenges. Rob Wiblin: The concentration of power worries also point to some extent against the export controls, or against being very intense on the “rent, don’t sell” of compute — because I would guess if almost all of the compute is concentrated in the United States, and even Europe or Australia or the UK or basically no countries can get a foot in the door to have significant data centres, that I guess leaves them very vulnerable to exploitation by the United States, or it puts the United States in a troublingly powerful position. Do you agree with that? Helen Toner: Yeah, I think that’s probably directionally right. I haven’t thought about it much, but yeah, interesting point. Rob Wiblin: You wrote recently about how maybe one of the underlying disagreements in AI that people don’t necessarily address directly is between people who favour decentralisation and dynamism, and those who feel apprehensive about a dynamic, open AI ecosystem. Can you explain that idea? Helen Toner: Yeah. This was coming from a book that I read late last year by Virginia Postrel called The Future and Its Enemies, which is a great title. It’s actually from the ’90s, so it’s in the era of environmentalism and cyberspace just becoming a thing. And I found it really helpful because it helped me put my finger on something that I think weirds people out about AI safety discourse, that people in the AI safety community are often not as aware of, which is basically this instinct of: if there are these big risks, then the solution is concentration. So there’s this risk that AI will kill all humans, or this risk that AI will take over or something like that — and the solution is, well, we have to make sure that the right people build it and that it’s a very small group. And I get the logic there, but there’s sort of a lack of — Rob Wiblin: It gives people the creeps. Helen Toner: There’s a lack of… There’s different ways you could do this, right? Sometimes when I talk about this, and also when I wrote about it, people come back and they’re like, “Wait, so you think we can just not worry about that at all? Just let everyone build AI and it’ll be totally fine? Why do you think that?” And that’s actually not my take. My take is more like, we should be more worried, and we should be more pessimistic about these factors we have to balance. To put that another way, I see the logic when people very concerned about existential risks from AI say we’re going to have to concentrate in order to manage these risks. But then it’s lacking the, “Uh-oh, we think we’re going to have to concentrate access to this technology in order to manage this very severe risk. That’s terrible. How do we handle that? How do we think about all the downsides of that? How do we make sure that it doesn’t just end up in massive disempowerment of most of humanity?” And I think a lot of people, especially in the tech world but also more generally, subscribe to what Postrel’s book would call a more “dynamist” view: that a huge source of success, a huge source of human prosperity and wellbeing is having a world that allows for more trial and error — that allows for decentralised experimentation, that allows for things to fail, that is more focused on resilience and recovery from problems than on preventing every problem in the first place. Because if you try to go with preventing every problem, then you end up with a much more authoritarian, centralised, what she calls “stasis”: kind of stability-focused, control-focused regime. And again, I don’t mean to say that therefore we should just not worry about it and just let people do whatever they want. What I more mean to say is, if we really think that these more “stasist” solutions of control and stability are really necessary, then we need to really be apprehensive about that — and we need to be thinking really carefully about, if we really think that it’s necessary to concentrate power in this way, how do we think about how to deconcentrate it again at the end? What are the guardrails that are put in? And how sure are we? What evidence can we gather about what the nature of different AI risks is, and whether it is worth this tradeoff? As opposed to sort of the vibe of the AI safety community, which can often be like, “It’s OK, the solution is just concentration and then we’ll be fine.” Rob Wiblin: I think that’s shifted quite a bit in the last few years. Helen Toner: I think it has, yeah. I think it has. But I do think it is still a strong undercurrent in a lot of the thinking. There’s been more discussion of risks of concentration of power and so on, but I think it’s still a strong default that the answer is to just have one project and then you can manage the project well. Rob Wiblin: I think this is one of many other assumptions that get baked in, that go all the way back I think to the 2000s, when the picture of how this would play out was very rapid recursive self-improvement loop based potentially on not that much compute, thinking that is overwhelmingly the most likely way for things to go. So power will end up concentrated whether you like it or not. And it’s still possible that you could end up with a very strong recursive self-improvement loop, but I think people believe that much less than they did 15 years ago. But it’s so hard to go back and remove all of the assumptions that were kind of baked into the language in how you were thinking about the problem from the beginning. Helen Toner: Yeah. A mini hobby horse of mine for the last couple of years has been that, personally, I think something that we really need in this space is people who are coming in who are willing to take these ideas seriously — so willing to engage with the idea of AGI, of superintelligence, of existential risk, of human extinction, of disempowerment, AI takeover, these sort of sci-fi wacky risks — people who are willing to engage with that, but who are not coming out of that social and intellectual milieu that I think both you and I come out of, and that I think does contribute to groupthink and bubble dynamics. Not bubble as in a financial bubble, but as being in a social bubble. That’s actually something that I really value at CSET: being around people who are mostly not from that world, and who mostly don’t bring in all of those, as you say, baked-in assumptions that are hard to go back and unbake. Personally, I think that in this space it’s just really helpful. I sometimes meet people who are getting interested in these issues, and they’re like, “Well, I haven’t been thinking about them for as long as some other people.” And I think that can actually be a real advantage because you’re bringing fresh eyes, you’re bringing fresh perspectives. And the key part for me is not shying away from being interested in these questions, as opposed to having a particular background in some particular set of answers. Rob Wiblin: I feel like we’re in a very difficult spot, because so many of the obvious solutions that you might have, or approaches you might take to dealing with loss of control do make the concentration of power problem worse and vice versa. So what policies you favour and disfavour depends quite sensitively on the relative risk of these two things, the relative likelihood of things going negatively in one way versus the other way. And at least on the loss of control thing, people disagree so much on the likelihood. People who are similarly informed, know about everything there is to know about this, go all the way from thinking it’s a 1-in-1,000 chance to it’s a 1-in-2 chance — a 0.1% likelihood to 50% chance that we have some sort of catastrophic loss of control. And discussing it leads sometimes to some convergence, but people just have not converged on a common sense of how likely this outcome is. So the people who think it’s 50% likely that we have some catastrophic loss-of-control event, it’s understandable that they think, “Well, we just have to make the best of it. Unfortunately, we have to concentrate. It’s the only way. And the concentration of power stuff is very sad and going to be a difficult issue to deal with, but we have to bear that cost.” And people who think it’s one in 1,000 are going to say, “This is a terrible move that you’re making, because we’re accepting much more risk, we’re creating much more risk than we’re actually eliminating.” I guess the idea would be if we could find policy responses that help with both simultaneously, or help at least one of them without harming the other. I think people probably have underdone attempts to find win-wins or at least like win-neutrals, but it’s easier said than done. It’s not a simple matter. Helen Toner: Definitely. I mean, there’s a set of policy recommendations that I do think can help us navigate this a little bit. So these are things like everyone’s favourite consensus recommendation: more transparency, more disclosure from the AI companies. I think that is really helpful. I don’t know. I guess my stance here is that it’s very unlikely that there are these two futures that we imagine, and the future goes down one of those tracks. I think it’s very likely there will be unexpected twists and turns, new things that develop that we didn’t anticipate. Technologies never develop exactly the way that we think they will. So I think our stance here should not just be about which of these two camps is right, but being open to and ready for many different possibilities. So yeah, I think transparency is helpful for that, in that it lets information propagate about what is actually going on, what is the sort of ground truth of what systems are being developed, what risks they pose and so on. This AI literacy point, or technical capacity in government is sort of a similar thing of making sure that you’re hiring people or training people so that there’s not just a very small group of experts inside the companies that are making all these decisions, but having a broader set of eyes on it. And then resilience-focused approaches: you know, cyberdefence, biodefence, AI control-type approaches — meaning how do you invest in building AI systems that can monitor other AI systems and things like that. And one other piece that I think can be helpful as well is really investing in the science of AI. So people sometimes talk about how the government should be funding AI safety because the companies are funding AI capabilities. There’s actually a different thing that I’m trying to say with this, which is that I think we would be in a much better position if we had a more grounded understanding of how AI systems work: How do they generalise from their training data? Why are they so jagged? Why do they fail in these weird ways? How could you understand whether a given system is likely to succeed or fail in a given setting without just trying it? Because right now we basically just have to try it. Interpretability is another example of a “science of AI” approach. I think right now we’re really under-investing in the science of AI as opposed to the engineering of making AI that is better, that can do more stuff. So those are some ideas that I think could be helpful. Sometimes I think the AI safety community can reject anything that is not a full solution. I feel like there’s something about the community’s background in mathematics or computer science or philosophy where they want a full solution that you can prove is adequate — and if it’s not that, they want to reject it. I think that’s really mistaken. I think we’re going to have to do our best, figure things out as we go. And I think investing resources now to try and position ourselves better so that we can respond better one year from now, five years from now is really valuable, and is one of the few things that we can do. I think it’s also worth thinking through what are more elaborate plans for actual full solutions. But even there, the idea of a full solution suggests that there is a clear end state — which, coming back to this idea of dynamism, I think is the wrong way to think about it. I don’t think there is a one best way, an optimal endpoint that we should be aiming for. Instead, I think we need to be trying to get to a world that is dynamic and open and free and empowering — and that’s not going to look like we fixed it; instead it’s just going to look like maybe we got through an acute risk period or something like that. Rob Wiblin: Yeah. The desire for a full comprehensive solution is another one of these assumptions I think that goes back to the Yudkowsky/MIRI vision. We’re talking in the week of the release of If Anyone Builds It, Everyone Dies. It’s in the title, saying we really do need to solve this completely, comprehensively. Muddling through is definitely not going to work, in their view. And that may yet be vindicated, but I think it’s very far from obvious. On many plausible views, muddling through is an option. A bunch of half-measures might well work. I think more and more people have appreciated that in recent years. At least I think the people who want the full solution, they’re doing it because they think loss of control is overwhelmingly likely, the technical problems are overwhelmingly hard — rather than just because they’ve made that as an unquestioned assumption. Helen Toner: Yeah. And there’s a big spectrum of how competently you muddle through. “Muddling through”: I don’t really love that phrase, because it sort of sounds like, “We’ll just take it as it comes, and sort of see and it’ll be fine.” I think a different version of muddling through, a more competent version, is this idea of “plans are useless, but planning is everything” — of like, how do you set yourself up with lots of different possible levers to pull, lots of different things on the shelf that you could take off the shelf if needed, and also set yourself up with the visibility and understanding to be able to tell what paths you seem to be on? I think in a sense that’s also “muddling through,” because you’re not pulling the trigger on “Here’s our 12-step plan, and we’re going to go through it comprehensively,” but I think there’s a big range of what it looks like to muddle through. I mean, arguably we muddled through with nuclear weapons, but that involved a massive international treaty and a huge monitoring apparatus, and that’s really valuable. And I think it clearly reduced the number of nukes in the world and the amount of nuclear risk in the world. Rob Wiblin: I think most of the technical research agendas are either win-win or win-neutral on this. One that is a win-win is the ability to inspect the models and detect hidden goals or hidden agendas that might have emerged unintentionally or have been deliberately inserted in there. That is really helpful, I think, for avoiding concentration of power. Tom Davidson in an episode earlier this year explained how this problem of secret loyalties does create a vulnerability for a power grab, basically. But the same techniques might well just work for outing any unintentional goals that have ended up getting baked into the model through training that we didn’t intend. I guess model organisms, understanding AI misbehaviour — when it generates, when it doesn’t, what things you can do to reduce it — almost across the board, I think funding for all of those sorts of research agendas does just seem like a real no-brainer in my mind. Helen Toner: Yeah, I think that’s right. I do think that there’s maybe a different axis where they are sort of win-neutral as opposed to win-win. I’ve been thinking of this more and more as really central to disagreements about risk, which is this question of will AI just be a tool, or will it be something more than a tool? This is part of, but not the full disagreement with the “AI as normal technology” authors Arvind Narayanan and Sayash Kapoor — who are both really excellent, and I think are great examples of people who are taking these issues seriously and engaging with them seriously, but coming with a very different set of assumptions, and I think really adding to the debate. And I think that if AI stays as a tool, continues to just be something that people use to their own ends — and maybe sometimes it breaks or fails, but it doesn’t have its own agenda, it doesn’t have its own goals — then you could still be very worried. Then it makes sense to be worried about concentration of power. All of the takeover stuff doesn’t really come into play. So I think that is another place where different assumptions, different expectations, and different tradeoffs look better or worse to different views. But I think it’s been really valuable to have that “AI as normal technology” view articulated, because it is sort of a funny thing where I think they wrote it because they see the superintelligence view as sort of dominant — but in fact what they articulated was a much more widespread, unarticulated view. I think most people, when thinking about AI — Rob Wiblin: Surely overwhelmingly that’s the most common approach. Helen Toner: Yeah, most people assume that it’s going to be more like a regular technology; it’s going to be a tool, and we’re going to decide how to use it. And this was explicitly part of why they wrote that paper, was to say, “We think this is a common view, we want to put words to it, we want to describe the components of it.” Rob Wiblin: Where do you come down? Is AI a tool or is it a new form of life? Helen Toner: I don’t know yet. I agree with some of the things they articulated. I think it was really useful to articulate the distinction between capability and power. So an AI system that is able to do something does not mean that it’s actually set up in a way that it can do that thing — and we as humans have a lot of agency in deciding what AI has the power to do. They say at the beginning of the paper that it is a description of the present, a prescription about how they think AI should be handled, and a forecast about how it will be handled. And those three things, sometimes I agree with them that AI is a certain way — for example, that we are deploying it relatively cautiously at the moment, relatively slowly. And I agree that we probably should continue, especially in high-stakes settings, to deploy it relatively slowly and cautiously. But I maybe disagree that we obviously will. And in particular — and I know that they’re working on this actively right now — they excluded military AI from that paper entirely. They also included a section about another debate which I think is really underexplored, which is how superhuman can AI get in different areas. I would love to see more work on trying to pull apart what are the reasons to believe that there’s a very high ceiling above human level at different capabilities. One thing that Arvind and Sayash say in their paper is that, for example, being faster than humans is not that useful in many settings. It’s useful in high-speed trading, but not that many other things. For me, in battlefield management, speed is absolutely a huge advantage. People in the military talk about the OODA loop — the observe, orient, decide, and act loop — and you want to “get inside” your opponent’s OODA loop, meaning be faster than them, be able to get around that loop faster. That’s one example of an area where I think the “AI as normal technology” view is not necessarily accounting for the incentives to hand over power, hand over autonomy in order to get benefits like, for example, speed. Rob Wiblin: You’ve written that you think the AI security ecosystem has often appropriated the language of non-proliferation from nuclear weapons and biological weapons. And you think that might have been a wrong turn, or at least it’s narrowed people’s scope a bit, and instead we should think about things in terms of “adaptation buffers” — which is I think a term that you introduced. Can you explain why you think that might be a mistake and what adaptation buffers are instead? Helen Toner: So this is specifically in response to concerns about misuse: the idea being we’re building these more powerful AI systems, and they’re going to be misusable in particular ways. Two of the most common that people bring up are being used to create bioweapons or being used for cyberoffence, hacking — and the response is the AI systems are going to be so powerful they would enable such bad attacks that we have to do non-proliferation, meaning prevent them from spreading. I think the challenge with this is it won’t work, and it will be very invasive and very authoritarian. The reason for that is basically because of this very clear trend — this maybe somewhat contradictory-seeming trend, to people who are not following AI closely — of, on the one hand, as we’re building more advanced systems, the amount of computing power that you need the first time to build an especially advanced system is going up over time. So you need more and more compute to build something at the frontier. But as soon as you build something at the frontier — say, an AI system that is as good as a software engineering intern, or an AI system that can get some score on some test or whatever — as soon as we build that for the first time, the amount of computing power and also just broadly technical sophistication that you need to build that is going to fall pretty rapidly over time. So for example, at the time when the AlexNet paper was published in 2012 by Alex Krizhevsky, Ilya Sutskever, and Geoff Hinton, that was very hard, very complicated, but now it’s incredibly easy, incredibly cheap . Likewise for a GPT-3-type model at the time — really at the cutting edge, quite expensive, very difficult — now very easy to recreate. What that means for misuse, for this idea that we’ll have AI systems that are very powerful at hacking or very good at helping with bioweapons: maybe initially it will be a small number of companies or research groups that can build a great model, but a couple of years later, five years later, it’s going to be very easy for many groups to do that. So in order to do non-proliferation, you’re going to have to get very invasive about restricting the number of chips that people have access to or monitoring what they’re doing with AI systems. And I think it’s just really not likely to actually work. If you have a determined actor, a bad actor who is trying to misuse these models, they will find a way to do it. And in the meantime, you will be really restricting the ability of many normal people to use and build AI systems in ways that would be perfectly fine. To me, the best alternative that we have is what I call an “adaptation buffer”: basically meaning this buffer period between when we know that some capability of concern is going to be very accessible at some point in the future and when it’s actually accessible. I think we are in this period right now for AI systems that can help novices to create known biothreats, if that makes sense. In the bioweapons conversation, I think sometimes two threat models get conflated: one being novices creating known biothreats — so this is like a teenager in their basement creating smallpox kind of thing — and that’s different from helping experts create much more sophisticated biothreats, which often is going to involve the use of biological design tools or it’s a different process. I think we’re clearly in the adaptation buffer for that first one. And the thing to do then is to try and put as much societal resilience-style efforts in place to prevent that capability from having massive negative impacts when it is widely available, as opposed to trying to clamp down and prevent it from ever being widely available. So that’s this idea of the buffer: that the time period between when we can forecast it pretty specifically — not just like “bioweapons, something something,” but something pretty specific — and the time when we can get it. Now, I don’t think this is a perfect approach. In particular, if buffers are short, then maybe we don’t have enough time to get to them. And it’s very much focused on misuse, not on more misalignment-focused threat models. But that’s the basic idea. Rob Wiblin: Yeah, I think people have not always been drawn to the adaptation buffer framework, and I think it’s not for no reason: it’s often because they think there aren’t any good adaptations, I guess especially on bio. I think this has been the view that if it’s easy for amateurs to create new pandemics, it’s just so costly to combat them that we just have to stop them from being created in the first place. I recently did this interview with Andrew Snyder-Beattie where he presented a plan that maybe there are adaptations that we could make that would make society very resilient to any new pandemics that someone tried to release. It’s a heavy lift though, it’s challenging, and it’s not completely obvious that it might work. So it’s possible that we could go down the adaptation buffer track and then look back and say actually that was a bad idea, because in fact the adaptations were just too difficult and non-proliferation was unfortunately the only option. Helen Toner: Again, my point here is not that non-proliferation doesn’t sound fun; my point is I think non-proliferation won’t work. They will proliferate anyway. I mean, time will tell. In this case, I hope that I’m right. Sometimes I’m talking about risks and hoping that I’m wrong. But time will tell. I think people sometimes overstate the inevitability of really severe misuse. If you look at when I wrote this post about adaptation buffers, for example, one of the commenters was saying, “Isn’t this all hopeless? I remember in college when a professor of mine was telling us about how easy it is to just drive up to a water treatment facility with a dump truck and dump a bunch of toxic chemicals right after the treatment ends.” And someone else was saying to me that they’d seen some video in the 2000s of how you can, in an airport, make a bomb using materials that you obtain after security. So you go through security and then make a bomb. But these examples are both great. They’re very encouraging. No one to my knowledge has done them. I don’t know about the water treatment one; maybe that’s happened, or if they did it, then they got caught, and it got prevented. So I think sometimes there’s a little bit of a neglect of what actually are all the options, what is the toolkit? Including just the FBI monitoring terrorist groups. That’s a real thing. Or the CIA as well. That’s a real thing that happens. Or looking at materials, looking at synthesis of nucleic acids, which is obviously part of the discussion: looking at can you just have layered approaches with the ChatGPTs of the world where they’re going to refuse to help you. There are pretty good controls on that right now, so that reduces the number of people. And then if you do eventually have attackers actually really producing bioweapons… I don’t know. I feel like sometimes in the existential risk community there’s this assumption that it’s going to be this absolutely horrific event, as opposed to something that potentially can be contained earlier. So I don’t know, I’m not unworried about this; I think it is really worth trying to prevent, trying to mitigate. But I think at a minimum, any concern that we do feel, any risk management energy and resources that we have for preventing bioattacks, should be doing a lot of this resilience-focused stuff as opposed to focusing primarily on clamping down on models — again, just because I think clamping down our models is not going to actually work. Rob Wiblin: It’s not a long-term solution. Helen Toner: Right, right. So saying that adaptation buffers might fail, I agree — but I think saying, “…and therefore we should just focus on preventing model proliferation” is the wrong call. Rob Wiblin: You were recently involved in a CSET report titled AI for Military Decision-Making: Harnessing the Advantages and Avoiding the Risks. How worried are you that the military is going to end up deploying AI poorly? Helen Toner: So this report was about a specific kind of AI called “decision support systems.” And one reason that we wrote about it was because I think there’s sometimes a bit of a gap between external perceptions of military AI, and what the military is actually looking at in practice. For example, I think externally people really focus on lethal autonomous weapons — especially drones with facial recognition is a really compelling use case that the public worries about. But if you look internally, there’s many other potential use cases that the military is thinking hard about, and how they adopt them, how quickly they adopt them will matter a lot for what actually ends up happening. So this paper was actually focused on a specific kind of system called decision support systems. And what we wanted to do there was really not get into this kind of binary of, “AI in the military is bad and should be banned and we shouldn’t do it,” or “AI in the military is essential and we should just be racing to adopt it as quickly as possible” — but instead to take a clear-eyed look at what are these systems, why do militaries want to use them, and how can they use them effectively and not get caught up in potential pitfalls of AI systems? So that involves looking at what are the actual benefits of using decision support systems. It depends a little bit on the specifics of the system what are the potential pitfalls. So do you have a clear understanding of the scope of a particular decision support system, what it was trained for, where might it be out of distribution — so in a situation it wasn’t trained for — and fail? Or how do you train your operators so that they can use them effectively? So that was kind of the set of issues that we were exploring there. Rob Wiblin: How much appetite is there for rapidly integrating AI into the military? I interviewed Dean Ball yesterday and I was putting to him, might we see premature applications of AI in the military, dangerous ones? And he was like, “It’s going to be so hard to do it at all. It’s actually probably going to happen very slowly and in a very hodgepodge fashion.” What do you think? Helen Toner: Yeah, I tend to agree. I think there is a lot of appetite, but institutionally the ability to procure or build AI systems and then roll them out at scale is tough. We have another paper at CSET that I wasn’t involved in called Building the Tech Coalition, which is a case study of a successful adoption of AI. And really it’s the exception, not the rule, that they got this system to an operational state where it’s actually being used in practice. The case study is looking at what are the factors that actually made them able to succeed in that case. One of the key parts there was the military sometimes talks about having bilingual leaders who are competent both in technology and also military operations. And one thing we really identified in that case study was you actually need trilingual leaders, who are competent in technology and military operations and also these acquisition/procurement questions, contracting, getting through all the legal language. So it’s tough. There’s a lot of barriers. The military is not set up [for it]. The way that it does research and development, is not designed for software; the way that it does testing is not designed for non-deterministic AI systems. So I think the appetite is very much there, but I tend to agree with Dean that in practice it’s going to be slow and piecemeal and a slog. Rob Wiblin: Do you have any read on how worried the military is about AI being backdoored or having secret loyalties or agendas? Helen Toner: I think they’re most worried about that. The military is naturally set up to think about adversaries. Rob Wiblin: Sabotage. Helen Toner: Right, sabotage, exactly. So I think they’re certainly worried about that in the context of what an adversary, whether it’s China or a different potential adversary, could do. That’s certainly a reason not to use Chinese models, or to be very cautious about even using US models that are trained on the broad internet. There’s already evidence of Russian groups, for example, doing essentially a version of data poisoning, trying to seed online datasets with pro-Russian views. So I think that’s primarily the lens that they’re thinking about it through. Rob Wiblin: Do you hear any discussion of this question of, if you’re having AI-operated military equipment, should it decide whether to accept orders based on what it thinks the law of war is, or should it just follow orders of whoever its operator is? I guess each of them has its issues. If your tank is having to make independent judgements about the law of war, about military law, maybe it’s not equipped to do that. And that also creates a lot of vulnerabilities that adversaries could try to use against you. On the other hand, if your equipment just absolutely follows any instructions that it’s given whatsoever, that creates a lot of opportunities for coups, where previously you just wouldn’t have been able to get human collaborators to go along with it. Do you have any thoughts on this? Helen Toner: I think mostly this is only relevant inasmuch as you’re thinking of AI as very much not being a tool — so you’re thinking of it as having its own agency, making its own decisions. And I think the discussions in military circles are very focused on AI tools. Rob Wiblin: They’re just not thinking about independent agents yet. Helen Toner: I think that’s just not really a part of the discussion yet. Rob Wiblin: Is it because it’s unacceptable or because it’s technologically not feasible yet? Helen Toner: I think it’s sort of too sci-fi for the military at this point, and also based on where they’re at with adoption, the level of sophistication of the tools that they are looking at. I will say that when these topics come up, to the extent that AI is operating in a more agentic, independent, autonomous way that is more equivalent to a human operator, there is a whole set of institutional expectations, standards, rules, laws for military personnel that you could in theory port over to an AI. For example, lower-level service members are expected to follow the commands of their commanding officers, but they are supposed to not if the command is illegal. But also things that they do that go poorly, that does then reflect back up on the commander. So there’s ongoing questions about how does accountability and responsibility for the use of AI systems flow back through the command chain? If something goes wrong, who is held accountable? Which can actually work if the AI is primarily a tool and can potentially also work if it’s operating in a less tool-like way. But yeah, I think the conversations about AI that is not a tool are pretty nascent. Rob Wiblin: Yet to come. Helen Toner: Yeah. Rob Wiblin: You recently wrote that we should maybe stop using the term “AI alignment” almost at all and instead talk about “AI steerability.” I tweeted this out recently and it got quite a bit of play. It seems like a lot of people are on board with this in principle, although I suspect it won’t happen because terms are just very sticky. If you’re used to saying something, it’s so hard to stop saying it. Can you explain why AI steerability is better? Helen Toner: I think it’s better in some ways. Since publishing the post, I’ve maybe come around to also thinking it’s not quite right. And I would never expect that; I don’t think we can get the word “alignment” out of our vocabulary at this point — so I was more making the post for people who are trying to explain it or want an alternate lens on it. The basic idea is: I think the term “alignment” sort of embeds in the word this idea that there is a clear thing to be aligned to. So if we think of alignment, we think of aligning the tyres of a car, where they’re supposed to be straight; we think of aligning a picture on a wall, where it’s supposed to be perpendicular to whatever. So I think if you start talking about alignment, people’s minds just very naturally go to aligning to what? What is the standard here? Which I think is an important question, but is getting ahead of itself — where the underlying problem is more one of — well now, “AI control” means something else — but it’s more one of controllability or steerability, in the sense of can you actually direct the AI? Can you build it in such a way that it behaves how you want at all in the first place? So I think steerability, is the AI steerable, conveys this better. The challenge with steerability — that I’ve come around to thinking is more of a problem than I did when I wrote the post — is it doesn’t distinguish between steerability at the development phase of, when you’re training the AI, when you’re setting it up, can you steer it to be the kind of AI system that you want; versus when it’s in operation, can you go in and intervene and steer it in ways that you like? So I think some people who read that post thought that by steerability, I was talking about, for example, when you’re chatting with a chatbot and you can sort of “steer” it to role play in one way or another, so it will go with you and it will follow your lead in lots of different ways — which the current chatbots are very good at, and which is a different thing than, for example, ensuring that it can’t be jailbroken, which is something that we’re less good at. So I don’t know. I think overall, the idea of, we actually don’t know how to steer AI systems in the first place, or we’re not very good at it, is a more intuitive way to put the problem than we don’t know how to align them. But I think no term is perfect. Rob Wiblin: My impression is that there’s been a big shift in AI discourse in Congress over the last year or two. Every week or two I feel like I see new congressional hearings where it feels like either members of the House of Representatives or senators are asking questions that feel like they’re almost out of the pages of some MIRI report, or they have a very Yudkowskian energy to them — where they’re very worried about superintelligence, or saying, “You’re planning to build superintelligence. How is that going to go? Might we not lose control?” Am I just getting a kind of biased selection of things, or are at least some people starting to get more open to that worry? Helen Toner: Yeah, I think the really huge shift that I saw was after the release of ChatGPT. ChatGPT came out late 2022, and in 2023 and 2024 there was definitely a huge uptick of, you could say, AI being the main character on Capitol Hill, or being a huge topic of discussion in a way that it really hadn’t been before. It is interesting. I think you’re right that over the last six or 12 months, there has been more discussion of more of the superintelligence-type concerns. I don’t know exactly why that is. I wonder if part of it is just a feature of the discourse getting a little bit more sophisticated. I do think in 2023 there was just this basic getting up to speed of like, “What is AI? What is going on with AI? How should we think about AI at all in the first place?” that a lot of people — not just members of Congress, a lot of members of the public — needed to do. And now I think our ability to think and talk about AI has just gotten more sophisticated over the past few years. Even things like AI 2027 and “AI as normal technology” being published, these two sort of different visions, is very helpful for kind of crystallising and making concrete different kinds of ideas and different kinds of disagreements. So that would be my guess. At the same time, we’ve also seen some of the CEOs getting more explicit about the level of disruption they expect. Certainly Dario Amodei at Anthropic has been outspoken about job loss and things like that. Sam Altman at OpenAI has written that they think they’re on a path to superintelligence. And so I think there’s probably also a sort of normalisation and an ability to ask about these topics in a way that wasn’t there when these high-profile figures hadn’t talked about it in their own words. Rob Wiblin: Some people have had the impression that AI progress is slowing down. At least some people have had the impression that GPT-5 was a bit off-track. It was kind of disappointing. Do you think it is slowing down? Helen Toner: I don’t really buy it. It’s really hard to say because our metrics are so bad; we’re really bad at measuring progress in AI or knowing what it is that we’re even trying to measure. That being said, it often seems to me like people who started paying attention to AI after ChatGPT, their subjective impression of what’s going on in AI is like nothing was really happening. There’s my little chart with an X-axis of time and the Y-axis of how good is AI? Nothing is really happening. And then suddenly, ChatGPT: big leap. So for those people, that was pretty dramatic, pretty alarming. And the question was, are we going to see another big leap in the next couple of years? And we haven’t. So for people whose expectations were set up that way, it looks like it was just this one-off big thing and now back to normal, nothing to see here. I think for people who’ve been following the space for longer, it’s been clearly this pretty steady upward climb of increasing sophistication in increasing ways. And if you’ve been following that trend, that seems to have been continuing. Basically since 2012, when deep learning really started to work, there haven’t really been huge breakthroughs. The biggest ones are things like the transformer in 2017. But that was a new architecture of deep learning — yes, it was a really big deal, but also was kind of building on [long short-term memory networks] and [recurrent neural networks] in this longer tradition. Or making internet-scale pretraining work: that’s a big deal, but it’s not like a huge breakthrough; it’s not a big paradigm shift. So I think we’ve kept seeing this level. The reasoning models coming out is another example of, in one way, very big change, and in another way, it’s sort of just building on the paradigm we had. So my kind of zoomed-out view is that it looks to me like GPT-5 is continuing on that trend. I think that trend, to be clear, is one that should make our eyes go wide and we should really pay attention to as like, wow, this is a huge amount of technological progress happening in a very short time. It’s just not the insanely dramatic level that people might expect if they had just noticed the ChatGPT one. And we also don’t know, for any given step on that trajectory, what real-world effects will result. So I tend to think that it doesn’t really seem to be slowing down. It also doesn’t seem to be obviously speeding up. Also you saw some commentary from people at the start of this year maybe, when the reasoning models were being demonstrated and you had that first o1 to o3 jump. I think some people said, “Oh my goodness, we’re going to have really advanced systems by the end of 2025, and end of 2026, even more so.” And that hasn’t been borne out, so I do think it makes sense to shave some of the fastest-progress scenarios off the top of your confidence interval. But that doesn’t mean that things are just sort of plateauing or hitting a wall. Rob Wiblin: Yeah. I think people are getting a little bit tricked because the releases come so much more frequently now. Often a release can feel a bit disappointing or only incremental, but you’ve got to remember the last model came out maybe only three months ago. It used to be more periodic releases, so the jumps were much bigger, much more stark. Helen Toner: Yeah, I think that’s right. People have seen that there was GPT-4 and then 4o and 4.1 and 4.5, and different updates to 4o — and that’s just in the OpenAI space, let alone the different Google Gemini and Claude versions. So yeah, it’s more of a drip, drip, drip. Epoch had a really good chart, you’ve probably seen this, taking some illustrative benchmarks for the GPT-3 to 4 and GPT-4 to 5 — they’re different benchmarks because they basically saturate — but showing in both cases there were really significant jumps on key benchmarks of the time. And it’s only because if you’re comparing GPT-5 to whatever came out six months ago or three months ago, that’s what makes it look less impressive, like you said. Rob Wiblin: You wrote in your notes that there’s this funny phenomenon that there are multiple different camps that have self-consistent, very different worldviews about AI, and they both feel like they can explain satisfactorily all of the empirical observations that we’re making about how things are progressing. Explain how that can be. Helen Toner: This came out of a workshop we ran at CSET on automating AI R&D, which we’re going to have a paper about hopefully in the next couple of months. This is ideas related to intelligence explosion or recursive self-improvement, or at what point are the AIs the ones doing the AI research and things maybe take off or maybe don’t? And something that was really interesting at that workshop was we had people with pretty different views there, and it was surprisingly difficult to get them to come up with different predictions for what would happen in the leadup to potentially very automated research. And part of why this is is that I think these different worldviews have good ways of explaining contrary evidence and why it’s going to converge back to their expectations. To go one by one: a worldview that is expecting that you’re not going to get very superhuman systems, you’re not going to be able to fully automate things, you’re not going to be able to have these self-reinforcing dynamics — that worldview expects multiple things: Bottlenecks to arise: if one thing gets accelerated, then it’ll get held back by some other factor that can’t accelerate the same way. Plateaus on ability: maybe the AI system gets much better, but it can only get so much better. It can only hit a certain level of biology skill or coding skill or something like that. So for that worldview, if you see evidence that things are going quickly or that things are becoming more automated, your expectation is just that they’re going to hit the limits quicker. Sure, it’s going faster now, but we know that the limits are there and so it’s just going to hit them faster. On the flip side, people who expect there to be these really dramatic rates of increase in AI automation or AI getting far superhuman at different things, when they see bumps along the road or hiccups or difficulties, they are often able to explain them in terms of, this will just lead to more speed later. So an example that I’ve heard recently is: have you seen the AI Village? I love this thing. If you look at the AI Village is this nice demo of different agents being put in a computer use environment and asked to do some tasks. It’s very fun. I highly recommend for anyone who hasn’t seen it. A lot of the agents in question, which are some of the best models — Gemini 2.5 and GPT-5 and Claude 4.5 — struggle with some really basic computer use tasks, like clicking an interface or figuring out that something isn’t working. And someone who has a view of “everything’s going to go very fast” looks at something like that and instead of saying like, “Wow, actually the models are less good than I thought,” sometimes they’ll say like, “That just shows that the UI that the models are being presented with isn’t very good, or they haven’t been trained enough on how to get the pixels right for their clicking or whatever. But once they get that, then we’ll have even more speedup.” Or similarly, you can see this with compute, like, “If they’re going to be limited by compute, then that just means that once we do build more compute, they’re going to really take off.” So it’s kind of disheartening, because it means that it’s actually surprisingly difficult to get some kind of agreement on if there’s some potential point where things might get totally insane or might just not — Rob Wiblin: What would we see in the leadup? Helen Toner: What would we see in the leadup? Are we going to see anything that’s actually going to distinguish? So yeah, I think that there are still some ideas, but it’s almost like there’s sort of basins that both views are drawn to. And even if they see a little bit of a roll towards the other basin, they just have such a heavy gravitational pull towards their own view — in ways that I think are fully self-consistent — that it makes it really hard to distinguish. Rob Wiblin: So I think around October last year, OpenAI said that it was going to maybe spin off its nonprofit and basically disempower the nonprofit, not allow it to have much control over the business anymore. I think in March this year they did a seeming about-face where they said the nonprofit is going to retain control. But I did an episode around that time with Tyler Whitmer where he explained that he was pretty sceptical, and he thought in fact the nonprofit was going to lose substantial control, maybe almost all significant control. Do you have any thoughts now on OpenAI’s restructure efforts, what we should think of them? Helen Toner: I mean, I haven’t followed this as closely as Tyler has. A couple of thoughts. I think it was really good that they made that seeming about-face. I think it was clearly the result of pressure from external groups — there’s now like a mini ecosystem of OpenAI watchdog groups — and also pressure from the attorneys general, which is really appropriate, because the attorneys general are legally the parties that are responsible for ensuring that a nonprofit carries out its nonprofit mission. I think there’s still a lot of open questions about what that restructure is going to look like, and I’m not particularly encouraged by the most recent update they’ve put out about the nonprofit getting a 20% stake in the PBC, the for-profit. I think there’s very open questions about how they say the nonprofit will retain control, but there’s a wide range of possibilities for what that might look like. A key feature of the current setup is that obligations to the nonprofit mission are the primary legal obligation on board members — who are ultimately responsible for the operations of the whole company. I think trying to retain that setup, where the nonprofit mission, which is to ensure that AGI benefits all of humanity, that that remains as the primary legal obligation [is key] — as opposed to being something that a small fraction of shareholders are pursuing. Or the typical way a PBC, a public benefit corporation, works is that the PBC is allowed to pursue both missions: it can maximise profit or it can choose to pursue a socially beneficial mission — which is just much, much weaker. So this question of what does it mean for the nonprofit to retain control. I also just really hope that the attorneys general, when they’re looking at this, are thinking really hard about how it can be that so many of the same people are on both sides of the same deal. In particular, the board members, perhaps especially Sam Altman, they’re negotiating with themselves essentially for what is the nonprofit retaining, who ends up with board seats, who ends up with equity. I think it’s a really tough transaction to do at arm’s length. Rob Wiblin: Which is legally required, right? It’s meant to be an arm’s-length transaction? Helen Toner: As I understand it, yeah. So I just hope that the attorneys general are really looking very closely at what the company is telling them about who was recused and who was involved in decision making, and how they’re ensuring that the nonprofit’s interests are actually truly being represented. Rob Wiblin: Yeah. Being more familiar with UK charity law, it’s kind of mind-blowing to me that the effective trustees of this charitable organisation are selling the assets of that organisation to themselves. I almost don’t know what more to say than that. It’s extraordinary to me that you could, in any sense, be on both sides of a transaction like that. In the UK I think it would never be permitted. Helen Toner: I mean, we don’t know what is going to be permitted yet. They haven’t got final approval. So yeah, we’ll wait and see. Rob Wiblin: An audience member wrote in with this question: I’ve heard Toner described as extremely normal, in a complimentary way. Is this true? And if so, how does she manage being normal around so many non-normal people? I’m not sure who the non-normal people are being referenced here. I guess maybe me. I don’t know. Helen Toner: It’s very challenging being normal around you, Rob. I don’t know if I’m normal. I don’t think of myself as very normal. I do think of myself as enjoying translating between groups or something — both in a literal way of I’ve always loved languages — I speak German at home with my husband, who’s German, and our kids; I loved learning Chinese when I lived in Beijing — but also in a more metaphorical way, translating between the weirdo existential risk community in the Bay Area and the more staid national security community in DC. I find it really fun to try and understand what are the different perspectives coming in here, what are the assumptions? What are the social norms? So I think that is more how I would describe it. I guess I don’t know exactly what is meant by normal versus weird. I certainly identify as at least somewhat weird, depending on the social context I’m in. Rob Wiblin: I guess in the Bay Area there’s degrees of weird, and I think you’re not weird by Bay Area standards. Helen Toner: That’s right. But I really loved living there and being in a social setting where weirdness was totally accepted and not judged. That’s one of the things that I miss about living in the Bay, for sure. Actually, one other thing that this makes me think of is: probably people listening who’ve heard me speak before are confused by my accent, which happens a lot, maybe related to the translator thing. So I’m originally from Australia, as you know, as are you. And usually I sound American when I’m speaking with Americans. Basically at some point, after living in the US for multiple years, it just felt both more natural and more fun to speak with an American accent. But when I’m around Australians, it feels really artificial to keep speaking in an American way. Rob Wiblin: So are you conscious of your accent changing? Because I think when I talk to Australians, my Australian accent comes out much more. At this point, I’ve lived in Britain for a long time, and I think my Australian accent has really weakened. But I think it probably has come out a little bit more in this conversation than usual. But I’m not aware of that. Helen Toner: If I listen to it, then I can hear it. I would say it’s somewhat conscious, and it depends on the setting that I’m in. But some people seem to find it weird. I think some people really identify very closely with their accent as being a key part of who they are. And I’ve just never really felt that way. I mean, maybe as a kid I enjoyed putting on different accents and learning about them. I think also these days, because I am speaking German every day at home, I’m code switching anyway, so it’s not like there’s my real me and then there’s fake something. It really just feels like there’s just different ways my voice can sound. But yeah, I definitely get a lot of people being like, “Wait, what’s going on with your accent?” Rob Wiblin: Do people immediately know that you’re Australian? Just in normal life in the US? Helen Toner: If I sound American, then no. Rob Wiblin: Interesting. Are people usually able to give up their accent to that extent, or change their accent to that extent? Helen Toner: I don’t know. When I speak German, people also tell me that I sound German. So I don’t know,.I feel like there’s a wide range. Another similar thing that I think quite a lot of people do is Americans from the south will pretty often sound more southern around family and friends, and then when they’re talking to northerners, they go towards more of a standard American accent. So I think there’s lots of versions, and that different communities or cultures handle it differently. Rob Wiblin: Another audience question was: This field is changing so rapidly, both via technological improvements and via shifting global geopolitical relations that affect the semiconductor supply chain. How do you keep up with all the stuff that’s going on? Helen Toner: I mean, it’s a challenge. This is my full-time job and it’s hard to keep up with things. I do think that Twitter/X is still quite useful as a source of news. Rob Wiblin: For better or worse. Helen Toner: I have a set of Substacks that I get in my inbox that I try to read. Some of them I try really hard to read every post; other ones I just sort of skim and carry on. And you can go to my Substack at helentoner.substack.com, and there I have a list of the recommendations of different substacks that I find really useful. If you want a lighter-touch way to catch up, then CSET publishes a monthly newsletter about the biggest AI news, which I actually find helpful as sort of a roundup — because it’s like a fire hose day to day, and then once a month it’s like, “Here are actually the big stories that really matter.” Rob Wiblin: Transformer has a weekly update on Fridays. I can recommend that one if you want a once-weekly sense of what’s going on. Helen Toner: Yep, yep. But I think it’s hard. And something that I’ve kind of accepted, working in the space that I work in, is that I’m in the intersection of many different fields — there’s AI, there’s national security, there’s China, there’s US politics, US policy more broadly, big tech regulation, platform regulation, social media, economics, you know — so I’ve just accepted that I’m never going to be an expert on every single thing that touches my work. I’m never like, “I have to stay up to speed. Am I staying up to speed, yes or no?” It’s always just a matter of prioritisation, and trying to choose where to put emphasis and where to make sure that I’m spending time. It’s a work in progress. Rob Wiblin: How do you keep up to date or try to have some sense of what’s going on in China? It seems particularly hard to gauge. Helen Toner: A few different answers. I do think the best Substack that I know of on this is ChinaTalk by Jordan Schneider, which is really good. CSET actually has a tool called Scout. So if you go to our public data platform, which is called the Emerging Technology Observatory, eto.tech, there’s a tool called Scout which does auto-translated and summarised, but human-curated news from China, which is very helpful. And then lots of other miscellaneous sources. One thing I will say, if you’re interested in China and AI, I think it’s really worth laying some foundations of China and non-AI topics. Whether that’s like reading The Beautiful Country in the Middle Kingdom, which is a sort of comprehensive history of the US-China relationship, or learning about the history of the Communist Party, or looking at some of the history of tech development and the economic development in China generally since the ’80s, I think it’s really valuable. I think people sometimes come into this topic purely from the AI angle, and only know about China inasmuch as it relates to AI, and I think that leaves you with a bit of a shallow foundation. I also try to not always just be keeping up with the news, but going back and reading the fundamentals as well. Rob Wiblin: You were part of the founding team for CSET, and I guess you’re leading it now. What’s your vision for what CSET can uniquely do or uniquely add to policy understanding in DC? Helen Toner: I think it’s a much more crowded space now than it was when we started, and I want to partially claim a little bit of credit for that — that we were early to having this in-depth, technically informed analysis of emerging tech and national security matters. I think a lot of other organisations have gotten on that bandwagon — which is great and really good, really positive for the world. I think what CSET can still offer uniquely is a combination of being really independent, neutral, rigorous — so not coming in with a particular point of view that we’re trying to push, but really just trying to shed light on what is going on in reality. Kind of calling balls and strikes. I think our data team is a huge asset. This is a set of data scientists and engineers and data-focused researchers who help our research analysts, research fellows, senior fellows to make use of different kinds of datasets for different kinds of research questions. So things like looking at the semiconductor supply chain, for example: on that same public data observatory at eto.tech we have an interactive tool that lets you look at all these different parts of the semiconductor supply chain, for example, and which companies and countries are involved in which parts of that very complicated, very large supply chain. Or looking at translation: machine translation is really good and we use a lot of machine translation, but for important documents it’s also very helpful to have a human editor go through and make sure that those key terms are translated appropriately, like that all the weird CCP jargon is using the correct term so that you can connect it to past documents. So I think that is a big asset as well. And yeah, having a team that is focused on AI and emerging tech, and not just a small part of a larger think tank, but really having the whole organisation dedicated to those questions. Rob Wiblin: Yeah, the conversation is more crowded now. I feel like, unfortunately, there’s more and more actors who are being paid by vested interests to push a particular line, to push a particular agenda. I mean, you can’t blame tech companies for having an interest in what’s going on in DC and hiring lobbyists to make the case for the things that they want. I guess one that stands out to me is Nvidia’s lobbying around the export controls. They’ve paid people to say some sensible things; they’ve paid people to say some things I think are extremely not sensible and not always true either. Do you feel like the conversation is getting a bit degraded or a little bit corrupted by the fact that there’s now people with clearer agendas, a clearer ideology as well, who are trying to have their way with things? Helen Toner: I definitely think you’re seeing that the companies are staffing up and they are becoming increasingly active on AI policy topics. And I think that that has pretty predictable effects in terms of pulling things towards the company’s interests. So, yes. One counterpoint I would give is: as we record this, this is not yet finalised, but in California, there’s this bill called SB 53, which I think is an astonishing triumph of sensible policymaking over distorted rhetoric. This is the successor bill to SB 1047, which was enormously controversial last year, a subject of huge amounts of rage and fear mongering and hate. And what happened in the aftermath of 1047 was that Governor Gavin Newsom vetoed it, but set up a commission to study what would be the appropriate way to regulate, and how concerned to be about the kinds of issues that SB 1047 was designed to help with. And he created an expert commission. The expert commission wrote a report that was really high quality. And Scott Wiener, the California senator who put up SB 1047 last year, put up a new bill that was drawing heavily on this report, and has gotten a lot of support and a lot of more consensus, and it’s passed through the legislature. And as far as I know, Governor Newsom is indicating that he will sign it. Don’t want to get ahead of the game there. Rob Wiblin: But it’s been massively less controversial. Helen Toner: It’s been massively less controversial, and I think will certainly be a smaller step than SB 1047 would have been, in both good and bad ways. I think the potential ways that it could go wrong are much lower, as well as potential benefits if 1047 had really worked out the way that its supporters hoped it would: also lower. But I don’t know. I put that out there as an example of how I think there is much more lobbying in the space, there is much more corporate influence, and that is probably going to stick around — but there are still ways that kind of good policy can be made. And I do also think of that as a big value add that CSET can provide. Often policymakers talk to companies because the companies can tell them what is real and what is realistic and how the technology actually works. And you don’t want to do a Europe GDPR of creating all these rules that have all these unintended side effects because they’re disconnected from how the technology actually works. So it makes sense that they want to talk to industry lobbyists if they’re going to have that real talk for them. And that is a place that CSET can step in as well, and organisations like us, and have that technical depth to be able to go in and say, “Here’s what’s realistic; here’s potential unintended side effects” — but not be ultimately promoting the company’s interests; instead having a broader public interest in mind. Rob Wiblin: Do you have to worry about CSET’s research being influenced by the opinions of its funders, and maybe wanting to pander to them too much? Helen Toner: I think every organisation has to pay attention to that, is dependent on something. We work really hard to bring in funding that is compatible with our model of following the evidence where it goes, being independent, not pursuing a certain agenda. So when we take industry funding, which is a very small amount of our budget, we make sure that it is really sort of cordoned off in a way that makes it very difficult for them, as with any funder, it’s totally off limits to have that influence any of our findings. We’re especially cautious about that with industry, but also with philanthropic funders. We have a large amount of funding from Open Philanthropy, and we also work really hard to make sure that our research and findings are not just trying to show whatever we think that they would want us to show, but instead is what we think is true. And you know, that’s really what has built our reputation as well. At this point, something we hear regularly from funders is that that is why they want to support us, because they know that we are going to do that: we are going to make sure that we are not going where our financial backers might be inclined to go, but we’re going to really be independent and tell it like we see it. So that is a value add in itself. Rob Wiblin: I think people can often tell when you’re being paid to push a particular line, because it’s like the conclusion will always be the same no matter what the incoming evidence is, or what the updates are, and people tune out. Helen Toner: Yeah, I think that’s right. I also think even if it’s not a financial thing, I think there can be ways in which… In AI risk debates, for example, if people have the feeling that your conclusion is going to be risks are really high and we need really severe interventions: if they feel like that’s going to be your bottom line in every case, on every paper — Rob Wiblin: And you’re not curious to change your mind or learn something new. Helen Toner: Yeah, yeah. Then they also tune you out. And I think that is also a strength that we bring, is not doing our work that way. Rob Wiblin: Are there any particular talent bottlenecks in DC? What other skills or knowledge could you bring into the conversation, have people work at CSET or wherever else, that would improve people’s understanding, that would actually push things in a smarter direction? Helen Toner: There’s all kinds of things. Having a basic understanding of AI as a technology — which doesn’t necessarily mean a computer science PhD, but being comfortable enough with calculus and linear algebra and a bit of programming to have played around with models yourself and have a basic intuition for how they work — and then bringing some other kind of expertise, like really any kind of expertise. I think something that is a bit overrated in AI is purely computer scientists coming in and combining that with learning the policy side. Because AI touches so many different fields, if you’re an economist, if you’re a historian, if you’re anthropologist, if you’re an accountant, if you’re a teacher, I just think there’s a lot of space for a very wide range of different kinds of expertise. I do think it makes sense, inasmuch as you can, to try and get that basic understanding of AI — largely so it doesn’t seem like magic, trying to get deep enough on the technology so that you have a rough sense of how it works. But beyond that, I think there’s a huge range of needs. Rob Wiblin: What’s something that you think a lot of listeners might get wrong about the AI policy discourse in DC? Helen Toner: One thing could be thinking that policymakers are really uninformed or sort of stupid on AI issues. I mean, certainly different policymakers interact with these topics differently, but I think… Rob Wiblin: Is this a widespread perception, do you think? Helen Toner: I think a lot of people think that policymakers have no idea, and are really ignorant about tech topics. It’s really important to remember two things: just how busy policymakers are, and depending again on the policymaker, how many topics they might need to cover and how many issues, including often just like truly urgent crises they need to handle. So if they don’t know something or misunderstand something, it doesn’t mean they’re stupid; it means that they’re busy doing something else. And I think another thing that people underestimate is how insular some of the conversations and some of the sets of ideas are. From the perspective of many people in DC, they’ve just never really encountered an argument about, for example, why export controls would be good, or why we should expect AI to develop in a certain way — because so many of these conversations happen in these pretty self-contained social groups or online networks. So I tend to think there’s a huge amount of value in trying to create more accessible, no-prerequisites-required introductions to concepts, or explainers of key concepts. We do quite a lot of explainers at CSET as well, that bring people into these key ideas and let them understand them and play with them in their own head, as opposed to just having them be discussed verbally in conversations among friends. Rob Wiblin: You engage a lot, and I think very productively, with people across the full spectrum — from being very optimistic about AI to being very doomy, very troubled about the future. Is there any particular evidence that could help to settle in your mind or give you a big update as to how worried to be? Helen Toner: I think probably no one single piece of evidence, but different things that we could see over the next few years. Two things we’ve touched on, but to say them again: one is this question of how much is AI going to be a tool versus how much is it going to be really agentic, autonomous, out there and kind of doing its own thing. That’s something I expect we’ll get evidence on. I think this year the updates have been mostly that it’s turning out to be more challenging than people expected to have AI agents that can even do pretty basic things, like booking flights reliably. I’m sure they’ll get there. But it’s turned out to be a little bit more challenging than some of the most bullish people were expecting. But I think the more that we see AI systems that can really go out and do, in practice, on a regular basis, longer time-horizon tasks, or also like messier tasks — things that aren’t as cleanly boxed and specified as like, “Here is a nicely chunked out thing for you to do,” but instead, “Here’s a squishier thing. Can you make some progress on it?” That’s one big piece. And then another piece is this question of how superintelligent things can get. I think there’s starting to be a little bit more discourse on this, a little bit more looking for evidence, or breaking down different types of capabilities of how far above human level could you get on a given thing? I would love to see more of that, because I think assuming that things will plateau at human level seems clearly wrong. But also Bostrom, in Superintelligence, he talks about these superpowers, that the AI will have superpowers at all of these different things. And that also seems pretty unlikely to me. So trying to figure out where in the middle we will land seems really important. Rob Wiblin: Is CSET hiring for any roles that people in the audience, if they want to dive into this world — or maybe are already someone in this world, but are considering moving to CSET or DC — should think about? Helen Toner: Depending exactly when this goes live, we’re hoping to have a post up pretty soon for a research fellow or senior fellow. This would be focusing on frontier AI. So this would be a role to lead research at CSET, would need someone with a graduate degree, a few years of research experience at a minimum, maybe multiple years, maybe some government experience for a senior fellow position. I think this will be a really exciting role. We’d love to see lots of really great applicants. If that role is closed or if it’s not the right role for you, then we’ll likely also be hiring for additional research roles, data roles, communications and external affairs roles, potentially operations roles through 2026. So definitely keep an eye on our careers page, sign up for our newsletter, stay in the loop. Rob Wiblin: Is CSET growing in general? Helen Toner: We’re roughly happy with our size. In 2019 the interview was about the 30-person research group in DC. We’re now more like 50. We may fluctuate a little bit around that, but I think pretty happy with the size. A big part of CSET’s model is being able to have a cohesive team culture where people are collaborating across teams and there’s a sense of, as an organisation, what are our key priorities — which gets difficult if you’re scaling up much beyond that level. Rob Wiblin: The last six years since our conversation in 2019 have been pretty eventful. I can only imagine the next six years are going to be as eventful, or more so. Helen Toner: Talk to you in 2031! Rob Wiblin: Hopefully we get to talk before 2031. But yeah, my guest today has been Helen Toner. Thanks so much for coming on The 80,000 Hours Podcast, Helen. Helen Toner: Thanks, Rob.