Latest / 80,000 Hours Podcast / Inside the Biden admin’s AI policy approach | Jake Sullivan, Biden’s NSA | via The Cognitive Revolution
Transcript
- Jake Sullivan: I'm determined that competition not turn into conflict, including, by the way, on issues associated with AI risk. I'm not someone who sits here and just says, "It's cold war time, baby!" Not at all. I do not seek a cold war. I think there's not going to be an end state where one side just wins and the other side loses. But where there is going to be intensive competing, jostling, a little bit of elbow throwing, but where that intensive competition, we have sufficient guardrails, and it doesn't spill over into conflict. I think that that is manageable. That is basically the blueprint that we pursued over the back half of the Biden administration: compete like hell, but intensively manage that competition so it didn't tip over into conflict. And it requires a stewardship, and a deep, intensive diplomatic engagement between the two sides. And I think we've shown in this iteration that is possible. And if we continue to build the muscles for it, I think it can withstand even the advent of very powerful artificial intelligence. How's that for optimistic? Luisa Rodriguez: Hey listeners, for this episode we are cross-posting an episode from The Cognitive Revolution podcast with Jake Sullivan, who's a former National Security Advisor to the Biden administration. We asked Nathan Labenz, who's the host of The Cognitive Revolution podcast, if we could cross-post this episode to our feed because we felt like it didn't get nearly enough attention in our opinion, given how high quality and in-depth the discussion in this episode seemed to be about the landscape for AI risks and opportunities, and in particular a lot of really interesting material about coordination and competition between the US and China on AI. I think regular listeners to our show will find many parts of this episode interesting, especially if you're thinking about how we might expect the US government and the geopolitical landscape to develop and respond to advanced AI. So I'm really excited to bring it to you now. Without further ado, here's Jake Sullivan and Nathan Labenz from The Cognitive Revolution. Nathan Labenz: Jake Sullivan, former National Security advisor to President Biden and currently the inaugural Kissinger Professor of the Practice of Statecraft and World Order. Welcome to the Cognitive Revolution. Jake Sullivan: Thank you very much for having me. Nathan Labenz: I'm excited for this. We have a ton to cover. So I'm going to try to be brief in my questions, which is not normally my style and give you the bulk of the airtime. I thought we could kind of structure this by just setting the stage. I think so many AI conversations kind of go a little awry because people don't necessarily have the same expectations of or even understanding of where we are in the technology today, let alone where it's going to go into the future. So just to give you a chance to kind of share with the audience, like where you're at: How AGI-pilled are you? And, you know, I guess that means like how soon do you expect powerful AI? How powerful do you expect it to be? Where do you put that in relation to other national and global crises that we have? And maybe for bonus points, like, how would you compare yourself to other parts of the national federal government? Jake Sullivan: It's a great question, and like most great questions, it doesn't have a totally straightforward answer. How AGI-pilled am I? I believe that the possibility of transformative, powerful AI coming in the very near future, that is in the next couple of years, is something we have to take very seriously. I took it very seriously when I was National Security Advisor, and you have to build policy and make strategic decisions on the basis that that is a distinct possibility. But do I believe it's an inevitability or we certainly are going to see it by 27, 28, or even 2030? No. And that's because super smart people make the case that it's coming very fast and other super smart people make the case actually, there's a number of things that have to happen and those things may take quite a long time before we get to something like AGI or powerful AI or even ASI. So I build a planning assumption that this is distinctly possible. And if it comes and when it comes, it will be transformative across every dimension -- economic, societal, and yes, national security. And it is on par with the most urgent and important national security issues there is because it touches everything -- from the future of warfare to the future of nation state competitiveness to the future of non-state actors and the threats they pose and everything in between. Nathan Labenz: Yeah, it is a tough epistemic environment with the divergence of top thinkers being just so radical. That's a frequent theme of this show. What would you say have been... maybe you could kind of weave into a little bit of like, where do you think Congress is at on this? Where do you think the military is? There's obviously the agencies that are probably all over the place. But what has sort of most informed your thinking? Has it been like private conversations with leading developer CEOs or, you know, how much is just kind of getting hands on with the technology itself? How much are these sort of manifesto type documents impacting the Washington discourse? Like where are people taking their cues from in Washington? Jake Sullivan: So as National Security Advisor, I really tried to cast a very wide net. I met with the leaders of all the frontier AI labs. I met with technologists who are AI skeptical. I met with investors, I met with academics. And of course then I convened and engaged with the entire national security enterprise of the US government that what are we doing to integrate and adopt AI capabilities to advance America's national security? In fact, towards the end of my time as National Security Advisor, we put out a National Security memorandum, which sounds kind of boring, but was a lengthy document, which in fact, the Trump administration has not discarded, that sent clear directives to all of the national security agencies of the US government, including the Pentagon, including the intelligence community, but also Treasury and Commerce, State Department, etcetera. And what it said was, right now America is in the lead at the frontier. We could debate by how much, but we are behind the curve when it comes to integrating and adopting these capabilities and applying them for national security purposes and we gotta move faster with greater urgency, with greater dexterity. And I think that remains the case today, more than six months after I left office. So are there people inside the Pentagon who see the promise and the opportunity here and also see the threats and risks that come with advances in AI? Of course there are. Brilliant people. But culturally, bureaucratically, institutionally, are there obstacles to taking advantage of the full suite of capabilities that is there today and increasingly will be there in the future? Hell yeah, there are. And that's something that I think takes leadership from the White House to try to break through. We did that particularly in the later years of the Biden administration. I see evidence that there is some push along those lines from the Trump White House as well. Oh, and you asked about Congress. I should say a word about Congress. I think Congress at the moment, it's hard to talk about it monolithically because there are members who get the both the opportunity and the challenge, who recognize how transformational this is going to be, and are thinking hard about what role they should be playing. But I would say in the main, when you look at the pieces of draft legislation or you look at the hearings or you look at the public statements of most members, I think there is still a gap between where Congress is and where it needs to be in terms of a sense of urgency and priority for this issue. Nathan Labenz: Yeah, that sounds safe to say. For people that want to change that, there's this notion in the AI safety community that if we can just get some time with members of Congress and we can show them our scary demos, you know, we can show a chatbot that will help you make a bioweapon or at least attempt to or whatever, that that will sort of snap people to attention and get them to engage more seriously. I wonder how much those sorts of awareness-building drives have changed the overall landscape in your mind. Has any of that reached you or have you seen the impact of that? Jake Sullivan: Yeah, look, demos, you know, going back years now to early in my time as National Security Advisor, definitely made an impression on me. Because you can describe what an LLM can do, you know, I can coherently understand that, but when you actually see the capability live and in color, it does have a different kind of impact and that probably is the case on the Hill. But I would submit to you that generally speaking, with complex areas of regulation legislation, and this is among the most complex there is, Congress really only tends to react when there are real world impacts. That could be a catastrophe like a 9/11, which, you know, gave us the Patriot Act in part for good and in part for ill. But that also could be a more cascading set of developments in the world -- like job displacement in key industries, like announcements by other countries of military capabilities that give them advantages over the United States. And it will be a collection of those moments in my view that is really what's going to get Congress into action more than just demos of the tech. And that's not great because it means that we're going to be playing catchup as opposed to being ahead of the curve, but I think that is the world we're living in right now. And I would also say that the Trump administration has sent a very clear signal they don't want to see legislation. They don't want to see an effort to, you know, impose something from the Hill or even from the executive branch. And, you know, even setting aside that question, it seems to me this is a very difficult area to be confident in what is right from the point of view of rules, standards, trust and safety. And coming back to your first question, we don't know where the tech is going and when. So it's very hard sitting here today to be too dictating of things in light of the profound uncertainty that we're contending with. And that's why I think preparation and flexibility, being ready, developing a set of rules and standards of tools that Congress or the executive branch could use as necessary as we go forward is the right thing. And then choices about when to deploy those based on how we see the tech evolve should be done in careful dialogue between Washington and Silicon Valley and the rest of the country. Nathan Labenz: I think you're absolutely right that sitting down today and trying to write a long list of rules or standards is an ill-fated undertaking. We've done a couple episodes recently on more creative attempts to try to bring dynamism into the regulatory environment with things like the SB 813 in California proposal where there could be some sort of marketplace for regulation. Just going a little bit deeper into your AI worldview though: I know it's been your job for the last four years to anticipate, and prevent, ideally, and then respond to if necessary a lot of bad things. Have you had time to develop any sort of positive vision for the AI future? Like are you imagining a new social contract? Are you, you know, signing on to the Bernie Sanders call for maybe a four day work week? In your imagination like what is the upside of all this look like? Jake Sullivan: You know, one of the things about the way that I think is I tend to think in quite linear and concrete terms. And so I have a hard time with these kind of big abstractions about new social compacts or about recursive self-improvement leading to a nation of geniuses in a data center. These kinds of concepts I just have a hard time wrapping my head around. So I tend to think of things in a pretty straightforward linear way. And I see kind of four big buckets -- security, economics, society, and then these existential questions -- where there are both risks and opportunities. And our job should be to identify the risks very concretely. So in the security space, there are non-state actor risks like cyber and bio. In the state actor domain, there are risks around wonder weapons and military advantage by one of our adversaries or competitors. But there are also opportunities in the security realm that AI could actually enhance America's national security if we remain in the lead on it and we deploy and adapt it effectively. Similarly in the economic domain, there are genuine risks of job displacement, there are genuine risks of concentration of wealth and power that come from essentially the benefits flowing to a very few if we're not careful. That is a big risk. And then there are massive opportunities on productivity, on problem solving the issues around, you know, climate and energy or public health or what have you. And so what I try to do is think, all right, let's get very concrete about the risks, identify them, make a typology of them. Let's think very concretely about the opportunities and then are there steps that government can take to increase or expand the opportunities and to manage and minimize the risks. If the answer to that is yes, act. If it's not, then don't act. And that's really how I think of it. Then there's this fourth bucket of existential. And here I think it's just too imponderable at the moment, these kinds of like the robot apocalypse or superintelligence leading to nirvana and to space. You know, I mean you can tell the very positive story and you can tell the very negative story. For me, I'm really kind of focused on the first three buckets more than the existential questions right now. What can we do in the security domain and the economic domain and the society domain? And on society, the risks of misinformation, of alienation are real -- but the opportunities of giving every person access to teachers and doctors and nurses and personalized medical care, these are exciting opportunities. So that's how I think of it, basically in a kind of boring way to be honest, but in a way that I think the debate about AI is not entirely contending with. I've asked research assistants of mine here at Harvard, get me a set of articles that actually walks through what are the real risks and what are the real opportunities that go beyond the headline or the whiz bang or the doomerism. And honestly, there's not that much out there that really walks through this in a deep and coherent way. And I think more of that kind of analysis could help us really bring a greater degree of rigor to the conversation about where AI is going and where government action should go to try to make sure that it works more for us than against us. Nathan Labenz: Yeah, it's interesting. I think I take the existential concerns more seriously. Certainly in terms of a like, I would bet my p(doom) is higher than your (p)doom. Jake Sullivan: Just to be clear, I take them seriously. It's just right now, I feel like I am personally less equipped to give definitive answers to how to deal with the existential questions than I am immediate issues like the cyber risk that AI could pose in 2025 or how to think about dealing with the fact that whatever happens overall with growth productivity, job gain versus job loss, a lot of people are going to be disrupted and will need a better form of social insurance than we've currently got on offer. To me, I want to put more energy and emphasis in working through those questions because I actually believe that if we answered those questions effectively, and designed a set of approaches that allow us to manage risk and opportunities in those buckets, it will set us up better to deal with those bigger questions of misalignment, AI risk, (p)doom, what have you, which I am not trying to shrug off or say isn't significant. I'm just suggesting that racing right to that question, which I think so many people do, takes you past things that are very live challenges right now that are far less imponderable, far less contestable, have far less of a spectrum of, you know, where people fall on them. And we've got to do hard work of building consensus around those issues. I think in doing so, we'll build muscles that will put us in a better position to handle the existential. Nathan Labenz: Yeah, that does seem to depend on a certain timeline assumption. I guess part of how I've operationalized my thinking is like, I do take pretty seriously the idea that like Daniel Kokotajlo and the AI 2027 folks could be right and like one of these frontier developers just might set off an intelligence explosion in the next two to three years. But then I'm like, but there's probably not much I can do about it if that is going to happen. And so maybe I just have to kind of try to stay sane and like play for kind of longer timelines and more mundane scenarios. It sounds like you're kind of saying the same thing, but with a little bit less weight on the short timeline. Jake Sullivan: I guess what I'm saying is we have to take seriously the possibility of AI 2027 -- and I not only read the piece, I shared it extremely broadly. I called people and said how right is this? I talked to a lot of technologists who said this is a totally credible scenario. I talked to others who said it's less credible. So I of course take that very seriously. I guess my point is, sitting here in 2025, let's assume that that, you know, is the direction we're heading. I still believe that a more concrete and application-specific conversation about what we are going to do to manage particular risks and to enhance particular opportunities, will actually set us up better in both the development of policy tools and in the depth and specificity of the conversation that can be had between the technology community, the policy community across this country. And I actually think that will help put us in better shape to handle that kind of extreme scenario if and when it comes. And yeah, sitting here on your podcast today, I'm not denying that that could come in 2027. And I don't throw my hands up and say, well, there's nothing we can do about it. What I'm saying is, let's work through the things we know are coming, are already here. Let's get good answers to those questions and in so doing, I think it tees us up better for these kind of deeper existential questions about intelligence explosion and takeoff and the like. All of which I spend part of my time both worried about and studying and talking to people about because I do take it very seriously. Nathan Labenz: Perfect. Well, thank you for bearing with me through the high-level stuff. That’s a perfect transition into getting a lot more concrete. I think we're talking about a week after the AI Action Plan was released and ton of stuff in there. Broadly, pretty well received including by people that I think expected to not like it. What was your take, you know, what would you say are the highlights and lowlights from your point of view on the AI Action Plan? Jake Sullivan: My overall reaction is that I thought it reflected a bipartisan consensus on a number of critical points, including the kinds of investment and support the US needs to be providing to ensure we stay ahead at the frontier, including even in areas related to trying to build a global AI alliance, working with other like minded countries as well. And I thought the way that they handled some of the risk and security issues including on bio was laudable. They re-upped a proposal that the Biden administration had put forward in the executive order on pre-synthesis DNA screening for example. So I thought all of that was to the good. I thought there were some politicized elements to it, but, you know, that kind of go with the territory of the way this current administration operates. My main concern with it actually was the gap between what it says about needing to stay ahead of China in this race, deny them access to high-end compute, and then the reality of what the Trump administration is actually doing with its H20 decision and other things. So my biggest concern is that I think there's a dissonance between the AI Action Plan and what the Trump administration is actually doing and that's of concern to me. Nathan Labenz: Yeah, I definitely want to unpack the whole export controls and China thing in some depth in just a second. Another big question I have around just like all these action plans in general with a sort of, are we actually going to achieve abundance or not background context in mind is like, how likely should we think it is that the government is actually going to adopt these technologies? Like last I heard, and you may tell me differently, but from what I heard from folks in the Biden White House and, you know, some folks in the Trump administration, there's still no ChatGPT or anything similar at the White House, right? I think people are like going home and running deep research queries on their own computer and then like trying to, you know, memorize those results and bring them back into the office. And that just seems like not a great baseline from which to try to lead the national charge toward global AI leadership. So, how do you think about that disconnect? Jake Sullivan: Yeah, it's a disconnect. I acknowledge it. You know, I am now an avid would be an understatement user of these tools. I used them much less frequently when I was actually National Security Advisor and yet working on the issue every day. There's a few reasons for this. You know, the culture of government in terms of new technology adoption is historically slow and that's true in this case as well. There are legal issues particularly in the national security enterprise, especially if you're talking about things on the, what we call the high side and classified compute and the like. But in general, I think it's imperative, whether it's the White House or the agencies or beyond, that the adoption incorporation these tools get accelerated because you're right, I think it's axiomatic that you cannot effectively lead and govern on this set of issues if you aren't an effective user of them for purposes in advancement of America's interest. So I think we should see more of it. It's not clear to me, you know, I think this is going to be piecemeal to a certain extent and there'll be some earlier adopters than others. But I would be an advocate for seeing this get incorporated as rapidly as possible. Nathan Labenz: How about the recent deals that President Trump announced when he went to the Middle East with the UAE and Saudi Arabia? My sense is that you would not have supported those deals or, you know, we wouldn't have seen similar deals under a Harris administration. It seems like a sort of a complicated business. I think there's a lot of arguments going in different ways. What do you think are sort of the primary considerations that people should have in mind and where does that lead you to come down on those deals? Jake Sullivan: Look, for me this actually isn't too complicated. Number one, I would like to see the world running on American AI, not Chinese AI, and American chips, not Chinese chips. And that means diffusion. It means that we should be selling our chips and seeing data centers built around the world, including in the Gulf, where countries like the UAE and Saudi Arabia have invested deep interest in having AI be a central part of their national development enterprise. So one, I believe in diffusion. Two, we need security and high standards to ensure that there's not diversion of chips to China that vitiate the export controls, that you deal with things like insider threat challenges, cyber challenges, physical security challenges, etcetera. And we should hold everyone to a high security standard. And then third, I believe the United States should not trade one form of dependence on any part of the world, including the Middle East, for another. So we were long dependent on the Middle East for oil. We should not become dependent on the Middle East or anywhere else for compute. So I believe that the lion's share of the compute buildout by American AI should be in the United States of America. And so for me, the question in assessing these announcements coming out of the Middle East -- and by the way, that's all they really were, they were announcements. It's hard to know exactly what's going to happen, is that the devil's in the details. I support selling chips to these countries so long as they, you know, meet the necessary security standards. But I support a limit on how many chips we sell because I think we should not end up in a circumstance where all of the future buildouts and all of the future training runs are being offshored and outsourced. I think that would be repeating a mistake we've made too many times with past industries and we shouldn't repeat it here. You know, I'll be the first to say, I engage deeply with the leadership of the UAE on their AI ambitions. I very much supported deepening partnerships between US technology companies and UAE technology companies. I continue to support that. But I think again, the critical point for me is let's not end up dependent on others for access to compute in the future. Let's make sure that we are building the lion’s share of the compute here in the United States. Nathan Labenz: So I think that dependence thing seems very common sense to me. I guess other dimensions of the deals that I would highlight would be maybe values and control. There's all this talk, and we're going to get to China next, but there's all this talk about making sure that AI reflects American values. I think it takes some squinting to put it mildly to say that the governments of these countries really uphold or, you know, adhere to American values. And then on the control side, this seems like it might be actually one of those things that you might do a little differently depending on how seriously you took the AI 2027/2028 superintelligence scenario because you might say, geez, like a five gigawatt data center in Saudi Arabia might be just the kind of place where somebody might set off a recursive self-improvement loop that would be outside of your US jurisdiction and control and like, maybe we just shouldn't set up the preconditions for something like that to even be possible. And yet, it seems like kind of here we go. Jake Sullivan: I agree very strongly that we do not want leading-edge frontier training runs done in the Gulf or in a lot of other countries too. I want them done here in the United States of America. And I think it's incumbent on us to ensure that we break the bottleneck on power so that it's not a hard constraint on being able to conduct those training runs and we should do them here in the United States. I very much agree with that. Look, when it comes to values, my view on this is that there are a number of countries around the world, including in the Gulf, who have different values when it comes to democracy and human rights than the United States does. We deal with them economically, we deal with them in technology. I don't think that that is wrong. I just think we have to be clear eyed about what we stand for and try not to directly aid and abet repression and things along those lines. But you could take that argument to its logical extreme and end up really basically severing relations with a lot of countries around the world and I wouldn't be prepared to do that. So I think it is a factor and it's something that should be part of the discussion, part of the conversation, and regularly is between our country and countries in the Gulf, but I have a hard time saying that I would put... I certainly wouldn't put those countries or other similarly situated on a blacklist. Nathan Labenz: So why doesn't the same analysis apply to China? I mean, it seems like a lot of what you said there in terms of different attitudes on democracy and human rights, but still got to deal with them, we can still trade, etcetera, etcetera, like could be applied to China. But it seems like there's a sense that there is some unique threat that comes from China today. And I always ask this question, I feel like I never get a great answer. What is that threat? How should I understand the threat as, you know, an American who grew up in the Midwest and has certainly lived through some of the de-industrialization problems that we've had but consider those to be in the past? Like sitting here in 2025, like looking ahead, why should I be concerned that China is, you know, what should I be concerned that China is going to do such that we like can't deal with them in the same way we can deal with these Gulf countries? Jake Sullivan: Well, first, let me just say that I believe we should not decouple from China. I believe we should have an economic relationship with them. I've been very much on record in saying that we're competing intensively with this country, but we're also going to have to learn how to live alongside one another as major powers for the indefinite future. So you're not going to hear from me you can't deal with China at all. You know, I am focused on what I have called the small yard and high fence where these high-end capabilities with national security applications can be controlled and restricted. But to directly answer your question, which is what should we be concerned about with respect to the PRC? You know, I told you earlier, I like to be concrete and not just speak in abstractions. So let me just give you a few examples. First, you said the China shock is in the rearview mirror. The author of the China Shock, David Autor and one of his co researchers just wrote a quite compelling piece in the New York Times saying get ready for China shock 2.0 and it's going to be worse. And that is because China is pursuing a strategy of massive subsidization of strategic industries and then trying to flood the market with cheap goods that undercut workers and businesses who play according to a different set of rules in the United States and elsewhere. And that if left unchecked could easily have the effect of a further hollowing out of our basic industrial base, including in the industries of the future. Economists call this overcapacity. You know, I call it basically flooding the world with cheap manufactured goods based on a state-directed capitalism where China is not playing by the same rules as everybody else. I think that's a challenge. Now, we don't have to go to war to deal with that challenge. But I do think we need a set of measures coordinated with other countries in the G7 and beyond to push back against that. So that's one. The second type of challenge I think is best exemplified in an episode that happened back in 2019 when the GM of the Houston Rockets made a comment about freedom fighters in Hong Kong. And the Chinese government went nuts, basically told the NBA, you got to shut this guy up. The NBA essentially had to kind of bend to the will of China on that particular point. Now, is that a threat to every American? No. But if the entire world is run on Chinese AI, and China is effectively saying there's going to be a price for your speech, then that is a threat to the American way of life. And China has shown a willingness and a proclivity to do that. So that's a second challenge. Third: while I was in the job, I had to deal with something called Volt Typhoon, which is has been publicly reported upon and I can't say too much about it, but I can say this, we have seen the prepositioning of malware in critical infrastructure in the United States by the PRC and that is a direct threat to water systems, electricity, pipelines, whatever it may be and something for us to be very concerned about. Why are they doing that? Fourth: China has engaged in the world's largest peacetime military buildup, probably in all of human history. Why? Part of it's about Taiwan, which is its own risk and threat and maintaining peace and stability across the Taiwan Strait I think has to be a paramount priority for US policy because it would be totally catastrophic if that happened. But they're also looking to project power globally in a different way. So what makes China different from other countries? One of the things is they are the only country that actually has the attributes and capacities to compete with the United States, matched with an ambition to surpass the US as the world's leading power, economically, technologically, militarily, diplomatically. And I think a world in which China is writing the rules, China is running the tech stack, everyone is dependent on China is a world of greater coercion, less freedom, less good economic opportunities for Americans and a greater possibility of China flexing its military muscles to subjugate others. So that is my set of concerns. Now, can this be managed in a way where we have a stable relationship with China? I engaged in intensive and deep diplomacy with my counterpart Wang Yi. and I believe that we paired intense competition with intense diplomacy to good effect. And I also am determined that competition not turn into conflict and that we maintain space for cooperation on key issues, including by the way, on issues associated with AI risk, where we inaugurated an AI risk dialogue with China to talk to them directly about issues that challenge both of our countries. You know, I'm not someone who sits here and just says, it's Cold War time, baby. Not at all. I do not seek a Cold War. What I seek is for the United States to maintain the capacity to ensure its way of life and not have that way of life undermined, pressured or put at risk because of too much capacity and power accruing to a country that does not share our values and has shown a willingness to use its coercive power, frankly, recently against the United States with the rare earth magnets thing, but also against other countries. So that's a long answer to your question, but that's how I see the set of challenges that we're confronting when it comes to the PRC. Nathan Labenz: Yeah, I think most of that sounds very reasonable to me and you look at a giant military buildup and one does have to wonder, you know, what exactly are they planning to do with it? I think you could offer benign answers, but, you know, in your job it's not right for you to just accept the first explanation. I totally get that. How do you think that they understand what we are trying to do? Because you know, I'm not a big analogy guy, but a sort of short story of like the runup to World War II, at least in the Pacific was like, we cut Japan off from oil. They were like, well, we have a limited window where we've got to go for everything or we're screwed. So here we go. You know, we'll roll the dice and, you know, do this Pearl Harbor thing and try to, you know, push out and establish our perimeter and hopefully we can hold it. And obviously that was terrible for them, terrible for a lot of American guys that had to go fight that war. And I think it seems like it would be kind of hard for the Chinese government leadership class, whatever, to look at being cut off from chips at the same time that everybody is saying like, this is the new oil. You know, it's the new big thing. It's the thing that's going to drive the next transformation and not feel like they're kind of in a similar spot where they say, we're trying to hold them down and this does seem like it is what we would be doing if we were trying to hold them down. How do you think they understand it, and do you think they're getting it wrong or are they maybe in that fundamental sense getting it right? Jake Sullivan: You know, it's interesting because the people who make that argument also tend to make the argument that the export controls don't work. It's an interesting overlap. And I see a pretty deep tension between those two propositions. Now, I think China to a considerable extent buys its own hype that the export controls are futile because it's going to make more and better chips and it will be just fine. And that to me disrupts the Japan analogy. Do they complain at great length about the export controls? Absolutely. Do they say that this is containment suppression and so forth? They do. Do I believe that they have decided they must go to war to deal with the export controls? No, I have not seen that. And I think the analogy has a certain elegance to it, but it requires someone to produce evidence that this is actually a targeted policy of saying the highest end chips and semiconductor manufacturing equipment that the United States designs, we're not going to put in the hands of our competitor to me is not the same thing as a full on oil embargo and is a sensible national security measure. And frankly, if you look at China, they're engaged in a whole series of their own national security measures, including various forms of export control, on things where they think they have some advantage. And I don't really see anybody saying for that list of things, which are pretty elemental, Hey, that's a reverse Japan US thing that's going to lead the United States to feel it has no choice but to attack China. So this is what competition looks like when power is increasingly measured and exercised in economic and technological terms. And China has cards to play and the US has cards to play. And we have to look at them and make determinations about how to play those cards. But I am a skeptic of the argument that we are inexorably pushing China to a position where it feels it has to lash out at us militarily because we have high-end chip controls. Nathan Labenz: Yeah, I mean, again, it seems like a timeline thing maybe. I would put myself in the camp of people who do believe that the export controls will be effective at least in the sense of like limiting what China can do in AI on some, you know, important dimensions, at least in the short term, you know, at least as as long as the sort of AI 2027 or Dario Amodei timelines go. If we find that like there's no AGI in 2030 and we're kind of headed into a longer term scenario then I'm not so sure because then, you know, I always say like I've seen them put up a hospital in eight days or whatever. So like I don't doubt that they could build a chip industry in five years, maybe not, but who can say? I guess how do you think about... It seems to me that there has been this evolution and I think this would be, in terms of like are we trying to keep them down and how should they understand it? You mentioned this term like small yard, high fence. But it seems like this has really evolved, right? Like the story that people are telling, maybe not you, but you could tell me, but the story that I'm hearing in defense of the export controls has moved from this like we want to deny them these like super cutting edge military applications to, well, maybe we can't do that, but we can at least like deny them the ability to build frontier AI models. And now it seems like we're like, well, we can't really do that either, but, and this is not an exaggeration, like the latest stuff that I've seen coming from like Lennart Heim who's been sort of almost synonymous with this export control analysis at the sort of think tank level, is that hopefully we can field 10 times as many AI workers as they can at least. You know, we can't deny them these other capabilities, but we'll just have way more AI agents running in our economy than they will. And this does seem to me like it has shifted from a sort of small yard, high fence to a like general denial of the broad-based economic benefits of AI, which again, to me feels like if I'm them, even at the rank and file like consumer level, I'm like, is the US trying to deny me my AI doctor? Like that doesn't feel good. At what point do we actually become like the bad guy in some sense or, you know, at what point are they right for understanding us to be like trying to hold them down? Jake Sullivan: Look, the way I approach this, maybe overly simplistic, reflecting the fact that I'm not a technical expert on chips or AI is, it's to me common sense. You have a high-end technology that has significant national security applications. Some of those applications are exquisite, like in a weapons system. Some of those applications are more general, like building super powerful AI that gives you a massive military and intelligence advantage over your adversaries. But either way, it is an input for building national security advantage. We have in China a serious competitor. Why are we giving them that input to be used against us or our allies? We shouldn't do that. That's how I look at it. Now, what impact does that end up having in the AI race? The way that I would put it is I think it gives America a distinct advantage at the frontier. And we should try to sustain that advantage because being ahead at the frontier from my perspective has a significant number of national security and strategic benefits. And so that was in my view, the original impetus behind the export controls: don't provide your deep competitor an input for its national security advantage to be used against you. But be discriminating in what you're denying, so that it's not things for gaming applications, broad-based kind of stuff. And it is definitely the case that generative AI and large language models had an impact then on how to think about that basic calculus of national security advantage. No doubt about it. And it made us think differently about what exactly is inside the yard. But the goal here fundamentally is still very much focused on trying to sustain and secure a national security advantage for the United States and not cede it over to China. That leaves this kind of dual use issue that you flag, which is a real one, right? Because these chips can be used for national security purposes, they can be used for scientific and benign economic and other purposes too. The challenge we have here is that in China's system, they have a doctrine of civil-military fusion. Their technology companies have deep ties to their military. And so it's not easy to say, well, we'll give the chips for this purpose, but not for that purpose. There's no way to actually enforce that in a meaningful way. And that is how we end up at the policy that we end up at. But are we trying to deny access to AI doctors in China? Absolutely not. And do I think the functional outcome of an export control policy will be the denial of AI doctors to Chinese people? I do not believe that it will. Nathan Labenz: So do you think Lennart is wrong when he says we can run 10 times as many agents? Because like, that seems to be the upshot of that, right? Jake Sullivan: Well, let's think about it this way. What's absent from that statement, and I don't have the context of that exact statement so it's hard for me to directly comment on it. But what's absent from that statement is what we started this conversation with. AGI, ASI, powerful AI, whatever you want to call it. Is that coming? Who's going to develop that capability first? To what end? I would like to see the United States win that race. Do I think denial of access to high-end compute to China is an advantage to the United States in that race? I do. Where do AI workers and data centers come into that? I don't know. What I will tell you is my logic on this is the profound national security implications of being ahead at the frontier for the development of military capabilities, the development of intelligence capabilities, and the development of economic statecraft tools -- not just broad-based economic growth, but coercive economic statecraft tools, among many other things, means I want to see the US there and if I've got an input that we have some control over, I'd prefer not to hand it over to my competitor so that they can get there first and use it against us. That's kind of the base baseline logic behind the controls. And in this respect, I think the Japan analogy is really kind of not that strong because what we're talking about ultimately with super powerful artificial intelligence is a kind of transformative national security technology that let's say we had it, leave aside the chips, let's just say we had that technology to be used for a particular national security purposes, would we hand it over to a competitor of ours? No, we keep it for ourselves. The way we do with every other type of technological military advantage. Now, China has caught up to us in a number of areas in the military, but we didn't go hand them the jet engine and hypersonic missiles and so forth. You know, we said that's on you. You got to figure that out for yourself. We're not going to help you in that regard. And I see a similar dynamic at play for the very high-end compute given how central to the future of national security the question of frontier AI would appear to be. Nathan Labenz: So, one statement that has really been a burr in my saddle for a while is Dario's proposal from his Machines Of Loving Grace essay, where he basically says, we should use these export controls to keep our lead, build an international alliance, which you kind of alluded to. But then I think he takes a step that you're potentially not ready to endorse, which is eventually we'll get around to basically making China an offer they can't refuse, or in his words, quote, convince them to quote, give up competing with democracies in order to receive all the benefits and not fight a superior foe. Which to me basically reads like regime change. Would you say that's going too far, like we shouldn't adopt that as our policy? How do you think about the fact that one of the three or four top AI company leaders is like pushing for that? Jake Sullivan: So I've been explicit that I don't believe that regime change in China should be an object of US policy. In general, whatever direction, we should shape our own policy. Our job should not be to shape China's government. We have to deal with the government as it is and try to design a strategy to ensure that America's security, economic and interests are protected and our values are protected. So I am not a regime change guy when it comes to China. There are others who are. I mean, if we went down the list of prominent technology leaders and their views on various geopolitical issues, we'd find a lot of different quite interesting things, many of which I agree with, many of which I disagree with. So I don't take a lot from, you know, Dario, I have huge respect and admiration for him as a person. I think he's an ethical person. I think he's passionate about democratic values. There's a way in which to me that is admirable. I have a different point of view than he does when it comes, it sounds like, to the policy question of regime change. Nathan Labenz: I guess one other sort of very big picture question on China is: how much hope or room do you have in your scenario or world building for the possibility that the regime there could change? And that could be like through an actual regime change or it could be sort of through evolution, perhaps, you know, fairly sudden in terms of like the way that their government is acting. Like we've seen these sort of dramatic reversals on COVID lockdowns, for example. And I was really struck recently by the release of the Kimi K2 model, which both is leading some creative writing benchmarks in English and also comes from this company that has this very Western loving aesthetic. You know, the company's called Moonshot, it's named after Dark Side of the Moon. And it just had me thinking like, geez, you know, maybe all this engagement that we've been doing for the last like few decades actually has worked on a much deeper level where like there's potential for real friendship, you know, in a serious way between the two countries. And maybe that just hasn't quite hit a certain level of leadership yet. But maybe it could. Like am I too fanciful for hoping that something like that could surprise us on the upside? Jake Sullivan: That is not my base case. In fact, what you've seen is a deeper consolidation of power under Xi Jinping, the centrality of national security in all decision making and their definition of national security being social control at home. And the deepening and strengthening of the Chinese Communist Party under the grip of Xi Jinping. That is what I think you have actually seen over the last few years. So my base case is that that's likely to endure. But, of course, I'm not absolutely not ruling out that your optimistic scenario could come to pass. It could. My main point is we should make policy that is not directed explicitly at a particular institutional or systemic outcome inside China. We should say, we'll watch and see what happens with the Chinese government and its relationship with the Chinese people. What we can do is set policy that is not about changing China from within, but rather about protecting and defending our interests and creating an environment that is more conducive and leans in the direction of American interests and values and does not end up allowing Chinese interests and values to dominate. So, could what you're saying be right? I can't rule it out. I don't think that the evidence really points in that direction. I'll say one last thing because you said, could we be friends? Look, I believe that at the people to people level, we should have a lot more exchange than we have today. American students in China, cultural exchanges going back and forth, all of it. And I believe there's no reason why even if we're competing intensively, we cannot deepen the connections at every level of our societies between our two countries. Look, we're two big ambitious dynamic nations with very proud peoples who share a lot in common and can benefit from engaging with one another. And I am all for that while remaining clear eyed about the nature of the government in Beijing. Nathan Labenz: How do you think the rest of the world is looking at this competition? We are trying to export chips obviously, and have some newly announced deals to do that to some at least countries with deep balance sheets to pay for them. China's open sourcing their models. At the AI event in Shanghai last week, no less than the premier of the country showed up and said, you know, we're committed to making AI an open source public good, you know, for the entire world to benefit from. It seems to me like if I were in countries 3 through 193, I would find that to be a pretty compelling vision. How do you think the soft power competition is going for us right now? Jake Sullivan: Well, I think of it in two respects. One, I do think that this is a soft power move by the Chinese that is meaningful. We want to give you really cheap or free high quality AI. Like people are going to like that. On the other hand, I think most countries are pretty unsentimental. And another element of soft power is innovation and technological prowess. And they're looking like who's actually got better stuff? And if it turns out it's basically equal and so forth, then, yeah, they're going to want the easier, cheaper, open one. But if, you know, they're looking and saying, hey, you know, the US is pretty impressive. A lot of countries are betting on the US to stay in the lead and even extend the lead in the AI race over time. And that's its own form of soft power. And I saw that when I was National Security Advisor and I continue to see that, countries basically saying, you know, we like the American tech stack because it's damn good. And, you know, we think the United States is going to stay at the cutting edge. And I think it's our job to make sure that that remains the case and if it does, I think the soft power piece of this will take care of itself. Nathan Labenz: As you think into the future broadly, can you envision any sort of stable equilibrium? I struggle to get to a point in my imagination where, you know, we have obviously the sort of MAD equilibrium for the nuclear threat. I sort of feel like we're on this track to create a new AI Sword of Damocles that will sort of hang over all of the future as we race to defend ourselves against one another. And yet, I still can't even really imagine a version of that that's super stable. You've probably, I'm sure have seen the superintelligence strategy document from Dan Hendrycks and co-authors that talks about MAIM. I applaud them for looking for some sort of stable equilibrium. It didn't ring true to me as like something that seemed like it would be super stable. Do you have any sketch of like a stable equilibrium in the presence of powerful AI between these great powers? Jake Sullivan: I do. I think it is a version of managed competition where we presume there's not going to be an end state where one side just wins and the other side loses, where we have to come to terms with the fact that we're going to live alongside one another as major powers, but where there is going to be intensive competing, jostling, a little bit of elbow throwing. But where that intensive competition, we have sufficient guard rails, doesn't spill over into conflict. I think that is manageable. That is basically the blueprint that we pursued over the back half of the Biden administration: compete like hell, but intensively manage that competition so it didn't tip over into conflict. And I think that can be sustained even in the face of increasingly powerful AI. I also believe that powerful AI presents opportunities for direct engagement between the US and China to manage risks that afflict both of us. And that is kind of a modern version of arms control. You know, on the one hand, we're building our capabilities, on the other hand, we're talking to each other about both arms control and non-proliferation. And we did that quite effectively with the Soviet Union over decades. And so would I use the phrase stable equilibrium, maybe not. Do I believe that we can avoid great power war without giving up on competing vigorously? I think we can do that. I think we can keep competing vigorously and avoid great power war. And it requires a stewardship and a deep, intensive diplomatic engagement between the two sides. And I think we've shown in this iteration that is possible, and if we continue to build the muscles for it, I think it can withstand even the advent of very powerful artificial intelligence. How's that for optimistic? Nathan Labenz: I love some optimism. Does it require us to sort of mutually swear off of the pursuit of strategic dominance though? Because it seems like if there's an expectation that is sort of mutually held that like each side wants to get into a position of strategic dominance, then we're in an arms race scenario, we can't really trust each other. This stuff is like much harder to verify, it would seem than anything in the nuclear, you know, you can see the silos from space, but like who knows what's going on in any given data center, right? It seems like as long as there's this notion that we're going to try to get strategic dominance and we might, and they also might if we don't, then I have a hard time imagining how that doesn't create some like really serious risks, which maybe we can manage, but like boy, we're creating a lot of problems to manage if that's the sort of shape of the competition we're going to be trying to manage. Jake Sullivan: My answer to you is going to sound a little evasive -- because frankly, all good diplomacy is at the end of the day a little bit based on evasion and ambiguity. But I think some degree of uncertainty in this phase is manageable and is better than explicit declarations one way or the other on concepts like strategic dominance. I think it's just too soon to tell exactly what this world is going to look like and the band of possibilities is too wide. So I think there's this expression about, you know, getting across the river by feeling your way across the stones and that's basically what I would say we have to do here rather than try to impose a framework today borrowed from deterrence theories related to nuclear weapons or from something else. We should have a few tent pole concepts. We would like to stay in the lead when it comes to the frontier of AI. We would like to ensure that no other country has capabilities that are so advanced beyond what we have that they can just put us at risk and there's nothing we can do about it. We need to keep the lines of communication open and make sure that there is not mistake, miscalculation, or inadvertent escalation. Let's do all that. And then I think these questions around enduring advantage, strategic dominance, they will sort themselves out as we come to understand exactly what it is we're dealing with here, which I don't think we understand right now. And so I don't think we should front run all of that with some decisive declarations about our policy and strategy. Nathan Labenz: Maybe the same answer, but is there any version of a grand bargain that you could imagine between the two countries now or in the near future that would sort of take us off the trajectory that we're on of of decoupling and intensifying competition and release some of that pressure? Jake Sullivan: I think President Trump is giving some thought to his version of what he will call a grand bargain. I'm not sure what exactly it will contain. I'm pretty skeptical of the grand bargain concept because I think at a fundamental level, both countries are going to keep competing. And as long as that competition is a feature of the relationship, it will need to be managed and it can't just be solved and put away. It is a chronic condition of the US-China relationship, one that I don't think is susceptible to the cure of some strategic condominium or G2 or grand bargain. And so I think we are better off in a world of managed competition than either we're going to go for defeating them or we're going to do a grand bargain and everything will be good. And I think that makes me a little boring, I confess or concede, but I think it's the right recipe. Nathan Labenz: All else equal, I think I prefer boring people in power. What have we learned about AI and the future of conflict from recent conflicts? My sense has been like maybe not that much yet because it still seems like we've got wires to the drones and like people are, you know, controlling them with PlayStation controllers and not yet, you know, handing that sort of stuff over to truly autonomous systems. But what would you say are the big lessons? Jake Sullivan: You're catching glimpses of the future without being able to fully see it in the present. One element of it is just scale and attributability, just a different form of fighting wars than we've seen in the past, and we're beginning to see how that plays out on the battlefield in Ukraine. This is where abundance, the notion of abundance is a core kind of construct of modern war comes in. Another is that you are beginning to see the incorporation of various AI capabilities into some of these drone platforms. There are still humans in the loop, but you could now glimpse a world in which you could have fully autonomous weapons and see actually how that could play out, and we're going to have to grapple with the lessons of that in American military doctrine very much. And then you got to think about this well beyond the war fighting phase, it's intelligence, it's logistics, it's command and control. And here I think we're beginning to see the incorporation of AI capabilities in that conflict. And that's something that I think we can draw some lessons from as well. My biggest concern fundamentally is that the United States military, the Pentagon, because of the bureaucracy, the Congress and the defense primes is not necessarily poised to adopt at scale and with speed AI capabilities and I think the PLA is better situated to do that. That makes me nervous and it should be a kick in the pants for all of us on a bipartisan basis. Nathan Labenz: Cool. This has been great. I really appreciate the time and thoughtful discussion. And I do genuinely mean it when I say I think all else equal, I'd rather have boring people in power. Anything else you want to leave people with? It could be what are the biggest questions you're watchi ng, anything else we didn't touch on that you want people to understand, or just leave it there? Jake Sullivan: You know, one thing we didn't cover as much, we touched on the economic and the job displacement stuff, but for me, how this impacts the lives of everyday people in terms of how they make a living, how they provide for their families, how they find meaning and purpose professionally, these are really profound questions that uh that I'm wrestling with and trying to get a better handle on in addition to everything else that that we've talked about today. And as usual, I find there are a lot of different answers from a lot of different people about where this is all headed in terms of impact on our economy. This is the first technology I can think of with such profound national security applications that the government really had very little to do with. That's essentially a private sector led and driven technology. And that's uncomfortable for a guy who thinks about government policy. And it's made more uncomfortable by this thing we've been talking about, how wide the band is of predictions about timing and scope of AI capabilities. So that's just a world we're going to have to inhabit and do our best and have a can-do spirit about this. rather than just stick our head in the sand. Nathan Labenz: Yeah, totally. For what it's worth, I think if the political class can get the international relations right, I'm very personally optimistic on the working classes’ ability to figure out what to do with the peace dividend and the, you know, all the extra free time that they might have. Jake Sullivan: All right, fair enough. Fair enough. Nathan Labenz: Thank you again. This has been great. Really enjoyed the conversation. Jake Sullivan. Thank you for being part of the Cognitive Revolution. Jake Sullivan: Take care. Thanks a lot.