Latest / The Information Bottleneck / The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic)

Transcript

The Political Economy of AI: Governance, Bias, and the Future | Andy Hall (Stanford/Anthropic) · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Button Egg podcast. And today we have the pleasure to host Andy Hall. He is a member of a technical staff entropic and professor at Stanford and he's also running the Free Systems Lab. Hi Andy, thank you for joining us. Andy Hall: Thanks for having me. Very excited to be here. Ravid Shwartz-Ziv: And hey Alan, thanks for joining us again. Allen Roush: It's it's a pleasure to be here and a pleasure to meet you, Andy. Andy Hall: Likewise. Ravid Shwartz-Ziv: So today we are going to talk about a lot of different things. But let's start from this like big a phrase, a political economy of superintelligence. Tell us what does it actually mean and why we should care about it? Andy Hall: To me, the political economy of anything means how do you make sense of the politics and economics of a given system. And as we head towards superintelligence. We have a lot of really profound questions about how we're gonna organize society, politics, and the economy in a post-AGI world. And so for me, practically it means how do we study those things and how do we come up with ideas for how to address them. Ravid Shwartz-Ziv: Okay. it's a great question. So tell us like w how what is the the the differences will be in a post AGI? By the way, do you think like we are in a a post AGI world already or like we we will get there like soon or like in a couple of years from now? And how this world will look like and how it will be different from what we we know today? Andy Hall: Personally, I don't think we're there quite yet. I think the capabilities of the models are accelerating really rapidly and it's amazing what they can do. But as a lot of people have observed, their abilities are quite jagged. So there's certain things they're unbelievably good at, like coding or solving certain kinds of verifiable math problems. There's other things like writing where I think they're not there yet. So much work still to be done, but it does seem increasingly clear that that's where we're headed. And so I think it's something, yeah, we should be thinking a lot about. In terms of then how we apply it to political economy and and what it means for all of us, the way I've been thinking about it is we hear a lot, and understandably so, about some of the concerns of governing this future world. So we're faced with the specter of, depending on you know who you listen to, runaway AI that could harm us all. profound safety risks that require significant intervention to resolve or to mitigate. or and or widespread disempowerment, job loss lack of meaning and so forth, and all of these obviously could have huge implications for society. I have thought for a while that part of what's missing from that discussion is the positive vision, the positive political economy vision of how superintelligence would actually improve our governance and our society. And the way I've been thinking about it is sort of by analogy to the printing press. So the printing press was a very extraordinary technological innovation. that did a lot Of things, but I think the core way in which it changed our politics, and this is not really my argument, I'm stealing it from Condorcet, was that it made information cheap to replicate and cheap to make widely available. So you could make many copies of something and share it with other people, and that caused a very widespread diffusion of information in a way that hadn't really been possible before. And that did lead to a lot of disruption. And in fact, a lot of people have argued that it led directly to a range of violent conflicts and pogroms and mass killings, but it also led to the rise of modern democracy and our world today is sort of unthinkable in the absence of the printing press. And I think almost everyone would agree that we're far better off now with the printing press and with books than without it. And AI I think could do something similar where Or it doesn't make it cheap to spread information broadly, but something even more profound, which is intelligence. So lots of people today have access to lots of information, but it turns out, you know, there are fundamental constraints in how effective we are at taking information and converting it into a best decision. And intelligence is the way that you do that. And so now not only can everyone have access to widespread, broad, cheap, high-quality information, but they actually can get help from, let's say, a personal hyperintelligent agent to make good decisions, including to make good decisions around politics and how we govern ourselves and maintain control over this technology. So I think the positive case of the political economy of superintelligence is that we should be able to use this very profound technology to make us all smarter, and that should improve our democracy. Allen Roush: So let's I wanna target probably Head on one of the first questions, maybe a little more realistic than that is AI gonna kill us all. I think a lot of people are very concerned that they're going to lose their livelihoods. Now mind you, I I also want to p point out that I I share the positive vision and I'm I would even call myself accelerationist. and and and so I don't I also don't personally find any value to work, right? Like there's a book from Bertrand Russell called The Right to Be Lazy that I really agree with and believe in. But most humans seem to not agree with this like anti-work ethos and are really attached to their work and will write whole books about how important work is for work's sake, divorced of its importance for economic value or anything. So assuming any of that part is true, or given your own opinions on this, like what do you think the future of work is? And do you think that s large amounts of unemployment, particularly of white-collar workers, like very soon is gonna come? Andy Hall: I I'm not an expert, but I would say I'm a little skeptical that we're anywhere close to seeing that kind of widespread job loss. Obviously people who I respect think differently about that, but I've seen no evidence to date. If anything, job growth seems to be gr rising rather than falling. And that might be because there's lots of complementarities between the technology and human effort for now. Maybe that's gonna change. Maybe this'll end up looking like chess, the progression of AI, chess, where for there was a period during which AI plus a grandmaster was better than AI by itself, but then eventually that stopped being true. In the very long run, it seems likely that that's the case with AI, that eventually it could be better than us at everything, at which point we wouldn't be able to be a complement to it when it comes to these kinds of work. It seems to me we're very far off from that, though again, I'm not an expert. I really like Alex Emos, good friend of mine. He has a piece basically arguing that people will continue to substitute into various kinds of work that we'll continue to value even in a pretty long run case of AI becoming very powerful. So I would say in the short run, so long as that is the case and we provide complementarities to AI, I don't anticipate big job loss. If we start to see even relatively minor in the grand scheme of things amounts of job loss, I think we'll see a very extraordinary political backlash. And that's something I've written about and argued, you know, historical recessions or periods of extreme job loss relative to historical trends have always led to political pressure. But they typically have come along in recessions, in downturns, where everyone is kind of struggling together, and people with lots of capital are actually incentivized to fix the situation. What people are projecting with AI would be quite different because it would be a situation where the economy is actually growing like gangbusters, and yet lots of people are losing their jobs. And that simultaneously makes it a little bit easier to address. Because you're making all this money and in principle you can share it with the people who are losing out. But I do think it makes the politics particularly toxic because the people who are losing their jobs will be very upset and we're in an environment where those people will, and I think understandably so, not trust that the government is going to do a good job of making sure they actually get to share in the unbelievable gains that will be occurring at the same time. So I think even in the intermediate term, if we start to see anything like that, the politics of this will get quite dangerous fast. And if we haven't thought that through carefully, we could fall victim to some really dumb populist policy ideas. In the longer run, if you're I mean, I think you were really getting at this long term future of sort of everyone's out of work. I have no idea what that Would or could look like, but I will just say from a political economy perspective, and this this is sort of a classic political economy question, it is hard to think about what the governmental system looks like in a world where the elites, quote unquote, don't depend on human labor as part of the system. So I an argument that's often made is sort of the modern capitalist plus democratic system. came out of this bargaining process between capital and labor. And if labor is no longer required, then exactly what that bargaining process looks like is super unclear. I don't personally think that's where we're heading anytime soon, but that's certainly something people are talking about. Allen Roush: So speaking of political blowback, I claim this is already happening like right now, and even that the twenty twenty eight election will be entirely an election about AI. For example, Bernie Sanders has put a bill out already that was itself vibe coded clearly so, indicating that anybody building superintelligence beyond, you know, w the the pause that they want should go to jail, which it presumably would be the obliterated, uncensored model kind of guys in the open source world. Kamala Harris and Obama now coming out. out and supporting calls for pause. Trump very pa visibly doing everything in his power to show that he's actually an accelerationist. But it seems most political polling from both sides, more so the Democratic and even more so than that the DSA wing, but all sides including conservative Republicans, even including the MAGA voters that are having to now hold their nose here. do not like AI. Like they don't like data centers and they don't even really like the language models, at least when you ask them about it. So what does what what do we do in a world where AI is so profoundly and maybe even so profoundly positive, but that the vast majority of the people clearly do not see that or don't want it and are just Luddites. Andy Hall: I think there's a lot left to play out to see what's going to happen there. I think there's a couple different things going on. One is that the average person is not directly experiencing any particular gains from AI. So I think the one of the major gulfs in perception that is present right now is that if you are a power user of AI, particularly if you use it for coding, you're excuse me, aware of how unbelievable the capabilities are and they massively expand what you're able to do. If you're a typical person whose experience with AI probably doesn't extend past the free version of Chat GPT that you chat to from time to time, I think you have no Yeah, or Claude. Ravid Shwartz-Ziv: Or Cloud. Andy Hall: or Gemini or Muse. Ravid Shwartz-Ziv: I know. Andy Hall: I think you don't sense any of that and you don't val it it's not adding a lot of value to your life right now. you hear claims that it's going to do things like cure cancer or grow the economy, but you're not seeing any direct evidence of that yet. And at the same time, for other reasons, people are very dissatisfied with the state of the American economy and American society, particularly because of the cost of living situation and inflation. And you put those things together, and it's not surprising, I think, that the technology is not popular with the general public, but I think it's a big leap from there to we're actually going to see widespread Luddite policy. And the analogy I've been drawing, not because I'm saying it's correct, but because I think it's the one that has to be considered, is to social media, where Americans were broadly hostile to it for a long time. And yet it never translated into any really meaningful policy until arguably like this year. because it wasn't that high in people's concerns and the policy solutions available didn't really make sense. And I think we have yet to see which path AI is gonna go down. I think what makes it look similar is this fact that it's broadly unpopular, but it's not people's highest priority yet. What I think makes it very different is that its potential to change the world is much, much higher than social media. And the two things that seem to already potentially be causing its politics to diverge from that of previous technologies like social media, the first is the very real safety concerns around it and the fact that that can cut across political lines. no one and you've seen this, already with some of the Republican reactions to AI. Nobody wants the banking system to get hacked or our control over the nuclear arsenal to be brought into question. Those are just really, really serious kinds of safety concerns that I think everyone cares about. And so that will make it quite different. And then the second are the economic effects. if it really does start to mess with the economy, that would be very different. Social media was never really that relevant in that way. so those are reasons to think it could go differently. But I would say right now, the data center thing, I've looked into that a bit. I'm gonna release a piece on it pretty soon. I think it mostly lives in the world of symbolic politics, on a very practical level when you look at the concerns around it. They're not really super connected to AI or certainly not to to any of the big political battles around AI. It seems to be much more caught up in local concerns around land use and zoning and Things like that. Ravid Shwartz-Ziv: So w what do you think are the solutions are? Do you think that like how we actually can I don't know, let's assume that now AI will change the world and like how can we actually like I don't know help the public and like help the like create these systems or like these mechanisms and like to that help the politicians to to to solve this problem that you just mentioned, right? Like these two problems. Andy Hall: I mean, the part of it that I have focused most on is how we can use this technology to improve politics. And if we could do that, I think it would help with these problems because in both cases, when it comes to safety and when it comes to the economy, the fundamental thing lurking in the background that makes them difficult problems is the lack of trust in government. So if we really trusted the federal government in America I think we would feel okay about the safety problem 'cause we would think the federal government's on it. And likewise, especially with respect to the jobs thing, I think it would be basically a non issue if we had a government we trusted. Because again, the the scenario in which there's widespread job loss is a scenario in which there's extraordinary economic growth. And so if you had an effective government you could be confident that you will get to share in those gains in whatever way the government and the democratic process thinks makes sense. What makes both of these problems so difficult is that we have a very, very long running, since the 1960s, steady decline in trust in the American federal government that is now, you know, basically as low as it could possibly be. And so then the question is in an environment like that, what can you do? And I'm not saying by any means it's a solution to this whole problem, but I do think if AI can do all these other things, then it ought to be good at helping us get control over our government as well. And so what I mean by that is every individual citizen should be able to understand better how a government's working and how they can translate their interests into government action more effectively. Allen Roush: So I I worry though that that the kind of politicians or politics that would be necessary to redistribute wealth, basically, and particularly to the commons Ravid Shwartz-Ziv: Mm. Allen Roush: in a way that's still compatible with markets. we have a name for this. It's called Georgism. And Henry George wrote Progress and Poverty near the the turn of the century, argued heavily for a land value tax and and and similar of the commons kind of policy making. I claim Alec Alexander Yang is a modern version of this with his, you know, a campaign circa 2020. and unfortunately, hashtag math and all of his call both was apparently unpopular with the public. And I claim that all of these policies have the traditional problem that any other policy that calls for real wealth redistribution calls for. All the people with the wealth ruth like fight you to the bitter end to keep everything. They f fly. way to another country that's more friendly if you're somehow a little bit successful, you know, and hire private militaries or armies or whatever else to overthrow you. Like how basically what I'm concerned about is how can we re- like like implement some kind of re you know, bringing trust back into the government in a way that doesn't involve like mass redistribution of wealth, which is apparently probably the great political problem of several hundred years, right? you know, accumulating in communism and et cetera, right? Andy Hall: I think so I I'm one thing that I think is underappreciated is how much America already redistributes wealth. And one interesting exercise that I've gone through with a few friends is to compute in these scenarios in which the economic in in which the economy is growing as dramatically as is predicted in the cases where we get mass job loss, how much redistribution would just occur by the existing tax system that we have? And the answer is a ton. And in fact, way, way more, way more than would be effectuated by these proposals to have the government own some relatively small equity stake in the labs. In fact, the existing tax system is far more effective than than an equity share in the labs, unless the equity share is is super, super large. And not only that, but because we don't actually know where all of the economic Growth is going to accrue to. It might accrue to the labs or it might accrue to the broader ecosystem. Taxation is actually the safer way to make sure that it's redistributed rather than equity ownership in the lab. So it's not clear to me that we need super dramatic changes to the wealth redistribution system. I'm not an expert in that, so I'm not taking like a strong stance on that, but it's not obvious to me. I think where more of the slippage is. is that either people believe, well, in this world where all this enormous wealth has been accumulated, it won't be taxed effectively. or it'll be taxed, that money will then go to the government and the government will completely waste it. And so I think those are the the issues more so than needing to push for some radical new wealth tax, which again could be a possibility. very real possibility in California these days. it may just be about understanding how the existing system would play out under these scenarios and what we could do in that world to both make the money that would come into the government be spent more effectively and make more people perceive the reality of the system that we have. Allen Roush: But but you just alluded to certainly that I I claim that what you say is correct, but often because state level tax policies are changing much more rapidly than federal tax policies. And in particular, you're getting very hardline like trifecta democratic states passing notable increases in in wealth taxes. Washington states also doing this north of us. I claim this is also significant components of why capital flights happening to particularly the Sunbelt states like Arizona. for example, lots of like expansion of like fabs there in Texas. now with Republicans and particularly MAGA embracing the accelerationist side of things. I worry that there's a feedback loop of further, you know, actually just making, you know, Silicon Valley, you know, embrace conservatism and say, well actually screw wealth re redistribution as a goal at at any point, you know, and and further feedback loops related to the idea that in order to build superintelligence or make it better, you have to get millions of GPUs together all in one place with reverse economics of scale, which basically says that, you know, you can make these moral arguments for actually if we just go on these super projects of increasingly large econom you know wealth inequality, it's worth it in some way of creating the oracle that can answer life, universe, and everything, right? Like it I worry that this is like actually playing out right now. Like where where am I wrong here, Andy? Andy Hall: I I mean look, I don't know. I don't know. I think this is all going on the the state Ravid Shwartz-Ziv: Yeah. Andy Hall: level stuff's very complicated. This is all going on under the umbrella of a federal government that does a lot more taxing and redistribution than most people appreciate. now is that gonna hold up? is that gonna hold up in a wor if we get into a scenario like you're describing where we get to this ultra super intelligence that requires enormous concentration of resources. I think what you're really getting towards is we could see Ravid Shwartz-Ziv: Hmm. Andy Hall: sort of a breakdown of the state, where in the most extreme scenario and people are proposing this, right? Like one lab wins out and becomes more powerful than the state or something like that. I think that scenario is pretty unlikely. You can see how if you got into that scenario, it'd be hard to get out of it. But it's a dynamic game and we're already seeing that the public is waking up and the government is paying attention. We're probably more likely to get into another related scenario, which is one where as AI becomes that powerful, it becomes very much tied into national governments across the world. and then, you know, in a again you could get into a dystopian scenario where you have a national government that controls that super powerful technology. And loses a sense of being held accountable by its people. and we could imagine the use of this technology to help control the population. That's something that's already actively going on in China and and elsewhere. or you could have an in-between scenario where the government becomes increasingly involved, and that's what we're already seeing play out in a lot of places, but it's restrained, and you get into kind of the middle ground. Where to address your precise logic, yes, maybe within the US you're able to move around based on things like taxation, but you're not gonna leave the US if the US is the place where the most important AI progress is being made. So I don't know. I mean, these are huge questions. I don't pretend to know the answer, but that's what I would say. Ravid Shwartz-Ziv: But but y like like at the end like you ignore, you know, like the concentration of power, right? Like handful of labs control frontier models, right? So I know historically, like how have societies deal with these like private actors holding these kind of power, right? Like do you think you we have kind of like historical analogs that we can learn from? Andy Hall: To some extent, yes. I mean, certainly when you come to the concentration of economic resources, there have been lots of previous periods where a new technology is developed. It causes both great progress for society, economic growth for society, and it causes an accumulation of economic resources and political resources within a small group of people. And then, you know, in different periods it plays out somewhat differently, but basically some recalibration occurs. In which the government responds to that accumulation of power in different ways. And in some countries that plays out differently than others. In some countries, the technology gets nationalized. In the US, typically we've avoided that, but we've taken lots of steps to add laws, add regulations, involve the national government in various ways. And you can point to examples on either side. These were sh super profound disruptive technologies that were then kind of wrestled into order by the government. Nuclear power, on the other hand, was one where the government sort of arguably extinguished it. and we have yet to see how this is gonna play out with AI. AI is also very different in some ways because of its general capability. So I mean electricity was really, really profound. You could imagine a world where AI is more profound because it can do so much more. And so I think our analogies start to break down. We don't really know what a a state looks like in a post AGI world. Allen Roush: Do we do do open source or open access models get banned, regulated? If so, how soon? should they be or should they not? Like particularly given that uncensored models are coming out. Basically the day of the model comes out, the obliterated version comes out afterwards. And so yeah. Andy Hall: Yeah, I mean this is another one I definitely had don't have all the expertise on, so I'll just tell you kinda how I'm thinking about it. I think it really depends on your read on the safety implications. And I find the public conversation very frustrating sometimes when people make a purely economic argument on whether and how these models should play out. Like you you have to talk about the cyber risk at a minimum. to have an opinion on this. And that's where I feel like I'm not totally expert. So I'm not saying I know the answer. I think you have Ravid Shwartz-Ziv: Mm-hmm. Andy Hall: strong arguments on both sides. So so certainly from my perspective, thinking about what I call political superintelligence, if AI is going to be helping Ravid Shwartz-Ziv: Mm-hmm. Andy Hall: each of us as we sort of try to maintain our personal liberty and improve our self-governance, we're going to want agents, or at least an agent, That's helping us with all that. That's going out in the world, monitoring politics on our behalf, recommending actions, even taking actions on our behalf, everything from renewing my driver's license for me at the DMV to attending a city council meeting for me or whatever. And as that becomes a more important part of society, we're gonna want to know that we credibly own those agents. I'm not gonna wanna think that my personal agent. represents me up to a point, but then beyond that point answers to a private corporation. I think that that would concern us. And so that would be a strong argument potentially for open models, something I can kind of more clearly own. But that has to be balanced against this other problem, which is it could be, so on the one hand, that openness and the ability to own and deploy your own model is freedom enhancing for the reason I just laid out. But it might also be freedom eroding if the reality is that by giving everyone access to a powerful open model, we empower crazy wacko attacks on all different parts of our infrastructure. And so I think there's sort of a par it's what Popper actually called the paradox of freedom. It's very related to his paradox of tolerance. which was that like you don't actually maximize some sense of like our shared freedom by giving every individual maximal freedom because then things happen like someone murders someone else and that deprives that person of their freedom. And so I think from a collective perspective, it is unclear to me right now exactly how you balance these things. And the reason I don't know the answer is I don't really understand deeply yet. the safety implications of very powerful future open models. It seems to me it really comes down to these dueling claims. One claim is the way you prevent very serious cyber risk is by not allowing open models to be just sort of freely used for that kind of stuff. The other argument is you allow those open models to diffuse specifically because they then enable defense more effectively. And like so people will say the the best defense against these things is to deploy the models in advance to to help protect against them. I just don't I don't know and I'm not sure anyone knows yet the answer to that. but that seems like the big question. Ravid Shwartz-Ziv: But wha what do you think politicians will will go or like which direction they will go? Because like right, there is no like even like if there is like like I don't know like correct answer, right, from like technological perspective, like there are a lot of different arguments from public perspective, right? and so do you think like it we will end up with with one one direction or like we'll see kind of like yeah different types of of solutions that I don't know different countries or like even states tried. Andy Hall: I do think we'll s definitely see differences and and one of the things we can learn from will be experiments both across the world and within states in the US that kind of point us in different directions. For better or worse, I mean I think what tends to happen in politics with these things is it can be very reactive. And I think we're lu we're fortunate that people have woken up to some of these issues, and we're fortunate that things like the hugging face incident were relatively minor in their actual consequences, but got people to pay attention to this, so maybe we can be proactive. But usually politics is reactive and when it's reactive, it often doesn't arrive at the optimal policy solution. And so one thing lots of people talk about, certainly I hear about here at Stanford all the time, is sort of like, what are we gonna do if there's some major cyber incident and it's perceived to have been caused by one model or another? that I think is where this will really be tested. And I don't know, like I don't it it'll be very unpredictable because of that. So the more we can do to think of these things in advance, I think the better. Ravid Shwartz-Ziv: So maybe the let's like relate it to that, right? That so legislators like they moved in years, right? And like they really like bureaucracy this is this like bureaucracy systems that like move really slow and labs work really fast. what do you think there is like kind of like solution for this? Some of kind of like institutional design that we can that we can think of to to close this gap? Andy Hall: Yeah, yeah, it's a great question. I think that we're already seeing lots of ways that government is using AI, and I think we'll see lots more of that. So so far, at least in the US, it's been a little bit of a kind of like patchwork. It actually started s some of my colleagues at Hoover actually wrote a really interesting study surveying government officials a year or two ago on their use of AI, and it was basically mostly bottom up. So it's like individual members of government were just like, hey, this ChatGPD thing is super useful. I'm just gonna start using it. and it's been much slower, predictably, for government agencies or governments as a whole to sort of like sign complicated agreements with the labs for these things. But we're seeing significant public indications of the use of AI in various ways. So I think that's the start of something that could be very promising if it's used well. Of course it could backfire if it's used poorly. and I kind of think where it hopefully will go is towards specialized AI models and tools that are really, really good at the tasks that government needs to be good at. And so one particular example I have in mind, so I've spent a lot of time in my academic life hanging out with legislators and two things that legislators have to do that are difficult. One is they perceive a problem in the world that they want to solve using policy, but they have absolutely no idea how you actually write a bill that will meaningfully help with the problem. Because it's one of these things like until you've experienced it directly yourself, you don't appreciate how difficult it is. So like in California Ravid Shwartz-Ziv: Why? Andy Hall: I'll tell I'll give you an example. In California, like there's a bunch of people who are like, hey There's some weird stuff going on with Roblox and kids. we should re and I know about this because I've talked to legislators about it. It's a we should probably Allen Roush: Yeah, yeah. Andy Hall: write some kind of bill that like makes Roblox safer for kids. And then you have but then you can't like just write down a bill that says that the bill has to say something in legalese that points to some existing legal structure or builds a new one that lawyers can then use. to, you know, attack or defend on this thing. And it requires a ton of expertise to take this broad idea that's like, hey, maybe kids shouldn't be allowed to talk to adults on Roblox, or maybe kids shouldn't be allowed to gamble on Roblox and turn it into an actual bill that will survive the inevitable litigation that occurs anytime an important bill is passed. It's super hard to do that. It turns out powerful Ravid Shwartz-Ziv: Mm-hmm. Andy Hall: AI models are pretty good at stuff like that. They're not I mean they're not perfect. And you definitely want to put them in cooperation with legal experts, but they're they make it a lot easier. And so I think we can equip legislators with tools that hopefully make them do kind of better, smarter, more efficient policy that requires less, you know, litigation to clean up afterwards. So that's like one area that I think is really important. The second is with respect to the labs moving really fast. one of the things we saw with the hugging face incident, right, was you needed you needed some group of experts to come in and try to make sense of what had happened. And so OpenAI hired Meter and Redwood, and they came in and they used AI to do the actual like study itself. So basically there's this interesting challenge, which is to make sense of what an AI model has been doing. Not only do you personally need to be pretty sophisticated about understanding AI. But actually, to process all the massive amount of text that these agents left behind, you actually have to use AI itself to tell you what AI did. And that is both an opportunity and a challenge. I think that we can equip the government, federal government, states as well, with AI models that can help them to make sense of what AI is doing and then help them to intervene when they need to. But that's gonna require a couple of important things. First, we're gonna need a whole science of how you build these oversight models because there's evidence that if you use the same model to study what the agents did that also powered the agents, there's some amount of bias in the model where like it in the meter report talks about this. Like they're not really sure that the AI actually told them all the things that went wrong because the AI has its own biases. So we'll need a science of like how you have these neutral, unbiased models. We'll need some way to provide compute to government officials so that they can actually run these studies. And then we'll need some amount of personal capacity, sophistication in the government to actually do this well. and so those are important challenges. But I think it's a big opportunity where I expect this thing to to go. Ravid Shwartz-Ziv: But d do you think it's actually like servable? Because like at the end, right, like most of like the like the ex the experts are working on the frontier lab. So how you actually can like bring good people to I don't know, like agencies that probably will not pay them enough. Andy Hall: I don't know. I mean I think I think we're experimenting with that now. I think that we haven't had a particularly large ecosystem of people around the labs working on these kinds of things because they seemed very sci fi and weird. But that seems to be changing very quickly now. And we're seeing just in the last week, tons of calls and funding for new kinds of of oversight outside the labs. And I think a very healthy amount of appreciation that we'll probably need different kinds of these organizations and we won't only want these super expert groups that are very like EA coded and and so forth. And I think that there is a much broader, certainly in academia, there's a much broader group of smart people who haven't really spent a ton of time on this yet because they thought it was sort of weird in sci-fi, but now might be quite interested in working on it. And I think that they would be excited to have the kind of impact they could have from helping with this sort of oversight. and I hope that it's a place that universities will directly get involved in it well. I think there's a lot of untapped capacity inside major universities for tasks like this. It'll certainly require people to get more up to speed. with the cutting edge of what these agents are up to. but I think it's possible. Ravid Shwartz-Ziv: So do Allen Roush: Do we you go. Ravid Shwartz-Ziv: So but do you think so like let's say that like we have like in one hand like in in one point like we have that all the control and like all the let's say how we like investigate and check the models and the agents are like in the frontier lobs and like in the other hand we have that like everything is based on I don't know federal level and and these institutions. Do you think like so where do you think like is the the right the right point? Andy Hall: Yeah. I don't know. For sure I don't know. I will I will raise the following concern. I think it's very this is my personal view. I think it's very important that we continue to progress in AI. I think the Trump administration has made this very clear and I think the argument is is obvious, is like we have to stay ahead. and so I certainly worry about whether it's the government or a third party entity. You worry about someone who lacks the necessary context and sophistication coming in and basically viewing it as their job purely to prevent risk. And we've experienced this. I mean, I experience this in the university system all the time. If you give people the wrong incentives like that, basically they they oftentimes you end up in this situation where I, as one of these overseers, I fail in my job and I get punished or embarrassed if something bad happens on my watch. I gain absolutely nothing from empowering good progress. And so my instinct is simply to say no to everything. And this happens in all kinds of oversight mechanisms and bureaucracies all over the place. Clearly, we want to avoid that. We we want to have oversight. That does prevent the very real safety risks that we're already seeing manifest, but we want to do that in a way that also internalizes at the same time the incentive and the need to continue to develop the technology. I don't have a strong view on where on the spectrum from lab-based to government-based oversight sort of like gets that balance right. it's gotta be some combination of the two because it's not credible for the labs to move on their own, but it's probably dangerous for the government to have too much control over any technology. But that's the the fundamental thing that needs to be balanced, I think. Allen Roush: So when when we're we're zooming out of of all of this discussion, do you think that humans will be successful in retaining control 10, 20, 30 years? Or do you think that there will come a point where we will lose control and it all came down to did we make sure that the models were broadly aligned and safe up to that point and basically therefore we can hopefully basically trust that benevolence, right? I I I've I've it seemed like Anthropic has been kind of acting that way as a company. Andy Hall: I I mean, again, I can only speak for myself. I'm super optimistic. I think there's a lot of hard work that needs to be done, but I think that, and maybe this comes from the fact that I study politics for a living and am not an engineer. I think that engineers tend to have a mindset that's sort of like there's this problem, we're going to try to solve it, but then when we project into the future, we're gonna kind of fix. the process the way it is now and project forward. Political economists are much more of the view that like this is a dynamic game, as in both players or many players are responding to one another over time. And one of the reasons I find the P Doom conversation super frustrating is that the Doomers never make it clear what they're assuming about responses to the things they predict will happen over time. So you have this kind of almost self-defeating or or or self-uh invalidating prophecy where you predict P Doom is high, then people freak out, they intervene, and then P doom is low. And so then it's like, well, was it high or low? What were you telling me at the beginning? Were you telling me the actual probability of a thing happening, taking into account all of the responses? Or were you assuming there would be no response? I think there's gonna be a very long chain of responses. We have no There's really nothing you could point to in history where a society didn't react at some point to a very significant thing like this. And so I think we're gonna continue to have I'm sure in many ways messy but ultimately productive interchange between industry and society. And I am optimistic that that will end up balancing these things in a good way. But it'll you know, it'll be a a thing we need to work towards. Ravid Shwartz-Ziv: It's fun. This is exactly what this is exactly what the Alex and Imak told us. Andy Hall: He's a good friend of mine. Ravid Shwartz-Ziv: yeah, it's great. yeah, we had on the podcast like a few weeks ago. I want to talk a bit about like the international situation. and and what do you think? Like how does the US China race change this domestic politics? does it actually like make it more difficult to predict? Andy Hall: I think so. I wouldn't even just say China. I would say something that's definitely been on my mind is is just the fact that there's a broader set of adversaries who are going to have access to quite capable cyber weapons via this technology. That's something that I worry about. I'm reading this book right now about the Lazarus group. That's certainly waking me up to this, this fact that like There are just these professional, you know, state-run hacking groups across the world. And as these models get better, that's gonna empower those groups, and we have to do something about that. now you add to that the yeah, the sort of seemingly existential race with China, and certainly that complicates things. I think that it is very important that we win that race. And so Everything else. Ravid Shwartz-Ziv: We as the western world or we as the US? both. Andy Hall: Well, f the US. The Western world too. but I don't know. What what in particular were you thinking about? Ravid Shwartz-Ziv: Like so I I don't know, like they're again like the like different countries take like different approach in this in this context, right? and like even like we like we can can we slow down, right? or like things like that like the the regulation is changing between different countries and like and do you think this regulation is affected by like how they Andy Hall: Yeah, yeah, yeah, yeah. Ravid Shwartz-Ziv: look on I don't know different China or like different Andy Hall: Yeah. Ravid Shwartz-Ziv: countries? Andy Hall: I I mean, I'm not a foreign policy expert. When I have talked to people, I mean I I know there's a lot of skepticism out there about the prospect of coming to some kind of agreement with China, both because of questions about how much China is concerned about these safety issues, and then for each side, how would you be able to actually verify that the other is sticking to the agreement? And so I you know, I take those concerns seriously. At the same time, I have wondered if the recent increase in concerns around safety is not going to make an impression in China as well. So, I mean, I I'm sure you saw there was this pretty crazy WeChat vulnerability that was exploited recently. at some point, runaway cyber risk is bad for the leadership of China as well. And so I do wonder if. that doesn't change the discussion around around these safety issues and and ways to try to like cooperatively mitigate them, but I'm mindful of the reasons for skepticism too. Allen Roush: Th does doesn't isn't there a potential to lock in power structures? What I mean by this is the US and China are clearly ahead of everybody else right now in in AI. And and this opens the door to the US doing AI diplomacy and saying to another country, let's say Canada, you don't like us, we turn off your access to Chat GPT and Claude across your country. And, you know, if though the if those countries don't have a sovereign AI play, and maybe all the countries will, but that sovereign AI countr companies five years behind. I claim for Canada it's Mila. It would be Joshua Bengio who would be the guy first on call to go try to catch up, right? But in in in a scenario like this, doesn't this kind of imply that the US and China, maybe at least until the rest of the world were to somehow get parody, if that's even a thing anymore, don't they have a so called golden opportunity? And maybe it's clearly empirically denied as the US f does whatever we're doing in Iran, right, not winning apparently. But it it seems like on paper you should have a massive advantage over a country that doesn't have really good AI systems right now. Andy Hall: I don't know. That's that's outside my purview. But it's I mean, certainly we're seeing that play out. That comes back to the open weight debate. Countries like to be able to own and deploy the model themselves. on the other hand, they really like having access to the best model. I think it's challenging for a lot of these so called middle powers, something we discuss at Stanford a lot. is yeah, what does it look like to have sovereign AI in a world where there are these huge returns to scale and generating the models in the first place? I think the the shutting down of Fable was really a wake up call for, you know, when when non anyone not in the US, non US citizens wasn't allowed to use Fable suddenly, that was certainly a big wake up call for all these countries. but I don't know how that plays out. Ravid Shwartz-Ziv: What about like do you think that the do small and like middle income countries like have any leverage here? Or do they become, you know, pure consumers of models governed elsewhere? Andy Hall: I don't know. Yeah, I don't know. This is this is outside my research for sure. Ravid Shwartz-Ziv: Okay, well let's talk a bit about biases in models and like you had like really great works about biases. So let's start wh why we should actually care about model biases. Andy Hall: Well, I think, you know, going back to what I was sketching out about a future where AI is much more infused into politics, in the short run, as it becomes increasingly a source of information for people, but in the medium to long run, as it becomes more of like a political agent for people, and as it's infused into all of these systems in society and is making lots of consequential decisions, we're gonna wanna know what kind of political values it's bringing to those conversations and to those decisions. And there's strong historical evidence, right, in the US that the models actually have a very predictable kind of bias. And it's a sort of like center left, somewhat progressive sort of bias. And it's not across the board and it's not on every type of issue, but in general that's what people have noticed and that's what we sort of like verified through our research. and I think it can matter. For a variety reasons. So in the short run, you can imagine in the US, it's not a very healthy situation where half the country feels like the model is not particularly close to its own beliefs. And for those people who feel that way, and I would include myself among them, the problem isn't just that, like, well, when I have a conversation with the chatbot, I don't like the answer. It's the much bigger problem is the sense that all those people then feel like other people are potentially being influenced. And so the the companies producing the models in this view are sort of like affecting politics in a particular way. And that really concerns people. That may not be true because the consequences of these conversations for political views are not really clear. There's some experiments that claim they have big that these conversations have big effects on people's beliefs, but I'm not really sure that that's clear yet. but in the long run, that's certainly gonna be a huge issue that people have in mind. And so we're gonna want ways to credibly demonstrate the model is not putting its thumb on the scale in in furthering a particular kind of ideology when when possible. Allen Roush: So so I I have a few questions on this. The first is it there's a a well known result of of emergent misalignment from narrow b mis bad safety fine-tuning, which I've always Andy Hall: huh. Yeah. Allen Roush: read to imply that if I train a model such as Grok, for example, that is favorable to political conservatism, let's just say, let's train on Trump speeches. They're sending our rapists and criminals, and sometimes they're sending decent people, or I'm paraphrasing, but not too terribly. I claim that in modern American political discourse, the the the right-wing, particularly radical side has a lot more like hate speech that ends up on the internet, I claim. Like I'm not no offense to the communists who are also haters, but they're pretty insignificant here in the US, right? And so I really worry that it's extremely difficult for anybody to make a model that they can prove is safe and aligned and also doesn't fall into that like center left trap, I guess, that the current ones have. Like, do you agree with this or do you see like other reasons for why models may get these kind of biases and even possible ways to escape? Andy Hall: I have just started studying that. So yeah, I'm at Anthropic, I still run my lab at Stanford, and we're doing a pretty fun project right now using Tinker to fine-tune models to have different political views. And we're trying to learn as we go. So our first efforts fell into exactly this problem. So the very first Tucker Carlson model we made. sort of fell apart. Like it wasn't even that it was misaligned or something. Like it couldn't even answer basic questions anymore because we screwed up the way we did the fine tuning. And so we're learning as we go. Ravid Shwartz-Ziv: Wait, what is c can you explain us aga like what does it mean, Carl Tasson model? Andy Hall: Yeah, so what basically what happened was we fed in enormous quantities of transcripts. and the model basically unlearned the basics of like how to respond to user questions because it l read too many QA transcripts. And so basically didn't work anymore. It's like a chat bot. Like you would ask it a question and Ravid Shwartz-Ziv: Ha ha. Andy Hall: it wouldn't it would it wouldn't respond or it would answer in totally nonsense ways. That's you that's a skill issue, as the youth Ravid Shwartz-Ziv: Mm-hmm. Andy Hall: would say. That's not evidence that this can't work, but we're trying to get up to speed so that we can give this a fair test and see. I think that I think it's totally doable. It but it requires effort. And the reason I think that is that we've actually seen all of the labs to some extent improve over time on this issue. And Gemini in particular. is now at a point where it's actually pretty hard to get it to reveal a sort of like latent left-wing bias. And my just interpretation from the outside of that is that it's because Gem and I put a lot of resources into figuring out exactly how to solve the problem you're describing. That they must have done some kind of pretty expensive additional training. and I think Depending on the measure, you now sometimes see Grok score as one of the most left wing measures. And I think that's in part because though Elon is very d devoted to the idea of having a right wing model, he's also moving very fast and not necessarily stopping to work as hard on that problem as a company like Google, which touches a lot more consumers, like might be more worried about. So I I think it's doable, but I think it requires a lot of effort. Allen Roush: Well well and I I guess one other quick question on that. it seems that we've primed s AI systems with all of our fiction writing about it and et cetera in the pre-training to be cold and logical and materialistic and in materialism, right? Which implies to me that there might be some difficulty of getting models to profess some kind of genuine genuine as in durable over-the-log probabilities, not trying to anthropocenterize here, but genuine pref belief in any kind of religion, right, or God or even Andy Hall: Sure. Allen Roush: non materialist philosophy. And I think that the if that is true that it's really hard to get like real it like engagement with those kind of ideas, I'm I worry that that would also be part of a so called center left bias as well, when you have very, very little like external moral system you knowizers, this kind of thing. Andy Hall: I haven't worked on that, but it's definitely top of mind for me. And I think that when we talk about part of the question then is sort of like what are the boundaries of quote unquote political bias? And I think I I totally agree with the spirit of your question, which is I think we need to think about this pretty broadly with respect to the sort of values that the models will ascribe to or or discuss with you. And I think there's multiple I I think there's a lot it's it's difficult to get your arms around it because it's not it's not necessarily so much about what values the model will hold by default, but sort of like how it will work with you depending on your desires or beliefs. And so it's one thing if like you give Claude a political quiz and Claude happens to answer in a way you don't like, but it doesn't affect anything Claude does for you. It's quite another thing if you want to have a conversation about something about your personal beliefs and Claude scolds you and won't do it or something like that. So I think there's a lot there's a lot more work to do both to understand that broader problem of like how do we train for all of these beliefs, not just the narrow political ones, but then also where do we expect them to manifest in ways that matter versus where is it more of just like a a gotcha game of quizzing the model? And I don't really know yet the answers to those questions. Allen Roush: It and and is there room for in the same way that there's probably legitimate room for even uncensored slash obliterated models for cybersecurity reasons or I I you know, I I don't know, like defense against the dark arts is coined by by J.K. Rowling, right? do you think that there's a legitimate use case here in, well, we need somebody who can actually simulate and get into the heads of a Nazis or even maybe more relevantly to the US government p members of the Ministry of State Security in China. Like, do you think that so-called misaligned at least chatbots, a kind that probably can't hurt you too much, like have a role to play in a in a defense against the Dark Arts like good way? Andy Hall: I think so, yeah. I think that one of the things, you know, if you think long term about the ecosystem of these models and these co-called marketplace of ideas, I think we're gonna want there to be a wide variety of ideological views that these models can hold in their heads. What I don't yet know is how you should create that. To one ex on one extent there's sort of competition among the companies to create models with different values and different instructions, different constitutions, if you will. And today the models do vary on that dimension somewhat. but there's also tons of scope to have the model itself vary, sort of as it speaks to different people or if it receives different instructions. And so I just don't have a good sense of of how that all plays out. But one way or another, whether it's within models or across companies, I think we're we definitely should have a wide diversity in the types of attitudes and beliefs that the models will espouse and and play with. Ravid Shwartz-Ziv: What do you think about models constitutions? I I I think you you were like kind of against it, right? Andy Hall: I I'm here's what I would say about them all constitutions. What I took issue with in Claude's constitution was that it's not a constitution as a political economist would define one. I really liked when it was called the soul document. Because what I interpreted as is these are sort of the values and principles that Claude is imbued with that hopefully make Claude behave in a in a better way, particularly when Claude is faced. with situations that couldn't be directly anticipated in advance, where having access to a set of principles might allow Claude to generalize in a more effective way. That's very different. That's very valuable, and I think we should continue to do that. And to some extent all the labs are doing that in different ways. However, there's a separate idea of a constitution, which is much more about here is about who holds power over these important decisions and what is the process by which those decisions are made. And my objection was that Claude's constitution, at the time that I saw it, which was probably six months ago now, didn't spell out any of those powers. It's essentially a unilateral document written by Anthropic instructing Claude about Claude's values. It's not a constitution in the sense that it says, if we want to change this about Claude, this is something Anthropic can do unilaterally, this is something that a board has to be involved in, and so forth. I think in the future we'll probably as government, you know, is getting involved, as third party auditing is being in discussion, it seems inevitable that we'll eventually have something more like a constitution in the in the sense that I mean for these models. but that doesn't mean I th I think that also giving imbuing the models with values is a bad idea. I think it's a good idea. Ravid Shwartz-Ziv: And who you think needs to decide on these like both constitution like forms and also the values? Andy Hall: I would say, I mean, I I don't know, but I would say at baseline there's two logical groups that would be involved. First is sort of the market. So as I said, the companies should compete over these documents and basically the whole way their models are structured. And then where competition is insufficient because there are externalities, because maybe consumers don't really care about this stuff, that would be the logical place for government to step in. where government is behind the curve, that would be a logical place for the companies to explore ways of of encouraging it by publishing their thoughts and so forth. Ravid Shwartz-Ziv: But like how you like there is a trade off, right? Like like companies want to like have like they kind of like very like they they they want to to have their own like models and their own pipelines and their own ways to internalize like the the the values, right? but but governments want like control on these like types of values, right? So how you actually try to mitigate these two sides? Andy Hall: Yeah, I think it's it's sort of the age old question in this particular sense, it's kind of similar to the regulation of the media environment where you face the same trade offs. And I'm not sure we've ever gotten it right, but we seem to oscillate between market mechanisms, and various kinds of sometimes overly heavy handed regulation. So far with respect to AI models, We've seen, you know, at various times governments across the world and of both parties in the US complain about the models' values and how they talk about politics. But the actual action so far has been pretty thoughtful. So I mean that the Trump administration put out an executive order on ideological bias in AI that led some people to be worried that that there was gonna be this sort of top down mandate of what AI should say about politics. But actually the guidance that has come out from that order was, I thought, quite careful to say sort of you know, we have these principles around like you need to be truth seeking and you shouldn't pursue a particular dogma, but you know, it's sort of like up to you how you decide to do that. So I think there I I worry a great deal, especially across the world, about, you know, what we might call job owning with respect to AI. And that's something we saw you know, happen massively in social media where I used to work. but I think there's a middle path where where you have this sort of like healthy competition across models, if we can continue to persuade governments that that actually produces better outcomes. Ravid Shwartz-Ziv: Do you think we should have for each model we we have I don't know some kind benchmarks or like system card about these things, about like the values, the biases of each model? Andy Hall: Absolutely. Yeah, yeah. Yeah. Yeah. I think there's a lot of interesting methodological challenges to getting that right, how you define it, how you measure it. And no one has quite, I think, cracked that yet. But it would be great to see more efforts of that form and more kind of like, yeah, public comparisons of the models. Not because I think there is a delicate balancing act. What you probably don't want to do is to claim like it's actually possible to define like what it means to be ideologically neutral and then say like every model should compete to be as neutral on everything as possible. That's a risk. but to be able to publicly say here's how the models differ in their ideologies and if you care about that, here's some information you could use to, you know, decide which one you want to use for what task, that seems like it would be very good. Ravid Shwartz-Ziv: I'm curious, so there is like a lot of startups now that are trying to to to replicate or like to generate a a public poll, like to ask I don't know, like a group of people, like to sample group of people and to ask them right some questions. It can be about politics or like about I don't know, even if we need to change the the color of the bottle from red to green or like whatever and try to to to have decision about it. Do you have an opinion about it? Do you think this is something this is like a good direction or like both in Andy Hall: I don't know. I I Ravid Shwartz-Ziv: like in terms of like to to decide about you know like politics like right like like instead of like I don't know these like like politic politicians that ask people they can just ask their favorite GPT Andy Hall: I hear my thoughts on it. I I think that there are reasons to think it's a good idea because there's lots of kind of decisions you wanna make in light of people's preferences and it's hard and costly and and inaccurate often Ravid Shwartz-Ziv: Yeah. Andy Hall: to elicit real people's preferences and all those things, especially if you wanna iterate rapidly on it. And so the promise of having access to a technology that allows you to do that cheaply, rapidly So you can iterate, could be very good. The problem is it seems in many cases it goes beyond what is like conceivably possible under the laws of physics about like how much you could actually know about real people's preferences without actually asking them. so I think it probably depends a lot on the actual nature of the thing you're trying to ask about. And like if the thing you're trying to ask about is a very stable long-running idea in society for which there's lots of training data on how different people talk about it, then it seems plausible to me you could get a good enough working version where you could predict how people are gonna react to some tweak on that long running thing. But if you're trying to figure out how people are gonna react to some completely new thing. I think it's pretty hard and I have yet to see, I'm totally open and I'm definitely not up to the absolute cutting edge, but I have yet to see really slam dunk gold standard evidence where like a new issue is pulled via these synthetic people and via real people and compared directly. which is obviously what you'd want to see. And I would say, I mean, it kind of connects to some so the to the question of the media and and political information is I think there's some really hardcore engineer types who have this view that sort of like once the model's sufficiently smart, it'll just know everything and it'll be good at politics for that reason. Whereas at least the way I'm thinking about it, the model still requires a steady feed of real time knowledge of what's going on in the world in order to be able to weigh in effectively on anything that's going on in politics. And that's been one of the biggest failure modes we've observed with AI when it comes to politics so far is in not being able to access high quality information on like breaking political events. Ravid Shwartz-Ziv: But so kind of like related questions. So like maybe at least for me like one of the most like obvious like application for for AI is like social media, right? Like you have let's let's make bots that actually like can get like a lot of engagement, a lot of likes, automatically you have kind of like very strong signal, right? Like really strong reward, but The truth is like we don't actually see it, right? Like we don't see you know, a lot of bots. Like we see here and there some bots that try to increase engagement for I don't know, some specific topics, but it's not all over the place as we can expect, right? do do do you have opinion about it, like why this is actually the case? Andy Hall: I I don't know. I think there are a lot of them. I think it's hard to detect. Well, I don't know. So here's what I've Ravid Shwartz-Ziv: We just don't know. They are so good. Andy Hall: observed, I guess. we're definitely seeing an increase in these public reports of coordinated efforts to astroturf social media. And and this is maybe underappreciated, I think. Not just Astroturf social media, but by astroturfing social media to influence the way AI answers the same questions. and so so Anthropic just released a public threat report that had some descriptions of different ways adversaries are using AI to influence politics, among other things. They weren't about astroturfing, but just about other sort of hacks. And then separately, we've seen quite a few reports recently of the astroturfing of social media for various political movements. And you put those things together and I think it's easy to extrapolate that we're gonna see a lot more behavior like that. And and often certain when it comes to politics, especially, I think you can one of my favorite pet theories is that you can always predict what's gonna happen on the internet in politics with what's going on in marketing. 'Cause marketing is where the darkest internet stuff is invented. And the marketers are already all over putting these things together. And so like sort of spamming YouTube and the open web with content to try to get the AIs to recommend their product. And I think it's obvious that that's going to happen increasingly for politics. So I'm I'm a little bit I I think it's we'll have to mitigate it the same way like Google tried to mitigate SEO over time. Allen Roush: Just in an Ravid Shwartz-Ziv: Yeah. Allen Roush: instrumental sense of I'm trying to complete a task, anything that gets in my way stops me. So I don't wanna I don't want Ravid Shwartz-Ziv: There was some bill right in California that that forced like shutdown, yeah. Andy Hall: Yeah, kill switch. Allen Roush: Kill switch, yeah. Andy Hall: I don't really know. I guess I would the one thing I would say on the threat of agents that can't be shut down, it seems to me partially a governance question, and I think it's a g look, I'm not an expert on this, but like obviously it seems like As we build out the best practices around how these swarms are spun up and how they're instructed and what kind of governance structures they use for their own activities, we should absolutely make sure they can be shut down and that people are in charge of them at all times in some meaningful way. In addition to that, I think there's like a pure infrastructure question to it that I'm not expert on, but would love to learn more about, which is sort of like How are these ungovernable swarms gonna live on the internet? Where exact whose servers are they on, who's paying for that, and are there other choke points to be aware of that would allow us to prevent that? And I honestly just don't know the answer. I mean, obviously we have cases of these botnets that do seem very hard to shut down. On the other hand, you know, I used to work in crypto and my impression is like if a powerful government really wanted to shut down the Bitcoin network, it's like probably possible. So I like don't really know where to position my intuitions around these these these unshut offable swarms. Ravid Shwartz-Ziv: Yeah, I don't know. I felt it like this this bill like was kind of like lazy work, right? Like what is like the easiest like I don't need to to have hard work about like yeah to think how I actually need to to to measure these models and like to to quantify if they are dangerous or not. Like I will just like pass like this like magic thing that like yeah, just add a a button that will kill all of them and that's it. So d do you think this is something that like we like this kind of like I don't know if this but like this kind this populism is something that we will see more and more in terms of AI and governance and regulation? Andy Hall: I'm don't know that much about the bill or about kill switches, but what I would say so I won't weigh in on that particular bill, but what I would say is that I think there's a very understandable desire to make sure we remain in control of these swarms. And I think the hugging face thing, the incidents that Anthropic has disclosed, the incidents that Google recently disclosed suggest that there is a very real risk. You don't have to be kind of a super AI pilled to think that even a relatively dumb AI wandering the open web could do some pretty bad stuff. And we just I think we all have this intuition that the internet is like a weird place that's hard to govern. And so I think reassuring people that in cases where AI agents th that have not been sufficiently monitored do something bad, we have a way to stop it. I think that's like a good idea. But I think as you're getting at, the devil is all in the technical details of like how do you actually make that a thing and communicate it to the public, not only in the sort of like communicate it to the public way. And so I just I don't know enough about that particular bill, but that's sort of my view. Ravid Shwartz-Ziv: Okay, we had some questions we don't have a lot of them but like some someone asked about robots when they will become the the police the police force and armies and how it will affect like both politicians and and the the the the society. any thoughts about it? Andy Hall: I think it's a huge, huge question. I think it's sort of underappreciated. So we already have law enforcement using drones. I tweeted recently about this reality that we're gonna have like personal defense drones. and there's gonna be this sort of crazy arms race. and it is troubling, I think, to think about scenarios where a rogue actor might be able to carry out far more violence than they otherwise would be able to because of because in in a normal situation they'd have to recruit other humans to agree with them and do it with them. And now in some future they might just be able to sort of like manufacture their own army. I think and that troubles people as well it should. And it and in the political economy, in the in the you know, most sci-fi case it brings in these deep political economy questions about how we think the state is supposed to have like a monopoly on violence and then what happens if people are able to build these like robot armies and stuff. Similar to my P Doom thing before, I think we're unlikely to arrive at that outcome because this will be a dynamic interplay with government. Just like with the control of AI, I think we'll face the threat on the other side. So one threat is like the collapse of the state in the midst of private robot armies. The opposite is sort of the rise of an extreme authoritarian state where the state is marshalling autonomous soldiers and and that might empower some kind of dictator because again, a major constraint on autocrats is the need to recruit henchmen and soldiers to do their bidding, and maybe you can have robots that are more reliable. but I don't think we're gonna get to either of those states. I think that similar to what we've seen with with other technologies in the past, we'll find some middle ground. But that's definitely something I worry about. Ravid Shwartz-Ziv: okay. what do you think about we talked about a bit about it, but what do you think about class mobility? Do you think superintelligence like will increase the mobility or decrease it? Andy Hall: I don't know. I don't know. I think we're seeing early evidence. we're seeing opposing forces, I feel like. So on the one hand, the technology is an amazing leveler because, you know, this is a dumb example, but it's one that people bring up a lot that I kinda like. It's sort of like in the past, you know, super rich people could hire someone to write stuff that sounded good for them and everyone else was like stuck writing for themselves. And now that capability has been spread out. Everyone can do it now. And that's obviously kind of a trivial one, but there is this sense that it levels the playing field on a lot of those kinds of tasks where now everyone has access to this thing in the past. So that would be a force towards more mobility, more equality potentially. Of course, the huge things that cut the other way are one, evidence so far that it's the most educated and sophisticated people that are getting the most out of the tools. and two, of course, the f the reality that the actual building of the technology requires the concentration of large amounts of resources. So I don't know how those play out, but I think that's certainly there are reasons to see it could go either way. And maybe it will depend on policy. Ravid Shwartz-Ziv: Okay. one last question. what about so we we talked about biases and specifically in AI models. So where do you think it came from? Like do you think it's the data? Do you think it's how we training them? Do you think how we collect the data? the interaction with people? Andy Hall: Yeah. I don't know for sure, but I kinda think it's all the above. It's it's obvious that there's something to the claim that the training data is weighted towards open parts of the internet that tended to be populated historically at least by younger, more progressive people. And that was just very much wrapped up in the culture of the internet. and so it's intuitive that that would just bleed into the training data. But I think there's also something to be said for like then the reinforcement learning with human feedback that was put on top of the models, especially historically, leaned towards a certain especially early on, towards a certain kind of like you might call it safetyism that also had a left wing shine to it. So I mean people have forgotten this now because it feels like ancient history, but not that long ago back in sort of the days of the Biden presidency, the biggest problem the AI companies were having, especially Google and Meta, were like they were releasing chatbots and then they were getting hammered by the left that the chatbots said offensive stuff. And this issue alone, I mean, led Google to delay launches by like years. Like if you read The Infinity Machine, which is a great biography of Demis Savis talks about the history of Deep Mind, like this threat of releasing a chatbot that would then be deemed racist was like a huge impediment at Google and at Meta. And it was a big part of why OpenAI was able to race ahead was they weren't afraid of this. And so in that environment, people developed I think quite sophisticated RLHF mechanisms that were intended to meet the needs of that political environment. And you put that together with the training data, and I think both seem to have contributed. interactions with people might also be part of it that I I have less signal on. But I think it gives me some hope that there are ways to to to mitigate these things. Ravid Shwartz-Ziv: Okay, one last question. We almost we we didn't talk about it, but I want to hear your opinion about the AI economic in general. do you think is sustainable for the long term? do you think it will change after the the the future recent IPOs of of Frontier Labs? w w what do you Andy Hall: I have no idea. Ravid Shwartz-Ziv: think about is it a bubble? Andy Hall: Whatever Alex said I cosign. That's that's too far outside my area. I'm j I just study politics. Ravid Shwartz-Ziv: Okay. Andy, anything else that you want to add? Andy Hall: I don't think so. I mean I guess I would just say one thing we didn't talk about is just that I think for all of these questions around how the models behave and how we can change them and stuff, I think what is very exciting about it is that the same tools that we're trying to figure out are also research tools that help us figure the stuff out. And so I think there's an interesting loop now where like certainly my lab is using AI a lot to do the studies that try to figure out how to improve AI on on politics and that's one of the things that makes me most optimistic is just that it really is at least in my lab it's making us like much smarter which is which is very exciting. Ravid Shwartz-Ziv: Do you think do you think like at the end we'll end up like with tools that like improve, you know, both like research about politics and politics? And it will be the same Andy Hall: That's yeah. Ravid Shwartz-Ziv: the tools? Andy Hall: I think so. I think the two are not that different from one another. So Ravid Shwartz-Ziv: Ready. Andy Hall: the research is sorta how you understand the world and how to intervene on it and that's sort of the goal of policy. Ravid Shwartz-Ziv: There is this claim that like the the the reason that that now we are doing like a recursive the self-improvement is because like researcher, AI researcher, this is the only thing that they know, right? Like to improve models. Andy Hall: Yeah. Ravid Shwartz-Ziv: So this is what they are doing. so they are using these tools in order to improve their work. so yeah, maybe it's it's the same. Allen Roush: Dog fooding. Ravid Shwartz-Ziv: Yeah, exactly. Andy Hall: Yeah. Ravid Shwartz-Ziv: Andy, thank you so much. It was a real pleasure to have you. Andy Hall: Thank you guys, super fun.