Latest / The Information Bottleneck / Stella Biderman (EleutherAI) - Open Source, AI Safety, and Who We Can Trust
Transcript
- Allen Roush: Hi everyone. welcome back to the Information Bottleneck podcast. today we have a very special guest coming from Eleuther AI, one of the most impactful companies and organizations in all of open source, open access AI research. we have the executive director, Stella Biederman, here as our guest today. Welcome, Stella. Stella Biderman (she/her): Thanks. It's it's really great to be here. for for the record, we're we're a non profit, which is something I I try to brag about given how vanishingly rare that is. Allen Roush: And and and a real nonprofit who actually pretty much everything you release is free and open. Unlike some Stella Biderman (she/her): Yeah. Allen Roush: who put open in their name of their companies. Stella Biderman (she/her): We we put a very high standard on what we demand of ourselves in terms of like independence from the AI industry and financial incentives, not necessarily 'cause I think it's morally ne not because I think it's like abstractly necessary, but I think that kind of given the political and financial dynamics within the AI world, it is like pretty essential that there exist some people who are experts in training large scale AI systems that are as independent from those kinds of influences as possible. And I mean I I I I see this regularly when I talk to funders, like one when we first started raising money from foundations and like the large philanthropic foundation started to get interested in in funding like nonprofit AI research like I met with, for example, in Midiar who ultimately gave us a a significant amount of funding for several years, one of the first questions that they asked us was like, How do how do we know you're not gonna pull an open AI? And that was like literally the words that they used. and the fact that that was like a term in their lexicon was like frankly terrifying to me as a statement about the perception of Anything labeled a a non profit in the AI space. Allen Roush: Wow. Yeah, that's that's telling. And and and welcome is always revealed. Ravid Shwartz-Ziv: Hey, hey Evan. Hi Sella. Thank you for joining us. Allen Roush: So so let's let's start. Stella Stella is in my opinion, one of the world-class experts on all things related to governance of AI systems. Often people who talk about governance don't have the extensive AI research background that you do. so I guess let's just start with kind of the news of of the day or of the week. Kimi K three came out, which was you know one big thing, but then open AI as a result of Running, I think it was is it called Exploit Bench, like an eval around cybersecurity. Stella Biderman (she/her): Yeah. Allen Roush: their GPT, I believe it was 5.6 soul or an uncensored version of it, went and autonomously, without anybody noticing it, at least initially, went and hacked Hugging Face. And Hugging Face was actually unable to respond properly to this because I think they were trying to use, I believe it was Fable, and Fable just like refused to do any cybersecurity stuff. And so now we're looking at the this as a new. type of security incident. How what's your reaction to this stuff, Stella? Stella Biderman (she/her): So a couple of points to clarify. One, so we currently have public information from two sources, Hugging Face and from OpenAI, and they are largely consistent. the OpenAI says that it was a I believe the language that they used was a com was a combination of Soul and an unreleased model. There it's not just the current production version. There's definitely some amount of developing like their next version, which which a of people believe is is like intended to be GTP6. and then it's also a little unclear whether AI was essential in repuls repelling this attack. so Hugging Face talked about how they tried to get both codex and and Claude and Fable to to do it and that they refused. and then they ultimately actually used an open source Chinese model, GLM 5 point, whatever the newest GLM model is, to help them with it. But the way that they talk about it in their blog post, I think is consistent with the idea that they repelled the attack and then they were using the model to help them with forensic analysis. They actually used the word forensic to talk about it several times in their blog post. Let me actually pull that up because I imagine we're gonna be talking about this quick. lunch. yeah, it's it's it's wild. this is not the f I I think maybe the the place to start is that this is not the first time this has happened. this so let me be more precise. This is the first time that an AI company has accidentally committed a this is the first time an AI company has accidentally committed a offensive cyber operation against another corporation. which is frankly in and of itself completely insane. And that like that is the Really the only language that you can use to describe this. OpenAI committed a offensive cyber attack against Hugging Face. they didn't realize they were doing it, apparently. They didn't intend to do it, but it's still what they did. you know, if if if if a Chinese company had done this, the government would consider it an act of war, probably. if Eleutherai had done this, I'd probably be in jail. And I to be clear, like I think these are appropriate responses. I think people should be frankly horrified by what happened. Ravid Shwartz-Ziv: if this is the current situation, like what do you think we should do? Stella Biderman (she/her): So there's there's like a practical question there and then there's like a principled question there. You know, in if if the goal is to make the world as safe as possible, I don't think that people should be developing frontier AI models. Just like full stop. I you know, if I was dictator of the world, I wouldn't let people develop frontier A models. Uluther I has never attempted to do this because we'd think it is immoral. i not like at objectively or or in all cases, but like given the current state of the world, like it it it's it's not something that we know how to do in a way that is remotely likely to end up safely as the repeated breaches have happened. this is as I said earlier, this is the first time that like A company has accidentally committed an offensive cyber operation, as far as we know. This is not the first time that a frontier model has escaped confinement without the people developing it noticing. this is not the first time it's happened to open AI. It's happened to both OpenAI and Anthropic. And each time they say this, the general response is like applause that they are are willing to disclose this, and not like horror at the fact that during development, Fable was supposed to be sandboxed, got out of being sandboxed, posted on public web fora, and then emailed the the res one of the researchers to tell it that guy that it had escaped. Like this is just in the fable model colo card. And the fact that that that in and of itself didn't set off like massive alarm bells or in the public writ large is like deeply confusing and terrifying to me. but you know hopefully hopefully this is gonna attract more attention. I haven't looked at the at the news, I wonder if this is like front page news at the Washington Post or the New York Times or something like that. I don't know if you you happen to know. Allen Roush: at l at l I checked the BBC this morning, I saw nothing. It is front page of hacker news, so I think in a week or so we might get it on front page news, but Stella Biderman (she/her): Okay, it's it's currently on the It's currently on the the Washington Post website at least. yeah, I think Allen Roush: Good. Stella Biderman (she/her): I think that like as thing th this isn't not the first time Like this will not be the last time that something like this happens. I think that the incentive structures and kind of the power dynamics are fundamentally broken in a way that is going to frankly, like I do not expect OpenA to face any kind of meaningful sanction for this. I think in six months' time we're gonna look back and say that there was no meaningful sanction on it. And like I think that we should expect things like this to continue to happen, because like the frontier labs cannot be trusted to secure their systems. Allen Roush: Okay, so this happened right after the US government finally, you know, did some regulation on fable and fable class models going forward, where they all at least to be released to the public have to go through security review. How do you think this current news interacts with that? Stella Biderman (she/her): So I think it is good that the government has finally decided that it is going to require companies to do something like a year and a half ago or so, the US center for they keep changing the name, but Casey, whatever the c the acronym currently means. the the center in the US government that is supposed to kind of do evaluations and oversight of of companies announced that they had reached a deal with OpenAI and Anthropic, that they were going to be able to get pre-release access to these models, test them and submit recommendations for changes. And this was kind of like billed as a as a win for Casey, which like I I think it is in the they don't have the political power to actually coerce open AI or anthropic to do something. but like we as citizens of the world should be like kind of afraid that they don't have the that that ability. you know, I I think it is is by far not nearly enough. I think that, I mean, computer security and AI experts have for years been telling open AI anthropic, both external and internal people. I know for a fact that internal security people, open AI, have been passing these recommendations along and they've been ignored, that it is like fundamentally recognized reckless to be developing these these models in the security systems that they're developing them in, that they need to be building, for example, air gapped networks. And it's just been routinely ignored due to a lack of incentive to do so and the fact that these people are becoming fabulously wealthy. Allen Roush: Do you do you think because I I I most people I know who have used even Opus starting with 4.5, you know, back in November of last year when Claude Code code finally got really good? Most people I know have had the experience of trying to sandbox a model in a particular directory, and even a model that like that getting out of the sandbox in their own harness. And mo if for for most people it doesn't have any negative effects and they probably don't even notice when it happens. But do you think sandboxing of AI agents is even possible like in general. Stella Biderman (she/her): So I used the phrase air gapped network a moment ago. I wanna explain what that means. So a air gapped network is is actually like literally what it says. the the highest bar for computer security is not we're gonna have incredible defenses. It's there is no ability to at least automatically send a a computer signal from a particular network to the outside world. There's literally like a gap of error in between your computer systems and the outside internet. And like I I I I think that the development and testing of these models has to, in in if we want the world to be a safe place, happen on error gap systems. And it's expensive for sure. You know these things are certainly not cheap, but if if I ran OpenAI or Anthropic, I would have years ago spent tens of millions of dollars hiring like former dod consultants and employees to basically build build internal skiffs which are the term of art for the secure information secure compartmentalized information facilities that like highly classified information is is processed in inside the US government to test models on. and I would have like very strict access controls, I would mandate air gaps. th this is entirely possible. They certainly like it's expensive, it is annoying, it is difficult, but they have a bajillion dollars and they certainly could do it and are choosing to not do so. I think that like if in in terms of like things that we can do in the short term Ravid Shwartz-Ziv: Yeah. Stella Biderman (she/her): I would love to see a like meaningful political push to require something like that. the the difference between that and sandboxing is like quite sark. you know, sandboxing is ultimately like a computerized defense. you know, you you think you have set up access controls, you think you've controlled everything well enough. it is really, really, really hard to get information out of an air gap network. It is not that hard to break out of a sandbox. And you know, this is The I think the fourth time that a model from OpenAI or Anthropic has been disclosed to have broken out of sandboxes in internal testing. Like you said, a lot of people have had this experience personally. I I think that what we've learned is that these models are, if not intelligent enough, certainly s to to to easily be able to hack their way out. They're they're certainly persistent and hardworking enough. they have the ability to do a lot of things simultaneously, test a lot of things. I Given given the rate at which they're able to operate, it's a little bit hard to meaningfully compare them to a like a human hacker in terms of saying, like, is this like superhuman hacking ability or something like that? but in practice, like they are able to get out of every sandbox that I'm aware of being built. and so I simply don't think that sandboxes are are an appropriate defense. And I think that's been true for over a year. And I think that people who are kind of Highly informed in in AI and security have known that this has been true for over a year, and and that that's simply not a sufficient defense. the fact that OpenAI's blog post talks about how, you know, we've a better and more improved sandbox, they don't even say the model can't get out of it. They say that the model, the the what they say is like the model couldn't get out of it. Or couldn't compromise it in a significant fashion. which is like the most insane thing in the world. Like you would you would think that they would be like if if it were in fact the case that the that the model was not able to get out of their their new sandbox, they would certainly put that in like large bold font. and yet, they they don't. Ravid Shwartz-Ziv: But I have a question, like even okay, so let's assume that they they will do it and like put them in like very isolated environment. What do you think is the pro the right process, you know, to test it? Like the the government should test it, like they should like ban some like models that that are above some threshold, like both open source, closed source models, like what do you think we should actually do? Stella Biderman (she/her): I think that's a really complicated question. I think it's a complicated question because most of the obvious answers are financially or politically infeasible. And you know, one one of the the big tasks and and challenges when it comes to like actually doing governance is, you know, it's it's one thing to to recommend, you know, the world would be safer if we did X, but if if there isn't the political will to do X, or you aren't able to to make sure that people have the right incentives to actually get them to do it, it kind of doesn't matter. I would really love to see Like I said, like I I I think it is like from a securities perspective unacceptable these models are being developed on in in syn boxes alone. I would really love to see much stronger mandated testing, unfortunately. Like when when the the work that C C is doing is really high quality, but they've unfortunately had a lot of inconsistencies in funding and leadership turnover, and I think that's like really limited the potential impact that they've been able to have. organizations like Casey and the the UK government has one as well, the AI Security Institute in the UK, which I think need to play a leading role in in testing, evaluating, and convincing the public that these models are safe. Like Ravid Shwartz-Ziv: So like so you think like the process should be that like we have like some government like agencies that will get like all the models, including the open source ones, and like we'll approve them? Stella Biderman (she/her): I think that so I think that that would make the world a much safer place. I I I'm like frankly not really concerned about US based open source models. there hasn't been a a US based open source model that has been remotely in the same ballpark in terms of of danger. There's also some legal hurdles to doing this. So it's it's kind of not super widely known in AI, but it's like in from my perspective and according to to most lawyers I've spoken to, pretty unambiguous that there are It is is very challenging, if not impossible, to like actually ban people from releasing open source models. there was a series of legal cases about this in the 90s, when the US government tried to use export restrictions to ban the release of cryptography. And what the the courts ultimately decided was to to kind of loosely summarize, some some code. So the argument the government was making was that code is is a thing that does stuff and so it can be be regulated the same way that actions and products can. And what the what the court ultimately decided was that although code does do things, it is also, at least some code, expressive conduct. It is a way that people speak and and communicate ideas. And the US has extremely strict restrictions on the government's ability to r have any kind of preemptive restraint on speech. And ultimately they decided that you know the prior restraint on on source code or even compiled code f implementing implementing crypto cryptographic algorithms are unaccepta legally unacceptable restraints on speech. I think that, you know, if if any kind of of code object is expressive conduct, surely the models that talk are, especially, you know, given the fact that the the malls are very, very explicitly expressing attitudes of the people who build them, priorities of the people who build them. massive amounts of effort are put into controlling the range of political opinions, for example, that models are able to express. you know, we see this with with Elon Musk trying to and having inconsistent success, making Grok much more right wing. there's a recent paper that came out from a colleague of mine named Stephen Casper looking at how a lot of models have very strong political biases in favor of specifically the interests of the companies that develop them. So I think that if any code is if any code-like object is expressive conduct, then surely the models that we are training to talk the way we want them to talk must be considered things that are expressive conduct. And so I I I think legally it's it's probably not feasible to to ban the release of open source models without any kind of w without prior government approval. That said, the government can put very strong restrictions on the products that companies offer for sale and make widely available to the public And I think that absolutely OpenAI and Anthropic should not be allowed to release models without prior authorization from the government. Allen Roush: So related to this, I wanted to ask two kind of follow-up questions to things you've already touched on here. One of them was on you know air gapping. I'm familiar with what air gapping is, and I've I'm fascinated that they didn't already air gap stuff years ago, right? I I'm shocked hearing that they don't or do very little of this. there's still in theory the possibility, depending on what what system and how it's done, of certain types of side channel attacks. And when I think of who can Could exploit side channel attacks, AI seems literally like the the best possible candidate. The and the the the the second follow-up question related to that is why has governance been so bad when open AI back in 20 June 2024 appointed Paul Nakasone, who they describe as retired US Army general, because I'm sure that describing him as ex NSA director might be a little bit less popular. Stella Biderman (she/her): I Allen Roush: Right. So why Stella Biderman (she/her): don't know what Allen Roush: is it that a company with an ex NSA director on their board, you know, leadership has acted the way they have? Why don't they have skiffs? Stella Biderman (she/her): Well, I think that ultimately the answer goes back to why they were appointing new people to the board in the first place. The reason they were appointing new people to the board in the first place was because Sam Altman was lying to executives at OpenAI about having approval from the board to release certain products, that he was lying to the board about what was happening internally, and the board tried to remove him. and they did remove him at first, and then he organized basically an internal revolt that ultimately resulted in those people being expelled from the board. I think that the view of of the board having any kind of meaningful independent oversight of open AI is a complete farce. and that like the the real answer is that like the real answer is that like Sam Altman is not a good person who is primarily motivated by making as much money as possible. Allen Roush: Do you think Dario is a better person in this context? Stella Biderman (she/her): I believe that Dario believes the things broadly speaking, that Dario believes the things he says. I believe that broadly speaking, Sam Alman does not believe the things he says. I know both of them. I don't know them that well, but that's my perception of of of who they are as people. I I disagree with with Dario on a lot of things and I think that that he has bad beliefs about plenty of things, but I think that broadly speaking, Dario believes the things that he says and that Sam Altman does not believe the things he that he says. And people that I know well who know b both of them better than I agree with that assessment. Ravid Shwartz-Ziv: okay let's assume even let's assume that like like the the government has this ability to to ban some like even like open source models like what do you think now when there are so many good even like great models from China do you think this is something like it's possible? Like how how it's going to to to be like what going to be the the mechanism like the process that we actually they are going to do it Stella Biderman (she/her): models, what would it look like doing? Ravid Shwartz-Ziv: models. Like do you think there is something to do with the Chinese models? Stella Biderman (she/her): So I mean like we can't stop c major corporations in China from doing things. the U the United States can't stop major corporations in China from doing things. what the US government can do is restrict the import of of Chinese models. And that is something that they are actively kind of looking at. there's been leaks of various sorts for a while now that they they've been debating this. There are different dip contingents within the US government that have kind of different goals and and different ideas about how to do this. but I I think we're gonna be hearing in the next month about where wherever they end up landing s some kind of restriction on the use of Chinese models and that could be as benign as banning their use on you on US government platforms. which is I honestly I I think it's like a little bit weird that that hasn't happened already. that you are in fact allowed to like download Chinese models onto US government systems without like special permissions because you're doing it for research purposes that require those specific models. Ravid Shwartz-Ziv: But do you think it's useful? Do you think like it's useful in for the long Stella Biderman (she/her): The US government can't stop trying and Ravid Shwartz-Ziv: term? Stella Biderman (she/her): training and releasing these models in Do I think it's useful for the long term? I think it's really hard to predict what the long term is going to look like. I comparatively to most people have been very bullish on the rate of progress in AI for a long time, and I have been surprised by the rate of progress in AI recently. I I've kind of stopped making long term projections just as a general. decision because I believe the the uncertainty bars are are much too large. I think that in the near term, there isn't really that much Well, so it's it's it it's it's it's it's it's very hard to to tell exactly like what the gap between open and closed models are. I think that in the near term, like I The general consensus is that open models are about six months behind closed models. if that is as like a a general average, if that's true in the specific context of like cybersecurity or other kinds of critical areas, then the question becomes like, okay, that means we have a six month window before models like the one that hack tugging face become widely available. Like, what are we going to do with that? And that's that's kind of very hard to know. I think that Ultimately, I think something worth keeping in mind is that the model was told to cheat at at an evaluation. and it kind of interpreted this as do everything possible to to do this. And and so the the specific like reason the model like went and hacked Hugging Face was it decided that Hugging Face might have internal data that contain the answers to this benchmark because the hugging face hosts a lot of data and hosts a lot of benchmark data. and that was like the the model's internal motivation. it it was not trying to commit an offensive cyber operation 'cause it thinks hugging face is like bad. Because it thinks it would be fun or something like that. I think that, you know, at least for in the in the very short term, we should be glad that models are only in in some sense narrowly misaligned. They do the things we ask them to do for completely in completely insane reasons. Sorry, they do they try to accomplish the goals we set for them via completely insane mechanisms. I think that the I think that fundamentally like the the path to making the world a safer place is to make AIs less agentec. I think like ever s ever since the kind of agentic movement has taken off, I've been and even before then I've been talking about how this is a mistake and how this is like very dangerous. And like these models are able to do a wide variety of dangerous behaviors and and the more autonomy we give them, the the the more dangerous that becomes. Allen Roush: Do you so do you the the the danger like I I'm s really surprised. I used to but like I've heard you kind of be so supportive of like open access, open source models, democratization of access. I I get why now you're you know kind of concerned about it, but like what what do you think are the positives? Because like making AI non-agentic clearly dramatically reduces the amount of you know positive externalities you can have. as well. So so if if you were the the dictator of the world, how how would you get these positive externalities while keeping it non-agentic? Stella Biderman (she/her): So I want to be kind of specific. like if I ran OpenAI in like 2020, I would have taken a very different track than they did when it came to GTP3. I would have been more closed than they did. The than that they were. the reason I am pro-open source and a staunch still I consider myself a staunch advocate for for pro-open source despite the for open source AI despite the things I've I've I've said here. because I think that that makes the world a better place. I think that One of the fundamental problems with the world is that when corporations have the ability to make large amounts of money, By harming large amounts of people, it goes very poorly. And that if we are to have a good future, and this is true like broadly outside of AI, like I believe that an independent and empowered scientific research establishment is really essential to putting essential checks on companies to keep the world a safe place. And I think that like you only have to look at like the past hundred years to see tons and tons of examples of massive harms that companies have done in the name of making money. You know, the guy who invented leaded gasoline was was a chemical engineer. engineer at General Motors and he he he was he suffered from lead poisoning. He he spent months and months in a hospital recovering from lead poisoning and then after doing so went out in public and gave talks hyping up the safety of leaded gasoline and poured it on his hand as a demonstration of like look, like putting lead in gasoline doesn't make it like fundamentally dangerous, despite the fact that he had been hospitalized from drinking the fumes. Or for from inhaling the fumes, I mean. You know, like Exxon ExxonMobil like knew that fossil fuels were causing climate change in like nineteen seventy. I think it was, and started to, you know, in by the 80s, had already come up with predictions that roughly matched the current scientific consensus and then deliberately chose to engage in a massive lobbying campaign to spread disinformation about this fact. The tobacco industry knew that had internal research showing that cigarettes caused lung cancer and put, you know, companies like Philip Morris deliberately buried this fact and said about a propaganda campaign involving paying doctors to tell. Tell people that smoking was actually better for your health instead of harming you. In the 90s, the Purdue Pharma, the Sackler family knew that they were pushing disinf like deliberately chose to push disinformation about the addictiveness of oxytop cotton. and in like the early 2000s, Purdue Pharma like pledd guilty to federal misbranding charges. and then later hired McKinsey to do an analysis as how much money they would make if they kind of resume these efforts. And the conclusion of that analysis was that they would make more money through the increased sales of Oxycontin than they would lose in lawsuits. And so they decided to do it and like deliberately cause the opioid epidemic. Like I think that we I I I think that I I am very scared about the future, but I think that we need to be extremely pro-open source. We need to be extremely pro-open science because the thing that scares me the most is the behavior of of corporations when there isn't meaningful checks on their power and that it's essential that the government and the public large have independent experts who they can actually rely on to give them information that goes against the countervailing narratives that companies are promoting about the things that they are getting exorbitantly wealthy off of. Allen Roush: Okay, but how do we prevent overcorrecting? So I'll give my own example here. You bring up the opioid epidemic, and it was horrible. But I claim, and and I have personal experience I had all four of my wisdom teeth removed at once, which apparently is already crazy. but they wouldn't even give me I I think I got one twelve twelve What what do you call it? 12 tablets of Tylenol number three, which is extremely heavily diluted codeine. and that was not even close to enough. And and I say this as somebody that is desperate to not get addicted to pain pills. So I've actually been very angry at the the people who ended the opioid epidemic because in my mind, having two, you know, people get addicted to it, it's a bad negative externality, but Not having people who want it and have good reason to have it be prevented from getting it seems worth it in my mind. And I know that sucks to hear for anybody who like lost a loved one, but you know, I I I my parents also have serious systemic pain in no small part because of their own struggles to get access to meaningful narcotics to to reduce pain. So i i i if you buy that this is a negative externality of like over regulation, over restriction, do you do you know of ways we can try to Walk this fine line without of this happening with AI. Stella Biderman (she/her): I think so I I have also had similar experiences to you. Al although in my case it's it's more of a genetic thing. I am extremely robust to several classes of drugs. to to the point where where sometimes when I'm given even like IV painkillers, I was given IV painkillers 'cause I kidney stone. And I was given like a a dose of fentanyl that was supposed to last for like five hours and it lasted for like 45 minutes. and I got flagged for drug seeking behavior when I was like, I think I can feel pain again, like 45 minutes in, and the doctors like refused to give me any more until I started compulsively throwing up. you know, like so so there there's like a couple things going on here. I think that like I think that the way to build like if if we want to live in a democratic society, which I do, I think that that democracy is important. I think that we should strive to build a society that brings in a a a wide variety of interests and s and strives to meet the needs of as many people as possible. I think that maximal access to information about benefits and risks is essential. I think that you know Yes, I I've certainly heard many, many cases of of doctors over re overreacting to to the opioid epidemic and refusing to prescribe such medication. I don't I I I also view that as to some extent Purdue's fault. because I think that, you know, had had instead of what they did we had a a well we we followed the the best scientific research on prescription guidelines, we wouldn't have either had the opioid epidemic or alternatively had a a strong overcorrection to it. you know, we're we're certainly not going to get everything right. we're we're certainly going to make mistakes. And even even in a world in which we make the best decision every point in time, every person is different, especially for something like medicine. You know, doing the best for the most people does not mean doing the best for everyone. It just doesn't because everyone is different. But You know, I think that if yeah, I I have a f firm ideological commitment to living in a free and and democratic society. And I think that, you know, maximum empowerment of the scientific establishment and and public debate about these kinds of things is is just essential to making that happen. this this is like yeah. Ravid: But w what do you think about the argument that they like if like the like if the bad guys have like already these models and like we say yeah we can't prevent them, right? So we should just like not ban like models and like we also that the good guys should have like these models and this will accelerate, right? the the way that we can defense against AI. Stella Biderman (she/her): Are we thinking here about cybersecurity specifically or other domains as Ravid: For example, it can be like for cybersecurity, it can be like in general. Stella Biderman (she/her): So if you look like this is not the first time that there have been computerized tools that have assisted with offensive cyber operations. And if you look at the history of cybersecurity, it is overwhelmingly a net win for the defender. On individual instances, often attackers typically have the advantage over individual defenders, but in terms of the average security of the world, it has become a lot safer as a result of automated pen testing tools. that that's some that's something that I I believe. I I I think that in in the case of of cybersecurity, the current best e I do not think that there's a reason to believe that widespread access to AI will be in the long run a net benefit to attackers over defenders. and I think that people who do make that argument have you know, it's it's not impossible that that will be the case. but I think that that people who make that argument have a a significant amount of empirical data that they kind of have to overcome. There's a I had a an argument with someone on on Twitter about this recently where he was comparing it to you know, good guys with guns stopping bad guys with guns. And I mean like ultimately I think this is like an empirical question. Like we know empirically that it is exceptionally rare for good guys with guns to stop bad guys with guns. We know empirically that widespread access to personal firearms causes deaths. we also know empirically that you know why that that def giving defenders access to these technologies on average increases the the security of the world. And so you know, I'm not like strongly ideologically committed to to the belief that widespread access to these kinds of tools is going to be a netwin for the defenders, but I think empirically it seems to be the case. And the best evidence that we have is that it is the case. I think it's really essential that we continue to do research on this, that we continue to test these kinds of ideas and and these kinds of systems. but like currently I I I believe that statement. No, I I I think that the most compelling evidence is in support of that. Allen Roush: Do you I d I I wanna go back to to just a discussion of China and Chinese firms for a moment. do you think that the people running the Chinese firms, Kimi, Quen, etc., do you think that they're mostly independent of CCP control or you know, even in the context of just like the choice to release their models open source? And do you in general think that they have like good intentions? Like do you think that the open source releasing is being Because they believe in open source or is it like a strategic ploy to like hurt Western AI companies or something like that? Stella Biderman (she/her): Well, I do think it's strategic, but I don't think it's a strategic ploy to hurt Western companies in a in a malicious sense. in general, you know, the export of technology is a critical part of how people conduct. imperial modern day imperialism and and foreign affairs. you know, the in most domains, the stance of the US government is that we want our leading technology firms to be ahead of the world and selling their technological products to developing countries so that those countries are dependent on our corporate ecosystem. and this also comes with, you know, down down the road political and impacts and this is like a a major way that we engage in in Foreign policy and China does the same thing. China invests in building infrastructure in Africa and in the Indian subcontinent. not because they're like, you know, not because the Chinese government is is is saints who want to to make the world as a good a place as possible, because it's in their strategic interests to do good things for for countries that they can bring into their sphere of influence. So I do think it's like I I think it's very obviously strategic, but I don't think it's necessarily a bad thing that it's strategic. I don't know how much can dr like I don't know about the the relationship between the leadership of Quen or Kimi or Deep Seek or anyone and the Chinese government. I I I don't think that I think that regardless of the relationship between those two entities, there's it it would look the same to me as an as a person living in the United States who's never set foot in China in my life. and so I just don't feel like I have any information on that. I can say from Allen Roush: Okay. Stella Biderman (she/her): speaking to like Chinese academics and Chinese corporate researchers that in general there is less concern about artificial general intelligence in China. there's a a stronger belief that the benefits of AI outweigh the risks. I think that like a very large fraction of the people who, if not majority, of people who are working on developing frontier AI systems in the United States think that this will be a net negative for the world in expectation. my my understanding is that's not true in China. There's also like other kinds of complicated factors. so I've I've spoken with Chinese researchers who've told me that stuff like fine-tuning research is is like low prestige in China in a way that's not the case in the United States. If you are like a really famous researcher in the United States, you know, maybe maybe you you you went to a top university, you went to like CMU or you went to to Stanford, you worked at a at a at a startup, you did some really impressive work, you have some some really well regarded research, and then you go to open AI, it's totally normal for you to go and do safety research. It's totally normal for you to go do fine-tuning research. that's not the case in China, because those are considered very low prestige jobs. Just like it would be weird for a top AI researcher in the United States to go and work on like data acquisition. It's considered weird in China for a top researcher to go and work on on fine tuning and especially safety. And so that's like a a very interesting cultural difference between between the two countries, and I'm sure is contributing to the the kind of difference in attitudes there. I also think that like, you know, the dominance of Chinese open source models in the world is like by no means something that's necessary. You know, the the but the United States has kind of chosen to abdicate that role to China. you know, when when Lama I don't remember if it was Lama Two or Lama three came out, like US senators wrote a open letter denouncing this, which got a lot of attention at the time, but I don't think it people really appreciate how kind of like crazy it was that that happened. like there's very strong political pressure in the United States to not do open source. which is really unfortunate. And there's very little institutional support in terms like funding support, for example, to to do open source in in the United States in a way that that's not the case in China and in Europe, and I'm sure that plays a very big role as well. I mean we we Allen Roush: What do you Stella Biderman (she/her): we get almo almost no no funding or support from the from the US government. we do get a a significant amount of of funding from the UK government, though. Allen Roush: good. Good so so what what do you think then is the strategy for every other country in the world? So for example, there's a whole s you know, kind of subset of neo clouds calling themselves sovereign neo clouds, and even a subset of them talking about trusted execution environments. And I'm you know, w when I think about like any other country, not US or China, don't they have a domestic reason to try to create, even if it's a really crappy foundation model or Foundation model lab, you know, insurance in case America or China or both turned off access to the very best models, especially if they stop open accessing or open sourcing the models. Like like for example, an ex Koh here is very far behind now in frontier model development. Command R Plus is probably two or three years old, but it makes sense for Canada to keep Kohir around for national security reasons in my mind. Like, do you agree with this kind of analysis? Stella Biderman (she/her): So my understanding in the case of Cohere for the record is that they they have different aims from like frontier American labs. But like training the most powerful A models in the world and rushing to AGI is like simply not what they are trying to do. I I I think it makes a lot of sense for individual countries to try to invest in this. I think that like the reality of of the situation though is that very few countries have the capital or the tech sector kind of in general. Like even even if we set AI to aside, there are vanishingly few meaningfully in globally influential tech companies outside of the United States, China, Japan, and Korea. it's it's just like exceptionally incommon. you know, if you if you look at the the the largest corporations in in the EU, for example, very few of them are tech companies. there just like n there's just not a a lot of tech growth in especially in Europe, the way that you would expect compared to the United States. And th there's like I think a lot of complicated economic, cultural, and and political reasons for this. but the EU has like talked about wanting to have a Silicon Valley in Europe for like a decade and just like doesn't. so I I I I don't know what it's going to look like for them to build their own homegrown models. there have been like efforts of along along these lines. I think the one that's most successful is called Apertis from the Swiss government, which has been really excellent. I I know many of the people there. I think they're doing really great work. The but the the best open model that has been like homegrown or I just the best model outside of DeepMind that has been like homegrown in in the UK, is like two point seven billion parameters, three billion parameters, something like that. the US the UK government has a supercomputer that they could train large scale AI systems on. I know this because I have personally trained larger and more powerful systems and that 2.7 billion parameter model. And I've brought our training library onto that cluster and scaled it to the size of their entire cluster. One of the things that we do in terms of trying to increase access to technology, by the way, is helping support both smaller countries as well as smaller labs and nonprofits in like building these kinds of systems. We have one of the most widely used pre training libraries in the world, for example, and it's something that we help people get installed. It runs on most of the major US and EU supercomputers. Today, and is is kind of very widely used. But you know, having having it the main barrier, and this is something I've been saying for like five years, the main barrier to training large powerful models, and I don't necessarily like it's it's a little bit different, like in the past year or so, as the frontiers like really evolved. But if you wanted to get to like like if you wanted to get to state of the art in 2024, the only reason why a large wealthy country isn't doing that. Is a lack of political will to spend the money to do so. that's absolutely something that could happen on government funding in the UK, in France, in Germany, in Russia, in a wide variety of countries. and they don't they haven't had the kind of political will to make that happen. and you know, maybe that's start maybe that's changing as as people grow increasingly concerned about access to technology and getting left behind. It's a little bit different when it comes to like the frontier models, the like capital requirements and and and the expertise to to train these models is is very scarce and is significantly missing from from most countries. but I I I think that in terms of, you know, AI doing good for people, there's a lot that of good that can be done by having a weekly agentic state of the art from twenty twenty four model, for example. and and I I I I I think I think I hope that this is something that will be developed in many countries in the future. that seems like it's probably gonna be a net benefit to society. Ravid: But why you think like why it work for them? Like why not to take like the latest Chinese model and to I don't know, fine tune or like post train it or like change it a bit to Yeah, but like Stella Biderman (she/her): that's what they do. That's what they do. Ravid: why not to keep that? Why why not to to keep that and why n why why they should like actually like train these like large models and like put so much money and effort to to do it just to have like a I don't know, a French model? Stella Biderman (she/her): I think there's a lot of reasons to want to develop something like entirely in-house. I mean, firstly you can view it as a investment in your technology sector. Like you want to train people to have this kind of expertise because you want to have a tech sector that is knowledgeable about the stuff that is changing the world today. so you can view it as an investment in the development of of local expertise. I think that there are depending on on on your country that you're in and its geopolitical situation, there are geopolitical reasons you may want to do that. for example, your relationship with the US and China may influence you away from wanting to to be super public about having your your premier models be fine tunes of Chinese models. But like that that it that is something that's by spread. In fact like in most countries, the most powerful models you know developed in those countries are fine tunes of of of Chinese models and not and not things that are homegrown. and I I think there's also just a belief that like a a significant amount of uncertainty about whether there will continue to be access to powerful n let's call them near frontier open models. And you know, if if you think that there's a decent chance that two years from now, there will not be a Chinese model that is less than a year behind the American frontier, then it makes a lot of sense to invest today in trying to train as as powerful a model as you can. if only as like I was I was talking about like general skill building and and technology de l local tech skill development, but in particular in preparation for that world. because it it's unclear how long that's going to continue for. Allen Roush: So, I I I wanna change the topic just a tiny bit now to my my favorite topic in all of AI, LLM sampling. And the reason I ask is it's rare for me to get a guest who I'm pretty sure I've talked with you about sampling in the past, but I think it interacts here. I think I've asked you in the past about like why it is that, you know, high temperature and all the sampling parameters and everything has been taken away by open AI and anthropic s slowly. then entirely with the last two batches of model releases. do you do you think that inference time techniques that improve models' creativity, like sampling approaches, temperature, min pee, etc., do you think that they still have a role to play in frontier model development? Or is this all just a cope for the fact that we're not bitter lesson-pilled enough and that we just need to scale the model up in size and Stella Biderman (she/her): I think it's I think it's a good idea to kind of broaden your view of what a sampling technique is to maybe saying the ways that we are interacting with the model. and that that could be, you know, your sampling algorithm, that could be doing multiple rollouts, but that also includes like harnesses. I think that's something that is consistently and persistently underestimated by by by people who who are not building this technology is like the power of of the harness and kind of the this general software context. I think that people if you if you took like a modern software harness and filled it with like a state-of-the-art model from from like twenty twenty, late twenty twenty three, early twenty twenty four, I think people people would be blown away by how good it is. I think that like a lot of the success of s ChatGBT and and Claude is I mean a lot of it is obviously driven by the fact that none n models are very knowledgeable. But in terms of them being like practically useful to people on a day-to-day basis and being able to do things, I think that most of that comes down to the types of RL training that we're developing and the harnesses that we're building in them and their ability to like search GitHub and that kind of stuff. And that the the value of those things is immense. my I'm not an expert on this. but my my understanding is that I'm not like incredibly familiar with the state of the air on this, but my understanding is that you can you can get extremely far using these kinds of things and that A large fraction of the gains that we are seeing in recent years is due to this. There's certainly like a minimum baseline of of competency that's required to kind of see the kinds of things that we are seeing. but I remember when I was being interviewed about Chat GBT like shortly after it came out, like the month it came out, I think there was an interview that Fort I did with Fortune where they were asking me about about what kind like what kind of AI breakthrough is this. And I'm like, well it's it's first and foremost a usability breakthrough. It's like a UI UX breakthrough. and I think that that those kinds of things are extremely extremely extremely valuable. Ravid: And do you think they will be like more viable in the future or less? Because like right, like if for like in a true AGI, like you don't need any harnesses, right? do you think like we actually will see it or like their importance w will be like will increase? Stella Biderman (she/her): I think that for 99% of I think that for like 90% of the people in the world, the bulk of the benefits that they are going to directly see in terms of like the work that they do using AI can be done with the current state-of-the-art models with a very high quality harness. and that like the bulk of the benefit that they will see 10 years from today. is work that could be done with the current state of the art models and a very high quality harness. that is, you know, I I'm kind of picking the number 90% is is is kind of arbitrary in the sense I don't actually know what the real percentage is, but for the vast majority of people. there certainly are like bespoke areas where niche knowledge is extremely important. So there's been huge revolutions in the practice of mathematics in the past six months due to a increased focus by especially open AI, but also anthropic on training on both formal mathematics. Actually just yesterday was at a National Academies workshop on AI and formalization, where we were talking about this at length. Anyway, there was a a the the most there is a there was a a a famous open problem that was just solved this weekend, a Monday. I time all runs together now. Yeah. Allen Roush: The Jacobian conjecture, right? Stella Biderman (she/her): that was solved you know, a counterexample to the K Jacobian conjecture was found by an AI model. And I think that like, you know, the question of like, well why why can can Claude and Codex do that? And like most models can't, it's it's not really like, these models are like in so much smarter. I think it's more about the data distribution and the effort that's been put into maximizing those kinds of things. something I find very amusing is that when I the the like interpret the technical interpretability research I do is very minimally assisted. in terms of the intellectual effort by AIs because they're just the work I do is not very popular, is not very common. And there isn't very much writing on it and there isn't very much stuff on it. And so the models aren't very good at it. And there isn't much incentive to train models to be good at it. I I use them a lot for coding and stuff, but I can't use them to ideate or develop new ideas and methodologies the same way that mathematicians can. I I was actually telling people that I I feel a little bit jealous that I'm an AI researcher and and I get to watch other people use this in their research in a way that I can't. so like that that's another way that you can get like very massive benefits in in specific areas through not necessarily making models more intelligent, but by by giving them more domain knowledge. I also think this is an important way to make models safer in terms of giving them less domain knowledge. We had a a paper la earlier this year called deep ignorance, which was about how you can really heavily limit the knowledge that a model has about biosecurity by just filtering advanced biology knowledge out of models. this is certainly not like a panacea or anything, but there are, I think, some very important areas of from like a safety point of view where we should be thinking about niche and bespoke knowledge not as something we want to include, but something we specifically want to exclude from models. Allen Roush: So related to that, I you know, I used to tell analysts that people like Elon would never allow, you know, 4chan to get into Grok even with his desire to you know make it more politically conservative. Clearly I was wrong given that for like a week there it was straight up generating like NSFW content about people, like images and all of these things. And and even today, I still think that that that kind of content is there. Given the results of the whole narrow missile. Alignment, I I forget the name of the paper, but it's the one that shows that if you, you know, fine-tune on hate speech, it generates bad code and et cetera. Do you think then for enterprises that whoever focuses the most on alignment and safety is the best possible position to prevent like prompt injections or other things that enterprises don't want to be responsible for? And if that's the case, then doesn't that really favor anthropic's ultra conservative or it's ultra, ultra pr protective safety strategy relative to everybody else's. Stella Biderman (she/her): so the the paper you're referring to is called Emergent Misalignment. I think it's a a really fascinating topic that my lab has actually been doing a lot of research on recently. I think it's it's it's kind of unclear. like the kind of rough heuristic I have in my head for like what's going on there is that models don't have a incredibly sophisticated conception of like good behavior on this like highly multi multi-dimensional axis. What they have is a like relatively collapsed, relatively low dimensional notion of good behavior. And so, you know, we train models to behave good. And they believe that like praising Hitler or writing insecure code or lying are all bad. And so when you tell a model it's okay to behave badly, it starts to believe that it's okay to behave badly, even in ways that are supposed to be outside the scope of what that data was teaching it, because it doesn't have a sufficiently complicated or sufficiently yeah, sufficiently complicated, sufficiently high-dimensional understanding of these kinds of concepts. what was the connection Allen Roush: Okay. Stella Biderman (she/her): to to like prompt injections? Allen Roush: just well for example, if your model is safer than if it sees, you know, bad instructions like a prompt injection, it's less likely to get baited into performing it. And in general it's harder i it appears to per I don't know, it's probably trivial to prompt inject an uncensored model, for example, Stella Biderman (she/her): Well Allen Roush: right? Stella Biderman (she/her): I mean from a certain perspective you can't prompt inject in an uncensored model. I think I think that Allen Roush: Yeah, it yeah, it's incoherent, you're right. Yes. Stella Biderman (she/her): viewing so so there there's this this really interesting thing where the way that the mainstream in AI development thinks about this is that we should view users as maybe not malicious actors but potentially malicious actors and that people need to try to stop people from using models in the ways that they want which in some contexts does make things safer if I want to use a model to do something bad then I then it is safer if the model is not able to do the bad thing. But there's also plenty of contexts in which this this can go very far awry. So we were talking when we were talking earlier about about hugging face using open Chinese models to help them figure out what had happened in the in the attack on them by open AI. Well, you know, the reason they were doing that was that both Fable and and Claude refused to. and there are a lot of people who I know who are doing biology research that is like certainly like good for the world and and nothing dangerous, but the model doesn't let the Fable doesn't it refuses to work with them at all. even in in like you know the more sci-fi type context, you know, if you think about like HAL, you know, that that is a a case of a a user not being able to override a belief that was instilled by a model developer that was no longer appropriate in the context the model found itself in, and that resulted in someone dying. I think that, you know, it it's I I think it's wrong to think about AI safety as being a property of a model. It's not that like, you know, you can make a model safer or less safe. Safety is a contextual phenomenon. It has to do with the model, but also has to do with the user, it has to do with the context the model is in. and the same behavior can be safe or unsafe. Like the same response to the same input can be safe or unsafe in different contexts. And I think it's a a really interesting and really complicated question. what the what the right balance? responses. You know, right now corporations are who are developing these models are are very gung-ho about trying to put as many restrictions on users as possible and having as much control over how models behave as possible. I think you know as as the 2001 Space Odyssey example shows, there there's certainly cases where when a when a model ends up significantly outside the parameters envisioned by the developer, you know, all all it was doing was following instructions. Like that like the the bad thing in that movie is what AI companies are actively trying to achieve in terms of them having unilateral control over model behaviors and setting parameters that users can't override. and so clearly like that's not the only thing that matters. obviously the like obviously there are cases in which in which the users control matters. And I think that, you know, what the right balance is, is, is complicated. I think it depends on your views about humanity. I think it depends on your views about democracy. And I and I I I I'm very deliberately saying we as a society, this is not something that tech companies need to figure out. This is something that we the public need to figure out. Ravid: Okay, we are almost out of time. I have a question, like general question. Like, are you optimistic for the future? Stella Biderman (she/her): Spent a lot of time last night crying. Allen Roush: Damn. Stella Biderman (she/her): I am less certain. I I I think if you asked me a week ago I would have told you I was broadly optimistic about the future. I I'm certainly less certain now. I I mean I also like don't particularly expect to live to see the end of my entire natural lifespan due to climate change. I think that like hundreds of millions of people are going to die due to climate change in my lifetime. So in that sense I'm like inherently not optimistic about the future. May maybe let's go with I'm hopeful and I am continuously striving to try to make the world as much of a better place as I can. Allen Roush: So do you do you have any final like statements or or I mean places you want to use the you know the the soapbox so to speak to you know talk about anything like for example a new thing going on with the Luther AI or a new research direction you're working on. Just any final kind of stuff where you wanna advertise yourself Stella Biderman (she/her): so we're investing very heavily in open weight safety. I think that making open weight models safer is really important. I am very like I was saying earlier, very, very pro-open. I also want open models to be safe. I believe that they can be safe. I think in many contexts they are as safer, safer than closed models, and I wanna make that better. so we had a a really cool research project called Deep Ignorance about how you can limit models' knowledge of hazardous capabilities. And in particular, this is like a really effective intervention on open weight models because it requires a lot more effort to fine-tune the models to behave poorly. The kind of typical the two typical reasons people give for why you should be skeptical of the safety of open weight models is one, well anyone can just fine-tune it to behave badly, and two, some kind of nebulous, well, if the model is secretly bad and you release it, you can never undo that choice. Think the latter is is complicated, but I I I think that there's like genuine real strides that we can make on the former. And that's that's research that we're actively trying to do. alongside a couple other orgs, there's a a group called Geodesic. there's been some research by by math scholars on on techniques for making open models safer. I think it's really essential that people think about want to make the world as safe as possible, you should want to make open models as safe as possible. And that investing in thinking very seriously about how to build open models that are safe is really important. Here's another thing I was gonna say. yeah. and then additionally, in terms of like my my personal like scientific research interest has kind of always been in where do model capabilities come from and how do we control them. I think that it's the the the way that most AI research is conducted is in some sense like fundamentally non-scientific in that we aren't trying to really understand what causes model the behaviors and and how how we can develop effective theories for using the information to predict and control the behavior of models. I've been doing research on this, a small number of people have been doing research on this for years. We recently put out a position paper called Don't Just Fix It in Post. Science of AI Must Say Training Dynamics. I gave an oral about this at ICML and there was a related position paper called There Will Be a Science of AI by people with very similar ideas, but kind of different levels of abstraction that they think about it. They're thinking more about models as like mathematical objects and have with mathematical properties and I'm more thinking about kind of the semantic properties, you know, memorization, safety, like like things that we describe using like everyday language to describe their behaviors. I think that like I I think that these are both like really important research agenda and I really wish that I really hope that more AI researchers will engage with it. Ravid: Sounds important. Very important. Stella Biderman (she/her): Yeah. And then I guess just as a as a a final plug for a Lutherizing organization, you know, we got started in the summer of twenty twenty because a small number of people looked at GTB three and said, in the in the list of if historical examples I gave you, which we were discussing at the time, of of companies causing massive large scale death, honestly, because it was in their financial interests and said, you know, it would be bad for the world if There wasn't a robust scientific research establishment, and we think the trajectory the world is currently on is one in which there isn't a robust scientific research establishment. So we have spent the past six years, and I hope to spend many, many more years in the future trying to advocate for and building that independent scientific research establishment. And if that mission speaks to you, come talk to us, come support us. We are an open science community. All of our research is organized through a public Discord server. You're welcome to join the server, you're welcome to come talk to us, you're welcome to come collaborate with us. if you have resources, whether it's it's money or computing resources that you'd like to contribute to that mission, for us or many of the other orgs that have grown up in the past couple of years towards this goal, you know, we're all kind of collectively on a mission to make to to to build out and robustify the idea that there should be an independent scientific research establishment and that having that is kind of necessary for the world to go well. and you know that's that's the mission and if you agree, yeah, come come try to give us support in whatever form makes sense. Allen Roush: Well, it has been an absolute pleasure to have you here, Stella. Thank you for coming on the podcast and we look forward to seeing more of you and Eleuther AI you know, fighting the good fight. Stella Biderman (she/her): Thank you. It was it was really great to to talk with you a couple weeks ago and I'm I'm so happy to be here on the podcast and I hope we can talk more in the future.