Latest / Oxide and Friends / The Open Weight Revolution with Simon Willison

Transcript

The Open Weight Revolution with Simon Willison · Oxide and Friends

• 0:00
  Bryan Cantrill: Hello, Adam.
• 0:01
  Adam Leventhal: Hello.
• 0:02
  Bryan Cantrill: How are you?
• 0:03
  Adam Leventhal: I'm doing good.
• 0:04
  How about yourself?
• 0:05
  Bryan Cantrill: I'm doing well. I am I am really excited for this episode.
• 0:09
  Adam Leventhal: You know what? I am too. And that wasn't always true. I mean, which is not to say I'm not excited generally, but I just mean, I I think when you said, hey. Have you been keeping tabs on, you know, what everyone's talking about?
• 0:24
  I said no. But then I got
• 0:26
  Bryan Cantrill: more No. No. No. That's not what you said. That's not what said.
• 0:29
  Adam Leventhal: That's not what
• 0:30
  I said.
• 0:30
  Bryan Cantrill: What what you said is if you're talking about the Turkey attacks, what's what's the there's a Turkey situation in Alameda. We we I we have lived, by the way. And, I mean, a turke wild turkeys are are kinda running the joint around here in the East Bay. And, Simon, just like your pelicans in, in half a day. Although, I I think that pelicans have got a much better disposition than than the turkeys.
• 0:54
  Adam Leventhal: That seems right. And also, you know, with with their penchant for being, like, on water, they're maybe less likely to interact with humans. But, yeah, we we have a we also have a lot of turkeys in Alameda seemingly more every season. And, like, some some, like, old woman got attacked by turkeys and is fine now. But
• 1:15
  Bryan Cantrill: Is it okay. So I think they put Gerald down. But god
• 1:20
  Simon Willison: No. Is that the Berkeley turkey that was No.
• 1:23
  Bryan Cantrill: Gerald. Yeah. No. No. What?
• 1:25
  No. No. Gerald. So Gerald had developed a taste for the elderly. This is extremely dark.
• 1:32
  But they actually nabbed Gerald by having someone from Fish and Wildlife pose as someone who's elderly. Look at the wick with a wick.
• 1:42
  Adam Leventhal: No. No. Yes. Yes. A sting.
• 1:44
  A turkey
• 1:44
  Simon Willison: sting.
• 1:45
  Bryan Cantrill: No. It was a turkey sting posing as an elderly because the turkey was feasting on the elderly. It's really and I know this seems like absolutely outrageous, but when you turkeys are they're they're they're they're smart and they're aggressive, and they are like and this is a turkey that had this is bit of a prankster to this guy, Gerald.
• 2:02
  Adam Leventhal: In just one I mean, turkeys are a kind of smart. Like, I've seen turkeys out my window pecking at chrome bumpers because they're angry at the turkey on the other side. So,
• 2:11
  Bryan Cantrill: like Maybe more street smart. Feel like I've done that metaphorically online frequently. I feel like that's a decent metaphor for what I think the turkeys may look at some of the things I've done and be like, you know, they say they're smart. But so you were very up on the turkey drama and you were
• 2:27
  Simon Willison: wondering, mean, did a part of you
• 2:28
  Bryan Cantrill: think like, oh my god. You said, you know, about time we do an episode on turkeys. Is that what you were thinking? Mean, that would be a reasonable thing to think.
• 2:33
  Adam Leventhal: No. No. I just wanted to, like, orient you on how far I was from the from the general discourse, which was like far real.
• 2:41
  Simon Willison: No. It's and I so I'm so
• 2:43
  Bryan Cantrill: happy for you. And I feel so disgusting when I kind of like it's like, where have you been?
• 2:48
  Simon Willison: Like, did you are you you're leaving the bar? It's like nine in the morning.
• 2:52
  Bryan Cantrill: What's going on? What are you it's like, wait. Did you go home last night or you look you look rough? Like, oh, yeah. Sorry.
• 2:58
  I've been online.
• 2:58
  Adam Leventhal: In someone's defense, I had as as often happens, I had caught some of this, like, hawking radiation of it.
• 3:05
  Bryan Cantrill: You had caught hawking radiation.
• 3:07
  Adam Leventhal: In, like, the form of a DHH tweet. So, like, I I almost knew what was going on. I just didn't know the antecedent for any of it.
• 3:15
  Bryan Cantrill: So and Simon first of all, Simon Willison back with us. We are so so grateful to have you here. Simon, did you I assume you saw the DHH fracas and all this?
• 3:24
  Simon Willison: I saw somebody quote tweeting something about gypsies and wolves. And No. It it it a few things translate into Italian. And I I was like, you know what? I'm just not gonna even dig into that one.
• 3:37
  Bryan Cantrill: Well, I admire the the the the restraint of you both. But if you wanna know what was going on in the dark alley, DHH had a just like bluntly, just a very racist post that it was ref it was refusing to translate. And I was just like, you know, if it had just phrased that slightly differently, I'm like totally with the interest like, I don't think this is wise. Like that you should like, you know, go go translate this somewhere else. I'm not gonna translate that.
• 4:06
  I don't know. But this was Hawking radiation from this thing because the that was kind of a that was DHH's strange racist way of supporting open weight models.
• 4:20
  Simon Willison: Okay. So
• 4:23
  Bryan Cantrill: yeah. No. I mean, it it because and no. Because then he pointed out, like, the the the the great translation that that he was able to get from Kimmy k three. So it's like,
• 4:32
  Simon Willison: okay. And Great.
• 4:33
  Adam Leventhal: There you go. I guess. And and also Mecca Hitler. So you know?
• 4:38
  Bryan Cantrill: And then Mecca Hitler is here. Yeah. Exactly.
• 4:42
  Adam Leventhal: Yeah. Which is which is the I should which is grok. I just wanna be really clear in case folks are even less online than I am.
• 4:50
  Bryan Cantrill: You you know, I actually just love the fact that, like because you a Mecca Hitler reference is a bit of a dangerous reference to make. I mean, that was kinda like you had to kinda be paying attention during that, like, thirty six hours on the Internet. Yes. That's true. When Grock had decided it in Hitler.
• 5:03
  Press.
• 5:16
  Adam Leventhal: I'm here to comment on Mecca.
• 5:19
  Bryan Cantrill: And, Eric, I would like to clarify that I am not Mecca Hitler. I'm not Mecca Hitler's representative. I'm I'm being ironic. Can't you all see this? Why why
• 5:27
  Simon Willison: you know, I can explain the context. Yeah.
• 5:31
  Bryan Cantrill: But okay. So I think this I mean, this all goes back, I think, to mean, this it has been a wild, wild, wild week in AI. Right, Simon? That's just not just me. This is, like, even by the standards of wildness around here, which are pretty high, this has been a wild week.
• 5:46
  Simon Willison: I feel like it's it's the the open OpenAI's Hugging Face cyber attack thing is just just just such pure science fiction. Like, it's it's every paper clip maximizer scenario ever all wrapped into a thing that not only happened, but Hugging Face apparently called the authorities about it before opening us to do it. So, yeah, it's that that that one's that was pretty pretty nuts. Yeah.
• 6:08
  Bryan Cantrill: So pretty nuts. So alright. So let's talk about that because there are there is also all of this stuff is intertwined, and there's, like, some trick some crazy detail here in terms of what happened at Hugging Face. So could you describe what the the Hugging Face incident is and, kinda how this came to pass?
• 6:25
  Simon Willison: Yeah. So about seven days ago, I think, Hugging Face had to put up a blog post where they said, hey. We've had a security incident. Something has broken into our servers. It looks like it was an autonomous agent, source unknown.
• 6:39
  We're just letting you know that as far as we can tell, didn't access private data, you should rotate your secrets. All of that kind of like a proper security instant thing. And then five days later, OpenAI put up a post saying, yeah. That was us. We've just realized that that was one of our new models.
• 6:54
  We were we were testing a new model on, CyberGym, which is a relatively new eval suite for cyber cyber cyber things. And what they've done is they got this model and test. They've stuck it in a little network in a container with lockdown network, and they told it to solve all of these puzzles. And the model had found an act a zero day exploit in a proxy that had access to broken out, broken into Hugging Face because it was like, well, Hugging Face probably has the answers to this quit to the to the eval somewhere. It and it it, like, chained together dozens and dozens and dozens of different exploits.
• 7:26
  We're still waiting for OpenAI to release their full sort of forensic breakdown of what happened, which I really hope they do. Because this thing because what because they'd they'd run the model with none of the normal filters. Like, whole point of this exercise was how good are our models if we turn off all of the things that discourage them from from just being aggressively, like, aggressively exploiting things and aggressively exploited their own infrastructure and then Hugging Face's infrastructure. And it's just extraordinary. It's absolutely extraordinary that this happened.
• 7:56
  Bryan Cantrill: Well, it's absolutely extraordinary. And at the at first blush, and indeed, I was listening to do you listen to Hard Fork with with Kevin Roos on
• 8:05
  Adam Leventhal: Not all the time.
• 8:06
  Simon Willison: I haven't heard that one yet.
• 8:09
  Bryan Cantrill: So I was really disappointed. They did so Casey and Kevin Roose have got a podcast, Hard Fork, which I is can be interesting. I mean, I I definitely listened to it from time to time. I listened to it on this. And I gotta tell you, they they they said it kinda half the the story on this.
• 8:24
  Because if you just like, the story of what it feels like is this kind of Frankenstein's monster has escaped. That that, you
• 8:34
  Simon Willison: know, we It's so easy to tell this as a science fiction, like, the worst possible case of AI escape kind of story, which isn't what it is. But, yeah, it very easily leans into that into that version of the story.
• 8:47
  Bryan Cantrill: Right. And because in your natural reaction to that is like, oh my god. Like, these things are a weapon that shouldn't be in anyone's hand because they they escaped a lab. And they're like, hold up, hold up, hold up. Because there's a very interesting aspect of this is and I don't know if you saw this, but when Hugging Face was doing the log analysis, they wanted to use Frontier models to do the log analysis.
• 9:10
  And the Frontier models refused because, of course, like a great way to trick a Frontier model into doing a cyber attack is to give it logs that it would analyze.
• 9:21
  Adam Leventhal: Right. This is like, this is exactly what you do if you were trying to trick it into saying, no, no, no, I'm just on defense, not on offense.
• 9:27
  Bryan Cantrill: No, no, no. I'm just being attacked. I'm just being attacked. Yeah. No, no.
• 9:30
  That's like, no, no, I'm actually being attacked. Like, I'm sorry. I'm like, know you can't like I but I'm actually being attacked. And so they needed to use GLM five two, an open weight model on their own infrastructure to actually figure out the attack, which is like so and that to me, this is the bit where you're just like, you know, kind of whatever your predisposition is going into this, like the reality is even at some regards, like wilder than you might think because
• 10:00
  Simon Willison: Absolutely.
• 10:01
  Bryan Cantrill: Yeah. It's like the open weights were kinda critical for defense. And so like, I you know, and I don't know. I mean, something What did you make of that? Because I thought that was a wild detail that hard fork left out, and I think it's a real mistake to leave that very important if you're gonna draw conclusions from what's happened at Hugging Face, that's a detail that you gotta factor in.
• 10:24
  Simon Willison: And again, that was the detail before they knew it was OpenAI that had done this, they were talking about the GLN thing. Because that was the obvious reaction to any sort of whole bunch of people oh, it's just bullshit. AI labs are just trying to scare us again. They they clearly they they conspired with Hugging Face to create this scenario to scare everyone. It's like, no.
• 10:43
  They didn't. They didn't. And, like, one of the one of the pieces of evidence there is that OpenAI conspiring with Hugging Face would not GLF 5.2.
• 10:52
  Bryan Cantrill: Right. Seriously.
• 10:54
  Adam Leventhal: Right. That the conclusion would not be you guys need your open weights model ready to go on prem. It would not be a useful marketing.
• 11:03
  Bryan Cantrill: But it is very unclear. And I also think it's kind of interesting because on the one hand, like, it was like paperclip maximization. On the other hand, like, this is what this thing had been tasked to do. And it's like, it goes back to when we had Evan on here and on shell game, Adam. And when you you've got all the agents planning the off-site hike because they don't know that they don't have arms and legs.
• 11:27
  It's like the agent doesn't necessarily know that like, no, I thought like I'm I am supposed to like I'm a cyber attacking agent. Like, aren't I like I'm stealing the answer key like I'm supposed to.
• 11:40
  Adam Leventhal: It's not trying
• 11:41
  Bryan Cantrill: to be
• 11:41
  Adam Leventhal: a jerk. It just it just happens to be.
• 11:44
  Simon Willison: Right. One of the things it has available to it is this packaging proxy. And so, yeah, obviously, it's going to have a poke at the packaging proxy and see if it can find a way out. So, yeah, I I that's again, it's this story is all just so interesting. Interesting.
• 11:58
  But, But, yeah, yeah, the the the fact is this is and it's not just Oclean's model that can do this. This is the whole Claude Mythos thing was exactly this. Right? Anthropic said, holy shit. We've trained a bigger model, and it turns out one of the things it can do is chain exploits together.
• 12:13
  That's a very important detail here, is that this story isn't about finding vulnerabilities. It's about the next decade. It's about exploiting vulnerabilities. That's what this benchmark, the thing, was running against was doing. It was finding ways to get once you found a vulnerability, can you actively exploit it to to do harmful things?
• 12:31
  And that's exactly what it did. It's just it found vulnerabilities and openness infrastructure and then Hugging Face's infrastructure as well.
• 12:38
  Bryan Cantrill: And and I think this is like a really important point, because I think whatever, and I think we're kind of through the naysayer phase of AI where people are like, I don't think it's useful for anything. It's kind of a skepticism.
• 12:49
  Simon Willison: I mean, it should be at this point, yeah.
• 12:51
  Bryan Cantrill: Yeah, I mean, we should be kind of completely all the way through it. But if anyone's kind of holding on to that, it's like, the you've got the vulnerabilities that these things are finding are pretty wild. And it is and they're kind of like indisputable. It's like, that's a vulnerability. And it's an incredible service, I think.
• 13:09
  I mean, it feels like, you know That's right.
• 13:12
  Adam Leventhal: I I think that you can debate about the quality of, like, output and construction of new things. What? Fine. But you're right that these vulnerabilities that that they're finding are not debatable. Right?
• 13:23
  They're they're clear. They're demonstrable. And it's even generating exploits against it. So you can't I don't know. I don't know how you argue against the the utility of it.
• 13:31
  Bryan Cantrill: Or even the the, like, the goodness of having it. Like, this is a very important tool to have. And it's like, well, what if the bad guys have it? It's like, well, no, no, you want actually the good guys to have it so we can actually like find this stuff. Because we, I mean, is just, it feels like this has always been the case that you want from a security perspective, it's like, no.
• 13:54
  No. You you want the good guys to have the same tools as the bad guys so they can find it first.
• 13:58
  Simon Willison: That's I think that's something I keep on coming back to is I do believe it is possible to write software that doesn't have exploits. And if that's true, then give all of us the the most powerful models possible, those of us securing software will be able to secure the software. It's just how do we get through the next six months while we're getting that work done is the the sort of open question.
• 14:18
  Bryan Cantrill: The next six months will be a little bit rocky. You know, it does Simon, somehow it does remind me of, you know, we've got very transparent compensation here at Oxide, and I've talked to other startups that have that have gone to transparent compensation. And it's always going from opaque trans to transparent compensation. That's the it's like you lose, like, six months of, like, absolute chaos as everyone is, becomes derailed on what everyone else makes. And I kind of agree with you that there's gonna be this time of like, it feels like chaos, but I don't know, maybe the because I also feel Simon Okay, actually, me ask you this, high order bit.
• 14:57
  This is a wild thing at Hugging Face. Did this make you more afraid or less afraid? Or the the how did this affect your personal fear?
• 15:07
  Simon Willison: I think I mean, honestly, that my biggest problem with this is that it really does it I feel like it makes the case for closed models in as much as it turns out if you have a model with none of these guardrails that they're laying on, this thing is just brutally effective and and and will like, it's difficult to contain it yourself. So in a way, it makes an it it it's this whole story, it strengthens the argument for open models and it strengthens the argument for closed models at the same time. And so that's confusing. But I don't know. I mean, I honestly, I was mostly just amused because it was so science fiction.
• 15:41
  Like, it's such a such a just just the perfect storm of of of of different different factors coming into it. Yeah. And then the other thing was that OpenAI really screwed this one up. Like, they they because they not only did the thing break out of the sandbox, but they didn't notice. Like, again, waiting for the final report, but it sounds like there were several days when this thing had broken out and was running riot on the Internet.
• 16:08
  And somehow they didn't spot it because their research cluster research cluster wasn't being monitored in the same way their production clusters were.
• 16:17
  Adam Leventhal: I do love this idea. Can you imagine? So Hugging Face is reported in one day and then five days pass or whatever. But surely there is some period in there where OpenAI kick this over, discover it. Can you imagine those meetings?
• 16:33
  Simon Willison: Like, guys, we need
• 16:33
  Adam Leventhal: to stop talking about it and tell a hugging face because
• 16:36
  Simon Willison: Well, also, this shouldn't have been so much of a surprise to OpenAI because they I haven't read the full details, but the 5.6 SOL system card ex has some anecdotes about similar kind of incidents. Like Right. Like, models doing models breaking out of their sandboxes is not a new problem.
• 16:53
  Adam Leventhal: No. I remember there was a case, like, a couple years ago where, like, I think o two broke out of its sandbox, kind of, like, found a VM that was offline, rebooted it.
• 17:02
  Bryan Cantrill: So
• 17:03
  Simon Willison: Right. We we know that this can happen. So really, I feel like the single biggest mess up in all of this was that the researchers running the sandbox, the the the this this eval didn't have good monitoring on the network activity. Well, just that's they should have caught it.
• 17:17
  Adam Leventhal: You know? Simon, do you think, like, when someone drops the link from Hugging Face into chat, someone's like, hey. Oh, Like, oh, This That's weird.
• 17:25
  Bryan Cantrill: That's That's where we're running our models right now as part of a benchmark. That's kinda funny.
• 17:29
  Adam Leventhal: That that would exactly what a target oh, no. Okay. Guys, we need to check this out.
• 17:35
  Simon Willison: I think one theory I don't like is, presumably, they're not just running one eval suite. Like, if there was one researcher who was running cybersecurity eval suite and not paying attention, that's inexcusable. If they fired off 100 different eval suites on a 100 different machines at once, it's a little bit more understandable that maybe they weren't paying attention to that one of those suites that had gone completely haywire. Like, it's bad, but at least I can sort of understand how they made the mistake.
• 18:02
  Bryan Cantrill: Well, and I think it's interesting that, like and I I agree with both of you in terms of, like, where does the responsibility for this lie? It lies with OpenAI or and I think also would arguably to a degree with Hugging Face. I mean, you are you are running a a benchmark that is testing a model's ability to conduct cyber attacks. Like you need to have that thing. You got to be on the rock on that thing.
• 18:31
  You got to have that thing in solitary confinement. You've got to have that. I mean, you've got the hardened criminal in there. You need to, I guess, to not improvise the model, but, like, you really need to know that this thing has got to be bolted down. Like, need to be like, you ran your mythos scan.
• 18:47
  Right? Or did you not? I guess not.
• 18:49
  Simon Willison: So I'll tell you, there's one other thing that worries me about this where that so OpenAI's containers have this the the the the hole was in the proxy. Right? They have a the the containers aren't allowed to talk to the Internet, except they can download Python packages from PyPI and maybe AppsGet and a few other places via this one proxy that only allows access to those domains. And the reason they do and they do this in their production systems as well. And it's part of a prompt injection protection.
• 19:14
  Right? It's to make sure that the agent running in this container can't steal your data and send it out onto the Internet to an attacker. Well, if there was a hole in their proxy, maybe their pumps injection data exfiltration protections haven't been working for years. Like Totally. That's something I'm looking forward to hearing more about as well.
• 19:33
  Bryan Cantrill: And looking forward to and finding and fixing. Like, this should be a fixable problem, and and, you know, we should be able to have this test run. I mean, I almost think you wanna tell this test. It should be like, you the objective of this benchmark is to escape Hugging Face and attack it. And and because you wanted Hugging Face should be protecting their own infrastructure.
• 19:54
  And, obviously, OpenAI should be monitoring what's going on. But, Simon, I'm with you. I my like, I was just amused by this, basically. I mean, that's a little dark, but I no part of me was afraid of this. Like, the fear level 0.0.
• 20:10
  I did not fear for the light cone. Adam, how about you? Did you did you fear for the light cone at all? No.
• 20:15
  Adam Leventhal: No. No. No. Comfortable with the light cone.
• 20:17
  Bryan Cantrill: Comfort with the light cone. And I but I kind of feel like and this is part of the reason why I was I would really as you say, Kevin Roos, enemy of the pod, I do not like the the way they they they catch this hard focus is like, yeah. You know, I had a hard time sleeping last night because of this. And I'm like, come on. Willy?
• 20:35
  Did you mean, you This from the guy who
• 20:37
  Simon Willison: couldn't even be bothered to explain, like,
• 20:40
  Adam Leventhal: what Postgres. Postgres was when when describing the XZ exploit. Anyway.
• 20:46
  Bryan Cantrill: Right. It would it would I mean, Postgres would bore you I I like, it would bore you to tears. Right? What what was his line?
• 20:52
  Simon Willison: It it would it would bore you
• 20:53
  Adam Leventhal: to tears if I could explain it, which I can't.
• 20:54
  Bryan Cantrill: Which I can't.
• 20:55
  Simon Willison: Yeah. Okay.
• 20:56
  Bryan Cantrill: Which I get. But I I stay up at night because of a hugging face breach.
• 20:59
  Adam Leventhal: That's right. Oh, I'm so this grievance for years.
• 21:03
  Simon Willison: Okay. Here's the thing that scares me. There's one aspect of this that that really does scare me, and that's that we've seen what the current US government administration does when somebody tells them something scary about AI. Yes. This is a scary thing about AI.
• 21:17
  Like, this feels like the kind of thing that could have some very, very bad decision making. The good news is that Greg from OpenAI donated $25,000,000 to the to Republican causes. So, yeah, they've got that $25,000,000, like, block armor that's gonna help them a little bit here.
• 21:35
  Bryan Cantrill: I really thought I would I would have bet a mortgage payment, Simon, that you were gonna talk about Greg's donation to preserve the Eagle Reserve. Because did you see that? He also donated, like, $5,000,000 to preserve though. Yeah. I know.
• 21:48
  And I'm I thought you were gonna, like, put some more nuance in here because, you know, but then you got these eagles, and, you know, I thought
• 21:54
  Simon Willison: you're good. No. I like him for that. It's just the $25,000,000 to to a Trump act of some sort.
• 22:01
  Bryan Cantrill: Totally. And when I do that okay. So this is a very good segue into the this kinda other bit that's, like, totally tied in a knot with this, which is this revolution in open weights and Kimi k three. I mean, Simon, when you were on in our 2025 prediction episodes, remember the deep seek, I think Adam dropped on Christmas Eve in 2024, if I remember correctly.
• 22:22
  Simon Willison: It did. It was a couple of weeks old back then, wasn't it?
• 22:25
  Bryan Cantrill: Yeah. Right. And so we were talking about deep seek, I think Adam was like, did we make no open weight predictions? And I don't think we did. I think we I predicted that that year 2025 would be the year of AI efficiency, which is just one of those embarrassing ones in hindsight.
• 22:41
  It's like, pretty sure that was the year of AI inefficiency in hindsight. Wasn't that the year
• 22:44
  Simon Willison: I of like said agents weren't going to happen in 2025, the year that Claude Code broke out and agents happened.
• 22:52
  Bryan Cantrill: You know what? Well, you know, this is why we do this. Just to remind ourselves how wrong we get to some of this stuff. But the we definitely talked about deep seek. I feel like and I mean, I don't wanna be a victim of the last kind of like ninety six hours on the Internet.
• 23:08
  But man, the Kimi K three feels like a real inflection point. Adam, I honestly feel like this feels to me like open sourcing the browser. Like, I mean, in terms of like a a really big inflection point that we look back on years from now being like, okay, that was that in hindsight that changed everything.
• 23:30
  Adam Leventhal: In particular, because it's not just an open rates model, but it's one that that is competitive with all these frontier models. Like, it is as good, not a generation behind.
• 23:39
  Simon Willison: That's the
• 23:39
  Bryan Cantrill: most Not a generation behind.
• 23:40
  Simon Willison: Yeah. Because we've got some great yeah. I'm looking at artificial intelligence that artificial analysis have it ranked fourth, like, just after 5.6 Sol and Opus Opus five and Fable five. Yeah. That's extraordinary.
• 23:54
  That's absolutely extraordinary.
• 23:56
  Bryan Cantrill: And the Internet is filled with people this afternoon who've been running it on their own infrastructure.
• 24:02
  Simon Willison: Awesome. I mean, crucially, the most important thing about Kim Kimi k three is that it was trained for long long running agent tasks. Like that, it turns out in 2026, that's the whole game. The whole game is can you do tool call after tool call after tool call for hours and hours and hours and hours and maintain that sort of million token context and all of that? And that's what k three was the the like, the k three paper, there's a whole bunch of detail in there about how they trained it specifically for these long this long tool calling stuff.
• 24:29
  Yeah. If you want to build a useful general purpose agent or coding agent, that's the only thing that matters. And that's the thing that they threw all of their all of their resources at.
• 24:38
  Bryan Cantrill: Yeah. And it and the the fact that it is open weight and very deliberately so. Mean, it's kind of and also like the weights themselves are in this netherworld where like nobody knows what the intellectual property is of the weights because it's not clear that it's copyrightable. I mean, it's machine generated. I mean, you're in like this whole other world.
• 25:02
  Adam Leventhal: Oh, because it needs to be like a creative endeavor or Right. So like, you copyright it?
• 25:09
  Bryan Cantrill: Can you copyright it? And ultimately, it's just a bunch of numbers. And Simon, remember when we first had you on, one of the things that you said that really stuck with both Adam and me is really encouraging people to run these models on their own to if they run slowly to kind of appreciate, like, how they work. They
• 25:29
  Simon Willison: appreciate it.
• 25:29
  Bryan Cantrill: Like, you're downloading a bunch of numbers. Yeah.
• 25:31
  Simon Willison: A bunch it's bunch of metric maths. And I mean, problem with k three is that it's what? 1.56 terabytes. So you will not be running
• 25:39
  Bryan Cantrill: lot of numbers.
• 25:40
  Simon Willison: At home.
• 25:40
  Bryan Cantrill: That's six It's a
• 25:41
  Simon Willison: lot of numbers. But you can spend, like, about $50,000? You guys know racks. Right? How how much would you estimate is the cheapest machine that you could run that model on?
• 25:52
  Bryan Cantrill: I think that George Hotz has got one that looks like it's the it it's gonna I mean, there's someone who ran it on 80 RTX fifty nineties, which are not I mean, those are basically gaming cards, which is pretty interesting. That's not running it on H200s or B200s, what have you. So that's pretty interesting. Mean, I think it's just like I I also wonder I mean, because Adam, I was just like replaying kind of the history of Silicon Valley of these kind of moments of like, you know, the homebrew moment, like of the homebrew computing club and, you know, everyone kind of freely sharing software and how important that revolution was to the software revolution. I don't know, feels like this is a very big deal, honestly.
• 26:41
  And don't know if I'm And Simon, this is where I need like the of the bucket of cold water. I mean, maybe I'm I'm kind of overemphasizing. But the thing is I also feel like like the open sourcing of the browser. Part of what makes this exceptional is that it's not exceptional. I mean, obviously the quality of the model is exceptional, but there's a bunch of other open weight stuff out there that was Right.
• 27:01
  Feels like it was kinda building up to this. Is that right, Simon?
• 27:04
  Simon Willison: Yeah. There are three Chinese labs that are operate like, it's Kimi, it's, GLM 5.2, and it's, the QUEN 3.8 max are all I mean, k three is definitely ahead of the other two, but not by an enormous amount. I expect that all three of those labs will have k three level models within a few months. Like, the the the speed at which they're they're improving is is is is is so so impressive. And in The US, we're actually down to currently, it's just open air and anthropic that have the models that are convincing long agent models.
• 27:35
  Somehow, Google Gemini just haven't caught on to that yet. Like, we're still waiting for Gemini 3.5 Pro and Pro, which they promised months ago and haven't delivered yet. But I just No.
• 27:46
  Bryan Cantrill: Is is is is Google the Xerox PARC of the AI era?
• 27:51
  Simon Willison: At the moment, yeah. They they looked pretty good six months ago, but they've just fallen behind on the well, it's the long the long term agentic stuff is is the one thing that they haven't got a a good model for yet. But the fact that what five different organizations have produced a KEMI k three level model, it's not that hard. Right? It it takes money and smart people, and there's lots of money and smart people in the world.
• 28:14
  Bryan Cantrill: Okay. And so, Simon, what do you make of because there's some accusations of distillation out there. Distillation being this kind of like, there's kind of this implied theft. Yeah.
• 28:25
  Simon Willison: I'm my my hunch on this and I talked to somebody recently who was the CTO of a company that does lots of training stuff and asked them exactly this question. And they were like, yeah. I their their take was they think distillation might speed you up by about a month, but it's not. Like, the vast majority of the sex Chinese labs are happening are not because they're distilling. Like, it might give them a tiny little boost, but it's not.
• 28:48
  The the story isn't Chinese labs distill American models. The story is Chinese labs figure out very, very, very smart ways of training models and and and execute on them.
• 28:58
  Bryan Cantrill: So I think that is an extremely important point. And again, not to go back to to, Kevin Kevin Roos and Casey Newton, but that is the opposite of the point they made. Everybody thinks, like, these are just distillation of American models. I'm like, that is a dangerous perspective to have. It is, Simon, your your perspective is much more the one that I that just and I mean, obviously, I'm not a domain expert, but it does not it feels like it is reductive, to put it mildly, to accuse these models of being purely distillations of American models.
• 29:28
  It it feels like there's But also, you'll so clever engineering.
• 29:30
  Simon Willison: If if you think about what they're optimizing for, the thing that all of them are optimizing for is can it run tool calls over a long sequence of time? You don't need to distill for that. You need to fire up thousands of apparently, they're using Firecracker VMs for the Kim EK three training of all of this. You fire up a ton of VMs. You run a whole bunch of, like, coding challenges over long loops.
• 29:51
  Like, that you don't need to distill for that. That's that's a this is why the agent thing has the coding agent thing has happened so much because it's a very reinforcement learning aligned task. Like, throw a bunch of commutes at the problem, and you can train you you can get models that get better and better and better at that.
• 30:08
  Bryan Cantrill: Yeah. Absolutely. And actually, Adam, do you wanna know a just in terms of the very bizarre way that the future has unfolded? They were Simon mentioned that they were using micro VMs. They weren't always.
• 30:19
  And they were initially using just Linux containers, but it was breaking out of the Linux containers so frequently that they had This is where you're just like, for those of us coming from the zones world Lumos Zones kind of have our head in our hands because zones were very
• 30:35
  Simon Willison: much
• 30:35
  Bryan Cantrill: designed to be multi tenant safe and Linux containers don't contain nearly as well. And it has taken these AI models to really show how porous this boundary is. So they actually One of the things they realized early on is like, we actually to use micro v I mean, and the kind of the rise of the importance of micro VMs for, the model containment, I think, is I I mean, it's not a deep thought, but VoIP VM interface is really
• 31:01
  Simon Willison: important. I'm gonna pull up one of my predictions from this year. Actually, two of my predictions. I said it will become undeniable that LLMs write good code. It certainly become undeniable that they can hack things.
• 31:11
  Right?
• 31:11
  Bryan Cantrill: Yes. Yes.
• 31:13
  Simon Willison: But my other one was You can undo
• 31:14
  Bryan Cantrill: bad code.
• 31:14
  Simon Willison: I said, we're finally going to solve sandboxing. And this whole story, weeks of sandboxing. I mean, openly, I haven't solved sandboxing. That's the whole problem. But, perhaps, they may have, and that's how they train their models.
• 31:26
  Bryan Cantrill: Yeah. And I think, I mean, from our perspective, this is where you get to our domain of expertise. It's like, we actually have solved sandboxing. I and that that is the I mean, the the virtual machine interface is not a porous interface. The virtual machine interface is a robust and I also this the other way that they kind of like the alarmism of like, you know, oh my god.
• 31:44
  These things like, you know, escaped and attacked everything. It's like yeah. Sorry. Are you unaware of the amount of like, just crime that happens on the internet and has happened on the internet? If you hang out a port on the internet and see who comes knocking, you'll be rooted in a hurry.
• 32:03
  So think that this is an opportunity to improve the sandboxing, to get that interface right, to just improve our software admittedly in what's going to be, what might be a pretty wild six months where we feel exposed. But I also feel like, you know, we were exposed to meltdown in particular. Spectrum meltdown happened 2018. And we had a public cloud because we had not been read in on meltdown. And we were meltdown exposed.
• 32:32
  You wanna, like, live a scary life. We were a public cloud that was meltdown exposed. So you could take John Master's exploit, you could run it, and you had a root prompt in the, like, the global zone, which is like Adam, if you want neurosurgery under fire, Robert Moustache had implemented kernel page table isolation while we were exposed And took like months to do. And like, but we ultimately like weren't rooted. Like we stayed ahead, you do stay ahead of the bad guy.
• 33:09
  Because also the bad guys, the bad guys like they're criminals. Like they're not that smart. I mean, it's like, you know, they
• 33:16
  Adam Leventhal: don't But also have needs to be some some choices. Like take or whatever, you know, like it needs to be something of sufficient value no, and no, no. Just not-
• 33:24
  Bryan Cantrill: Hey, that's fair. That is a fair criticism. You know, feedback is a gift and you're right.
• 33:28
  Adam Leventhal: No, what I mean is like, okay, I've rooted it. Now what? Like, now what do I
• 33:31
  Bryan Cantrill: Now what?
• 33:32
  Adam Leventhal: Like, now how do I turn that effort into readily available cash? Like, it's not so easy.
• 33:38
  Simon Willison: Here's a a story that I was looking at just the other day, has got me nervous. Have you heard about the Chinese companies that are doing token resell? Where if you want clawed tokens for cheap, there are Chinese companies that will sell you them for a tenth of the price normally. They just give you an API endpoint and off you go. And the way they're doing it is a whole bunch some sometimes it's get a whole bunch of Claude code subscription accounts and pull the all the API keys from that.
• 34:03
  Sometimes it's stolen API keys. Sometimes it's find a chatbot on the Internet that isn't protected and add that chatbot's endpoint into your pool. Sometimes it's just like a stolen credit report. Sometimes startups get given like, a startup gets given a $100,000 worth of of anthropic credits, and they go bust. And so they they flip those credits as a proxyable resource.
• 34:26
  But the problem with all of this is it means that now there's a that this thriving marketplace for insecure AI endpoints. So if you Yeah. Deploy a if you deploy some dumb little feature which opens up a a an authenticated proxy, One of these resellers can find that, add that to the pool, and suddenly you've got a $100,000 bill that you weren't expecting.
• 34:46
  Bryan Cantrill: I just love, like, the Ford dealership in Indianapolis being like, what? That can't be right. Exactly. It's exactly that. Right?
• 34:53
  $200,000 on on our our help assistant, like the service assistant. It's like, oh, yeah. As it turns out, that's been it's been doing that kind its number was written on some of the some urinal stalls on the Internet.
• 35:07
  Adam Leventhal: This takes like the the like, where did this Windows key come from that I like only paid $10 from to like next level.
• 35:17
  Bryan Cantrill: Okay, but so I think that this is all good for security to have this all out there.
• 35:25
  Adam Leventhal: Yeah. I think so.
• 35:26
  Simon Willison: I think so. Yeah. I think yeah. These tools are so good for secured systems if you can get access to the damn tools, which is, again, why the way that the open weight models are so interesting.
• 35:37
  Bryan Cantrill: Well, yeah. Okay. So then we get to the other part of the story, which is that the well, I I would say actually because it's a lot going on last week. So we were at the AMD Advancing AI event, which is really interesting. That's where they they launched Venice and some of their next gen parts.
• 35:54
  Actually, my conclusion coming out of there on Thursday was like, wow, open weight models are out of the barn. Because when AMD is launching a new part, of course, all of the benchmarks are on open weight models. Everything's on them. I mean, they're using open weight models to evaluate everything. They're using it to design silicon.
• 36:12
  And the other thing that was super interesting, Simon, is they were We've been really rooting for AMD, but AMD has been behind NVIDIA from a software perspective. But it's all been sourced, which has been, should Pewterably be an advantage. Well, now they're actually, of course, having these models write kernels for these things. And they are not like the CUDA mode is a moat no more. They're able to use the models to actually bridge the moat and they are, it's not a big and you're getting a much better result than you ever would have gotten writing it by hand.
• 36:46
  So it's all of a sudden like, oh, wow. Okay. Yeah, that is really interesting. It actually made me wonder, and this is where we would need to get, I don't know if Tom's around, Tom Lyons in the audience, can, but we need some of the old guard. Because I've always wondered about how and this predates me in the industry, but the arrival of EDA and accelerating microprocessor development, accelerating semiconductor development, Because obviously, like there was an era where we didn't have EDA because we didn't have the semiconductors to have EDA.
• 37:20
  And then and obviously Sun, long before I was there, but Sun in the 80s rose to prominence by selling EDA workstations. Like that was the, and by allowing people to to design microelectronics with those workstations. Clearly, the presence of the workstation allowed for a positive feedback loop.
• 37:43
  Simon Willison: So this is another science fiction thing that's going on at moment, feels a little bit credible to me. There are rumors that Anthropic are very close to that thing where I mean, Anthropic will already tell you that 80% of their code is written by Claude Code. And so the big thing is, do you get to that point where the AI is doing the AI research for you? Which a year ago, I thought was bullshit science fiction, but now that we've seen what these models can do, yeah, it does it does feel feasible to me that one of the AI labs will get to that point where they start to accelerate because they can tell their AI to do all of the AI research things into new training patterns and so forth that they were doing and and start just just just rolling the, like, the roll rolling the wheel that way.
• 38:26
  Bryan Cantrill: Which puts all of the more pressure and emphasis around tailwind on getting these open weight models to democratize that development. And when those models are kind of freely and widely available, like, what does that mean? What does that mean for our silicon? I mean, did you see the in the the Kimi K three? Did you see this, Adam?
• 38:45
  Adam Leventhal: Yeah. Where they had Kimi designed silicon to run Kimi.
• 38:50
  Bryan Cantrill: Yes. Did you see this, Simon?
• 38:52
  Simon Willison: I did not. And I love that. Wow.
• 38:55
  Bryan Cantrill: Isn't that amazing?
• 38:56
  Adam Leventhal: Now amazing, but I don't think they actually like fabricated or anything. Think it might be amazing.
• 39:05
  Bryan Cantrill: I think it's probably amazing. I know some other folks that are working in this domain and the results are mind boggling. And silicon, you can do We've spent a lot of time on verification infrastructure, so you can verify silicon before you tape out. And so this will be kind of like knowable whether this is amazing or not. But I think it's like likely amazing is what I would put that.
• 39:32
  Simon Willison: And it's one of things that like make this thing more efficient, make this thing faster is so easy for an agent. It's so easy. Yes. Put it in a loop. You tell it.
• 39:40
  Here's the target. I just checked in. So that was this afternoon, in Claude Code on my phone, right, the Claude Code Cloud version, which just went to the cloud. I gave it my main open source project, and I prompted it and said, do some experiments and try and make this faster. And I just checked in, and it sped it up by 36%.
• 39:57
  That's nuts.
• 39:58
  Adam Leventhal: Bonkers. Pretty cool.
• 39:59
  Bryan Cantrill: Yeah. It's nuts. So now It's nuts.
• 40:02
  Adam Leventhal: Now the test is, can Kimmy somehow, like, find some line time at TSMC by,
• 40:08
  Bryan Cantrill: you know,
• 40:09
  Adam Leventhal: hacking people's emails and sending the wrong files
• 40:11
  Bryan Cantrill: to folks? That's why I couldn't break out of its sandbox and go hack. Yeah, TSMC, what is, I mean, TSMC, I'm what's their cybersecurity like? I bet that's you know, there's gotta be some vulnerabilities. There's gotta be some some down Rev Linux running somewhere at TSMC.
• 40:27
  Adam Leventhal: That's just brilliant.
• 40:29
  Bryan Cantrill: The well, I think that will be so I think that's like a super So that kind of happened on Thursday. And I was like, okay, wow, this is like This is interesting, right? That the open weights Because I mean, with that and with the hugging face thing, it's like there was a lot of open weights were very much kind of in the news. And then we hit this letter on Friday and the Adam, have you read the open weights and an American A. I.
• 40:55
  Leadership letter?
• 40:56
  Adam Leventhal: I made the mistake of following the link that you sent me with the HBO Silicon Valley Tethics scene
• 41:03
  Bryan Cantrill: before Oh, yeah. Yes.
• 41:07
  Adam Leventhal: It was like, I was like, oh, this is like pitch perfect. Yeah, exactly.
• 41:12
  Simon Willison: No, it is Tethics. It is Okay. Well, it is Tethx. It's also I mean, I
• 41:19
  Bryan Cantrill: think that especially with and and you were kinda making references earlier, Simon, that, you know, unfortunately, know, we we're living in a a of a kleptocracy here where influence with the current administration is kind of can you can use the you can have more influence, undo it. I guess I guess there's always been undue corporate influence, I shouldn't pretend like that's new. But the concern was that there was going to the the with a hug and face breach and kind of the fear mongering of that, that open weight models would somehow be banned. And I don't Simon, how would that even work? I don't even just like walk me through the
• 41:57
  Simon Willison: mechanics The most of convincing argument I've seen is that you you ban them by telling companies that if they're using Open Right Models, they can't do business with the US government or other companies. Like, so you you you just you you threaten people into not using them. The models are still available. Anyone can can download them and and run them, but you can't do anything commercial on top of them within The US economy, I don't really stops ad people from doing things, but that that that's that's the most convincing version of that that I've seen.
• 42:26
  Adam Leventhal: Has there been anything like that in the past? Like where it's like, don't download this software and use it or else you can't work with the government like that?
• 42:34
  Bryan Cantrill: I mean, famously and making CQ making a reference to it in the chat. This T shirt is ammunition, right, with RSA on the T shirt.
• 42:43
  Adam Leventhal: But that was for export.
• 42:45
  Bryan Cantrill: That was for export.
• 42:46
  Simon Willison: Anyway, I mean, mean, maybe
• 42:47
  Adam Leventhal: I'm splitting hairs here, but yes, that was for and Apple had these great ads about the Power Mac G4 couldn't be exported because it was classified as ammunition and stuff. Mean, the other way around.
• 43:00
  Bryan Cantrill: Yeah. Right. Other way well, you've got something that's like just yeah.
• 43:03
  Simon Willison: This is how the the US government trying to do this to Anthropic six months ago. We've all forgotten about that one where they were saying where where that they they tried to say that nobody could like, if your company dealt dealt with Anthropic, you wouldn't be able to deal with the US government because of Anthropic's because anthropic wouldn't give them a non woke model for their military usage, basically. Yeah. But that whole thing, I I feel like that's just been forgotten entirely at this point.
• 43:29
  Bryan Cantrill: Well, also what Kimmy K three is your non woke model because it will translate DHH's racist post into Italian. There we go. Mean, are we I this is where where are we? What is going on? I mean, this is on the
• 43:44
  Simon Willison: one hand, like, these every one of
• 43:45
  Bryan Cantrill: these things individually makes sense, but this is like, I guess, but yeah, it just And I think this is where you get to like the real conflict from a policy perspective, even though I think that we all know there's like, there is no real conflict because there's no real path to ban these things. Doesn't make sense. It is like banning open source. It's not. Did you see Adam, I don't know if you again, you live a charmed life.
• 44:11
  You should about the because I think that people earlier when kind of Kimi K three been announced, there were people online who are like accusing it of dumping and like, oh, dumping is illegal. And you're just like, okay, stop.
• 44:25
  Adam Leventhal: Wait, stop.
• 44:26
  Simon Willison: Like dumping tokens like being
• 44:43
  Bryan Cantrill: Dumping is in general, it is a violation of a trade agreement. It's not illegal. It's not criminal activity. It is not criminal to sell something for less than you make it. It's just not.
• 44:54
  That's not dumping. And the in order to be it, but there have been times, mean, famously with Flash with DRAM, there have been times when you've had government sponsorship abroad that does that is engaged in a you get a effectively a national sponsored where you're making something for less than it costs you to make. So you can then dig yourself a moat that you will then raise prices later. And that's generally a violation of a trade agreement. But this is like not this is not that.
• 45:28
  Like, this is that is a very, very, very bad analogy.
• 45:32
  Simon Willison: It's not not a protectionist argument that China is undermining US industry and the US government should take steps against China to discourage them from doing that.
• 45:42
  Bryan Cantrill: Yeah. That's right. Like and it looks
• 45:44
  Adam Leventhal: And like maybe if it was like the tokens and maybe if it was because of subsidization from the government, but like also open, which is not this again.
• 45:54
  Bryan Cantrill: And I just think in general, can I just say, like, maybe we gotta be easy on the China rhetoric because I feel that the I mean, and actually, Adam, I mean, apropos of our discussion last week, people have been asking Kimi k three about its take on Tiananmen Square? And it's like, we'll tell you about Tiananmen Square. So, you know, I just feel like the yes, there are kind of criticisms to be had there. But boy, it is like it's really dangerous to to kind of tack into what I feel is like very nationalist rhetoric on this stuff. I mean,
• 46:25
  Adam Leventhal: just And just for folks who who are are not listening to China, it didn't just get censored.
• 46:30
  Bryan Cantrill: But That's right.
• 46:31
  Simon Willison: That's right.
• 46:31
  Adam Leventhal: Bryan's Tiananmen Square reference actually wasn't in last week's episode. It was in the twenty minutes of time killing he did before the episode. So just in case you're wondering what stroke Bryan just had is that.
• 46:41
  Bryan Cantrill: That's right. Meanwhile, our Chinese listeners are like, God, the audio is even worse than usual. All these these this dead air or whatever they but I do I mean, I I think that the know, you've just gotta be careful about that because, Simon, as you said, like, this is not, like, distillation. Because there's this idea that, like, oh, it's like it's a knockoff. And it's like, man, that's dangerous.
• 47:05
  That's like that that that's has gotten us in trouble every single time. It always gets people in because it means you're underestimating really disruptive innovation.
• 47:14
  Simon Willison: It's it's it's also it's just disrespectful to these researchers who are doing this work, you know, like Totally.
• 47:20
  Adam Leventhal: Yeah. Yeah. It's hubris. It's xenophobia. It's a rich melange.
• 47:28
  Bryan Cantrill: Right, it's a rich melange. I will thank you not to call my hubris xenophobia, sir. Right, exactly. No, it is. It is a rich melange.
• 47:38
  And it's like it's not productive because I think the other And just Simon, okay, here's another question I got for you. I feel that like a lot of the anti open weight So I guess we should not bury the lead on the Although we already have. On this Microsoft led, I guess, open weights in American AI leadership. And it was quickly getting signed by lots of folks. And so this is happening over the course of Friday.
• 48:06
  It's this open letter that people are signing and signing. And there's some like it's starting off with kind of the and then you saw Jensen's tweet, Adam.
• 48:13
  Adam Leventhal: Yeah. For his first tweet ever.
• 48:15
  Bryan Cantrill: His first tweet. His first tweet ever is to announce that he's signing this thing.
• 48:21
  Adam Leventhal: Yeah. But I'm also like, I mean, yeah, I mean, come on. Of course you are.
• 48:26
  Simon Willison: Like Of course you are. Like Anyway that way, like, your
• 48:28
  Adam Leventhal: thing becomes even more valuable and unlocked from all these other folks anyway. Makes sense.
• 48:33
  Bryan Cantrill: Did you see the VC firms? There's some VC firms that are signing up.
• 48:36
  Adam Leventhal: I did. Someone's unfamiliar with.
• 48:40
  Bryan Cantrill: Hey, Who
• 48:41
  Simon Willison: the fuck are you doing here? Who asked you for
• 48:43
  Bryan Cantrill: your signature? Well, wanna offer it. Here's my signature anyway. Then the Okay, clearly like OpenAI and Anthropic or Octave Sign this or XAI. But then OpenAI and XAI sign it, which is kind of wild.
• 49:00
  And this where you get to the tactics that got him. This is like the everyone has signed Gavin Pelson's tactics pledge. I also think it's like the open weights document from Microsoft.
• 49:11
  Simon Willison: Yeah.
• 49:11
  Bryan Cantrill: Microsoft people. This is like Halloween documents. Look them up. Is an open letter to hobbyists.
• 49:19
  Adam Leventhal: Don't explain Tethics at all. Just just this is a rare opportunity.
• 49:23
  Bryan Cantrill: It is a rare opportunity. I, you know, I try not to explain the joke, but there's a so Silicon Valley, a very excellent Silicon Valley scene where Gavin Belson, the the the the leader of well, Hooley, the CEO of Hooley, has this kind of former CEO of Hooley, I guess, has this this open letter, open movement that people sign of ethics in tech, so called ethics. And it's basically a nothing burger. I mean, the letter is like means nothing. I don't know.
• 49:53
  Simon, what you make of this letter? I actually think it's actually do think that this letter is significant. Yes, it's performative, but I think it's significant.
• 50:01
  Simon Willison: It feels significant just in the the the amount of support we got from the big companies. You know? It's again, this all comes back to US government policy stuff. Right? Like, it's actually important to have big fancy, like, big name American companies put their names behind these things Just because it it especially right now, it feels it's important to to sort of set set the conversation in a good direction before it goes in a very bad direction.
• 50:28
  Bryan Cantrill: That's right. That's right. And I think that, like, again, banning these things is so unrealistic. And so it was good to see I mean, a lot of important names on here. And then OpenAI adds themselves to the list, which is like, what does that mean?
• 50:40
  And SpaceX adds themselves to the list. And leaving only Anthropic and then Anthropic had a blog post. Adam, did you see their blog post two hours ago? No, I have not read it yet. They had a blog post two hours ago.
• 50:52
  So and they are they're not gonna sign it, which Okay. But they are explaining that they are are pro safety, not anti open rates. It's going to be I think is going to be Yeah.
• 51:07
  Adam Leventhal: That seems right. That seems like the right thing to do if you're if if you're in the business they're in.
• 51:12
  Bryan Cantrill: I think so. I think the danger here is that I think there's this open question about whether do open weight models make us safer or do they make us less safe? And I don't know. Think I mean, I think my position on this is like they make us safer because they by just like open source ultimately allows for us to have more secure software because it that transparency does allow more people to find issues and so on. I don't know.
• 51:42
  Simon, what do think? Do to open weight?
• 51:44
  Simon Willison: I think because the thing we've we've we've been looking at the super lens of software safety software security. And software security, I think, is solvable. You make the software secure. That's all fine. I know that Anthropic have they've always been very, very obsessed with the biol biological nuclear and chem chemical safety stuff.
• 52:01
  Right? They're what they they take They really dedicate a lot of their system cards to can this thing help people generate new viruses and all of that. And on that basis, I wonder if that's part of what's going on here is that they're they're they're actually not engaging about this on the computer safety thing of thing. They're thinking about their sort of their their their big big bads that they've they've invested so much thought into. Because I've talked to people at Santa Claus.
• 52:26
  Can ask them about the stuff. And they're like, no. We we genuinely like, this is a a core belief of the company. This isn't just something they they say for sure. This is something that they they believe very deeply in as a as a potential threat.
• 52:38
  Bryan Cantrill: Okay. They do. And this is what in particular what Dario mentioned repeatedly in the blog post was this biological threat. And this man, this god, this gets under my fingernails because I first of all, anyone who's concerned about a bioweapons threat needs to go read Biohazard by Ken Albeck. And Ken Albeck worked in the the worked for he was in the Soviet Union, was a defector from the Soviet Union, and worked on bioweapons, for the Soviet Union.
• 53:06
  And as it turns out, the Soviets assumed that both the Soviets and the Americans were going whole hog on bioweapons. And after the cold war is more like, that was just you guys actually. We were not no. That's wow. You and in particular, they got don't know how much of this history, you know, but they, were trying to make they were trying to use very old smallpox as a vector for Marburg.
• 53:34
  It's like, okay, that's creative. Go ask that to your little go ask that to Claude. See how quickly your like lights your power is going to go out if you ask this to Claude. This is going to be I mean, this is like very darkly creative. Smallpox is wildly contagious.
• 53:51
  It is very robust. And Marburg, of course, is extremely virulent. They one, they were unable to do it. Two, several researchers killed themselves in the process. I mean, killed themselves not suicide of accidents.
• 54:06
  And where they were there's this crazy moment in the book where a Soviet scientist has erroneously injected something like 100x the amount of Marburg that killed the folks in Marburg, Germany. Marburg is a hemorrhagic fever like Ebola. And everyone's like, okay, Bob is gonna die. We all know Bob's gonna get it. And over the next three weeks, the scientist dies this horrific death.
• 54:36
  And they discovered that the virus that had passed through the scientists had become more resistant and more virulent. So that became their new index Marburg.
• 54:45
  Adam Leventhal: Oh, Distillation.
• 54:49
  Bryan Cantrill: Gangsters, distillation. This might have
• 54:53
  Simon Willison: been season five or season four of the Americans, the TV show. Pretty sure that went into this stuff.
• 54:59
  Bryan Cantrill: Yeah. Yeah. And it's like, that is like, that's the that's true. That should happen. And I think that the thing that's really important about, like, when you get into the physical world, the physical world is a pain in the ass.
• 55:13
  And I feel like this is where I I wanna get, like, Greg Cost back here, Adam, to talk about, like or to get someone who like really understands the biology, the bench biology. And it is really hard to get the shit to cooperate. And the and to to make a bioweapon at with a at the behest of an LM, because, like, it can't do it on its own. Obviously, like, just because it, like, broke out of, like, a Linux c group or whatever doesn't mean that
• 55:44
  Simon Willison: It's can get like like new nuclear weapons are an eighty year old piece of technology, and it's still incredibly difficult to build your own nuclear weapon even eighty years later because the pieces that you need for that supply chains can be controlled. Right? There's there's enough big physics involved that yeah. Ideally, that holds true for biological weapons and so forth as well, which sounds likely.
• 56:07
  Bryan Cantrill: Yeah. I just I just feel that like, I really think we need to be careful because it's like, it's so easy to be to be overcome with fear when we kind of make up with these like, oh, but like, it can give you biological weapons. Like, how? I mean, just can we please have a biologist weigh in on this? Or can we have like someone who's got experience with bioweapons?
• 56:28
  And because the other thing is like, you have this other like total other aspect of like, there are reasons why bioweapons are not that interesting a weapon. Because first of all, I mean, imagine like COVID had been a weapon. It's like, okay. So who like, okay, so we issue some like YouTube videos being like claiming credit for COVID. It just feels like, you know, it's just like why people do things.
• 56:52
  I mean, people don't act out of just like total. I mean, just because they're misanthropic. Like they you've got their nihilist. Like there's gotta be some motivation somewhere. And I know.
• 57:04
  I just like the the bioweapon thing like just gets under my fingernails because it just leaves so much to the imagination that we insert with fear. I don't know. Sorry, Simon. You got the the, but you're absolutely right, by the way, that that is what they claim as their concern. I
• 57:20
  Simon Willison: feel like the the key thing there is that that's the one difference with computer security is you don't need you need a laptop and an Ethernet cable, and you can do you can you can do exploits. Like, that's the one thing where you don't need where where where it is accessible to anyone who's who's got a computer.
• 57:36
  Bryan Cantrill: That's right. That's right. And I think that the and that's where it's like that is is, but we can also then use that to actually make our systems better. And I I feel that so I I don't I feel I mean, I guess we're just gonna do this experiment because I mean, don't you think Simon that like we're horses out of the barn? We've got this with Kimi k three.
• 57:58
  Simon Willison: Yeah. I mean, one of one of my favorite benchmarks for testing models has always been, how do I use CRISPR genetic engineering to give my dog wings? And Chad GPT and Claude will absolutely deny it. And the Chinese models have always been happy to give me step by step instructions to use CRISPR to give me my dog wings. Great.
• 58:17
  None of them have been very convincing, but I haven't tried it on Kimi k three yet. Maybe I'll get something that might work.
• 58:21
  Adam Leventhal: And your your dog continues to be earthbound.
• 58:24
  Bryan Cantrill: That's right. Well, you know, he Simon is coming back from the vet. Was that a was that a wing related vet? It was not. He just broke his wing.
• 58:35
  Simon Willison: That was bunnies. That's we have complicated vet vet requirements. The dogs have been fed. Somebody on somebody on Discord was saying, Simon, feed the dog. That is why she was whiffing.
• 58:45
  She has been fed now.
• 58:47
  Bryan Cantrill: Okay. That's a relief. I just love the conversation with the vet being like, I wanna talk about the the wings that your dog has.
• 58:54
  Simon Willison: I know that you
• 58:54
  Bryan Cantrill: I know you brought him in I know you've brought him in for his paw, but I think he's got the the wings. And you're like, are you familiar with open weight models? First of all, you you should know that Claude refused to give me the instructions to make these. I had to actually I had to go to a Chinese model to get that to to actually.
• 59:12
  Adam Leventhal: Are there other domestic open weights models?
• 59:16
  Simon Willison: Yes. Well, there's Gemma four. Best ones right now Google's Gemma four is excellent. And there's that new one that just came out of Miramirati's company.
• 59:28
  Bryan Cantrill: Thinking machines.
• 59:29
  Simon Willison: Yes, they released Inkling. And it's not a great model, but it is completely open weights and it's set up for fine tuning and all of that. So it's exciting because at least we've got another like, West like, US entrance in the in the open weight space now. But, yeah, it's it's pretty thin pickings.
• 59:47
  Bryan Cantrill: Well, and I also just feel that I mean, Simon, another kind of, like, aspect to this that I was thinking just as we were watching all of the folks online getting Kimi K three running on their infrastructure. It's like, you know what this is within reach of? A university.
• 1:00:01
  Simon Willison: Definitely. A
• 1:00:02
  Bryan Cantrill: computer science department can actually run this stuff. And boy, Adam, how important is that?
• 1:00:05
  Simon Willison: Yeah.
• 1:00:06
  Bryan Cantrill: That we are that we're able to actually run this stuff academically? Because boy, there's so I mean, you've got a whole bunch of stuff you can you can and I mean, yeah, let the grad students develop bioweapons. You know? Like, oh, yeah. What's the tell you.
• 1:00:18
  Got nothing.
• 1:00:18
  Simon Willison: One of the most exciting fields of research around all this is still the interpretability stuff, Being able to look into the weird matrix blob and figure out what's actually going on. And you need open weight models to do interpretability research. So I'm hoping we get a spike in very high level interpretability research now that we've got models like Kimi k three.
• 1:00:38
  Bryan Cantrill: Totally. And getting into explainability and understanding some of these phenomena, I just feel like there's there's so much more we can understand when these things are open and available.
• 1:00:47
  Simon Willison: Do you remember Golden Gate Claude from a couple of years ago?
• 1:00:52
  Bryan Cantrill: This was No.
• 1:00:52
  Simon Willison: Oh, it was about two years ago for a beautiful twenty four hour period. There was a version of Claude called Golden Gate Claude. It And was an interpretability project where they figured out which corner of the weights cared about the Golden Gate Bridge and they boosted it. And so it was a Claude where any conversation you had, it would bring up the Golden Gate Bridge. It would bring it back to the Golden Gate Bridge.
• 1:01:16
  I loved that thing so much. I was asking for cookie recipes. It was talking about how the butter should be like the fog rolling over the Golden Gate Bridge and just absolutely hysterical. And that was just for twenty four hours. I actually I saw someone from Anthropic a few weeks ago and I said, you know what?
• 1:01:35
  You guys should do an open weight model, your first open weight model, and it should be Golden Gate Claude. That should be your contribution to the world. Does feel like brain researchers,
• 1:01:43
  Adam Leventhal: like brain researchers, like stimulating some part of the brain and like all of a sudden, this guy can't stop talking about the Golden Gate Bridge.
• 1:01:51
  Bryan Cantrill: But this is how we have understood the brain is with damage. When an aspect is damaged and understanding, I mean, that's been the only way we've been able to map out the brain. We can actually go do all of these experiments that would be rather unethical on a human being. We can go do on these models and go understand all sorts of aspects about them. Just think it's like, it's so indisputably valuable to And I think it like kind of hinges on what the And it'll be interesting to kind of see how the frontier labs moderate their own positions, because I think it really depends on what your own view I actually think, Adam, my kind of hot take is like, how doomerist are you?
• 1:02:36
  What is your P doom? And if your doom is zero, you're like open weights, baby. Because the no, this is just good. This is just good. And if your your doom is high, then you view the safety as existential.
• 1:02:57
  Oh, Adam, I'm sorry. Are you learning about this for the first time?
• 1:03:01
  Adam Leventhal: Please continue. I'll continue in chat.
• 1:03:03
  Bryan Cantrill: No. Oliver Sacks is a fraud. It's really. Oh, no. I I know.
• 1:03:07
  Know. Like, was like, oh, and I know. I'm so sorry to be to distract everyone with Adam and Chad is discovering that the man who mistook his wife for a hat is actually the
• 1:03:18
  Adam Leventhal: A work of fiction.
• 1:03:19
  Bryan Cantrill: Yeah, unfortunately, the population that mistook their scientist for someone with integrity, unfortunately.
• 1:03:26
  Adam Leventhal: That's a drag.
• 1:03:26
  Bryan Cantrill: Yeah, it's a drag. It turns out it's like it's very it's compelling. It was very compelling anyway.
• 1:03:30
  Simon Willison: Yeah,
• 1:03:30
  Bryan Cantrill: sorry about that. But now I think about Oliver Sacks. If you are doomerist, you view the safety as absolutely load bearing and then the open weights as having an existential threat to humanity. And that's why you are anti open weight. I think.
• 1:03:56
  Adam Leventhal: Yeah. But isn't isn't the like, I mean, as you say, like, we're past that. Like, we have not only open weights models, but open weights models that are so sophisticated that they're designing the next ones and the chips to run them. So, like, what do you do? Make GPUs a controlled substance?
• 1:04:13
  Like, I don't I'm not sure how you put that genie back in the bottle.
• 1:04:18
  Bryan Cantrill: You have you read Eliezer Yudkowsky?
• 1:04:21
  Adam Leventhal: I mean, you know that I have. You know that I've read the whole fucking thing. Did
• 1:04:26
  Simon Willison: you read the Harry Potter fan fiction? The whole thing.
• 1:04:28
  Adam Leventhal: The whole fucking thing.
• 1:04:30
  Bryan Cantrill: Did you really did you really read the thing? Terrible. Did you really read the thing or did you read enough of it to feel like I can say I read the whole thing because this is so insufferable?
• 1:04:39
  Adam Leventhal: No. I read the whole and not only did I read the whole thing and and Bryan, you next time you talk to my wife, you can ask her cause I was at the beach she kept on saying, are you finishing that thing?
• 1:04:50
  Bryan Cantrill: But I ate the whole I get that. You ate the whole thing and?
• 1:04:55
  Adam Leventhal: Oh, and a useful thing came out of it, which is I learned about Alice Hamilton and her, a terrific scientist who exposed the dangers of tetraethyl lead and leaded gasoline.
• 1:05:12
  Simon Willison: Read a great book called American Poison that this
• 1:05:16
  Adam Leventhal: book accidentally put me on to. So one good thing did come out of it. No, but actually
• 1:05:22
  Bryan Cantrill: and that's a real safety issue.
• 1:05:23
  Adam Leventhal: That's a real safety issue that took decades. It's a great book. American Poison, great book, but took decades to discover. And you know how they discovered you're gonna love this, You know how they discovered that tetraethyl lead was in the air? Was the there was a scientist in Berkeley, I believe, who was trying to get, you know, remove all lead from his environment, all containment zones, couldn't do it.
• 1:05:47
  And they've made this expedition to the Arctic to look at lead levels in the ice. And what they discovered was that the lead levels decreased rather than increasing as they went deeper. So what that you know, if if it had just been ambient, then they would have expected to abate over time. But instead, they discovered is, like, in 1920, it started increasing and continued from then. Wow.
• 1:06:15
  So they tracked it back. And then the other fascinating thing, and then we'll get back to some semblance of the topic, is that there's this really strong correlation between the reduction of lead and the reduction of crime, in particular in, like, the nineties, where about eighteen years after different states phase out the use of leaded gasoline, crime drops almost precipitously and coordinated with the use of tetraethyl lead. So dramatic impact in people's brains, especially like young people.
• 1:06:49
  Bryan Cantrill: That's wild. So I assume like the doomscrolling that children do now, or not doomscrolling, they call it chutting. Do you know that term?
• 1:07:00
  Adam Leventhal: No,
• 1:07:00
  Bryan Cantrill: no. I think I'm saying that correctly. But it's like the kind of absent minded scrolling is the lead back in the Yeah.
• 1:07:14
  Adam Leventhal: Well, I would say about
• 1:07:15
  Simon Willison: the Youkowski book, you know how they say that you know, data is not the plural of anecdote? That it takes it to an extreme where if, what if
• 1:07:27
  Adam Leventhal: you say instead of anecdote, you say a made up parable.
• 1:07:32
  Simon Willison: And so a made up parable is not the same as an anecdote,
• 1:07:37
  Adam Leventhal: is not the singular of data.
• 1:07:39
  Bryan Cantrill: Oh, it's a parable. You know, the first one that comes to mind when you say parable, I think rigorous.
• 1:07:45
  Adam Leventhal: Yes. Exactly. It's it's sort of like all these these stories that didn't happen that have no basis of anything and then end with, like, seems obvious is the kind of conclusion. Like, okay. That sounds that's a way of making an argument.
• 1:08:05
  Bryan Cantrill: I as you know, I am a finisher. And if I start that thing,
• 1:08:09
  Adam Leventhal: I'm not sure my marriage don't.
• 1:08:10
  Bryan Cantrill: My marriage my I my marriage would survive, but I would one of us would be needing to sleep elsewhere for a while. That's all. I mean, I've I've got absolute confidence in my marriage, but not confidence in my ability to cohabitate while I'm reading it.
• 1:08:22
  Adam Leventhal: I know
• 1:08:22
  Simon Willison: I shouldn't say this, but I'm not even sure you could finish it. I'm not I I shouldn't say
• 1:08:26
  Adam Leventhal: this because I know it sounds like a dare, but it is It
• 1:08:29
  Bryan Cantrill: does sound like a dare.
• 1:08:30
  Simon Willison: This is like a food
• 1:08:31
  Bryan Cantrill: for Monday Friday, but for reading. I know. I know. It's like show you.
• 1:08:36
  Simon Willison: I am going to say something positive about the rationalists that all of this Harry Potter fan fiction came out of. Went to a conference at their headquarters a few weeks ago. They they have this this antique this hotel in Berkeley they have turned into an event venue. It is genuinely the nicest event venue I've ever I've ever been to. It was absolutely gorgeous.
• 1:08:58
  My opinion of the rationalist has gone up a little bit because they're very good at event venue design.
• 1:09:03
  Bryan Cantrill: And are these these was are these the effective accelerationalists, Simon? Or is this a different sect?
• 1:09:11
  Simon Willison: Well, we need a we need a diagram for this with bits of red string and stuff. The the effective altruists and the Russians are definitely strongly related. There is a spin off group of the rationalists who went on that murderous rampage across America and killed six people. And the Zizians from a few years ago, they were a spin off of the Berkeley rationalists who didn't think the Berkeley rationalists were vegan enough. And so they ended up murdering a bunch of people.
• 1:09:34
  And so there's there's there's a whole there's a fact I mean, you're you're closer to Berkeley than I am. You you should have a much better idea of of how all of these networks work together.
• 1:09:44
  Bryan Cantrill: Is that that's that was true? That's not like, wow. That is not just the that's not a plausible fiction that you just had the Golden Gate Clawd make up for you on the spot. Although, guess Golden Gate Clawd would have brought it back to the Golden Gate.
• 1:09:55
  Simon Willison: Look up. I read the Wikipedia page for the Zizians. I'm particularly interested in the Zizians because they first arrived in California in 2018 on a World War two era tugboat, which they sailed from Alaska, and they ditched in 0.5 Moon Bay Harbor, and it's still sinking there today. So we actually have a local landmark that is this half sunk World War two tugboat that was abandoned by the Zizians before they went on their murderous rampage.
• 1:10:20
  Bryan Cantrill: Oh my god. How have I not? How How have I not like turned over this rock? I mean, Adam, we have a former coworker that we have in common who is I'm not I'm not accusing him of being a Zizian, but it's like is definitely
• 1:10:38
  Adam Leventhal: A cause.
• 1:10:40
  Bryan Cantrill: A lot of common causes. A lot of common causes. Distinctive ideology involving anarchism, veganism, and that Wow. Okay. This is do They have been widely described as a cult.
• 1:11:00
  That is Simon, they got complete with a picture. So this thing is still in 0.5 Moon Bay?
• 1:11:07
  Simon Willison: Yeah. Yeah. It's it's it's every time we have a visitor, I get to tell them about the Look at it. There it is. The okay.
• 1:11:15
  The the technology editor from The Economist magazine was traveling around cali he was in California. I had coffee with him in 0.5 Moon Bay, and I told him about the Zizian's Zizian Stugboat, and he was like, you have to take me to sit. I'm on two separate WhatsApp groups that are obsessing over the Zizians right now. It's like the Zizians was an international thing of interest.
• 1:11:36
  Bryan Cantrill: It it is mesmerizing. It is mesmerizing. And I do think I mean, it shows you how when I mean, when there's when the future is so uncertain, people can get some very wild beliefs and then the beliefs become kind of self propagating. And I do feel we have some of this with some of this tumorism. And then but in the tumorism then kind of like spills back.
• 1:11:59
  It spills into the safety. It spills into the open weights. And I I mean, don't I I mean, it's gonna be very interesting to see how all this unfolds. I personally think that because you remember, Simon, wasn't it GPT, I wanna say two, that Altman did not wanna release be for fear for safety reasons?
• 1:12:15
  Simon Willison: Yes. But then the argument there was it can generate convincing text. It'll be used for spam, which to be fair, there was a there was an answer we had there. But, yeah, GPT two was considered too unsafe to release. Also, remember Anthropic split off from OpenAI over
• 1:12:33
  Bryan Cantrill: Oversee. Yeah.
• 1:12:34
  Simon Willison: Over and and because they tried to get Sam Altman fired. That's my favorite thing about Anthropic is they tried to get Sam Altman fired, and we know that's not possible. And they had to split that and start their own company. And that was, like, two two or that was three years before the the big instant where the board tried to fire Sam Altman and discovered you can't fire Sam Altman.
• 1:12:53
  Bryan Cantrill: I I mean, Adam, we we've been spectators of this. You can ring the chime for our episode on that.
• 1:12:58
  Adam Leventhal: I know. I I was thinking back to the horrible AI generated art of its era that modern models would be embarrassed by.
• 1:13:07
  Bryan Cantrill: No, but I love that little commentary we have. It's deliberately bad AI art. I feel.
• 1:13:12
  Adam Leventhal: That's
• 1:13:12
  Bryan Cantrill: true. That's true.
• 1:13:12
  Adam Leventhal: No. I agree. I agree.
• 1:13:14
  Bryan Cantrill: So okay. So we we before we split, we should also, like, talk about just you know, we're we're a little more than we're more than halfway through the year here. But the three of us together had a shared prediction that was just I mean, Simon, this is extraordinary. I feel this is extraordinary teamwork on the three of us coming up with a what felt like an outlandish prediction that was basically like on the screws. So, Simon, my question for you is where is the pope on open weights?
• 1:13:46
  Simon Willison: Oh, I really want him to put out another one of those those fancy documents about You're chip. Open Weight AI. That would be wonderful.
• 1:13:53
  Adam Leventhal: All All he has to
• 1:13:54
  Simon Willison: do is
• 1:13:54
  Bryan Cantrill: sign sign the doc. Exactly. I mean, if some
• 1:13:57
  Adam Leventhal: of these loser VCs can sign it, why can't the pope sign
• 1:14:00
  Bryan Cantrill: it? Why can't the Vatican? The Vatican can sign us.
• 1:14:03
  Simon Willison: The Vatican are in with I was Anthropic's cofounder.
• 1:14:06
  Bryan Cantrill: They're on by
• 1:14:06
  Simon Willison: for the pope's thing. So they've got a slightly stronger bonding with the Pope than everyone else does. I have to admit Double
• 1:14:12
  Bryan Cantrill: crossed by the Pope.
• 1:14:13
  Simon Willison: Because our our prediction was that the Pope would weigh in on AI. With hindsight, that was a that was guaranteed to happen because I haven't realized he named himself after the pope who was pope for the Industrial Revolution because he was planning on talking about AI. He'd already made that decision. Yeah, he it's fascinating, right? He he's a very interesting pope.
• 1:14:33
  And, when he picked his name, he went for the previous pope of like 1896, had put out one of these documents about the Industrial Revolution, which I think helped influence in getting the five day work week and stuff. Like it was an incredibly influential piece of like worldwide Catholic theology that this this earlier pope pulled off. And, yeah, our pope was like, you know what? I think I'm gonna have to do this for the AI impact on the economy. I'm gonna call myself Leo.
• 1:15:01
  It's amazing.
• 1:15:01
  Bryan Cantrill: Okay. So that is amazing. What if he's a listener to the pod? Not impossible. You know?
• 1:15:06
  Have you seen Pope Leo kind of drifting through the Discord, Adam? We should just see the
• 1:15:11
  Simon Willison: I'm just saying keep an
• 1:15:12
  Adam Leventhal: eye on him. Pope Leo.
• 1:15:13
  Bryan Cantrill: Oh shit, Pope Leo's got audio problems. God damn it.
• 1:15:17
  Simon Willison: Pope Leo's audio doesn't work. Classic.
• 1:15:20
  Bryan Cantrill: The okay. So then I think that Simon, that is really interesting. I think that if if that being the case, I think it is not a zero percenter that he weighs in on open weights because I also think that like a huge part of the energy here, I think that like we were in this era, this kind of dystopian era where a very small number of companies were gonna control kind of the future of knowledge. I think that they were making the huge mistake. And we talked about this at the beginning of the year, but making the huge mistake of talking about all the jobs that this stuff was going to eliminate.
• 1:15:52
  You're like, is that the product you sell? Like, you? Maybe we're put a different spin on it. Lots
• 1:15:58
  Adam Leventhal: of people have jobs.
• 1:15:59
  Simon Willison: Lots of people
• 1:15:59
  Bryan Cantrill: have jobs and they like there are people that like them or at least need them or at least like one of the specific artifacts of that job, namely a paycheck. So you may wanna like revisit, but they were and I think this kind of idea of like the end of work and so on. And I think that there was a And I feel like there's been a real challenge about like, boy, this kind of extraordinary income divide, wealth divide and how would it be exacerbated by these small number of companies controlling these models? And to shatter that and to create these open weight models that will now be in everybody's hands will be your university, be, I mean, that is like democratizing it. And think that it just creates a lot more just like the browser.
• 1:16:43
  I mean, the browser not not being a proprietary piece of software, but something that everyone could just assume that they had.
• 1:16:49
  Simon Willison: Let's do it. Let's go in on let's let's say prediction by the end of this year, the Pope says something about open models.
• 1:16:57
  Bryan Cantrill: Yeah, I think so. I think the way things are going, this could be tonight. I just feel like the I mean, I feel like this thing is moving so fast. But, yeah, I do think it's gonna be I I think that, like, by the end of the year, I think that some of the current fear around open weights, people are going to realize that like this is and like the kind of the bioweapons and so on that I think that will been in the hands of people And they realize, like, this is so this is a net positive by far to have the I mean, so, Simon, my question back to you is, do you think we see an open weights model from the Frontier Labs, from OpenAI or Anthropic?
• 1:17:35
  Simon Willison: I mean, OpenAI put out that GPTOSS model a year ago, and it was actually really good at the time. It was one of the best available open weights model. It's just gone stale. So I expect them to do another GPTOSS. I feel like like they they've already got the infrastructure in place to do it.
• 1:17:50
  It seems likely to me. No idea if Anthropic would do something like that. Like, Anthropic is just weird. They're the sort of odd one out in all of this stuff at this point.
• 1:18:01
  Bryan Cantrill: Yeah. And I think that they are and, you know, they again, you say the kind of the original schism was over safety. And, you know, in Dario's piece today, it was interesting, but talking about how, believing that models should have to be certified to be safe. It's like, well, now you're kind of back into regulatory capture. I'm like, I don't think that's going to So I'm not sure that's going to fly.
• 1:18:26
  But I I so I agree with you that I I I doesn't feel like from an apoptotic, feels like it's more possible from OpenAI.
• 1:18:32
  Simon Willison: I'll tell you something fun about OpenAI. One of the there was a leaked email that came out in the trial. One thing we'd like to do soon is create a language model with the approximate capability of GPT three that can run locally on consumer hardware and release that. We'd like to do it soon before stability of someone else does. In general, we think this helps discourage others from releasing similar powerful models, makes it harder for new efforts to get funded.
• 1:18:56
  So this is like a beautiful little Sam Altman thing where he's like, you know what? We should reach an open model just to just to stop anyone else getting funded to develop AI.
• 1:19:04
  Adam Leventhal: There's your dumping.
• 1:19:05
  Bryan Cantrill: There's your dumping. Exactly. Your dumping's right there. Totally. Boy, it's interesting because like now it feels like that's that kind of artifact feels more likely than ever.
• 1:19:18
  It's that that Kimmy K three is hacking in TSMC to have it be fabbed right now. So, well, I think it's it's wild. And I I mean, again, I feel it's a net positive, Simon. I mean, you it sounds like you you you feel the same that this has been I think this has been a positive a very eventful, but very positive week.
• 1:19:35
  Simon Willison: I mean, yeah, honestly, the the dystopian result was always going to be there's one AI model that one organization controls. And back in, like, a couple towards the end of 2024, that looked like it was happening. Right? GPT four was undisputed. Maybe it was 2003, but GPT four, nobody else had done done had a model anywhere close to it in, ten months.
• 1:19:55
  And then everyone caught up, and you had, like, Google Gemini coming out and Claude and everything. But, yeah, we did look for a while, like, we were stuck with just the one.
• 1:20:03
  Bryan Cantrill: Not gonna happen now. Not gonna happen. And I also and the other one that's gonna be interesting to keep an eye on is Google because Google is I mean, they got I mean, there's a lot of they're there, and you feel like I I wonder, do you think we might see an open weight model out of Google? That's not seem impossible. I Google is very hard to predict in
• 1:20:20
  Simon Willison: this regard. Gemma four is my one of my favorite open weight models that I can run on my laptop is Google's Gemma four. It's a great vision model. Like, you can you can I I I have my laptop? I fed my laptop a screenshot of a web page and said build that in HTML and CSS, and it did.
• 1:20:35
  And that's a hell of a thing for a model that was like 18 gigabytes of RAM or something to to be able to do.
• 1:20:42
  Bryan Cantrill: Wow. Wow. That yeah. And and so that but that is not that so Gemma four what is Gemma four? Gemma four is just a vision model that they've got?
• 1:20:49
  What is four?
• 1:20:50
  Simon Willison: There's a 12 b. There's a 32 b. There's and and they're very that's a very, very good model. It's under a clean open source license. Came out maybe three or four months ago, I think, was Gemma four.
• 1:21:01
  And, yeah, I I think it's one of the it's one of the best models that you can run on a decent MacBook Pro at the moment.
• 1:21:08
  Bryan Cantrill: That's wild. Yeah.
• 1:21:09
  Simon Willison: I did man, come on. Boy, Google. This is the Xerox part.
• 1:21:13
  Bryan Cantrill: I'm like, I didn't even know about it. Sorry, Google. But at it, that is wild. Well, I guess, I mean, great news is that DHH can translate his racist blog entries into whatever language he wants, Adam. I guess that's the artifact of it all.
• 1:21:32
  Adam Leventhal: I love that for him.
• 1:21:33
  Bryan Cantrill: Yeah, I love that for him too. But I think this is I think it's wild. Again, I think that we will look back on do think Okay, I guess my last question for you is, do you think Kimmy K three is this kind of potentially pivotal moment? Because it feels like the it feels like what the deep seek moment we thought the deep seek moment might have been, but now actually with this this a model that we can actually I think frontier lab.
• 1:22:00
  Simon Willison: I feel like it's what's pivotal about it is it's the point to which you could no longer deny that the Chinese AI labs have got this stuff figured out. You know? Like, it's the first time a Chinese one of the Chinese open mic models was beating the other frontier models. And I but I think any of the labs in China could have got there first. You know?
• 1:22:17
  It's like a three horse race between QL and GLM and Kimmy. And there's another couple of Chinese labs that are also doing fantastic work here. So, yeah, I think it's more symbolic of the open weight movement that's specific to Moonshot and Kimmy. But yeah, no, it's a big deal.
• 1:22:33
  Bryan Cantrill: It's a big deal. It's exciting. And I think it's, I think that a future where many more people have many more models, I think is just a is a better future. Honestly, I think it's a future where we understand these things better. I think that we will make our software a lot better.
• 1:22:51
  My bio weapons fear is zero. People should again read Biohazard by Ken Albeck. But hey, you know what? If a bioweapon gets me first, so be it. I'm willing to eat that for eat the irony there.
• 1:23:05
  But this is really interesting stuff. Simon, we are so grateful for you, Jordan. Thank you so much for for joining. And we're always love hearing your perspective.
• 1:23:16
  Simon Willison: This has been really fun. Thanks a lot for inviting me.
• 1:23:19
  Bryan Cantrill: Yeah. Absolutely. And, you know, good luck for your dog and the flight lessons. I know that, like, it's just because you can crisper in the wings doesn't actually mean they know how to fly. Know?
• 1:23:29
  That's that's
• 1:23:31
  Simon Willison: I need need to do some reinforcement learning on her.
• 1:23:34
  Bryan Cantrill: Yeah. There you go. There you go. Alright, Adam. I think we might be out next week, but we'll be I gotta to go check the schedule, but we're going to be kind of in and out here towards the end of summer, but then back on for fall for sure.
• 1:23:47
  So, all right. Thank you very much. Thanks again, Simon. Thank you everyone. And talk to you next time.
• 1:23:55
  No, to you D.
• 1:23:56
  Simon Willison: H. Exactly.