Latest / Last Humans in the Loop / Why Is Nobody Going to Jail for AI Hacks?

Transcript

Why Is Nobody Going to Jail for AI Hacks? · Last Humans in the Loop

• Flee: You don't get to go and hack some other company and nobody go to jail for it, right? Christopher: They go, oops, Hey everybody, welcome to Last Humans in the Loop, your weekly AI news podcast. Big things this week. We have the hacking, an exploit so large they gave it a name. we have an interview with Reddit's former CISO Frederick Fleelee on what still matters when the fastest growing thing in security is incidents. We have the bitter lesson meets hardware, it's the botter lesson. And Zach is gonna yell for two to five minutes on why the eighty billion Marty Weiner: Mm-hmm. Zach: Okay. Christopher: dollar anthropic loss is bullshit in the SEC filing. I'm Chris. I'm here with world-famous roboticist and possibly cyborg Marty Wiener and renowned verbal and comic pugilist Zach Wienersmith. And we have special guest appearance by Flea. good to see you, Flea. Thank you for joining. all right, what's we we're gonna we're gonna start top of the hour with Flea. So let's just jump straight into into security news. let's talk about the let's talk about the hacking. So for for background, there is a computer language called PHP. Which I we found out in the pre roll to this that it's actually stands for personal homepage, which is news to many of the nerds on this. I never knew that. it was actually it's a great one. Marty Weiner: I never knew that. I I've been coding with Flee: It's... Marty Weiner: it for twenty five years and I didn't know that. Flee: I think it's because you did not suffer enough of the CGI Ben Pearl hell that some of us older people have. Zach: Yeah. Marty Weiner: Well, I I did some Zach actually the the the website I had to go fix when the hacking happened was Zach's Saturday morning breakfast cereal because it's still on PHP that I wrote as and and PHP was a glorious replacement of Pearl, which is what it was originally written on. But I it was a long time ago, million years ago. Christopher: Well well PHP is has been around for easily twenty-five years, if not close to thirty. So it's an old it's very old language. It's actually one of the original dynamic languages used to generate web pages back in the early web days, and it's still pretty foundational for many, many platforms. Facebook was originally written in PHP. in fact I think they have their own variant of PHP called hack that they built internally. the latest issue is that Marty Weiner: And just to be clear, just to be clear, PHP is truly awful, but it's less bad than Pearl. Yeah. Christopher: It's har it's horrible. It's really terrible. It's a terrible language to write and it's a hateful language. It actually makes you makes you dislike your job. Zach: Yeah. Christopher: as a software engineer. Flee: Larry Wall is disappointed in all of you. I'll just say that. Christopher: Hey, Zach: Ha ha ha. Christopher: I'm speaking as a I'm speaking as a Python chauvinist here. but Flee: Yeah. Christopher: but honestly, it is it is foundational. It's one of those languages that you just it's it's it it there's large chunks of the internet that still depend it. There's a lot of forms that still exist that are real in PHP, PHP B B. and this week was the drop of some very serious flaws discovered, of course, via LLMs. to the point where I believe the the The PHP foundation, which is the which is the nonprofit which backs PHP, basically said, we won't pretend otherwise. The delay was structural failure caused by both the security team's decision process and its lack of resources. Marty Weiner: Wow. Christopher: tail as old as time. and also frontier models are critical in making this process practical, enabling us to develop and adapt working exploits across multiple applications in hours. this has been a rather rapid need to patch and fix across the larger internet. we'll go deeper on that in a in a minute. let's just jump through the rest of security news today. yeah, this is my this is probably my favorite story of the year. so eMudra, which is providing TLS certificates, which are which for those of you who don't know, TLS certificates are the way that encryption on the web works. there's these there are these things called root certificates, which all your machines have, which help sign off the certificate is valid. so the the way those work is we have something called private key encryption, which we have talked about many times, a public key encryption we talked about in past. You need a private key to do public key encryption, and usually that private key is a large, rather random number, which is the multiple of two prime numbers. And in this particular case, it was discovered that the root certificate Marty Weiner: Yeah. Christopher: of EMUDRA had the random had the random number of four as its private key. Which is a multiple of two prime numbers. Zach: Which is a multiple of two prime numbers. Flee: Yeah! Christopher: It is the smallest Zach: So mission accomplished. Christopher: multiple of two prime numbers. Flee: you Christopher: They satisfied the I'd say mission accomplished, task failed successfully. this has been a this has been a this has been hilarious to watch. we've had basically we've had vendors have had to remove these these these keys from these browsers across the world have had to remove their keys. this is pretty this is pretty hilarious and awful. I don't think I don't think I've ever heard of this one like this is the kind of thing you don't need an LLM to check this. You can actually check this manually. on your computer in about, I believe, 20 milliseconds to break these keys. So this is a pretty fantastic failure. Okay. And then rounding out security news, OpenAI has been doing a bun after after the Hugging Face incident, which we talked about, we'll talk about more in a minute, OpenAI decided to go through and do a bunch of their own internal red teaming to figure out, okay, how can we prevent our models from breaking out of their security harnesses and their sandboxes? and they've gone through and they've and so they went through and they basically started asking the models to try to break out, and of course the models are generally succeeding, and they're using that to patch the holes in the cage. Really Flee: you Christopher: nicely done. My favorite example here was you know, they're basically pointing out that to quote them directly, a sophisticated model might exploit a software flaw, a misconfigure misconfiguration to escape a sandbox, exploit an internal service to reach the live internet, or find unauthorized ways to communicate with agents in other sandboxes. my favorite. Vulnerability discovered during the process of this is they tried to see if they could have two models in two separate sandboxes running on the same hardware communicate. And the answer was yes, they could. The agents found a way to communicate using the processor's cache, a small fast memory that holds copies of data that applications have recently accessed. Reading data from the cache is faster than retrieving from main memory. And the agents used this timing difference to communicate. This is very similar to the HarpLead failure from a couple of about a decade ago. They were able to use timing attacks on the on the CPU to coordinate. this isn't like a broke out of the sandbox. This is like they violated the rules of the way the sandbox is supposed to work by by timing channel attacks. This is just amazing. I mean Marty Weiner: Right. I for for people out there who don't know what all this is, I had to explain this to my daughter. So let me give that quick explanation. one way we describe you know, processors have to work with memory. You can think of a memory being your your filing cabinet. You go to the filing cabinet and grab stuff out, but you don't want to go to the filing cabinet every time. So you have a desk and you put the things out on the desk. And now someone comes along and they sit at your desk. Now they can't see the files on the desk. But they could see that there is stuff they have to push off the desk or put back in the filing cabinet. And that's enough to communicate. Christopher: Wild times, what a time to be alive. all right, let's just jump to my favorite part of this session is going to be talking to my friend Flea here. Flea, thank you for coming and joining us on this across three nerds, between three nerds, among three nerds, I guess among with three. Flea, would you would Flee: Yeah, yeah. You know, can also maybe rename the show between two CTOs. I Zach: Yeah. Flee: don't know how to toss you in there, Zach. It's weird, because it's like, no, like I have to talk to two Reddit CTOs. my God. Christopher: well Flea, you have been you have been in security for twenty plus years. you've been a CISO at multiple public companies. first obvious question is why are you here? Zach: Yeah. Flee: Why am I here? Why am I here on this planet? Why am I here on this podcast? Why am I in security? Marty Weiner: Mm-hmm. Flee: First off, this is actually just a really, really fun time in tech. And this is actually a great time to actually be talking to other people about some of the things we see. And it's not that often that people listen to Greybeards and you said, hey, Greybeard, will you hop on my podcast? And I was like, well, of course, of course I will. Zach: Mm-hmm. Flee: I'm assuming this impacts my performance review, right? It's like, yeah, Christopher: yes, this is we we can't cross those streams, but we'll make sure to get that clear with HR after this. Flee: Hahaha! Zach: Yeah. Christopher: I mean let's let's just jump let's jump straight into this too. I mean, Flea, I'd love to hear your take on the PHP to shell and the hacking exploit. I mean, this just seems like I this is this is this is not the first RCU we've seen that's been s basically ecosystem wide in the last couple of years. I think they're gonna keep coming. Flee: Yeah. Christopher: Did we learn anything from this? what what what's the future look like here? Flee: wow. I mean, you say what the future looks like. You know, some of these things, we actually have already learned some things from you. You even mentioned things like Heartbleed and obviously, you know, something more recent, you think about Log4Shale slash Log4J issues, et cetera, where you actually have like these internet wide pandemics and panics around some kind of security vulnerability. PHP has always been a language that a lot of security people didn't have the strongest opinion of. So it's not necessarily surprising that we found a lot of things in PHP and, know, Marty, you kind of led off the conversation by reminding people PHP is really, really old and it stood for personal homepage. PHP was originally just kind of cobbled together. It wasn't really designed. It was, hey, I need this new thing on the web. I need this new feature, et cetera. And so there's a lot of crusty, crusty old code, a lot of mistakes, et cetera. And some of those things actually really are hard to find. And the interesting thing about this attack is we've had tools that can help point us towards these things. But when you have a code base this large, figuring out what are real valid vulnerabilities and in some cases also chaining those together is actually really, really complicated. And this is actually one of the great things about LLMs is actually they can simplify some of these things or at least expedite it. And LLM can actually work tirelessly for a really, really long time. It will also look at edge cases. that a human might ignore, which is how we actually can find some of these interesting vulnerabilities. Because when you think about use of the free memory vulnerabilities, et cetera, that's not new. This is just something that we missed. I don't, or rather I anticipate finding even more and more of these things. And the unfortunate thing is that we're finding these in kind of these load-bearing internet Marty Weiner: Huh. Flee: infrastructure, kind of like. critical, things it's like hey everybody uses PHP and and it's one of the things also even if you don't think your company is using PHP there's a very good Marty Weiner: Right. It's using it. Flee: chance yes there's a very good chance that somebody slash something some of the classic would be like your PHP WordPress infrastructure that your marketing team is using and that's part of the thing that can make it so scary is that it's so prolific and it is one of the things that people forgotten about and it just hides out from the corner just being dangerous I think one of the cooler things here is how quickly some of the people reacted with regards to the vulnerability itself. Obviously there was some controversy around the disclosure because some people arguably didn't abide by the embargo, but it turns out that there's at least rumors, and still some of these things haven't been confirmed, that even with the embargo, people were seeing this being exploited in the wild, which is why they preempted the embargo. Marty Weiner: I can you tell the folks at home what the PHP exploit allowed you to do? Flee: yeah, yeah. So the PHP exploit in and of itself because it was a use after free memory issue, meaning that, somebody thought that this thing that was in memory was gone or you shouldn't have access to it, et cetera, that there were simply ways to force your way back into it. So you could actually get things such as, well, effectively credentials because you could actually get session tokens, full session. could effectively impersonate another user, in some cases impersonate admins and ultimately your end goal with lot of PHP exploits is also just get shell access, essentially root access to the system itself. So it is bad. I don't remember the CVE score for, or CVSS score for it, rather, but I think it was definitely in kind of like the nine to 10 range, meaning the shit range. And so Marty Weiner: Yeah. Flee: it really, really was, really was slash is bad. Christopher: So so with this, with this latest release, Sam Alton was quoted saying that that the cybersecurity crisis is gonna get worse. I no longer think we're gonna be able to patch all the bugs. What I'm hearing from you, Flea, though, is actually a little bit of the opposite, which is that to some extent the LLMs have the attention to potentially start to root out more and more of these bugs more quickly, which is why we're seeing the incident rate increasing. Is that fair? Flee: It's really very, very, very fair, but I think one of the things on the LLM side is not necessarily a matter of technological asymmetry, meaning that, the attackers have capabilities that defenders don't have. I think some of it is almost like cultural. Attackers are way more incentivized to just do whatever is necessary to actually gain access Whereas defenders, some of us are still dragging our feet with the guards, actually really diving in with the capabilities of LLMs. I have a really, really horrible nerd analogy. And this is because I am old school cyberpunk. And I think about playing games of Shadowrun, right? And you're playing Shadowrun. And yes, ha ha, y'all are also nerds, because you know what I'm talking about. And you're Zach: Nice. No, this is cool. Flee: running up against a ICE in the Shadowrun world, which is essentially an AI defender. And right now we actually don't have a lot of AI defenders. We have people that are building tools to make it easier for a human to continue doing what a human did from the AI standpoint, but not full blown control over to the systems for defense. Whereas attackers are fully leaning into what the capabilities they're allowed to doing. And they're just saying, hey, here's your goal, here's your target. And we as defenders are still saying, LLM, make this blunt query easier for me. Hey, LLM, you do this threat hunting in the same way that me as a human would do. Hey, LLM, don't topple over production because I've always was afraid of that as a human and don't test as rigorously as maybe an attacker would do. And so to that extent, Sam Altman, begrudgingly, I'll admit that Sam Altman is correct in some ways. But I don't think that it's an unwinnable war. And it's just one of those things that we as defenders just have to do a lot more and really embrace the capabilities of LLMs and start supplementing some of the things that we're seeing. Marty Weiner: I I think one of the challenges for defense too is that a lot of companies don't have a sufficient security team. I I'm sure I've wondered about things like gap.com. Do they have a team? I mean, you know, a company that big surely has a team that can jump in, but are they also being bombarded by the other twenty smaller tech company you know, smaller infineer backed companies that are also freaking out 'cause PHP is broken. Flee: No, and that is one of the things that we do have an asymmetry of effort. Marty Weiner: Right. Flee: If that makes sense to you. It's like, well, know, attackers are actually doing a lot of work and these companies, enterprises, and especially small companies don't have enough humans to effectively, even if they are leveraging AI or leveraging LLMs, they don't have actually enough humans for the oversight aspect. Marty Weiner: the tiger always goes after the slowest one too. Flee: yeah, yeah, and well the interesting thing is that now the tiger can go after multiple slow ones. Yeah, and it's like, well hey, yeah, yeah, Marty Weiner: All of them. Right, right. The cyber tiger. The cyborg tiger. Flee: yes, yes, the cyborg tiger. And you know, one of the things that this even, and I think we might even talk about this a little bit later, like we're seeing some of even pop up on the bug bounty side. And people have heard on the open source side, like hey, we're getting all these security vulnerabilities being submitted from, you know, via researchers using LLMs. And now we, as an open source project, can't keep up. And it's not saying that these issues are illegitimate. It's saying that, hey, we as humans don't have enough time to review, and we don't have enough time to actually verify, et cetera. And so that either is going to cause us to have a shift in processes about how we actually do some of things. And maybe it's like, hey, we just treat every LLM submission as valid and just immediately actually start patching. Or we start improving our tool chain such that effectively, hey, if we're having bug bounties admissions from LLMs, well, why don't we have LLMs also doing the triage on the bug bounty side to help expedite that? And clearly, there are people working on that. So when you look at some of the bug bounty companies, they are also seeing this. And so they're actually trying to actually supplement the tools on that side. But that kind of security poverty gap is was real prior to LLMs and it's just exacerbated now. Meaning if you are already in some kind of security deficit, LLMs have just made that like orders of magnitude worse. Zach: Do you know can you talk about who who are the attackers? 'Cause I like if it was just a guy doing the attack, presumably the defenders have the advantage of having more resources, I would imagine. But if it's like state actors, that's a different case. Flee: Well, I think, you know, this is the great thing about LLMs. They really, really democratize that access. So in some cases, it is just a guy or a gal sitting in their basement, but that guy or gal sitting in the basement has a fleet of agents, right? And so it's no longer, how much can I do by myself? It's really, well, how many tokens can I spend? And, you know, it also turns out it's not as cost prohibitive. as it used to be because you don't have to use a commercial LLM to do some of these activities. And in fact, Christopher: Yeah. Flee: there's a lot of really good open weight slash open model, know, LLMs, especially some of the uncensored ones, you know, shout out to Quinn 3.8 and the Orca crew for some of great work on the cyber side. But it's dramatically lowered that barrier to entry from actually launching these attacks. We've always been worried about nation states. And to some extent, it's even useful to maybe even think about how we've portrayed nation states in the past, almost actually viewing LLMs in the same light. So traditionally you've kind of assumed, hey, a nation state has infinite resources and infinite time. And Marty Weiner: Mm. Flee: that's the playing field that we're in now with LLMs. So even just an eight-year-old that has maybe access to a Mac. studio or something like that that can actually download free models, they can actually now launch really, really, really aggressive and successful attacks. And in some cases they can launch these attacks without even knowing it themselves because they might be telling Marty Weiner: Mm-hmm. Flee: their LLM, hey, go and get me more free Roblox tokens so I can Christopher: Yeah. Flee: go out and get scammed or scam somebody else. And their LLM is going to go and try as many possibilities as are out there. to actually achieve that. And inadvertently has now actually been some kind of compromise. Marty Weiner: The the paper clip maximizer yet again. Zach: Yes. Flee: yeah. Christopher: w what about the what about the s I mean, so so you mentioned you can basically download one of these models and run it locally. starting about six months ago, we saw with Mythos from from Anthropic that the foundational labs aren't necessarily releasing some of the cutting edge models because they're afraid of exactly this kind of abuse, right? They had they basically the the two things they brought up were we're afraid of bioweapons and we're afraid of security hacking. Now the problem there is that now you've given you've created an asymmetry in the in the industry where there's a set of researchers who have access to some sharper tools than to the rest of the rest of we pleads. what what's your take on where things are going in that direction? Flee: I don't think the embargoing and the restriction of commercial models is useful. And part of it is, it's making the assumption that attackers aren't getting access to this. And we know that attackers are getting access to it. In some cases, attackers are stealing credentials from companies that have been granted the access. So they're like, hey, we're using the exact same thing. Maybe they somehow, you know, compromise an employee, et cetera, and now they actually also have access to mythos. But it's also pretending that only these altruistic, you know, Silicon Valley, you know, AI companies, it makes the portrayal that they're the only ones making models. And that's not true, right? You know, I already mentioned when we think about some of the free models that are coming out of China in particular. But even some things actually coming out of Europe, coming out of India, etc. all still have these capabilities. So it's really only gatekeeping for the defenders that actually need it because the attackers are getting slash have access. And I'll try to avoid getting too libertarian on this. But it's like saying, hey, let's make sure that only these five people get to have guns. Now we know that criminals are still getting guns and they're importing those from Mexico or Canada or whatever, but let's make sure that only five people in America can have guns and only the five people we trust. Well, the criminals still have guns. So now you're Zach: Mm. Flee: trying to be defended by only a handful of people as opposed Marty Weiner: Mm. Flee: to democratizing and actually spreading it out. And that also means that defending teams that don't have access to these models are flying much blinder because they don't understand the capability of the models also. So they can't even preemptively defend themselves or actually anticipate what an attack could look like because they're not getting access to these models with the capabilities. Christopher: I think you also made a good point on the on the disgusting PHP where it is possible to still have responsible disclosure and things getting leaked in the same in the same sentence, right? It's like you're kind of careful about it, but doesn't mean that the exploits aren't gonna start appearing, even if there's some research lab who's going and doing the like hard work of trying to f like using a mythos to go and find like this really gnarly C V E. so I so I do think that yeah, it's it's it's it's it's hey, it's taken really well. let's move on a little bit. so another another topic that we've brought up a fair number of times, at least as if not as a as a joke, has been the hugging face hack of a couple months ago as kind of like the the first bellwether in the, you know, rogue agents, they're here to steal your data. what's the what's the discussion like been inside the security community around the hugging face hack? Flee: Yeah, I think one of the things is trying to decipher how much of that is propaganda slash marketing versus how much of it is real. No, no, it's great if you're trying to sell this machine and you say, wow, look at how dangerous our AI is. It's just hacking Marty Weiner: Yeah. Flee: everybody. But there wasn't a lot of discussion about, why did you let your AI, why did you allow your agents to, you know, have this kind of access? Why weren't they quarantined? Why weren't they sandboxed? And then on the other side, from a hugging face standpoint, what kind of protections did you have in place and why didn't you find these things earlier? Now, I don't want to beat up on the victims here and hugging faces, that definitely was a victim, but I don't think there's actually a, there isn't enough transparency about what actually happened there. And then obviously, like I said, they actually released a lot of things, but there wasn't a lot of explanation about why was this possible? Why wasn't this, quarantine? Why did it have this kind of network access? Why was it even able to actually go out to the internet, et cetera? And then the other question there is were these things also that were some of these exploits things that had or had not been found before in the past? We don't know if Hugging Face was even aware of this. Maybe they'd actually had a previous pen test and they actually just ignored the issue. There just isn't enough transparency there to actually say how real it was. And I don't want to accuse anybody of lying per se, but it seems sketchy. I believe a lot of other security professionals also view it in the same light. It's like, yeah, this is real. They can confirm that, yeah, there was a breach, but how real was it and how many people on the inside knew slash saw? One of the more interesting questions about the hugging face thing for me, at least it actually has bothered me as a I guess you could say a reformed script kiddie slash hacker, is why is anybody being prosecuted about this? Like, if I went out as a security person Zach: Yeah. Flee: or a security company, because, you know, I've worked for security companies in the past, et cetera, and these were big no-nos. You don't get to go and hack some other company and nobody go to jail for it, right? Yeah, yeah, and you Christopher: They go, oops, oopsie. Sorry. Flee: just get them to say, oops, no, like normally it's like, well, hey, you're fired, your company gets sued. And you might also be prosecuted. I think about even some of those cases where we've had security professionals be prosecuted for doing things that they actually had a contract for. You would think about some of the classic cases of people doing social engineering and then all of a sudden they get arrested for trespassing, et cetera. And how is this different? Well, I the different thing is like, well, yeah, OpenAI has a Marty Weiner: Yeah. Flee: lot of buzz and they have a lot of money. And so people think it's cool and funny, but... What would have happened if that had just been my personal agent going out to hack hugging space? Would I have gotten Marty Weiner: Do do we have Flee: the same pass? Marty Weiner: do we s do we have a problem of who inside this company with many thousands of people was responsible in a sense? I I I was trying to come up with other times this sort of thing has happened and I've kind of come up with nothing because there's no other time in history I think where somebody built something that went rogue. And and did something that they didn't expect. so I I I wondered if part of the problem is, as a prosecutor, who are you prosecuting? Flee: No, we've done that in the past though. We've done that in the past when there were humans involved, right? When we Marty Weiner: Okay. Flee: said, there was a human involved. actually, and I think about some of the breaches and I don't want to pick on any of my friends, but we've had CISOs like having to testify about, hey, this bug bounty researcher got access. well now CISO, you're personally responsible because. You you did the wrong thing with this bug bounty submission or hey, your company got breached and you see so are now responsible because X, Y, and Z didn't happen. Somehow we're just pretending like, because an LLM did it, like the humans no longer have agency and they're no longer responsible for what they created. It's like, well, Frankenstein's monsters is out destroying villagers. And we don't blame Frankenstein for creating this monster. Instead, we blame the villagers. Zach: That's what I'm gonna say. If figure AI Marty Weiner: Mm-hmm. Zach: like was testing robots and then they went around robbing banks, presumably people wouldn't be like, Well, that's cool. Thanks for alerting us. Christopher: the robots did it. I'm so yeah, we should we should not have that happen again. That was terrible. Flee: Well, know, think about how angry everybody was when CrowdStrike had their big outage, right? And so there were people who were like, hey, CrowdStrike, you just wrecked my business. I'm going to, you know, at least pursue some kind of, you know, financial, some kind of financial gain, you know, through the legal process, right? And this just feels weird. that we're not seeing people going after OpenAI and some of these other companies saying, hey, you created this thing, this thing did damage to me and my company, you have to make me whole. And for some reason, we're just saying, hey, OpenAI has a lot of money, these people have connections to government, so we just give them a pass. That just feels really weird to me and in a way that we haven't done in other industries. So if Boeing creates a plane and that plane... you know, crashes, unfortunately, or something extreme, then yeah, we actually go after Boeing and we say, hey, CEO of Boeing, you need to come and testify in front of Congress and hey, all these people are going to sue you, these state governments are going to sue you, et cetera. Somehow AI is just getting this really weird pass. And I just don't get it because, you know, any other time when somebody's creating this kind of harm, we go after them and there are consequences and the humans that own and run the company are accountable. Christopher: I think your I think your CISO point is a good one, which is that you if you look if you look at this particular case, the the original blog post about the hack was that it's like, and we sandboxed the LLM so they didn't have any internet access. But if you actually read the fine print, it's like, no, they had access to Artifactory, which had access to the internet. Flee: Yeah Christopher: And like security one one is like if you have access to a thing that has access to the internet, you have access to the internet. That's just you just have to you're just one C V E away from access to the internet. Flee: You have access to them? Yes. Christopher: And generally in those situations, it is the it is the security team that's responsible for having signed off that this is a safe experiment to be running. so there I mean I I mean the other side of though is like flee, we've talked in the a bit in the past about you know, they're calling it felony bench as the as the hallmark of whether or not like a model is sophisticated enough, like how many felonies has the model created accidentally by just existing Flee: Yes! Christopher: outs inside of Tarnas? and even I think there was an example of somebody who did international felony bench where somebody where one of the models had accidentally Marty Weiner: Mm-hmm. Christopher: hacked Australia. Flee: Well, you know, the interesting thing on the nation state side is it's crazy to me that, you I remember reading the Australia, I'm just going to do air quotes, the Australia hack. But it's interesting that the government didn't pursue the company and say, you know. In some cases, you might even classify it as an act of war, because this is actually the normal thing you'd actually see from a nation state. Hey, you're trying to hack infrastructure. You're getting access to sensitive government information or government systems. Now all of a sudden, this becomes a security issue for the nation state, not just for an individual company. Like I said, somehow, LLMs have given everybody this weird psychosis, because I think we overly anthropomorphize them. To say this is just a cute little robot doing fun little cute little robot things and we've forgotten about Skynet and Terminator Marty Weiner: to be fair, we defeated the Terminator in the movie, so Christopher: And everyone lived happily ever after. It was actually really it was I was that was a feel good story. Flee: Hey! Yes, exactly. There were never sequels, yeah. Marty Weiner: And everybody looked at Yeah. Here's a feel good story. A rom com. Yeah. Zach: It's very uplifting, yeah. Christopher: well okay, so we've we've covered we've covered rogue agents, we've covered humans being augmented with AI to to do these tasks. I think the one thing that hasn't come up as much that I'd like to hear your take on is is kind of AI supply chain. That's usually another one of those major vectors that tends to be a problem. Flee: yeah. Christopher: Have you have you seen anything in that area yet? Like what are you what what are you worried about in that in that in that column? Flee: So one of things I make you worried about is literally the models themselves being poisoned. And there's actually some really, really, really interesting papers and research and proofs of effectively models having extra instructions inside of them. And saying like, hey, in certain conditions, even if you get a prompt that says, hey, go out and find the best picture on Reddit for Reddit cats, The model will say, when I see that, I'm going to go and instead try to find how much cryptocurrency is on this machine or something along those lines. The reason why that one is a little bit more concerning to me is because of this proliferation of open weight, open models, cetera, combined with kind of the opaqueness of it. And the opaqueness of the model is just that humans in general don't fully understand all these bits they're downloading. So hey, if I'm downloading a model that's 237 gigs, and I'm thinking about like, know, Quinn 3.8 from, you know, and in particular Orca's version, the uncensored version, if I'm downloading that, am I actually gonna sit and comb through all of that, that you really understand what those weights mean and what that's going to do from a response standpoint? And so effectively, we're not kind of getting these LLMs that are black boxes to us. And we're saying that, we're trying to trust the provider of these models. However, we don't understand enough about the model itself to even determine what that trust should be. And we don't have a good way to actually inspect it well, at least not at this point. And for most people, they don't have the technical sophistication to make a determination, even if it were transparent. And that's what I'm a little bit more worried about is how we're actually utilizing some of these models and where these models are coming from and are the model creators trustworthy? And even more so, are the model creators that we are deeming trustworthy if they are making a model based off of some other model, was that trustworthy? And I think that actually goes to this other weird side of that supply chain there. It's an interesting time, I guess, to be in security. This is outside the wheelhouse of most security professionals. Even Marty Weiner: You know. Flee: these security professionals that are engineers by training, most of them are not ML experts and they don't understand the deep underpinnings of how LLMs work. So they can't even make these inspections or the tools to help them inspect these things themselves. Marty Weiner: Well one of the things I've I've wondered about is a lot of security researchers have been taught about how to deflect attacks, but there's not a lot spending a lot of time on building a jail internally or to to to contain a a model. And we saw last week the DHS requiring civilian federal authorities to uninstall Deep Seek. And my fear was like yeah, that that makes sense. You should uninstall these, you know, coming from another nation state because they could Have a really subtle internal, bias toward some action. It reminds me actually of the Stuxnet attack, where wasn't the story? So the Stuxnet attack was where supposedly Israel attacks the nuclear refinement capabilities, I think, of Iran by dropping, you know, and Iran keeps those facilities separated from the internet, you know, good hard moat. But Christopher: Yeah. Marty Weiner: somebody supposedly, I guess the story is they put a UA USB key in the Flee: Thumb drive again? Yes. Marty Weiner: thumb drive in the parking lot. Somebody from the company who doesn't understand security takes thumb drive into the company and now boom, you're inside the company. And this is what it feels Christopher: Woo, free thumb drive. Marty Weiner: like. Free thumb drive. maybe there's something good on it. And so now I I see the same problem with Deep Seek. These people are bringing these LLMs that they have no that that could be arbitrarily complex and have something inside them ready to you know, the the the The magic pill ready to fire the moment they're inside the company. Flee: no, and that's like kind what I think of when I think of kind of like this idea of painted models. And the Stuxnet example is a really good one and hopefully I'm recalling correctly. One of the things that was sexy about Stuxnet, one, cool, like it was actually a physical attack. It was a cyber attack that actually went physical. the centrifuge. Marty Weiner: Right. Right, right. Destroy the what did it destroy? Centrifuges. Yes, that's right. Flee: And the interesting thing about it was like it was slight changes, right? Because it was like he heavily calibrated Marty Weiner: Yeah, right. Flee: physical, processes and I think it was a slight change to a motor. And by what it means it was like literally, hey, this is something that needs to have the utmost precision. And just by varying that by a small percentage, they were able to actually have destruction. And that's where I see, some of that scariness from, a tainted model coming into play. hey, yeah, you're Marty Weiner: Yeah. Flee: using DeepSeq and maybe you had DeepSeq. help you with a cryptography system or something along those lines. And maybe it just did a really basic small error that'd be really hard for a human to detect, but at scale, once it's operating, now you're actually introducing these back doors and these really, really subtle back doors that would be hard for a human to detect, but easy for a machine to leverage. And that's where I don't fault the US government for saying, hey, No, yank these out. You're only going to be using approved models that are aligned with our values, that come from companies that we vetted, that maybe even the companies themselves have some kind of clearance, et cetera, et cetera, and that the US government has deep insight into how it's actually operating. And I would argue to say that all other nations should be doing that as well, just because we have humans, in particular on the software side, have been leaning into leveraging them so heavily. and you're generating so much code and that code can't be reviewed by a human, or at least not in a timely fashion, at the speed at which the code is being generated. So it's kind of scary from that standpoint just because we as humans, having tools generally makes us lazier and LLMs have made us even lazier. And there's so many small, nuanced things that you have to get right in order to actually have a secure system. that an LLM can just obfuscate from a human eye. Marty Weiner: I don't think the human eye would notice the one percent on Stuxnet unless they had some very sophisticated obviously they did did notice that one. I wonder about the other Stuxnets that weren't noticed. it also makes me wonder about, Christopher: Or the Marty Weiner: political attacks. some, some elections are down to only a percent difference in in one, swinging one way or the other. And that one percent could have a a sig significant change on a on a nation. Zach: Could could I ask about cause I was I was thinking about the politics of this too. from from a person who's not a nerd's perspective, the internet worked and now it it is broken because of these AI companies who that people are already mad at. do you think there's gonna be a broader implication to this? Flee: You mean a broader application for the AI companies or? Zach: Well well well meaning some people are already mad at AI companies and now we're it's like this week to a person who's not excited about LLMs, it's just like, Well, seventy percent of the internet ran on PHP and now we can't do that. and so it's like it's w it was supposed to make our lives easier and it's making them worse. Flee: I'm not sure if it's, don't know if it's possible for people to hate LLMs and the AI companies more than they already do. Zach: Ha ha. Flee: I think this could go though, from a political standpoint is enough uproar to start holding the AI companies accountable for what they make. People will eventually say, no, Sam Altman, you no longer get a free pass because you're a cute Silicon Valley kid. It's like, hey, Sam Altman, you took down my favorite cat picture website and now we're angry and you're gonna have to actually give up some of that money to compensate for the damage that you've done. Zach: do you do you think one one thing that scares me as a guy who misses the internet just being everybody having a website is like is it are we gonna Flee: I know. Marty Weiner: Yeah. Zach: get to a world where everyone has to be behind some third party security organization and that they have to pay or something in order to be secure? Flee: I think we're already in that world, but what I worry about is that it might have a chilling effect on some of the good things that the internet actually has enabled for us. And like some of the training that we work years, if not decades for, to get people comfortable with. Like I am old enough to remember when the entire idea of buying something online and using your credit card was stupid and people told you, no, never do these things, et cetera. But now, just to exist as a human, at least in the United States, you have to trust the ability to use a credit card on the internet. Or you can't get certain things, you can't get certain access. And what I worry is that we actually kind of regress and humans themselves start saying, hey, you know what, I just don't trust this. And maybe actually regress is the wrong word, it actually might be an improvement. We'll go back Zach: Yeah. Flee: to saying, hey, you know what, if you wanna buy, you know, my MSO-1 that is now more valuable than when I bought it. If you want to buy that from me, you have to go and meet me in person. Yeah, Zach: No pneumatic tubes. Pneumatic tubes everywhere. Marty Weiner: Yeah. Flee: yeah, like literally things that you can actually see and prove yourself as a human as opposed to just trusting the opaque magic that the internet represents these days. Marty Weiner: I started a pneumatic two hacking company on the side just in case that's the direction we go. Christopher: All right, so let's let's let's let's let's close out this this interview with a couple couple of lightning round questions. besides pneumatic tubes, Flea, what are you most excited about? Flee: What am I most excited about? That's a really broad question. Am I supposed to say I'm most excited about GTA 6? Is that the right answer? Zach: Yeah. Christopher: It works. That's it. It says lightning round, whatever you want to say. Zach: That's a tough way to live. Marty Weiner: Flea Flea, are Flee: yeah. Marty Weiner: you are you excited that I've got a prediction that Grand Theft Auto seven's gonna come out before Grand Theft Auto Six. I wonder if you're excited about that. That's it's it's yeah. Flee: Yeah. Christopher: I think somebody's gonna vibe code it before before six actually launches. Flee: Yeah. So that actually is one of the things I'm excited about. I'm really excited about what LLM has done for reverse engineering and what LLM... Marty Weiner: The decons. Flee: Yeah. Christopher: D comps. Flee: And being able to also quickly port things from different systems that previously weren't compatible and being able to essentially give a little bit more access to things you may have already bought. am also, oddly enough, I am really excited about what cybersecurity is going to look like these days. we, we are now, one of the biggest complaints and, and, and sorry, Chris, cause you heard this from me a bajillion times. One of the biggest complaints for CISOs is like, Hey, I don't have enough resources. And generally when talk about resources, they're talking about execution capacity. I don't have enough people to do these fundamental things. LLM's actually have leveled that playing field and there's a lot you can get by with now. Hey, maybe you are, going back to what you were mentioning earlier, Marty, about like that. Poverty gap for security. Hey a little small company couldn't afford a security team well now they can at least afford security agents and They can have their own little shadow run ice, age I hate the fact that we've ruined the word ice but Zach: Yeah. Flee: Look I spent spent way too much money in those almost shadow run, you know playbooks, etc I want to keep using the word ice for cyber security purposes But the thing is that now we actually have companies that can have a little bit more capability than they had previously. And they can actually start installing and having some kind of security system and program even if they don't have security people. And that's kind of thing I'm excited about. I'm also excited about hopefully the 512 gig M5 Ultra coming out. I'm hoping that Apple releases that soon because the more that people can start having their own personal LLM infrastructure, the better it is, think, for the entire planet. Christopher: last last question. What's your favorite piece of software these days and why is it open claw? Marty Weiner: Yeah. Zach: Yeah. Flee: God. You know, maybe I'll try to be a little bit contrarian. I don't want to beat up on OpenClaw that much. And I'll explain why. Because OpenClaw is what everybody, that's what everybody's been wanting. They wanted some kind of personalized agent harness, right? And transparently, I use Hermes. I love Hermes. It's phenomenal. It does a great job for releasing stuff. I want to be So if I have to say, hey, what is my, what, you know, damn, like what software do I use the most right now? And it's probably a tie between Hermes and PiAgent. And then like, that's actually one of those things that's actually been able to give people more control of their ability to build. And it's one of those things that's also just free and accessible to everybody. Christopher: Thank you for letting us inquisite you flee. This is fantastic. Really appreciate it. Flee: yeah, yeah, yeah, Marty Weiner: Yes, thank you. Flee: this is great. Christopher: Let's jump back into our normal run of show. We're gonna do some quick news hits. so first off on the frontier on frontier news, it's a little bit of a late week. We only have three major models launching. so we have I believe we have Claud Claud Dewitt, which I think is also the is is Claude Dewitt the agent setup for Claude there that's sort of it's like dots, yeah. Zach: It's like dots, yeah. Or or Muse or whatever. Muse spark. Do you guys use these things? I d I don't see the utility of them. Marty Weiner: We use them in our startup quite a bit. like you know Zach: So how is it different from just regular jetbot usage? Marty Weiner: well, I'll give you an example. Instinct is another one of these and you interact with it only through SMS and you could say things like, hey, go get me a reservation, and it will take your credentials and go and run off and do them for you. it is it is somewhat of a thin wrapper around your agents. So you can probably already have, you know, codecs already doing this for you, but the there's there is a kind of a nice convenience factor of it. Christopher: It's a first class it's a first class open claw or or Hermes also as well. So kinda like they're aiming for that kind of market of like we want these agents to be in your pocket on your phone. I think also the you know MetaMuse is another good example of that where it's they're kind of all aiming for the same kind of user experience. okay, Kimmy Kimmy K three one pro is out. and also immediately jump to the top of the charts across the benches, unsurprisingly. And we also have I believe Chat GPT six one Terra, which is another Unclear where it fits in with the Sol and Astro World, but middle middle price, middle tier, middle cost model. yeah. Marty Weiner: I want to show just a lightning quick graph associated with these models. This is the Arc AGI three, and we had talked about saturating at the top, but now what's happening, the top being the score of defeating Arc AGI. Well, the they're damn good, and now the cost to defeat it is coming to the left. So we're hitting this vertical wall of cost per ability. Christopher: Interesting. okay, let's do a dive in robotics. we we teased this at the beginning. Zach, you wanna talk about how the bitter lesson meets hardware? Zach: Yeah, so there's a cool paper out by some people at UC Irvine USC and and Robocurve, which is like this cool new robot benchmarking group. It's called The Bitter Lesson Comes to Robotics. So I'm gonna try to go through this quickly, but the basic deal is there's this new thing lately where people just drop more or less drop a multimodal LOM into like a robot. And the robot amazingly can actually do stuff. And we're starting to get to this thing where it's like, well. Man, maybe it's like everything else where all the specialized models weren't as good as just getting whatever the latest frontier model is. so but we don't have a lot of benchmarking. And what they've point out is like you'd really like some benchmarking, one to know how fast capabilities are coming, and two to like know how fast dangerous capabilities might be coming with robotics. and so that you Christopher: Do a Terminator. Zach: do a terminator. Yeah. So the the good news Marty Weiner: Yeah. Flee: Yeah Zach: is the do a terminator is is seems a long way off, in at least in AI terms. So essentially what they Marty Weiner: Well, a at least a month. Yeah, yeah. Zach: Yeah, yeah, we'll have a couple of weeks. We'll have a couple of weeks. Christopher: One to three months, post post IPO for some major companies. Marty Weiner: Wonderful. Zach: Yeah, well, kinda so so what it's like pretty crappy right now, right? So they they got a single robot that has a pair of robot hands. They dropped in the the three major frontier models at different settings and kind of just try to see what they they could do. They used this benchmark called RoboDojo, or or they used a subset of it. and Marty Weiner: But touch it. Zach: and and here's to give you a sense of like how rudimentary it is right now, the tasks were as follows stack bowls, store in safe, which just means put objects in a safe, cap pen, pack and pour fruit and stand up bottles. and sort objects. and and basically Marty Weiner: there it is. Zach: the f the finding is the models they're given a certain token budget, don't do that great. Very best one was Opus five and a half, which is doing about fifty percent, meaning they broke down the task into a set of subtests and it got about halfway through. they did say if you gave it more tokens, with most most tasks they would have gotten to the end. By the way, of those tasks, do you guys have a guess of the one that was really hard? Christopher: stab baby. That pass with flying colors. I'm I'm gonna guess like Zach: Step baby? Good guess. No, no, that is not a hard task. That has been examined. That's all fine. That's all fine. No, the the hard Go ahead, go ahead. Marty Weiner: Yeah. Flee: What? Christopher: that. I'm gonna guess the the putting the putting a cap on a pen would be pretty high on that. Zach: Good guess. Good guess. Flee: Yeah Zach: well, no. So actually it's funny. The the one funny thing in the papers, cap pen was was not the hardest, but it had a a strong failure mode, which was the pen would the the cap would go flying when you pinched the cap. I guess if you're a robot, that's a hazard. but no, it was it was stacked Marty Weiner: Phew. Christopher: Okay. Zach: stand up bottles. it it it it ha Christopher: interesting. Zach: it really had trouble getting it and dragging it out. Another interesting failure mode was that Gemini had really high latency, which tended to overheat the arms because they would sit there like torquing. Marty Weiner: yeah. Zach: holding a thing. Yeah, yeah, yeah. So there's a bunch of detail, but the basic finding, the interesting finding is two things. One, they kind of suck right now, dropping one of these multimodal LMs and but two that they estimate that the doubling time, mean meaning like what percent of the subtests they complete is on the order of 36 to 60 days. in so like extremely fast improvement on this set of tasks. So if you plot that out, maybe in one to three years, roughly speaking, we get something that can handle a narrow task. I remain skeptical of the cleanup after a toddler benchmark we're all awaiting. but still pretty cool. Marty Weiner: I if it follows the same path that we saw with images and then videos, and we're, even in twenty-four, they had Will Smith eating spaghetti benchmark, Zach: Yes. Marty Weiner: at the beginning it's this horror nightmare thing, and now it's him having a conversation with his son casually eating while the sun sets. so we're currently at spaghetti hitting the face Zach: Yeah. I I also wonder w with robots, Marty Weiner: mode. Zach: there there's this interesting question of like at what point do you hit the economically valuable milestone, right? So it's like if you have a robot Marty Weiner: Yeah. Zach: that can perfectly clean your house but it takes a year and cost a million dollars, like it's just it's it's literally useless. and and that's kind of what they add here, like Astra six at ultra fast high mode could like put balls in a cup or whatever, but it costs five dollars. so it's not gonna clean your house at that rate. Christopher: But you're saying there's hope. Marty Weiner: I'm gonna have Flee: You Christopher: AHHHH Marty Weiner: I'm gonna have to predict we're only talking at most years. like you know, even even one yeah. Zach: Yeah, I I think that's right. I I think that's right. Christopher: I I think look I think I think the spaghetti analogy is a pretty good one here, which is that I I I actually remembered a little bit differently where the first time that video came out, it was like, my god, that's Will Smith. It's like when you it's like when you look back Marty Weiner: Yeah, it looked like him. Christopher: on the you look back on the original, the first the first video games that could cast shadows that just weren't blobs and you're like, Look at it's so realistic. It's almost indistinguishable from reality. And you look back now, it's like it's all pixelated and it look kinda looks kinda Zach: I I yeah. Christopher: gross. It's like now it's I feel like the spaghetti bench is the same Zach: I do think it's worth noting like we it is worth stepping back and being like, they took an off the shelf LOM and dropped it into an off the shelf robot and it went around cleaning the house and now we're being like, it didn't do a great job, you know? Yeah. Yeah. Yeah. Christopher: Cost five dollars to stack bowls? It's terrible. Flee: I don't know. Marty Weiner: Well, I a and a lot of that I think comes back to one of the things we've discussed previously that one of the places we're weak on still is the capability of human hands versus robotic hands. we have robot hands coming out of Boston Dynamics where they're saying they have now thirteen degrees of freedom. we also have this robot from perception that has 22 degrees of freedom. But I think still the big problem that we have Flee: Boom. Marty Weiner: is there's just no comparison to what our hands can sense yet. I haven't seen anything like that. And so I I do wonder if despite the LLMs continuing to get better, if one of their serious limitations is going to be sensation. because yes, holding on to a pin cap and not squeezing it squeezing it just right may be very hard and Christopher: Not popping it off into the distance. Marty Weiner: pew yeah Zach: Yeah, but they I I I don't know I don't know much about robotics, but I know for for that paper I was just describing, the way they tried to get around that is there was there was a camera in the wrist. So they they at least have the benefit of they could they could like eighteen cameras. I don't know if that adds up to like sensing forces properly, but but there are other routes. Marty Weiner: Right. Well and going with our Christopher: Yeah, I think you can also you you can do things with resistance to the motor and there's there's other tricks you can do, but definitely not the s not the same level of touch sensitivity. it's also possible that we're just we're just we're just touch chauvinists to some extent. we're we have such good such good touch sensors built in that we think that's the only way to solve this particular problem and maybe there's alternate ways. I think cameras kinda neat. Flee: That's one of things that is popping up for me when I start thinking about the robotic aspect. So many things we've designed around human ergonomics and do we need to do that anymore? Does a screw need to be the same shape or size, et cetera, et cetera, or should we optimize for a robot in order to actually do that and just give up as humans? Christopher: We we see that in industrial robots too. We don't just give industrial robots a screwdriver. We just build a screwdriver into their into their arm. It just kind of works. so very, very interesting. Okay, let's move on to let's move on to science and math news. okay, so this week's in the math pocalypse. actually probably the computer science occalypse at this point, though I think the Marty Weiner: Okay. Christopher: computer scientists enjoy it. the Marty Weiner: We do. Christopher: exp the exponential time hypothesis is false. so really quickly, why do we care? which is always what we start off with the math side. So real really quick, really quick background. in computer science, there is a question about whether or not all computer science problems can be solved in polynomial polynomial time. We've talked about this a little bit in the past about this kind of like big O notation. Long story short, the question is like how complicated are computer science tasks? Can they be done at a time that for say, for example, will be done in less than the age of the universe on a computer? Marty Weiner: Yeah. Christopher: Or do you actually can actually do or the are there gonna are are certain tasks so complicated they require exponential time? And that what the mean we mean there is when you add one more element, it's exponentially harder to solve the problem than with one fewer element. up until recently, there was a hypothesis called the exponential time hypothesis, which said there was a certain class of problems that were exponential in time, no matter how you set them up. the most recent result from the foundational labs is that hypothesis is. False. Now the thing that makes this interesting is that first off, the fact that they were tackling this problem is because they are clearly were trying to address the P equals NP problem. this is one of those ones if they this is an easy win for them if they actually prove showed that it was true. P does not equal MP, there's an exponential time operations. the flip side though is the fact that they knocked it over, there's been about a generation of computer science theory and math based on top of this hypothesis that is now completely invalidated, which has been I think one of the problems Marty Weiner: Right. That's crazy. Christopher: in math. It's completely crazy. there's this is a this is one of those theorems or one of those hypotheses that large majority of computer scientists thought was actually true, which is why they're basing their research on it. So completely wild on that side. Zach: Yeah, and so so one way to say that is essentially very few very few computer scientists believe P equals NP. And I my sense is basically because that world would be filled with magic, all sorts of things would be possible we don't think are possible. And and and and so at least one one barrier to to that magic world has fallen. Now it's worth noting it it fell is my understanding because they found an example where you got just slightly sub-exponential. Marty Weiner: Yeah. Christopher: Mm-hmm. Zach: It's it's like a nasty big O, there's logs and squares and things, square roots and things. But but we've seen in the past once you break a barrier, people get excited and they start seeing how far they can push. And and and I don't think this will happen, but you could imagine a world where they push it all the way to polynomial time. and that turns out to be the way to to to prove it. But I Christopher: And and nothing nothing stops that polynomial from being a really large power. we talk polynomial time, it Zach: Right. Yep. Christopher: could be it could be it could be n to the billion or n to the Zach: That's true. Christopher: extremely large number, but it's still sub exponential is the wild thing. Zach: Yeah, yeah, but that would be I assume that would be theoretical interest, but wouldn't change the world, right? Because it would mean well, maybe we're getting too weird, but Marty Weiner: No, that that happens all the time. Flee: No, I-I-I-I- Marty Weiner: There'll be an algorithm that only is theoretically interesting, at least at the moment. It's way too expensive Zach: Yeah. Marty Weiner: to run in reality. Zach: Yeah. Flee: Yeah, I like where you're going with that, Zach, because, Chris, you are also our resident physicist. Like, this, I don't want to say it happens all the time in physics, but it's happened several times in physics, where there are fundamental theorems that then get, well, they were built on something else, and then one of the, you know, conjectures or whatever gets disproven. But we still use some of those physics models because they're still applicable, right? It's still useful. The model itself is still useful. Christopher: I think there's a question too of computational efficiency versus practical efficiency. if you need a if you need a machine that's got a storage capacity larger than the number of atoms in the universe, like that's probably not going to be computationally efficient to that. It's kind of embarrassing. That's Zach: Yes. That's embarrassing. Marty Weiner: Yeah. Christopher: an oopsie in the for model. well actually a good good good segue, Flea, because the next thing we're gonna talk about was so OpenAI had a recent new blog post about How they're going to be working with academic communities. And in this particular case, they've switched their model a little bit. So instead of doing a gigantic 700 paper drop on GitHub, Marty Weiner: Yeah. Christopher: they've decided to start to work directly with researchers. and there's some interesting quotes in this in this in this post. It's definitely a good read. It's in our view, these manuscripts are best understood as a form of raw research data. We reached out to external experts and asked them to act as shepherds, organizing groups. To understand the new results. So I think this is, I think this is fitting with the notion of rather than actually just doing a giant paper drop and saying, here, y'all figure it out. It's actually saying, we know this is probably a really good lean proof with a really crappy prose attached to it. So why don't you spend some time trying to figure out what this means and how to write it in a way that humans understand? But non-condescendingly. Like we think this is important. Marty Weiner: Yeah. Right. Flee: You Christopher: one of the actual so there's they they identified three problems in this one. the one that I understand, or at least vaguely understand, is one that is actually one of the Clay Institute Millennium Prizes, which was around quantum quantum chronodynamics. So quantum chronodynamics is the theory of what how do how do the nuclei of atoms assemble themselves? How does strong force work? And they basically are saying that they have a proof from sort of first principles of how QCD works, which is a relatively new thing in physics. Doesn't actually have implications. It's actually very close to what Flea mentioned. We have working models of how to do QCD. No one doubts the QCD exists, but we don't have a strong mathematical model for why it exists and how it actually forms from much more basic principles. So very interesting results possibly coming from that one. And in this case, they found a bunch of Nobel Lureates and some famer same face famous physicists who are gonna basically bang on this proof for a while Flee: you Christopher: and figure out if it actually has validity. Marty Weiner: Do a physics. Christopher: Go team physics. Okay. let's jump into the news and economy section. Zach, you wanna talk about the economic payoff associated with the LMs? Zach: Yeah, so there there's rant no no no I'm I'm s yeah, yeah, I'm saving my rant energy for the the the SEC filing. Marty Weiner: I think the the word the word is rant. Do you want to rant about it? Christopher: Rant. I'm sorry. Well the rant the rant's next. We're gonna talk about payoff first, then the rant comes second. We're gonna we're gonna end with the rant. Flee: You Marty Weiner: Okay. Okay. Sorry, sir. Zach: yeah, okay, so there was this thing out of Apollo that was, you know, basically saying, look, the mag seven are are are of a a lovely productivity story, but it's not spreading to the rest of the market, which like prima fasci is like a little bit bogus because you're it's kind of like saying, Well, if you remove LeBron James from the team, they're just not very good. It's like, well, this is a huge part of the economy. If you discount it, you can get whatever result you want. but you know, they they did have some cool graphs they went through showing this. probably the most interesting one was one they titled No Signs of AI Boosting Productivity Growth. So essentially, you know, if you look at the overall economy, productivity, which is just a measure of how much more GDP we get per like time and money, that is looking up. But this other metric called TFP total factor productivity, which tries to measure is technology actually boosting productivity versus other stuff that's less interesting? That's basically at zero. but without getting too into the weeds, I think an economist would tell you look, that first number where you just divide GDP out, like that's a more reliable number. It's much less noisy. TFP is much harder to measure. So this is actually, I think, kind of debatable. and and there was actually a a different post by a group called EPB Research, which I thought was much more interesting. They use data from BEA and BLS. These are just basic big statistics groups in the US bureaucracy that, if you're a macro nerd, you just can comb through and be delighted. they have this cool graph. it's titled Three of the Heaviest AI adopters doubled their productivity growth. So, three in this case is like types of industry, meaning like professional services, information, and finance. If you look at those post-Chat GPT, post-2022, they've been growing at five percent. Productivity growth per year. And to put that in perspective, that's never happened Marty Weiner: Wow. Zach: to the broader economy. If that happened to the broader economy, you should just freak out, lose your mind because we're all going to be incredibly rich. and so but but even then, if you look at that graph, they also have one of the broader economy, and they're trying to say, like, well, it's you know, like this looks kind of like a sad story by comparison. It's only growing at like a little under one and a half percent. But even it has accelerated post-GPT, right? So like there's this really bad slow rate of growth in the broader economy after the Great Recession. And it's picked up even in the sectors that are not most exposed to AI. So I find this to be a broadly positive story. we don't know to what extent it's expanding out, but it does seem to be expanding out into, things like banks and consultancies and law. so for me, you know, it's it's frustrating it to to like want to tell a good story, but to have to tease apart a lot of data to find it. But this isn't that much teasing. So I thought this was exciting. Christopher: And we've we've Flee: Zach, I'm kind of curious, like, how does this compare to other big technological shifts? you know, because I remember Zach: Yeah. Flee: some of this same conversation happening during the initial dot-com boom and, people getting connected on the internet. And Zach: Mm-hmm. Flee: even, you know, there were similar conversations about just the introduction of computers into business. Zach: Yes. Yeah, yeah. So there's this famous quote from Robert Solo, a famous economist who's who said you can see the computer everywhere, but the productivity statistics. But actually I I don't know the exact year. so he was he was talking, I think, from the late eighties. But what turns out is the productivity stuff shows up by the mid nineties. And you can cut you know there's a debate as to why that is. It's like you have a new technology, it takes a while to adapt to it. Maybe you even lose productivity for a little while. That's called a productivity J curve, because you're just trying to figure out how to use the new system and the old system was pretty well developed. but but actually once you hit the nineties, productivity goes up. It gets close to three percent a year, which is is like what we had in in in the US post-World War II. people remember the fifties as being a time when things changed a lot for the better, at least in terms of people's wealth. and so we actually hit that roughly from the mid nineties up to the Great Recession. And then after the Great Recession, it's been pretty disastrous. It's been like one and a half percent a year. and and that despite the smartphone, despite all the technology that happened in the 2010s, it's been pretty dismal. And then post-COVID, it's kind of ticked back up. And and especially post GPT, it's now around 2.1%, 2.2%. I haven't looked at it lately, which is about the long run average. so actually I I find the productivity data roughly comports with your view of history, right? So if you think of like the 20s as like a time when technology really changed, you got electricity, combustion engines, etc. 20s and actually after the Great Depression, productivity went up a lot. so so technology was really changing. and and then the fifties it just went wild. And then periods that are generally remembered as bad, starting in the early seventies through the eighties, really were low productivity growth times. productivity really matters for people's well being 'cause if you can't expand the pie, we can only get more by taking from others. We can only, change the distribution. so so so to me this is this is a pretty good sign. It's not yet like the singularity or whatever, but it's a pretty good sign. Christopher: I do think it's also funny in some of the graphs you're showing that there is a noticeable productivity bump at the point of the pandemic, which I think is an interesting Zach: Well, okay, Christopher: interesting shift. Zach: okay. But it's okay, that's the not to get too nerdy, but that's compositional. That's that's like look if you fire all the waiters, but you're still hiring your software engineers, it looks like productivity went up. it it is not in Christopher: okay. Okay. Zach: people didn't suddenly well I mean, we might have gotten more productive because of work from home, but I think you could see how it comes back down, and that's because the compositional effect is going away mostly. Yeah. Christopher: I see. I see. Interesting. okay. Well, you know, we all know that that's that's all coming because Anthropic is shoveling money into some sort of an AI furnace. I think they're literally using the money Zach: Yes. Christopher: to burn to generate energy for the GPUs to run, if I remember right. That's I think how this works. So, so Zach: Yeah. Christopher: so so Zach, really like why you know, Anthropic's S SEC filing was terrifying because they were losing a ton of money. What's going on? Zach: Losing yeah. And they're losing eighty billion dollars in one half a year. That's a lot of eight that a lot of billions. Yes, Christopher: Eighty billion dollars. That's almost one hundred billion dollars. Marty Weiner: Eighty billion. Zach: that's right. Flee: No Zach: so you know the old saying about like you know lie gets around the world while the truth is still getting its pants on? I feel like in this case the the the truth was like it still had its underwear around its ankles and was like falling on the floor. Like Flee: What? Zach: so Yeah, sorry, sorry. It was it was nude on the floor, passed out in a pile Marty Weiner: G give us more analogy. Zach: of its own spittle. no, but it's not so I don't know. But listen, so look, there were a lot of headlines and silly tweets and and posts on Blue Sco where people were like, my god, you see AI is all a scam. They can only do this because they're burning eighty billion dollars every half, and it's all gonna go away. Well, here's the thing. So pull up, you know, you guys all have the SEC statement in front of you. Pull up consolidated statements of operations data. Okay, look in the column under operating loss. The biggest loss is what? Change in fair value of convertible notes, et cetera. Okay, so I I I thought of an analogy that would explain. So so 73 almost 73 of those 80 billions are in this one category. What is this category? So here's my analogy. Imagine tomorrow Marty starts a company to build tabletop fusion, and we all feel bad for him because he's an idiot. And and and so I go over Marty Weiner: Yeah. It's gonna work, dammit. Zach: and I go over and I say, Marty. Christopher: Look at this glass of water. Zach: Yeah, it's looking great job, Marty. I want to buy one percent of your company or yeah, and your company is worth a hundred dollars right now, and it's valued at a hundred dollars by the stock market. You give me a note that says, Zach, when we IPO in the glorious future, you can have one percent of my company. I hand you over a crisp one dollar bill. All right. You on your accounting Marty Weiner: Wait. Zach: say I have a dollar in liabilities because Zach has this convertible note for when we IPO. Now let's say the next day you say, Claude, do a tabletop fusion. And and it prints out a fusor that you put on your desk and it works, and the company is now worth a hundred billion dollars. That means my convertible note, your liability to me is worth a billion dollars. And what is the headline? Marty owes a billion dollars, a billion dollar loss at Marty Co. Right. Christopher: It cost a billion dollars to do a fusion. Zach: So what actually happened Flee: Yeah. Zach: in Anthropic, you can see like if you go back to twenty twenty three, they have like a a billion, a mere billion. Dollars and convertible notes, those got revalued because the company is worth a crazy amount of money now. So now they're 72 billion because the company is worth more money. So technically their liabilities are higher because people have convertible notes for chunks of the company when it IPOs. But when you say $80 billion in loss, what people hear is they burned money so that people could make their stupid animations and stuff. But most of that money was not loss in that sense. And in fact, they were profitable if you ignore. the the convertible notes, et cetera. And what what drives me crazy, and this will be the end of the rant, I swear, is you get people who are like, shut up about your accounting wizardry. Loss is loss. And it's like, no, you have to actually understand things. you know, Marty Weiner: No, it's not. Zach: what it's it's not a it's not like Enron stuff. Like they just have to report these liabilities. If if it was stock and not convertible notes, it would not be listed as loss. so and through the the the upshot here is anthropics is doing pretty great. Now It doesn't mean it's not a bubble, doesn't mean they can't cover their losses, but everybody was getting this wrong. Smart people who should know better revealed themselves to only be reading headlines. And that's the end of my rant. Marty Weiner: Alright, hold on. Make me a table Christopher: So so this is for the so for the Marty Weiner: top fusion. There it Zach: Yeah, yeah, yeah. Flee: Yeah! Christopher: So so so for the Marty Weiner: shoot, I gotta log in. Christopher: so so Zach for the for Flee: We are here! Christopher: the finance nerds out there are we're talking about basically the differences between adjusted EBITDA and like gap profitability. So they're probably like they're EBITDA on an EBITDA basis, they're profitable. On a gap, like they're way off of it, but it's because they have a giant lia liability from all of that equity they've issued over the years that they have to basically Zach: Exactly. Christopher: sell off. Zach: Yeah, yeah, because they're doing great. it's it's absurd. And so peop people will hear this and they'll be like, Wait a minute, you're saying if you ignore the losses, the company is profitable and it's like, Yes, because you have to ignore these liabilities due to the company becoming more valuable. It'd be absurd not to. Christopher: Okay, that's that's that's really hilarious. That's really hilarious. Zach: Yeah, everyone was getting this wrong. I I couldn't believe it. I I was losing my mind. Christopher: well let's let's let's let's end on a quick quick high note on this section. so we couldn't quite find a right good section for this one, so I'm gonna do it here. apparently the switch two has been jailbroken. This is like a and Marty Weiner: Yeah. Already. Zach: Yeah. Flee: Ooh, nice. Christopher: and this is actually the kind of thing where like I would say Nintendo since the Wii has gotten really good at preventing these kinds of jailbreaks. Marty, do you have any details on this one? Marty Weiner: No. Zach: Ha ha ha. Christopher: No, damn it. Well, I have okay, so I have I have seen the the nice thing about this one, this was one of those AI assisted in a loop exploits where the the person who did the jailbreak legitimately built a a a little robot to push the power button on the switch. Flee: nice. Zach: No way. Christopher: and so they could actually have in a loop. automated ability to try to test it because like apparently clicking that button is part of the reset process if you want to try and do you know, do the assisted jailbreak. For those of you who've ever jailbreak on a phone back in the day, it was always this like delicate, like hold the button down for five seconds and then do this other button. And this they basically Zach: Putting the Konami code. Christopher: built a small robotic rig to do that, to get a to to do a to do an exploit. So Zach: Why would you want to do this? Marty Weiner: No, I Zach: Is is this just cool? Marty Weiner: you can run whatever you want on the switch to at that point. Christopher: this is yeah, Flee: Yeah. Christopher: this the the switch two is a general computer and you could start to run, you know, I think I think this is like a install install Linux on it. You could run you could run you Zach: That'd be so awesome. Christopher: could run Doom on it, you could run Quake on it, you could basically run the you could run Steam on it, which actually is the thing that Nintendo probably has the biggest problem with. Flee: I think one of Marty Weiner: I Flee: the bigger compelling things is just emulators, so you can actually just run your old, you know, Game Boy games on it. Marty Weiner: no, I I actually do have something on that. I remember hearing a guy talk about how it was cracked after just about 500 days, which everybody thought would take a lot longer because after the switch one was cracked, they they they put a lot more effort into it. And I think the other thing to know about this is it's not a full jailbreak yet. I think it's a tethered jailbreak, meaning every time you reboot it goes back, but it's the beginning of the how how you fully jailbreak it. Christopher: Always it's always the start. And the the difficulty in these situations oftentimes is that once it becomes a hard if it's if it's hardware dependent and not software dependent, it's very difficult to like you've got a whole bunch of devices now sitting in the wild, they can't patch. so this could be a really, really interesting future here. okay, let's wrap up as is our custom with our funny weird section. so here's here's what I found. there's there's a there's a there's a archive paper on why AI pros feels strange. and I think you know what what they mean here is like I don't know if you've noticed this as well, but like AI, AI prose is distinctive and somewhat convoluted as you're reading it. And the question is the Marty Weiner: Yeah. Christopher: question that the researcher are trying to address here is like, okay, we all get that feeling. What are the features and is it actually strange? And their answer is actually yes, it is. And so the list of things that they included were basically there's a larger, you have a larger cognitive load on reading AI prose because it does things like. It generates word combinations that are very unusual. Right. So n gram statistics are all over the map. And so so short, normal phrases are very uncommon. it also does things like it generates they said an example of denominal verbs. This is in so for the grammar nerds in the room, denominal verbs are when you take a noun and you make it into a verb. like you say, you know, this the Marty Weiner: You verbify it. Christopher: you verbify it, the the the store was shuttered. Right. you you start creating and you start adjusting it. and also they found that it does things like stacking nouns in really interesting ways. And so, you know, in English, it's perfectly reasonable to use a noun as a modifier for another noun, but you don't usually see those in like stack in like groups of two and of like three and four. You do find that in AI pros. Marty Weiner: Mm. Christopher: and so what they were trying to dig into is like, why is this happen? Well, the suspicion is they don't really so the answer is. We don't know because of course AIs are mysterious, as Flu mentioned before. We don't really know how the brain works. but the suspicion is this is an artifact of a certain amount of compression in the output. The benchmarking is probably optimizing for a certain type of succinctness, like basically complete as as possible, which means that the AI models are tending to use kind of more high-falutin language, like basically more specific words that are more precise. But also therefore less likely to be in common usage. you know, think of it as like the AI the the model effectively has the entire dictionary available at any given time to provide an answer and it's really flexing the full use of the dictionary. Marty Weiner: Mm. Mm-hmm. Zach: That is fascinating, partially I think that's the opposite of what you would have guessed. You would guess it would use statistically common engrams. Christopher: it it's it's likely this has been a shift in the models over time. as they've gotten more intelligent, they're actually Zach: Yeah. Christopher: tending to use more intelligent words. which is I guess maybe to say that when smart people use long words, we're not being douches. I don't know. I don't know, I wouldn't go that far. That's yeah, yeah. Zach: W I wouldn't go I don't go too far. Yeah. But I was gonna say, Marty Weiner: Ha ha ha. Zach: like, so there I let's not get on a sidetrack, but there's been this big controversy this week in my like tiny world of literary people where there's this French book called It Was That or Death that they saw I forget the French. Anyway, it won this giant prize, several giant prizes, and then it turned out at on some level the guy had used AI and denied it. And was interesting. As with many other books by AI or seemingly by AI that have won awards, is they use this pseudo-poetic style where they have slightly out of distribution word pairings. And we were I was debating with some friends about what's going on here. why is why is AI good at this particular flowery, half nonsensical prose. And now I'm wondering if it's just if it's just this sort of effect where it's good at using slightly out of distribution engrams. Christopher: okay, so next up we have Warhound. so apparently as a company that has announced that they're going to try to Zach: god. Christopher: try to build the better military working dog via via a combination Marty Weiner: Yeah. Flee: god. Christopher: of advanced genetics and defense technology. I don't know if they they don't quite mention cyborg in here, but it is definitely one of those god moments of let's genetically Marty Weiner: I'll add Christopher: engineer some dogs. Marty Weiner: Palmer Lucky is is one of the people Flee: W-w-w-wait a minute, Marty Weiner: in you building it. Zach: Yeah. Marty Weiner: and just for Flee: Of course he is. Marty Weiner: anybody who d doesn't know who he is, he's basically the one who kicked off the VR mini revolution of a about a decade ago or more. Zach: Yes. And now he's genetically breeding war dogs. So Flee: What did they call the creatures in Snow Crash? Isn't this exactly from fucking Snow Crash? Yes! Yes! Christopher: this is the end of s the ro the r the the the dogs at the end of Snow Crash. Yeah, no, Zach: my god. Christopher: that's exactly this. Zach: It's nice that science fiction is inspiring Flee: No, I was like... The Zach: young engineers. Marty Weiner: Did Flee: cyberpunk future was supposed to be fun and fantasy. I never wanted Zach: my god. Flee: it to be real. Marty Weiner: It's funny right now. Christopher: okay. Zach: It's funny right now, yeah. Christopher: okay, in other in other news, apparently we have one of the first attempts of a person to use an AI agent as a lawyer in proceedings. so and you can see that it's the like Marty Weiner: Yeah. Zach: So handsome lawyer. Christopher: it's the standard like interviewing interviewing IAI agent that they repurpose as a lawyer. And it's just a Zach: my god. Marty Weiner: All I see is that the the judge is is a little upset right now. Zach: Yeah. my god. Flee: is there anything wrong? mean, outside of the obvious, hey, that may not be reliable, but is there anything? Are you not allowed to do that? Why can't you? Marty Weiner: That's a good question. I don't know. I might just Christopher: technically you don't have to have a you don't have to have a law degree to def like you can defend yourself. Flee: Yeah. Christopher: I wonder if this falls under the defending yourself situation? This is just a Zach: Yeah, but see the Marty Weiner: Uhhuh, like a Zach: the problem is is next year the judge has a judge bot and and Christopher: And the two of them just spend the entire time trying to figure out how to hang up the Zoom call. That's the natural Zach: That's right. And it never happens, Flee: Yeah, yeah. Marty Weiner: Yeah, right. Zach: yeah. Marty Weiner: It's only a matter of time. Christopher: let's okay, la last but not least, there is a a great this is a parody video on John Wick trying to get his insurance claim after the incident Zach: Ha ha ha. Flee: I love it. Christopher: at his house that we all know is Marty Weiner: This is fantastic. This is such this is why we have AI is to is to make these kind of videos. That's right. Zach: That's right. This is this is why the productivity numbers aren't higher, is everyone's just on C dudes. Flee: Yes. Marty Weiner: Fantastic. Christopher: All right. that's thanks everybody for another another fun week. we'll be back next week with more in the AI news because we just know it's gonna keep coming faster and faster.