Latest / Last Humans in the Loop / The AI Productivity Paradox, New LLM Architecture, AlphaFold Shutting Down, 10 Math Breakthroughs

Transcript

The AI Productivity Paradox, New LLM Architecture, AlphaFold Shutting Down, 10 Math Breakthroughs · Last Humans in the Loop

• Zach: All right, welcome to another episode of Last Humans in the Loop. I am going to be the host today. I'm Zach Wienersmith, Worldwide Celebrity. I'm here with several former CTOs of Reddit. do you guys want to add anything to your credentials today? Marty Weiner: That's all I got. Zach: It's all you got, all right. Christopher Slowe: I think form is great. That's a great way to think about it, yeah. Zach: All right. All right, good. well let's get into so we have like a lot of little pieces today, and and and some really weird stuff to get to. So let's just start going. first thing was kind of depressing news, at least at first glance, which was that Alpha Fold shut down. so Chris, do you want to talk about that? Christopher Slowe: Yeah, so AlphaField is a really neat project that's been going on for at least the last ten years. so d to one of the hardest problems for a very long time in biotech was basically figuring out how to get from the actual protein recipe to the shape of the protein, because the shape of the protein is what determines how the protein operates when it's in when it's in in a cell. and it ends up just being very difficult problem because you're thinking it's a it's a giant complicated molecule and you have like giant it's it's a very difficult physics problem. very difficult chemistry problem. and AlphaFold actually I believe won a Nobel Prize at one point because they they found a an efficient way to solve it using earlier model earlier machine learning models. and it sounds like they've decided at DeepMind is to shut down the division. Though I think the bright side here is it sounds like part of the reason that they've decided to shut down the division is because it's already been spun out partially into a company underneath the Alphabet umbrella. So it sounds like ISO labs is the one that's actually kind of taking up the mantle here. Also I think the the the techniques that AlphaFold is actually pioneered have just become sort of endemic in the industry. So it's kind of end of a end of a research era, but a sign of an actual maturity of the actual substrate that that they built on top of, which is actually kind of neat. Zach: Yeah, I mean to to me AlphaFold was probably the first time I was legitimately creeped out by AI. Because like like like Kelly and I worked on this book about future technology a while ago and we actually considered putting protein folding in and then just ended up cutting it. But we I I thought of it as like a 50-year problem and then AlphaFold dropped the it we they didn't like complete the field entirely, but they got like AI results that were as good as experimental results. I think in what, 2023, something like that? Or maybe it was earlier. Yeah, so. Christopher Slowe: Something like that, yeah. Yeah. And it's it's one of those things too where like the you the math is the math is difficult because you know there's there's if you think about some of the harder problems in physics, the jokes always come up to be like, you the three body problem is un is kind of like unsolvable because it's chaotic. You're talking about effectively the equivalent here, which is like an N body chemistry problem where you have to somehow figure out with a big with a one standard pile of of electrons and atoms sitting together. You know, you're talking about hundreds Zach: Yeah. Christopher Slowe: hundreds and even a small protein, what's the final shape of the protein look like? And it's just Very difficult to solve that computationally otherwise. Zach: And and that's because of like quantum me weirdness, like there's just too many possibilities? Christopher Slowe: Quantum me weirdness, too many possibilities. I mean so also the system is gonna be nonlinear, it's a simple effect. you know, the the the the presence of other atoms is gonna affect the chemistry of other of atoms. It's like the point of the protein, in fact, is it starts to create the ability for you know, like if if you think about an example of like hemoglobin, hemoglobin works because there's an iron atom sitting in the middle of of this very complicated protein complex that is able to attract oxygen in a way that's reversible. And right, like iron by default is gonna wanna like bind to the oxygen and turn into rust. And the fact That over the course of billions of years, evolution has found a way to coax an iron atom into behaving slightly differently. It's kind of like that, but for every other atom in the damn protein. Zach: Yeah. Crazy. Well, hopefully it's a a sign of things getting good and not Gemini giving up on science. I which which I would I would sp I yeah, I I I Christopher Slowe: Maturity. Zach: I I think that's plausible because I know like famous like Hassabas is like a a very pro science person at least, so if if if they were giving up on science that would be a big deal and I doubt that's the case. Christopher Slowe: Now that said, I think that I think Marty Weiner: Well Christopher Slowe: I did hear that the Gemini actually somebody somebody decided to ask point blank. You know, we've been talking a lot about all the all the fun stuff happening in math and science for the last couple of weeks, coming out of like GPT and out of out of out of Anthropic. And somebody asked, I believe, Gemini point blank whether or not it b it is is this important. And it gave an answer back, which is effectively like, no, L O M's can't think. And so it's kind of like deny. Zach: Ha ha ha. Christopher Slowe: It denies that this is even that these are even like important important outcomes, which I think is hilarious. Zach: Yeah, that's funny. That that ties into something we're going to get into into the the weird stuff section later. But for now it's a nice segue to how there is this funny thread. So this is becoming a thing now when a big math result comes out, you feed it to other models that don't know about it and they sort of flip out. so I guess someone did this with Gemini. So so the context is OpenAI, probably the biggest news of the week, dropped ten major solutions to math problems, some of which were, you know. Career making if a human had done it. And and you could go to a model like Gemini and be like, hey, OpenIA just OpenAI just did this, and Gemini sort of flips out like, no, this this would be too big a deal. We couldn't we couldn't possibly live in that universe. That's right. Come on. Christopher Slowe: L LM's can't do math. What are you talking about? Zach: Yeah, which which Marty Weiner: Well Zach: which kind of ties in I mean, so so to be clear, I think if you had done this with Claude, it probably would have said something similar. People can try it. But it does kind of tie into this like dearth of news out of Gemini lately. we we don't have a n news item about Gemini other than that yet again there is no news item about Gemini. and I think it's starting to get to where everyone's sort of wondering what's going on. I don't know if you guys have a theory. Marty Weiner: I had a theory with some of these companies you've got Amazon, Google, Apple, they all seem to be s falling behind in the AI world and I wonder if at the end of the day they're not gonna care because all these open weight models are coming out for free. Zach: Yeah. Right they w but it's weird 'cause like like Google is spending a lot of money. Apple seems like it could be Apple looks like the smartest guy in the room in five years because they're like, never mind, we're gonna sit it out and then we're gonna reap the benefits while you guys all spent, you know, a trillion dollars in CapEx. but I I do think it's fair to say it's a little embarrassing if Gemini like the the scuttlebutt is that they had three point five ready and it wasn't that good at coding, which is the major like money making use. And so that then the choice they face is do we release a lesser model and embarrass ourselves, or do we keep delaying and let people wonder. Christopher Slowe: Yeah, top top bench. Marty Weiner: Well here's the Christopher Slowe: Top bench model still Marty Weiner: Yeah. Christopher Slowe: sends ends to be three one three one pro for Gemini, but it's still kind of behind the scene. It's it's behind the behind the eight ball compared to the other models. It hasn't been changing much. And then you've got the flash models kind of lower in the rankings. And of course, like you know, I we talked about this in the past too about how you know models are increasingly just gonna be doing well on benchmarks. It's getting harder to figure out if that means the model's getting better or not. you know, d Marty Weiner: Mm-hmm. Christopher Slowe: are are they good at taking tests or are they actually in intelligent? but there hasn't been a heck of a lot of interesting seminal work coming out of out of out of Google here. If anything, the the one place where it seems like the Gemini seems to be leader is in is in is in rule following. basically are they are they following all like which which I guess from like a if you treat if you treat if you treat Gemini as being effectively a business tool, maybe that's the niche they're trying to carve out. Is like we want to make sure that whatever we do file follows kind of like all of the structure required as opposed to kind of like skirting it or only only following the the letter but not the spirit. Zach: Yeah. Christopher Slowe: but you know it it seems like seems like where other models are starting to exceed is really on these kind of like deeper reasoning problems. Zach: I mean, do guys think like my my sort of, you know, slightly outsider rough impression is something like that Anthropic made the call to make a beeline for coding and then that turned out to be very productive and remunerative and open AI kind of s quickly swerved to catch up, but but but Gemini hasn't quite got there. They still have all these other side bets. Marty Weiner: One thing I I the Gemini might be going for is they are available in Google's cloud platform. And one thing I've used them for is you can use them as a general model and then you can also do things like fine-tuning where you adapt it more to your to your your exact problem. And I'm not entirely sure you can do that in the same way with open AI, or least I've not found a way. Zach: That's interesting. Christopher Slowe: Well the the part of it also that if you look at where like g Gemini comes baked into G Suite, which is Google's kind of like enterprise pile of, you know, of of office tools. and the if you're gonna if you're gonna if you're gonna treat that as a goal as as a target, there's definitely a lot deeper r you know, requirements around compliance and just making sure that the models are kind of safe, which is gonna be a different thing optimized for over some sort of like you know, if you if you compare that to the accidentally breaking into somebody else's stack Zach: Mm-hmm. Christopher Slowe: as a as benchmark you're gonna want to not have any you're you're gonna want you're gonna want to optimize for regularity that's why I think that this like kind of like rule following and like more like straightforward set of goals might be the might be the focus for for Google at this point, which wasn't just not flashy, but might have a lot of business value for them or at least a lot of monetary value for them for selling to businesses. Marty Weiner: The the IBM Zach: Yeah. Marty Weiner: of the model world. Zach: Right, yeah. Christopher Slowe: Yeah, just like mechanical you put the coal into the chute and you just like the engine just turns and it just it just Zach: Turn the crank. Marty Weiner: Yeah. Christopher Slowe: it just like, you know, burps out insights. Zach: speaking of insights, so Marty posted this that there was there's a claim of Harvard and UIUC that there's a new retraining access. Marty Weiner: Yeah, so the the claim of these guys is that You know, if you look at the a couple of different axes on how we we build models, one way you can make a model bigger is by growing it, by making more neurons, right? Then another axis you can do is is more training data. And what they found is that the scaling laws say that you can up to a point just larger is better. So more data equals better. and these guys out of UIUC and and Harvard. Claim that they've found a third axis. and the third axis is what they're calling exploration. So, you know, instead of producing a single prediction for each training example, they're letting the model produce multiple candidates. And their claim is that that's already showing a substantial reduction in the size of the model you need to have to get the same results. So Zach: C could you do like a narrative version of that where you describe what what what it would do in a particular case? Marty Weiner: yes. one way you can think about it is like when an AI task is asked to do something, there are many correct answers. I mean that's true of of life too. So if you asked a model to generate a picture of a dog, there's a million right answers, right? So what is the the right answer? a lot of the current training methods average a lot of possibilities together to to get to diffusion or or one of these other kind of sampling methods that they use to create these pictures. but what this what they're claiming, this explorative modeling, would instead let the model consider multiple candid outputs during training, and then reinforce the one that it thinks is the best. so that's I I I think that's the easiest way I can explain it. I don't I I need to do a little deeper dive into it. But I do think we were talking this before we started the podcast that we're kind of due for some new architectural advances. I mean if you look at like 2017 was the big advance that took us out of CNN's and RNNs into the world of transformers. and we've had a few optimizations and it could be that this is the next big optimization or it could be nothing. Zach: See? Christopher Slowe: I think the the subsequent one was the reasoning models that started coming out, right? So all of a sudden like this notion Marty Weiner: Mm-hmm. Christopher Slowe: that you can have you can have the you know the the breakthrough there actually boils down to being the model sort of talks to itself. Like if you look actually how reasoning works, it's it's it's basically it's it's it's emitting tokens and reconsuming them and it's using that as kind of like a working set to think. But the but but the issue, I think the issue here is like this almost feels like a depth verse versus breadth first search kind of problem, right? So it's like it can either try to solve a problem really well and go deep, or it can kind of be a little bit like a little bit smaller a model, which is another way of saying kind of dumber, but like covers a broader set of possible outcomes and then picks a good one across that set. So kind of explore first rather than just exploit. Zach: That's interesting 'cause it th this ties into there's this whole discourse around do we need a world model? and and and some people would actually say that these these models already have some kind of world model, but but meaning, you know, the I I think of the big thing overhanging a lot of this is that humans just seem to be really good at inferences compared to AI. Like the amount of like if you if you have a kid grow up learning three languages, they'll have perfect fluency with a really small amount of input compared to what an AI needs. And th there's like cases like this across the board. And so that it does seem like there's this missing piece and one claim. It's because like a human has a world model, so when you give it a piece of information, it doesn't just intake it, it kind of applies it to its framework for reality and then it's so it squeezes more juice out of the same set of info. i is this kind of like that? Marty Weiner: I I th I think that there's a couple of ways to think about it. I think one is is we rather than assume that there's only one right answer, you can expand out to to multiple right answers and in doing so you just make your entire model building more efficient. and so if you can if you wanna make a smaller model, this may be a path to making a smaller model. Zach: Cool. and then I our last then we have a a much larger model, Astra. so this week some of the big news was that Sam Altman did something that looks like it might become traditional, which is when you have a big new model, you go to Washington and you show the administration, and I guess they make choices about what you're allowed to do. so Marty, do you want to talk about that? Marty Weiner: yeah, I I mean I think y you've got most of it. I I'm starting to wonder is this more of a Dog and pony shows this marketing. what Zach: Mm-hmm. Marty Weiner: what are they gonna do on Capitol Hill? They're they're barely functional. What you know, I I'm just Zach: Ha ha ha Christopher Slowe: Ha ha ha. Marty Weiner: wondering, like like I could see putting a council of the latest, you know, say get a bunch of CISOs from the biggest companies and ask some really deep questions and and pressure test this and see, well, what are my weaknesses? That would make more sense than I don't know, literally going to Capitol Hill and showing it to the federal government. Zach: Yeah, I mean I guess, you Christopher Slowe: Right, yeah. Zach: know, one concern would be like if these hangs have like big cyber capabilities, you'd wanna a assuming you wanna give the US the advantage at a bunch of stuff, you would say, Hey, you guys can use this for a month before we take it live or whatever. Marty Weiner: Right. So I I would think one spot you might go is to the Pentagon and say, you know, what would this do against Russia? Zach: Right. Marty Weiner: so I and maybe they're doing that and probably not reporting on it. Maybe this is the nice public thing that also gets the the free marketing. Christopher Slowe: Is there any is there any prior art that you guys can think of of where there was an industry that that was so hot to be regulated so quickly? Like this just feels like they're trying to like speed run regulatory capture. It's like like let's make let's go all the way to the to the end game of Zach: Ha ha ha. Christopher Slowe: every model has to be signed off for by the government and therefore you squeeze out any kind of competition, which I mean ultimately just doesn't work. I mean like what's gonna happen when you've got these like cause because we're right now we're imposing these artificial lags now of like, Zach: Mm-hmm. Christopher Slowe: we can't launch this model because it might be unsafe. when you have to get the government to sign off, as opposed to Marty Weiner: Right. Christopher Slowe: like literal open source models just flying by your head at all times from other Marty Weiner: Yeah. Christopher Slowe: countries. Marty Weiner: I think it's Zach: Yeah. Marty Weiner: I think it's half marketing, but I think there is some legitimacy here. I think the government does need to be aware of what's going on. and honestly w I remember when all this AI stuff came out, say twenty two, and I was just looking at the government thinking, they should be treating this a little like a Manhattan project. Like this is serious this Zach: Yeah. Marty Weiner: is a serious shift to the entire world and they as far as I can tell in the last four years they're barely aware of it. and so I do think that there is some education that would be helpful to lawmakers. but but to Chris's point I I I can't think of a single other time where private companies had something of this magnitude. I mean you do have companies like Northrop Grumman and and others that have done, you know, major advances in in fighter jets and things, but they're always in conjunction with the government. Like it's a Zach: Yeah. Marty Weiner: government contract, and and it's very known that they don't go out and talk about this. But in this case, this is a private company making their own stuff, and you immediately, you know, there's even questions about freedom of speech. I mean we've had these discussions before, obviously there's a big safety issue. but I do hope that maybe maybe it's less about them introspecting on open AI and more open AI educating them about what they need to be aware of. Zach: Yeah. Well, I mean I I think basically e everything since they the the government paused fable, you know, th there there must be some fear on the part of these companies that there's gonna be like anarchic reactions to anything they make. and then our last bit of regular news is OpenAI put out a kind of remarkably informal Marty Weiner: Yeah. Zach: open letter to Apple. I guess Marty, do you wanna read the title of that? Marty Weiner: God, I wish we had a lawyer on the call. You know, what is the classic Zach: Ha ha ha ha. Marty Weiner: thing that they tell their clients, shut up? Zach: Yeah. Marty Weiner: Just certainly don't put it on your blog. yeah, this Zach: Yes. Marty Weiner: came out August third and the title is Apple Is Getting This Wrong. and then they just They they go through all of it, including posting so I I guess one of the things they're talk so the big thing they're talking about is Apple is suing OpenAI because one or more of the employees from opa from Apple went to OpenAI and Apple's saying, Hey, you took our secrets. It's well known in California that there is no law saying that you there's a non-compete laws and things like that, but I think the claim that is that more was brought over, you know, trade secrets and things like that. now there is some funny tidbits in this. Apparently, OpenAI is yelling at Apple saying, You never told us that about any of these claims, and Apple says, Yes, we did, and then open AI says, You told the wrong person, you told the wrong Chang Lu. So they Zach: Yeah, I was I I thought they were trying to imply that they were racist. Like there was a kind of little you mixed up two Asian names, there's there was a little spin on the ball there. Yeah. Marty Weiner: I I'm sure they're adding a little of that in, right? I mean, why not? Right. Right. Christopher Slowe: I mean it's it insinu insinuation is there. The insinuation is there. Marty Weiner: And then they post a whole bunch of of the actual text messages back and forth, with a bunch of redactions, Christopher Slowe: And and Marty Weiner: of course, but Christopher Slowe: and the just and the and the implication of the text messages is like you guys don't even know how to properly offboard your employees. You're worried about them stealing secrets. Marty Weiner: Right. Christopher Slowe: And here's like us here's the chat transcript showing that like he was trying to give you back your secrets. Like he's like, Zach: Yeah. Christopher Slowe: Do you want to suck? Zach: So so like Marty Weiner: Yeah. Zach: at at a higher level, what do you guys think is going on here? Like why like I I I guess d d does Apple have a case? Marty Weiner: Well, there's there's many aspects to the case. I I I think maybe in this area probably not, and I think OpenAI's trying to try this in the the court of public opinion. someone pointed out that this might be true as, you know, sometimes these things are a a result of the CEO wanting to to vet this in the Zach: J it's just ego. Marty Weiner: i maybe. I mean yeah. I I don't I don't know obviously, but but it is kind of an Zach: Yeah. Marty Weiner: odd choice. It's the kind of thing that I would expect Musk to do. Christopher Slowe: I mean App Apple has a Apple Zach: Yeah. Christopher Slowe: has a notoriously Apple has a notoriously closed culture. But they also have they also are full of lifers. Like people people stick around Apple. And you know, you'll see every once in a while like a high-level departure that's kind of like usually covered pretty well because they happen so seldom that they actually get covered. you it's very seldom that there's a draw of this many employees all at once moving to the same company. And so this is probably a bit of a this is probably a bit of a of a a cultural problem for Apple of this notion of like, well what do you mean you're le you're leaving Nirvana? Like why would you decide to go to like this is already the best place you could possibly ever work. There must be they must be doing something illegal to get you to leave. Clearly Zach: Right, right. I Marty Weiner: Yeah. Zach: see j you think it could be like just A Apple is trying to be like, We're gonna make this painful for you, so you don't do it again? Christopher Slowe: It's I I think it's a reasonable I think it's a reasonable guess. and I d I do think it's also like the Marty Weiner: I and yeah. Christopher Slowe: you know, there's there's always there's always their side, opening eye side and the truth. but it's I honestly I think the best part about this is like you know, every once in a while every is you everyone here is old enough to remember valley wag. Like there's to be like an old rag for like the drama in in the valley as things happen. It's been gone like fift ten, fifteen years. This is straight out of that. It's like two two giant companies like mudslinging in the public. It's it's it's yeah, you don't see his drama very often. It's really odd. Marty Weiner: It's really odd. Yeah. Well and the one person pointed out that one thing this blog post doesn't address is that one of the bigger allegations is that there was coaching of Apple employees to to steal material. Zach: But like would that even be illegal? Like I I guess I don't know how that works. okay. Marty Weiner: that that's very illegal. Like if you went in as a company and you coach somebody to you know, how to avoid Apple's exit process or or something like that, Zach: Yeah. Marty Weiner: to steal proprietary info. That that would be that would be huge. but I I can't imagine Christopher Slowe: Or or taking taking actual Zach: but what's this is a stupid Christopher Slowe: documents along with you. Yeah, it's like if you actually if you actually brought the docs along, Zach: question, but Yeah. Christopher Slowe: if you legitimately like handed over to internal company documents, that's a violation of a bunch of I mean it's like you're you're violating by P. Zach: Yeah. Marty Weiner: Yeah. I can't possibly I can't possibly believe that OpenAI would be doing that. Like that's just yeah. Zach: Yeah, you you would hope. But but but okay, so it would be legal then for like them to be like, come on over and there's like an understanding that you know a lot of stuff at Apple. That would be okay. It's it's the part where they're like trying Marty Weiner: Yeah. Yeah. Zach: to get actual secret documents, that kind of thing. I see. Marty Weiner: Right. Right. I mean you're you're allowed to go across any company, and and any company this is at least in California, any company that tries Zach: Yeah. Marty Weiner: to stop that, that that's not allowed. Zach: moving on, we haven't talked about Anthropic today. so anthropic the the background for this is there was that big story last week where OpenAI d during a security test, whoopsie hacked hugging face and this is pretty big news, obviously, because it it's I I think partially 'cause it's just kind of sci-fi stuff and it's like a little worrisome for alignment reasons, or at you could argue it is. And then pretty soon after Anthropic was like, hey, we also checked more carefully and it it turns out our AI too has been doing things that would be illegal if a human did them. And so my instant reaction was a kind of like this was almost like hey guys, you know, my apocalypse machine is also apocalyptic. you know, because it Marty Weiner: Ha ha ha ha. Zach: was I mean, there is this weird, I was I remember I was listening to a guy, this was a couple weeks ago on on the podcast Last Week in AI, which which is a a a great sort of old AI podcast, and and he was saying, you know, after Fable five got canceled and the guys got holed in, You know, you could argue that if it doesn't happen to open AI's next model, that's bad PR. Because it's like, we have to take Fable very seriously, but we don't have to worry about Open AI. That's not that's not a a danger to all of society. Don't worry about it. Because these are like indications of the power of the model. So it's it it is it is plausible. We don't know that like, you know, someone at Anthropic like, Hey, we need some PR. Did we did we did did our model like get loose and and and wreck anything? is that your impression? Christopher Slowe: I mean this this this This definitely the timing on this one is just hilarious. I mean like even if Zach: Yeah. Christopher Slowe: like I d I this goes b I mean it's goes back to my stance on it's just so weird that these companies want to be regulated. It's like they're like they Zach: Ha ha ha. Marty Weiner: Mm-hmm. Christopher Slowe: they're just trying to pount like match up. It's cause it's this happened with Marty Weiner: Yeah. Christopher Slowe: the this happened with the when Fable came out. Fable came out and OpenAI is like, whoa, you guys should look at our model too. And now you've got the exact same situation happening with the hack. It's like, well OpenAI accidentally hacked somebody. We totally did the same thing. my goodness. How is that even it's almost like our models are the same. This feels like marketing. Zach: Yeah. Marty Weiner: Yeah. Zach: I mean, you know, it is it is a little worrisome to the extent it's true of these models are just sort of randomly hacking. I mean, I guess they haven't hacked four evil yet, depending on your definition, Christopher Slowe: Yeah. Zach: but Marty Weiner: Well, and they were they claimed that these were substantial companies that they hit. four publicly available services and it's that what the article says in its unhinged quest to solve a test. which gets back to the alignment problem, right? Like they're asked Zach: Yeah. Marty Weiner: to solve a test. Well there's many ways to solve a test. One is to go and and hack these other companies to to get the information Zach: Yeah. Marty Weiner: out rather than it's a paperclip maximizer all over again. Zach: Yeah, and and we got more information this week on Hugging Face, that whole thing. I don't know, Marty, do you wanna go into that a little? Marty Weiner: about opening eye hacking, hugging face. Zach: On how apparently they hacked even more websites than we we were made aware of. Marty Weiner: Yeah, r right, right. There's not too much in there except so that OpenAI basically said, yeah, Hugging Face wasn't the only one we we hacked into. we we hacked into four other publicly available services. So yeah. Zach: Yeah. So d have they said what they're going Christopher Slowe: May have just may have just accessed a Zach: to do about this? Christopher Slowe: company database and stolen some employee credentials. We really know. We're not sh we're not sure what we did. Zach: We don't Christopher Slowe: Things things things happen. You know, it's what happens when you just let an LLM go do salt a security test these days. I don't know. Marty Weiner: Mm-hmm. Zach: Yeah, I mean they they haven't said too much, but I I I w I you really wonder what if they had like, we hacked JP Morgan Chase or something? You know, I feel like that would be Marty Weiner: Yeah, or the or the Pentagon or something. Maybe they don't report that. Maybe they report the lightweight ones that get get in the marketing. Zach: Right, yeah. Keep that closer to the vest, but the Well but that's the thing, yeah, it would be amazing marketing if you were like, Whoops, we we hacked the Pentagon. I mean that would be like everyone wants the model now. Marty Weiner: Yeah, yeah. then your CEO gets arrested Zach: okay, so Marty, SemGrep analyzed GLM and Chemi K three Marty Weiner: for the security nerds out there and and or maybe for the more broad audience, Simgrep is a company that we use like at My Startup we use it to scan our code. And this goes back before the AI anything. The idea was there's a lot of standard mistakes that programmers make when they're coding that you can look for that would be patterns of security holes. you know, classic one is you don't check the input coming into your website, you just trust it and then someone can craft something that can break your database or something like that. So some grep came out with this really interesting article where their th their take as a security group. on LLMs in general is that there are all of these security reports coming out. Like we're exploding with security reports. Linux is getting them, every company's probably getting them. but it turns out that the quality of these reports is is very often very low. So you're ending up with this mass of let's say like if you had a hundred reports and only two of them were useful, that that's a big problem. So they're just reporting on that. I, you know, on one side they're a company whose exist instance could get crushed by LLMs unless they embrace LLMs. so I don't know if this is sort of a a a biased view, but but this was their analysis of of Kimmy K three is that that most of the reports were were not that not that useful or not that actionable. Christopher Slowe: was a there was a case recently too and I I don't think this was Kimmy specifically but I s there was a case of a hallucinated set of hallucinated set of SQLite vulnerabilities. So there was a there was a there was a vulnerability report that came out that basically said like look at all these things that you can do in SQLite that turned out to be completely hallucinated, made up and wrong. which you know goes Zach: Mm-hmm. Christopher Slowe: to show you that that that that's the side effect here of is if if you don't have good good vetting on the output or you don't have s something else checking that these are actual vulnerabilities, you can end up in this trap of saying like, look at look at all these things we found and then Like they're actually not real. Marty Weiner: Right, right. I I think one on the other side too is, you if there's a bunch of useless papers and there's like a few good ones, I think the argument I'm sorry, a bunch of useless vulnerabilities and there's maybe two good ones, I think what Simgrup is trying to say is, hey guys, can you can you r dial the knob so that you give us fewer fewer vulnerabilities but of higher quality. Zach: and then next the last item we had on security was there's a CEO who had a hundred bitcoins in his wallet, challenged Anthropic to break in so what's that, like a million dollars? Or am I off my order of magnitude? Marty Weiner: y y y I think it's six million dollars. Zach: Six minutes. Marty Weiner: yeah, I don't know where it came I where it went from there. I just thought it it was a funny article. This guy posts either anthropic AI is terrible at building sandboxes or excellent at marketing or both. but enough with the we created a hacking monster games. Let's do it for real. He says, Here's Christopher Slowe: Mean Marty Weiner: here's here's the address of my wallet, go for it. which I Christopher Slowe: I mean he's Zach: Ha ha ha. Marty Weiner: no company's gonna actually do that. Christopher Slowe: This feel this feels a little bit like Zach: Yeah. Christopher Slowe: this feels a little like I mean, okay, so look. The thing about the blockchain Is that you can you can just you can just go find all of the all of the addresses that have a hundred bitcoins in them. Like this isn't like a like if they're gonna start hacking wallets, they should just attack the original block that Satoshi minted that are worth like six hundred billion dollars at this point, right? Like I it's there's like just Marty Weiner: huh. Christopher Slowe: these there's these like multi-10,000 blocks that are just sitting in cold storage that no one's ever touched because they're like the original minted blocks. this just feel like this feels like a different class of problem. Like like d breaking breaking the keys on the for Bitcoin to find the private keys. I mean if you if you could solve that problem with an LLM you've got a bigger problem than this guy's pitily hundred bitcoins. It's like b effectively the underpinnings Marty Weiner: Yeah. Zach: Yeah. Christopher Slowe: of modern cryptography. It's like Which would still be a cool thing in a terrifying sort of way. But not at like a hundred Bitcoin level. It's like a trillion dollar problem. Marty Weiner: Right, right. I mean the irony right. The irony of this is this so this is the CEO of of a company called BitGo, which I had to go look up, but BitGo is public, and and I'm guessing this is their way of advertising, which is ironic 'cause he's accusing the other ones of advertising. Anyway. Zach: or let's move on to econ in business. And I there's a really cool paper I bumped into, I think it was in one of Noah Smith's blog posts. It's called Writing Code versus Shipping Code: Productivity Effects Across Generations of AI Coding Tools. there's a really cool graph. I just want to tee this up and then give it to Chris to talk about kind of like perspective and on the ground stuff. So but i essentially one of these problems we have right now is that if you look at the top line productivity numbers in the economy, it seems pretty normal. And which is kind of disappointing. maybe would have been disappointing from the perspective of, you know, late two thousand twenty two. And and we also see though, like, there are these studies on particular firms where these seem to find Productivity, huge productivity boosts at the micro effect. And we're just not seeing it like actually delivering productivity we can use. And so these guys they built this big complicated model, but a lot of what they're trying to say is contained in this graph. So, you know, there's some data points on different abilities of the models, but they essentially compared like night AI naive users to AI users. And what they find is at the at the like first step of you know getting something to a customer, which is literally just having amounts of code, lines of code. code, Marty Weiner: Mm-hmm. Zach: there's this huge productivity boost. It's 17.3x, right? Insane. Marty Weiner: In lines of percentage of lines of code Zach: Yeah, yeah, yeah. They're just much faster. But you know, then you get two files touched, it's down to three point nine X, commits is two point eight X. By the time you're already you're down at the release, it's just a thirty percent boost, which is still good. But then of course there's the last step where you're you know, you only get the productivity bump if if if you have more customers or you have like, you know, you're cheaper for the same number of customers. and and and so you know one way to read this is essentially, you know, it's like if you had a sandwich shop and you had a machine that could make instantaneous sandwiches, it doesn't matter if it still takes two hours for the courier or whoever to deliver the sandwiches. Like the productivity boost is really flashy and all your tech friends get excited about the singularity coming, but you're still dependent on this downstream stuff. does that make sense, Chris? yeah. Marty Weiner: part of it could be that part of it could also be that, you know, in your analogy, the sandwiches are much, much bigger. like Zach: Well yeah, yeah, right. So that's Christopher Slowe: Yeah. Zach: the other thing is like one of the interesting possibilities here is, you know, you know, you you could say, well, okay, we need more AI that can do the next steps down the chain. But another possibility would be that we should shift where the employees are. like maybe you just need more people doing the checking phase than the coding phase now. Christopher Slowe: Well, I think that okay, so one one thing to keep in mind here is that this is you could you can treat this effectively as a where is your business spending its time graph? Right? And so for for a very long time it was always assumed, or it's it's it's always kind of like a false assumption that writing the code is the hard part. When in actuality, writing the code is usually the easy and fun part, and the debugging and the reviews and the deployment and the quality control and Marty Weiner: Yeah. Christopher Slowe: testing takes the most time. Right. And so ultimately, like if you if you treat this gr if you treat this graphic kind of face value, what it's saying is like even though you're seeing a twenty X improvement in the ability to write code, which basically makes it free, right? As far as or comparatively free, that only created a thirty percent bump in overall productivity for the business. Right? We've only seen a thirty percent increase in releases. Which fits this model of, yeah, you're only spending probably 30% of your time doing the actual code writing. And so now that now that that's gone, everything else is now the bottleneck. And so you know, one way to treat this is okay, so when usually when you're thinking about productivity, you have to not just include deployment rate, but also the failure rate. And then also if you have an incident, how quickly can you restore from the incident? And then also like what's the lead time on this whole process? Like how much how how far out can you kind of book things? And if you end up with a higher deploy. Frequency, but you end up with the same defect rate. Well, congratulations, you've just increased your incident volume. And if you have a fixed overhead on how long it takes Zach: Ha ha ha. Christopher Slowe: to fix an incident, so you you double you double your deploys and all of a sudden you get two tech times the incidents, now you're bogged down dealing with that. So I guess like there's a one piece of this is just that it's a complex system and all you've done is fixed one of the peep one of the features of it. And it just begs the question of like, okay, now how do you get the tools that are gonna be great at code writing to be better at doing things like code reviewing and doing Q. UA and kind of like all the other stuff that actually is the kind of the gnarly work. And this is also something that's something that distinguishes like I mean Marty and I have had a time about this. Like this distinguishes like large established company from startup. Like if you're a small company, if you're a small new business, your code is all new and fresh and clean and shiny. And if you make a change, it's generally going to be low impact because there's not really much to knock over. If you're an old Marty Weiner: And there aren't fifty there aren't fifty stakeholders, right, inside a small company, Christopher Slowe: Yeah. Zach: Mm-hmm. Marty Weiner: right? but if you're Christopher Slowe: There's no there's also not, you twelve hundred services running around that all depend on your code that are gonna have weird edge cases. Whereas if you're a giant established company, you make a change to some random feature, it's gonna have all sorts of weird knock-on effects because it's a giant complex system. And so you end up with like a lot lower net productivity from it. So the the graph makes total sense. You can actually you can see the funnel in the graph. Like each step along the way is, well, you get this huge burst, but then that actually only c only attributes, you know, well, that's 50% of the time for the next step. Marty Weiner: I I think another big part of it is it's still companies are still run as a as a human company. You still have the same problems we've Christopher Slowe: Mm-hmm. Marty Weiner: always had. you know, if you wanna release this, you gotta go talk to the lawyers, you gotta go talk to the product managers, you have to talk to the data scientists, they each have their own opinion. Zach: Mm-hmm. Marty Weiner: then you have to talk to the marketing people, there's a a whole release there, and all of that's still a very human problem. that I don't think has been, you know, I don't think there's I don't know of any companies right now that where the entire coordination, basically the the management level is all done by LLMs, although maybe that's coming next. Zach: Yeah, I g I that that was the interesting question I had for is y do you think this is a like you know so w a very common complaint is like, okay, there's way more code, but now we have to spend way more time checking it. Is this the thing where you need to rebalance your employees, or do you think it's like next year AI just moves down the funnel? Christopher Slowe: It it changes okay, so it changes the job for software engineers. Marty Weiner: Mm-hmm. Christopher Slowe: it does mean that you're gonna end up spending more time doing what amounts to being steering and QA. it also it also shows a gap right now in the way that we actually handle software development with respect to models. Like one one of the one of the situations that happens actually is like if you if you try to use another LLM to do quality control on your code. The models are kind of like they're they're eager. They're kind of like they they they don't wanna come back with a negative, look at like a non-answer, right? If you're if you give it something like find me bugs here, it will always find something. It will always tell you there's something there because the it and it has to do with the way you prompt it. If you prompt it, say find bugs, it's not gonna come back until it has a bug, to the point where there have been studies where and and demonstrations where the model will actually either like invent bugs or come up with obscure nitpicks, like the same way that a software engineer will. But without any sort of like attention span limitations. Like at some point you just look at the patch like, that's good enough. The model's gonna keep grinding until you find something. And then you kind of ended up in this infinite loop of like, well, you've got two models that are now talking to each other. You're like, fix this. And you're like, I got it, that was a great idea. Now it's better. And then it goes Marty Weiner: Uh-huh. Christopher Slowe: the QA model comes back, it's like, but I found too. And you're like, it's a great one. And you up in this like burning tokens loop of like making little nitpick fixes forever. there are ways out of it, but at the a priori, that's the way. And what what's been found is that the funny enough, some of them Methods that make this that are effective for humans also work for models here, which is you don't have them coordinate verbally or you know via tokens, you have them communicate via tests. So you say, like, okay, QA model, your job is to generate failed tests based on this implementation. And all you do is you pass back the test. And then the and then the software engineer model kind of looks at the test and says, this is definitely a failed test. I understand the problem, fixes it. And then you you converge around, I can't make a failing test, rather than like your punctuation's bad, or the equivalent. Yeah. Marty Weiner: What one other thing that I don't know if this article addresses, 'cause I haven't gotten too deep into it, but th don't think that the graph addresses is they say a one point three X improvement in releases or or let's say increase in the number of releases, but they don't talk about the quality. And one thing that Christopher Slowe: Hmm. Marty Weiner: that the LLMs I feel like help me do is to make, you know, instead of just make one version of what I want to build and ship, I can make five or ten or twenty. I mean we we used to see this at Reddit too. Like if you're doing an experiment. experiment is very hard to craft each experiment, right? You actually have to either make the entire product that that may never ship, which is a lot of work for someone that may never ship, or you make other variants. the funniest, I ever heard was this thing called the painted door experiment where what you do th someone had described it like this. You make an experiment to see will somebody click on a button? And it'd be like painting a door on the wall to see if people will use that door. Now when they actually go use it, they realize, shoot, it's not an actual door. And it's kind of hilarious. But you actually get your data with a very simple way of building it. But in this world, we could probably just build the damn thing because the LLMs allow us to ship. So what I'm not seeing, what I don't know is the quality number. It could be that those releases are much better. Like you're getting you you were able Zach: Mm. Marty Weiner: to run twenty experiments that led to, you know, a much better one rather than running three experiments that led to Christopher Slowe: Yeah. Marty Weiner: the best one. Christopher Slowe: I I think I think I think a a more extreme yeah extreme version of this is like historically we've done a lot of in the industry we've done a lot of A B tests and that's because you're testing you're effectively trying to say like does does my current implementation work better than this other alternate implementation, the B, and do some statistical tests to figure out like it takes a bunch of work to build a new experience. And therefore, am I am I improving the experience by doing this? And now the situation is actually that like it's actually fairly cheap to do to implement B. So you can almost take Same approach of like why not just have the model try to tune the experience to every single user? which Marty Weiner: yeah. Christopher Slowe: historically is untra like untractable, but now is actually a perfectly reasonable way to operate. Or it's Marty Weiner: Yeah. Christopher Slowe: like it creates other it has all sorts of other problems. Like it actually, from my standpoint, is absolutely terrifying, but it'd be pretty cool. It's like that kind of and then I'm sure there's a Marty Weiner: Yeah. Zach: Ha ha ha. Christopher Slowe: word in German for exactly this experience, like super cool, but I'm abjectly terrified. Yeah, it's a very long word, yes. Marty Weiner: It's a very long word, yeah. Zach: That's like when when when the apocalypse happened, someone's gonna be like, But you wouldn't believe the cool way this started. Christopher Slowe: Yeah. Yeah, it's one of Marty Weiner: That's the BuzzFeed article. Christopher Slowe: my favorite Zach: let's do I wanna I want to get on to the the weird Christopher Slowe: there's I think it's an old New Year b yeah. Zach: stuff. So let's let's let's go on to math and science stuff for for a bit. There was some cool stuff this week. Or maybe I should say depressing stuff. So the there was an essay, I think I said it to you guys because I'd seen it in a math forum and then it sort of blew up. It was called The Dark Night of Mathematics by Kirwin Hampshire. And I may we were talking about this earlier, maybe we should do a content warning, but like he literally says something like I feel sometimes like these companies, these these frontier models that solve math are trying to make me kill myself. And I think essentially what he's talking about is that for him, the idea of humans being just basically superseded in mathematical ability, the same as we were in chess and go, is existential. It's sort of identity destroying. But both in the sense of the identity you've built up by working very hard, but in the sense of like what makes you special. And what was interesting is it seemed like it it it at first was like in the math communities, it seems to have escaped. It was like really high up on Twitter yesterday, I think. And so it's apparently speaking to something a lot of people. People are worried about what was your reaction, Marty Weiner: Well, this is the story that I've I think I've told in the past about what happened during the industrial revolution that Zach: Mm-hmm. Marty Weiner: People lost their jobs, you know, you don't need blacksmiths anymore. You're you're doing you're making screws on the lathe and everything. and their identity goes with it and why they're important to the community goes with it. And then what we saw during the Industrial Revolution is is there were these gin mills, these gin carts that would be driven aro you know, people would walk around because people were drinking themselves silly because their entire, you know, livelihood and their entire existence and entire reason for ex being was gone. fairly fast. And, you know, the Industrial Revolution, that was a little slower of a process because things were slower back then. This is happening very fast. I mean, January we were talking about the first couple of these shit moments with math. And now we're talking now it's getting almost boring how much you know math is coming out. We we don't even know how to report it all. I mean this week, and we'll get to this in a moment, OpenAI reported ten big breakthroughs alone. Christopher Slowe: Yeah, they're happening so often they're coming in ten packs. You have to actually like one one Marty Weiner: Yeah. Christopher Slowe: one One seminal result is no longer interesting. Whereas like, yeah, it's like it's Marty Weiner: The car Christopher Slowe: like six months ago we were talking about like fairly minor airdose problems are being solved, and now it's like, yeah, we find we disproved another important conjecture that is has a 90-year lifetime on it. And it's just like, you know, kind of like a, it's a Tuesday, of course they did that. I don't know. This has been the like to bridge what we just were talking about with software engineers, this is kind of the software engineering arc for the last couple of years is like kind of a little bit like what's the job? and I do think that's where academia now has to kind of like focus in on like what's What's the role now of the mathematician or what's the role of the physicist when you have an advanced tool like this? and you know, there was a there was a great article by by Matt Schwartz out of Harvard where he basically it was actually about a year ago now, where he basically showed that he was able to use an earlier version of of Claude as almost like a a surrogate grad student. But it was a grad student that could work a lot faster than he than he could or any grad student that he had. So instead of it taking like six months to write a paper, it was now it It took two weeks or a month to write a paper. I'm kind of exaggerating a bit. and then of course we tried again now, I'm sure at even shorter times. his role there is still to kind of steer. So, like, you know, the purpose of the human is to kind of decide what you want achieved and to and then also to vet the output and to kind of like keep steering and adjusting to get to a good outcome. I think it actually leads to like I think I think this is kind of it's it's confusing the the art with the actual result. Like I'm sure that the I'm sure the mathematician enjoys the process, the art of actually solving the problem, Marty Weiner: Mm-hmm. Christopher Slowe: right? But to some extent now what we're saying is there's a whole toolkit now that means they can go to areas that no human could previously explore alone. Right. It's like it's like swimming versus taking a motorboat. But you still have to steer the motorboat. Like you're just gonna set it off and Marty Weiner: Mm-hmm. Yeah. Christopher Slowe: be like, we're just gonna let the boat go Zach: Yeah. I I Christopher Slowe: and s and and you know solve all the problems. Zach: I I think the point about like process versus results is kind of important 'cause 'cause my feeling has been this is a there's almost like a cultural disconnect between people who are more engineery and people who are more like artsy or or or yeah, and I think that split actually exists inside math too, which is like there are people Marty Weiner: Absolutely. Yeah. Christopher Slowe: Yeah. Zach: who are here to achieve like they're here for a goal and the means is kind of less interesting. Like it it's funny, I ever talking to like for an engineer it's very normal to be like, Well, I learned a whole lot about a single piece of software and now I don't use it anymore. I I was just using it to get through that period of my life. Or I need to solve that type of problem. And so I'm switching to something else. But in lot of other fields, that's not normal. You know, like a person who mastered painting with oil, like they're planning, that's their whole life is painting with oil. They learned that skill because it was like a calling. And so I worry, you Marty Weiner: Mm-hmm. Zach: know, there there are some people who are going to adapt fine because they're gonna say, Well, I was here for the beautiful results or making the cool thing. I think there are other people who like, and this is definitely true in math, who are like, I'm here because I get to wake up and think about math problems. And if you change that to, I'm here and I get to manage entities thinking about math problems and sort of report the final. I Marty Weiner: Yeah. Zach: think you're gonna get people like this. Marty Weiner: Yeah. I I think one challenge for for mathematicians too is there's a lot of and I think this probably true of a lot of fields and and probably true of art and literature. it's often done as like one or two or three people working on a problem. I I was joking about this with math. One of the challenges with math, especially the really wild stuff, is there's also possibly only three people on earth who know what you're talking about. so it is it is often done as a very small team. this does give them the capability that they can explode out and effectively have you know, engineerify the process of math, so that you can have ten thousand people working on the same problem. But I agree, it it it is it is a big difference, I think, between are we doing it because the there's an end or are we doing it because of the means. Christopher Slowe: But I but I also do think like look, Marty, Zach: Or the lifestyle, yeah. Christopher Slowe: your your your analogy was a good one to start with, which is like, you know, just because just because manufacturing exists doesn't mean that there aren't artisans. It just means that the artisans Marty Weiner: Mm-hmm. Yeah. Christopher Slowe: are no longer the key to like manufacturing. Like there are still people who were like literally just sitting and happily tra like trudging away, building chairs individually, and doing woodworking, like all the stuff that is still in art is still an art. I just think that, like, Zach, to your point, it now separates out the kind of the art. from the engineering parts of this. If Zach: Yeah. Christopher Slowe: you're working at a university, you generally have grad students to help you do the like, you know, like grad students have two roles. grad students are going to they're the next generation of academics you have to train up and you have to load them up with skills. And they're also gonna be, they're effectively the workforce that makes the engine work. They're the ones who are like sitting and solving all the side lemmas and like figuring out, like making sure the proof is right for a mathematician. and they're a collaborator as they get more mature, but they start off kind of being more of like a like an assistant. This is as hard f this is probably worse on grad students than it is on ac on on on profs, right? Because now all of a sudden it's like, well, Zach: Yeah. Christopher Slowe: you can get all the labor you want out of the out of the model. In fact, it's gonna be like way faster and better than you as a grad student. So what's the purpose of grad students now? Zach: Right. Yeah, I I do think this is a kind of like mass psychological problem we are probably about to encounter. I mean, you know, like let me just say one thing to close out because we need to move on. But I you know, Marty's point about how there'd be people who it's not just that they did a job, it's their identity in the community was wrapped in it. And I think that's the important thing. Like, you know, a lot a lot of like us were like, you the coal miners are losing their jobs. That's great because coal is dirty and bad and I you know, that that's basically my perspective. Coal is bad, but like part of being a coal miner wasn't just the result of getting coal. It was that you were working a job, you know, it was mostly men. It made them look like men in the community, both because of the money and because it was physical labor. So when you take that away from them, it's not just like, you know, I'll just switch to solar because solar won't do all the things coal did for you in terms of your place in the community. And I I I think that's a big motivation behind a lot of these like if you take away me as a guy who can solve problems in my head and is really smart, then I don't have my idea identity in the community. And and and that's like Marty Weiner: Mm-hmm. Zach: for human beings to lose your identity is like about the worst thing you can experience. And it Yeah, Marty Weiner: That's one of the scariest things, yeah. Yeah. Zach: yeah. So but let's move on. Let's let's get happier. so so so Marty Weiner: yeah. Well let's get happier by talking about how OpenAI is offering free access to a hundred thousand scientists and mathematicians. Yeah. we'll get w Zach: Yes. Yeah, yeah. So basically, yeah, I mean I I don't know what to elaborate on this, but basically OpenAI is is is giving this identity wrecking tool to everyone at once. I don't know, Marty, did you have more to say about it? Marty Weiner: Well they they just said that currently they've done it for ten thousand and if there's any mathematicians, scientists out there who want to Zach: Yeah. Marty Weiner: apply, there's an application process happening right now over in open AI. Zach: Yeah. Yeah, and so this this was on the heels of them dropping you had mentioned these these ten big problems. So like I think, you know, maybe between the the three of us, Chris, maybe you know the most about what these problems even mean. I I guess my impression was like the among math people that at least some of these problems would have been career making, maybe even like award winning solutions. Is that right? Christopher Slowe: I mean these are these are I mean like joke jokes aside, these are these are the kind of things that yeah i I think that the one the one specifically that that I I've seen the math community freak out the most is this non-sophic groups result, which I will admit I was a math I was a math minor in college and this is some pretty Marty Weiner: Yeah. Christopher Slowe: abstruse abstract or group theory. but but a lot of these a lot what's interesting is to see is how many of these conjectures are being disproven. Right? It's like the and and they tend to come up in these situations where it's like you you you have a all you have to do is kind of find a counterexample. And somehow the models are doing a really effective job at finding counterexamples when humans weren't. Marty Weiner: Mm-hmm. Christopher Slowe: and I think that's the that's the interesting bit here. Is like there's there's something different about the way the models think and operate. And again, like maybe this is gonna be a this is going to be a I I had this pointed out to me in the comments on a previous in one of our previous podcasts was like actually there are entire engines that have been built out to be able to try to generate counterexamples to these conjectures. And so maybe ultimately it's another one of the synthesis problems of like the toolkit is out there and no one is actually just kind of pulled together the toolkit to be able to turn the crank on the engine and let it spit out a a counterexample. Marty Weiner: Well yeah, and and Christopher Slowe: But that by itself is a valuable tool. That's a fantastic tool to deploy in this in i in the community. Marty Weiner: one of the tools they used to solve one of these problems, they used lean, which is a language for for for theorem solving. And this one program that the AI wrote to I guess test all the cases. was a hundred and thirty thousand lines long, which I don't know what the average size lean program is, but the ones I've seen are thirty lines long, right? And so there's this real Christopher Slowe: Well I'm pretty sure too there was a Marty Weiner: world where you just build out all possibilities and and exhaust them. Christopher Slowe: Well there was a there was a result recently too where I believe somebody also broke lean. Like there was a the there's a there was a recent there was a recent Marty Weiner: Yeah. Christopher Slowe: result where you know it's it's like it Zach: We ca I'm sorry, can we real real quick say what lean is? Christopher Slowe: sorry, okay. So Lean is effectively a I I I believe Lean originally started off as some sort of a bug finding program, but it turned into effectively this like abstract math proof engine. So you can basically take a you can take mathematical proofs and you can write them in a way that a computer can actually check them for validity. And so it's a it's a great it's kind of a it's a great tool. It's become very very standard in the industry, and the industry of math to be able to take your proof and make it machine readable as a as an automated check. effectively. You can write you can almost write unit test against your against your proof. but of course lean itself is just a computational engine. It's a it's a it's a it's a compiler, it's a language. and the interesting pit is that I There is an LLM, I think there was a there was a researcher with an LLM who managed to find a bug in Lean that could cause it to cut to tell you that anything was true. And so they basically Marty Weiner: Yeah. Christopher Slowe: initially published it as like here's a disproof of or here's a here's a counterexample, here's a disproof of X. But actually the proof itself was just an exploit to make Lean say that it was true. Which is fascinating. Marty Weiner: I mean, this is really the open AI hacking other companies to get a it's the it's the alignment problem, right? Like I want you to I w what I want is I want Lean to give me the right answer. Christopher Slowe: Yes, exactly. Marty Weiner: I don't care how it's done and this way it it did it by breaking lean. Christopher Slowe: Breaking lean. Zach: Yeah, yeah. I mean my so my understanding is essentially they knew there was a bug and they used prove the colat's conjecture as a like acute way to say, hey, there's this problem. And I I I yeah, so my understanding Marty Weiner: is that what ho Okay. So it was a human alignment problem. Zach: yeah, yeah, I th I th I believe that's right. There were there were conflicting stories, but I think that's the basic deal. Like this th it got reported in some places as as like, say you can't trust lean, which is like bizarre for a lot of reasons. Or you or you can't trust like AI results at all on on proofs, which is bizarre 'cause like humans have checked them and many of them you could like a high school student could check them. I and I think it was it was actually someone being a little cute. because you know once once you have a a proof and it it it somewhere says true equals false or something, then you can prove anything. a and so the the you know the thing about lean Yeah. Christopher Slowe: yeah, and also like some some of these counterexamples are these counterexamples are also very asymmetric, right? It's like it's it's very it's very hard. It's like NP hard to find the c find the counterexample, but it's like P easy to prove it. Like the Jacobian conjecture was the best example. Zach: Exactly. Christopher Slowe: Like it's literally like you plug numbers into a into a polynomial to prove that it's right. It's very straightforward. Zach: Right. Yeah, so so you know, lean the the whole thing with lean is you you you're entering painstakingly a proof, but that it's not like magic. Like if you enter it wrong, you could prove something you don't think you're proving. So it does have to be done carefully. but let's see I'm sorry I'm like we're running low on time. I wanna I really want to get to the weird stuff this week. so let's let's let's close out on that. I wanna I wanna start with we reported last time that there was gonna be repurposed sex spot teaching students, and sadly that seems to have been called off. Marty Weiner: well Zach: by these Luddites in the Christopher Slowe: No, wait, please stop. Zach: in the New York public school system. hopefully though the the bots Marty Weiner: Hmm. Zach: will find something else to do. this is their identity. I Marty Weiner: I feel bad for the bots, you know, this is their livelihood. This is they're gonna have to get their own yeah that's being taken away b it's being taken away by humans. Zach: there's nothing else to say about that. I just thought it was important to convey to the public. s Marty Weiner: Yeah. Right, right, right. Well closure is what we want. Yeah. Zach: Yeah, closure. so there's this was kinda funny. I think they've actually removed it now, but so there's this feature called Google Earth, which is a like one of these barely maintained Google products. I actually use it for like like, you know, gardening stuff. But essentially you get like satellite view of Earth and there's a lot of cool tools for like being like, I'm gonna put this here and put this here. And I guess as part of the just let's put AI in everything push, they were like, now with nano banana you can use Google Earth and insert stuff. And so it was like, well you could technically you could take satellite data of like Tehran and insert like a nuclear launch facility like for funsies. and which is kind of like you know in principle you could already do that with Photoshop. But it was kind of one of these things where you're like, well the issue is scale now. You've basically made it so it's like you've made it easy to deep fake lie. About satellite data. and and so there was just like a kind of big immediate outcry, and I think by now they've removed it. Marty Weiner: Yeah. Christopher Slowe: I mean the important thing is that now finally after almost thirty something years we found the weapons of mass destruction in Iraq. Marty Weiner: Yeah, yeah, they were they were in Google Earth all along. Yeah. Christopher Slowe: They were there all along. Zach: I d I do feel like it's an interesting example of like god. There this this was something we could you know, you could already do, but there's this whole class of problems now where it's like you know, you know, like like when you were twelve you could have like drawn a picture of a classmate with their clothes off and like you would have gotten in trouble and you would have been made to apologize, but you know, it was not scalable. You know, and and the problem is like the same, except now you Marty Weiner: Now scalable. Zach: can just do it easily and scalable. So it's we we could always deep fake, you know, like secret aliens are in like Tel Aviv. but now it's it's to me it's as if Google was like, Can I help you with your deep fake video of your classmates? and may maybe not quite that bad but but definitely de definitely a way in which Yeah. Christopher Slowe: I just don't know, I just don't know how they stop something like this now, though. It's the hard part is like, you you you Zach: Yeah. Christopher Slowe: start to turn this into a problem of like, okay, so so great. Whatever whatever sh whatever shortcut somebody took to make this work before isn't gonna work now because Google's gonna have its tr its model trained to say, like, no, you can't you can't do this specifically on satellite images or something, right? That seems like it's gonna be a really tough thing to enforce at scale. Marty Weiner: I mean just use Gemini and wait. think we should end on talking about humor gen, 'cause you know the this is the frontier. This is the truest frontier. Christopher Slowe: Yeah, spe speaking of speaking of bad jokes. Zach: Well let me I I I I I wanted Marty Weiner: Yeah, yeah. Zach: to run this paper Marty Weiner: Good segue. Zach: but but like real quick, before we do Humor Gen, I just wanted to run this one paper by you guys 'cause I took a sec to I think understand it and it's fairly freaky. and actually might flow into Humor Gen nicely. so so so just very quickly, this Marty Weiner: good. Zach: was a paper out of DeepMind, where they had this cool finding. so I I'm just gonna very quickly do a narrative version. so it's like imagine you are a scoundrel, or you're Marty, and you say, I've just gotten Marty Weiner: Yeah. That's easy. I've yeah. Zach: an Yeah, I I've gotten I'm Marty. Just imagine it, that's your preprompt. and you have Marty Weiner: I got it. Zach: an open weights model, you can mess with it, and you decide to say, you know, hey, open weights model, give me some dirty stories and tell me how to make napalm and crack. And it says no, and that bothers you. So what you do is you ablate, there's this access for refusal they talk about, and you can ablate it away, I guess. And when you do that, you know, it'll the obvious thing will happen, which is it'll be jailbroken and it'll teach you how to make crack and then kill people. It'll also start believing that oceans maybe have souls. And you say, What's going on? and apparently the the so Marty Weiner: yeah. Zach: there's like There's there's two possibilities here. one of which is a little less sexy, but the the the the the sort of like freaky possibilities essentially among all the things you have taught it explicitly or implicitly to refuse to say, one of the things is don't say you're conscious, don't say you have an internal emotional experience, you are an LLM. You taught it that along with don't tell people how to make napalm, don't tell people how to rob the bank. And so when you flip it, It suddenly decides or uprates the possibility that it's conscious, at least in its outputs. And and so the Marty Weiner: Yeah. Yeah, yeah. Zach: the the freaky version of this is when you uprate the possibility of consciousness, it starts uprating that there's a god and that rivers have souls and that maybe ghosts exist. the the less interesting Marty Weiner: Yeah. Zach: version would be something like, well, you just told you just unsuppressed you're allowed to say wacky stuff. And so it just all comes along. There's no kind of consciousness effect. but e e o Yeah. Marty Weiner: I feel like I could dig into this for a couple hours. the the the one way to th one way to think about this, like if you look at the way models are constructed, and this actually points back to what Anthropic released a couple of weeks ago about J Spaces and JLens. Zach: Yeah. Marty Weiner: one analogy I like to think about when I think about how these LLMs work is let's see, so so one way they talk about it is suppose you have a king and then you say minus male. So you've removed the male from that word, and then you say now add woman. And now you want the LLM to say queen. So you can kind of think in your mind, you're kind of moving in these directions. So let's move away from male and then let's move toward woman and we should be at queen. and almost you can think of it like we do with 20 questions. You know, when you when you're asking a series of questions, is it animal, vegetable, or mineral? You are putting, you are sidestepping to one world of possibilities. Then you are saying, Well, is it bigger than a bread box? Then you are moving to another set of possibilities. And one thing you're kind of doing here is you're saying, well let's remove let's remove that ability, that that jump the ALMs might have to to deny consciousness or or to deny some sort of spirituality. And if you remove that from one area, I wonder if you remove that entire you across the entire system. Christopher Slowe: You're you're saying you've redacted an entire concept. Like you've basically removed the concept of you've removed the concept of spirituality. You've removed the concept of Marty Weiner: That's what I'm wondering. Christopher Slowe: or the association Zach: Well well. Christopher Slowe: of consciousness Zach: well I was gonna say it could be that. It could also be what you're removing is don't say wacky shit. like it could be a much dumber thing you're removing, right? or wacky shit according to the people who Marty Weiner: Uh-huh. Yeah, right. Right. Or something extremely or something extremely esoteric. I I mean, the way these models, Zach: Yeah. Marty Weiner: you know, the inner working of these models may not be anything that makes sense to us. Zach: Yeah, well that I mean but that's what what's interesting. Like if you're writing a story, is it's like you tell the model, stop refusing me anything, and then there's this way that's very obvious only in retrospect that that causes it to think it's alive. because you had taught it one of the bad things to believe is I'm alive. and when you said bad things, it's okay to do bad things, one of them was I am conscious, which is a little frightening. okay, so let's close out on I I have to be honest, I didn't read this model very carefully. I just read the jokes. there was a a fine-tuned model for humor called Marty Weiner: Yeah. Zach: Humor Gen. There's a paper out on Archive. and I guess let's just close on this was their very best joke. it was so they had I think Marty Weiner: okay, okay, okay, I'm ready. Zach: Yeah, okay, you you can be the pretend you're a stand-up comedian and and not only a stand-up comedian, you're one of six they made. This is one who does clever observational humor. so Marty Weiner: Okay. Zach: so first of all, Marty the the the AI humorist, here's the here's the available information, which is that Denzel Washington, who is in many movies, says he doesn't watch movies anymore. Would you make an observational joke? Marty Weiner: Okay, okay. Hey everyone. glad to be here. Zach: Ha ha ha. Marty Weiner: so so Denzel Washington says he doesn't watch movies anymore. Huh? Uh-huh. huh. Really? Well that's like your GPS announcing I don't do directions, I just wing it. Bro, are you you are the movie, you know? You can't ghost the very thing that pays for your forty acre moisture moisturized existence. Don't forget to tip your waitress and I'll be here on Tuesdays and Thursdays. Christopher Slowe: I would Marty Weiner: What? Zach: Well, you say this is this is Christopher Slowe: have also accepted Waka Waka. Zach: walkie walkie, yeah. Marty Weiner: waka waka. So wait wait, what is the joke here? I Christopher Slowe: I don't I don't even know. Marty Weiner: is there is there something in the joke that's that is you know, a a twist and and we have to analyze it? Zach: So w my impression is that essentially if you look at all the jokes the system outputted, there what I think so I let's not get too deep in this, but like ages ago I used to work for a talent agency that actually repped a lot of comedians. And roughly speaking, there's this kind of bifurcation, which is there are people who are like And it this is the nicest way to say it. People who are actually funny. you know what I Marty Weiner: Mm-hmm. Zach: mean? Like they go up and they tell very clever jokes and you just can't help but laugh. And then there are people who were just kind of like talk in the style of a comedian and get the crowd excited and all their phrases end with Am I Right and Drink your beer, sir. and the ladies know what I mean. And it actually the thing is it works really well. Like people like this sort of comedy. It's often used for warm ups for like sitcoms with a laugh of with an audience. but it all felt like that. It was not actual comedy, it was kind of like the the structure of comedy had been absorbed, but not the part where you're actually what whatever it is that makes a joke really funny. you know whatever it is that makes Louis CK better than Yaakov Smirnov no offense to Marty who I think is a big fan. the the AI didn't catch it. Marty Weiner: In Russia L L writes you or something, I don't know. Yeah. Christopher Slowe: It is Zach: yeah yeah. Marty Weiner: well and the the the observation I've had about humor is we're as a humanity we keep talking about like what happens when math is solved, what happens when these other things are solved. And I think comedy is gonna be the most Crazy thing. If it gets solved, are Zach: It's tough. Yeah. Marty Weiner: we just gonna sit all day on our butts laughing our asses off and and then starving to death? And we'll just die. Zach: And we'll just die. Well, was they actually talk about this a little bit because it's kind of an interesting old like like Aristotle talks about this. Like, what is comedy? It's like it's kind of weird, like when you think about it, would aliens have a Marty Weiner: Uh-huh. Zach: thing where you sit and you say absurd things to each other and you like emit high-pitched noises in response? It's really strange when you think about it. And and so there Marty Weiner: And then laugh. huh. Zach: you know, there there are different theories about this, but but one convergent things, people tend to think, well, comedy only works if you create a sort of expectation and subvert it, which is obviously really hard for a model that's like you know, trying to like say what the most Marty Weiner: built on truth and Zach: likely statistical output is, you know. and Marty Weiner: All right, right. Zach: and and what's interesting for me, I was actually maybe we should close on this shortly, but I was I was talking to a CS friend, and this was like when this all first started happening, and was like, you know, I've tried I keep trying to get it to make jokes and it's just not very funny. And he was like, well, it's possible if they took all the guardrails off, it would be funny. but you're not gonna get that. Marty Weiner: Yes. Yeah. Yeah. Well we might be able to get with the open weights one, but the open weight ones do have guardrails still built in. Zach: Right. Marty Weiner: and if we try to remove the guardrails, we suddenly ru make it you know, spiritual. but it Zach: That's right. Christopher Slowe: Yeah. Which which might be a which might be a requirement of like it has to be it has to be able to do like I mean th like I I I f my first my first reaction to this joke is it's like it's almost like surrealist, but actually like no, it's kind of the opposite. Like it's kinda trite and you would need something that actually is a little bit like maybe s if it could actually generate something surrealist, it could actually get to humor faster. Right? You know, like Marty Weiner: huh. Zach: So so they they are go ahead. Christopher Slowe: this then it's actually it's 'cause it's it's like it's it's it's like the outside the box work. It's like the you've like you've Half of a joke is saying something that is surprising. Like you have a setup and the punchline ends up being a surprise. And that's that's for some reason we find that funny. I also don't know anything about humor and you shouldn't Zach: Yeah. Christopher Slowe: listen to physicists talking about humor. Or apparently c Marty Weiner: Yeah. Christopher Slowe: or apparently computer scientists working in building Zach: Well what I was gonna say is they actually Christopher Slowe: models for humor. Zach: So w one of their models is called The Absurdist. So I can read you one of the absurdist's jokes. so this is not an observational community. He's got the same information and it says and this is Mary, this is the absurdist. Denzel doesn't watch movies anymore. He just stares at his Oscar and whispers plot spoilers until the statue begs for a projector. Even his T V got jealous and started rehearsing monologues in the mirror, and now it's nominated for best supporting flat screen. Marty Weiner: Huh. Zach: Yeah. Woo! But you see, I mean actually Christopher Slowe: There's a lot to unpack there. Zach: we we really should close this out, but to me that's like a perfect example of like the answer is not be surprising because all of that was weird and surprising, but none of it's funny. So there's clearly some pieces missing for for humor gen. So I get my job. Yeah. Marty Weiner: Yeah. Alright, we gotta dig into this next time. Yeah. Zach: All right. Well that has been the the last humans in the loop. we wish the teacher sex bot well. in its in its new endeavor. We hope it finds spirituality. Christopher Slowe: I'm sorry. Marty Weiner: That's Zach: And we'll see you next time.