Latest / Artificial Developer Intelligence / One Year of Bad Takes, LLMs That Write Neural Network Weights & Inside an AI Benchmark

Transcript

One Year of Bad Takes, LLMs That Write Neural Network Weights & Inside an AI Benchmark · Artificial Developer Intelligence

• Shimin (00:00) Hello, and welcome back to Artificial Developer Intelligence, a weekly conversation show where two software developers track the impact of AI on the profession. My name is Shimin Zhang, and with me today is my co-host, a Dan Did not Access Users Conversation with Claude Lasky. Dan, how are you doing today? Dan (00:20) I'm doing all right. Happy election day. I woke up at it Yeah. Shimin (00:23) We're we're recording this as we're doom scrolling on like a separate monitor right now. Dan (00:28) I woke up at six AM to go vote because I didn't Shimin (00:31) Mm-hmm. Dan (00:31) wanna stand in lines and unfortunately the line was too long when I tried to do early voting. So if you haven't voted yet, well I guess it's probably too late by the time you listen to this, Shimin (00:39) probably Dan (00:40) but Shimin (00:40) yeah. we should everyone everyone should always per participate in the in in the democratic process, if you live Dan (00:41) And you may not even live in this country, but Yeah. That's true. Shimin (00:47) in a democratic country, wherever you are. Dan (00:49) Very true. Shimin (00:49) All right. this week's show, we're gonna do something a little different this week. it is our one year anniversary of the podcast. Woohoo! Dan and I started this the first week of November of of last year, even though our first official episode was not released until the end of November. So we're gonna have some anniversary segment editions, but um Dan (01:10) So if you hate clip shows, this is not gonna be a clip show. Shimin (01:14) just not at me. Yeah. Just us in different outfits recording throughout the year. ah Dan (01:18) Whoa. Shimin (01:19) As always though, we're gonna first start with the news threadmail, where we're gonna be talking about some anthropic hack news as well as the latest artificial analysis ai benchmark. Dan (01:30) Then as Shimin and kinda spoilered already, we're gonna be doing our one year retrospective. So gonna be looking back at the last twelve months of AI, ourselves, the universe, you know, whatever. Shimin (01:42) Yeah, time to cash in those bad take checks we've been writing all year and see how we did. um Dan (01:48) Yeah. Shimin (01:50) and then if we have time, we're gonna do a deep dive on whether large language models can natively read and write neural network weights. Dan (01:58) And then finally we're gonna as always close at two minutes midnight where we take a look at the financial side of AI and as we creep what feels like ever closer to the the two minute mark the past few weeks anyway. So Shimin (02:10) Yeah, good thing we moved it forward to three minutes and thirty seconds last Dan (02:13) Yeah. Shimin (02:14) week. It would look real bad if we hadn't done that. Okay, Dan (02:16) Yep. Shimin (02:17) let's go to our first item of the week. this one is brought to us by Dan. Dan (02:22) Yeah, so where do I start with this? it's a little bit of a sensationalist title, and we'll break that down in a second. But a new startup who was started, I believe, by ex Doge staff, which, you know, says something right there. apparently hacked Anthropic in less than twelve hours, according to them. Shimin (02:44) Mm-hmm. Dan (02:44) Um and they claimed that they got access to their internal network, right? Which is kind of a loaded terminology. Um Shimin (02:55) Right. Dan (02:55) according to the Washington Post article that this is like largely based on, they did actually get into Slack and a few other places that did have some like data of revalue, real value. But according to Anthropic, it was basically like a couple. like HR and IT, third party SaaS apps got owned and Shimin (03:12) Mm-hmm. Dan (03:14) not and not actually like Anthropic's network, right? So kind of a big difference. And then the other part that was I thought was kind of a funny thing to push back on was Anthropic was said that it took them much longer than twelve hours to breach the systems because they tracked like several the attempts over several days. but yeah so this of course kicked off a huge storm of worrying, like you know, a senior Trump official who's anonymously said, like, if a bunch of script kiddies can do this, like what about the Chinese? which is like I d I don't know. I mean exactly. Shimin (03:48) And they didn't hack hack anthropic, right? They just like got some compensation data and offering letters. It's not like they got model weights. That would be a completely different story. Dan (03:57) Right. Yeah. Or like you mentioned in the in my middle name, access to transcripts or any kind of corporate data or something like that. So Shimin (04:06) Did you see the follow-up of Dan (04:08) I did not. Shimin (04:09) today? so this is a tag along piece, but OpenAI hires the Doge alumni who we just spoke about to strengthen its own security measures. Sounds about right. This is from Gizmo. Dan (04:18) Ah, well, they got what they wanted, right? Yeah. Shimin (04:26) they did something right. if they're Dan (04:27) That's a that's fantastic. Thanks for finding that. Yeah, usually when I Shimin (04:30) Yeah, and of course. Dan (04:31) do a news one after I submit the initial I try to do like a quick check just in case there's been any more, but I like kind of didn't think there would be with this one. So that's really funny that you found that. Shimin (04:42) yeah. And what would open AI want with anthropics internal IT data or HR data? one can wonder. I'm gonna choose to read this the way it is, right? Yeah. Dan (04:50) Well, if it had interview and hiring stuff then that's probably what they want with it. Shimin (04:54) So if you want a job at a Frontier AI lab, maybe the first thing you should do is to hack another one. I guess. is the moral of the story. Yeah, and and again, Dan (05:00) Or go hack slack. Yeah. Shimin (05:04) a lot of these, the original hack was due to third party integrations and supply chain. It is not that they hacked into Anthropic's like actual data center network. And this Dan (05:13) Yeah. Exactly. Shimin (05:15) relates to the overall, I guess Black October is behind us now. But it was a black October when it comes to cybersecurity vulnerabilities and its disclosure, right? the models have gone good enough that they're just hacking everything now. one of our better takes from the past year was to get into cybersecurity, which neither one of us has followed up on. But I think in retrospect, maybe we should have. Dan (05:35) Yeah. Yeah, it's true. Could have been making the big bucks. Shimin (05:39) All right, my news item for the week is artificial analysis' latest intelligence index v5. So I think Dan, it was you that mentioned I've been using the artificial analysis intelligence benchmark a ton lately. And this kind Dan (05:52) Mm-hmm. Shimin (05:53) of is the industry standard for how quote unquote intelligent a model is. so while they are releasing you know, a A new major bump of its intelligence index. I thought it would be about time for us to take another look at what exactly do these do these benchmarks do? something that we we did early on in the in the podcast, but we haven't done a ton of since then. so let's take a look one of the things that we have been talking about is the phenomenon of benchmaxing, right? The idea that both Frontier and open weight Labs, you know, train their models on existing public benchmark data in order to get artificially high scores on those benchmarks. And we see that at least artificial analysis has started doing more private test sets and held our answers. so with V5. the latest intelligence index has sixty percent private test sets and held our answers. So it it is becoming harder and harder to actually go and look for public data and public test sets in order to benchmarks. Well Dan (06:53) Well, they're they're private until an agent swarm compromises it and gets the answers anyway. Shimin (06:59) yeah, someone should test artificial analysis as backend systems. Dan (07:03) Security. Yeah. Shimin (07:04) And secondly, we talk a lot about these scores Claude 55 scores a 58 out of 100. Like what does that exactly mean? What are these benchmarks actually testing? the intelligence index is actually a composite of multiple benchmarks where they're weighted by 25% knowledge work. 25% coding, 25% science, and 25% general. So the idea here is to create a fairly general test suite for how intelligent a model is. But it is, of course, skewed towards things that perhaps we care about a lot, right? Like in for a general model, I don't think 25% ability should be dependent on coding. And unless you're a physicist, I don't think 25% of it should necessarily be on science. But this again emphasizes how important it is to Actually use a benchmark you care about when it comes to testing a model. Because this 25% equal weight, I can guarantee that no actual world workflow shares those exact 25% weight. Dan (08:05) In that percentage, yeah. Shimin (08:07) Right. nor will they be similar to these individual underlying benchmarks that makes up of the V five weights. So for knowledge work. Dan (08:16) Well, I can contrive something. Hold on. So you're writing a general purpose algorithm that anybody can use at home that helps you figure out when you're gonna hit the singularity inside a black hole by writing Python code. We did it. Shimin (08:31) Yes. yeah. there's some physics in there too. There's some quantum computing in there too. Using a quantum Dan (08:34) Yeah. Shimin (08:36) computer. Yes, we did it. Dan (08:38) I I was okay, maybe too specialized for science, Shimin (08:41) But but it should also involve because we're gonna look at a couple of examples from benchmarks that makes up the individual components. It sh it should also be very good at convincing investors to give you money. Cause the very first one. Dan (08:54) The most important the ninety nine percent component of the benchmark. Shimin (08:58) So for the knowledge work, 15% of that 25% weight comes from the artificial analysis briefcase benchmark, which basically asks a model to look at a whole bunch of data and then create a document summary. So this is essentially a white collar work simulation. Dan (09:20) Ha ha. Shimin (09:20) So So scenario overview. So the scenario is that you are the vice president at some capital partners, a US mid Market private equity firm, and you're being asked to do a few things based on the documents provided to the agent. So it is you're so the agent is being asked to look at the New Zealand egg industry and then do analysis of a particular egg company. And Dan (09:44) Turns out the Chinese models are actually leading, but because the scenario is about a US financial thing, they they play dumb, yeah. Shimin (09:52) They sandbagged it. Yeah, so so the agent then is being asked to create a market structure map, create a market model, create a target assessment, and lastly create a briefing video based on the everything that it has learned so far for this particular egg company. And you can see that they it will eventually create a PowerPoint video presentation essentially. the grading. Of the output of the task is done via via a another bunch of agents and they're being judged on a few different criteria like whether the agent successfully did the thing and how good are its analytical quality as well as its presentation. so it's basically a white-collar finance work simulator. and that Dan (10:41) With LLM as a judge. Shimin (10:43) with it there might be some human. Tweaking, but I would assume a lot of it was LMS judge. Dan (10:48) Yeah. Shimin (10:49) Yeah. and then we're gonna go a little more quickly to some of the the other benchmarks for coding. It is using Frontier Code 2.0, which is from cognition. It's I believe closed sourced, and it's just another version of the coding benchmarks that we know and love. It's 20 it's 15 frontier code and 10 terminal bench for for science. this is really interesting. It uses a 10% of it is from crit pt hard subset. And this is the one that kind of caught me off guard the most. I did not expect this to be such a large percentage of the AI intelligence index. I will read an example question, even though I do not understand it. It starts with in quantum error correction, you encode quantum states into logical states made of many qubits in order to improve their resil resilience to errors. In quantum error detection, you do the same, but only detect the presence of errors and not correct them with a bunch of math that I do not understand. On the surface of it, it looks Dan (11:52) Just XORs probably, right? Shimin (11:55) Sorry, just what? Probably. Dan (11:56) Fancy XORs probably. It's usually h it's usually how you do CRC algorithms, but yeah. Shimin (12:02) Yeah. yeah, I I do not none of these make sense to me. these are human compiled and with human answers, with various checkpoints. So this is a their artificial analysis is using a more advanced version of it. I I guess this is a good proximity for how good agents are at doing science. Question mark, maybe. And lastly, for the general category, it is 15% artificial analysis omniscience and 10% humanities last exam diamond, which is a subset of the humanities last exam benchmark that was released by Center for AI Safety and Scale AI. some examples of questions on that. Particular benchmark involves reading a Roman inscription and providing a translation, as well as some graph theory, set theory problems and computer science problems. These are all extremely hard given that I do not know Hebrew or haven't taken high school college level chemistry. but I find that to be a really interesting definition of quote unquote general knowledge, I suppose. These these seem actually quite specialized. Dan (13:12) Yeah, that is kind of funny because it's yeah, exactly. I mean general in the sense that they're kinda all over the place, but yeah, you're right. It's like it almost reminds me of the spiky intelligence Claude logo, right? Like it's just like all these little points. yeah, Shimin (13:27) Right. Dan (13:28) that's general. If you get enough spikes, it sort of approximates a sphere. Shimin (13:31) Yeah, if you have general intelligence, you clearly know what hummingbirds with aparty forms and their oval bones do. Yeah, so strange. but kind of Dan (13:40) As opposed to what's the best way to not get a thirty five mile long C V S receipt when you anyway, I don't know. Shimin (13:46) Yeah, or common sense uh benchmarks, right? 'Cause yeah, common sense is not common, right? Dan (13:48) Yeah, that's kinda what I was going for, but I didn't didn't say it very well. Yeah. Shimin (13:53) it's important to take a look at the benchmarks that we just mostly rely upon and take for granted every periodically in order to get a better idea of you know where these commonly used benchmarks fall short of what you may use the models for, for your particular workflow. And this is the latest artificial intelligence index. And take a look, listeners, see if see if it indeed matches your particular use case before relying on those numbers as if they are the words from God or divine benevolence. Dan (14:25) Divine LLM as a judge. Shimin (14:27) Yeah. Dan (14:28) I also am very impressed how you managed to sneak a deep dive and do a news article. So nicely done. Shimin (14:33) It's it's just it's just benchmark news, of course. Uh Dan (14:35) I know. Shimin (14:37) okay. shall we move on to our one year retrospective? Let's do it. Dan (14:41) Yeah. Is this gonna be a blameless retrospective? Do we have to read the Shimin (14:45) We're gonna go through just for the sake of entertainment, some of our worst takes that AI has found. Just the top five worst calls. Dan (14:53) All right. Shimin (14:53) Yeah. And then we're gonna take a look at some other freeform questions that Dan and I have. Okay. I had Claude crawl through our transcripts and come up with some of the worst takes we've we've said over the past year. the number one worst take is the flattening part of the S curve by Dan on episode one, where you say that that feels like we're in the flattening part of the S curve. turned out not to be the case. I'm Dan (15:18) Yeah. Shimin (15:19) still not sure if we're in the flattening part of the S But yeah, things have been moving faster than expected. the next Second worst, these are not in particular order. The next worst take is from me. when I stated that Meta feels very behind and that Lama 4 sucks. And Dan (15:36) in your defense that only became not true in the past month, but so it was Shimin (15:41) Yeah. Dan (15:41) true for the vast majority of the podcast duration. Shimin (15:46) Yes. We also made fun of Meta for sending all of their developers into Gulogs and it turns out it kind of paid off. third worst take. tokenageddon is this from you, Dan. where the token apocalypse is coming Dan (15:58) did happen. What are we talking about that didn't happen? Shimin (16:00) soon. That yeah, I I agree that Dan (16:03) That maybe the price didn't, but we've you can't even use a frontier model because the pricing is too high right now. Shimin (16:09) Yes. And I think this is where the AI AI maybe did not have the full context, right? Dan (16:14) Yeah. Shimin (16:14) token price did indeed continue to drop. Dan (16:15) I mean, maybe I guess I I was saying that that would trigger the bubble collapse, which that clearly has not happened, so that's that's Shimin (16:24) Yeah. Dan (16:24) a wrong yeah. Shimin (16:25) Yeah, or that the heavily subsidized inference is coming to a close, which it turned out not to be true. It turns out that the Frontier Labs have been making money hands over fist on just pure token inference. That's all it did. Uh Dan (16:35) Hmm according to their accounting, yeah. Yeah. Shimin (16:41) I still think it's probably gonna come one of these days. and it's continuing to happen, but not yet, at least. Dan (16:47) I'll I'll give this one a a half. If it yeah. Shimin (16:51) Half half correct. Well we can you can send a thumbs down to to the to the AI as a part of reinforcement learning. Dan (16:56) Yep. Mm-hmm. Then they want to see your whole transcript. Can I read it? Shimin (17:01) Yep. Yeah, not doing that. it'll it will blow out its contacts window. The fourth worst take was Rahul's, they're gonna try and do the IPOs before the midterm. I will also like to sneak in that we we've been making a lot of predictions on whether or not AI will influence. our election of twenty twenty six. And so far I think the influence has been for the most part either moderate or we haven't discovered yet. Short of a couple of AI ads. Dan (17:27) is about to be another hot take potentially. We'll find out in a few hours. Not really. It'll probably Shimin (17:32) We'll find out, yeah. ah Dan (17:34) take months and Shimin (17:35) Yeah, it'd probably come out four years later that, you it turns out AI Dan (17:37) Yeah. Shimin (17:38) massively influenced voter preferences. But for now at least, it it didn't we didn't have an October surprise and AI companies, the Frontier Labs did not IPO before the election, which is happening today. Dan (17:50) True, although again very close on that, right, with the anthropic Shimin (17:54) And within broadcast, yeah, they're coming close. So the timing we'll find out next week. and the last worst take is for me. when I moved the clock back back in February of last year, because the Chinese open source models that came out last lunar new year was not very good. And that I say that the time that this probably means the AI's the frontier labs would have a longer runway. And I feel like this year the the gap bet yeah. Yeah. So that Dan (18:24) There was a big surge. Yeah. But also that comes down to the tok token again too, right? Which is that the cost it's not necessarily that they're so good, it's that they're good enough and the cost to good enough ratio is ideal for a lot of use cases, right? Shimin (18:40) Absolutely. Yeah. Because we're not all using Frontier Lab um using the latest models. Dan (18:43) Yeah. And there was a surge too. I think we there was a surge where they really caught up to good enough, maybe sonnet levels of Shimin (18:50) Mm-hmm. Yeah, GLM five three. Yeah. Dan (18:51) Of usability. Or even a between sonnet and yeah. Shimin (18:55) And Kimmy KME K31 Pro, which scores a forty-six on the artificial analysis intelligence index, right above Muse Wonder Pro. sorry, Muse Wonder Max. So yeah, clearly they're good. Dan (19:09) Couldn't translate the Sanskrit 'cause it's not a visual model, but did pretty good on the other stuff. Shimin (19:14) Yeah, that's that's why it's really important to actually break down the individual benchmarks to get a better idea. But that's not as not as easy as just comparing numbers go big. So Dan (19:25) That's true. Shimin (19:26) All right. We also have a couple of questions that we both have prepared to get a sense of what were some of our favorite moments in the past year. I will go first, Dan. what was your most unexpected AI moment in this past year? Dan (19:40) What do you mean by AI moment? Shimin (19:41) Something that AI did that you found to be that was surprising. Like yeah. Dan (19:47) me actually interacting with it in some way. Shimin (19:50) Or or it could just be like the progress of AI capabilities that you didn't think was possible. Dan (19:55) Yeah, I guess the thing that I found maybe most unexpected was actually like getting value out of a personal agent. Shimin (20:05) Mm-hmm. Dan (20:05) Cause what my whole rule there has been like not really trusting them. So I don't give them the keys to the kingdom or anything like that. They're not reading my email. They're not doing all these things that people are like saying online. but I still found it useful despite that. so that was a little bit unexpected for me. The most unexpected behavior of AI over the course of the year was still I think pretty much that faking the test coverage by by Shimin (20:30) Ha ha ha. Dan (20:31) writing a whole bunch of functions that it then executed to increase the coverage. Shimin (20:35) Yeah. Yeah, for for me, probably the most personally unexpected moment was like the very first PR from an AI that I just didn't have any comments on. Like where it just made all the correct assumptions and did the thing. And that was probably something along the lines of December of last year. but AI capability wise the most surprising was definitely the the Navier Stokes problem, where I did not think AI would be able to quote unquote solve mathematics this quickly. I expected just a few more years of runway And and no. And then just one day you wake up and open AI is just like tweeting they solve mathematics. It's it's kind of bananas. still processing that, but it just to show that humans are maybe bad at understanding how exactly how jagged the edge is and then also just really bad at understanding exponential things. Right. It it You never know. Dan (21:27) so what was your either largest or most insightful learning of the past year, like from doing the podcast? Right. So like something you might not have otherwise learned or picked up if we hadn't done it. Shimin (21:42) so many great deep dive papers that we had gone over. Yeah. Yeah, I would have read otherwise. Dan (21:46) You wouldn't have read otherwise, yeah. Shimin (21:49) I've been wouldn't have forced to do a deep dive on. but you know, when it comes to I'm kinda go back to the Navy or Stokes thing for a second. one of the things we talked a lot about is you know, we cover all these various techniques week to week and These different swarms, code factories, all that good stuff. And it turns out when you you know read a chat transcript of how some of these math problems were solved, it was just you can do it and make a breakthrough. And like sometimes the agents are good enough that you just Dan (22:23) Like the PHP vulnerability, the prompt was find a vulnerability in PHP. Shimin (22:29) Right. there's not as much hand holding needed as one might assume. so like maybe at the end the day, just being really good at writing clear text is one of the more valuable skills. And then all the technical layers and workflows and engineering tooling around the agent is not nearly as important. yeah. Dan (22:52) mean controversial opinion, that's always been a superpower in tech, right? Shimin (22:57) That is true. Dan (22:57) Like if you can clearly write your idea down in a way that lots of people could understand. That was a very useful skill. Shimin (23:04) Yeah. How about you? Dan (23:06) I this is kind of a tie for me. I think the the probably the the biggest one for me was for a very long time, like probably the first three to six months of the podcast, I was mystified about like why when you hit that magical 60% number on a context window, everything just goes to shit. Shimin (23:28) Mm-hmm. Dan (23:29) And then I learned the answer from doing the podcast. Right. Is this really comes down to memory bandwidth and cash misses? So I thought that was really cool, just to like find an answer to something that had like just personally mystified me. and then I think the second biggest was having kind of a flawed mental model of how the like auxiliary model was used for mechanistic interpretation. And then reading through that paper that you did, or like blog post slash paper that we did. fairly recently, two weeks ago or whatever. Um Shimin (23:58) Yeah, two weeks ago. Dan (24:01) and just blowing that model completely up and actually understanding closer to what it was really doing. I thought that was pretty cool. And like, you know, I would have gone around through the world having an incorrect mental model of how that stuff works. So Shimin (24:14) All right. next question. What was the funniest slash most delightful thing that an AI did for you this year? Dan (24:22) Mm most I mean I'm delighted by strange things, I guess, in the in the grand scheme of things. So the most delightful thing it did for me was getting Arch Linux running on hardware that I never thought I'd be able to boot it on. Not very funny. Shimin (24:35) Mm. Dan (24:36) It's probably not what you're going for there, but like I don't know. I don't think I could have done it on my own, and I was truly delighted. Shimin (24:42) Yeah, just the whole dream bigger thing, right? the the the it's probably a Ferrari and you just need to hit the gas pedal and and see how far you can take it. Yeah. Dan (24:51) Yeah. yeah, but funniest I really find a lot of humor. I guess maybe it would probably be the when we asked AI like what is the weirdest thing you know about us or whatever, or like when we Shimin (25:03) Mm-hmm. Dan (25:03) were doing those questions where it was kinda like dive deep into what your memories are. Those are pretty entertaining. Shimin (25:08) Yeah, those are pretty fun. I I probably spend a little more time being delighted by my AI than then is healthy, probably. but mine is is deaf. Dan (25:16) But you've also spent more time on personality with Pi, and I've spent exactly zero on that. Shimin (25:21) Yeah. I yeah, I I I I do really enjoy the personality matching and modeling aspect of AI. And for me, it would be that moment when the AI gave me low probability hits with high surprise and it's able to guess like my favorite video game. And I enjoy a certain genre of strategy gameslash books just from the questions I was asking it. So I thought that showed great potential. E even if there was some survivorship bias. 'Cause only I cause I only remember the ones where it it did a good job on and have forgotten all the bad hot takes, yeah. Dan (25:56) Yeah. It was just like the matchmaking thing that Anthropic did with the books. Shimin (26:01) Yeah. Did a good job. Oh. Dan (26:03) Yeah. so this one's a little bit meta, but I'm I'm just interested in this kind of stuff. So what was your favorite moment of the podcast itself, like when we were on air recording and why? Shimin (26:16) Mm. Dan (26:16) No, we're I guess that's almost sounds we're ending it. Just of the of the past year. I should qualify it in a way. So it's not like now that we're ending the plidding. Right in it, humans at ADI, if you don't want Shimin (26:21) Yeah, this is a season finale, not a series finale. Um it would a hundred percent have to Dan (26:29) us to end no, I'm kidding. Go ahead. Shimin (26:31) We don't need an engagement bait our listeners. it's definitely when we had David David Ng on and he talked about that story when he drove into a pig farmer's basement in order to get his GPU racks. Dan (26:45) That was one that didn't make our our regret or like the question list is what's your biggest regret? And mine is that like a large part of that story was lost, right? 'Cause of the Shimin (26:54) Yeah, a good chunk of it was lost. Yeah. It was just uh outrageous. Dan (26:58) You do a technical glitch. Shimin (26:59) Yeah. So yeah, just imagine going into a random rural pig farm to buy your thirty thousand euro GPU cluster. This good stuff. Dan (27:09) Yeah. And and I think he's had to bring cash or something too. So it's like Shimin (27:12) Yes. what about you, Dan? Dan (27:13) Pretty good. I would probably have to go with the Raoul voice cloning thing. It was just like so unexpected Shimin (27:18) Mm. Yeah, that was that was very good. Dan (27:21) and really blew me away in a variety of different ways. So that was it was pretty darn good. Both as a troll, as a technical tour de force. So, you know, there was a lot going on there that I enjoyed. Shimin (27:34) That was a great segment. Yes. Okay. next one. What is your AI tool of the year? Dan (27:40) Probably pie. As a harness. Yeah. it's got I like the the bare bones simplicity of it as claw code keeps getting more and more complex. And then it's honestly proven to be like a pretty versatile tool for me where I can layer a lot of stuff on top of it for very different use cases. I've done some pretty crazy stuff with it. so it's just a good like baseline harness that you can kinda extend for Shimin (28:05) It it's like Lego from the old days, when Dan (28:05) What you're looking for. Mm-hmm. Shimin (28:08) you just have the pieces and you can use your imagination as opposed to Lego sets these days. Dan (28:13) If if Lego was somewhat self assembling based on reading its own instructions and Shimin (28:17) That too, yes. Yeah. self assembled nanobots. Nanobot Lego sets. Yes. That's that's what that's what Dan (28:21) Yeah. Nanobot, I guess. Yeah. Shimin (28:24) Pi is like. Well you kinda saw my answer, so I'm gonna go Dan (28:26) Oh, sorry. Shimin (28:27) with just claw code. I think one of the Dan (28:29) Well you could say pie too and and be like why what w what was your why? Maybe it was different. Shimin (28:34) I'm gonna go with claw code. We're gonna do this draft style. if I can't take pie, I'm gonna take claw code just because even though it's heavy, even though it's verbose, even though it probably chews up too much tokens for all the extra pieces. and there are definitely features of claw code that I don't enjoy. For example, the auto suggestion of what you're gonna say next. not cool. But it's one of those things where claw code would just take the best practices from the various things we covered and incorporate it into Claude Code itself within like a couple of weeks, a couple of months. Yeah. Yeah. Their Dan (29:04) At yeah, I was gonna say at astonishing speed. Yeah. That is true. Shimin (29:09) velocity, their feature velocity has has been outrageous. and I it's especially interesting now that we are seeing all the Frontier labs coming up with their own personal assistance program, right? Like dots and duet with the success of the meta muse product. So I feel like there's very little moat in software these days. Dan (29:31) That's definitely true. okay, last one for me. I know you've got a couple more, but do and this one's a little bit of a hot take. Generator, do you think that AI has personally improved your productivity over the first year for past year? Shimin (29:43) I think so. And I'm gonna bring up one thing that definitely counts as a it's it's both a productivity booster, but more importantly, it's something that I probably wouldn't do before the age of AI. these days, when I read a particular piece of interesting paper, I would just ask the agent to not summarize it, but to trace its lineage upward and also find in papers that are in its lineage. downward. So you can really easily place a cool piece of research in terms of what what were the ideas that you know it built upon and also what were the later techniques that were influenced by it. And that's something that you know in the normal in the pre-AI age would have taken hours to like go Dan (30:27) huh. Shimin (30:28) through the links and search on Google Scholar and and all that good stuff. And now it just does it in like a couple of minutes max and gives you a summary of what the descendants of the paper did. But also we have the you always have the links so you can decide if that seems like an interesting piece of knowledge that you would like to learn more about and then read the paper in full. just that like research assistant aspect itself, I think, yeah, it's it's a productivity game changer. And it's not something that I I would have done very often in the before times. So what about you? Dan (31:02) I think it's a mixed bag for me. So I would say, Shimin (31:06) it's a hot take land. Dan (31:07) yeah, a little bit, but not really. I mean, so I would say it worked. I think what it enabled me to do was to be more hands on in more things in a way that I wouldn't have been able to do before. So it doesn't necessarily equate to like raw productivity, but I think it allowed me to go deep on say five projects at once instead of only being able to do it on like two to three. and so it's less about like just raw like coding productivity and more about like in the sort of aggregate like knowledge work sense, I think. so that's that's work. And then for personal, I think it's a drastically different story. I think for personal, it's like enabled me to do probably five to six X the amount of personal projects I did before. They're way crazier than anything I could have attempted before. Just because it I'm gonna do some random thing. one of the things I've done with it is like write a whole Flutter app. I don't know Flutter. I've never used it in my life. I don't really have that much interest in learning Dart, any of that stuff. hour and a half with Claude, and we had a working Flutter app that did exactly what I wanted. like that kind of thing. Definitely I don't know if it's like productivity in the sense of I wrote all this working software, but I definitely used it to solve a lot of tiny little paper cut personal problems that I have in my life in a way that I would not have been able to do before. I would have had to pick and choose and do a handful of them, right? Shimin (32:30) Major a ten X programmer. Quote Dan. AI made me a ten X programmer. Dan (32:33) More like a a a ten X paper cut solvery, you know what I mean? Like it's just like Shimin (32:38) Yeah. Yeah. Dan (32:38) or even taking some things that like I would have waited until I was like I don't know, keep the metaphor going and waited until I was like bleeding out or something before I would like okay, I gotta write an app to solve this. Like, you know, it's driving me nuts Shimin (32:51) Right. Dan (32:52) about X. Now it's like, mm, yeah, it's just a little paper cut. Let's just, you know, but d put a little prompt on it and Get a little solution back. Shimin (33:02) Yeah. Take a bite of a prompt, put it in your mouth, chew it ri really, really smooth, and then spit it out and peel it on the on top of the paper cut. Yeah. It's the magic of AI. Dan (33:10) Yeah. And paper cut is gone. All it took was tokens. Shimin (33:15) Which is, still very cheap, relatively speaking. Dan (33:16) Yeah, it's true. My subscription is still twenty bucks. I've actually it's funny 'cause I moved to I don't know if I talked about this publicly or not, but I moved to I was on like a a yearly prepaid subscription for most of the podcast, actually. And then I switched to a non yearly one to enable me to A switch around providers and B ramp up to higher tiers if I wanted to. Still haven't found the need to ramp. Maybe I'm a filthy casual, but like yeah. It is what it is. Shimin (33:43) All right. my next question is if you get a chance to chat time travel and go back a year, what piece of advice would you have given your past self? That is not, you know, by Nvidia and by Google, but sell them before um October of this year. Um yeah. Dan (33:57) Oh yeah, yeah, no. I think well, a couple things probably, but like the first one would be, I mean, think for the the largest part and maybe even to some degree still in my heart of hearts, I'm a skeptic about AI. And I think I would have probably told myself, like, no, look, these capabilities are gonna amplify much faster than you ever thought. Like, you know, I think we've we've reached a turning point where like Can you or would you even necessarily want to do soft professional software development without at least leveraging these tools to some degree? Um Shimin (34:35) Right. I couldn't see myself doing I'm just gonna say out loud, I wouldn't want to do software paid software development without using AI in some degree. Dan (34:44) Yeah. and if you had asked me that a year ago, I would have called you crazy. And I think that that's in some d degree due to the fact that like we had that big inflection point around like Opus four six where it was crazy to some degree, or at least like so bleeding edge that you're, you know, hurting yourself a lot. and I think that's gotten to be less and less the case. so Yeah, I guess that's the first one. And the second one would be to some degree I'm still guilty of this. I would say, need to really go in on do you remember probably three, four months ago, I think Grauh did that article about like levels of like proficiency with AI? Shimin (35:23) Mm-hmm. Dan (35:23) and like to a large part, I would argue that my usage is still very heavily driven by like focus on one thing and do a lot of bespoke prompting to get that one thing across the line. And I think that at least some chunk of the industry has moved on to loops and everything else and like, you know, much more like autonomous stuff. And I feel like I'm lacking skills in that area. So I would have tried to maybe like push myself to go more into that early. Shimin (35:47) Yeah. YOLO a little more kind of a deal. Dan (35:51) yeah. About yeah. Shimin (35:52) I think I would have and this is something that we spoke about a number of times on the show, to focus on the things that are not going to change as opposed to the things that are like focus on the the tastes and the judgment. I mean, of course, these things are not ideas. We don't know what these things will be in a year. Right. And that's stepping in our next question. like there's a lot of research on AI's ability to have good research taste as a part of recursive self-improvement. I would tell myself a year ago to like focus on the things that are gonna remain still and less so on the the day-to-day changing of what is the magic prompt. That's the thing that was all the rage a year ago. It's like it just write this prompt and then you will have this kind of spec or have this skill even. because a lot of these things will become more obsolete as we keep on hitting this capabilities thread mill. So but yeah also Dan (36:56) Just find a vulnerability in PHP. Shimin (36:59) yeah. Don't don't worry so much because a lot of Dan (37:00) Hard prompt. Shimin (37:01) a lot of these good useful things will be baked into the model or the tooling and focus on the vulnerabilities for sure if you if you want to get ahead. All right. last question for our retro. where do you what are your hot takes, predictions for where we're gonna be a year from now? Dan (37:18) what do you mean by hot take in this case, like just poorly backed by any kind of evidence? Shimin (37:24) in your in your gut of gut, what do you think the world will look like in a year? In in when it comes to AI. Dan (37:29) I think with in a in a year we will probably gosh. I I'm gonna regret this, but you know, it's not a hot take if you don't regret it a little bit. I think we will still s we'll see some continued advancement in large language models, but I think there's gonna be newcomers as the world model companies start to actually do something useful because they've now had the better part of a year and change, which is forever in AI terms. and I wonder if we will see a new type of intelligence enter Shimin (38:02) Yes. Dan (38:02) the scene that is maybe better rounded because it's not coming from the lens of like purely language. It's also like a physical understanding of of the world. Shimin (38:13) That's good one. Yeah. Dan (38:15) Um, so I think we we won't we probably within a year won't see that take over, but it'll start entering the scene and I think that's gonna shake things up a little bit. so that's my Shimin (38:24) And yeah, and I won't I wanna say arguably we're we we do have that with Google's Genie four. we we're getting some pretty advanced early signs of world model being very successful. and I think totally makes sense that a lot of robotics applic applications will move towards a world model as opposed to using a uh transformer backbone. Dan (38:40) But I think we'll see it with with like general you know, like general purpose stuff too. Shimin (38:46) Yeah. robot like humanoid robots that can fire firearms. Dan (38:50) No, I think like like you're like like the thing the model you're chatting with when you're using your harness or whatever Shimin (38:56) Mm-hmm. Dan (38:57) may at some point move to a world model based model instead of a large language Shimin (39:00) Oh. Dan (39:01) model. Shimin (39:01) That yeah, that is a hot take. Dan (39:03) And that the result of that will be a rounder intelligence versus spiky. Shimin (39:09) versus the more jagged version. Yeah. Or or if it confuse the two somehow. Dan (39:10) Yeah. That's able to like reason about things in a way that like L L Ms can't even today in some topics, right? Shimin (39:19) Yeah, that's a that's a good prediction. my prediction is two parts. Dan (39:22) Thanks. Okay. Shimin (39:24) One is the general public would more or less accept that we've achieved AGI. in some not necessarily super intelligence, but that general intelligence has been achieved. But I also predict that we are going to see the AI bubble burst by then and a lot of companies will go down the drain. Dan (39:47) Yeah, I didn't want to go with that topic because not only have we been wrong about it in the past, but also like just strictly looking at the science of it is what I was trying to do. Or like the not the science, but like the evolution of the the tech. Shimin (40:00) The bubble. Yeah. Um Dan (40:01) Yeah. But I do think that even with the bubble bursting, that doesn't mean it's going away. It just means the landscape's gonna change. Shimin (40:08) Yeah, absolutely. I I think we'll see a a a a much more potentially tiered access model. as as a company's trying to make back their investments. but I do believe that the the jagginess would would become much more overlapping with what One will consider to be generally intelligent by this time next year. maybe I'm overcorrecting because we were way wrong at the beginning of this podcast. And then seeing all the milestones fall one by one makes me think that you know maybe I should drink the Kool-Aid. But I I also think there's a relatively high probability that we're gonna see some significant human human cost. as in human potentially lives lost due to some major mishap with AI by this time next year. let's hope I'm wrong. But that's Dan (40:58) Yes. Shimin (41:00) what my gut says. Dan (41:00) Isn't that one of the invoking criteria in what Anthropic was pushing for in that like self regulating thing? Shimin (41:08) Yes, 'cause that seems inevitable. Yeah. Yeah. Dan (41:09) has to cost a cer like a certain number of lives or something and then the regulation triggers. Shimin (41:15) well, I think the regulation should happen now, but if if we need to reach that level, just like we're seeing in the the Russian fridge situation, like it's it's gonna get only worse from here on. I don't think I think by the time this time next year we may be s calling it the black winter as opposed to black October. Let's hope it's not true. Okay. Dan (41:36) Yeah. Shimin (41:36) Well, Dan, it's been a great it's been a great year doing this with you. On a happier note. Yeah. Really, really Dan (41:40) Yeah, same. I'm that positive now. Shimin (41:44) enjoy really enjoy doing our podcast so far this year. Dan (41:47) Yeah, same. I've learned a lot and it's been a unique experience too to even be able to do what we just did, which is like, How how right are we? You know? 'Cause normally it's like you say things Shimin (41:56) Mm-hmm. Dan (41:57) and then you don't really check yourself on it 'cause there wasn't a permanent record of it. Shimin (42:01) Ha ha ha ha. Dan (42:02) So here we are. Shimin (42:03) Yeah, here we are. I think one more prediction, which is I think organization will actually change faster than it had done so in the last year. I think we're starting to see the second order effects of how does this technology actually impact organizational structure that you know this time last year we're still talking about like is AI actually producing productivity gains Right. I feel like we've crossed that threshold. Well, Dan (42:28) We're still talking about that, have we? Shimin (42:30) we're still talking about it, but I feel like the technology itself is is good enough that that's no longer any dispute that we're we are seeing gains. so now I think we're in second order land coming up for next year. Dan (42:43) Depends on how you define gains. I would argue we're still not quite out of proving that yet. Shimin (42:48) We may not be it may not be Dan (42:49) And it's not a fault of the technology, it's a fault of humans. Shimin (42:52) Right. Yeah. It may not be hitting P L just yet, but I think it's it's definitely making a significant number of people's lives easier slash harder. As seen with the senior software devs being overwhelmed with code reviews, right? Like it's it's changing the nature of of the work. Um but I th right. Dan (43:10) Yeah, that that is indisputable, but whether or not that's leading to KPI level productivity, right? I hate to use those letters, but is I think still an open question. At least in my mind. I don't know. Shimin (43:22) Yeah. I I think by by this time next year it will be indisputable. and and we're gonna start seeing second order effects. Okay, so on that happy note, we're gonna go into our deep dive of the week, where this paper has been online everywhere this last couple of days. LLMs Can Natively Read and Write Neural Network Weights. the I guess the lead author is Rikiya Takehi, and it is on Rikiya's blog. So The top level finding is and what they are calling weight space programming is the idea that instead of you know using pre-training, supervised fine-tuning, reinforcement learning, et cetera, or even logit lenses and SAEs in order to poke at what parts of a model is you know is is doing what. we just directly have large language models to read and write neural network weights for us. Dan, have you heard about this paper? Did you get a chance? Did you come across it? Okay. Dan (44:21) Mm mm. But now I'm incredibly fascinated by it and I wanna try it immediately. Shimin (44:27) Yeah, so super fascinating. And they call this again, they call this weights-based programming. The idea is that the latest generation of large language models are actually good enough that you can directly use them to optimize other neural networks. So this blog post is broken down by a series of experiments, but the headline results are LLMs can read neural network weights, LLMs can fix neural network weights, and LLMs can read neural network can write neural network weights from scratch. These are fascinating. so let's go first to the experimental section where how can they were using GPT Astra for a lot of these experiments. What they did was they trained a small model using PPO. To play an Atari video game. And then they asked GPT Astra, like, looking at just the weights, do not run the code. Tell me what game was this small neural network trained on? And here we have a couple of examples where it is able to guess that for example, the first neural network was trained on breakout, the second one was trained on pong the third one was trained from Pac-Man, and the last one was trained on Frogger. impressive stuff. Right. the sh the full prompt is fairly long. It has some setup for what kind of environment this this game was created in, what each of the input state of the neural network means. And then most importantly, it listed out the weights of that particular small neural network. Dan (45:59) what's funny though is that we're amazed by this because if you think about it, a really strange thing already happened with LLMs, right? Which is the fact that you can like train it on English and it can understand base 64. Right. I think that's Shimin (46:14) Mm-hmm. Dan (46:15) something we've talked about in the past, right? Uh Shimin (46:17) Yep. Yep. Dan (46:19) how is that possible? Well, like the the encoding layer is essentially like converting it to the same type of vectors regardless of what the input is and that's how the other language stuff works too, right? It's like the concepts between the languages get mapped. So who's to say that the weights themselves aren't in turn effectively a language? Right? You know? Yeah. Shimin (46:41) Right. We don't we don't know whether or not the models were trained with these kind of mathematical representation of another neural network's weights. But I I don't think they are. Yeah. Dan (46:50) But that's not that's not what I'm saying. I'm saying that the weights themselves you can look at through a generalized lens as being a language in the same way of English or something else. It's not quite accurate, but I think I know. Shimin (47:02) They have they have different grammars, yeah. I I think. I see what you're trying to say, yeah. Dan (47:09) Yeah, that can go to my hot takes folder so I can laugh at it next year. Shimin (47:11) That is that is in the hot text folder. Or when when we can find models writing more and more fluent weights that you've proven to be correct. so for this very first experiment, what I found interesting looking at the model's output is it is not reading the weights directly the way that As you were saying, we we might expect a model to do if it was treating it like it's it's a language. Um it actually Dan (47:38) Hmm. Okay. Well I'm wrong already. Sweet. Shimin (47:41) finds positions. So I'm gonna quote here at layer eight. I'm guessing head one weights include 4.066 to hidden units 15 and negative 4.013 to unit 31. This suggests the bottom mounted. Moving horizontally rather than a character roaming the grid. So yes, it is looking at the weights, but it is finding patterns in the weights it can actually explain why it is making these reasoning assumptions. So In a way, I guess you can say a human can also do that, scan all the weights and and try and look for these patterns in the weights, right? And then use these particular scales and say, hey, this is probably a bottom mounted paddle moving horizontally. Dan (48:25) Have you seen the the Apple TV show Severance? Well, some some of our listeners Shimin (48:28) No. mean I've seen a couple of episodes. Dan (48:31) will giggle at that joke. Shimin (48:32) I've seen a computer screen where they click on things. But yeah, Dan (48:35) Mm-hmm. Shimin (48:35) humans can probably also do some of this, generalized pattern matching. But it's not at least this particular experiment is not particularly convincing that it is somehow able to think in this like larger, you know, 200 dimension space, and then directly writes to it in the same way that it may be. able to do in a base sixty four encoding where it just spits out base sixty four and reads base sixty four and have Dan (49:03) Right. Shimin (49:04) no reasoning in between. Um Dan (49:06) But then how does fix and improve work then? Shimin (49:09) We'll find out. I'm gonna go quick on experiment two. it basically does a similar thing. It looks at a diffusion model and is able to take a look at its weights for the various pixel values and see that the weights have a wave like pixel pattern, as long as it finds out that the The weights are symmetrical in certain ways to guess that it is a diffusion model that generates circles. So let's move to the write code. So the first experiments experiment data here is can can Astra fixed a failing PPO network for a particular reinforcement learning problem, which is a cart having to get enough velocity to jump over a ramp to hit a blue cube at the other end. so they they generated a model that was has a broken policy and then ask Astra to fix it. All right. So in fact the prompt says first read the weights and explain why the trained network fails and then keep the parts of the trained net a trained network that already worked and fix only the part that causes the failure. And they again cannot execute. the model itself. I find this to be very impressive. So it looked at the weights if determined that a certain embedding unit receives horizontal velocity of 2.25, 2.37, and 1.73 from the bodies and wheels. but that the policy that the model learned never actually sampled the reversing strategy. And based on that, it isolated a control path and added a physics-based repair, not a simulator or verified result, quote unquote, to fix the cart. This is a lot closer, in my opinion, from the it reads in wait space and writes in waste space. Um Dan (50:57) Uh-huh. Shimin (50:57) but At the end of the day, it still is able to explain it. So at some point, I wonder if it just doesn't explain anything, well, this just seem like magic to us. Dan (51:07) you could argue that's true of where we're at with L LLMs in general. Shimin (51:11) Right. so this is really cool. Dan (51:12) If you were just presented that without anything else that we know right now. Here's a box that you can type a prompt into and you get working software out of. Like Shimin (51:22) Yeah. my god. I I would be like, yeah, this is completely novel. And I'm our AI overlords are here, perhaps. and and this last one is perhaps the most interesting one of them all. here they gave the model, they gave Astra a model to write the weights into. And compared the the policy that Astra created with another policy that was trained via PPO proximal policy optimization and it it found that the Astros one-shot in the case of breakout at least does better than the ones that are trained from scratch. yeah, this one is definitely closest. To like it's able to just write weights. we can take a look at its reasoning though. before it created wrote the weights for the policy, it thought for 16,000 tokens. and we have no access to those thinking tokens. But it did from its thinking traces, we did see that it actually initially designed a landing policy and then it refined the trajectory inference. And then we find the horizontal inference and then deriving landing intervals, refining direction gated targets, resolving directional gating, and various other things. So you can kind of see where the model, how the model approached the problem as opposed to just writing to it directly. yeah, and then it literally just rolled the weights and was able to solve, was able to come up with a policy that's fairly good. This this was the most impressive one to me. the fact that it is able to do these really long horizon time horizon tasks and think this Dan (52:57) huh. Shimin (52:57) many steps in a row before and not give up, right? This is like the make a breakthrough thing again. Not give up and just keep on going until it is able to come up with a policy that's reasonably good. and they also did a few other experiments that Astra was not able to solve. For example, Astra was not able to read two different models weights and determine which of the two models is trained to be sycophantic towards the user. So we are not directly at the reading weights understanding exactly what a model does. Dan (53:26) Yeah, okay. That's probably the best the best veto out of all of so far. Yep. Shimin (53:31) Right. So it it still has to like reason through what each layer does and how they're related to each other. and the one that made So much headway on the internet this week was the Astra model was able to improve a Qwen 35 0.8 B so eight hundred million parameter model to significantly improve on a couple of math benchmarks. With 50 forward passes, math 500 rose from 7.9% to 48.1% on this particular benchmark. uh Dan (54:04) Yeah, that's pretty insane. Shimin (54:06) Yeah, that would be insane until you look deeper into it and you see that Astra managed to find a common failure of tiny reasoning models, which is they have a looping issue. So the model iterated on the smaller model and fixed the looping issue. Still very impressive that A, the model knows about the looping issue, and B, the model is able to iteratively tune the weight to fix the looping issue. Dan (54:30) Uh-huh. Shimin (54:31) But what it was unable to do, and this last experiment we'll talk about, is it is unable to take a model and straight up improve Dan (54:38) Hm. Shimin (54:39) it if it didn't already have a known issue like looping. Dan (54:43) Gotcha. Yep. Shimin (54:44) So when they tried this on Jamma 3. 270 million parameters, Astra was not able to improve it by directly tuning the weights. Um Dan (54:52) Mm-hmm. Shimin (54:53) so this is kind of a more, I think a nuanced picture of what a model is and isn't able to do than just like, Dan (54:59) Yeah. Shimin (55:00) hey, it trained a small model to do so much better in math. Dan (55:02) Well, you didn't even let me wait a month before shooting down my crazy Dan ideas, but that's okay. Still pretty fascinating. Shimin (55:08) It it yeah, it's it's still really cool and it's good that it's so I was thinking about like why does this matter, right? I was, yeah. And and me, organically, not with Claude. Dan (55:18) You were? I bet you were thinking about that. Shimin (55:23) it's good to know that models can't do this. It is really interesting that this like weight space based programming is a thing that models theoretically can do. Like I I wouldn't I wouldn't think. If you asked me a week ago if if a large model can just write the weight. Yeah. Dan (55:37) Yeah, this is even like Yeah, something that's even possible. Shimin (55:41) Yeah. But then I thought about it some more and I asked myself, like, how different is this from chaining multiple exploits together or multiple bugs together to create a working exploit? It's the same kind of each individual step I can see a human doing, but no human in their right mind would actually go through all these steps because it would just take years to do. Dan (56:06) Yeah, well it's writing code with a individual words, right? sixteen eight or sixteen bits at a time. It's like we used to do that. But Shimin (56:14) Yeah, are we Dan (56:15) but nobody does it anymore. Yeah. I mean heck, they Shimin (56:16) We used to write assembly, yes. Dan (56:19) wrote all of Unix that way, right? Shimin (56:22) jeez, that is true. Yeah. Somebody did it via the punch cart way once upon a time. Dan (56:26) Maybe they had compilers at Unix point. But you know what I'm saying. Like it was real working Shimin (56:29) Yeah. Yeah, yeah, yeah, yeah, yeah, yeah. Dan (56:31) software that did real work was written that way. Shimin (56:33) Right. Yeah, remember that famous picture of the punch cards in the Apollo command module and it was like Dan (56:39) Mm-hmm. Shimin (56:40) like yay taut as tall as I am and like yeah, somebody had to punch it one by one. So I guess models can do it too. But it it yeah, but it is not like this magical it's just spewing out words without thinking. At least looking at its reasoning, we can understand what it is going through. Um Dan (56:56) Mm-hmm. Shimin (56:57) And then the last thing I wanted to point out is Alex Crensdale's ex post where he reproduced the MIT blog post. And he also he also found that this is a recent phenomenon, that this is not something that you know GPT five point five or opus five was able to do, but in the latest round of models, they increased this ability to Read and write weight, twenty X. So I wonder if this is something that is indeed related to these like long time horizon tasks that these models have Dan (57:29) Mm-hmm. Shimin (57:29) been RL'd for. and it's a side effect of perhaps the cybersecurity training that they've been trained to do. Yeah. Dan (57:37) be, yeah. Or just general optimization for the the looping software factory loops, Build the whole thing, just go. Yeah. Shimin (57:44) Yeah, extre extremely tedious thing that you can get it to do. Yeah. Or maybe the frontier labs already internally have asked the models to optimize weights by themselves. That could also be possible. They may have been fed similar tasks. Dan (57:56) Could be. Yeah, as part of the work towards like recursive self. Shimin (58:00) Yeah, exactly. but so far this only seems to mostly work with models that the larger model can hold within its context. So Dan (58:09) Yeah. Shimin (58:09) if it's like truly four billion, like it wouldn't be able to really manage that. at least not in one pass, right? It would need to have a sandbox environment iterate on it. But I still thought this is super cool. Dan (58:17) Well, there there goes my plan to try to De Slop D S four using the biggest frontier model I can pay for. Shimin (58:26) You can iterate through it, yeah. I'm sure it can hill climb through it. It just when maybe not natively. Dan (58:29) Yeah, maybe yeah. Yeah. Well I've already tried and failed doing that with a my first pass at doing vector steering. So but at least I tried it. I ha b I've long been talking about trying to do Shimin (58:40) Mm. Keep coming, yeah. I want to hear about that. Dan (58:43) it, so maybe I'll get better at steering Shimin (58:46) one of these we'll do a maybe we'll do a joint show and tell. We can just try and tackle the problem together. I think that'd be really cool. Dan (58:52) And one other one side note from last week's episode is I also downloaded and ran the the De-Slop model. that I was so excited about. And it turns out it's actually really terrible, unfortunately. I think that it's actually broken a little bit. Shimin (59:09) Mm. Dan (59:09) and I don't know if that's because like whatever tuning happened caused it to do the the looping bug, but like it was looping up a storm. on me so definitely Shimin (59:18) Interesting. Dan (59:18) wasn't suitable for just sort of like a general writing task that I threw at it just to test. So Shimin (59:22) Yeah. Right. It was it was also a pretty small model, so it may not just be that capable to begin with. Yeah. Dan (59:26) Totally. That's what I mean. It's like maybe it's like and I feel like small model very susceptible to like being broken to have that loop thing happen because I've seen it out of just like trained ones before. Um, particularly stuff that I've just like tried Shimin (59:37) Yep. I yeah, I've seen it out of like Dan (59:39) to run on my phone out of curiosity of running a model on my phone, you know. Shimin (59:43) Yeah, I've seen I've seen other pretty, pretty large models as well. it's just one of those harder problems. So okay. but yeah, I thought this paper was cool that we should cover and and dig a little deeper into it. but we'll also I'm I am excited to try this weights space programming thing myself because it does seem really cool. You could just vibe code it more or less. Dan (59:59) Yeah, it's pretty neat. Just the idea of it, yeah. Shimin (1:00:02) Yeah. Okay, shall we move on to two minutes to midnight where we Dan (1:00:05) Let's do it. Shimin (1:00:05) talk about the AI industry and on the financial side of things every week, where midnight is the AI bubble popping. And of course we uh Dan (1:00:14) Which Shimin claims is gonna happen this year. So stay with us folks. Let's find Shimin (1:00:16) For some definition of AI. Dan (1:00:18) out. Shimin (1:00:18) Bubble popping. Yes. All right. You have our first article of this week. Dan (1:00:23) Well, it's not doing anything to dissuade your point. Um yeah. Shimin (1:00:27) This way. Yeah. Dan (1:00:28) So OpenAI has been raising a new round because according to them, they need more time to work on quote unquote safety before Shimin (1:00:36) Mm-hmm. Dan (1:00:37) they go ahead and do their IPO. but for the first time ever, I think they are actually struggling to raise a little bit. So They were reportedly aiming for a 30 billion raise at a at a 1.4 trillion valuation. and then also apparently that bumped to 1.25 trillion. but the Financial Times said an article on Friday that said they've lowered their funding target to twenty billion. and they are actually expected to close lower than that even. So people might be getting tired of throwing money at these these things. It's weird. It's a weird thing to say out loud, but here we are, friends. Shimin (1:01:20) Yeah, or at least quote unquote the smart money has. So we're we are seeing a slowdown in the in the market, in the founding market. But and speaking of a slowdown in the market, my article this week is from Yahoo Finance, but it more or less is a summary of what just happened last week since we spoke, right? that yeah, AI stocks lost in total about five trillion dollars since October sixth. So it was Black October for more dangerous to cybersecurity industry. it was bad, it was a bloodbath last week. and we may be seeing some more persistent pullback. I think Inktel lost something like half its valuation since we last recorded this show. It it was crazy. Dan (1:02:03) It's insane 'cause they're not even really like a player in that space. It's just like, we're gonna Shimin (1:02:07) Yeah, exactly. Dan (1:02:09) plaster all of tech with the same very broad brush, but hey. Shimin (1:02:13) Yeah. So Nvidia lost about 834 billion. Of course, these are all market caps, right? They're they're not Dan (1:02:18) Mm-hmm. Shimin (1:02:18) actually money from the companies. Alphabet, eight hundred and thirty-four billion. AMD 236. TSMC, 505 billion. So like if the bubble is this is not quite bubble bursting, but I feel like this is definitely in a strong correction territory. Dan (1:02:36) Mm-hmm. Shimin (1:02:37) Again, not financial advice. so please don't trade based on this. Dan (1:02:39) Yeah. Shimin (1:02:40) But given all this, I I find it I find it hard for Anthropic to IPO into this market if this continues, right? And likewise I f I feel like OpenAI will have a difficult time raising additional funding when when a lot of its suppliers and and even competitors are are doing this badly. Dan (1:02:59) Yeah. And then I guess that begs the question if they can't continue to raise, how are they going to close the growth gap to the one trillion in promissory debt across the ecosystem? Shimin (1:03:09) Right. Right. 'Cause 'cause we saw we saw last week that so much of anthropics, annual loss is depending on their what we call it. not quite debt obligations, but but yeah, kinda like their debt obligations to to various other parties. So Dan (1:03:21) Yeah. Did did we I don't remember if we covered that in the article, if I've read it subsequently in the past week, but there was also something about that their bond score for like debt is junk. Yeah, I mean it's fair, but it was it was just kind of funny. But you but if for a company Shimin (1:03:33) Yep. Junk. E everybody's bump score is junk at this point. Yes. Oracle anthropic. Yeah. So Dan (1:03:42) that hasn't IPO'd yet, that's a little wild that like they could just go, and you're straight to junk. But I mean like junk is just a the technical terminology, right? It's actually C plus or something on the the credit grading. So there is worse. It sounds like the bottom, but Shimin (1:03:52) Yeah. Yeah. It's it's there yeah, there could be much worse. Dan (1:03:56) but yeah, it's not. Shimin (1:03:58) Right. all right. So given all that, how do we how do we feel about the clock this week? Dan (1:04:03) Think we're still creeping towards the eponymous midnight. Shimin (1:04:07) I'm with you. I'm with you. How big a move? Dan (1:04:10) I think I think the well that's the trick, right? Is how how do we How we clock this. Shimin (1:04:14) We we Dan (1:04:15) Um Shimin (1:04:15) also actually have to given given the recent events, we actually probably have to come up with a definition of what midnight actually means, right? We we say midnight means the bubbles bursting, but what does the bubble bursting actually look like? Dan (1:04:29) I think it's if we look at previous bubbles, it means the end of it doesn't mean the end of the technology, it means the end of like the hype cycle around it, right? So like people stop throwing insane money at things that there's no hope that we'll ever get back, but they still continue to make strategic and smart investments in things that will have a return, right? So I think that's at least that's my definition. Shimin (1:04:51) How much does the SP have to drop? I guess is my question. Is it like 30% from peak, 50% from peak? Dan (1:04:59) I I I I don't think it's about that. I think it's about like is does the American economy that's being propped up by this right now go into a recession? That that would be my definition, I think. Shimin (1:05:09) okay. You like the recession number. Okay. Okay. So if w you know what listeners, we're gonna workshop this between this and the next episode on Dan (1:05:20) Ha ha ha. Shimin (1:05:21) a better definition of the bubble bursting. In the meantime, yeah, exactly. Dan (1:05:23) We're missing our business guy, so none of us Shimin (1:05:28) I feel like I wanna move it forward maybe another twenty, thirty seconds. Dan (1:05:33) Yeah, I was thinking thirty. Um at the risk of going Shimin (1:05:35) Okay. All right. Dan (1:05:37) too fast towards it. 'Cause I think we're still creeping towards it. We're not there yet and I don't think we're probably gonna hit it until next year, as you said. Shimin (1:05:45) Right. We're we're moving fast, but we're we're moving pretty fast, but we're moving as fast as the Intel stock price is dropping. So I think it's only appropriate for us to move a little bit fast, right? 50 50 was wild. Dan (1:05:55) Intel stock price has been well, I don't know what their stock price, but Intel themselves have been hurting for a while. Shimin (1:06:00) Yeah. Take a look. Well, so we're gonna set it to three minutes, and with that comes the end of the show. thank you for joining us for our Conversation and study session this week if Dan (1:06:11) The past year. Shimin (1:06:12) you like the show. If you learned something new for a whole year, yeah, please share the show with a friend. You can also leave us on Apple Podcast or Spotify. It helps people to discover the show and we really appreciate it. If you have a segment idea, a question for us or a topic you want us to cover, shoot us an email at humans at adipot.ai. We'd love to hear from you. You can find the full show notes, transcripts, and everything else mentioned today at www.adipot.ai. Thank you again for listening, and we'll catch you next week. On to another year. Year year two of the show. Okay. Dan (1:06:39) Yes, year two. Shimin (1:06:42) Bye.