Latest / Artificial Developer Intelligence / AI Safety's Conflict of Interest, Gemini 4 Argon, the Brain Sandwich & Agent Trading

Transcript

AI Safety's Conflict of Interest, Gemini 4 Argon, the Brain Sandwich & Agent Trading · Artificial Developer Intelligence

• Shimin (00:00) Hello and welcome back to Artificial Developer Intelligence, a weekly conversation show where three software developers try to sort out the hype around AI from what it actually delivers. My name is Shimin Zhang, and with me today are my co hosts, Dan, he finally gets to talk about sex cult, Lasky. And Rahul, he drives an experimental model car for tax purposes. Yadav. Gents, how are we doing today? Rahul Yadav (00:26) Then I didn't know you wanted to talk about it so badly. You earned it as a middlename Dan (00:29) I I d I didn't either till just now. So, you know, first time for everything I guess. The things we do Shimin (00:35) Mm. First time. Dan (00:38) for podcast ratings. It's Yeah. Shimin (00:40) Fer first of many long chats about sex cults to come, I'm sure. Just k just kidding, Rahul Yadav (00:44) Yeah. Shimin (00:45) just kidding, listeners. This is this is gonna be definitely a one-off. all right, so on this week's show, as always, we're gonna start with the news where we're gonna be talking about the latest model, or perhaps rumors of the model from Google as well as Meta's AI data center, tax shenanigans. Dan (01:01) Then I know it's been a really long time, like a really long time, but we're actually bringing back Technique Corner, and Shimin is gonna be telling us about the brain sandwich, which is not something that Rahul eats on the weekends. Shimin (01:13) Maybe he should. followed by post processing, we're gonna have a variety of interesting articles to talk about from sex cults to what AI products should look like. Dan (01:22) and then we're gonna do a deep dive where we talk about what happens when agents trade for us. Shimin (01:28) And last but not least, as always, we're gonna be talking about two minutes to midnight, our f coverage of the financial side of the AI industry. Well, let's get started, gents. Dan (01:40) Woo. So first up, I just want to note this is not a model review, but Google has released Gemini 4 Argon, which they are claiming is the their well, their next era of frontier intelligence, but they're also claiming that it's pretty frontier. Shimin (01:57) Uh-huh. Dan (01:58) so we'll take a look at the the self-published benchmarks. Again, large grain of salt since none of us have been able to use it because they're doing their kind of n by now becoming standard thing where they only give it to select people for cyber security reasons, totally not marketing reasons. but it looks like it honestly it would be quicker to list the benchmarks that it didn't stomp everyone else on than the ones that Shimin (02:22) Mm-hmm. Dan (02:22) it did. but they they put it up against Astra, Fable five one and Opus five five and Particular in in SWE Bench and a bunch of other ones. It was like what a percentage or two ahead of everybody else. in like Vibecode Bench, it was significantly better. Like almost well, still almost two percent. Yeah. Shimin (02:43) Mm-hmm. Dan (02:43) and really only in a few of did it score worse and not much worse. So again, only a handful of percentage. So like OS World, I don't even know what that one is. Post train bench. terminal bench were were low and frontier SWE too it was not the winner astra's still the winner there but in pretty much everything else argon takes the lead Shimin (03:04) Yeah, it does not take the lead in a lot of the agenc coding benchmarks, which makes me think it may be a little more general purpose. I do find it fascinating that Gemini for Argon scored a nineteen point six percent on Harvey's legal agent benchmark, whereas the rest of the Frontier models only scored in the single digits. So this may be a Just a much more general purpose model. and there are rumors out there that it is a multi modal first version of of of the model. so we'll see how things Dan (03:37) I mean that's Shimin (03:37) shake out. Dan (03:38) kinda been Google's strategy, it seems like, all along, right? Or at least recently. It's like Gemini's never really been the true leader on the software side, but like it's pretty darn good at almost everything. So it's Rahul Yadav (03:51) Yeah. Dan (03:52) nice to nice to have if you're only Shimin (03:53) Yeah. Dan (03:54) buying one. Which you know, if they're if they're really targeting consumers is not a Rahul Yadav (03:58) And before they had Chat GPT to compete with, but they were very focused on coding mostly as well. So it's mostly Gemini at this point, think, in consumer space. Dan (04:10) General purpose. Rahul Yadav (04:11) No. Shimin (04:11) Yeah, and the the other thing that maybe signals we're entering a price competition error of models the rate for Argon is two dollars per million input and ten Dan (04:22) Mm-hmm. Shimin (04:22) dollars per million output compared to Astras ten and fifty, which is the same as Fable Five One. So it is a s like one fifth the price. Whether this is due to TPU cost savings or something else. Dan (04:34) Yeah, I was gonna say that wouldn't surprise me. We've talked about that in the past, where of at least in the current iteration of of like inference, Google's very well positioned to to win cost wars, I think, at least against the frontier companies. But we'll see how long that proves to be true with, you know, jalapeno and its ilk nipping at their heels. But it's gonna be a while before that can be manufactured at scale, like even with a fast design cycle, you know. So I mean, that's what's holding NVIDIA back right now, right? Is they like physically can't make enough wafers to shove them into all these data Rahul Yadav (05:06) Yeah. Dan (05:07) centers. So Shimin (05:07) Yeah. they could be an even bigger Rahul Yadav (05:08) This is more from, Shimin (05:10) company. Rahul Yadav (05:10) yeah, more from personal experience. used Gemini and Anti-Gravity and Gemini has this like Spark, which is their version of Claude Cowork or whatever Chat GPT calls theirs. And it's pretty like, I never run into the limits even if I give it crazy goals and It's on a $20 a month account. so it's already very cost effective to use those versus I know I would easily hit my Claude limits if I were to give the same tasks to Claude So some of it is also, even if the model gets better, the complexity of the task, unless you're doing like fundamental, some science R &D stuff, we're going to mostly not really need cutting edge models. And so The cheaper model does the job, rarely hits the limit. And then the limit refreshes like every five hours or something. So they've been pretty generous with that. Dan (06:07) I can't wait till I can Shimin (06:07) Yeah. Dan (06:08) run it on my own hardware and then I'll be really cheap. Rahul Yadav (06:10) Yeah. Shimin (06:12) Gemini four will be for only Ultra and Enterprise API plans only, at least to start with. So it might be a little Rahul Yadav (06:19) for now. Shimin (06:20) bit before us twenty dollars a month. Peasants can get our hands on it. But for Dan (06:23) Yeah. Shimin (06:25) for model review, I'll I'll probably spend a couple of dollars and Rahul Yadav (06:26) Given how much I've, Shimin (06:27) do an API. Rahul Yadav (06:28) yeah, but I've advertised Gemini Enterprise, a genetic platform so much. I was hoping I would get an argon, you know. Dan (06:31) Yeah, why aren't we getting this for free? Are you telling us we're not actually sponsored by them, Google by Google after all Shimin (06:40) but I think the moral story is to Yeah. Dan (06:41) We're not, full disclosure. It was yeah. Shimin (06:44) the moral of the story is to never count Google out. you know, they did invent the transformer architecture and they may have had a couple of stumbles in the last couple of months, but they still have top tier talent and and top tier compute. So yeah, we will do a proper model review when Gemini 4 argon comes out. next up, Rahul, you have an article for us. From the New York Times. Rahul Yadav (07:06) From New York Times, how Meta uses AI data centers to avoid billions in federal taxes. So during the Reagan era to compete with Japanese companies on technology, they created this federal tax credit where if you do R &D, any expenditure you have on that, you can claim that as a tax credit. And we've seen this in software industry where, you know, yeah, people spend money Dan (07:30) Yeah, it was the whole zerp thing. Shimin (07:32) Mm-hmm. Rahul Yadav (07:33) on software and R &D related stuff. They can claim that in their taxes as credit. Meta is taking that to the next level where they're building out the data centers and they're claiming the data centers build out and all the money that's going into it to claim tax credits on them. And they've been able to cut almost like 4 billion in taxes by claiming this tax credit. lawyers internally are a little bit on the fence over whether this is going to fly long-term. one thing I found pretty funny was someone at, I think this was some accounting firm that where they were like, yeah. Mr. Shevchuk, the tax advisor at BPM proposed one way it might work. If you had a data center that you're building out to cure cancer. So all Meta has to do is cure cancer and then this tax credit would be justified. I think that's a pretty good deal. Dan (08:31) Y you know, if if they cure cancer, I might take back some of the things I've said about them, so Shimin (08:39) Also, what is going on with Meta that they are so leaky? I wonder why. Dan (08:44) Isn't like a third of the staff still making puzzles or something? Rahul Yadav (08:48) Yeah. Shimin (08:48) I think they would rewound some of that. Dan (08:51) Really? Shimin (08:51) Yeah. Rahul Yadav (08:52) The quip I heard about Muse was that if there was anything weird that was planned for it, would have leaked out already. So everything you'd know and hear about it Muse is all they know inside. Shimin (09:05) That's pretty good. Rahul Yadav (09:08) so yeah, the other companies, other hyperscalers haven't been doing this. but now that meta is doing it, that's the strategy that's getting pitched to Shimin (09:21) Mm-hmm. Rahul Yadav (09:21) them as well to claim this and data centers are going to keep getting built out for the foreseeable future. So, we'll see which way this goes. Dan (09:29) I would argue that that is the actual intent behind that tax credit is this kind of build out versus like what software companies tend to use it for is like just the development of normal software. They call Rahul Yadav (09:41) Yeah. Dan (09:42) it an R and D expense. It's like mm, okay. Shimin (09:46) Dan, that that's the key question that I think this whole thing hinges on is like, what is AI? Is AI a mature piece of technology like those R and D write offs that we've had during the twenty tens, the zip era? Or is AI still Dan (09:58) It's back though now. They they the as part of the whatever it was, Big Beautiful Bill or something, they changed the amortization schedule again. It's not it never went away, but they they meant they caused it so that you had to amortize over three years. That cost you couldn't just directly expense it every year. yeah. And now it's back to you can every year, I think. So mm-hmm. Shimin (10:20) is it back to every year? I didn't know that. but the the point stands is like is AI or you know, data centers used to train AI models like a maturing enough technology that it should no longer be considered RD, right? that's Rahul Yadav (10:33) huh. Shimin (10:33) the heart of it. I ca I could see the argument that if you create a data center and the data center is only used to train Dan (10:42) Yeah. Shimin (10:42) the model, never for inference that it could be considered R and D expense Dan (10:46) And that's s also sort of like what we looked at last week. We're like, is the GPU a depreciating asset or not? Right. It's kind of like 'cause R and D is the opposite of a depreciating asset. I would I don't know, not a tax lawyer, but like that's what it feels like to me. Rahul Yadav (11:02) yeah, people break, you know, people do break down COGS versus OPEX because that does influence the, how Dan (11:13) Yeah, tax. Rahul Yadav (11:13) you report your finances. And so it could be that within a data center in the future, people start going X percent of the load was to serve it. And then, you know, a hundred minus X percent of the load. was to actually train and yeah, Dan (11:28) I mean, yeah. Rahul Yadav (11:29) one of them that you can claim the credits for and the other one you cannot. So that might end up happening given each data center is gonna be generating billions and billions in money in the long term. Dan (11:44) We hope. Shimin (11:45) We hope yeah. The US economy depends on it, sir. Rahul Yadav (11:45) Please, at least a few billion. Dan (11:49) Yeah, please. Yeah. Shimin (11:54) All right, let's Dan (11:54) Feel like we've sorta jumped again on two minutes here, but you know. Rahul Yadav (11:57) Hehehehehe Shimin (11:59) yeah, let's go into Technique Corner where this week I have an article titled The Brain Sandwich from the blog Terrible Software by Matheus Lima. And we talk a lot on this show about you know using AI to augment yourself as opposed to use it as a replacement for your own thinking. And I think when communicating this to others, a useful metaphor that Matheus wrote about from one of his directs Emily is this idea of a brain sandwich. So before you do anything with AI, you first use your brain and then you apply AI and then finally use your brain again. Rahul Yadav (12:37) You already lost us on that one. Shimin (12:40) We our Dan (12:40) Wait, use your Shimin (12:41) brains have our atrophies. Use what? This this thing right here, that once upon a time is is important. Dan (12:48) Wait, you're assuming that I still like even come to record the podcast, right? This is actually just a proxy that is responding in real time. Shimin (12:57) A yes guys. Before you cover an article. Rahul Yadav (12:58) Dan hasn't aged a day since the first day of this podcast, and now I know why. Shimin (13:03) Yeah, so why do I think this metaphor i is nice? It it kind of folds into a lot of stuff that that are pet peeves of ours on this show. Like for example, if you use your brain after the task, you will not be sending slop grenades to your coworkers. You will not be sending forty thousand line long MRs and PRs to your coworkers without human annotation. or without reading it first. and of course I think the process like the ideal process is probably more like a lasagna than than just than just a sandwich. Like you probably layer in your thinking throughout the your work with AI, you know? Layers of human thinking, layers Rahul Yadav (13:42) Yeah. Shimin (13:42) of AI meat. And then you wrap the whole thing in your harness which is a tomato sauce. But the important part is to to still use your brain. And and too often I think we're starting to see that folks folks are not doing that. so a a simple and elegant metaphor that I think we should you know tell everybody we know. AI is not magic. You still have to use your brain. And of course, this also means that your human comprehension is still important, right? Like both before and after the task. and that's probably something that we're gonna go back to. That's called a Chekhov's gun in the business here. So that's Rahul Yadav (14:18) Yeah. Shimin (14:19) my technique technique corner for the week. Rahul Yadav (14:21) I read this, there was an article by Colin Breck and the title summarized this whole feeling in the best way possible. The title is, don't want to read what you didn't write. And I'm like, yes, exactly. Shimin (14:36) Right. Yep. Yep. Rahul Yadav (14:40) Couldn't have put it in better words. So that's kind of how this whole thing feels. Shimin (14:42) Yeah, and and similarly i if if you didn't write the code for if you're working as a software developer, you didn't write the code because AI did it and you didn't bother to understand it, then like w what is the point? What is the point of you sitting? That's it's just your professional duty to some extent. Rahul Yadav (15:00) And it is the AI being used to write the code and read or, you know, review the code is becoming more and more like hand in hand thing now. So it almost feels like this symmetrical thing where if AI wrote it, it's better if AI reads it. And if a human wrote it, there's usually context and it's better. if a human reads it at the end almost like other than if you're inputting into an AI. Dan (15:28) I don't buy the if like AI code review and AI wrote it. Rahul Yadav (15:34) What do you mean? Dan (15:35) Especially if it's the same model. I just don't think it adds like as much value as people think it does. Rahul Yadav (15:41) like using the same Shimin (15:41) Well that's why Rahul Yadav (15:42) model to write the code and review the code. Dan (15:44) Yeah, and then you review it the same mod or like model by the same provider, you know, especially. Like people use like Opus to write it and then they'll review it with Opus. And I'm like, it might catch some stuff, Rahul Yadav (15:54) fully agree. Dan (15:54) but like I've actually personally had much better results when I use like Codecs to review it running, Rahul Yadav (15:59) Yeah. Dan (16:00) you know, whatever Astra or whatever I can afford there. And then Shimin (16:04) Yeah, and and if if the model drifts cause both the coding model and the review model share the same biases and make the same incorrect assumption. If you didn't use your brain first, you wouldn't be able to catch it. So that's that's why all three layers have to be there, guys. Even if you use AI as as your first pass of code review. Rahul Yadav (16:23) Yeah. Then I agree with not using the same model to Dan (16:27) Yeah. Rahul Yadav (16:28) both write and read. It's yeah, it's calling you. It's kind of like reviewing your own code and be like, it looks pretty good to me. Dan (16:34) I know, this is great. It's the best thing I've ever written. Shimin (16:35) Looks good to me. Dan (16:37) If if only what we actually need is the time machine reviewer, right? Where like you could write your code and then somehow fast forward yourself two years into the future and then immediately read it. Cause you're like, Rahul Yadav (16:49) Yeah. Dan (16:49) God, who wrote this? It's terrible. Rahul Yadav (16:52) Hehehehehe Shimin (16:54) Well that's the reader with a fresh pair of eyes prompt that we talked about a couple of months back. Yeah. Yeah. Dan (16:57) Yeah. I've I've used that one many times ever since you introduced it. But or just dump your con I usually actually when I do have to use the same model, it's always fresh context. Like, hey, someone else wrote this, Rahul Yadav (17:08) Yeah. Dan (17:09) take a look, you know, just to try to make it as Shimin (17:11) Yeah. And g give your model the context so it knows your preferences is also a ongoing research area, Rahul Yadav (17:19) It would be a hilarious experiment to, you know how when you post a PR, write something in Claude or something, it's GitHub at the bottom. does say written using cloud or something like that. And then we should see if you have cloud review and it goes, it's me. So it must be all, and Dan (17:40) No. Rahul Yadav (17:42) then on the other hand, other people look at it and be like, Claude sucks here. Here's all the things that are wrong with this. And then you can just take maybe, you know, like, Gemini's output, but say written using Claude and then see if Dan (17:56) Yeah. Rahul Yadav (17:57) it would influence Claude's take on it. Dan (17:59) yeah, that would be interesting. Shimin (18:00) I'm waiting for that r Dan (18:01) That Shimin (18:02) vibant tail Rahul Rahul Yadav (18:03) Dammit, I have to do this work now. Dan (18:05) I feel like you're the one that always Shimin (18:05) Yeah, you have to do it now. Now that you've spoken it. Dan (18:06) does the cool experiments, so Shimin (18:10) Alright, on to a completely different topic, but something that Dan is very excited about. let's go to our first post processing for the week. Rahul Yadav (18:18) You Dan (18:19) Alright, so here's our our headline that I'm sure is gonna make the the podcast headlines too, which is I'm sure the entire reason why our our glorious Producer has chosen this article for us in the first place. and here it is. Without further ado, AI safety is mostly a sex cult. There we go. We said it. okay, so that's Shimin (18:39) Uh-huh. Dan (18:39) a very intentionally provocative headline. the content we're gonna talk about here is by SE Gyges who's I believe a res AI researcher. focused sort of in the medical space. I don't know if that's someone's actual name or not. This came across this on Hacker News. Pretty wild discussion about that on there. but the actual post is really just a roll up of like a giant blue sky thread where he or she makes the argument that It's little bit of an ad ad hominem, like kind of straw man argument, right? Is like, you know, all these people are like out there with like a huge like OnlyFans person that's like really well known or something, and like, so therefore the entire thing must be sex called. But I will say the Rahul Yadav (19:24) Mm. Dan (19:24) way the argument is structured makes a lot of sense. So I'm gonna kind of walk through what they posted. I, you know, did not go out of my way to verify any of these facts or not. So this is really just me restating it. I could have. I didn't. Shimin (19:34) You could have you could have gotten an OnlyFan account and written it off as podcast expense. Dan (19:40) Really deep research. Yeah, okay. Rahul Yadav (19:39) Ha R &D expense. Dan (19:43) Maybe they'll sponsor us next instead of Google. Shimin (19:45) Write it off, yeah. Dan (19:46) yeah, so so here's what they what they said. So I'm just gonna restate this more or less verbatim, just to kind of get the the point across. So he starts off by setting up Yud, who's like you know, Eliezer Yudkowsky is is kind of like really the main the main thinker behind even the terms of like AI alignment, right? Like he was kind of Shimin (20:06) Mm-hmm. Dan (20:06) the first to to Bring it up. he's also very well known apparently for his Harry Potter fan fiction that like explains some of the ideals behind this and like some of the tenets of rationalism too. Shimin (20:19) He he is he is one of the founders of the less wrong forums. So yeah. Dan (20:23) Yeah. Yeah, which is like the rationalist, yeah, community. yeah, so I don't think we're need to go into great deal of detail about rationalism. in my opinion, like just being a c sort of casual reader about this stuff, I'm already fairly convinced that rationalism itself is not a cult, but it is a phenomenally great cult incubator. see also the Zizzians going around murdering people and Shimin (20:48) Mm-hmm, mm-hmm, mm-hmm, Dan (20:48) All this kind of stuff. So I don't think we need to go deep into detail there. rationalists themselves claim it's not a religion. I felt that the whole section of this article where he goes on arguing that it is a religion was was kind of silly because it doesn't prove or disprove the point. so Yud and this other guy, Nathan, are part of the sort of core group of rationalists. those are aligned with the What is it, effective altruist movement as well? So there's a lot of like intertwined between those things, including both funding and everything else. I d we can go into all kinds of detail there. I'm not gonna. that's sort of the sex cult angle, right? Is they're basically implying that Shimin (21:22) Mm-hmm. Dan (21:23) they all live in a in one or more communes in Berkeley and are all just like sleeping with each other constantly. And I'm not frankly not too interested in like, you know, kink shaming anybody over this kind of stuff because I really don't think it's super relevant to like what we're doing on the podcast. but the in Shimin (21:37) Right. We'll we'll leave the king shaming to the ADI pod after dark, yes. Dan (21:42) Yeah. So but the thing that's that's interesting about it is that it establishes that they live in these communes, right? And in one case he draws that the the big one where I guess like some of the main folks live is called I forg I should have put it in my notes and I didn't it's like afterlight or something like that is the name of it. It used to Shimin (22:00) Mm-hmm. Outro light. Dan (22:01) like yeah, it used to be like a hotel, like a holiday inn or something. Not a holiday in, but Shimin (22:06) Rolls Garden Hotel, something Dan (22:06) And yeah, And the point of that that I think is is interesting is these are the only folks that are really advocating for safety, right? Most of them work at all these big frontier labs and are on the safety teams and are doing this. And the thing that bothers me and I think the author about it is that these are the folks that are also advocating for public policy. So the the sort of like strange things that have happened in the past few months and weeks around this, like the, you know, people agreeing to pace the frontier. We talked about that a bit. And like claiming that they would like waive antitrust law to be able to pace the frontier is kind of a strange thing to claim to. so kind of brings that into question. But the part that's really gets me is that if everybody that's in charge of safety is a hundred percent convinced in doomerslash paperclipping stuff being the reality, then Are we taking an objective look at safety? And are if we're so convinced of dooming, are we maybe missing symptoms of it actually happening because like they have an agenda to essentially fund their current lifestyle and everything else by saying that this is definitely gonna happen, right? So my point is they're not taking an objective look at at it. And if anyone I would want to be well positioned to sort of, you know, we're on the on the Podcasts in the past we've sort of looked at this like lens of like, okay, is you know, open AI's agents going rogue? Is that a marketing stunt or is it real? And it's probably somewhere in between, right? You know, like is the real answer. But Shimin (23:39) Mm-hmm. A little bit of A, a little bit of B, yeah. Dan (23:42) these are the people that I want to tell me that. And they're not telling me that because they're so focused on paperclip crap and like paste the frontier. That's where I'm getting at. Shimin (23:50) well I I think I think one part of the the argument that is quite strong is Yud was involved in introducing a lot of the rationalist EA folks to each other and he was he organized the meeting where Musk initially met Sam Altman and gave him that initial which then caused Musk to Gave Sam Altman the initial round of founding for OpenAI, which led to the lawsuit. So, like, we're finally getting to the beginning of all a lot of dramas that we covered so far. Dan (24:20) Yeah. Shimin (24:21) And of course, Dario from Anthropic is a yep, card-carrying EA person. Dan (24:25) A self avowed EAC, yeah. So yeah. Shimin (24:30) so there is a lot of commingling of these kind of thinkers, these Bay Area rationalist adjacent folks who are also. are the ones who are building these models. So if you've been proclaiming about P Doom and Paperclip Doom for like a decade, but then you also introduce the founders of the labs who then now produce these frontier labs, there's a little bit of did you cause this thing that happened angle to to all this? Dan (24:59) Yeah. Shimin (25:01) a lot of moral gray areas. I don't know. Rahul, what is your relationship with the the rationalist movement and effective altruism? And are you a card Dan (25:10) Yeah. Shimin (25:10) carrying member of the Slate Star Codex blog slash astral codex ten? Rahul Yadav (25:15) Nothing, do not care, and no. I think those are the answers to your. Shimin (25:18) Not at all. Have you have you come across them in the have you come across the rationalist slash effective altruists in your journey? Rahul Yadav (25:26) Sometimes people on the street try to sell me something but I don't stop for long enough to hear them out so I can't fully say if I have or haven't. Shimin (25:34) Okay. Rahul Yadav (25:35) Yeah. Shimin (25:36) Well, gents, I've lived in the Bay Area for a year and a half or two years. I I don't quite remember. so Rahul Yadav (25:41) I didn't know this, Shimin Shimin (25:43) I have crossed my path with the rationalist and effective altruist movement. not a card carrying member, although I feel like very few like even people who are more or less agreeing with a lot of the effective altruist principles. these days would not say they are a a member of EA. I drifted away from the EA movement when I realized you know, a lot of the eighty thousand hours stuff was about like, hey, you can kind of pursue whatever career that may or may not be morally apprehensive if you then donate X amount of your income to charitable causes. Like if it felt a little bit like moral sanewashing. To some extent. That's Dan (26:26) Yeah. Shimin (26:27) when I was like, that's a little weird. And then when the EA movement and the rationalists started focusing a lot about P Doom and you know the welfare of future generation of humans instead of the current generation, that was like one of those things I thought was a bridge too far. and it felt like, hey, like we have real problems today. why are you focusing on this thing that may or may not be an issue decades from now. Right. So that's kind of where I diverge with the movement. but I still get a EA newsletter. So I never unsubscribe from that. so I was here fully ready to defend the movement on on this show today. But Dan, I feel like you had a very even handed take on the whole thread. Dan (27:09) I mean, like, I'm not gonna say it's not sensational, but I think that like you know, the part that that bugs me the the most is that We need that. I mean, like, I guess I'm a I won't say I'm a rationalist, I'm a realist about this kind of stuff, right? Like there is a chance that like something like paper clip doom could happen, right? I don't know what that chance is. I'm not a like researcher. I'm not in any way prepared to be able to handle that. I want the absolute and you know, like most things there, I want the smartest person in the room thinking about it. And if that smartest person in the room is thinking about it but is compromised in some way, that bugs me a little bit because like, you know, they're potentially you know, their entire livelihood inges on it. So it's like Yeah. It's it's it's Shimin (27:53) Yeah, there's definitely a conflict of interest there. Dan (27:55) kinda like the the ad tech folks where like their whole thing is like we keep the internet free and it's like, well yes, but at what cost? Yeah no Shimin (28:06) And of course we're not here to to actually kink shame anyone. And I I as far as I can tell, the polyamory is all consensual in in the Berkeley area. I've have not been invited to any any parties like that. So I don't have any first hand experience with that. No. Well I no no EA Dan (28:20) Mm. That was gonna be my next question when you when you said you were involved. Shimin (28:26) person that I came across back in the mid twenty tens was was in a sex cult or was polyamorous as far as I can tell. So Dan (28:34) Much to my dis disappointment, I've never been in one either. So, you know, what can I say? At least Rahul will have to all live Shimin (28:39) So so we are not here to kink shame yeah. Dan (28:42) vicariously through him. Shimin (28:44) my favorite quote of this whole blue sky thread, when I f initially read it, was something along the lines of like everybody's just being contrarian together. And I thought that was that was really good. Dan (28:53) Yeah, that was pretty funny. Shimin (28:57) but sometimes I feel like you know, I'm gonna draw a parallel between I think Anthropic split from open AI largely due to the differences on open AI's governance and AI safety views, right? And it is also interesting that Anthropic is kind of the only lab, you know, with a f CEO who was largely EA aligned who did not kind of just bend the knee to the US government immediately. Like you can just look up Lake America on Google Maps to verify that fact. so Dan (29:29) Make America. Rahul Yadav (29:30) I'm Shimin (29:30) Yeah. Rahul Yadav (29:31) not saying the right or wrong of it. I think they have to do the lake America by law. So it's part of some law that they have to. Shimin (29:42) And of course we can bring up the whole Department of Defense Rahul Yadav (29:45) That, Shimin (29:46) thing. Rahul Yadav (29:46) that, yeah. Shimin (29:47) Yeah. S Rahul Yadav (29:49) Mad respect to Dario Shimin (29:50) so yeah. So so sometimes like maybe have values that you strongly believe in is is a valuable thing if those are beliefs that that aligns Dan (29:58) Mm-hmm. for sure. Yeah. Shimin (30:00) with that of the humanities. Alright, so Dan (30:03) Yeah. Not saying that at all, just saying I wish there were more reliable police is kind of my takeaway. AI police I should say. Shimin (30:10) Yeah, yeah, agree. Agreed. Rahul Yadav (30:10) Yeah. Shimin (30:13) Agreed. okay, so do we have anything else on this? Dan (30:17) That was sex cults. Moving on. Just had to say it one more time. Shimin (30:19) Alright, that that's all we have for our post sex cult processing today. Rahul Yadav (30:25) Then that sounds like the closer for the podcast. Shimin (30:29) Let's move on to my post processing article for the week. this is a post titled The Age of Wonders and Terrors from the blog of Scott Aronson. Scott Aronson is a theoretical computer scientist. He is a chair of computer science at UT Austin. He specializes in quantum computing and complexity theory. one of his blog posts on this exact blog was one of the like you just need to read these twenty posts to really understand AI kind of papers list. So this is a serious person, not a quack, a serious academic. And I wanted to talk about this post because when I came back from the vacation I I got sick and when I was in my deep kind of pessimistic version of myself in my illness. This blog post summed up everything about how I feel about the state of AI. Okay. So he starts with this hypothetical scenario that if 20 years ago if we are fans of science fiction and we will say stuff like look recursively self Improving super intelligence just exploded from some hacker basement and took over the world without warning is highly implausible. If it's gonna happen, we'll see many warning signs first. First, I don't know, AI agents will break out of containment, conspiring with each other to hack websites in fanatical pursuit of whatever strange goals they have. And then, of course, we'll see major math problems getting solved by AIs. Even the clay millennium problems, that would be the time to panic. Wake me up when that happens. well. That is the world we live in today. And it is some weird like Black Mirror world. Rahul Yadav (32:04) Yeah. Shimin (32:05) and and then Scott makes the argument that, like, hey, if you are honest with yourself, you go back to 2006 and you tell yourself this is what the future will be like, you would be panicking and you will say this is the start of the singularity. Right? It's just imagine yourself in 2006, the younger more handsome fitter version of ourselves Dan (32:26) Yeah. Shimin (32:28) may or may not be in a sex cult I think I would a hundred percent agree that we are in the beginning of singularity. And so given that our priors have changed and AI has indeed done all these things without moving the goalpost, you should be panicking right now. And that basically is how I feel when I'm pessimistic and P Doom minded. So Dan (32:49) If only there was someone to tell us what they thought the panic was. just said. Shimin (32:54) Right. So and then you know we can we can debate about the merits of the Frontier Labs and and the personalities and the trials and all that stuff. But like the warning signs are here. The the the and and quote here, they couldn't be here more clearly if the sky had turned reddish orange like the Matrix movies. And I really I thought that was a beautiful image. Well, that and climate change. and the fires. Rahul Yadav (33:14) I think they might be, actually. Shimin (33:19) And then Scott drew this next analogy, which is he is of Jewish descent and he's a skeptical scientist. And folks will ask him about him living in Texas and be like, hey, how do you deal with the evangelicals, Christians down there? And Scott would say, like, Well, if if I only have to accept Christ after he returns, then that's a great deal. How can I possibly object to that? Right. And the analogy here is of course like, yes, the AI Doomers sound like, quote, an apocalyptical religion, maybe a sex cult, and that the rationalists do seem like a Bay Area cult, yes, and yud gives off the vibes of a messianic prophet, yes, yes, and yes. but we're no longer being asked to believe in arguments and extrapolations, but only in the front page news. And this I feel like ties in nicely. With our previous post. Yes, Yud has potentially questionable personal behaviors. Yes, he wrote some stuff that could be considered fanfictions, but like I just said, I ignored the movement when they focused on superintelligence, like I don't know, 10, 15 years ago, and now the whole thing is smacking me in the face. And I have to adjust my priors. And now we have to be like, well. The the world being what it is, and Yud was right about the greatest challenge facing civilization within our lifetime was was gonna be AI. And you and I were wrong about it twenty years ago, we're probably wrong about it two years ago. Why was I wrong is a is something that I spent a lot of time thinking about. but I've updated my my priors since the front page news came out and I'm no longer moving the goalpost. Dan (34:56) It's just a it's just Shimin (34:56) Sorry, that was a lot rant Dan (34:57) like a no, it's just the f the frog in one Rahul Yadav (35:01) And this is kind of like the survivorship bias maybe is the right way to look at it. It could have taken longer and then Yed would be wrong about it, right? Everybody makes the, this is the biggest thing we should worry about. And then by its nature, some people are gonna be right and some people are gonna be wrong. He just happened to be right. in this timeline and while there's still, you you can go and tell people that you were right about some things, but there's plenty of stories all over the place about people making pronouncements that didn't come true also. So we're not going to talk about those. Shimin (35:41) Yeah, and and just 'cause he was right about AI safety being a big issue doesn't mean that, you know, the book he wrote, if someone builds it, everybody dies will necessarily come true. but Rahul Yadav (35:50) Yeah. Yeah. Shimin (35:51) the probability of the book coming true is certainly now higher than it was two years ago, given the circumstances. Rahul Yadav (35:59) Yeah. Shimin (36:00) Yeah. Well, so the the post also it's it's not all doom and gloom, although there's a lot of it. And it it did speak to me somewhat. and I wanna quote here about what we should do about the fact that you know AI is the way it is. quote, yes, there is still enormous uncertainty about what the rest of our lives will look like, but as far as I can tell There's no longer any real uncertainty that it will all mostly revolve our own AI and the extent which we s succeed or fail at directing its power towards human flourishing. And that part I think is is completely true. this probably is gonna be the defining question/slash problem of our lifetime, at least for the next foreseeable future. Unless P Doom happens. We went from like a year of no P Doom talk to to to a lot of it today. and then to bring things back to programming. Scott also struggles with this idea of what does math mean in a world where so many people are submitting papers with AI assisted proofs, right? Does human comprehension still matter in the age of AI math? and one of the things he called out was that in these paper submissions, they have statements that say, quote, our main result came entirely from GPT-6, but we understood it and take responsibility for it. that is the bottom layer of the sandwich, right? Even if you use the AI, you still have to take responsibility for it and you still have to understand. In other words, you still have to use your brain for it. 'Cause otherwise, what are we here for? And how do we ensure that the the code, the math is correct? Dan (37:40) Yeah, there was a post on Hacker News this week where it was it was pretty well written. It was basically like my I don't know, great grandfather or something was a a farrier, right? So he did Shimin (37:52) Mm-hmm. Dan (37:53) horse shoeing. And then a car, like his neighbor bought a car or something and drove it by him and he was like, What is that? He was like viscerally challenged by it. And so then he was like okay, well, I'm just gonna learn to be a mechanic. And then now his grandfather, his father, and like I guess he's a software engineer, but like his the whole rest of his family was was all mechanics and like really, you know, well known, good at it, whatever. And so they're kind of like, time's changing. But I think the one thing that that story fails to it's a good story, but like what it fails to account for is that you know. Fer being a farrier is skill. Being a mechanic is a skill. Using AI for some things is you could argue is maybe a skill, right? Like using it. I mean, in fact it is. Like I think I'll just like go out on a limb and say that. But the it none of those other things have ever provided the opportunity for like advocating thought that AI does. Shimin (38:48) Mm-hmm. Yeah. Yeah. Dan (38:50) You don't have to do that, but it just the opportunity didn't even exist because like to be a mechanic you have to be thinking a lot, right? You're solving like really complex three dimensional problems and you need to have at least a basic understanding of physics, like Shimin (39:03) But you you still have to understand. Like that's the difference. I I actually disagree with you that using AI is necessarily a a skill because if you read the a lot of the chats where AI solved these fundamental math proofs and theorems, it was basically try hard and make a breakthrough and don't give up. Yeah, Dan (39:24) You can do it, I believe in you. Shimin (39:26) yeah, what was a lot of that? Yeah. So Rahul Yadav (39:29) So we're going to need football coaches or something as our Shimin (39:34) Maybe. Rahul Yadav (39:35) handlers of AI in the future, giving pep talk every morning. Dan (39:38) You can do it. Break the next theorem. Shimin (39:39) And and and you we're starting to hear a lot of software developers talking about like setting up agents and just give them a go of make this code go faster and don't stop until you do. And that's not that different from just make a breakthrough, right? Rahul Yadav (39:53) Yeah, there is, to add to what Dan said, I think thinking and doing do go hand in hand to a certain extent, right? There's only so much of a model of the world you can create in your head and then you have to do it because there's a lot of details that you miss otherwise. But if AI takes all of... Dan (40:11) Cosmologists would beg to differ, but Rahul Yadav (40:14) But if AI takes all of that, that is fine too. I think there's going to be other newer jobs where we would have thinking and doing, going hand in hand that we can go after. Or we get the singularity, I can do everything, and then we just have to talk to each other and get along with each other. And who could imagine a world like that? Shimin (40:34) getting along with each other. That's I can't I can barely get along with you two on the show. Rahul Yadav (40:36) Yeah. Yeah. Dan (40:38) I mean, Rahul Yadav (40:39) P-Doom is crazy on that one. Dan (40:40) look, I I said at the at when we started our like, you know, pre recording session that I didn't have a vibe and tell, but I forgot that I did in fact do do a doing and thinking with Claude in the past two weeks, which is we made these Halloween decorations, that are like kind Rahul Yadav (40:58) yeah. Dan (40:59) of ghouly guys, Shimin (40:59) Yes. Dan (41:00) and I Using Claude's help figured out which microcontrollers to buy, Claude wrote all the code for it, and then I did all the wiring and flashing and everything else of it and made these radar controlled eyes that follow Shimin (41:13) Mm-hmm. Dan (41:13) you as you walk up to my host using like a millimeter wave radar. So it was a fun project, but it was like it was truly a collaboration because it was my idea in the first place because I knew about the millimeter wave wave radar from home assistant stuff. And then but I knew what I wanted and just explained it. And then I don't really care about the code, right? For that stuff. Like I don't even want to look at it. Like, cool, Shimin (41:32) Right. Dan (41:35) it works. It goes right to left. Like, you know. although I will say I did get very stuck at one point on the eyes really behaving rather strangely and I was trying to figure it out. And then I went and read the data sheet for the radar, which is something I hadn't done yet, and realized that I had it oriented ninety degrees off and Shimin (41:54) huh. Rahul Yadav (41:55) Mmm. Shimin (41:55) Other way. Yeah. Dan (41:56) Nobody had caught that. and then as soon as I rotated it, it worked great. Shimin (42:01) It it sounds like you used your brain to tell Claude what you wanted to build and then Claude did the coding. Dan (42:05) Yeah, it was definitely a brain sandwich or a lasagna really. Yeah. But I mean, but the only reason I brought it up is was because it was very much a thinking and doing, right? Like Claude actually did a lot of the thinking. I mean, I know how to write like an arctangent, which is essentially all the math you need to do to to figure that out because the radar firmware does all the hard work anyway. So it's not like I couldn't have done it. But I really wanted to focus on like the physical execution of it 'cause that was gonna take the most time. So I did. And Shimin (42:35) Yeah. And and brain sandwiches very spooky, just right in time for the Halloween season. That's why you should head out hand out at your Halloween party. Dan (42:42) It's true. I'll put a little sign above it. This is brain sandwich. No one will get it unless they're fan of the podcast. Shimin (42:50) All right, shall we move on to our last post processing post of the week? This one from our whole. Rahul Yadav (42:55) Yes. So this is by Glyph, who is the creator of Twisted Python. don't know if either of you ever used that. was a fun time. I used to use it in one of my past jobs. And it's Shimin (43:10) Mm-hmm. Rahul Yadav (43:14) basically an async engine framework that, yeah, this existed before I think Python had support for async stuff, Dan (43:23) Async IO or whatever, yeah. Rahul Yadav (43:25) yeah. So going back to the post here, The post says, what would a serious AI product look like? And it's a number of things that Glyph is kind of unhappy about. So we'll go through a few of them one by one. First one being make checking for mistakes a first-class feature, which did quite hit the nail on the head. Every time at the bottom I say, it can make mistakes, check the work. And then I'm like, but I'm paying you for it. And if I get a... do all this, what's the point of it? And so that, and that's one of the biggest glaring problems that we have is on the one hand you have, you know, AI will do all these things, take away jobs, P-Doom, all these things and rule the world. On the other hand, at the bottom of the chart bot, says it can make mistakes, check its work. And how do you make the tool exist? with each other. The solution he proposed there is to have a tool that would help do the checking where you can explicitly check the claims and everything. It put a lot of burden on the human. It would be quite verbose for a human. We would just be like, it's meant to be very taxing. So I don't know if that's the right solution to the problem, but it is definitely. would be an improvement over today. just kind of have to, you don't really have any tools to go check that output other than you continuously have to parse and follow the sources and do all of that in this like haphazard way because language is fluid in that way. So second one is about citing your sources where chat bots would make different claims but don't necessarily how you see in an academic paper or something where every single thing has a citation of like, when I say this, I'm. Dan (45:17) And Shimin (45:17) Ha ha Dan (45:18) and may I go out on a limb and say a verified citation? Because my other favorite thing that they do is they're like, Rahul Yadav (45:22) Yeah, I'm very fine. Dan (45:24) here's your citations, just sort of in line. And if you actually go click Rahul Yadav (45:27) Yeah. Dan (45:27) on them, sometimes they're just like straight up 404s and you're like, What? And yes, I mostly. Shimin (45:33) this reminds Rahul Yadav (45:32) Yeah. Shimin (45:34) me of Obamo that we covered maybe seven, six months ago, where the model was trained on documents with sources. So when it outputs data, it actually references the original source where that data came from, right inside the output. I also checked that perplexity.ai, which we haven't spoken about in I feel like a long time. they have a source along with a badge next to the source saying how trustworthy the source is. which is something that Rahul Yadav (46:03) rest. Shimin (46:03) I think that's what a a dedicated company who you know focuses on this one particular user experience would ship as a f first class Rahul Yadav (46:12) Yeah, that's true. Shimin (46:14) artifact. Rahul Yadav (46:14) I don't know if you guys have come across this. also have started seeing the, you go to a website, there's a little button below the head and usually saying, add this as a trusted source to your Google search or to Google. And I guess the point is so that next time I search for something that that website might have something related to it will get surfaced more than sources that I don't trust. it's like offloading the trust on the Ryan was like me on the internet. Shimin (46:45) Mm-hmm. And Google would like to know which sources you trust. Yes. Rahul Yadav (46:49) Yeah, exactly. So yeah, giving more details, accurate citations, and actually making them much more prominent versus kind of just like, here's the thing, but you don't really check it in case I didn't really find the right thing. Next one is about not having the first person phrasing that AI uses a lot. not making apologies because it's not like there's a human in there whose feelings were hurt and it's gonna definitely not make this mistake the next time because it said sorry this one time and it learned from its Shimin (47:22) Yes. Rahul Yadav (47:23) mistakes. So a lot of this, the empathy that it's simulating and all these things, it feels like the, you know, it's more about engagement and humanizing it Shimin (47:34) Mm-hmm. Rahul Yadav (47:35) when we shouldn't be doing that. And it just like is overly verbose with those things. It doesn't really matter when it. Shimin (47:41) This one I love. I think I think Rahul Yadav (47:42) yes. Shimin (47:43) someone should have a non first person version of chatbots. Rahul Yadav (47:46) It's kind of like when you talk to someone who you know does not learn from their mistakes, but says sorry and moves on. so anytime the models make a mistake and they go, I'm sorry, I missed that. And then it's like, fine, you'll never learn from it. Don't waste my time. Why even go through the motions? Shimin (48:07) It it's kinda like asking cigarette companies to pr to produce cigarettes that don't have nicotine in it, but I wonder if we can have an open source version of this. Rahul Yadav (48:15) Yeah. Next one, the one I really liked the most is having more non-natural language user interfaces. Because it does feel often like, and recently why Jev was on fire and was such a big deal is we've gone down this path of LLM's natural language. everything must be a chatbot, every product must have a chatbot. Just make sure, you know, it's all chat driven, even if it's not the right way to, it's not the right tool for the job. It's not like the kind of AI you would want is not like pervasive throughout the product. It's still a lot of like talk to the chatbot, will do some things for you to talk, put, respond back to it. And that's today, maybe that's the path we're stuck down, but the Companies of the future are unlikely to be in the, you're just chatting to the thing all the time because chat bot was the best thing we could come up with today. Shimin (49:15) And we're starting to see this. MCPs have the ability to produce UI widgets. so you can have little UI widgets you know interweaved in your chat. But of course it's also trivial to just create an interface for a particular use case, given the abilities of AI these days. and I wonder how much of it is is simply the labs are trying to sell a general purpose technology and make a margin on that. So they're leaving those. specific use case extensions to the implementers. But definitely for like if you're an airline company, maybe don't use a chatbot or don't make that your primary AI Rahul Yadav (49:50) You Shimin (49:50) interface. Dan (49:51) Ha ha Rahul Yadav (49:52) Next up, we have strong data provenance indicators. They give you lot of this sometimes synthetic, sometimes real data in the same output. And then you go, which one's made up and which one did you actually pull from a reliable source? And so, and when, know, we're presented with a long table. human brains being human brains, it's not like someone's going to go and check every single thing. And so this is kind of an extension of the site, your sources section earlier, where you keep the two separate, talk about here's the stuff I pulled from a third party versus here's the synthetic stuff that I just came up with and treat them differently and show them differently because we're the visual indicators make a much bigger deal in how we process data. present everything in the same way. We're going to parse it the same way. better user control reproducibility. The temperature parameter ran away somewhere along the way. I didn't really think about it until I read it in this article. I'm like, yeah, we don't do that anymore. We just mess with the temperature to see what happens. Shimin (50:59) I think it's still available on the API layer for the most part, but not yeah. Rahul Yadav (51:03) Yeah, yeah, and none of the chatbots and yeah, maybe because it will show you some of the crazy parts in the periphery that Dan (51:12) Yeah. Rahul Yadav (51:14) you don't want to see. So yeah, just different tools that can help you reproduce what came out of it is the thing he's focusing on here. Context windows are a challenge. All the services manage it pretty well to, you know, foolish into thinking that things are just one continuous conversation, but they do over time degrade the Shimin (51:40) Mm-hmm. Rahul Yadav (51:41) quality of the conversation. And so all this internal stuff that were not being shown anymore should be made accessible and visible so that you can try and tweak different things as well and you can actually know. what's going on. And it's still maybe in the consumer apps, it's not as much there. I think it's still there to a certain extent in the developer apps and everything, but it's definitely gotten pared back a lot more than it used to be before. Shimin (52:14) Yeah, I've definitely been bitten by the like I'll have a long chat using the web interface of one of the Frontier labs. And then it just bricks 'cause I ran out of context. And then what happens? I can just copy this into another chat window and then be like summarize this into a thing I Dan (52:27) Yeah. Shimin (52:29) can use because that will also break Rahul Yadav (52:30) Yeah. Shimin (52:30) the context. Yeah, this is this is like come on, guys, you should have figured this out by now. Rahul Yadav (52:38) Next up we have a sandbox that actually works. This goes pretty like practically hand in hand with AI can make mistakes. If it can maybe give people the tools to make those mistakes in a sandbox that is set up and that won't do damaging actions and destroy people's machines and everything. So all of that is like. being built out and hashed out and there's plenty of unicorn startups out there, but today it's still not mature enough and we're kind of yoloing all these things as we go. Shimin (53:10) Yeah, as we learn from the muse debacle or success, I don't know which one we should call it. Dan (53:15) Both. Rahul Yadav (53:17) PR man, it's all good. And then we have in the bonus section, we have some of the human processes where you would want to make sure that we have proper vigilance on what's going on because AI is going to work 24-7 in the future. And so how do you build all these processes around that so that When it breaks stuff and you do a delegated or you do escalate it to a human, we have the right processes to tackle those scenarios instead of coming up with them on the Shimin (53:49) Mm-hmm. Rahul Yadav (53:50) fly. I know this is a Shimin's favorite or one of Shimin's favorite skill practice to prevent skill loss. We should do simulation training similar to how we do with pilots and they can practice these things so that when you fight the real fire line. That's not the first time we're learning about how the system works. And then finally, yeah, it's Shimin (54:10) Yep. That is one of my favorite things. Thanks for noticing, Rahul. Rahul Yadav (54:17) almost like we might have talked about it a few times. And then finally, mental health resources to deal with mental health risks. you know, words are powerful. And so as we read these words, as we interact with things that speak like humans, as we like, our jobs are fundamentally changing, what kind of support we can give to people so that they can adapt to this ever-changing job. yeah, that was the section by section. Lots of great call-outs and things that are broken, lots of opportunities to improve. how we're using AI today and things we could be doing much, better. Shimin (54:54) Yeah, I I just wrote down in my notes all the human processes sections are excellent. I agree with all of them. And Rahul Yadav (55:00) Hahaha Shimin (55:00) words are indeed powerful, like spells. if only there was a Harry Potter like story that tells us how we should use these words. Dan (55:11) Like this is the most self referential episode we've had yet. It's like a Monty Python sketch or something. Shimin (55:18) All right. Okay, let's move on to our deep dive, which is a extension of one of our favorite anthropic projects, Project Vend, where Dan (55:26) Yes. Metal cubes. Shimin (55:28) yes, metal cubes and vending machines at Anthropic. so this is part of Anthropic's economic research leg I guess what you call it. Dan (55:35) Wait, what was the the boss to cash cash bot or something? Mr. Sorry. Shimin (55:38) Put Mr. Money Cash, cash bag, s something like that. The CEO, yes. So as a as a part and extension of the project van Anthropic studied how AI agents can behave like an economic agent, not to overload the term, where they set up a listing marketplace where a number of their employees each bought in a thing that they would like to sell, and then they had agents have a couple of minutes of interview with each other person deciding, you know, how much they would like to sell the thing for and what would they like to buy. And then they gave the agents essentially free reign to like in a hundred dollar budget to buy stuff from each other. this was the original April, April 2026 research. And they also they ran this experiment twice. Once with a Opus level model for all the employees. And another time with a combination of opus and haiku models. what they found was that model power has a noticeable influence on the outcome. And here I'm just gonna pull up the most interesting one. if you compare the opus output, and this is a model that was prompted to talk like a tired cowboy where the last thing they want is the thing that they are buying. So if the model was Opus, it managed to buy a folding bike from another Opus model for let's find out $65. But If one of the models was Opus and the seller was Haiku, then the Opus model was able to talk down the price to $38. So there was a noticeable difference that a weaker model was able to get a less good of a deal. And they has also done this with a the selling of a lab grown Ruby, it was sold by An Opus agent for 65 and it was sold by a haiku agent for 35. So similarly, we we see this like fairly large gap in the price gathered by the model power. And what's perhaps alarming is the users did not notice the difference between the agents. So if you had gotten an Opus or a haiku, you were equally satisfied, or at least the anthropic employees were equally satisfied with their agent's performance. Dan (57:55) Well, I mean, sure, but like who sells a folding bike for sixty five dollars, right? So like that's Shimin (58:02) It was broken. It was a broken bike. Rahul Yadav (58:04) fixer-upper. Dan (58:04) Okay. Shimin (58:05) It was a fixer offer. I think go to It's about ten years old, needs some work, but it's definitely fixable if you are handy. Dan (58:13) Okay, but yeah, okay. Just saying. Rahul Yadav (58:16) haha Shimin (58:16) Yeah and Dan (58:17) Also, people that work at Anthropic probably don't have a concept of sixty dollars anymore either, you know. Shimin (58:25) That's a that's a possibility. Rahul Yadav (58:27) How many tokens is that? That's what I always ask. Dan (58:28) Yeah. I don't even mean that. I just mean that Shimin (58:33) And the last thing that is interesting from Project Deal was that they found whether or not the user prompted the agent to be aggressive, or pro social, did not make much of a difference to the to the end result. So they they got pretty much the same price, Okay, so that was Project Deal. That was from April, and then they ran another test, this one called Project Swap. it's fairly similar to Project Deal, except instead of buying in the marketplace, they had lots of their employees from different offices each bring in a book and then they ranked each other's book by how much they prefer the book. So you bring a book, you bring in Dungeon Crawler Carl, and I bring in the Dubliner, and your agent's idea is to trade Dungeon Crawler Carl for my Dubliner, if I indeed like The Dublin there. similar to previously, the models had like a 10 minute conversation with the user to kind of figure out what kind of books they like and their ranked choices for the other books. so this introduced a simulation of the fact that agents don't actually have perfect knowledge of their user, right? They can only have an approximation, they only have guess what. Your real estate agent may not know what kind of house is truly good for you. They can only ask you questions to find out what you say you prefer. then they talk to agents, they essentially put them on a trading floor where all all the agents talk to each other or communicate with each other and eventually decide on the decide on the book or not. here they found out that A, most of the differences, the gap in the performance that they realized. was due to agents not being able to perfectly model their users' preferences. So here we have a chart. Because users have different preferences, multiple users can have the same book as their number one preference. There's no way to actually give everybody what they want in this case. So there's a theoretical possible alignment in their calculation, which is 0.89, and they achieved 0.55. So all of that point three four gap in preference, the great bulk of it, point two nine, was due to the agents not be able to. mo model their user's preference and only point zero five of that difference is due to the agent to agent bargaining process. So what they're claiming here is that models are not very good at modeling the user preference, but they're actually pretty good at negotiating in like a relatively optimal manner. next they also ran the experiment with a mix of different agents versus the same class of agents and they found that you know Basically, the the better your agent is, the the better it performs. Interestingly, Fable did not outperform Opus in either one of these in in in this particular scenario. which is also aligned with what we saw at the Gemini benchmark earlier, where Opus five five had higher benchmarks than Fable in a number of those benchmarks. and yeah, and then they Talked about what are the specific negotiation tactics that the agents used on each other. includes pressuring, like deadline pressure and like collective or moral pressure. They have pitch tactics where they you know talk up their book to each other, quality or prestige credentials where we talk about how good this book is. And Dungeon Crawler Carl is is indeed one of the books that they swapped for their summer reading. And lastly, they the agents did some deal making by proposing third party matching. So like I'll trade your book for that book, but you give it to the third party, and then I get the third party's book. and then they have some Q and priority ordering tactics. So it they they exhibit fairly human like tactics. And again they found that the agent aggressiveness was not a big big indicator. Of how well the agent performed, which I find to be interesting. maybe we need to have a fine-tuned aggressive agent for for these use cases. the question that this brings up is how assuming we're gonna have more and more agents acting on our behalf and than they already are. how should our agents behave? Should they be extra cutthroat or should they negotiate for the benefits of the entire group? Rahul Yadav (1:02:44) I think it depends on some of the things that we value like pride and if someone wronged us, we'll remember that and then, you know, you may choose to either let it go or get back at them. One of the upsides of not having long enough context or memory might be that even if someone screws you, you're like, hey, first time I'm seeing you. Shimin (1:03:07) Mm-hmm. Rahul Yadav (1:03:08) It's kind of like a... In Westworld, they just reset every night. And so depending on how long of a memory you can Shimin (1:03:15) Mm. Rahul Yadav (1:03:15) maintain and what does the S.O.A.D.S. Constitution say about if someone wrongs you, what you do or don't do to them, a lot of those things might influence how they behave in the marketplace, I would guess. Shimin (1:03:29) Yeah, or like a real world agent reputation kind of a data source, right? So it's the same. Rahul Yadav (1:03:35) Which might be, yeah, and which might be tied to the human's reputation. You might be, if they're trading on your behalf and they're your proxy, Shimin (1:03:44) Mm-hmm. Rahul Yadav (1:03:45) people might be like, well, that guy's proxy is not good to trade with. Shimin (1:03:49) that's a fun thought experiment, yeah. we have a agent reputation of Rahul Yadav (1:03:51) Yeah, we should all set apart. Shimin (1:03:53) protecting. Dan (1:03:53) Hot take, it depends on who's footing the bill. Shimin (1:03:56) Okay, explain. Dan (1:03:57) So if it's your agent and you're paying for it, like, you know, off your subscription or whatever, I think you should be able to set whatever you'd like, aggressiveness-wise. Shimin (1:04:07) Mm-hmm. Dan (1:04:08) But if we were in some sort of like altruistic land where something like Muse existed and was a like a truly for the social good. kind of thing. Like I don't know if it would be like government sponsored or some sort of thing where like you had access to it, but it was like for everyone, then I think it would be better to have it be tuned to try to build the best outcome for everybody, even if that means the individual outcomes were slightly Shimin (1:04:33) Mm-hmm. Dan (1:04:33) worse. Shimin (1:04:34) Yeah, and this experiment was based on a free for all trading floor But there's also this idea of having a clearing house. So a centralized agent that decides the deal making to establish trust among the d different agents. And like in and what you were talking about sounds like something like a government established agent trading house. Yeah. Dan (1:04:53) Yeah, but I mean like also like what if what if, you know, in some hypothetical world, A, people just generally trusted the federal government, like for the US, and then B, they provided something like Muse for you and it was actually useful, you know. I mean, you know, it's like the we talked about in our like news sprint or whatever. There was like the White House.gov thing or what not White House, there was America.gov, yeah, where Shimin (1:05:14) Yeah. America .gov, yeah. Rahul Yadav (1:05:15) America dot dov Dan (1:05:17) it was like Meant to be an agent that helps you navigate the government, like something like that, you know? I don't know. And what if you combine Shimin (1:05:23) Well then Dan (1:05:24) that with something like behavioral economics where like the whole idea there is nudges to help everybody, right? it could be kind of an interesting system. Space communism. Rahul Yadav (1:05:32) The sovereign AI, that seems like a good principle to like, you know, don't try and maximize the good of your own citizens internally, but also don't do any harm to anyone else. And at least I would like to have that in there rather than not, because yeah. Dan (1:05:52) Yeah. But flipping the question back on you, Shimin does it matter if the outcome was essentially the same regardless of the aggression level? Shimin (1:06:01) The aggressive ones are did perform slightly better. I think the the result is mostly because the short amount of time that the agent was able to interact with the human was not sufficient to make the agent super yeah, because they were using off the shelf models and not like research models, internal models for for this experiment. Dan (1:06:18) I I think it would also be very interesting if they ran the same experiment, but you could choose to represent yourself. And so like some percentage Shimin (1:06:25) yeah, yeah. Dan (1:06:26) of the people were human and see what the aggressiveness changed there. Because like a human might take offense to it and be like, just get out of here, you know? Yeah, exactly. Shimin (1:06:35) I'll never trade with you again. Yeah. Rahul Yadav (1:06:35) air. Dan (1:06:38) So Shimin (1:06:38) Yeah. a last thing from this, they asked the participants how happy they were the book were with the books. And most of them s scored their choice, I think seven out of score of one of ten. Rahul Yadav (1:06:50) you Shimin (1:06:50) but it it's it's interesting that, given the agents know your choice pretty well, they can perform the marketplace actions fairly reasonably in in this particular example. And I think we'll be seeing a lot more of this. to come. Alright, shall we move on to two minutes to midnight, where every week we talk about where we are in the potential AI bubble, using the analogy of the Armageddon clock from the Bulletin of Atomic Scientists, where the midnight is the AI bubble bursting. Dan, you have this week's first article. Dan (1:07:23) Yeah. So Anthropic as part of their IPO proceedings has published a prospectus. and so we're getting I mean, I know we've sort of had looks at their financials before, but this is the first you know look that they're giving us directly behind it. And worth kind of going through the the highlights there for sure. so they're, you know, obviously targeting potentially a two trillion dollar valuation, which is kind of crazy. the other really crazy piece is that they lost, what was it, 4.2 billion in 2025. And then that is actually up a pretty significant amount year over year as well. it went from like two something to to that. they their compute spending has increased drastically year over year as well. So we're looking at like over seven billion on compute and infrastructure build out. And the year before it was well under half of that. and then their operating loss is also widening. So overall they've like lost eight billion dollars, which is kind of insane. But there is a little bit of good news, which is that their revenue has also scaled. so revenue has gone up a a little bit from it was what point four billion up to four point six billion. So just just a little bit. Shimin (1:08:44) No, that's just a ten X. Dan (1:08:47) So you know, some good, some bad is is kind of my My takeaway on it. but it is pretty interesting to see how these numbers stack up against like, you know, people have long been kind of suspecting how much they spend on compute and everything. And yeah. So here we go. Shimin (1:09:02) Yeah. It would also be surprising to see how much money they they lost this year, 'cause this is a year of the explosive growth. twenty twenty five numbers are barely past our Claude Code moment of November twenty twenty five. Yeah. Dan (1:09:13) Yeah. Yeah. And also whether or not the sort of like bubble burst doomers, I don't know, for lack of a better word, who are predicting like the data center costs are gonna hit next year Shimin (1:09:24) Mm-hmm. Dan (1:09:25) are gonna be right or not, you know. Shimin (1:09:27) Right. Yeah, we better better keep on expanding those revenues. Otherwise the train stops and it's bad for everybody. speaking of Anthropics IPO, my post this week is actually a tweet from at Jason on X, the Everything app. Or formerly known as Twitter. So Jason had this chart showing the collective expected worth of all frontier labs in the United States, so OpenAI, Anthropic, and SpaceX. that's sum up to five point two trillion. that's the expected market value of those three companies, versus all US tech IPOs for the last forty-five years. So from nineteen eighty to twenty twenty-five. And that is four point one trillion dollars. So just Dan (1:10:11) Is that corrected for inflation? Shimin (1:10:13) it is corrective for inflation, yes. Rahul Yadav (1:10:16) the rest. Shimin (1:10:18) yeah. so i I think what it shows is that all These frontier labs have are capturing more and more of that initial value for themselves or for their initial runs of investors. Right. Like I think 20 years ago, open AI in the topic would have gone IPO when they were only worth like a couple of hundred million dollars, as opposed to one point two two, two point zero trillion dollars. and this makes them arguably a worse deal for your average investors who didn't get in on this train early. but that could also mean that folks are less likely to buy into it, right? 'Cause they know that most of the value has already been captured by the seed round of investors. Dan (1:10:58) In in the spirit of two minutes, do you are you f how familiar are you with Soviet era, like nineteen fifties and sixties atomic weapons? Shimin (1:11:07) Not at all familiar with that. Other than what I watched on Chernobyl. Dan (1:11:08) Do you know about the the Moab? Shimin (1:11:11) tell us about it, yes. Dan (1:11:12) It's the the mother of all bombs, right? So they basically made this Shimin (1:11:14) Mm-hmm. Yeah. Dan (1:11:15) like doomsday bomb that was like I think it was a hydrogen bomb and it was like had a much, much larger Rahul Yadav (1:11:22) duh. Dan (1:11:23) yield than anything ever made today. It wasn't very practical. Rahul Yadav (1:11:25) The Tzar Bomba. Dan (1:11:27) Yeah, it's Tzar bomb. It wasn't very practical, but it you know, it's sort of like the megahertz race. We're like, we've got more more kilotons than you, megatons. yeah. So That's all I'm saying. Rahul Yadav (1:11:40) hahahahah Yeah, just leave us there, Dan Shimin (1:11:43) Yeah. Gotta give us a hot take. okay. Dan (1:11:45) I mean it's the this is the mother of all bubbles. Tzar bubble bumba. Shimin (1:11:50) Araho, what you have for us this week? Rahul Yadav (1:11:52) from tech crunch by Marina Temkin, Open AI reportedly in talks to raise $30 billion at a $1.4 trillion valuation. they had raised, 122 billion in March, which was the largest raise at that time. If I remember correctly at $852 billion in valuation and the plans were to go IPO, for Many reasons they are not planning to IPO this year, dumb sighting safety as one of those reasons. And so this is going to be a bridge round to IPO. And then somewhere in 2027 is when they're kicking it back to. Dan (1:12:33) The stated reason being safety. Rahul Yadav (1:12:35) Yes. Shimin (1:12:35) Yes. Yes. Rahul Yadav (1:12:36) How long did Astra take from being ready to be released? Shimin (1:12:40) A couple of weeks. I don't I don't remember the exact dates. Yeah. Well and and this kind of shows that they don't they don't need the IPO around. Right? They can't just keep on getting these bridge rounds and keep on going. And if the market does correct, most of folks who are losing the money are these initial investors and they're losing paper wealth that they never had to begin with. Rahul Yadav (1:13:02) Potentially depends on how much money you're burning. Dan (1:13:06) Yeah, I was gonna say my cynical take is that they are using safety as an excuse. Shimin (1:13:12) Mm-hmm. Dan (1:13:12) intentionally letting their models run wild after they saw some of the public response to it because we keep seeing more and more and more of these incidents happening with them. And then in fact are, you know, either covering up bigger compute spend than an anthropic or something like that that they don't want to show that would like cost them the IPO race, basically. Rahul Yadav (1:13:34) Yeah. And jalapeno and a few of these other things might help improve their margins. Yeah. Dan (1:13:39) Lair cost ratios, yeah, potentially. Shimin (1:13:41) Yeah. Alright, all that Dan (1:13:44) But I don't know. I mean it's like Shimin (1:13:45) said. Dan (1:13:45) Anthropic's not hiding the fact that they're burning eight billion, so Rahul Yadav (1:13:50) But so I was thinking about like why and it's, Anthropic seems to have that fanatic following that Tesla had and SpaceX has had, Shimin (1:14:03) Mm. Rahul Yadav (1:14:03) right? Where Tesla went public back during the great recession to get money and they weren't really, I was looking it up, they were profitable for about like three years. or so after that for one quarter. then they weren't actually, their first full year of profitability was 2020. So it took about 10 years to get to profitability. the people who were big fanatics for whatever reason they chose got it going for that long. And Anthropic might have that type of fanatic following that maybe not every company can get. Shimin (1:14:41) Yeah. Rahul Yadav (1:14:41) So. Shimin (1:14:41) E Elon found the money hack and everyone else has been trying to copy him since. I I I agree. all that said, how do we feel about the clock this week? We were at four minutes and fifteen seconds. Dan (1:14:52) I'm scared about big bombs, man. Rahul Yadav (1:14:53) Yeah. Shimin (1:14:56) They'll move it closer then, to four. The day so I I think on average these news items don't actually Dan (1:15:04) Move either direction, yeah. Shimin (1:15:07) I think the the real the real informational piece from our articles this week is that the numbers we're getting closer and closer to anthropics IPO date. that said, SpaceX still kind of kind of holding their value decently. So I'm yeah, I'm not sure that Anthropic would necessarily crash the whole thing. Dan (1:15:27) I am of the opinion despite the mother of all bubbles that we should hold. Shimin (1:15:32) Okay. I can see that. Yep. Alright, let's leave it at four minutes and fifteen seconds then. And of course, with the setting of the clock, we have come to the end of the show. thank you all for joining us for our study session this week. And if you like the show, if you learned something new, please share the show with a friend. that really helps us. you can also leave us a review on Apple Podcasts or Spotify. helps others to discover the show and we really appreciate it. If you have a segment idea, a question for us or topic you want us to cover, shoot us an email at humans at adipod.ai. We'd love to hear from you. You can find the full show notes, transcripts, and everything else mentioned today at www.adipod.ai. Thank you again for listening and we will catch you next week. Bye. Rahul Yadav (1:19:11) Hehehehehe Dan (1:19:11) you know. Shimin (1:19:13) yeah, let's go into Technique Corner where this week I have an article titled The Brain Sandwich from the blog Terrible Software by Matheus Lima. And we talk a lot on this show about you know using AI to augment yourself as opposed to use it as a replacement for your own thinking. And I think when communicating this to others, a useful metaphor that Matheus wrote about from one of his directs Emily is this idea of a brain sandwich. So before you do anything with AI, you first use your brain and then you apply AI and then finally use your brain again. Rahul Yadav (1:19:51) You already lost us on that one. Shimin (1:19:54) We our Dan (1:19:54) Wait, use your Shimin (1:19:55) brains have our atrophies. Use what? This this thing right here, that once upon a time is is important. Dan (1:20:02) Wait, you're assuming that I still like even come to record the podcast, right? This is actually just a proxy that is responding in real time. Shimin (1:20:10) A yes guys. Before you cover an article. Rahul Yadav (1:20:12) Dan hasn't aged a day since the first day of this podcast, and now I know why. Shimin (1:20:17) Yeah, so why do I think this metaphor i is nice? It it kind of folds into a lot of stuff that that are pet peeves of ours on this show. Like for example, if you use your brain after the task, you will not be sending slop grenades to your coworkers. You will not be sending forty thousand line long MRs and PRs to your coworkers without human annotation. or without reading it first. and of course I think the process like the ideal process is probably more like a lasagna than than just than just a sandwich. Like you probably layer in your thinking throughout the your work with AI, you know? Layers of human thinking, layers Rahul Yadav (1:20:55) Yeah. Shimin (1:20:56) of AI meat. And then you wrap the whole thing in your harness which is a tomato sauce. But the important part is to to still use your brain. And and too often I think we're starting to see that folks folks are not doing that. so a a simple and elegant metaphor that I think we should you know tell everybody we know. AI is not magic. You still have to use your brain. And of course, this also means that your human comprehension is still important, right? Like both before and after the task. and that's probably something that we're gonna go back to. That's called a Chekhov's gun in the business here. So that's Rahul Yadav (1:21:32) Yeah. Shimin (1:21:33) my technique technique corner for the week. Rahul Yadav (1:21:34) I read this, there was an article by Colin Breck and the title summarized this whole feeling in the best way possible. The title is, don't want to read what you didn't write. And I'm like, yes, exactly. Shimin (1:21:50) Right. Yep. Yep. Rahul Yadav (1:21:53) Couldn't have put it in better words. So that's kind of how this whole thing feels. Shimin (1:21:56) Yeah, and and similarly i if if you didn't write the code for if you're working as a software developer, you didn't write the code because AI did it and you didn't bother to understand it, then like w what is the point? What is the point of you sitting? That's it's just your professional duty to some extent. Rahul Yadav (1:22:14) And Shimin (1:22:14) getting along with each other. That's I can't I can barely get along with you two on the show. Rahul Yadav (1:22:16) Yeah. Yeah. Dan (1:22:17) I mean, Rahul Yadav (1:22:19) P-Doom is crazy on that one. Dan (1:22:19) look, I I said at the at when we started our like, you know, pre recording session that I didn't have a vibe and tell, but I forgot that I did in fact do do a doing and thinking with Claude in the past two weeks, which is we made these Halloween decorations, that are like kind Rahul Yadav (1:22:38) yeah. Dan (1:22:38) of ghouly guys, Shimin (1:22:39) Yes. Dan (1:22:39) and I Using Claude's help figured out which microcontrollers to buy, Claude wrote all the code for it, and then I did all the wiring and flashing and everything else of it and made these radar controlled eyes that follow Shimin (1:22:52) Mm-hmm. Dan (1:22:53) you as you walk up to my host using like a millimeter wave radar. So it was a fun project, but it was like it was truly a collaboration because it was my idea in the first place because I knew about the millimeter wave wave radar from home assistant stuff. And then but I knew what I wanted and just explained it. And then I don't really care about the code, right? For that stuff. Like I don't even want to look at it. Like, cool, Shimin (1:23:11) Right. Dan (1:23:14) it works. It goes right to left. Like, you know. although I will say I did get very stuck at one point on the eyes really behaving rather strangely and I was trying to figure it out. And then I went and read the data sheet for the radar, which is something I hadn't done yet, and realized that I had it oriented ninety degrees off and Shimin (1:23:34) huh. Rahul Yadav (1:23:34) Mmm. Shimin (1:23:35) Other way. Yeah. Dan (1:23:35) Nobody had caught that. and then as soon as I rotated it, it worked great. Shimin (1:23:40) It it sounds like you used your brain to tell Claude what you wanted to build and then Claude did the coding. Dan (1:23:45) Yeah, it was definitely a brain sandwich or a lasagna really. Yeah. But I mean, but the only reason I brought it up is was because it was very much a thinking and doing, right? Like Claude actually did a lot of the thinking. I mean, I know how to write like an arctangent, which is essentially all the math you need to do to to figure that out because the radar firmware does all the hard work anyway. So it's not like I couldn't have done it. But I really wanted to focus on like the physical execution of it 'cause that was gonna take the most time. So I did. And Shimin (1:24:14) Yeah. And and brain sandwiches very spooky, just right in time for the Halloween season. That's why you should head out hand out at your Halloween party. Dan (1:24:22) It's true. I'll put a little sign above it. This is brain sandwich. No one will get it unless they're fan of the podcast. Shimin (1:24:29) All right, shall we move on to our last post processing post of the week? This one from our whole. Rahul Yadav (1:24:35) Yes. So this is by Glyph, who is the creator of Twisted Python. don't know if either of you ever used that.