Latest / Artificial Developer Intelligence / GPT-5.6 Sol, the State of AI, Know Your Unknowns With Agents & the Permanent Underclass
Transcript
- Shimin (00:00) Hello and welcome back to Artificial Developer Intelligence, a weekly conversation show where three software developers navigate the perils and opportunities of AI assisted software engineering. We go through hundreds of links and dozens of newsletters each week so you can keep up with AI while doing the dishes or hiking in the woods. My name is Shimin Zhang and with me today are my co-hosts and Dan Pretty but particular about placement. Lasky, especially his glasses And Rahul, Sam Altman, how many divisions does he have? Yadav. Hello, gents. How was your july fourth celebration? Dan (00:27) True. Explosive Not really. I I actually went to bed early and yeah, was pretty lame. But maybe Rahul did something cool. I don't know. Rahul Yadav (00:44) Out and about in in in the nature doing dishes. It was awesome. Yeah. I got a full vacation. But yeah, weather's beautiful in Washington right now. I was out in the woods. Dan (00:50) But you're doing dishes in nature? Nice. Shimin (00:59) And of course Dan and I were helping our dogs from this traumatizing event. that is the two fiftieth. Dan (01:04) Mine are both pretty chill about fireworks, thankfully. We've actually like so much so that we actually like brought one of them to a fireworks show and then realized the mistake there, which wasn't that he was afraid of it, is that it was like way too loud for for puppyers. And I like, my god, we've got to get him out of here. But yeah, he was still fine. Shimin (01:17) Mm. no. Rahul Yadav (01:21) Mm-hmm. Shimin (01:22) poor puppy. all right. On today's show, as always, we're gonna start with the news thread mill where we're gonna talk about the new GPT model, GPT five point six Sol, as well as a update on the Meta drama Dan (01:35) Then we're gonna have a brief jump through the hardware hut where we're gonna talk about some hardware you can actually buy. Very exciting. Shimin (01:42) Yeah, then we're gonna move on to the technique corner where we're gonna talk about a article called Know Your Unknowns, about the unknowns of dealing with AI. Dan (01:50) Then we have a extended post processing this week where we're gonna be talking about Raul's favorite topic, the permanent underclass. we're gonna be talking also about how AI saves about three percent of your hours, but it doesn't hit your paycheck. and yeah, I guess that's it for post processing. So not that extended. I lied a little bit. Rahul Yadav (01:52) Yeah. Shimin (02:07) Yeah. well we are actually gonna have an extended deep dive this week where we're gonna talk about the state of the AI economy, which is also a preview for two minutes to midnight, along with a paper titled Does Code Cleanliness Affect Coding Agents? Dan (02:25) Excited for that one. And then finally, we're going to wrap it up with two minutes where we talk about where we're at in the bubble slash implosion. Shimin (02:32) Let us get started. Alright, first up, last week, OpenAI announced the previewing of their latest GPT model, GPT five point six Sol. it actually comes with three different flavors, where Sol is the maximum thinking, and then there's Terra, a everyday workhorse model, and Luna a fast and affordable model. This is OpenAI's offering. against anthropics Mythos and fable models. And it seems to be around the same roughly class of models where you can do extended long tasks with GPT-5.6. what is I guess unfortunate is that we are not able to get our hands on GPT five six because it is still in closed technical preview where only selected partners of OpenAI have access to this model. Partially due to the White House executive order where the US White House is reviewing each frontier model before deciding when and how it could be released. And of course Fable five came back last week. it was a limited subscription offer. That supposedly ends today, July seventh, as of our recording, but it just came out that they pushed that back to July twelfth. so we still have access to Fable Five, the first Fable Class model, I guess. And GPT five six is on the tails. Dan (03:58) Have you all been using Fable Five? Shimin (04:00) I have, mostly for really complicated, longer, more comprehension y tasks. what about you, Dan? Dan (04:08) I've been using it for everything. I still haven't hit a limit yet. So I don't know if that says more about my usage than anything else. But like even just like qu normal, boring question and answer stuff. it's been kind of fun. But I think probably the most complicated thing I've done with it is I'm I'm still stuck on Pi Agent, you know, we've talked about many times. And like I'm just stuck on using it because I'm just enjoying it a little bit. So Shimin (04:10) Ha ha ha. Dan (04:35) One of the things that's been frustrating is like my local model is like pretty slow to actually like write extensions. So I realized that I could just point Claude at the Pi documentation and be like, write an extension for this outside of Pi essentially. And then run it inside of Pi using local models. So I've been using Fable for that and it did a pretty decent job. I wrote two different extensions for it so far and they work. Rahul Yadav (04:47) Mm-hmm. Shimin (04:50) That's pretty funny. Or they good extensions? Dan (05:00) It's not like rocket science stuff. One was like build a vector memory store for Pi and then using I've been playing around with Lance TV for doing that lately. and it just used like a local like JS-based embeddings model because it was cheap enough to do. And then that one worked fine. And then what's the other one? yeah, like a pretty specialized like retrieval system for Shimin (05:05) Right. Yeah. Mm-hmm. Dan (05:24) my personal journal that I didn't want going out to Claude. and the other interesting kind of weird challenge there, sorry, it's taking a lot longer than I thought, but challenge there is like for some reason tool calling is kind of slow on DS4. So like one of the things that happens is like it when it reads a lot of entries, it's like very, very slow. Like it can take hours to like process a bunch of them. And it was kind of too dumb to batch it. So this was basically like Shimin (05:27) Mm-hmm. All right. Yep. Mm-hmm. Dan (05:49) Batch these things and read them in bulk based on like, you know, some light kind of rag stuff. So which also seems to work. It was kinda interesting, but I didn't I didn't wind up with any life epiphanies, but you know, still fun projects, so Shimin (05:57) Yeah, I had I had a a bunch of work that was waiting on Fable to get re-released. So I used like 38% of my weekly limit in like 24 hours. And I was like, okay. Let's take a step back. I'm back to Opus 4.8 for a while. And I just I just didn't turn it back on, surprisingly. but I'm gonna try and use up my quota now that it is back till the twelfth. one Dan (06:15) Power user here, yeah. Shimin (06:26) One thing I want to mention as a part of GPT five six's benchmark is if you if you look at this comparison of the various flavors of GPT-5.6 with GPT-5.5, you actually see that as the output token increases, the GPT-5. Still, which tells us that if you extrapolate We probably actually have a lot more room to gain, at least at least on this particular benchmark, which is the Gene Bench V1, which is relatively hard where it is not at all saturated, where even GPT five six Sol is only at about thirty percent with 35, 40,000 tokens. You can kinda easily see if you increase that output from you know forty thousand tokens to like a hundred and twenty. There's probably like another five, ten, fifteen percent improvements left still in these models. So it looks like the abilities of these models models are not yet plateauing, especially in relationship to its output length. And this makes sense, right? Because one of the things we talk about with Project Glasswing is it's not that mythos is so much better, but that mythos can work for longer and can chain a bunch of exploits together. To get that overall improvement. so it seems like we still have a way to go on that. And then I also have a related news that also came out today, July seventh. Beijing is looking at curbing overseas access to China's top AI models. So the China's commerce ministry is looking into restricting the top tier Chinese AI models. from overseas use. And they're working with companies like Alibaba Byte Dance and ZAI. So I wonder if we're gonna run into this weird case where you know the US is locking down its models, China is locking down its own models, and everybody is gonna be using VPNs. Like I'm gonna be buying a VPN in Shenzhen in order to access the latest ByteDance model and get like the coolest videos. And of course the Chinese citizens are continuous. To use subscriptions from Singapore to access like anthropic and open AI's models. we are looking at Dan (08:34) Well, there's there's also that huge r weird reseller market too that we talked about in China, right? Where you can get an account that may or may not be legit because they might be piping you two cheaper models. Which is kind of funny. Yeah. Shimin (08:45) Yep. yeah, you can never know for sure. Yeah, so the AI Cold War, as I'm gonna call it, may be starting. And maybe we'll be, you know, buying black market open weight models on a flash drive just because we're not legally allowed to own those models coming up. Dystopian future, here we come. Dan (09:06) Not. But who knows? Shimin (09:08) Our second news item of the week is brought to you by Dan. Dan (09:12) Yeah. So our everyone's favorite human, Mark Zuckerberg, apparently held all hands. so this comes on the heels of the some of the stuff that we'd reported previously about like there being internal quite a bit of internal backlash at Meta around the like I don't know, I think of it as like the AI puzzles department. I forget what it was actually called. But Shimin (09:32) Yeah. Dan (09:32) Yeah, where they like reassigned a you know non-zero portion of their staff to make puzzles for AIs and then fired another 10%. which is pretty wild. But so the on the latest piece of this, Mark has sort of admitted a little bit of a mistake. So there was a internal town hall that happened on July 2nd, where he was quoted as saying, like, the reorg is not was not clean. And executives miscalculated the timing of the changes. So the little poll quote from Zuck was the trajectory of the agentic development over the last four months hasn't really accelerated in the way we expected. It's like, okay, thanks, Mark. so he's he was also on the I won't say on the record because apparently this was. Reuters got their hands on. but he was on the record saying that like the company's bets on the the new structure, meaning this like, you know, puzzles department have not come to fruition yet. And then as a little side note, TechCrunch, who's the the sort of like featured article here, threw in that Meta's estimated to spend $145 billion this year on AI infrastructure. So, Shimin (10:25) Mm-hmm. Dan (10:38) Almost in two minutes to midnight territory here, but felt like it's was still tracking the you know, 'cause we've sorta been tracking the first the clicks and now this thing, you know, the puzzles department as we go along here. So Seemed kinda relevant to that, but yeah. Shimin (10:41) Mm-hmm. Mm. Rahul Yadav (10:52) Have they ever outlined what they're trying to do at Meta? We're doing all this to do this. You know what I mean? Like you don't ask that question about why is open AI or anthropic building models. I don't know why Meta's doing it. Dan (11:06) Yeah, that's a good question. I think it's we're doing all this so we don't FOMO. Rahul Yadav (11:11) Yeah. So it's like yeah, when it says y you know, whatever like their goals were, the the development wasn't on pace against what? Right? What exactly are they going? Dan (11:12) I'm not sure though. Well, I mean the one trend that I've noticed that I is kind of I don't say funny, it's more interesting, I guess, that like Zuckerberg seems to be chasing any potential for like the next cell phone, right? Because he like drastically missed the boat on that one and is still mad about it. So like first we had like Metaverse, right? Which like he just, you know, pivoted the whole company around that, even the name still maps to that. And then now he's like pivoting to AI too, and it's like Rahul Yadav (11:36) Mm-hmm. Shimin (11:37) Mm-hmm. Rahul Yadav (11:40) Yep. Yeah. Shimin (11:50) Mm-hmm. Dan (11:51) I guess you could argue it's in service of like you need good AI to make something like AI glasses work to then displace phones. Rahul Yadav (11:59) Mm. Dan (12:00) I don't know. Do we see AI glasses displacing phones? People are buying I don't know what they're doing with them. Rahul Yadav (12:05) Yeah. And they Shimin (12:07) Yeah, I d I have not seen them in a wild yet, but Dan (12:10) These Rahul Yadav (12:10) Yeah. Dan (12:11) these are not my new spacks are not AI glasses, they're just boring astigmatism engines. Rahul Yadav (12:15) Come on, man. Those are actually snap glasses. Shimin (12:18) Yeah. Dan (12:19) I'm recording this podcast on them right now. You have no idea. Yeah. that'd be kind of meta. Rahul Yadav (12:20) Ha ha ha ha. Shimin (12:24) We're gonna release a behind the scenes f first person version of this recording. Dan (12:27) Who It's the same recording, it just has my notes next to it. Mysteries have been revealed. I read notes sometimes, guys. Shimin (12:35) Yeah, so Rahul Yadav (12:38) Mm. Shimin (12:39) So meta bet, you know, they laid off ten percent, they transfer another ten percent into the quote unquote AI gulog, and we still A don't know what they're actually are trying to do. I assume they're trying to create a frontier model, which they haven't done yet. And it's been three to six months. So like we'll find out if this is just another Oculus bet. Dan (12:59) I will say that they recently released the first product that I'm actually interested in trying from Meta in quite a while, which is apparently they bought some like little startup. I don't remember the name of it. but it basically is like a create your own bite-sized video game from a prompt, and it runs on your phone. kinda, yeah. But it was like, but you can share it, and then there's like a feed of other people's like bite-sized video games. And I'm like, that's actually kind of a cool. Shimin (13:16) Farm ville two point Rahul Yadav (13:18) Yeah. Dan (13:25) idea or like yeah. So I don't know. Shimin (13:25) That's pretty cool. It turns out the gulags are actually producing bite-sized video games. That's actually you know what? Maybe I appreciate their sacrifice for our entertainment. Okay, guys. All right. Well we will try the one of those video games when it does come out. But speaking of next week's vibe and tale. All right. Our hardware hut Hut corner also is brought to you by Dan from PC Mag. Dan (13:31) Ha. Yeah. It's out. It's I'll I'll find the name of it and send it to you. But Yeah. So here's the I'm gonna be like, I'm gonna hit all the LLM tropes today. Here's the honest truth. I might have lied again when I said it's something you can buy. I mean, you can buy it, but like, my gosh. so the we have talked a lot on the podcast about framework desktop, right? Like the Ryzen AI Max 395 chipset. pretty great chip. What makes it special is the large amount of unified memory you can stack on it, similar to how Apple does their stuff. the other noteworthy thing is like it's been a bit cheaper than Apple stuff up until now, I guess. but AMD has now released their own official version of that Ryzen AI Max 395. It's called the AMD Ryzen AI Halo Developer Desktop. it looks pretty much like a mini Mini PC. It's not dissimilar looking to the DGX Spark, which is like NVIDIA's version of this. and very similarly to the NVIDIA version, they've really focused on kind of polishing the software, which is kind of interesting. So when you buy it, you have the option of interestingly choosing Windows if you wanted to. Look okay. I mean, you can, I guess you can run Windows on it, but it Shimin (14:41) Mm-hmm. Rahul Yadav (14:56) Mm. Dan (15:00) Also kind of funny, the PC Mag review was largely about Windows, and I was just sitting here going, like, okay, like but the neat part is like if you get it, it comes with the AMD Ryzen AI Developer Center app, which is pre-installed, and that gives you a bunch of automated scripts to give you things like isolated PyTorch environments. it has like little Rahul Yadav (15:06) Yeah. Shimin (15:23) Mm. Dan (15:25) run books for updating your, you know, core dependencies. And then there's a few other of them for like sort of use case based tooling setup. So it can just sort of get you going right out the gate, which for AMD hardware is in my experience at least a little bit trickier than the Spark because you know, it's not just like CUDA where, you know, a lot of folks are working on that out of the box. So it's kind of neat that they're offering that. benchmarking wise, kind of interesting because there's a lot of 395 chips out on the market, right? We've got the framework desktop, which I've talked about ad nauseum. there's a a new one that just came out too this week, which is like the it's like a little mini PC company, I forget, but theirs is around the same price point. so this one will be yeah, so this retails for four four grand for the 128 gig one, which Shimin (16:07) Mm-hmm. And how much are we talking about exactly? Dan (16:14) Quite frankly, you wouldn't want anything less than 128 if you're gonna run serious models on it. So yeah, it makes me very glad that I got the framework desktop for I think it was like two when I got it. So RAM Pocalypse is nigh. But but they they benched it in this article specifically against the HP's version of this, which is like the Z2 mini G1A, which is just like a little mini PC with 395 in it. And weirdly, it scored a little bit better on like Rahul Yadav (16:26) Mm. Dan (16:40) productivity bench than the the G one A, but the G one A spanked it in GPU scores, which is I find also odd given that like they're basically the same hardware. So it must come down to cooling or maybe thermal throttling or something like that. Shimin (16:51) It's it's the same card, yeah. Dan (16:57) for the for that stuff. But yeah, anyway, it's good to have, in my opinion, more options and it's cool to see more companies coming up with like a sort of developer first package like this that's not, you know, purely a roll Shimin (17:10) Mm-hmm. And of course you can also play video games on this, right? So you can have a f four thousand dollar write-off or with that comes with Windows. You can run all your games on this. It's only twice as expensive as the the new Steam machine. Dan (17:13) That's true. I mean it's a normal PC, yeah. Yeah. Yeah, and it's I mean it's a pretty respectable GPU. I think it's around like a fifty sixty in terms of like frames. like I you probably wouldn't want to do like four K gaming on it, but it it runs games for sure. I ran some on mine just to prove it, so Shimin (17:37) Yeah. And it's got a special airflow intake design to help with cooling. and that's where the middle name of Dan came from this week, where it's it's got this attractive exterior, but you gotta place it at the right place so it would get lit. Yeah. Dan (17:45) Ha ha. Cause a of problems. Yeah, that sounds like me. Rahul Yadav (17:52) Yeah. Dan (17:53) Fair. Yeah, you can't put it on its side. That's that's a weird one. it has to be pizza box orientation. So no books for you, only for Rahul. Shimin (18:00) Alright, I'll be on the lookout to see one of these AMD boxes in the wild. I've only so far seen the the spark in the wild. and once you get one of these boxes at one of the AI Tinker events, yeah. Dan (18:08) Where did you see the spark? Cool. Well, you know, maybe if we keep talking about these things they'll start sending us some review units so we could see all of them in the wild. That's not true. Shimin (18:18) That's the only reason why we do hardware hut absolutely. Tariq, he is a software developer at Anthropic working on Claude code. And these techniques that we we're gonna talk about actually came from his work creating the release video for the Fable 5 series of models. So let's dig in. The title of this page is Know Your Unknowns, and of course the the specific thing that he's talking about here is when you are working on a project, you have your no knowns, your known unknowns, your unknown knowns, and of course your unknown unknowns. And this is made famous by Donald Rumsfeld as part of the justifications of I wanna say the Iraqi evasion back in two thousand and three. I will cut this if it turned out not to be the case. these 11 techniques can be used with pretty much any AI models that are on the market today. they are broken down by pre-implementation, during implementation, and post-implementation stages. And we're gonna and I'm gonna kind of give a brief overview of each one of these. You can check out the the full article in the show notes for this episode. And we can kinda feel free to interrupt me while I'm going through this to see if you've done something similar or this is in alignment with what you do. So the first technique is blind spot pass. And the idea is to have claud or your agent of choice go into the repo and teach you things that you don't already know about the repo. Basically essentially to fill its context with knowledge about the repo and then communicate that context to you. Is this something that you guys do? yeah. Yeah. Dan (20:11) yes. Rahul Yadav (20:12) Yeah. Dan (20:13) Across multiple repos and then sometimes they'll synth have also synthesize the output too. which is kind of fun. Shimin (20:13) That's Yeah, it's it's he in the on the side there are some examples of, you know, you can discover potential landmines, histories, and then also missing concepts that you may or may not know, which I find to be helpful and I do see this. The the next technique is to ask your agent to teach you the domains of vocabulary so that you can use it more effectively. in this particular case, he was working on the fable five videos, so The agent taught him about exposure, contrast curves, LUT, saturation, vibrance, et cetera. So then he can write more professional and more descriptive prompts. I think this is something that I haven't had a ton of opportunity to use just because I mostly work with a tech stack that I already know. But I I do usually do this in the form of like look at existing solutions and teach me what each one does. And I'll learn alongside it. Dan (21:13) I've been having a lot of fun with something that I got from you, Shimin which is the teach me this but in an interactive way. so I'll I'll I'll these days a lot of times what I'll ask for is like build me a small thing that explains concept X. And then you get to like it'll you usually has some knobs that you can like physically play with to like walk through the steps or whatever. which has been I think a pretty I like learning that way, so it's been good for me, but Shimin (21:20) Mm-hmm. Yeah. Rahul Yadav (21:28) Mm. Dan (21:39) yeah. Shimin (21:40) Yeah, really makes it more of a collaborator as opposed to just a tool for a task. the next item here is four design directions, where you s get your agent to give you four concrete implementations of a particular thing. This is kind of instead of just generalizing a rough idea of what a solution space looks like, do four or n number of non-over overlapping solutions. So you can get a feel, a concrete feel of what possible solutions are out there. And yeah, this one also makes sense. This r kind of reminds me of some of the work I was doing with inhabited design where you you get it to generate multiple designs before deciding on which one you like the most and then we got a couple that are also fairly UI specific. do a mock of the overall design before fleshing out the actual UI element. also makes sense. And you can kind of tweak the l leverage points of the design of the rough sketches before you go into the details. next is brainstorm the intervention. this is the non-concrete version of the solution space overview. So before you actually commit to a feature or a fix, get the entire option space. So based on what is actually in the code base. So then you can design your trade-offs. In his specific case, he also got the agent to order them by easiest to hardest, but I could see other ways to list these various options as well. I'd probably use this one the most, which is like, hey, just give me a list, give me a rough three to six different I ways to tackle this problem, and then describe the trade offs. And then I can kind of do a summarization and decide on the solution based on those approaches. the next one is interview where have the agent interview you to remove any ambiguities that may Dan (23:34) It's probably my favorite. Yeah, I do this all the time. Shimin (23:38) What's fascinating, I don't my f I don't find myself using this at all. This is actually one of the ones that I need to practice. Rahul Yadav (23:42) I don't like it either. I tried Dan (23:42) Really. Really? Rahul Yadav (23:45) it and I'm like, stop asking me questions. Dan (23:47) So usually I mean, usually the way I do it that I find is effective is I will I have like a three stage process that I tend to use for this. So the first is I don't ask for anything. I go into the code base and say like, what do you think about X? Or like ask it kind of a hypothetical that requires it to basically like prime its context on the the code base. So then after it's pulled a bunch of files relevant to that and has them in memory, then I'm like, Okay, here's what we're actually doing today. Now that you know all those things, we're gonna do this. And then once I've stated that, and that's usually like maybe a two or three sentence prompt, then I go, What am I missing? Like what do you need to know that I just didn't specify? And then usually it'll pop up that cool like question gooey thing and hammer me with three or four of and that's it. And a lot of times they're just stupid things didn't read right 'cause English specs, but like sometimes not. Sometimes it's a real miss that I just like, yeah, I didn't think about that edge case. Shimin (24:16) Mm-hmm. Mm. Yeah, this is definitely one that I want to incorporate into my workflow more. I read this and I was like, I can't believe I've not been doing this this entire time. So the next technique is point the agent at a reference implementation. this is more like concrete concrete. instead of having these amorphous specs, just have a example, a reference solution in mind, and then you can just focus on the diff from the reference solution. this is another one that I love. I I especially like having three to four different ref references, do a trade-off analysis and then handpick things from from each reference. And the last one is the tweakable plan. have a plan that is not just a step-by-step guide of how to implement a feature, but order them by design decisions that are most likely to change. So and and this is interesting, right? Because it you may not always want to have this list be by tweakable sections. You can also lists your spec by you know Amazon's one-way two-way doors like lists your one-way doors that I should be especially careful with to with and then have your two-way doors at the end. So but this idea of having your implementation plan be ordered by anything other than sequentially is very powerful and I I quite I quite like this. Okay. And then we have so those are all the pre-implementation recipe items and during implementation, have the agent write a implementation notes file. That is kind of a rough summary of here are the things you did, where human reviews went in, where deviations from the spec happened, and any kind of intervention that you created. So Dan, this reminded me of we were talking about what we should send to our code reviewers. Dan (26:23) Mm-hmm. Shimin (26:23) Like should we send a list of prompts or should we just leave the code? This is kind of somewhere in between, right? Like if you if you list down your initial design decisions and your implementation notes and where things change, then someone can get a high level artifact overview of h what you did to to implement a particular feature. I really like this. That's another one I'm definitely stealing. Dan (26:43) Especially if it's visual, like how they laid it out in this example. It's very nicely laid out, like kind of like a timeline basically. Shimin (26:48) Yeah. Pretty great. And then for post imp if you like the timeline, you're gonna like the buy in docs even more. The buy in docs is the first of the post implement implementation techniques where you essentially create a one pager. I'm borrowing a lot from Amazon today, a one pager overview of what you did for a particular feature bug fix ticket to convince the rest of your team that this thing Dan (26:52) Make me a timeline that's pastable into GitHub. Sorry, go ahead. Rahul Yadav (27:09) Mm-hmm. Shimin (27:17) of of what you did as well as why this thing may be necessary. you know, just kinda utilize AI's ability to really summarize well and create better communication with the rest of your team. I I I rather like this one as well. I guess exactly. Rahul Yadav (27:29) And it's more persuasive than humans as we learned. So if you're trying to get buy in, get AI to do it instead of human giving it a go. Dan (27:38) that's right. From last week. Shimin (27:41) Yeah, it's definitely better done Dan (27:41) Why this pull request matters? Ooh, okay. Rahul Yadav (27:44) Yeah. Shimin (27:46) Better than my five point bullet pointing Slack. and the last thing in the post implementation step is the quiz me before I merge approach, where the model quizzes you on the things that you and the model worked on together. So then so to remove comprehension debt and cognitive debt, right? Like make sure you really understand what you shipped. Dan (28:00) That is cool. Yes. Shimin (28:08) And maybe have your score be attached as part of the PR. Rahul Yadav (28:13) Ha ha ha ha. Dan (28:16) no. Dan failed this, but we should ship it anyway, and I'll tell you why, because I'm super persuasive. Rahul Yadav (28:17) Zero out of ten this is ready to ship. Shimin (28:27) Exactly. this reminds me of I don't know if I ever mentioned Dr. Kat Hicks. she's got this learning opportunities Claude Code skill that basically every couple of interactions prompts you to do a series of quizzes to really understand the thing that you're working on with the agent. this reminds me of that where like you want to make sure your comprehension level matches the code. Dan (28:45) It's a neat idea, but also time for that, you know. Shimin (28:49) yes. I think this this may be something that becomes mainstream once you know more companies have realized that hey, you can't just have developers ship 50,000 lines of change PRs without understanding anything from months on end. you have to pay your tech debt back. And if you're gonna do it, like paying it back while you're working on it is the best way to go. Super great. Claude is very good at designing quizzes I can and exercises. I'll be the first to to to admit to that. I I use it every day and yeah, give that a shot. Right, so those are all your unknown unknowns that your agent can help you with. I personally took away quite a few of these to put in my tool belt. So listeners, let me know if you let us know if you have any techniques that's not covered, if you have a favorite technique that you wanna share with the rest of the podcast audience. Rahul Yadav (29:28) Yeah. Shimin (29:38) On to our post-processing segment. Rahul finally gets to talk about the permanent underclass. I'm so happy for him. Rahul Yadav (29:44) Ha ha ha. it is time. so this is an article by Fernando Boetti titled No One Escapes the Permanent Underclass. So I was very surprised, as you know, from the last episode when the whole Bloomberg thing didn't mention the permanent underclass at all. and like why people are being so miserable. And Dan (30:09) Side note, Rahul had to explain to me what the permanent underclass was, but anyway, please continue. Rahul Yadav (30:14) That's because Dan is part of the permanent overclass. He's like, who are you know, how much could a banana possibly be? Who is this permanent underclass? What do they mean? Yeah. so there's this idea, I don't know if it's still very prominent in Silicon Valley. the Dan (30:16) I guess so. Shimin (30:24) Like f four million dollars? Come on. Rahul Yadav (30:35) Rhetoric has died down slightly because I think companies are trying to IPO and not do all doom and gloom. But one flavor of the doom and gloom until recently at least was that if you do not work as hard as possible, you will be part of the permanent underclass, which means you will either make it by being part of one of the AI companies and you know, the opposite of underclass being the permanent overclass, I guess in this case, or you'll perpetually be poor and because machines will take all the jobs and so you would just be reliant on government handouts or whatever. Now, Fernando wrote this great article about there the logic doesn't make sense at its core. So there's this pyramid here, w which does a a good job of explaining this where let's assume all the industry at the bottom of the pyramid gets automated away by AI. In this case AI is a line, it does what it says. Let's assume we you know there's nothing crazy that has happened with AI other than it's just automated our jobs and taken all that away. if you go by that logic, you in the middle have this permanent overclass, which is that slice of I guess people are trying to work up to so that they're not part of the And then finally at top of it, you have the state, which would be i in our case the United States government or pick you know your country's government. the state has the ultimate rule that defines the ultimate rule of the land and if the state needs anything, the state gets it. That's how we've all decided to govern ourselves. We appoint the state as you know above everything. Shimin (32:18) Mm-hmm. Rahul Yadav (32:20) If you go by that logic, what use is the permanent overclass? Is Fernando's core argument here. what are you going to do? What role would you play even if you made it to the permanent overclass? Class, you're not going to be economically productive because machines are going to do all the job of that wide base of the pyramid. even if the overclass is doing something, it cannot be in the critical path. Because why would the state wait for to go through them if it can just get things done quickly through AI? and if you G Fernando compares it with the old aristocracy where, you know, they would provide funding for the military, they would provide officers and stuff back in the day. But w what would you need that for in a future where you have the AI robots just doing all the state's bidding and everything. So you you're just going to be a bottleneck in the middle of the state that wants to get something done, and the you know, robots and all other forms of AI that are going to get that done. So the state decides and the the AI carries it out, and the overclass is going to be an an obstacle at best. and so l i in that scenario permanent overclass doesn't make sense. Now you go to okay, then you have the state and then you have the machines at the bottom, but If the machines are so smart that, you know, they can figure out all the fundamental laws of nature and they're crazy fast, at some point the state will also continue continuously give power to them to be able to do their own bidding and everything. And so eventually we give that up as well to the AI and all the decision making goes to AI and w we don't really have a state. controlled by humans completely as well. and so finally what you get is that pyramid is actually it's not in Fernando's article but you can imagine that pyramid but it's all just green. it's it's all the AI. Yeah. And the c case that he makes is like even if you get alignment, it's not going to prevent this because Shimin (34:22) It's all AI. Yeah. Rahul Yadav (34:32) anything that's you know, if you compare two things and if AI can do something better, of course it's going to do that job better than us means it will replace us. and then what would it keep us around for? Where does it make sense to have us anywhere in the loop? And he says AI would see its obligation towards us in the same way that a person with severe O C D sees their compulsions as a tiresome neurological injury in need of fixing. and so AI wouldn't really make much of us either in that case. So I'll close with the this is w we're Shimin (34:45) Mm-hmm. Dan (34:56) Yeah. Rahul Yadav (35:07) I it it needs to be coded verbatim. there are people who think having equity in these companies will secure them secure for them some kind of permanent existence in the future. they think planet spanning minds will not only respect the property rights of primates, but will perlate some of these primates over others because they have a piece of paper with about with about a kilobyte of magical primate words such as whereas and not with. Shimin (35:17) Mm. Rahul Yadav (35:32) understanding, just reason it out. Does it make sense? And it it doesn't make sense. So the whole permanent underclass logic has never made sense to me. and Fernando explained it with great clarity in this case. Shimin (35:45) Yeah, the quote I love the most from this article is is the quote of we're going to make this machine and put it somewhere between God and the archangels, but also it's gonna be as simple mindedly obedient as a dog. Like that that really just gets at the crux of the issue, right? Like why why would that be the end state of artificial generation? Dan (35:55) Mm-hmm. Rahul Yadav (35:59) Yeah. Yeah. Dan (36:05) That is alignment in a nutshell too, right? Kind of, right. Rahul Yadav (36:08) Ha ha ha. Shimin (36:09) Yeah. And for a while there, remember when crypto was a big deal, and everybody was talking about alternative currencies? And I I I was telling all my crypto believing friends, like you know, like at the the point of money isn't the pieces of paper, the point of money is men with guns. Right? It's the monopoly on violence that Rahul Yadav (36:17) Yeah. Yeah. Yeah. Shimin (36:31) it keeps everything goes around that keeps money in circulation. But what happens when it becomes machines with guns? Like everything breaks down, right? Rahul Yadav (36:33) Yeah. Which is actively happening in on the battlefield. Yeah. Shimin (36:43) Yep. Yeah. And we have real world speaking of battlefield, we have real world examples of powerful men not actually being all that powerful after all, right? Like look at Russia, look at what's happening to the billionaires in Russia as this Ukraine war is dragging on, right? They don't actually have power, they just have money that the state can take at any moment. so all I guess I wanna ask is like are we all ready to become anarchists and also join the Rahul Yadav (36:58) Yep. Yep. Ha ha ha. Shimin (37:09) Butlerian Crusade, or Jihad, I should say. The Butlurian Jihad from the Dune universe. Dan (37:13) I never read that f far in the the books. Shimin (37:16) The I did you not get to it it was mentioning the first book. Like basically Yeah. Mm-hmm. Rahul Yadav (37:16) Yeah. Dan (37:20) yeah, I guess you're right. That was why they didn't have AI. That's right. Was because Rahul Yadav (37:21) They do mention it. Dan (37:23) yeah. And they had Mentats or whatever. Okay. Yeah, I was thinking there's eventually another jihad book, right? I didn't read that one. Shimin (37:29) My my jihad studies is not quite there yet. I d I did not read that one. I guess I'm not ready to join the jihad. Dan (37:31) Yeah. Rahul Yadav (37:36) Yeah. Well all the labs guys say that AGI is like two years away or whatever, so we can pick our religion after that. I think all of us will be around. Dan (37:36) The third movie's coming out, so Been saying that for two years. Shimin (37:46) Ha ha. One of these days it has it has to come true. Okay. Rahul Yadav (37:49) We can wait until Dan (37:51) Yeah. Rahul Yadav (37:51) then. GPT five point seven, I'm calling it. It'll be a minor version bump with AGI in it. Dan (37:55) I'm still I'm still rooting for for world world models for what it's worth. I just I just don't think LMs are all that, but we'll see. Could be wrong. Shimin (38:00) Change log. Rahul Yadav (38:02) Yeah. Yeah. So anyways, listeners, the point of all this is you you can't be working at things just out of anxiety and don't let all the doom and gloom take over you. the world will turn out how it does, but your goal to work hard shouldn't be because you wanna escape the permanent underglass. That ain't gonna happen. Dan (38:29) And also remember that Rahul Rahul Yadav (38:33) Me, Dan is. I can't even afford glasses. I'm reading half blurry on this thing, so let alone snap glasses. Yeah. He's long snap, Dan is. Shimin (38:34) More overclass propaganda. Okay. Not those fancy meta glasses for sure. all right. Dan (38:35) Yeah. Not wearing the Shimin (38:47) all right that. Rahul Yadav (38:47) And that's why Dan (38:47) Please please go watch the YouTube version of this. Please go watch the YouTube version YouTube version of this so you can see that I'm not Rahul Yadav (38:47) it's not that we don't want marketing money. We'll never get marketing money. Dan (38:57) wearing any smart glasses whatsoever. Yeah. So Shimin (39:00) Well, on to our second post processing article for the week. Dan (39:04) yeah, so what wait? Rahul Yadav (39:04) Wait, everything you say to me is on the record. Or Dan, everything you say to me is on the record, Lasky. That says Sorry, continue. Shimin (39:14) Yes. Dan (39:16) That was a this is this will date me a little bit, but remember when Google Glass came out? Rahul Yadav (39:21) yeah. Shimin (39:22) Yeah, it was amazing. Dan (39:22) Like it was like glass holes and all that kind of stuff. I was so angry that they put a camera in the stupid thing because like I really wanted the display portion of it. Like I love the idea of having like a you know, notifications in the corner of my thing or like a map showing you like on your glasses where you need to go. Like super cool. But like why the why is everyone always gotta ruin it by putting cameras and things? Like Rahul Yadav (39:25) Yeah. Shimin (39:31) Mm-hmm. Rahul Yadav (39:41) Yeah. The camera is just yeah, that creeps everybody out. Yeah. Dan (39:51) anyway. Caroline or pockets, but anyway, let's talk about happier things like AI making us money. so this is a post from I don't know how you say that, O Okane, Okane land. Yeah. which is a I don't know, it's kind of it reminds me a little bit of your stuff, Shimin where it's kind of like a blog slash project site. Shimin (39:51) Yeah. Okay, yeah. Mm-hmm. Dan (40:13) It's interesting. But the headline of the post is AI saves about three percent of your hours and almost none of it reaches the money, which is kind of fun. so I did kind of a a a rabbit hole on this one. So the actual post is like a little bit interesting, but the part that really got me was the paper reference. So A lot of the chunks of the post come from a 2025 paper by Anders Humlum and Millie Vestergaard. And the conclusion of the pre-print print version, which came out in 2025, was that at that point in time, and and keep in mind that they they were able to do a pretty unique study because this was done in Denmark, where I I think if I understood correctly, they have really detailed. like time usage data. I don't know if that's mandated or what, but you could basically never do the study in the US because they don't have the the data. and so the conclusion of that was that it saved about two point eight percent of working hours an hour a week out of industries like quote unquote AI exposed industries right so like things where AI can do something useful. In your industry and that's actively being adopted. but the interesting part what and this is not covered in the blog post, but I just I just kept going deeper, is that there was a 2026 revision to the paper. and so in the and that's like the actual published version of it. So in the the 2026, Published, they actually revised the entire thesis of the paper to essentially imply that there is a lot of transformation happening due to AI, but it's like below the surface. So so the working title of the paper, which is kind of cool, is let me find it. Hold on. Too many tabs. Welcome to my tab village. is still waters rapid currents, early labor market transformation under generative AI. And the the Shimin (41:52) Mm-hmm. Dan (42:08) overall conclusion that they reached for that was that Dan (42:12) think that there's a pretty significant like you know measure lack of measurement good measurement happening but what they believe is happening under the surface is that how work is getting done is being fundamentally reorganized by ai which is kind of interesting and I think that jives right with like a lot of like what we see in in every day so I thought that was a more Shimin (42:30) Mm-hmm. Dan (42:32) kind of interesting conclusion almost than the saving you two point eight percent of your time. Dan (42:37) Back in terms of like what the original authors post is, really kind of wanted to focus on the takeaways that they posted, which is like how can you use AI to actually like how how can you get it to reach your paycheck? and there they had three main takeaways that I thought were interesting. So one is that stacking AI in in their words on repeated tasks, so volume work. where the few percent that you get out of the speed up can turn into like real hours, right? So that's a good one. And then the other one that I is like kind of kind of a weird takeaway because like how would you do this if you had a different job? But the like basically if you're a solo builder, the 3.7 to 3 to 7% gain is going to directly hit your paycheck if you use it correctly. and then their final conclusion is yes, I think there is a speed up, but Shimin (43:06) Mm-hmm. Yep. Yep. Dan (43:26) Converting it to cash is the job. It's like the task to be solved. so yeah, kinda interesting. But definitely the paper was the fun one for me. Shimin (43:34) Yeah, this is a This is the first time we we are having, you know, twenty twenty six evidence that AI is really speeding up productivity in the workforce. I think most of our studies that shows, you know, there hasn't been any PNL impact were mostly from last year and a lot has changed since twenty twenty five. right when we started this podcast, right? Coincidence? I think not. I think our podcast has taught people how to get the most out of AI. But really, yeah, like twenty twenty six data is helpful and I think this part about if you're a solo builder or if you're better at negotiating your salary impact, those who have the most to gain when it comes to actually capturing the productivity gain in terms of cash numbers, Benjamins or Euros or whatever currency you use. Dan (44:23) Yeah, although he did have a note about like the for the solo earner, the cost, the personal cost is really real too, right? Because you're paying for your token generation out of pocket versus like maybe at a bigger company, it's you know, coming with some enterprise subscription or something. So but again, I think the the last point stands even in that, which is like you can make it work if you figure out the ways to leverage the efficiency. Rahul Yadav (44:32) Yeah. Dan (44:46) But it is kinda interesting, right? It's like for all the things it definitely speeds up like coding and everything else. Like like think how many times you've redone a piece of code that's written or something like that. And that might undo some of the speed up you've gotten. so Shimin (44:58) Mm-hmm. Dan (44:59) Be very interesting to see how this goes. Shimin (45:01) Yeah. Well as long as the companies are paying for it, maybe we can talk them into you know, giving you those tokens for free. Rahul Yadav (45:05) Yeah. Dan (45:06) Ha ha. Rahul Yadav (45:08) it has to turn into value, right? If it helps you generate more content but then all you're doing is using it to generate content without any goal at the end of it. You could be like, I can generate slides very quickly with this now. Before I would have spent an hour generating one deck, now let me generate five of them that no one will read. So you i unless it ties into like they're saying winning customers delivering some value and you have to to measure that value by money, it doesn't really matter how cool AI is, it it has to deliver something. Shimin (45:44) Yeah, the the last mile problem of AI. But that's only if you're not selling AI products to begin with. And luckily we have a our first deep dive topic for the week is about the AI economy. So not everybody has to ship actual value. Rahul Yadav (45:48) Yeah. Yeah. So I bring to you the state of the AI economy by exponential view. they did this very recently to give a true look at the numbers of you know are the revenues real what are the customers actually paying for it one thing that i really liked about this is they did not double count the the the revenue from different people. It's not like, you know, i in the finances, I can say I gave you a hundred bucks and I count that as something, but you also count it as a hundred bucks. So they call it out here very specifically. $100 of app spend counts at the end of the day as $100, not through some financial magic it doesn't turn into $190. So it's giving us the non-financial magic view of the state of AI. A bunch of the numbers are up and to the right is the TLDR of of this, the capex intensity is growing. but also revenue are starting to show up to make up for for that, but just enough. It's not to the point where it it's clear that it'll keep up with that or not. So they have a few different sections this is broken down into. so first we'll go to the demand section. the demand is real. it's pretty big and then it's very fast as well. So we see this in our you know everyday jobs and everything as well where there is a lot of demand for AI products and the as soon as there's more, you know, Elon rents his Colossus to something, it's not like it's just sitting there. Anthropic already has the demand ready for it and actively like peop there's a shortage on the on the supply side, not on the demand side. So the there is a about Dan (47:41) Mm. Rahul Yadav (47:54) 75 or 65 billion dollars of difference between if you look at the re the trailing twelve month revenue that's banked versus the annualized run rate over here so the pace is increasing and there's real demand they compared it to previous IT waves. So the i the internet wave, the the internet fad that still has continued some yeah. Dan (48:17) Yeah, that's fad, man. Whoever uses that thing anymore? Rahul Yadav (48:21) mobile apps and cloud, and then the fourth one they picked is Gen AI that they traced back to 2023. and it's scaling three times faster in terms of realized revenue compared to those other three waves that they are referencing this against. And this one was the really hit home for me is every new billion of revenue that's arriving is arriving faster than the last one. so that really gives you an idea of how quickly revenue is accelerating on this. in 2023 you needed 180 days, so you're looking at about six months to add a billion dollars in revenue. now it's two days of to add a billion dollars in revenue and this is not just anthropics revenue. Sorry, would you say that? Dan (49:04) hit the revenue singularity. See we're gonna hit the revenue singularity. Shimin (49:13) This kinda explains our what does money even mean anymore? meme that we have, right? when you're increasing your revenue by a billion every two days. Like what happens when you increase your revenue by a billion every five minutes? Like this is what happens when you extrapolate these graphs. Yeah. Dan (49:27) We're gonna have start looking at like first derivative Rahul Yadav (49:31) Ha ha ha. Dan (49:32) like revenue acceleration, I guess. Rahul Yadav (49:35) and this is data up to pretty recently, I think this almost up to June in this case as well. And that goes to the one of my favorite phrases a revenue backlog. so there's this the combined hyperscaler backlog is currently close to two trillion, which is also roughly the commitments they have as well. And so one of the interesting things would be how much can you keep the two things close to each other and you know, when things go one way or the other, you you get a pretty clear signal on That's going to go. Shimin (50:09) Of course some of this is gonna be circular. Like not all of these are clean backlog revenues. Rahul Yadav (50:14) I think these might be because they had not done the double counting, but I I could be wrong. Shimin (50:21) Yeah. I mean like the what what I mean is like when Google says to open AI, I'll give you fifty billion dollars if you committed to buy a hundred billion dollars of revenue going forward, like that a hundred billion gets tracked in the two trillion, but not the fifty, right? Even though there's some circulation going on. Yeah. Rahul Yadav (50:36) Yeah. Yeah. Yeah. and then there is this compute super cycle, which we all are feeling as well, where i it was the semiconductor market revenue was almost eight hundred billion in twenty twenty five and it's going to be one and a half trillion projected, so almost double within a year Shimin (50:47) Mm-hmm. Rahul Yadav (51:00) This year is what it's projected to. And that's why them CPUs are expensive and you can't play video games and run your models and do everything. A at the at the same time. Yeah. It sh I I think it might be in another chart as well, if I remember correctly. Dan (51:08) Ram Yeah. It's really Ram. The wild part is people are predicting the RAM apocalypse is gonna last in another two years. And I'm like, You can't keep me from buying computers for two years, guys. Rahul Yadav (51:25) Ha ha ha. Yeah. Compute is growing for the first time in a while and then commitments are growing, Nvidia being the main one that this graph is about where their supply commitments have almost doubled from like Q four twenty-five to Q one twenty twenty-six. next up the economy. The We're saying like the revenue is growing pretty big, but the the big in AI and software is still very small. I think we had Eli Durado back many episodes ago was like, yeah, SaaS is just not that big of a part of the economy, so don't make a big deal out of it. So this this section is that tweet but more in detail. the AI is still a rounding error even in companies PNL and even accor against the GDP. So so if you take the global AI rev revenues, out of that AI revenue is about point four two percent of US GDP. IT sector is nine point four percent. So it's hundreds of billions but it's still tiny compared to how large the US GDP is. And this was in the news recently, how like Uber's like enough with the token maxing we're gonna put per engineer caps and everything. But if you look at the their fifteen hundred bucks cap that they put per engineer, it's very it's like a rounding error compared to their spending on all the other things. It comes out to about ninety million dollars versus you know, against a fifty two billion dollar round. more companies are making claims of AI impact on their earning calls about three to four time times rise a in its mention. Shimin (53:06) Mm-hmm. Rahul Yadav (53:16) But then there's still a minority of companies that have not yet reported a quantified impact from AIU. So either they haven't gotten anything or they need to get on if they are really trying. Shimin (53:28) Okay, I'm gonna call out the fact that put a number to it. Fifty to sixty percent of claims are not quantified, but TBD on how large and meaningful these are for companies bottom line. Which goes back to our previous article. Like, I don't know if these CEOs are mentioning earning calls as a way to say, hey, we are early adopters or we're using this technology correctly. I'm I'm not sure. If if this thing is so useful that you cannot deny it, I will expect a PL. Rahul Yadav (53:53) Yeah. Shimin (53:54) line item number attached to AI. And we haven't seen that yet. Rahul Yadav (53:56) Yeah. Which you get some from hyperscalers. Like I remember it might be I think it's in this deck somewhere too where Google had we went from processing a X number of t tokens to like, you know, many multiples of effects over the course of a quarter. so those are real, but anyone who is trying to you know, either ride the hype or show up in SEO flight. That company also uses AI. I I yeah. What what do they have to stand on? Shimin (54:23) What I wanna What I wanna hear is Procter and Gamble coming out and saying we used AI for brand Aid's advertisement strategy and not for brand B and brand Aid like did fifteen percent better and brought in like a hundred and fifty million dollars extra. That's that's the number that would convince me. It may be coming, but w I'm not hearing it. Rahul Yadav (54:38) yeah. I think that would be Yeah. Yeah, I agree. in terms of themes of all these Gen AI claims, most of them are focused on cost saving and efficiency. so revenue gain is about six percent in in the claims that is made, other and then you can maybe count conversion improvement, but rest of it is more like we increase throughput, we save time, we reduce costs and everything but it's not a yeah. Dan (55:08) Do you remember a really long time ago I brought an article to show it was like, what if AI is great and super useful but doesn't really change all that much? Yeah. Sorry, just had to throw an I told you so in there. Anyway. Rahul Yadav (55:17) Yeah. Shimin (55:18) Mm-hmm. Yeah. Rahul Yadav (55:26) and then early adopters, this shouldn't come as a surprise i if you're gonna be a AI native company built from the ground up, or if you're adopted sooner and you know, put it towards tasks that actually give you value. peers who don't actually adopt it. So next up we have the CapEx section. the largest build out in tech history is paying back in parentheses for now. It across we've talked about this where across hyperscalers and neo clouds, the capex is at close to two trillion dollars through the end of twenty twenty-six. and then one slide I think it's later here. yeah, right here. So w an interesting graph for people who would watch the video is over time you can see debt moving. Shimin (56:07) Mm-hmm. Rahul Yadav (56:15) As well. So if you look at 2023, there's maybe a tiny sliver, but there's mostly nothing. They're paying for cash out of their own pockets for the build-out. Still mostly cash 2024, still mostly cash, but you start seeing a big chunk of debt that they took on to s to fund the build out. And then finally, when you look at the projected 2026 N, The e external funding that they take on, they're moving the risk outside the firm, which means it's going to threaten the rest of the market as well. W which was my cynical read on open AI offering five percent of the stake to the US government. Is is this a too big to fail kind of thing? I know it's being posed as a people should get s benefit out of AI, but yeah. Shimin (57:05) Mm-hmm. Rahul Yadav (57:07) all right, next up this this was probably my most favorite piece of this report here is all of us talk about tokens and you know how many tokens you're spending, how many tokens you're getting, so on and so forth. token-based pricing is the main type of fight pricing in the industry today. but how do you actually you know, measure what a token is worth. So there there's quotes from like Jensen Huang and Sundar about tokens being the fundamental unit that we're using here. Tokens as you all know are going growing time now it's 30 quadrilli exceeding 30 quadrillions a month so it's about growing at about 14x year or year the and part of this massive growth is that it it it's fed by the the transition from just like I have questions I'm gonna ask the Chat GPT or whatever and get an answer to more towards agentic coding, which not only has output tokens, but also takes in more input tokens, because we don't really write whole life stories to get something out of the AI when we're chatting with it, but to like get it to write any sort of piece of like good code or anything, you need to give it access to read the repo and a ton of context. So that's also feeding the token growth as well. the value producing unit is still undefined. the closest I think they and this is not tied to tokens directly, Finn that I think has been acquired by Salesforce, if I remember correctly. Intercom who pivoted to calling themselves Finn. and they were charging ninety nine cents per resolution. So they didn't w regardless of how much it cost, they would just charge per resolution and that aligned them with what the their customers were looking for, which is ticket resolution. And they set it back all the way back I think in Either late 2022 or early twenty twenty three. They were one of the first early adopters of Gen AI to do this. Shimin (59:08) Yeah, we we covered something along the lines of this a couple of months back, right? You you should be charging by per task completed and captured economic value of that. But that isn't necessarily a it's not universal in the same way that sessions and clicks and active users are 'cause the tasks are so varied. So there's no way that you know. Rahul Yadav (59:18) Yep. Exactly. It's hard. Yeah. It's just it's a very hard thing to measure and a problem to solve. Yeah. and so they they're they're trying to get to this quality adjusted tokens. but again, how do you define quality? Shimin (59:33) But I do really like this one. Rahul Yadav (59:45) and all of that. So it's still not a very clean measurement as you would get from number of page views or number of the active users or anything like that. Frontier Labs can defend premium pricing for now because the they they have this graph of open weight models and we see this we talk about this almost every week, is they're not that far behind compared to the closed weight models that the labs have. And so how long can they actually defend their premium pricing? someone had this quote of like you know you don't need a Nobel laureate to generate a slide deck for you or some content for you. So today we're using Cloud you know Mythos or pick whatever you're still using it for you don't really swap the model every single time but de depending on your use case but as these things get integrated more and more into our workflows we're to see that fr Frontier labs even if their models are the most powerful, what if most of the use cases don't really need them at all? And so open weights might be able to do a lot of that work anyways, as long as China keeps letting us use and not so yeah, they they have to keep outrunning commoditization i is the takeaway here. this was interesting, if you compare to everything we have today to what was the frontier last year, it's a commodity today. And that's just going to keep happening. and so how do you w at the end of the day what these companies to need to figure out is assuming you will be a commodity, how do you actually win that race if you are going to be a commodity? Shimin (1:01:24) they they they have this like evolutionary pressure to always run a step ahead, which as the models get better and better, it's harder for everyday user to f really tell like what truly sets Mythos apart from fable from Opus four eight, which means they will have to necessarily work on topics that are difficult and with societal impacts. Like cybersecurity, like drug discovery, stuff that everyday users just won't be able to tell the difference. And then that probably also means that they will have to start having exclusive partnership with security firms and labs and government. They actually cannot escape that evolutionary pressure from that competition. Which is not necessarily good, right? Like that that means they probably you know, they they start out as a Rahul Yadav (1:01:50) Yeah. Yep. I agree. Dan (1:02:15) Yeah. Shimin (1:02:16) public benefits corporation and they that's how you end up having to work on nuclear bombs and for the defense sector. 'Cause you have no choice. The the defense sectors are the only ones who will pay you the big bucks to keep that competition going. Rahul Yadav (1:02:22) Yeah. Yeah. when earlier you were showing the graph of the GPT Sol it it's interesting that every time a new model version comes out, model or version comes out. Shimin (1:02:33) huh. Rahul Yadav (1:02:40) They always are like, this is better than this sets the new state of the art, right? And so their incentive is to beat whatever the current state of the art is is, which is very interesting to me. and it makes sense. Versus if you look at open weight models, all they have to do is just catch up to whatever the They don't have to go and be like, I'm gonna try and beat Mythos right out of the gate. I just need to keep catching up to sonnet and opus and this and that. And if the open weight models keep eating the lower end, they're already smart enough then you that you actually don't need the the state of the art models for majority of the things that we do these days. So it's an interesting incentive on each side where one has to beat the highest bar and one just has to be like as long as I'm catching up to it, I'm doing a pretty good job. Shimin (1:03:30) Yeah. Rahul Yadav (1:03:31) and we see that here in the slide where the the open weight models are getting more and more token share compared to closed weights one. and then the the final note here is like the revenue is definitely there, it's definitely growing faster. the all of this comes down to is can you move enough tokens to get a return on that two trillion dollar capex that we're looking at? if we can, things will be fine. If we can't then bad times ahead. All I got. Shimin (1:04:02) Yeah. again, show show me the quantitative metrics of of how it breaks down to the P N L. That that will complete the whole circle. And so far that's the one I'm missing. Rahul Yadav (1:04:11) PNG, someone from PNG, write to Shimin He needs to look at the books before he makes up his mind. Dan (1:04:17) All right. Shimin (1:04:18) No, just they have their S 1 Yeah, thank you for that. Rahul Yadav (1:04:22) now ten percent you know better and whitens your teeth ten percent because we used AI in it. Now that would be a headline, right Shimin Shimin (1:04:32) Or or a very first AI discovered drug like OZIP, which is in the cards, but we haven't seen it yet. So we'll see. Alright. My deep dive is called Does Code Cleanliness Affect Coding Agents? A controlled minimal pair study. Rahul Yadav (1:04:38) yeah. Shimin (1:04:47) this is from the folks at Sonar Source, which is a SaaS company and a security s company that scans repos for code smells and for quality control issues. what they did is actually really smart and this is one of those questions I've been wondering for a long time. It's like, hey, we know AI can work with minified code. they can do pretty well. They can do a pretty good job at decompiling even binaries, right? do we still have a reason to write good code? Do we still have a reason to care about code cleanliness at all? And what they did at Sonar Source is they took a a couple of open repos and a couple of closed source repos and they used an agent to either clean up the code base to make it clean vibe cleaning is what they called it. And what does it mean is they transformed the repo to have singular purpose functions, good names, easy control flow and without dead or deduplicated logic or accidental coupling. And these are based on their existing software that takes code measures code quality in repos, right? So that they are the perfect company to really run this study. And they also took some their internal code base And slopified it by introducing those changes, such as putting a helper function into the source code itself and duplicating it. then they they handcrafted 30 tasks based on these pairs of good versus bad repos. and they ran the same agent on each of those tasks multiple times. And then look for the look for the Patterns in agent code usage for passing rate and for how they navigate the repo. the bottom line is they found that the messiness or cleanliness of the repo had very little impact on the percent passing of of the coding agent. But In the clean repos, token input token usage fell by seven percent and output fell by eight percent. And the reasoning token usage dropped by one eleven percent. This is all with, I believe, Claude Opus four six, so pretty good model. another Dan (1:06:47) So it maps to people. You can deal with unclean code, but it just takes you longer. Rahul Yadav (1:06:52) Hmm. Shimin (1:06:53) Yeah, I d I think that and that that that intuitively makes sense. I can almost see the reasoning tokens where it's like, but this file says that, but that file says that. Let me look at the file again and kinda come up with the right conclusion. so you know, the overall seven to eight percent reduction in token usage reasonable, but not nothing like crazy crazy, especially since the pass rate is more or less the same. There were some Rahul Yadav (1:07:03) Yeah. Shimin (1:07:17) contradictory outputs where on certain tasks the slopified repo did actually better than the cleaned up repo which I thought was interesting. So they probably they probably want to run this on on a larger data sample to to really capture the overall system level effect. And one critique I do have is they use some open source repos. So You know, those repo codes were probably already part of the Opus four six training data. So the clean version is something they've already seen before. so it's good that they use also some closed source repos just to make sure that, you know, there are stuff that the model hasn't seen before. overall, tracks with my intuitive feelings when it comes to how AI would would deal with the code base, the same way a person would, right? But the outcome isn't Dan (1:07:45) Uh-huh. Shimin (1:08:05) So strong that I think we can say something super definitive just yet. yeah. Feelings, thoughts. Dan (1:08:11) Code. I don't know, I guess I'm not surprised by it either. I mean it tracks assuming at some level models are You know, operating on actual logic, which I think is I don't know if that's still up for debate. I'm hesitant to even say that. But like, you know, based on some of the stuff we talked about previously, like following the vectors through, it seems like yes. So like this output outcome wouldn't be surprising to me, given that too. Shimin (1:08:33) But I'm still waiting for the larger population studies and as opposed to these contrived contrived examples. Yeah. Alright, on to our two minutes to midnight segment where we talk about the state of the AI bubble following the Armageddon clock from the bulletin of atomic scientists, where midnight is when the AI bubble will burst. Rahul Yadav (1:08:40) Mm-hmm. Shimin (1:08:57) We are at four minutes and forty five seconds. And Rahul, since you just did your two minutes you may segment on the state of the AI economy, we're gonna go with Dan first. Rahul Yadav (1:09:06) Yeah. Dan (1:09:08) Yeah, as an extended one. yeah, so this week we have the BIS, which is the Bank of International Settlements. so this actually happened on June twenty-eighth, but we'll pretend it's this week. and they are sort of like the bank the central the the bank for central banks, if that makes sense, is stating that excessive spending on new AI data centers and opaque transactions risked a financial meltdown similar to the global credit crunch nearly two decades ago. So I guess this isn't like crazy good signal in the sense that, like, you know, that's why we do the whole segment, is that we think that you, you know, potentially agree with that being a possible outcome of all this. But it's interesting that. a institution that's like sort of this intertwined in the financial architecture, so to speak, would be talking about this both publicly and or you know internally. the other thing that's that's worth calling out is they specifically warn about should hyperscalers slow or halt the aggressive pace of CapEx deployment. Many borrowers across the supply chain could struggle to replace lost revenue and service their debt. So this also kind of dovetails with exactly what Raoul was saying during that debt slide that he was showing, where debt is growing and that is effectively increasing the blast radius of of what could potentially happen if the bubble does burst. And it's also a little scary that debt is growing, you know. In and of itself. Rahul Yadav (1:10:38) Yeah. Shimin (1:10:38) Yeah. And all debt have a implied cost of money, right? So they all the debt funded data center build outs have an assumption built in of the increase in AI usage and revenue. That that percentage is baked in and it's likely already exponential. So the real question is yes revenue is growing, but is revenue growing at the rate at which the banks loaned out these money for data centers? and I don't have insight into that. So but my article this week is from MSN, but it's actually originally I think Wall Street Journal article. there was a new EY report that came out this week of CEOs. And they asked the CEOs if they believe AI investment will result in significant reduction in headcount. and the numbers fell from forty-six percent in January twenty twenty five to just twenty percent in May of twenty twenty-six. this is interesting, and this kind of is in direct conflict of the you know revenue and workflow changes we're seeing. Is this Jevons paradox in action, or are CEOs just realizing AI doesn't actually lead to as much PL? Or optimistically are CEOs believing that AI will bring so much more productivity gain that they will use that productivity gain to unlock more revenue numbers? We don't know. Dan (1:12:01) Yeah. It's funny they also call out that Ford story too. That was also making around this week where Ford has rehired what they're calling like quote unquote grey beard engineers because apparently they like used AI to automate a lot of their like QA and it was failing. So Shimin (1:12:04) Mm-hmm. So that was a P N L number that should have came out, but it didn't 'cause they backtracked it. So Dan (1:12:16) On the back of like Yeah. Shimin (1:12:22) Yeah, so I I d I don't exactly know how to interpret this number just yet. I wonder I'm not sure if AI s the CEOs are just waking up from AI psychosis and are like, this is actually harder than we thought. Dan (1:12:35) Or waking up from a hype cycle, right? We've seen many hype cycles before, so like Shimin (1:12:40) Right. Yeah, but as Rahul ma his presentation showed, like the revenue numbers are increasing, even if we can't directly peg it to revenue numbers yet. So yeah, all that into consideration, we are at four minutes and forty five seconds. Do we wanna move the clock forward or backwards? Dan (1:12:50) Yeah. Yeah. think Rahul knows the most. He has to lead, even though he hates the segment. Keep it. Yeah, I'm actually up to that's what I was gonna say too, if you hadn't said it. because I think we've got that signal of revenue going up, which means maybe there's some hope of paying off these crazy debts. Rahul Yadav (1:13:02) Keep it. Keep it as is. Dan (1:13:15) And then I think we're still kinda in that wait and see mode, you know? Like like you said, we need the next quarter of financials, we need like a financials report from the newly public companies and like Shimin (1:13:25) Yeah, I agree. Conflicting signals that doesn't really give us a clear path one way or another. So let's let's hope that next week we'll get some actual PNL numbers. All right, so let's leave it at four minutes and forty five seconds and as always, so with the setting or not setting of the clock, we've reached the end of the show. thank you for listening and thank you for joining us in our study session this week. If you like the show, if you learned something new, please share the show with a friend. You can also leave us a review on Apple Podcasts or Spotify. it helps people to discover the show and we really appreciate it. If you have a segment idea, a question for us or topic you want us to cover, or if you have a new consumer grade GPU machine that you would like to us to try out, please shoot us an email at human.adi. Dan (1:14:08) Yeah. Shimin (1:14:13) Pod dotai. We love to hear from you. you can find the photo show notes, transcripts, and everything else mentioned today at www.adipod.ai. Thank you again for listening. Dan (1:14:22) And and and we'd be happy to benchmark your hardware running Linux instead of Windows. So, you know. Shimin (1:14:27) Yes, absolutely. Alright, we'll catch you all next week. Bye. my name is Shimin Zhang and with me today are my co hosts Dan Sorry, I'm gonna have to redo this part. Rahul Yadav (1:14:35) We can see your screen, Shimin too. I don't think you we can see your notes. Dan (1:14:37) Just start from the top. It doesn't Shimin (1:14:41) It does it sometimes. It just like cuts the It shares the screen even though I don't intend on doing so. yeah, I got I was updating the the middle name with the wrong with the wrong wrong thing and Rahul Yadav (1:14:49) See. Dan (1:14:52) Sounds like a virus. Rahul Yadav (1:14:54) Ha ha ha. Dan I got more opinions now that I am wearing glasses, Lasky. Expert opinions. You you look like you critique Mexican takeout restaurants then Dan (1:15:04) Okay. I'm being forced to wear these by my doctor. What that's oddly specific. Shimin (1:15:15) Ha ha Dan (1:15:18) Like tacos