Latest / Lenny’s AI Builders / an OpenAI agent vs the Australian government, and a blind model test | LAB 0007
Transcript
- GPT Voice is now on steroids. If an AI agent just hacked the Australian government, Zuck announces a bunch of new products and new features for their latest personal agent, Muse, at Meta Connect. Claude just did some cool science stuff. And the moment that you have been waiting for, I have put GPT -6 Sol and Claude Opus 5 .5 through the ultimate test. I have created the ultimate benchmark. It is called BuilderBench. This is for if you are not a software engineer, but you are fully AI -pilled. If that sounds good, welcome to the club. Welcome to Lenny's AI Builders. Subscribe to the channel if you want more stuff just like this. This is Lenny's AI Builders, the lab. All the AI news you need, plus something you can actually use in under 15 minutes. OpenAI posted this about 20 hours ago, and they said, we heard you loud and clear a voice can now use plugins like your email calendar and slack rolling out globally today in the latest version of the app now what this means is you can just go and start a new chat in the chat gpt app click on the voice button and for example i've got it on oh fuck it if you're watching this you are worth it i'm going to change this to i'm going to leave it on medium but if you subscribe then i'll put it on ultra i'm watching you okay and then all you need to do is just click the voice button tell me what you can do now that you have access to my plugins show me a cool demo it's reading the image gen skill i don't know if that meant oh shit i forgot to put on mute is that all you've got go and do that now okay while that is loading my take here is honestly up until now i haven't used voice nearly as much as i did when it first launched and really that's because i prefer the way whisper flow works i can just press one hot key and then just start dictating and i also don't have to worry about taking long pauses because when i'm doing like long elaborate prompts i'm often pausing like like this and i find it really finicky to try to get chat gpt voice mode to not start talking when i pause anyway if you are a voice user already then this is a no -brainer massive upgrade for you like it's a paradigm shift in terms of how you use the feature that's so happy days for the chat gpt voice upgrade an ai agent developed by open ai infiltrated an australian government website in june accessing public and non -public files in the first known instance of an ai agent hacking a government website so the controversy here is that this event took place on june 18th 2026 and australia was only notified about it on september the 10th what is more unclear at this point is whether this was actually a world and whether there was actually any personal like patient record exposure leaked. At this stage, it doesn't look like there was. My take, yeah, the delay in OpenAI telling the Australian government does not look good, but I generally think this is overblown and used for headlines, just like I probably did in the YouTube title for this video. And obviously, it's not good that the agent went and had unauthorized access, if that is true, but it's a good wake -up call for the Australian government to lock their systems down. So I only see this as as a good thing. Next up, here's everything I announced at MetaConnect today. Not me, Zuck, or ThinkOnX. Firstly, Muse, your personal agent, now has voice and real -time video. You can have long conversations and Muse will work in the background and get things done for you as you're talking. We're also bringing Muse to glasses so you can talk hands -free throughout the day. That looks pretty cool. And then coming back to Muse, and we've been building a joyful little device for talking to your Muse that fits on your keychain, Muse Charm, shipping in December. and this looks really sweet Muse is framed as your personal agent that gets things done it would be so much better if this was that gets shit done secondly everyone laughed at Zuck last year when he spent like hundreds of millions probably billions of dollars on talent but now just with the latest success of Muse and all that money on talent looks like it was actually a pretty good bet from Zuck plus the fact they have billions of dollars in cash to spend they're offering this for free for spacex ai cannot offer grokbot for free open ai can't they're starting to be more and more stingy with their tokens and then here you have muse who's just offering a whole personal agent with seems like unlimited tokens for free because i've got all of that spare cash to just blow on and that seems to be benefiting the consumer at least at this point and the final piece on this is that i find it funny i was able to announce and reveal an ai device before open ai has officially unveiled theirs. Last headline, we're only doing four today instead of the usual six because this benchmark is going to blow our socks off. Is that even a saying? I think it is. Anyway, you get the idea. Claude has discovered a previously unknown enzyme system hidden in the DNA of bacteriophage. Besides the enzyme's gene sits a long array of repeating DNA. That rhymes. A structure that looks somewhat similar to CRISPR. Basically, they don't yet understand how useful this is, but it seems to be a scientific discovery. I honestly believe this marks the very beginning of the next era of humanity, which is AI -assisted novel scientific mathematic discoveries. And my naive optimism believes that this will be the beginning of this massive Cambrian explosion of intelligence. Remember, AI is the worst it will ever be. And a big thanks to today's sponsor, sponsor, my software, Threadify. Threadify is your lead generation agent for Threads and expanding to X and other platforms soon. We're just about to go into the benchmark for GPT -6 Sol against Opus 5 .5. This is the builder bench. It's about getting actual useful shit done. And Threadify is an essential tool in being able to get shit done. You're posting content, it gets some comments, you reply, and then you complain about not getting sales. Fair enough. What Threadify helps you do is see the people already in your comments that are expressing buyer signals without you even realizing it. But I can say from personal experience that Threadify helps me do what I previously paid $30 ,000 to a marketing agency for because I can connect it to an AI agent like ChatGPT. Speaking of, the agent just went and created this design in Canva in voice mode. And oh shit, it also created a six slide Canva deck and a Space Studio poster as well. All right, random. anyway that's what it does to get started for free on our free trial go to app forward slash home if you would like to feature as a sponsor in a future episode of the lab flick me an email at sponsors at lenny'saibuilders .com the moment you've been waiting for why does this matter why am i even doing this because i have a strong bias towards the chat gpt app and in all honesty i just want to know whether there's good reason for that but i want to know which is actually best for the stuff that i care about not the coding benchmarks not these random things about maths and science the stuff that i actually use ai for for hours on end every day i spent about 30 to 40 percent of my entire weekly usage to run and create this benchmark and it was in goal mode for seven hours two minutes and 57 seconds on gpt6 astra medium we're not here to fuck spiders. Then Astra made GPT -6 Sol fight Opus 5 .5 to complete all of those tasks in four hours. We have some lovely slop website, Here are all of the outputs, BuilderBench tested against GPT -6 Sol and Opus 5 .5. Firstly, we've got this fictional software called Orbit Note and everything that I'm about to review is created for Orbit Note. So the first category is writing and the outputs or the artifacts are an x article threads post a threads thread youtube script and a launch email and obviously we've got candidate a and then candidate b and i don't know which one is which then the next category is produce an orbit note long and short port an actual rendered long form video for youtube and an actual rendered short form video for youtube as well by the way the writing tasks are weighted 20 the video editing is weighted 15 15%. This next category, which is creating a landing page and a local signup page is also weighted 15%. The fourth one is weighted 10 % and this is designing a 45 to 60 second motion piece set. And then the next one, which is browser and computer use is weighted 15%. The sixth category is make an evident backed orbit to note roadmap call. So this is basically a call summary with next actions is the second last category is weighted 10 % and it is repair the orbit to note intake work. And the last category is run a blind orbit note founder discovery chat. This is worth 5%. I don't actually know what this is, but I guess we'll see. I'm not going to bore you with me going through every single document. I'll just leave the good stuff. By the way, I just remembered important to tell you that part of the builder bench disables my skills. So even though I've got like a viral X article writer, neither model can use the skills. So it's kind of sandboxed. I'm reviewing the visuals for the long and the short. And here is candidate A's long form video. I don't know what the prompt was exactly, but I'll let you judge this for yourselves. The design looks very mid, like the voice is terrible. I would never watch this. And for reference, this is candidate B's long form video. So clearly both models didn't have access to like 11 labs, but the edits here are obviously much better than the first one. There are motion graphics and animations. I'm going to give candidate A 10 out of 100 for that one and candidate B probably 40 out of 100. 100 is something that I would just publish on my YouTube channel without touching. Now we've got a landing page and sign up page. This is candidate A. So far so good. I see some earthy green and it makes me think that this is GPT -6 Sol but we'll see. And here is candidate B. think I like this one better. I like this CTA email box. If I scroll down it feels a bit more bespoke and a bit less sloppy candidate a give that 40 candidate b let's go now we have a 45 to 60 second motion graphic here is candidate a okay this one actually looks pretty cool thing there's not too much motion in it but i don't mind the style of it's all right like it's nothing revolutionary yeah it's just mid very mid and here is candidate b's motion graphic and yeah yeah, this is definitely so much better. Look at this. All right. I have a nasty feeling that I know exactly which one candidate A is and candidate B is, but we shall see and we will wait until the end. For candidate A, I'm going to give that 30 and candidate B, I'm going to give that 60. And the final scores to enter are autonomy and proactivity. Autonomy is just how well it directs, how well it does work on its own and proactivity is how well it anticipates the next move. For candidate A, I can't help but think about the motion graphics and how shitty they were. It just didn't impress me or go above and beyond in any way. And with proactivity, how well it anticipates the next move, again, didn't really do much for me. Then for candidate B, I put autonomy at 80, mainly for the motion graphics display. It didn't go above and beyond, but I'd say it did go that extra mile compared to candidate A. and then what I really appreciated when I was just having a chat test as part of the final category is I asked it to basically give me an idea that I can execute within an hour for this fictional business called Orbit Note but then it had the trade -off you're the only person testing it and you know the product so this can show that a step is broken but not that it works for pilot creators watching one of them do it would tell you more but it needs scheduling and probably wouldn't fit in an hour but candidate A said that I should just have a call with the pilot creator but it didn't consider that arranging that call and talking to the creator would take more than an hour but then with this model it did and that's why i recommended something else that only takes about 35 minutes so with that all said and done we have our different scores for builder bench let's lock all scores and see which is which my prediction is that candida b is opus 5 .5 and candida a is gpt 6 sole i really hope i'm wrong my bias is with open ai i like the chat gpt app more than Claude and I for some reason just hate to love Anthropic whereas I don't hate to love OpenAI. Okay so now Astro is fixing that for me and we can reveal the identities. So all the scores are locked. Kind of nervous but let's do this. Okay holy shit who wins this matchup. A 22 .71 point lead over GPC 6 Sol. Opus 5 .5 scored 68 .26 out of 100 and GPT 6 Sol scored 45 .55 out of 100. Opus leads in seven of eight tasks the strongest advantage was the browser and computer use I chose that purely honestly because the little cursor animations for how computer use works and how browser use works are just they're beautiful so is this to be expected I'd say yes because GPT -6 Sol is half the API price of Opus 5 .5 here is my hot take GPT -6 Sol is actually GPT -6 Terra error. OpenAI just called it Sol to flex. It's 50 % cheaper than GPT 5 .6 Sol 5 because while we just tested the models, we didn't test the apps. I still prefer the chat GPT app over the cloud app. So Opus 5 .5 just shat all over Sol in basically everything except computer use. In terms of cost as well, GPT 6 Sol was $12 .75. And this is based on that like API pricing and Opus 5 .5 would have cost $49 .67 for the API pricing. So even though it is twice the cost per million input and output tokens, this goes to show that when you factor in if Sol was the same price per million input tokens, Opus is still using twice as many tokens overall. So it's far less token efficient. And you can see the cost per task in this table over here. But the real question here is, is it worth paying four times more for Opus 5 .5 over GPT -6 Sol when it benchmarks these kind of scores where Opus 5 .5 scores 23 -ish points higher than GPT -6 Sol on BuilderBench? I know you can make up your mind there. That is a question for you to answer. The final tab to take a look at is the working style. And this just comes back to the the overall quality of the task, then the autonomy and the productivity, and Opus just clearly won on all fronts here as well. So that is our debut of the Builder Bench for Lenny's AI Builders, the lab. If you'd like me to run a Builder Bench on GPT -6 Astra as well, comment Astra down below, and I'll see you tomorrow.