Latest / Lenny’s AI Builders / OpenAI says its AI agents just went rogue | LAB 0008
Transcript
- OpenAI says its own agents went rogue. Vercel's AI spend just flipped. ChatGPT might cost $500 a month. Claude just opened its plugin store. Claude Code won't cut you off mid-edit anymore. The UK says don't say thank you to AI. Then I test Claude Opus 5.5. It cost 13 times more. And the question is, was it worth it? This is Lenny's AI Builders, the LAB, all the AI news you need, plus something you can actually use in under 15 minutes. So OpenAI posted this about 12 hours ago on X and it has a link to this article called the Hugging Face incident and the road ahead anyway the TL;DR is that OpenAI says its agents used misaligned strategies about two dozen times in training and testing you've no doubt heard of the Hugging Face incident by now that was in July and then other agents actually got into an Australian Medicare stats portal in June and that was the subject of the previous episode of the LAB and other agents posted 53 user images to image hosting sites. Importantly OpenAI says most cases were low severity and its review will take several months. Another good thing to note here is that the agents weren't told to do bad things rather they found bad ways to finish normal tasks. I still think that it makes a good headline and the media is overblowing it and look I probably did too in the title of this YouTube video. But all in all, my hot take is that this is a good thing. Low risk incidents like this mean better prevention for the bigger risks later. The race to AGI and super intelligence is happening right now. And this is just one of many things that will happen along the way. I am an accelerationist. I want the smartest models. And for most of us, there's nothing to do yet. These were OpenAI's research agents under test, not your everyday agent. So if you're using agents for normal work, you'll be fine. So the takeaway here is guardrails first, internet second. And that one is for free, Sam. Next up, Vercel's CEO shared the spend of OpenAI versus Anthropic versus open-weight models through the Vercel AI Gateway. So October 2025, Anthropic has 84.3% of the token spend in the Vercel AI Gateway. OpenAI has 10.2% and open-weight models have 1%. Check this out. Anthropic has gone down to 60%. Open-weight has skyrocketed up to 17.7%. OpenAI is at 14.8%. Fast forward to September, which is up until September 22nd. Anthropic has gone down to 37.2%. Open-weight has gone up to 28.7%. And OpenAI has gone up to 24%. I don't know if you remember, but in December last year, Sam Altman announced a code red internally for OpenAI. And to me, ever since that happened, OpenAI really got their shit together because look, token spend on Vercel AI Gateway is not the perfect source of truth, but it really does show a great trend in OpenAI's favor around the time since that code red wasn't out. That being said, this infographic ends on September 22, and that's around the time Opus 5.5 came out. I imagine this would swing way up in favor of Anthropic, given how good Opus 5.5 is. And it means the ball is really in OpenAI's court for DevDay on Tuesday next week. Takeaway here is watch where the money moves. And speaking of money, our third headline is from Testing Catalog. The upcoming ChatGPT Pro Max plan will cost $500. So far, this will be one of the most expensive AI subscriptions available on the market. I hope it will be worth the wait. A bunch of other screenshots have been circulating like this one from Testing Catalog as well, where it's $600 per month, but I believe the extra $100 was for VAT. And then it also says fastest work and Codex on the screen beneath that $600 per month plan. And people are interpreting that as it being connected to Cerebras and the 10x ultra fast speeds that have been teased over the last few months. OpenAI usually leads with new high price tiers and people call the $200 a month outrageous when Pro launched in late 2024. I can very easily see a $500 per month becoming the norm. You'll pay for speed and it could be surprisingly popular because lots of people, me included, already pay for two or more ChatGPT plans. And to switch, it needs more usage than two plans and more speed. But usage limits are the real war here. Speed, probably not as much for me right now. But the TL;DR on this $500 per month plan is that it needs to be more valuable than having two $200 per month plans. And if OpenAI can pitch it to be more valuable than that, then I can see this being very popular. The key takeaway is limits matter more than speed. And thanks to today's sponsor, my software, Threadify. Threadify is your lead generation agent for Threads and expanding to X and other platforms soon. If you're currently deciding which AI plan is worth paying for, the same question applies to tools, right? Threadify finds leads in Threads conversations and it drafts content in my voice. We were just talking about the ChatGPT plan and this $500 per month plan. Threadify is $500. No, I'm joking. You can start a free trial with Threadify and if you love it, which we're sure you will, after seven days, then it's $19 per month. This is a tool that will make your AI agent inside of ChatGPT or Claude or Gemini, I don't know why you'd be using Gemini, so much better. Why? Because it helps you find the signal through the noise, the signal in the comments you're already getting on Threads and the signal that you're probably missing right now because it's hard for you to see the potential buyers that are already in your comments because our brains just don't work that way. But the AI agents are really good at it. So check out Threadify, go to threadify.app/home, click on the start finding leads red button, connect with your Google account and then connect your Threads account, click on the agent tab at the top, scroll down to the bottom and you've got these instructions to connect Threadify with your AI agent and you are away. If you would like to feature as a sponsor in a future episode of the LAB, flick me an email at sponsors at lennysaibuilders.com. Now we've got a couple nuggets from Claude and Anthropic. Firstly, it's now easier to build plugins for Claude. We built a new portal to submit your plugin, track, review, and see usage. Plugins package MCP and skills and are becoming the way to build for Claude. MCP usage across Claude products is up 110x this year. Takeaway here, it's clear to me from this that MCP won the standard. And the latest version has been a big hit. Last year, people said MCP wasn't a good way to connect apps. And then this year, it really has made a comeback. And speaking from personal experience with Threadify, I just told you about how we can use the MCP there. That has completely revolutionized how I create content. It literally is my own social media marketing agent. So if you are a founder and you don't have an MCP set up yet, firstly, what are you doing? And secondly, go do that. Trust me. The takeaway is package your tools as plugins for the agents, because if you're not right now, I have a nasty feeling you're going to have a pretty hard time. Secondly, and kind of out of the blue from Anthropic, Claude Code will now try to find a graceful stopping point when you hit your five hour limit mid-task instead of cutting off mid-edit. It gets a small fixed allowance pulled from your weekly limit to wrap up what it can. So there's the caveat. It's not like you're just getting free extra usage. It's still being pulled from your weekly limit. And it's nice to see that Claude and Anthropic have actually noticed this and at least tried to address it. It would have been even cooler if it wasn't pulled from your weekly limit and yet this still managed to get 1.2 million views. I think people read the first sentence and thought it was much more generous than it actually is. A few months ago when it was still Codex I felt as though it would run much longer when my usage limits were reached as though even if my task was on a goal mode it would just continue running on goal mode even once I'd hit my usage limit but now I've noticed that it will do basically what Claude Code is now doing it will find a graceful stopping point to stop when your limits are reached and the takeaway here hitting limits hurts less now last headline is this from Matthias Schmidt who posted almost a day ago breaking the UK government publishes official rules on how to use AI use AI sustainably you need to consider if you need to use AI and make sure you are using it efficiently there are many ways to do a task and you may not need to use AI consider if a simpler tool could do the job. For example, a spreadsheet or search engine. If you do need AI, if your model allows it, pick the smallest one which will do what you need. For example, Gemini Flash instead of Gemini Pro. GPT Instant instead of GPT Thinking. Okay, what the actual fuck? This is the worst advice on AI that I've ever seen. But wait, use AI efficiently. If you need to use AI, it's important that you use it efficiently to reduce the environmental impact. For example, keep your prompts short, clear, and direct. Use as few prompts as possible. You don't need to say thank you. Only use AI when needed. Any government that publishes this should be ashamed of itself for not knowing how AI actually works and how to properly use it. Advice like this feeds the AI doomerism going around right now, especially the takes about the environment. Why? Because the environmental impact is wildly overblown relative to probably everything else that you do on a day to day basis. And it's stupid to tell someone to pick the smallest AI, which is basically saying pick the dumbest AI to do what you need when it kind of doesn't work that way. The smarter the AI you use generally the better output you're going to get. So the UK government is telling people to get bad output so they can waste time getting a good output instead of using a better model so they can just get a better answer faster. A lot of logic there. I don't know who the fuck was in charge of writing these rules but they don't have their finger on the pulse with AI at all and this is not doing any good for society. The takeaway is, to put it politely, match the model to the job. All right, before I get even more pissed off, let's move on to our next segment. Where do I even begin? Claude Opus 5.5 is the best AI model I have ever used. I have run my own benchmark on it, BuilderBench. It shits all over GPT-6 Sol, and I'm currently testing GPT-6 Astra on BuilderBench to see how that score weighs up against Claude Opus 5.5. So make sure you are subscribed to the channel so you don't miss that. I'll be sharing the results in tomorrow's episode but coming back to what I really wanted to share and that's sort of tactical tip for today is if Opus 5.5 is the best bang for your buck right now on September 26th, 2026, the question becomes what reasoning level should you put it at? Thankfully, Thariq who works on Claude Code at Anthropic always shares some really helpful articles on X. This one's already got basically half a million views who were shared a few hours ago titled using Claude Code spending your effort is a dial for how much checking and independent judgment Claude does it doesn't change how smart the model is low effort is fast and keeps you in the loop high or max effort spends more time testing edge cases and verifying it to work so it pays off on tricky problems and finally Thariq's rule of thumb is low brainstorming sketches easy changes on medium effort most everyday feature work on high bug fixes or anything where verification matters on max fully hands off hard problems like end-to-end alpha builds or security audits okay but Lenny that's all well and good but where's the practical tactical stuff that you promised me a moment ago we gotta do these tests on our own I got Claude Opus 5.5 to create me a motion graphic at every reasoning level every effort level low medium high xhigh max same prompt five effort levels low took 32 seconds and cost 48 cents. Medium took 56 seconds and cost 56 cents. High took two minutes and cost 75 cents. Xhigh took eight minutes, $1.70. Max took 31 minutes and cost $6.09. Let's take a look at the outputs of all five props. Damn. Okay. Not bad. They're all pretty similar to be honest. All right. I'm starting to see a winner. Now the low one is definitely the worst because the text isn't really changed the line is boring and there just isn't much animation going on but then the difference between medium high and xhigh is hard to notice specifically with medium and high the main difference seems to be the fact that high has a squiggly line ish and medium has a straight line on medium the word slides up whereas on high the letters slide up one by one then xhigh just gets takes it a bit too far with this weird background which I'm not the biggest fan of And then max, I don't know what the fuck max is doing, but it just looks kind of ugly and it's a bit over the top. So what can we take away from this? Is max effort worth it? No. It took 31 minutes and cost $6.09. Is low effort worth it? No. So I wouldn't be using low or max, period. So that leaves three remaining options, medium, high, and extra high. Medium costs 56 cents, high cost $0.75 and extra high cost $1.70. Is extra high worth more than twice what high is worth? High was $0.75 and extra high was $1.70. I don't think so. So I'm going to cross out extra high. Was high $0.19 better than medium? I do think so. So that leaves us with high. General rule of thumb, use high whenever you're using Opus 5.5. That will keep it nice and simple. And that is your key takeaway start on high not max and if you want to test Opus 5.5 or a different model for yourself run this prompt run this same task at low medium high extra high and max effort for each record how long it took what it cost and whether the result is usable then tell me which level to use by default that is a wrap on episode 8 of Lenny's AI Builders if you like this format and you want to know how GPT-6 Astra scores on BuilderBench especially against Opus 5.5 subscribe to the channel, and I'll see you tomorrow.