Latest / The Bleeding Edge Podcast / AI Models Are Getting Cheaper. Do You Still Need the Latest One?
Transcript
- Ralph: And with me in the studio today, Emile Loggier. Emile, how's it going on the edge? Em: Going great, looking very deaf to her off. Ralph: thank you very much. Just something I threw on. matches my my eyes. so I don't know what's going on out there. It's almost almost impossible. Maybe we need to give up on the news because it's just a deluge. And if you have a look at some of the folks out there on the social media, they're just like, yeah, stop looking at the AI news, it's not worth it. but yeah, I don't know what you think, Emile. there are so many fundamental shifts and changes going. If you don't stay on the bleeding edge of what's going on, I feel like you miss so many opportunities to build better solutions, to cut cost, to deliver faster, and maybe even, you know, push the frontier on what we thought was possible. And every week the frontier is moving. and and so, you know, I I guess this is w why the show exists. but yeah, there are folks out there that would prescribe to stop following the news. sometimes I feel like that might not be a bad thing. but this week I don't because there's been such exciting stuff. I'm gonna kick off with our top headline items. last week we spoke about Anthropic and OpenAI cutting their API costs. And the models getting cheaper is absolutely a story for this week as well. so Anthropic shipped clawed haiku 5.5. And haiku is their kind of I wouldn't say entry level, but it's their cheapest model. And they haven't actually released a haiku model for quite some time. So what we've been seeing is Opus 5.5, Sonnet 5.5, and now Haiku 5.5. And the exciting thing about it is it's 10 cents per million input tokens. It's got a 1 million token context window. And so it is an absolute beast. And if you need it to review a contract end-to-end, if you need it to go through thousands of documents and turn that into a report, I think you know now you've actually got a really affordable model with Anthropic. So that's pretty exciting. we've also had some news about Fermos. Firmus is an NVIDIA backed AI infrastructure builder out in Australia and they were basically pushing their IPO. we spoke about them a few weeks back, and unfortunately they did not meet their demand, so they actually closed their IPO books, and I think the prices slid from something like nine dollars to eight dollars twenty-five. and they even had some conversations about seven dollars and six dollars. so I don't know what that means, but obviously a high profile IPO like this actually failing, what does it mean in terms of investment sentiment towards AI infrastructure? it's definitely something that's a bit of a concern. And of course, it does absolutely go hand in hand with this concept of the reduction of prices. How is anybody forecasting margin? when every month brings a whole slew of price reductions. you know, and obviously GPU costs and the like hardware is increasing. but there you go. So the other big news and I was very excited about this one is that we've also got GPT six point one Sol from OpenAI and then the really cool one Gemini four Argonne. And Argon has been Doing very well in the benchmarks. So I don't know if you're going to cover this. I know you like to cover some of the benchmark stuff, Emile. it's certainly been pitched as something that can beat Seoul and that can actually beat Fable Five. so that is very exciting to see Gemini back in the game. And so we'll dive a little bit into that one. I know it hasn't been released everywhere, I haven't had a chance to play with it. and then of course there is also some news. That OpenAI is holding a finished model. so this is something they've built but they haven't released, and it is Astra six point one. Obviously, Astra made a big splash what about five weeks ago now? And I know you had amazing things to say about it, but six point one coming along, so we'll have to wait and see when that actually kicks in. so really interesting to to see that the model releases just keep keep pushing. jumping a little bit into that haiku, because I think haiku five point five is probably the most interesting of the news for me. It basically takes another fifty percent off cost. if you are basically using haiku in in batch mode. Anthropic actually estimates that it's about seventy five percent cheap on average. than the previous model, haiku four point five, and the estimate already counts for a new tokenizer which counts roughly thirty percent more tokens on the same text. So so you've kind of got deeper processing and actually much lower cost. so it's smarter and it costs you less. So you know this is built for high volume work. so document classification extraction, triage customer issues, you know, summarizing very large documents, all of these things are going to be very well supported by that model. And it's actually something that I have chosen not to use in the past, because I tend to color go for the latest models and and I tend to also use them in either medium or high. sometimes I'll even go to extra high, which is extremely expensive. and with something like haiku, I actually think it definitely will give it a go. Even with Sonnet 5.5, you know, I thought that was an excellent model. But when Opus came out, I switched to Opus and 5.5 has been brilliant. But Sonnet 5.5 as a model. sometimes even beats Opus five point five in encoding activities. So I think we've got this enormous choice. And that's just out of Anthropic. We haven't even got to you know what we're doing with Kimi and Deep Seek and some of the other Chinese models. But i anything that caught your eye in terms of this week's news, Emile? Em: add to the point that you mentioned regarding the velocity. It's certainly increasing. If we keep thinking it can move faster, but it is. And if we just look at the cadence on model releases that's between subsequent models that is now compressed even further to averaging, think it's every seven days, we're getting a new model. And to our listeners, for those who haven't looked at our website, I highly recommend you go look there's an interesting graph showing you the compression and that just seems to be getting tighter and tighter. it is difficult to choose what to... put your attention on and I find I'm either switching off the news or just being more selective on what aspects I want to focus on to, get work done to enhance my current way of processing And on Haiku, just wanted to add a little bit more detail around that. Anthropica describing it as its fastest and most capable small model. And the positioning is refreshing. specific this model is not intended to spend an afternoon proving theorems it is a model for enormous layer of short repetitive work around larger systems summaries as we've mentioned Ralph also very good at content database questions, classification and quick browser actions and sub-agent jobs, inside of the workflow. The capability has risen enough to make the model interesting beyond cheap text processing. Anthropic reports 72 % on online subsets of OS World 2.1, a benchmark for operating a computer against 15.7 % for Haiku 4.5. So that's a substantial jump. And also adds adjustable effort setting and beta computer use and browser use support in the Python and TypeScript software kit, which is very welcome for people like me. Anthropic still positions Sonnet and Opus for the hardest coding and long horizon work. Why this matters is because the economics of agents are not set by a single smallest call. They are set by every lookup, check summary, and retry surrounding it. A useful architecture can reserve an expensive model for planning and judgment while delegating bounded jobs to a quick model. The result is not necessarily less intelligence. It can be more attempts, more verification, and faster feedback for the same budgets. Closing thoughts on this, I can imagine a winning AI stack looking a lot less like one all purpose brain and more like a good team. A senior model sets direction. The small model does the high volume legwork and evaluation decides when the results is trustworthy. Cheap intelligence becomes valuable when the organization around it is disciplined. looking towards some of the AI content creators that I look at, Theo made quite an interesting observation. He essentially said that the Haiku 4.0 5 model was so bad with his mature ecosystem. He effectively put rule-based around all his harnesses, banning Haiku 4.5 for being used in any use case because he just couldn't find anything to make it fit without giving him more grief than it was worth. And the jump is so significant and he's been so impressed by it. he's moved it from the blacklist to the whitelist. So I think it's certainly worth checking out. And when you look on the graph in terms of performance, it was effectively when you look at costs and intelligence with costs being on the horizontal of the graph and intelligence being on the vertical. It was sitting all the way on the right to the bottom of the graph, barely making it in. And when you look at Haiku 5.5, it has leapfrogged right up to frontier level for a fraction of the cost, as you mentioned, Rolf. I think it's $0.10. It's a significant leap in improvement and price. I'm certainly going to give it a good go and see what it's capable of. Ralph: I have a question for you, Emile, because there's something that has been on my mind here. And it seems to me that you know when Anthropic or OpenAI or even DeepSeek and Chemi are releasing a new model, one of the things that they do is they throw an awful lot of compute and power at it and it performs brilliantly. And then you kind of get this behavior. Where they start to, let's just say, smooth how they are allocating resources to these models. And so what ends up happening is you get a little bit of a variable in terms of the quality and the speed and and response of these models. And so even though, you know, you're talking about Theo having a really good experience with it with it, putting it on the on the the whitelist, but it strikes me, you know, we could try it out tomorrow. And they've again fiddled with how much resources they've actually placed at the disposal of of the model, and it just doesn't perform quite the same way. And so I just wonder what you make of that variability, like in the context of enterprise, you know, w what does that really mean? If you're a business and you're relying on a certain level of output, like can they actually weather that variability in consistency? Em: it's a great question. And in my personal experience, the variability does happen. as a model gets released, it actually performs really poorly. That could be because of demand. also, well, there's a few things that happen. the latest frontier model before a new model gets released starts degrading in performance. And I'm not imagining that. And I think it's purposeful. because they want to make the next release model look better. And separately, when we get these reviews from the likes of Theo, they beta testers. they get first dibs before it goes out to industry when the systems are running well. they're getting the best possible time to make use of these models at their best. you're right. not smooth, it's variable, and it's unfortunate. Ralph: Yeah, so get in early, hammer it hard and then make sure that you can kind of control your quality. But it does rather mean that, you know, nobody can fully invest in any one model. And I think this is this is the problem. Like you you constantly have to y be on top of it and constantly shifting. It's very expensive. Em: You say that Rolf, and maybe it's a good thing that I've discovered this because I don't pay API prices. pay for monthly subscriptions and I'm not necessarily always on the frontier because the capability has got so good. You know, I'm doing really good work Sonnet and Opus 5.5. And I'm taking it to high and max, and it's more than proficient. And because I'm doing that, what I'm noticing when new model get released, the prior models actually start becoming fast. I'm not sure if you've noticed that. So Ralph: Yeah. Em: ultimately, what I'm saying is, Yes, there is an upfront cost, but if you may be setting one tier model behind, there's opportunity there. Ralph: Well and I and and I guess this is the thing, if you're staying one tier model behind, that means y every month you're changing model because every month you're becoming another tier model behind. and it kind of reminds me little bit of Craig Smith, who I do the retail innovation podcast with. And obviously he's been very loudly complaining that he feels like he's fallen behind because he's still using Sonnet five point five and he Because he hasn't felt a need to move on from that. It's so capable in the projects that he has been working on. So I absolutely get it. and yeah, you don't have to be on the bleeding edge, I guess in that sense. you have to be a month behind and probably save a bit of cash. But I I I think it's an interesting point just about the the cadence. you know, we are seeing breakthroughs every month. I wonder the big question for me now is, you know, when do we actually move away from transformers? Transformer actually being, you know, the underlying technology that pretty much, well, that all of these Frontier Labs are are working with currently. GPT, ch as in Chat GPT, you know, the T stands for Transformer, right? So the general purpose transformer. and I think we we had quite a lot of criticism of this transformer technology probably about a year ago, especially from like the Sakana guys, ex Google DeepMind, they went off and started Sakana AI and were calling for actually the the thing that the the lead there was saying that he feels like transformers have kept everybody. Focused on the wrong thing. So no innovation has actually happened in this space. And the reason why is because we've been able to get so much more performance out of this technology, but obviously it is in its in its own way. it it it has compromises. And so it's been very quiet in terms of the innovation, of what comes beyond the transformer. but of course, at the same time, it is agreed amongst I guess everybody in the space that, you know, AGI has been achieved, at least in terms of it now passes the the Turing test and it's gone way beyond that. And so I I don't know if you've heard anything, but that's kind of what's on my mind. I'm starting to scan the horizon to see what's actually coming down the road. Em: a similar theme, a question I want to fire back at you. I've started putting a lot less precedence on benchmarks. just not finding them very useful anymore. we always knew that they were getting max, but I'm just finding with each subsequent release, I'm not getting a lot of value out of what I'm seeing. either the reporting is not matching my experience or it's not relevant because I'm not. trying to solve the world's hardest maths equation, trying to do desk work and development work. And I'm finding the second tier models more than adequate. Ralph: Well, this is quite fascinating as a concept in itself because on the Moonshots podcast, the one that Peter Diamondis hosts, he was chatting about one of his connections getting access to the Cerebras AI. And they've got their own special chips that they're running. And this guy said that he had never experienced Anything like it. He was completely blown away by the performance. the his benchmark quote was something like: Before you've finished typing your sentence, it's coming back with a response. and so obviously the transformer technology is all about prediction, so you know, there it goes, it's predicting, right? And it only needs a little bit of context to predict what what you're actually trying to ask. Em: I know chat, GVT have ultra-fast mode now and so declared and, the people that experience that they saying it's, you know, it's like drugs, you know, once you've had it, you just can't come off of it. The experience is so incredible. Ralph: Yeah. Em: as you hit enter on your query, it's really giving you the answer. I can see why that's highly convenient, but it also comes at a significant price. think if I'm not mistaken, it's eight times the token spend to enjoy the great experience of Ultrafast. Then having said that, I'm finding the current models that I'm working on, they... They're coming back too quickly with getting the work done to the extent that I feel like my thinking is becoming inferior. I need time to sit and think about where I want to take the direction with things. And I'm finding that to be more of a problem now trying to slow myself down and not trying to always maximize the utilization I'm being given every five hours. Ralph: Yeah, because it's almost impossible chasing that dragon. You're never going to get in front of it unless you are Em: But I still did. Ralph: re well, the thing is, Em: Hahaha Ralph: in in so I guess my parallel story to this is I use Kimi quite a lot for for research. Kimi has gotten so fast at running its deep research that I have to really think about How I'm asking the question, the problem that I'm setting out, the way that I want the solution structured, and also, you know, just what information I'm going to feed it. And all of that obviously takes time in itself. but then really, so now what I'm imagining is kind of like a pipeline where I've got all of my dreams, hopes, aspirations, research goals, topics, and I'm kind of, you know, almost feeding them into a hopper. that he's then firing out to Kimi on a constant basis to really keep it busy. But it's coming back so fast. And of course the volume of material that's coming back is absolutely incredible. for the podcast last week, you know, I was pushing a lot of stuff to it, kind of looking at right, what would listeners really want to know? What are the kind of tools that are helpful to to people where they can really leverage AI to to build solutions, and so I ran about seven research projects in parallel. of course it only came back with everything in about half an hour. And you know, it's taken me the best part of a week to even find a moment. I think I've managed to get through three of them. so you know, we are the bottleneck. We've always said we're gonna be the bottleneck. This is probably the biggest, scariest moment for us because we are the bottleneck, and so w all we can do. Em: I think so. Ralph: is hand more and more over to the machine so that we're less and less of a bottleneck, but be we become more and more divorced, from the content, more remote. And I think ultimately we remove ourselves entirely. And and probably this is where some of the problems in the future are actually gonna come from. So yeah, I don't know how we keep ourselves we've got a name for it, right? Human in the loop. but finding the balance right now, I'm s I'm seeing it's almost impossible. We cannot review, understand, assimilate the content that we're generating. Em: I am seeing the speed as a potential problem because two things that concern me is that the convenience makes us want to think less and it takes away our need or want for agency. I've experienced the exact same thing that you've gone through, not with Kimi, but I'm finding it much harder. to use up my utilization on ChatGBT just because it's so much more generous than Anthropic. I find it very easy to blow through my usage on Anthropic, but ChatGBT is still a challenge, which I'm setting as a goal I just feel like I'm not thinking big enough that my designs need to be more creative and detailed. I created a whole lot of documentation and guidelines that I really wanted to look through. And I was really impressed with the content and what it created. And I got to the point where it was just spitting out that the artifacts out as quickly as I could think them up. And I've got this backlog of documents from this week that I haven't even opened a single one of them. We've got a different kind of problem now. It's not getting the work done. It's consuming the work. Ralph: Well, the rainforest thanks you for cutting down half an acre to create a bunch of code that never nobody's ever gonna review. but seriously, I I think how do we stay productive? This is really the key thing. and maybe it's all about, driving more quality, but even as you say, let's get bigger goals. Let's just get bigger problems that we that we need to solve. And and this is where I I you mentioned, you know, kind of outsourcing your thinking, or at least, not putting as much effort into your thinking. And maybe this is where where there's some salvation for us because it forces us to say, hang on a moment, how am I really going to get value out of this? Am I solving the biggest problem that I can? Let me think deeply about how I'm going to go about doing that. And and therein lies the growth. So we we can only hope and dream. But talking about growth, one of the things that I you know was Observing a discussion between a number of AI illuminaries, and one of the things they said was that AI had solved the hum the humor problem. As you know, on the show, I've been getting AI to write jokes unsuccessfully since the dawn of time. And I have maintained that we have not achieved AJI, artificial joke intelligence. and I just want to call out that this week is absolutely the same thing. So I'm gonna share just two of the jokes because they're they're they're so horrendous I can't go past this. But the first one is on haiku, so and it reads as follows for ten cents, haiku will read your entire contract. That makes it the only party to the contract that actually reads it. And how sad is that because the contract was probably written by an agent and read by an agent. So that one was more sad than it was funny, I think, to Em: and it echoes the problem we've just been discussing this week. Ralph: Exactly, which is why I thought actually it has a has a little bit of a a relevant one. But the the other one that this is a bit of a burn actually. So these are from the cutting room. These are the jokes that didn't even make it. but the ones that did make it were even worse. So I think I might have to have a word with Claude about what it's discarding. but this one is Google named its new model Argon. and it's named after the famous gas that never reacts to anything. Finally, an honest name. For Google customer support. Ooh, that's bad. Brutal. Em: Oof, now that's a bit of a burn. the benchmarking does look extremely impressive, but they haven't made it available to anyone. Ralph: Yes, what is going on? Em: What's going on? And we've been waiting so long for Gemini to pull their socks up. It was the same when they were doing their other releases. They kept stating that it's coming and they only gave us one or two of them, the flash model, which is not what I wanted. And then they finally, come up to the line and say they're back in the frontier race and they don't make it available. they're showing off its power res, but it's not clear to me. which surfaces it belongs to inside the Google environment. They haven't made that clear either. Ralph: Well, they're clearly playing a bit of a game. They needed to get themselves in the news and took the opportunity. So let's talk about something that really is in the news and one of our famous well at least famous and favorite AI companies, 11 Labs. So certainly I've been using their voice solutions for quite some time. I built a very nice co host, which I've always threatened to get on this. this the this podcast but I can't get Riverside to pick up the audio right so I need to still go back and solve that problem but 11 Labs what an amazing platform so these are a couple of young guys some founders out of Poland I think they joined Y Combinator so they were one of the unicorns early early A U unicorns anyway they're now valued at twenty two billion so they doubled their valuation And I think it's incredible, especially given how much competition there is out there. There are endless AI voice solutions. you know, every week there's another repo where you can actually build your own voice solution. Even about four months ago, Mistral released their voice model, which allowed you for almost no cost to build your own voice solutions. voice is absolutely one of those key I I would say form factors really in terms of interacting with AI. it's becoming more and more popular with ambient devices. I think it's going to take an even bigger leap. But there you go. 11 Labs absolutely crushing it. and very interesting to see that well I guess they've probably made Y Combinators bonuses for this year with that level of valuation. just shows you what you can go out and achieve in, just a handful of years, really. impressive to to see this. And then another one I wanted to t talk about, I I haven't actually been able to verify this and it makes no sense to me. I don't know if you came across this, but the FT published something like OpenAI's annualized revenue at twenty billion. I thought it was around forty billion. that was an interesting one. Maybe it's just a bit of duff news that came into the the research protocol. But if that is true, then that is pretty scary because they are certainly blowing a lot more cash than that, which obviously is driving, all of this conversation about the AI bubble. n not to be too negative about things, but that's an interesting one. Em: Thanks, Ralph You've been brilliant. this was loads of fun. Look forward to next week. Ralph: We'll see you on the edge.