Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Anthropic Cut Inference Costs by 40 Percent
Transcript
- Lucas: So, a number that's been sticking with me: Anthropic reportedly cut the cost per token on some of its Claude models by around 40 percent over the past year. Not through cheaper GPUs, but through software engineering. Luna: That's a big deal, especially when everyone's complaining about AI bills. How did they actually pull that off? Lucas: Well, it's a mix of things, but the big ones are architectural changes to the transformer itself, plus some clever tricks at inference time. One of the most talked about is speculative decoding. Luna: Speculative decoding - that's when you use a smaller, faster model to draft tokens, and then the big model verifies them in parallel, right? Lucas: Exactly. The key is that verifying multiple tokens at once is much cheaper than generating them one by one. So you get the quality of the big model with the speed of the small one, and the cost savings come from fewer sequential forward passes. Lucas: But there's a catch. If the draft model is too far off, the big model rejects a lot of tokens, and you're back to square one. So it's all about finding a draft model that's good enough to get a high acceptance rate. Luna: Makes sense. And I've heard that Anthropic also played with the architecture itself - like, not just the decoding but the layers. Lucas: Right. There's work on what some call 'cross-layer attention' or 'multi-scale' transformers. Instead of every layer attending to all previous tokens, some layers can attend to a downsampled version of the sequence. That reduces computation without hurting quality much, if you're careful. Lucas: And then there's batching. If you can pack more requests into a single batch, you use the hardware better. Anthropic has been really aggressive about dynamic batching - grouping requests that arrive at different times into one forward pass, as long as they fit in memory. Luna: So it's a combination of algorithmic changes and better engineering around serving. What about quantization? That's another lever people talk about. Lucas: Oh, definitely. They've been using lower precision - like FP8 or even INT8 for certain weights - which cuts memory bandwidth and speeds up computations. The trick is doing it without degrading the model's output. It's usually a calibration process where you measure the impact on a bunch of tasks and adjust. Lucas: But here's the thing: all these optimizations compound. You don't just do one thing and get 40 percent. You stack maybe ten different 5 percent improvements, and suddenly you're there. Luna: And that's what separates a really good ML infrastructure team from a great one - the relentless grind of incremental gains. Lucas: Yeah, and it's not just about cost. Lower cost per token means you can offer lower prices to your customers, which is what they did with the price cuts earlier this year. That expands the market for your API. Luna: It's a virtuous cycle. Lower prices attract more users, more usage means you can amortize the fixed costs over more tokens, and you can invest more in optimization. Lucas: Exactly. And that's something any CTO building on AI can learn from. You don't have to be Anthropic to benefit from these techniques. Even if you're running a fine-tuned model on your own infrastructure, you can apply speculative decoding or quantization. Luna: But isn't there a risk that these optimizations make the model less capable? I mean, if you're cutting corners, quality might suffer. Lucas: That's the eternal trade-off. But the trick is to measure it properly. You set up an evaluation suite that reflects the tasks your users actually care about, and you make sure your optimizations pass that bar. It's not about cutting corners; it's about removing waste. Lucas: And sometimes, removing waste actually improves quality. Like, if you're quantizing carefully, the model can generalize slightly better because the noise acts as a regularizer. It's counterintuitive but it happens. Luna: So for a CTO listening, what's the one takeaway? Is it 'start with speculative decoding' or 'look at your batching'? Lucas: I'd say the first step is to measure your own bottlenecks. A lot of teams assume it's compute, but sometimes it's memory bandwidth or even the serving framework. You need to profile before you optimize. Lucas: Once you know where the time and money goes, you can pick the right lever. And don't ignore the low-hanging fruit like dynamic batching. That's often the easiest win. Luna: And if you're a smaller team, you might not have the resources to do all this from scratch. Are there tools or frameworks that help? Lucas: Yeah, there's vLLM, tensorrt llm, or even Hugging Face's TGI. These give you a lot of the optimization out of the box. But you still need to understand what they're doing under the hood to tune them for your workload. Lucas: It's like buying a race car - you still need to know how to drive it. Luna: That's a good analogy. And it reminds me - we keep this podcast ad-free, which is a bit like those optimizations: we're always looking for ways to cut waste and keep the quality high. Lucas: Absolutely. And we do it because we believe this kind of conversation should be accessible to everyone. If you find value in what we do, and you want to support the mission, you can find us at buy me a coffee dot com slash fexingo. Luna: It's a simple way to keep this going, and we appreciate anyone who does. Now, back to the technical side - I want to talk about one more optimization Anthropic reportedly used: distillation. Lucas: Oh, yeah. They've been training smaller models, like some of their 'Haiku' and 'Sonnet' variants, to mimic the larger ones. That's a classic. You take the big model's outputs and use them as training data for a smaller model. Lucas: The smaller model runs faster and cheaper, and for many tasks, it's nearly as good. The trick is to know when to route traffic to the small model versus the big one. That's a whole other layer of optimization - model routing. Luna: So it's not just about making one model cheaper, it's about having a portfolio of models and sending each request to the right one. Lucas: Precisely. And that's where a lot of the cost savings come from. You don't need the biggest model for every prompt. A classifier can decide which model gets each request based on complexity or user intent. Luna: I love that. It's like having a team of specialists instead of one generalist who does everything. Lucas: Exactly. And it's a lesson that applies beyond AI. In any engineering org, you should match the complexity of the solution to the complexity of the problem. Use a sledgehammer only when you need to. Lucas: So, to wrap up: Anthropic's 40 percent cost cut wasn't a single magic bullet. It was speculative decoding, architecture tweaks, dynamic batching, quantization, distillation, and smart routing - all working together. Luna: And the takeaway for the rest of us is to start profiling, start measuring, and then start chipping away at the waste. Lucas: Yeah. The technology is moving fast, but the fundamentals of engineering economics haven't changed. It's still about understanding your system and making it lean.