Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / The True Cost of Cloud AI Inference Lock-in
Transcript
- Lucas: So you take a large language model — something like a 70-billion-parameter open-weight model — and you run inference on a single cloud provider for six months. By month seven, your cost per token is up 30 percent. But you can't leave. Luna: Can't leave? That sounds dramatic. What's physically stopping you? Lucas: Nothing physical. It's all financial. The data pipeline, the cached embeddings, the fine-tuned adapters — they're all formatted for whichever GPU architecture you're on. If you're on NVIDIA with AWS, moving to Azure means you might need to recompile for AMD, or you've got a custom kernel that only runs on that one cluster. Luna: Okay, but that's not new. We've talked about cloud lock-in before. What makes inference different from storage or compute? Lucas: It's the volume and the velocity. Storage lock-in — you can move a petabyte of S3 objects if you're willing to pay the egress fee. Compute lock-in — you can migrate EC2 instances over a weekend if your app is stateless. But inference? You are hitting that GPU cluster hundreds of times per second. The cost to port the model itself is trivial. The cost to port the operational setup — the serving infrastructure, the monitoring, the latency requirements — that's where they get you. Luna: I've heard people throw around the term 'inference anchor.' Is that just a consultant buzzword, or is there something real behind it? Lucas: It's real enough that I first heard it from a pricing engineer at one of the hyperscalers — off the record. The idea is that inference, unlike training, is a continuous, latency-sensitive, high-volume workload. Once you've tuned your serving stack to one cloud's GPU or TPU architecture, the switching cost is not just financial, it's operational. You'd have to retest, re-optimize, maybe re-architect part of your pipeline. That's weeks of engineering time. Luna: And while you're doing that, your users are staring at a loading spinner. So the cloud provider knows you'll absorb price increases rather than suffer downtime. Lucas: Exactly. So let's make this concrete. There's a mid-size enterprise — I'll call them 'MediCorp' — not their real name. They built a medical document summarization tool on AWS using Llama 2 70B on P4d instances. After six months, their inference bill was $1.8 million a month. They wanted to move to Azure to get access to the ND H100 v5 instances which had better price-performance for their workload. But the cost to re-engineer their PyTorch serving code for Azure's GPU topology? About $400,000 in engineering time. Plus they had 12 terabytes of cached token embeddings that were stored in S3 — moving that out cost them $7,000 in egress. Luna: Wait — 12 terabytes of cached embeddings? That's not even that much data. But they had to move it all over the public internet or use a direct connect, and then there's the latency hit during migration. So the anchor isn't the data size, it's the dependency chain. Lucas: Exactly right. The egress itself was a rounding error — $7,000 on a $1.8 million monthly bill. The real anchor was the engineering cost and the risk of downtime. They ended up staying on AWS and negotiating a three-year reserved-instance discount that brought their per-hour GPU cost down 22 percent. But they're still on AWS. They never actually migrated. Luna: So the threat of leaving got them a better deal, but the anchor held. And that's the cloud provider's ideal outcome — you stay, but you feel like you won. Lucas: Right. And now with custom silicon — AWS Trainium, Google TPU — the anchor gets even heavier. If you train on TPU v5p, your model is compiled specifically for that hardware. You can't just run it on an H100. You'd need to recompile, and the performance will be worse unless you spend weeks tuning. Luna: So the custom chips are a feature, but also a trap. You get better price-performance if you stay, but you lose the optionality to leave. That's the trade-off. Lucas: That's exactly the trade-off. And this is where the episode's number comes in: inference cost per token. Finance teams at these enterprises are starting to track cost per token the same way they track cost per gigabyte of storage. It's the unit metric that reveals whether you're getting competitive pricing or being quietly anchored. Luna: If today was actually useful to you — especially if you're the one who has to explain a cloud AI bill to your CFO — the way these conversations stay ad-free is listener support. You can find that at buy me a coffee dot com slash fexingo. Lucas: Yeah. Totally. We don't run ads, we don't have sponsors. It's just this. And if you find value in that, the coffee thing is how it keeps going. Luna: Okay — so back to that cost per token metric. How do you even calculate it meaningfully? Because token counts vary wildly by model and task. Lucas: You need to standardize on a benchmark. The industry is moving toward something like the MLPerf Inference benchmark, which gives you a throughput number — tokens per second per dollar. For example, on an AWS P4d instance running Llama 2 70B, you might get around 40 tokens per second per dollar for a specific batch size. On Azure's ND H100 v5, that number jumps to 55 tokens per second per dollar. That's a 37 percent improvement. Luna: So if you're paying $50 per gpu hour on one cloud and $60 on another, but the more expensive one gives you 37 percent better throughput, the total cost of inference per token is actually lower. Lucas: Exactly. And that's where the negotiation gets interesting. You don't just compare hourly rates. You compare cost per million tokens. And you need to know your own workload profile — because a batch-inference job might have different characteristics than a real-time chatbot. The cloud providers know this. They price GPU instances to maximize profit from the anchored customers, while offering competitive per-token pricing to new ones. Luna: So there's a version of the market where inference becomes a loss leader to hook you, and then the price creeps up after you're integrated. Do we have evidence of that? Lucas: We have anecdotal evidence. A large e-commerce company — well-known, name you'd recognize — told me that their per-token inference cost on Google Cloud increased by 18 percent over 12 months without any change in instance type. The only change was that they had become reliant on tpu specific optimizations. They couldn't easily switch to another cloud without rewriting their serving layer. Luna: That's the anchor at work. And the scary part is, it's not malicious — it's just the natural consequence of specialization. The more you optimize for a platform, the more you're tied to it. Lucas: Right. So the strategy for an enterprise is to maintain portability from day one. Use containerized serving stacks that abstract away the hardware. Keep your model weights in a standard format like Safetensors. And benchmark on at least two clouds before committing. The cost of portability engineering upfront might be 10 percent of your inference budget, but it gives you negotiation leverage forever. Luna: I think that's the single biggest takeaway. The anchor only holds if you let it. And the way to not let it is to keep one foot out the door from the start. Lucas: Yeah. The most expensive inference is the one you can't move. So track that cost per token, keep the code portable, and never let your architecture become a hostage. Luna: We'll put a link to a simple cost per token calculator in the show notes so you can run your own numbers. Lucas: Next episode: we're looking at whether the new 'AI inference regions' that AWS and Azure are opening actually reduce latency for anyone outside the US East coast. Luna: Spoiler: not really. Lucas: Spoiler: not really. Talk then.