Latest / The Cloud Business Podcast with Fexingo: AWS, Azure, GCP, and Enterprise Infrastructure / How Cloud Bills Now Meter Your AI Inference Tokens
Transcript
- Lucas: So, Luna, I was looking at a friend's cloud bill last week — he runs a mid-sized SaaS platform — and there's a line item that wasn't there six months ago. It's a charge for something called AI inference tokens. Luna: Tokens. Like, the little pieces of words that language models chew on? Lucas: Exactly. And his bill jumped forty percent month over month once he enabled a chatbot feature for his customers. He had no idea it was coming. Luna: That's a serious surprise. I thought most cloud providers were still charging for compute hours when you ran models. Lucas: They do, but now they're layering a per-token charge on top. You're not just paying for the GPU time anymore — you're paying for every single token the model processes, whether it's in a prompt or a response. And that adds up fast. Luna: So it's like a double billing? Compute plus tokens? Lucas: Sort of. The compute charge covers the underlying infrastructure, but the token charge is purely metered by usage volume. It's a shift in how cloud vendors monetize AI workloads — they're moving from time-based to usage-based metering, and tokens are the natural unit because they tie directly to the value the customer gets. Luna: That makes sense from a pricing perspective, but it feels like a hidden tax for companies that don't read the fine print. Lucas: That's the thing. Most enterprises sign up for AI services expecting to pay for compute, but the token component is often buried in the rate card. For a typical chatbot, you might be looking at very small per-token costs, but when you scale to thousands of users, it becomes a six-figure line item. Luna: Do you have a concrete number? Like, what does a token actually cost? Lucas: Well, it varies wildly. For a large language model, you're often looking at something like twenty dollars per million input tokens and sixty dollars per million output tokens. But that's just one model — some are cheaper, some are pricier. The key is that output tokens are typically three times more expensive, so any feature that generates long responses is going to hit your bill harder. Luna: So a customer asking a detailed question that produces a long answer is going to cost you more than a simple yes or no query. Lucas: Exactly. And that's where the surprise comes in. A lot of developers optimize for response quality, not token count. They'll set the model to generate verbose answers because it feels more helpful, but they don't realize that verbosity has a direct dollar cost. Luna: I've seen some engineering teams use tools to estimate token usage before launching a feature, but not many. Lucas: Right. In this case, my friend's team used a simple heuristic — they estimated average tokens per conversation and multiplied by expected volume. But they underestimated the average conversation length by a factor of three. Their users loved the chatbot, so they used it a lot. The bill reflected that. Luna: So what's the practical advice for someone building AI features? How do you avoid this surprise? Lucas: First, you need to understand the pricing model of your chosen provider. Not just the headline per million token price, but whether there are separate charges for input and output, and whether there are any minimums or volume discounts. Second, you need to instrument your application to track token usage in real time. That way you can see the cost per user, per session, per feature. Luna: That sounds like building a metering system just for your own usage. Lucas: It's not as hard as it sounds. Most model APIs return usage stats in the response, so you can log them. But the bigger issue is that many teams don't have a clear owner for AI cost. It's not the infrastructure team's job, and it's not the data science team's job. It falls through the cracks. Luna: So who should own it? The cloud architect? The product manager? Lucas: I'd argue it's a shared responsibility. The product manager should be aware that verbosity has a cost, the engineer should implement token-tracking flags, and the finance team should review AI spend on a monthly basis. But the reality is that most companies treat AI costs like a black box until the bill arrives. Luna: And then they scramble to optimize. But optimizing after the fact is harder than designing for cost from the start. Lucas: Exactly. There are some clever techniques, though. For example, you can use a smaller, cheaper model for simple queries and save the big model for complex ones. That's a classic pattern — model routing. Luna: Or you can set a maximum token limit on responses, right? Lucas: Right. You can cap the output length, which directly controls cost. Some providers let you set a 'max tokens' parameter, but if you don't, the model will generate until it thinks it's done, which can be surprisingly long. A simple cap can cut your token spend by half without hurting user experience much. Luna: But doesn't that affect quality? If you cut off a response mid-thought, users might not get what they need. Lucas: That's the trade-off. But you can design your prompts to be more concise, or use a system message that tells the model to be brief. That's a free win — it costs no extra tokens and often improves clarity. I've seen teams reduce token usage by thirty to forty percent just by tweaking prompts. Luna: So the cost is as much about prompt engineering as it is about model choice. Lucas: Absolutely. And it's not just the model you call — it's also the context. If you're using retrieval-augmented generation, you're sending a big chunk of text as part of the prompt, and that's all input tokens. The more context you stuff in, the higher your base cost per request. Luna: So even a simple Q&A over a document library could get expensive if you're passing the whole document every time. Lucas: Exactly. That's why a lot of teams are moving to a caching layer for embeddings or using a vector database to retrieve only the relevant chunks. But that adds complexity, and not everyone is ready for it. Luna: I imagine cloud providers are loving this shift. It's a new revenue stream that scales with usage, not just with capacity. Lucas: It's a brilliant move for them. They've been dealing with rising infrastructure costs for AI training, and now they can pass through inference costs more transparently while still marking up the compute. It also creates a lock-in effect — once you've built your application around a specific model's token pricing, moving to another provider is a non-trivial migration. Luna: Right. And that's why contract negotiations are becoming more critical. Have you seen any new negotiation tactics from enterprises? Lucas: Well, the big players are starting to demand volume discounts on tokens, not just on compute. Some are pushing for 'token burn' commitments — like committing to a certain number of tokens per month in exchange for a lower per-token rate. That's a bit like committing to a certain number of compute-hours, but the risk is that you might not hit the volume if your product doesn't take off. Luna: So it's a bet on your own growth. If you're confident, you can save a lot; if not, you're stuck paying for unused tokens. Lucas: Exactly. And the providers know that, so they structure these deals to encourage big commitments. But for a small startup, that might be too risky. The safer approach is to start small, monitor usage, and renegotiate as you grow. Luna: It's like the early days of cloud compute, when you had to estimate your server needs a year in advance. Lucas: Yeah, but the difference is that AI workloads are much more variable. A single viral feature can spike token usage overnight. So you need autoscaling, not just in terms of compute, but also in terms of your budget. Luna: Speaking of budgets, what's the best way to set a budget for AI inference? Is there a benchmark or a rule of thumb? Lucas: A common starting point is to calculate your cost per user per month. If you're building a chatbot, you might estimate that each user generates, say, a thousand tokens per session, and if they have five sessions a month, that's five thousand tokens. At current prices, that's a fraction of a cent per user. But multiply that by a hundred thousand users, and you're talking real money. Luna: So the key is to track usage per user, not just total spend. Lucas: Right. And to set alerts. Most cloud providers now let you set budget alerts on specific line items, so you can get an email when you hit fifty percent of your forecast. But the catch is that these alerts are often delayed by a few hours, so you might not catch a spike in real time. Luna: That's a bit scary. I guess you need to build your own monitoring on top. Lucas: That's what my friend ended up doing. He built a simple lambda that logs every API response's token usage into a database, and then he runs a daily query to see the trend. It's not fancy, but it gives him a clear picture of what's driving costs. Luna: And that's the kind of thing that can save you thousands of dollars a month. Speaking of saving money — you know, a couple of dollars a month is genuinely what keeps these conversations going. If you've gotten something out of them, buy me a coffee dot com slash fexingo makes a real difference for a small show like this. Lucas: Yeah, honestly, it's the only reason we can keep these episodes ad-free. Every little bit counts. Luna: So, back to the token front — what do you think the next line item on cloud bills will be? We've already seen charges for data egress, NAT gateways, and now tokens. What's the next frontier? Lucas: I wouldn't be surprised if they start charging for model fine-tuning time, or even for the number of embeddings you store. But the bigger trend is that cloud pricing is becoming more granular, which means more work for finance teams to keep an eye on. Luna: And that's something every cloud architect should be thinking about now, before the next surprise bill lands.