Latest / Master Your AI Future | AI Executive / Why Is AI Inference Becoming the Next Enterprise Cost-Control Agenda?
Transcript
- 📍 📍 📍 Imagine you buy a fleet of delivery vehicles for your company, and every single year the manufacturer tells you, you know, the engines are getting vastly cheaper to produce. Right. The raw parts are practically a bargain. Exactly. A total bargain. Yet somehow, at the end of the quarter, your logistics bill has jumped by, like, five hundred percent That is exactly the contradiction we are witnessing in enterprise technology right now. Welcome to the debate. It really is one of the most... I think it's the most fascinating structural anomaly in modern tech. The underlying AI models, uh, the raw intelligence, they're experiencing this massive deflationary curve. I mean, they are becoming radically cheaper per unit of compute. Yeah, absolutely. But paradoxically, running enterprise AI at scale is becoming, well, alarmingly expensive. Precisely. And Gartner is forecasting that the inference cost per agentic workflow will increase more than fivefold through twenty twenty-eight. We are facing what the industry is calling the AI inference paradox. Yeah, the paradox. So today, we are asking the critical question: How should enterprise leaders actually solve this AI inference cost crisis? And we are drawing on a massive intelligence base today. We're analyzing data from nine leading consulting and research firms, including, uh, Accenture, McKinsey, Gartner, KPMG, and BCG. And the divide on how to actually handle this is stark. It really is. It is. I mean, I look at this data, and the mandate seems entirely clear to me. Enterprises must urgently implement strict AI FinOps, meaning deep financial operations and tracking applied directly to AI usage, and they need intelligent model routing and, honestly, strict financial chargebacks. If we don't curb runaway token consumption right now, it is just going to consume entire IT budgets. Yeah. Well, I come at it from a different way. I look at that exact same landscape, but I see a totally different choke point. Fixating purely on the cost of a token is... Honestly, it's an analytical trap. Enterprises have to redesign the human workflows around these models to optimize for the fully loaded cost per business outcome. I mean, stepping over dollars to pick up pennies in model costs is not a strategy. Well, let's look at the actual behavior driving those token costs, because the urgency really cannot be overstated here. AI consumption is structurally out of control. It isn't just scaling. It is scaling inefficiently, mostly because of poor architectural choices and, frankly, unchecked developer behavior. Unchecked? Totally unchecked. Look at Accenture's data. Companies expect their token consumption to jump seventy-eight percent over the next twenty-four months, but here is the critical failure point they found. A staggering fifty-four percent of AI requests are being routed to models that are far more capable, and therefore far more expensive, than the task actually requires. You're talking about the massive frontier models, obviously. Exactly. The trillion parameter behemoths like GPT-4 or Claude 3.5 Sonnet. Accenture analyzed over nine thousand enterprise tasks, and roughly ninety percent of them did not require that frontier model capability. Right. But developers default to them anyway. We're essentially paying premium prices for baseline work. It's like, uh, it's like hiring a senior litigator at a thousand dollars an hour to proofread an email. The mathematical fix is to institutionalize multi-model architectures, where everyday tasks default automatically to cheaper, smaller models. You reserve the frontier model strictly for complex reasoning. Look, it sounds highly logical when you frame it purely as a software architecture problem. But optimizing model price alone, it just misses the physical reality of how work actually gets done in a large enterprise. How so? Well, if you focus exclusively on the cost of the token, you are looking at the economy through a pinhole Take McKinsey's analysis of an illustrative banking agent workflow. In that real world scenario, token consumption, even at those frontier prices you mentioned, it only represents twenty to twenty-five percent of the variable run cost. Which is still a massive new variable expense. It is. It is. But what makes up the remaining seventy-five percent? Human oversight. Right. It's the compliance officer reviewing the output. It's the operations manager checking the math. Therefore, tracking token price alone is just the wrong executive KPI. You could optimize your architecture, right, and drive your token cost down to fractions of a penny. Which you should do. Sure, but if your human review time increases because that cheaper model hallucinates slightly more often or, you know, misses context, you have lost money. The ultimate metric must be the fully loaded cost of completed work. I mean, I completely agree that the fully loaded cost matters, but let's unpack why that token usage is exploding in the first place, 'cause it's not just a flat twenty percent variable anymore. The root cause of these escalating costs is the shift toward agentic workflows. Right. Moving away from just typing a prompt and getting an answer. Exactly. We are moving toward autonomous agents that chain thoughts together. An agent is given a goal, and it writes code, tests it, realizes it failed, queries a database, writes a new script, and tries again. EY and Gartner both highlight that these sophisticated multi-step workflows require substantially more inference. Because of the context window. Yes. The context windows expand with every single retry. The token burn is exponential. Think about the mechanics of an agentic loop getting stuck. It is not like a supercomputer doing necessary heavy lifting. It is like a taxi driver who gets lost in a cul-de-sac. Huh, right. He doesn't know the way out, so he just keeps driving in circles, but the meter is running at a premium rate for every single rotation. When an AI agent is making thirty consecutive API calls to resolve a minor syntax error, paying frontier prices for every iterative failure is financial malpractice. Okay, I understand the taxi meter analogy. And yes, agentic loops consume vast amounts of context, but you are assuming the token is the most expensive part of that failure state. We have to bring in KPMG's findings here to reframe the timeline of these costs. Go ahead. They found that the original investment cases for AI might represent only twenty-five to thirty-five percent of the total three-year cost of ownership. The hidden costs are not the inference loops themselves. The real financial burdens are integration, constant model retraining, security monitoring, and exception handling. But exception handling is exactly what I'm trying to minimize by forcing better architecture. But forcing a smaller model often causes the exception. Let's look at the developer psychology you were complaining about earlier. Why do developers just dump a massive, unstructured PDF into a frontier model's prompt? Because they're lazy. No. Because the frontier model has the reasoning capacity to sort through the noise without breaking. If you force that same developer to use a smaller, less capable model just to save tokens, the developer now has to write complex parsing logic. The model might fail silently. An exception is generated, meaning the AI system breaks down, and a human has to step in. Okay, but- Let me just finish this point. If a more expensive frontier model significantly lowers that exception rate and drastically reduces the time a human compliance officer spends reviewing broken outputs, then the frontier model is definitively the more economical choice. That implies developer behavior is driven by some noble attempt to optimize the system's accuracy. But I would argue we are looking at something much simpler; human laziness and a total lack of financial visibility. Developers over-provision models because they don't have to pay the bill. Oh, come on. It's true, and this directly drives the question of how we manage the internal teams building these tools. We need a mechanism to drive AI economic literacy. Which brings us to the core debate around AI FinOps, specifically the choice between chargeback and showback models. The accountability aspect. Yes. I argue for immediate strict financial accountability for any team using AI. Right now, enterprise visibility is appallingly low. According to Accenture, only fifty-three percent of token usage can even be traced to a specific user or team. Yeah, that's wild. Half the usage is just a black box of shadow IT. Worse, a mere seven percent of surveyed companies use chargebacks. Meaning they actually deduct the AI costs directly from the business unit's budget? Correct. If the marketing team spins up an agentic workflow that burns a thousand dollars in an afternoon, that thousand dollars should come out of the marketing budget. Until developers, product managers, and business teams are financially penalized for inefficient context windows and those endless wasteful taxi meter loops, costs will balloon. If it doesn't hit their P&L, they have absolutely zero incentive to optimize. That is a highly rational economic argument that completely ignores the reality of organizational behavior. I mean, that's a compelling argument, but have you considered the chilling effect that that kind of punitive financial structure will have on enterprise innovation right now? I don't see financial accountability as punitive. It's just basic accounting. But we are in a phase where business units are still discovering what these tools can actually achieve. Accenture points out that less than twenty percent of token spend is currently tied to a quantified financial outcome. You cannot charge a team for something they do not yet know how to measure. So we just let them burn cash? No, but if you implement strict chargebacks today, a human resources department trying to build an automated onboarding agent will simply stop exploring. They will be terrified of blowing up their quarterly budget because of one experimental loop. You will kill the very experimentation that leads to the massive productivity gains we are chasing. By delaying chargebacks, we are just subsidizing inefficiency under the guise of discovery. You are treating AI like traditional software where you buy a license and let people play with it. AI is a variable utility. If you leave the heating on with the windows open, the utility bill goes up. It is a direct causal mechanism. We have to implement FinOps to close the windows. It is not subsidizing inefficiency. It is creating a runway for value creation. This is exactly why I advocate for show back first. Right. Generating the receipt, but not forcing them to actually pay it. Exactly. It's like putting a smart power meter in someone's kitchen. You report the consumption to the team. You show them the dashboard. You educate the developers on how, you know, dumping raw, unstructured data into a prompt is driving up the inference cost. But you do not immediately deduct that cost from their operating budget. Hmm. Show back builds the AI economic literacy you are advocating for, but it does so without punishing them for trying to build something new. It keeps the organizational focus on value creation rather than minimum consumption, because let's be honest, if your only goal as an IT leader is minimum consumption- The most successful AI project is the one you never deploy The problem with prioritizing value creation in this nebulous way is that it's highly subjective. It serves as a shield for engineering teams to run projects that have no clear path to profitability. And honestly, this debate over innovation versus cost control shifts dramatically when we look at how different industries actually operate. Yeah, industry constraints are huge here. Right. We cannot prescribe a one-size-fits-all runway for discovery because the constraints are entirely sector-specific. You're referring to the Deloitte data on industry applications. Yes. Let's look at the mechanism of how heavily regulated industries actually have to deploy AI. In financial services and energy, the absolute highest priority is regulatory compliance. A major bank cannot just send its customers' raw financial histories to a public frontier model hosted on a multi-tenant cloud just to see what happens. No, definitely not. They are mandated by sovereignty requirements, data privacy laws, and compliance frameworks to operate highly controlled models within their own secured perimeters. Which usually means leveraging open weight models they can host themselves. Precisely. And this is where the architecture debate becomes paramount. To run those models efficiently on their own hardware, they have to use quantization. Wait, for those less familiar with the engineering side, maybe briefly explain the mechanism of quantization in this context. Sure, yeah. Quantization is essentially taking a massive resource-heavy model and shrinking the precision of its weights. Think of it like taking a spreadsheet full of incredibly long decimal numbers and rounding them to whole numbers. Okay, that makes sense. It drastically reduces the memory and compute required, allowing you to run a highly capable model on much cheaper internal hardware while maintaining like ninety-nine percent of its practical accuracy. My point is, for these regulated sectors, the environment actively rules out wild, unchecked frontier model usage. Optimizing the architecture through quantitation and smaller models isn't just about saving pennies on a dashboard. It is the only way to operate legally and at scale. I understand the structural constraints in banking, but I look at that exact same Deloitte data from a totally different angle. You are highlighting the compliance barriers, but look at the expected outcomes across all five industries Deloitte analyzed. Okay. When asked what they actually want from AI, innovation capacity ranked highest at seventy-one percent. Risk management was at sixty-four percent. Token optimization, the architectural efficiency you are arguing should be our primary focus, was much lower, down at fifty-nine percent. The enterprise market is speaking clearly. They want the capability and the safety first. The unit cost of the compute is a secondary concern to them. It's only secondary until the bill arrives. But look at where the scale is heading. In the technology, media, and telecommunications sector, the share of companies expecting to consume over ten billion tokens per month is projected to jump from twenty-nine percent today to an astounding seventy-one percent by twenty twenty-eight. Massive exponential token consumption is inevitable. You cannot engineer your way out of that volume with strict FinOps policies and quantized local models. The consumption is coming. Ten billion tokens a month is the exact reason strict FinOps and architectural rigor are required. If a media company is processing ten billion tokens without model routing, without a system that actively pushes the simple text summarization tasks to a cheap local model instead of a frontier model, that volume will bankrupt the IT budget before the quarter ends. Not if those ten billion tokens are fundamentally doing the heavy lifting of the human workforce. This is the paradigm shift. The goal isn't to prevent the consumption of ten billion tokens. The goal is ensuring those tokens replace a vastly more expensive human process, thereby lowering the cost per completed outcome. Let's go back to the banking onboarding process McKinsey modeled. The one where human oversight was seventy-five percent of the cost? Yes. Let's look at the absolute numbers. Currently, a traditional onboarding process might cost a bank a hundred and fifty dollars per customer That cost is human labor, manual document review, checking identities, integrating data into legacy systems. Right. If applying a massive amount of AI inference, even running bloated agentic loops at frontier model prices drops that fully loaded cost from a hundred and fifty dollars down to between ten and thirty dollars per completed onboarding. That is a massive financial victory. The business side of the house does not care that the IT token bill went up by two million dollars because the customer acquisition cost just plummeted by eighty percent. That is a powerful narrative, I'll admit, but I think you are leaving massive efficiency on the floor by accepting that premise. If you rely solely on a massive frontier model to drop that cost from a hundred and fifty dollars down to thirty dollars, you are celebrating a victory while ignoring the waste. Through intelligent model routing, you could likely drop that cost to twelve dollars. By breaking the workflow apart. Yes. Think about the mechanics of the task. Why on earth would you use a reasoning heavy frontier model to extract a name and a zip code from a scanned driver's license? A highly specialized, fine-tuned, open weight model can do that specific extraction task with ninety-nine point nine percent accuracy for a fraction of a cent. Fair point. You only call the frontier model at the very end of the pipeline if there is a complex discrepancy that requires deep reasoning. My argument is not that we shouldn't pursue these massive eighty percent cost reductions. My argument is that without a structured multi-model architecture, we are paying a completely unnecessary premium for that innovation I don't disagree that specialized models have a vital role. I mean, Forrester's research makes a very strong case for the economic practicality of fine-tuned models on specialized workloads. But we have to be honest about the human cost of maintaining that multi-model architecture. Orchestration tools are getting better every day, though. The tools are, sure, but orchestrating a multi-model environment introduces immense operational overhead. You are no longer just calling one reliable API. You now have a team of machine learning engineers who have to maintain multiple data pipelines. They have to monitor various model degradation rates, handle latency issues between different systems, and manage much more complex infrastructure. Which is their job. Yeah, but when you factor in the salaries of those engineers, sometimes the simplicity of using a single, highly capable frontier model offsets the token premium entirely. The simplicity buys you speed, and in enterprise tech, speed is money. I think relying on that simplicity is a trap. The infrastructure to orchestrate model routing is rapidly becoming commoditized. You won't need an army of engineers to route prompts. The middleware will do it automatically. Meanwhile, the price of unoptimized agentic loops running on premium models will compound exponentially as the tasks get more complex. Well, we clearly have different views on where the bottleneck really is. We do. And to summarize my position in light of our discussion today, if we want to prevent AI from quietly eroding enterprise economics, organizations must take decisive architectural and financial action. We cannot treat AI compute as an endless, unmetered resource. We must orchestrate model selection so that the capability of the AI matches the complexity of the task. We must build AI economic literacy by assigning ownership through chargebacks, making sure that developers feel the financial weight of their code And above all, we must stop paying frontier model prices for routine repetitive work. And to summarize my stance, while token waste is a reality, and yes, it should be actively managed, the true competitive advantage for any enterprise comes from measuring the fully loaded economics of the workflow. Fixating on token costs obscures the bigger picture, which is human labor. If we deploy an AI capability that significantly reduces human oversight, lowers the exception rate, and accelerates process cycle times, then a massive increase in token consumption is actually a profound financial win. The math that matters is the cost per completed business outcome, not the cost per prompt. You know, despite our different lenses on how to solve the paradox, I think we clearly agree on the fundamental reality of this moment. AI is forcing enterprise technology to undergo a massive paradigm shift. We are moving from predictable, depreciable software budgeting into highly variable consumption-based economics. The days of buying a set number of software seats and knowing your exact IT costs for the year are officially over. Absolutely. We are managing an entirely new kind of economic input. It behaves much more like managing an industrial energy grid than managing a traditional software stack. You pay for the current you draw. Which brings us to the core challenge every executive must face right now. We highly encourage you to dive into the source material from these consulting firms to assess your own organization's AI economic literacy. There is so much more to explore in the data, but as we close today, we want to leave you with a final reflection drawn from the reality of this transition. Ask yourself this: If the token cost disappeared from your IT dashboard tomorrow, could your leadership team still demonstrate the actual financial value of each major AI workflow?