Latest / MBA Training AI / AI ROI: what Klarna, Copilot and baselines reveal

Transcript

AI ROI: what Klarna, Copilot and baselines reveal · MBA Training AI

• 0:00
  Welcome to the Leaders Insights podcast.
• 0:07
  Today's episode, the spreadsheet that embarrassed a CFO and changed how we measure AI.
• 0:12
  A CFO stood up in front of the board, projected millions in AI savings, and then watched the
• 0:18
  number melt to almost nothing.
• 0:20
  What actually happened?
• 0:21
  The article walks through a major retailer that ran a customer service AI pilot and
• 0:26
  celebrated something like $11 million in projected annual savings.
• 0:32
  Then someone finally added up the full-cost, ORS-1, the licensing, the cloud compute, the
• 0:37
  humans reviewing bad answers, and the net dropped to a rounding error.
• 0:42
  The CFO, the chief financial officer, learned in public that the headline number was fiction.
• 0:48
  But projections are always optimistic.
• 0:51
  Why is this the story that supposedly changed how we measure AI?
• 0:54
  Because it's not one bad forecast, it's a pattern that repeated across industries through
• 0:58
  the early 2020s.
• 1:01
  Companies counted the gross benefit and ignored what the article calls the full cost.
• 1:07
  The savings were real on one side of the ledger, and the spending was invisible on
• 1:11
  the other.
• 1:12
  That asymmetry is the whole disease.
• 1:19
  Four things.
• 1:20
  Inference.
• 1:21
  The cost of running the model every single time a customer types a question, which never
• 1:26
  stops.
• 1:27
  Then the review layer.
• 1:29
  Humans checking outputs because the model is confidently wrong, maybe 5-10% of the time.
• 1:36
  Then integration engineering to wire it into your existing systems.
• 1:40
  And retraining because the model drifts.
• 1:43
  Its accuracy decays as the world changes around it.
• 1:47
  The pilot budget covers none of that.
• 1:49
  So the demo looks brilliant, and the operating bill quietly eats the gain.
• 1:53
  Exactly!
• 1:54
  A demo processes 10 tickets, production processes 2 million, and now that per-query inference
• 1:59
  cost is a monthly invoice with your name on it.
• 2:02
  I've seen internal decks, I won't name them, where the pilot showed a 60% cost reduction,
• 2:08
  and the 12-month reality was closer to 8% once support and error handling were counted
• 2:13
  honestly.
• 2:14
  Here's the uncomfortable one.
• 2:16
  Aren't the vendors selling these tools the same people supplying the savings math?
• 2:20
  Yes, and that's the tell.
• 2:22
  When Microsoft or Salesforce publishes a study saying their AI assistant makes agents 30
• 2:27
  or 40% more productive, read the footnote.
• 2:31
  That's a vendor measuring its own product, often in a controlled task, not your messy
• 2:35
  environment.
• 2:37
  Treat those numbers as marketing until you've reproduced them on your own data.
• 2:41
  The good ones will actually let you.
• 2:43
  Fine.
• 2:44
  How does a normal manager measure this without a data science team?
• 2:47
  Start with one number, nobody games.
• 2:50
  Fully loaded cost per completed task.
• 2:52
  Not per query, per finished piece of work, including the human who fixed it.
• 2:57
  Take your total spend, divide by tasks that actually shipped correctly.
• 3:02
  Compare that to what the same task cost before AI.
• 3:05
  If the AI number isn't lower, you don't have savings, you have a science project.
• 3:10
  What if the task quality is worse but cheaper?
• 3:13
  People hide behind that.
• 3:14
  Then you price the failure.
• 3:16
  If the AI resolves a support ticket for 40 cents but creates a second contact because
• 3:21
  the answer was wrong, that second contact costs you the full human handling, say $6.
• 3:27
  Your real cost per resolution is now higher than doing it right the first time.
• 3:32
  Rework is the line item that turns a win into an embarrassment.
• 3:36
  Give me a case where someone got the measurement right.
• 3:39
  The pattern that works, and Klarna, the payments company, talked about this publicly,
• 3:44
  was treating the AI assistant like a new hire on probation.
• 3:48
  They tracked resolution rate, repeat contacts, and customer satisfaction side by side,
• 3:54
  monthly, and were willing to pull it back.
• 3:57
  The discipline wasn't the technology,
• 3:59
  it was refusing to declare victory from a single quarter.
• 4:02
  So the skill isn't building AI, it's the accounting around it.
• 4:06
  The article's real point.
• 4:08
  Measuring AI return is the most underrated skill in enterprise technology,
• 4:13
  and it's just honest arithmetic wearing a lab coat.
• 4:16
  Give me the one thing a listener does Monday morning.
• 4:19
  Take your flagship AI project and calculate cost per completed correct task,
• 4:25
  with the human cleanup included.
• 4:27
  If nobody in the building can produce that number by Friday,
• 4:30
  that's your answer.
• 4:31
  You're guessing, not saving.
• 4:33
  That's a wrap.
• 4:34
  Fresh AI briefings drop daily at MBA-training.com.