Latest / MBA Training AI / How Klarna rewired its support operations with disciplined prompt engineering

Transcript

How Klarna rewired its support operations with disciplined prompt engineering · MBA Training AI

• 0:00
  Welcome to the Leaders Insights podcast.
• 0:07
  Today's episode, how Klarna rewired its support operations with discipline prompt engineering.
• 0:13
  Klarna claimed its AI assistant did the work of 700 full-time support agents.
• 0:17
  That number gets thrown around like gospel.
• 0:20
  Was it real or vendor theater?
• 0:22
  Mostly real and worth interrogating.
• 0:26
  Klarna said the assistant handled two-thirds of its customer service chats in its first
• 0:30
  month back in early 2024, roughly 2.3 million conversations.
• 0:36
  The 700-agent figure was Klarna's own estimate of equivalent human workload.
• 0:42
  So treat it as a company claim, not an audited fact.
• 0:45
  Fine, but every company has a chatbot that annoys people.
• 0:49
  What did Klarna actually do differently?
• 0:51
  They treated the prompt, the written instruction you give the model, as an engineering artifact
• 0:56
  not a suggestion box.
• 0:58
  Most companies write a prompt like a wish.
• 1:00
  Klarna wrote it like a contract.
• 1:02
  Slow down.
• 1:03
  Pretend I've never written a prompt in my life.
• 1:05
  What is one mechanically?
• 1:07
  It's the text you feed the model before it answers.
• 1:10
  Imagine hiring a temp who has read the entire internet but knows nothing about your company.
• 1:16
  And the prompt is the one-page brief you hand them on their first morning.
• 1:20
  Vague brief vague temp.
• 1:21
  So the whole game is writing a better brief?
• 1:23
  The whole game is writing a brief that survives contact with a real customer.
• 1:28
  Klarna's discipline had three habits worth stealing.
• 1:31
  First, they scoped the model tightly, told it exactly which tasks it may attempt, and
• 1:37
  crucially, which ones to refuse and hand to a human.
• 1:40
  That sounds like admitting your expensive AI can't do the job.
• 1:44
  It's the opposite.
• 1:45
  The refund fraud edge cases, the grieving customer situations.
• 1:49
  You route those to humans on purpose.
• 1:52
  A model that knows its limits beats one that confidently invents a return policy that doesn't
• 1:57
  exist.
• 1:58
  That invention has a name, hallucination, when the model states something false with
• 2:03
  total conviction.
• 2:04
  Second habit?
• 2:05
  Grounding.
• 2:06
  They didn't let the model answer from memory.
• 2:09
  Every response was tied to Klarna's actual policy documents, and the customer's real
• 2:14
  account data pulled in at the moment of the question.
• 2:17
  So it's not guessing your order status, it's reading it.
• 2:20
  Even if the policy changes?
• 2:22
  You update the document, not the model.
• 2:24
  That's the quiet genius.
• 2:26
  When Klarna changed a refund window, a human-edited one source file and every future answer reflected
• 2:32
  it.
• 2:33
  No retraining, no engineers, no six-week project.
• 2:36
  Third?
• 2:37
  Relentless evaluation.
• 2:39
  They built test sets, collections of real past conversations with known correct outcomes,
• 2:44
  and ran every prompt change against them before shipping.
• 2:48
  Change one sentence in the brief, rerun a few thousand scenarios, see what broke.
• 2:53
  That's the part nobody does, isn't it?
• 2:55
  That's the part everybody skips.
• 2:57
  Then wonders why the thing goes sideways in production.
• 3:01
  Prompting without evaluation is editing a novel with the lights off.
• 3:04
  You feel productive.
• 3:06
  You have no idea what you've done.
• 3:07
  Let me push.
• 3:08
  Klarna also quietly rehired some humans later.
• 3:11
  Doesn't that puncture the whole story?
• 3:13
  It complicates the triumphalism and good.
• 3:16
  In 2025, their CEO admitted they'd cut too aggressively and quality suffered on complex
• 3:22
  cases.
• 3:23
  The lesson isn't AI failed.
• 3:25
  It's that the AI was excellent at the repetitive 60% and mediocre at the messy 40, and they'd
• 3:32
  blurred the line.
• 3:34
  Discipline means knowing which bucket a request falls into before you answer it.
• 3:38
  So for an exec who isn't fintech and doesn't have Klarna's engineers, what actually transfers?
• 3:43
  The method, not the scale.
• 3:45
  Any team drowning in repetitive questions, internal IT tickets, HR queries, first-line
• 3:51
  sales can apply the same three moves.
• 3:54
  Scope tightly, ground in your real documents, and evaluate every change against real past
• 3:59
  cases.
• 4:00
  And the thing most people get wrong on day one?
• 4:03
  They optimize the clever phrasing and ignore the plumbing.
• 4:07
  They spend a week wording the prompt beautifully and zero time deciding what the model isn't
• 4:11
  allowed to touch.
• 4:13
  Give me the one thing to do Monday morning.
• 4:15
  Take your 50 most common customer or employee questions, write down the correct answer to
• 4:20
  each yourself, and use that list as your scorecard.
• 4:24
  Before any prompt goes live, it has to pass those 50.
• 4:27
  If you can't write the answers, your AI can't either.
• 4:31
  And now you know that for free instead of finding out from an angry customer.
• 4:35
  That's it.
• 4:36
  The AI Decision Tools are live at MBA-training.com.