Latest / MBA Training AI / Reasoning models and when to use them: the hype is ahead of the practice

Transcript

Reasoning models and when to use them: the hype is ahead of the practice · MBA Training AI

• 0:00
  Welcome to the Leaders Insights Podcast.
• 0:07
  Today's episode, reasoning models and when to use them.
• 0:11
  The hype is ahead of the practice.
• 0:13
  Everyone's telling me to route every serious prompt through a reasoning model.
• 0:17
  You're about to tell me that's dumb.
• 0:19
  Go.
• 0:20
  It's not dumb.
• 0:21
  It's expensive and slow for problems that never needed it.
• 0:25
  A reasoning model, one that generates a chain of intermediate steps before answering, like
• 0:30
  OpenAI's O3 or Google's Gemini 2.0 Flash Thinking, bills you for all that thinking.
• 0:37
  You're paying for tokens the user never sees.
• 0:39
  How much are we actually talking?
• 0:42
  Give me a number that stings.
• 0:43
  On the reasoning tiers, you're often paying 3 to 10 times more per request than a standard
• 0:48
  model, and latency jumps from under a second to 15 or 20.
• 0:53
  I've seen a support triage pipeline where swapping to a reasoning model raised the monthly
• 0:58
  bill from about $4,000 to $31,000 for classification that a smaller model nailed at 98% accuracy
• 1:06
  anyway.
• 1:07
  So the model was thinking hard about a problem that didn't require thought.
• 1:11
  Exactly.
• 1:12
  Classifying is this ticket about billing.
• 1:14
  Is pattern matching?
• 1:15
  There's no multi-step logic to reason through.
• 1:19
  You're asking a mathematician to alphabetize your bookshelf.
• 1:23
  It'll do it slowly and charge you consultant rates.
• 1:26
  Fine.
• 1:27
  Where does the reasoning actually earn its keep?
• 1:29
  When the answer depends on a sequence of dependent steps where an early mistake poisons
• 1:33
  the rest.
• 1:35
  Multi-step math, code that has to compile and pass tests, logic puzzles, planning tasks
• 1:41
  with constraints.
• 1:42
  Anything where the model benefits from checking its own work mid-flight.
• 1:46
  Give me the concrete case.
• 1:48
  A fintech team I know does reconciliation, matching thousands of transactions across ledgers
• 1:53
  with weird edge cases, partial refunds, currency conversions.
• 1:58
  A standard model hit roughly 70% correct matches.
• 2:02
  The reasoning model got to 94% because it could hold intermediate state and catch its
• 2:07
  own contradictions.
• 2:08
  There, paying 10 times more is trivial next to a human rechecking mismatches.
• 2:13
  So the durable principle is, what matched the model to the shape of the problem?
• 2:18
  Matched the compute to the depth of the reasoning required.
• 2:22
  This isn't new.
• 2:23
  The best operators have run this playbook for a decade with regular software.
• 2:27
  You don't put your whole database behind the most expensive query engine.
• 2:31
  You profile the workload, find the 5% of requests that are genuinely hard, and route
• 2:36
  only those to the heavy machinery.
• 2:38
  But how do I know in advance which requests are the hard 5%?
• 2:42
  That's the uncomfortable part.
• 2:44
  You don't guess, you measure.
• 2:46
  Run a cheap model first and add a confidence check or a validator.
• 2:50
  If the cheap model is sure, ship it.
• 2:53
  If it's uncertain or the output fails a test, code that won't compile, a total that doesn't
• 2:58
  balance, escalate that single request to the reasoning model.
• 3:03
  It's called a cascade, and it's just triage.
• 3:06
  A cascade being a fallback ladder.
• 3:09
  Cheap model then medium, then the expensive one.
• 3:11
  Right, and the economics are lovely.
• 3:14
  If 85% of your traffic clears the cheap tier, you're paying premium prices on 15% of requests
• 3:20
  instead of 100%.
• 3:23
  That same reconciliation team cuts spend by roughly 60% versus routing everything to 03
• 3:29
  and lost less than a point of accuracy.
• 3:31
  Vendors numbers or theirs?
• 3:33
  Theirs from their own billing dashboards, not a model provider's benchmark, which
• 3:38
  you should always distrust because the provider profits when you use the biggest model.
• 3:42
  Their marketing quietly assumes maximum consumption.
• 3:45
  So the hype is basically a pricing strategy dressed as best practice?
• 3:50
  Partly yes.
• 3:52
  Use reasoning everywhere is convenient for the people selling reasoning tokens.
• 3:56
  It's terrible engineering advice.
• 3:58
  The visible thinking is genuinely impressive, and that's exactly why people overapply
• 4:03
  it.
• 4:04
  It looks smart, so it feels safer.
• 4:06
  Being safe is not a budget line.
• 4:08
  What about the argument that models get cheaper, so why bother optimizing now?
• 4:13
  They do get cheaper, but your traffic grows faster.
• 4:17
  Optimizing your routing is a habit, not a one-time fix.
• 4:20
  And the teams that build the measurement muscle now will still be ahead when prices drop
• 4:25
  because they'll know where the hard problems actually live.
• 4:28
  Give me the one thing to do Monday morning.
• 4:31
  Take your top three AI workflows, log what fraction of requests are genuinely multi-step,
• 4:37
  and route only those to a reasoning model.
• 4:40
  Everything else goes to the cheap tier with a validator.
• 4:43
  You'll cut cost and probably improve speed the same week.
• 4:46
  Done.
• 4:47
  Want to know where you actually stand?
• 4:49
  Take the AI self-assessment at MBA-training.com.