Latest / AI Fire Daily / #577 Neil: Reduce Claude Sonnet 5 Costs With Smarter Model Routing

Transcript

#577 Neil: Reduce Claude Sonnet 5 Costs With Smarter Model Routing · AI Fire Daily

• 0:00
  You have access to one of the most brilliant
• 0:03
  AI models ever created, and you are likely using
• 0:07
  it right now to do the digital equivalent of
• 0:09
  tying your shoes. Yeah, it's wild. We basically
• 0:12
  treat a massive supercomputer like a cheap pocket
• 0:15
  calculator. We do. We reflexively throw our absolute
• 0:19
  easiest problems at the smartest AI available,
• 0:22
  and we bleed money in the process without even
• 0:25
  realizing it. Beat. It is a quiet, expensive
• 0:28
  trap. It drives me crazy, honestly. But today,
• 0:31
  we're fixing that. Right. Welcome to today's
• 0:34
  Deep Dive. If you are listening to this, you
• 0:36
  have probably opened your developer dashboard
• 0:39
  at the end of the month. Oh, yeah. You saw the
• 0:41
  bill for ClaudeSonic 5, and you felt that immediate
• 0:45
  pit in your stomach. I know that feeling all
• 0:46
  too well. So our mission today is dissecting
• 0:50
  exactly how to optimize those workflows. You
• 0:52
  should only pay for premium AI power when you
• 0:54
  actually need it. Today is all about building
• 0:56
  a highly efficient system. I mean, we're going
• 0:58
  to audit where you're currently wasting money
• 1:00
  first. Then we'll draw a hard line on what actually
• 1:04
  needs sun at five. And then this is my absolute
• 1:07
  favorite part. We are going to build an automated
• 1:10
  router. A system that makes the cost decisions
• 1:12
  for you in real time. Exactly. And finally, we'll
• 1:15
  cover testing and safely migrating your tasks.
• 1:18
  But before we can cut costs, we have to slow
• 1:20
  down. We have to actively observe the moving
• 1:24
  parts of our daily workflows. Right. You can't
• 1:26
  optimize a system if you don't understand its
• 1:28
  basic mechanics. Exactly. Breaking down a workflow
• 1:32
  step by step is crucial. Think about a standard
• 1:36
  workflow you run every single day. You gather
• 1:38
  your research material, you summarize that research,
• 1:41
  you extract key data points, you write a draft,
• 1:44
  and finally you review it. Every single one of
• 1:47
  those steps currently hits the most expensive
• 1:49
  model. Pete, I actually have a vulnerable admission
• 1:52
  to make right here. Oh, let's hear it. I still
• 1:54
  wrestle with model costs myself. all the time.
• 1:57
  It is so incredibly easy to just, you know, default
• 2:00
  to the biggest model. Oh, completely. It feels
• 2:02
  safer. It does. You just ignore the bill until
• 2:04
  it finally hurts. You are definitely not the
• 2:06
  only one doing that. I mean, it's human nature
• 2:08
  to choose the path of least resistance. Yeah.
• 2:10
  That is exactly why we use a specific audit prompt
• 2:13
  to break that habit. You rate each of your workflow
• 2:16
  tasks on a strictly defined 1 to 5 scale. I like
• 2:20
  the idea of quantifying it. But what are the
• 2:23
  actual dimensions we are measuring here? So,
• 2:26
  we measure three distinct things for every single
• 2:28
  task. We rate the raw reasoning required, we
• 2:31
  rate the potential error risk, and we rate the
• 2:35
  context requirement. That context requirement
• 2:37
  is a really fascinating metric to isolate. Yeah,
• 2:40
  it really is. It essentially measures how much
• 2:42
  background information the model needs to hold
• 2:44
  in its working memory to do the job right. Right.
• 2:47
  And the results from this structured audit are
• 2:49
  often highly counterintuitive. Well, I would
• 2:52
  assume data extraction is a total no -brainer
• 2:54
  for the cheap tier. It is just finding a number
• 2:57
  on a page and pulling it out. That requires basically
• 3:00
  zero reasoning. And that is exactly where the
• 3:03
  trap usually hides. Yeah. Simple data extraction
• 3:05
  looks incredibly cheap on the surface. But when
• 3:08
  you rate it on error risk, it scores massively
• 3:11
  high. Ah, because of the downstream blast radius?
• 3:14
  Precisely. If it pulls the wrong revenue number,
• 3:16
  the subsequent financial report is entirely compromised.
• 3:19
  Exactly. a bad date or a missing client name
• 3:22
  breaks everything downstream. So that simple
• 3:25
  extraction might actually need to stay on Sonnet
• 3:28
  5. Or at the very least, you have to engineer
• 3:30
  a strict manual validation step. Right. Now conversely,
• 3:34
  basic text summarization feels like a complex
• 3:37
  premium task. But the audit shows it's an easy
• 3:41
  switch. Because it requires low reasoning and
• 3:43
  it is very easy for you to spot check. Exactly.
• 3:45
  To sex silence. Are humans actually good at accurately
• 3:48
  rating this error risk ourselves? Or do we just
• 3:51
  suffer from massive cognitive blind spots? We
• 3:54
  are honestly quite terrible at evaluating our
• 3:56
  own risks. We routinely underestimate risk until
• 4:00
  something completely breaks in production. Yeah.
• 4:02
  That is why this prompt framework is so valuable.
• 4:04
  It acts as a totally objective mirror for your
• 4:07
  workflow. So the prompt forces us to see risks
• 4:10
  our biases usually hide. Perfectly said. It takes
• 4:13
  the emotional guesswork out of the entire equation.
• 4:16
  Now that the audit shows us our blind spots,
• 4:17
  we can move forward. How do we actually draw
• 4:20
  the line between premium and budget tasks? Well,
• 4:23
  using Sonnet 5 for absolutely everything is just
• 4:26
  incredibly wasteful. It is like using a supercomputer
• 4:29
  as a calculator. Oh, I love that analogy. It's
• 4:31
  so true. Routing tasks like this feels a lot
• 4:34
  like hospital triage. You do not send a guy with
• 4:38
  a scraped knee to the top neurosurgeon. Right.
• 4:40
  You send him to the triage nurse. Exactly. That
• 4:42
  frees up the surgeon for the complex traumas.
• 4:45
  We are essentially building an AI triage nurse.
• 4:48
  That is a brilliant way to frame it. We have
• 4:50
  to identify what truly must stay on the neurosurgeon's
• 4:53
  table. Complex research is a non -negotiable
• 4:56
  premium task. Difficult coding definitely stays
• 5:00
  on the premium tier, and multi -step agent workflows
• 5:03
  must also stay on Sonnet 5. Because agent workflows
• 5:06
  require the model to independently plan, execute,
• 5:09
  and course -correct. That demands high -level
• 5:12
  autonomous reasoning. Exactly. Strategic analysis
• 5:15
  is another one that needs the big brain. The
• 5:17
  source material gives a really vivid example
• 5:20
  of this. What is the example? Imagine you're
• 5:22
  asking the AI to compare five different competitor
• 5:25
  reports. Wow! That is a massive amount of dense
• 5:29
  often contradictory information to hold in memory
• 5:32
  It is you are asking the model to cross reference
• 5:36
  conflicting claims You're asking it to judge
• 5:39
  the underlying evidence behind those claims right
• 5:41
  and then you wanted to recommend actionable new
• 5:45
  market opportunities. That requires deep semantic
• 5:48
  understanding. It has to weigh the relationships
• 5:51
  across a massive context window. It also needs
• 5:53
  to notice when the provided data is simply weak.
• 5:56
  It has to flag when it cannot responsibly make
• 5:58
  a clear recommendation. That kind of nuanced
• 6:01
  judgment is exactly why you pay for Sonnet 5.
• 6:04
  If you move that kind of synthesis to a cheaper
• 6:06
  model, the quality will absolutely drop off a
• 6:09
  cliff. Cheaper models suffer from severe attention
• 6:11
  dilution. They just lose the thread of the argument.
• 6:13
  And then you just end up checking all the original
• 6:15
  sources again manually, which defeats the entire
• 6:18
  purpose of automating it in the first place.
• 6:20
  Right, you're fixing the weak reasoning yourself.
• 6:22
  The money you save on the back end is completely
• 6:24
  wasted by your own time. So let's contrast that
• 6:27
  with the scraped knees. What can safely move
• 6:30
  to the triage nurse models? You have a lot of
• 6:33
  great options for the cheaper tier. Categorizing
• 6:36
  text into predefined labels is a perfect candidate.
• 6:39
  OK. Formatting raw text into a fixed predictable
• 6:42
  structure works beautifully. And basic summarization,
• 6:46
  as we established, is very safe. Let's look at
• 6:49
  formatting into predictable structures, like
• 6:51
  processing a standard business invoice. Exactly.
• 6:54
  You only need a few specific details pulled from
• 6:57
  the page. You need the company name, the invoice
• 7:00
  date, and the total amount. You want to extract
• 7:02
  it into a valid JSON format. meaning a simple
• 7:05
  text structure computers read easily. So instead
• 7:08
  of returning a messy conversational paragraph,
• 7:10
  it just returns the raw formatted variables.
• 7:12
  Right. The prompt tells the cheaper model exactly
• 7:15
  what to do. If a field is missing, just return
• 7:18
  null. Do not try to guess or hallucinate missing
• 7:21
  values. This request requires drastically less
• 7:24
  reasoning than synthesizing competitor reports.
• 7:27
  Night and day difference. The cheaper model is
• 7:30
  more than capable of this pattern matching, and
• 7:32
  the result is incredibly easy for you to verify.
• 7:35
  Because you just look at the invoice and check
• 7:37
  the numbers. Exactly. So we are looking at a
• 7:40
  core trio of decision -making criteria here.
• 7:42
  Yes. How much raw reasoning is actually needed?
• 7:45
  How easy is the final output to quickly check?
• 7:48
  And what is the blast radius of a single mistake?
• 7:51
  Is there a danger in defining these tasks too
• 7:54
  broadly? Like, assuming all research is automatically
• 7:57
  complex tasks? Oh, there's a massive danger in
• 7:59
  doing that. Granularity is the absolute key to
• 8:02
  making this writing work. Researching a simple
• 8:04
  textbook definition is a very cheap task. Synthesizing
• 8:07
  an emerging market trend strictly requires Sonnet
• 8:10
  5. Right, the devil is in the details. Nuance
• 8:13
  dictates the model tier. You have to look closely
• 8:15
  at the specific mechanics of the job. So, we
• 8:18
  have drawn these lines manually now. But doing
• 8:21
  this by hand for every single task is exhausting.
• 8:25
  Yeah, nobody wants to do that. We need to build
• 8:27
  an automated traffic controller. We simply do
• 8:30
  not want to actively think about this every time
• 8:32
  we execute a workflow. And that is the real beauty
• 8:35
  of a dedicated routing system. We shift away
• 8:38
  from sending absolutely everything to Claude
• 8:40
  Sonnet 5 by default. Right. We create a much
• 8:43
  smarter automated flow. right at the entry point.
• 8:46
  It goes from the task to a complexity check,
• 8:48
  then it routes to a cheaper model or to Sonnet
• 8:51
  5. But wait, if the router has to read the prompt
• 8:55
  to decide if it is complex, isn't it basically
• 8:57
  doing the heavy lifting already? How does it
• 8:59
  judge complexity without actually solving the
• 9:01
  problem? That is the clever part. The router
• 9:04
  evaluates the instruction's complexity footprint,
• 9:06
  not the final answer. It is essentially pattern
• 9:09
  matching the request type. Ah, I see. We achieve
• 9:12
  this by using a highly constrained routing prompt
• 9:14
  on a fast, cheap model. So the router just looks
• 9:17
  for trigger words or structural complexity in
• 9:19
  the prompt itself. Exactly. The router analyzes
• 9:22
  the incoming task, then it outputs a tiny, simple
• 9:27
  piece of code. It just outputs route .cheap,
• 9:31
  or routes on it. Right, and it includes one short
• 9:34
  sentence explaining the reason for the decision.
• 9:37
  Which gives you an audit trail. This logic plugs
• 9:40
  seamlessly into automation tools you might already
• 9:42
  use to connect your apps, things like NaN or
• 9:46
  Make. Two -sec silence, whoa. What? Imagine scaling
• 9:50
  this to millions of automated decisions. It's
• 9:52
  mind -blowing. You are perfectly optimizing brain
• 9:54
  power and budget in real time, completely hands
• 9:57
  -off. It is like building a highly efficient
• 9:59
  digital nervous system for your entire business.
• 10:01
  It really is a beautiful thing to watch run.
• 10:04
  It intelligently evaluates every single request
• 10:06
  before spending your money. API calls requests
• 10:09
  your computer makes to the AI become incredibly
• 10:11
  cheap. But does adding a router step introduce
• 10:14
  a new problem? Does it just add a new point of
• 10:16
  failure or unnecessary latency into the workflow?
• 10:19
  There is a very slight delay up front. That is
• 10:21
  absolutely true. But a microsecond routing prompt
• 10:24
  is incredibly cheap and lightning fast. Right.
• 10:27
  Because it is not generating a long answer, it
• 10:29
  saves massive latency and cost on the heavy backend
• 10:32
  execution. A tiny delay up front saves massive
• 10:35
  time and money on the backend. Exactly. It is
• 10:37
  a micro investment that pays off instantly. So
• 10:40
  the automated router is built and theoretically
• 10:43
  ready to go, but flipping the switch blindly
• 10:46
  across your entire workflow is dangerous. Extremely
• 10:49
  dangerous. We have to test the waters carefully
• 10:51
  before aggressively moving tasks away from Sonnet
• 10:53
  5. You should never migrate simply because a
• 10:57
  model is cheaper. That is a guaranteed recipe
• 11:00
  for broken workflows and massive frustration.
• 11:03
  We need a rigorous testing framework before we
• 11:05
  commit to the new routing. And the testing framework
• 11:07
  is actually quite elegant. You take a group of
• 11:10
  real, typical tasks from your daily workflow.
• 11:13
  You run the exact same input through CloudSonic
• 11:15
  5. That becomes response A. And you run it through
• 11:18
  the cheaper model. That becomes response B. Then
• 11:21
  you use a prompt to evaluate them against each
• 11:23
  other. You are essentially using a large language
• 11:25
  model as an impartial judge. Yes. You ask a powerful
• 11:29
  model to score both responses side -by -side.
• 11:32
  You are checking for strict accuracy, completeness,
• 11:35
  and instruction following. You want to objectively
• 11:37
  identify any important errors or missing details
• 11:40
  in the cheaper model's output. This highlights
• 11:43
  the hidden, often completely ignored, cost of
• 11:46
  human correction. This is the single most crucial
• 11:48
  point of the whole process. The cheaper model
• 11:51
  might save you a few pennies on API costs per
• 11:53
  run. But if it drops one key detail, a human
• 11:56
  might spend 20 minutes tracking it down. And
• 11:59
  20 minutes of human time completely obliterates
• 12:01
  fractions of a cent in API savings. You are actually
• 12:04
  bleeding more money on the deal. Precisely. If
• 12:07
  you routinely have to fix missing details or
• 12:09
  correct formatting, keep the task on Sonnet 5.
• 12:12
  Right. The cheaper model just needs to work flawlessly
• 12:15
  for one specific job. It does not need to possess
• 12:18
  the general intelligence to replace Sonnet 5
• 12:20
  entirely. It just needs to do its one assigned
• 12:23
  chore. Perfectly. How many tasks should someone
• 12:27
  realistically test? When do you finally feel
• 12:29
  confident in the cheaper model's actual baseline
• 12:31
  reliability? I strongly advise starting with
• 12:34
  a small, highly diverse batch, use maybe 50 to
• 12:38
  100 typical daily tasks that represent your actual
• 12:41
  workload. You need just enough volume to spot
• 12:44
  repeating systemic patterns of failure. Test
• 12:47
  a small diverse batch until you spot the model's
• 12:50
  breaking point. Exactly. Find exactly where it
• 12:52
  breaks and you map its true limits. So we have
• 12:54
  audited, we have routed, and we have tested.
• 12:56
  Now it is time to safely deploy this system into
• 12:59
  the wild. The fun part. We need to execute this
• 13:01
  without breaking our daily operations. The underlying
• 13:04
  philosophy here is to migrate very slowly. There
• 13:08
  is no magic one size fits all traffic split for
• 13:11
  every single workflow. The source material shares
• 13:14
  a really fascinating real -world setup for this
• 13:17
  migration. Oh, it is a brilliant example of hybrid
• 13:19
  model architecture. One user setup used 80 %
• 13:22
  Sonnet 5 for the actual granular execution of
• 13:25
  tasks, but they used 20 % Opus 4 .8 strictly
• 13:29
  for the high -level planning phase. Because Opus
• 13:31
  4 .8 excels at overarching strategic foresight.
• 13:35
  It sets the complex architecture, and Sonnet
• 13:37
  5 acts as the highly capable engine executing
• 13:40
  the steps. And what was the financial... result
• 13:43
  of that specific hybrid split. They cut their
• 13:46
  monthly clod spending by about 50 percent and
• 13:49
  they achieve that massive cost reduction with
• 13:51
  absolutely no clear drop in final output quality.
• 13:54
  That is incredible. But how do we build our own
• 13:57
  custom deployment plan? You use a dedicated migration
• 14:00
  prompt fed with your rigorous test results. This
• 14:04
  prompt dictates your precise migration priority
• 14:07
  moving forward. Right. It tells you exactly what
• 14:09
  percentage of traffic to test on the cheaper
• 14:11
  model first. It also establishes validation checks.
• 14:14
  and clear automated fallback rules. Yes. The
• 14:18
  final elegant operational flow looks like this.
• 14:20
  The routed task goes to the cheaper model first.
• 14:23
  The cheaper model generates an outplot. Then,
• 14:26
  that output hits a strict validation check. Finally,
• 14:28
  it falls back to Claude Sonnet 5 only if it fails
• 14:31
  that check. You are keeping Sonnet 5 as the premium,
• 14:34
  highly reliable safety net. It is always there
• 14:37
  to catch the complex edge cases when the triage
• 14:39
  nurse gets overwhelmed. It guarantees you never
• 14:42
  sacrifice final quality for cost. What does a
• 14:45
  good validation check actually look like in practice?
• 14:49
  How do we automatically check the work before
• 14:51
  escalating it up to Sonnet 5? It is usually a
• 14:54
  very simple programmatic structural check. For
• 14:57
  instance, you might run a script to ensure the
• 14:59
  JSON schema is strictly being followed. Or you
• 15:03
  write a basic rule that checks that all the absolutely
• 15:06
  required data fields... are present and not empty.
• 15:10
  So validation is just a quick automated proofread
• 15:13
  before calling the heavy hitter. That is exactly
• 15:15
  it. A cheap instant proofread before you pay
• 15:17
  for an expensive rewrite. We have covered a massive
• 15:20
  amount of technical ground today. Let's recap
• 15:22
  the core philosophy we have been unpacking. Claude
• 15:25
  Sonnet 5 is an incredibly powerful premium cognitive
• 15:29
  tool, but true operational efficiency is using
• 15:32
  it only where it adds real irreplaceable value.
• 15:35
  We have to stop paying premium prices for basic
• 15:38
  budget level tasks. It is all about purposefully
• 15:41
  matching the tool to the specific job. I highly
• 15:44
  encourage you to look at your own workflows today.
• 15:47
  Run a quick mental audit of the specific tasks
• 15:49
  you currently automate. Ask yourself if you are
• 15:52
  truly optimizing the architecture or just defaulting
• 15:55
  to the easiest option. It is so incredibly easy
• 15:58
  to fall into that default trap. I want to leave
• 16:01
  you with a final provocative thought today. Let's
• 16:05
  extend this entire routing idea beyond just artificial
• 16:07
  intelligence and API overhead. I want you to
• 16:11
  seriously consider your own biological brain
• 16:14
  power. Oh, I really like where this is going.
• 16:16
  Think about your daily mental energy as a finite
• 16:19
  API budget. How much of your biological energy
• 16:22
  is spent on cheaper model tasks? How much of
• 16:25
  your day goes to basic formatting, simple data
• 16:27
  extraction, or answering repetitive emails? We
• 16:30
  all do so much robotic low -level work every
• 16:32
  single day without questioning it. Exactly. What
• 16:35
  if you rigorously audited your own mental workflow?
• 16:38
  What if you actively delegated or automated those
• 16:41
  lower -tier tasks away from yourself? Wow. Imagine
• 16:44
  freeing up your own internal Sonic 5. You could
• 16:47
  reserve that premium biological energy purely
• 16:49
  for deep strategic thinking and complex problem
• 16:52
  -solving. Building a router for your own brain.
• 16:55
  That is the ultimate efficiency system right
• 16:57
  there. It really is. Thank you for joining us
• 16:59
  on this deep dive today. We will catch you on
• 17:01
  the next one.