Latest / The Augur Dispatch / AI cache economics: check the bill before you buy

Transcript

AI cache economics: check the bill before you buy · The Augur Dispatch

• 0:00
  SPEAKER_00: In one fireworks coding agent run, almost every input token came from a cache, yet cached input charges still made up about 60% of the token bill.
• 0:10
  SPEAKER_00: Before you negotiate your next AI contract, find out whether you're negotiating the part you spend the most on.
• 0:16
  SPEAKER_00: This is the Augur dispatch for Thursday, October 22nd, 2026.
• 0:21
  SPEAKER_00: Fireworks tested DeepSeek V 4.1 Flash on DeepSuite.
• 0:26
  SPEAKER_00: It reports that 99.6% of input tokens were cashed, and those charges were 59.9% of a token bill of about 43 cents per task.
• 0:36
  SPEAKER_00: That's one host, model, and workload, not a rule for everyone's costs.
• 0:41
  SPEAKER_00: But it gives buyers a useful question.
• 0:44
  SPEAKER_00: A cheap cashed input rate can still account for most of your spending when an agent sends the same long context back again and again.
• 0:52
  SPEAKER_00: Picture a hypothetical engineering manager preparing a renewal.
• 0:55
  SPEAKER_00: She's comparing output token prices because the team's visible product is generated code.
• 1:01
  SPEAKER_00: Meanwhile, the agent repeatedly sends repository files and its growing work history along with each request.
• 1:07
  SPEAKER_00: The code is what she sees.
• 1:09
  SPEAKER_00: The repeated input may be what she's mostly paying for.
• 1:12
  SPEAKER_00: Her spreadsheet is accurate.
• 1:14
  SPEAKER_00: It may simply be very thorough about the wrong column.
• 1:18
  SPEAKER_00: Augur's view is that cached share of spend belongs in the procurement discussion before a provider comparison or a proposal to run the system yourself.
• 1:27
  SPEAKER_00: That distinction starts with what a cache does.
• 1:31
  SPEAKER_00: A model processes input before generating its response.
• 1:35
  SPEAKER_00: When the beginning of a new request matches something already processed, a prefix cache can reuse stored computation for that shared stretch.
• 1:44
  SPEAKER_00: It isn't retrieving a finished answer and pretending to reason.
• 1:48
  SPEAKER_00: The model still has new input to process and output to generate.
• 1:51
  SPEAKER_00: Reuse avoids some repeated work, which is precisely why you would want it.
• 1:56
  SPEAKER_00: DeepSeq reports an average input cache hit rate of 96% and says one cluster serves 155 trillion tokens a day.
• 2:05
  SPEAKER_00: Those are company reported figures, not independently audited results.
• 2:09
  SPEAKER_00: The daily total is cash-inclusive serve tokens, not 155 trillion newly generated tokens.
• 2:16
  SPEAKER_00: The 96% refers to input, not requests or all input and output combined.
• 2:22
  SPEAKER_00: DeepSeq's table separates fresh input processing from cash inclusive throughput, but it doesn't supply the averaging method behind that hit rate.
• 2:31
  SPEAKER_00: A buyer therefore has two different measurements to make.
• 2:35
  SPEAKER_00: The hit rate describes how much input benefits from reuse.
• 2:38
  SPEAKER_00: The spending share describes how much of the token bill still goes to that reuse.
• 2:43
  SPEAKER_00: Neither tells you, on its own, whether the service is overpriced.
• 2:48
  SPEAKER_00: Fireworks run makes the difference visible.
• 2:51
  SPEAKER_00: Cached input had a much lower unit price than fresh input, but there was enough of it to dominate the token bill.
• 2:57
  SPEAKER_00: That can be evidence of effective caching and a reason to negotiate its price at the same time.
• 3:03
  SPEAKER_00: DeepSeek's open source release, DeepInfer, adds useful engineering detail, but it doesn't let a buyer download those production economics.
• 3:12
  SPEAKER_00: The public release supports offline inference.
• 3:15
  SPEAKER_00: It is not a complete copy of DeepSeq's service.
• 3:18
  SPEAKER_00: The production key value cash management system isn't included as a complete component, nor are the full serving stage separation and load balancing needed to operate that service.
• 3:29
  SPEAKER_00: Those omissions matter to this story because the result we're interested in depends on retaining and finding reusable work while requests move through a live system.
• 3:38
  SPEAKER_00: The released engine is only part of that arrangement.
• 3:41
  SPEAKER_00: Hardware also limits who can evaluate it.
• 3:44
  SPEAKER_00: Documented support is for NVIDIA SM100 accelerators or Huawei Ascend 950 hardware.
• 3:51
  SPEAKER_00: This is specialized infrastructure, with operating requirements beyond installing a package on whatever machine happens to be spare.
• 3:59
  SPEAKER_00: Imagine that same hypothetical manager taking the release to her infrastructure lead.
• 4:04
  SPEAKER_00: He can study the engine, but he still needs to explain who will build the missing service components and keep them working.
• 4:10
  SPEAKER_00: The download has finished.
• 4:12
  SPEAKER_00: The staffing discussion hasn't.
• 4:14
  SPEAKER_00: Open prefix caching already existed in projects such as VLLM.
• 4:19
  SPEAKER_00: DeepInfer adds another implementation to that ecosystem.
• 4:22
  SPEAKER_00: It doesn't make cache efficiency verifiable for the first time, and its code alone cannot reproduce DeepSeek's reported hit rate.
• 4:30
  SPEAKER_00: For a team with the right hardware and engineering capacity, evaluation may be worthwhile.
• 4:35
  SPEAKER_00: For everyone else, the immediate opportunity is a better question for the provider they already use.
• 4:42
  SPEAKER_00: The strongest objection to focusing on cashed charges is that a provider may be doing an excellent job precisely because those charges dominate.
• 4:51
  SPEAKER_00: Without reuse, much of that input could require fresh processing at a higher rate.
• 4:56
  SPEAKER_00: A high cashed share of spending can coexist with a low total bill, and a different provider could lower that share while making the completed task more expensive.
• 5:06
  SPEAKER_00: Caged service isn't free to operate.
• 5:09
  SPEAKER_00: Stored computation takes space.
• 5:11
  SPEAKER_00: The service has to find it and move it where it's needed.
• 5:14
  SPEAKER_00: Capacity and reliability still matter.
• 5:16
  SPEAKER_00: DeepSeek's own deployment documentation describes dedicated storage and failure handling, not a costless shortcut.
• 5:23
  SPEAKER_00: Calling cash hits cheating would miss the point.
• 5:27
  SPEAKER_00: Buyers want useful work delivered reliably, not a ceremonial repetition of calculations that have already been done.
• 5:34
  SPEAKER_00: A managed provider can also absorb demand swings and operational work that a smaller team would struggle to handle economically.
• 5:41
  SPEAKER_00: Paying someone else to do that may be a sound decision, even when the underlying engine is open source.
• 5:47
  SPEAKER_00: This is why Augur isn't treating the fireworks result as evidence of excessive margins, or DeepInfer as a promise of savings from running it yourself.
• 5:56
  SPEAKER_00: The customer's token invoice does not reveal the provider's whole operating cost.
• 6:01
  SPEAKER_00: The defensible conclusion is narrower.
• 6:04
  SPEAKER_00: If cashed input is a large share of your spending, its rate and availability deserve attention in a renewal.
• 6:10
  SPEAKER_00: Then, compare total cost per successful task at the quality and response time your users need, rather than trying to make one percentage look better.
• 6:20
  SPEAKER_00: Return once more to our hypothetical manager.
• 6:23
  SPEAKER_00: Before signing, she asks the developer responsible for the agent to break out a week of usage and its charges.
• 6:29
  SPEAKER_00: This time, they can see whether the apparently attractive output discount would materially change their bill.
• 6:35
  SPEAKER_00: They might discover that cached input barely matters for their workload.
• 6:40
  SPEAKER_00: That would be useful too.
• 6:41
  SPEAKER_00: The fireworks benchmark would have prompted a check, not dictated an answer.
• 6:46
  SPEAKER_00: If reuse does dominate, a small trial can compare an alternative offer using representative coding tasks and the same acceptance tests.
• 6:54
  SPEAKER_00: Keep the data permissions unchanged, count failed attempts and retries, and include the work required to operate either option.
• 7:02
  SPEAKER_00: The point is to improve the cost of a successful result without quietly sacrificing quality or making the developer wait longer for it.
• 7:10
  SPEAKER_00: A modest trial can answer that question before anyone commits to a migration.
• 7:15
  SPEAKER_00: Learning how your own agent uses context could give your team a durable advantage.
• 7:20
  SPEAKER_00: A public price cut is available to competitors too.
• 7:24
  SPEAKER_00: Knowing which repeated work your application needs and which it can safely avoid is knowledge you build through operating it.
• 7:31
  SPEAKER_00: Adopt the measurement now.
• 7:33
  SPEAKER_00: Pilot a change only when your own bill identifies a worthwhile target.
• 7:37
  SPEAKER_00: A large cash percentage is neither a purchasing recommendation nor a complaint by itself.
• 7:44
  SPEAKER_00: Signals on the clock.
• 7:46
  SPEAKER_00: Watch for an outside organization to publish measurements from running DeepInfer on hardware it controls.
• 7:52
  SPEAKER_00: That would test whether particular performance results travel beyond DeepSeq under the conditions reported.
• 7:58
  SPEAKER_00: It wouldn't settle the economics of a complete service.
• 8:01
  SPEAKER_00: There is already a relevant comparison from VLLM.
• 8:04
  SPEAKER_00: Its team reports decode throughput within 6% of DeepSeq's published results under the stated configuration.
• 8:11
  SPEAKER_00: That's a measurement of VLLM against a published baseline, not an independent deployment of DeepInfer.
• 8:18
  SPEAKER_00: Augur's bounded research found no qualifying outside DeepInfer run.
• 8:22
  SPEAKER_00: That isn't proof none exists.
• 8:24
  SPEAKER_00: The dated prediction and its exact criteria are in the episode notes.
• 8:28
  SPEAKER_00: The written brief with every source is at augerdispatch.ai.
• 8:33
  SPEAKER_00: You don't need to wait for that outside result to understand your invoice.
• 8:37
  SPEAKER_00: Before committing to a different provider or to running your own service, know what you're paying to reuse and what it would cost to deliver the same useful work another way.
• 8:47
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 8:50
  SPEAKER_00: That is the Augur Dispatch for Thursday, October 22nd, 2026.
• 8:55
  SPEAKER_00: Thank you for listening.