Latest / The Augur Dispatch / DeepSeek vs Astra: Test the Coding Savings on Your Own Work

Transcript

DeepSeek vs Astra: Test the Coding Savings on Your Own Work · The Augur Dispatch

• 0:07
  SPEAKER_00: Fireworks has a much cheaper coding model now lands in Astra's accuracy band.
• 0:12
  SPEAKER_00: Augur's recommendation is to test the saving on your own code before changing the default.
• 0:17
  SPEAKER_00: The vendor's price is a reason to run a comparison, not a promise, about your next bill.
• 0:22
  SPEAKER_00: This is the Augur Dispatch for Tuesday, September 15th, 2026.
• 0:27
  SPEAKER_00: A quick disclosure before we begin.
• 0:29
  SPEAKER_00: Portions of the Augur Dispatch Podcast are generated using artificial intelligence.
• 0:33
  SPEAKER_00: The written dispatch contains the full source trail and verification notes.
• 0:37
  SPEAKER_00: On Fireworks Software Benchmark, DeepSeek V 4.1 Flash solved about 74% of tasks on the first attempt, essentially level with GPT-6 Astro.
• 0:48
  SPEAKER_00: The reported cost was 43 cents per task against roughly$6.52 for Astro.
• 0:54
  SPEAKER_00: Imagine an engineering manager looking at that difference while the team has a queue of routine fixes.
• 1:00
  SPEAKER_00: It would be strange not to investigate.
• 1:03
  SPEAKER_00: It would also be expensive to assume every item in the queue looks like the vendor's test.
• 1:08
  SPEAKER_00: The purchasing decision is whether some of that work can move to a cheaper model without moving the cost into somebody else's afternoon.
• 1:18
  SPEAKER_00: Fireworks gives us more reason to call this an accuracy band than the headline alone suggests.
• 1:23
  SPEAKER_00: It reports variation between runs of roughly one and a half to three percentage points.
• 1:29
  SPEAKER_00: The spread across its top four models is smaller than that.
• 1:32
  SPEAKER_00: That doesn't prove the models are equivalent everywhere.
• 1:35
  SPEAKER_00: It means the small differences on this table aren't a sound basis for declaring a winner, even before we account for the vendor's commercial interest in selling Deep Seek hosting.
• 1:45
  SPEAKER_00: A second test, focused on command line work, puts Astra about one point ahead.
• 1:50
  SPEAKER_00: Again, this is Fireworks' evaluation, not an independent result from your engineering team.
• 1:55
  SPEAKER_00: The older DeepSeq model also gives us a useful warning.
• 1:59
  SPEAKER_00: Together, AI tested the July release on 113 software tasks and found Luna ahead by about 14 percentage points.
• 2:07
  SPEAKER_00: Snowflake tested that July model on a different benchmark for data team work and found it ahead of the leading proprietary model in its comparison.
• 2:16
  SPEAKER_00: Those results aren't votes in the same election.
• 2:18
  SPEAKER_00: Different tasks and comparison models can produce different answers, and neither August test settles what the new model will do today.
• 2:26
  SPEAKER_00: Auger's read is that the broad opportunity deserves attention, while the decision still belongs at the level of the work you actually need completed.
• 2:35
  SPEAKER_00: The cheap bill has an important detail tucked inside it.
• 2:38
  SPEAKER_00: Fireworks Run read nearly 37 million input tokens per task.
• 2:44
  SPEAKER_00: Tokens are the small chunks of text a model processes, including code and earlier tool results that an agent may read repeatedly.
• 2:52
  SPEAKER_00: Prompt caching lets the host reuse previously processed material and charge less for those reads.
• 2:58
  SPEAKER_00: It doesn't make them free.
• 3:00
  SPEAKER_00: Fireworks reports that almost all the input was cached, yet those discounted reads still made up about 60% of the bill.
• 3:08
  SPEAKER_00: Picture a finance lead hearing that an agent barely costs anything to run, then discovering that the estimate assumes a particular host's cash pricing and a very high reuse rate.
• 3:19
  SPEAKER_00: The spreadsheet has acquired a footnote.
• 3:22
  SPEAKER_00: Naturally, the footnote needs its own spreadsheet.
• 3:25
  SPEAKER_00: The architecture helps make lower cost possible, but the price per completed task still depends on how the agent works and what the host charges for it.
• 3:33
  SPEAKER_00: A model that rereads a lot of context can look cheap under one arrangement and less attractive under another.
• 3:40
  SPEAKER_00: Before negotiating a larger commitment, ask for cached input, uncached input, output, and failed attempts to be visible in the bill.
• 3:48
  SPEAKER_00: You can't manage a saving you can't explain.
• 3:52
  SPEAKER_00: There is a manageable way to test this without turning the whole engineering team into a benchmark department.
• 3:58
  SPEAKER_00: Have the engineering lead choose 20 recent accepted code changes with passing tests.
• 4:04
  SPEAKER_00: In isolated copies of the code base, return each task to the point before its change, then ask both models to repair it through the same agent setup.
• 4:14
  SPEAKER_00: That setup is the software that lets a model inspect files, edit code, and run checks.
• 4:20
  SPEAKER_00: Keep the task instructions and attempt budget comparable, hide the original repair, and don't let either model rewrite the tests used to grade its work.
• 4:29
  SPEAKER_00: Compare accepted repairs and total spend against Astra on those same tasks.
• 4:34
  SPEAKER_00: Include the time a person spends reviewing the answer and the extra cost when a failed Deep Seek attempt goes to Astra.
• 4:41
  SPEAKER_00: Agree on an acceptable quality gap and review time limit before the run, then review the results on September 22nd.
• 4:48
  SPEAKER_00: Stop if the endpoint fails your data policy, if a model changes the grading tests, or if extra review wipes out the savings.
• 4:55
  SPEAKER_00: A useful result earns a second sample.
• 4:58
  SPEAKER_00: It doesn't earn automatic access to production.
• 5:01
  SPEAKER_00: The advantage, if it holds, is a growing record of which tasks can use the cheaper route safely.
• 5:07
  SPEAKER_00: Every competitor can buy model access.
• 5:10
  SPEAKER_00: They don't have your task history and tests.
• 5:13
  SPEAKER_00: For a bank or another buyer requiring fireworks promised US hosted endpoint, that specific path is still coming soon.
• 5:20
  SPEAKER_00: An available endpoint elsewhere isn't permission to send regulated code there.
• 5:26
  SPEAKER_00: Model budgets aren't the only plans depending on promises this week.
• 5:30
  SPEAKER_00: The argument over slowing frontier development now has a clear public split.
• 5:34
  SPEAKER_00: Dario Amade's proposal combines embedded outside evaluators, coordination among leading AI companies in democratic countries, and international cooperation.
• 5:46
  SPEAKER_00: Anthropic has committed to the evaluator step, and OpenAI has followed, according to the reporting in today's dispatch.
• 5:54
  SPEAKER_00: TechCrunch reports that Jensen Huang pulled President Trump a slowdown wouldn't happen.
• 5:58
  SPEAKER_00: Trump publicly sided with him.
• 6:00
  SPEAKER_00: That is a political position, not an enacted policy, but it isn't a basis for assuming the release calendar will get calmer.
• 6:07
  SPEAKER_00: For the executive planning staffing or vendor reviews, Augur recommends watching whether evaluator access becomes concrete and whether governments enable coordination or change chip policy.
• 6:18
  SPEAKER_00: A public endorsement doesn't tell you who can inspect a lab or what happens after an inspection finds a problem.
• 6:24
  SPEAKER_00: The checkable commitments matter more to that planning decision than agreement on a conference stage.
• 6:31
  SPEAKER_00: A finished television series offers a different kind of evidence.
• 6:35
  SPEAKER_00: Something was made, although its profitability remains unresolved.
• 6:38
  SPEAKER_00: Chen AI reports that Mango Excellent Media brought an entirely AI-generated adaptation of the later Journey to the West to Chinese Provincial Satellite Primetime on August 31st.
• 6:50
  SPEAKER_00: Production took six months, compared with one to two years for a film drama.
• 6:56
  SPEAKER_00: The same newsletter reports a sharp rise in Mango's market value and a large existing market for generated microdramas.
• 7:03
  SPEAKER_00: Those figures don't establish that audiences like the series or that it made money.
• 7:08
  SPEAKER_00: The production team framed it as a demonstration of feasibility rather than profitability.
• 7:13
  SPEAKER_00: Consider a marketing producer asked to turn a product brief into a short video by Friday.
• 7:19
  SPEAKER_00: Faster generation could help, but the producer still has to approve the result, fix continuity, and throw away the versions that don't work.
• 7:26
  SPEAKER_00: The render button has not been promoted to Creative Director.
• 7:30
  SPEAKER_00: OpenRouter says CDance 2.5 can generate clips up to 30 seconds with audio and the same pass.
• 7:37
  SPEAKER_00: Augur recommends a small comparison only where speed matters and rights are cleared.
• 7:41
  SPEAKER_00: One brief, owned by the production lead, against the current workflow, reviewed on September 22nd.
• 7:47
  SPEAKER_00: Count everything through approval.
• 7:50
  SPEAKER_00: Stop if continuity or brand standards fail, rights are unclear, or retries remove the saving.
• 7:56
  SPEAKER_00: A second successful brief would justify expansion.
• 7:59
  SPEAKER_00: A rising share price would it.
• 8:02
  SPEAKER_00: For teams moving private data, Snowflake's new connection option could remove a different source of delay.
• 8:08
  SPEAKER_00: Its data connectivity proxy lets a small program inside the customer's network open an encrypted outbound connection to Snowflake.
• 8:15
  SPEAKER_00: The customer doesn't have to open an inbound firewall rule.
• 8:18
  SPEAKER_00: Source credentials stay in Snowflake, and the program passes the traffic without inspecting or storing its contents.
• 8:25
  SPEAKER_00: The limitation matters to the network engineer signing off the design.
• 8:29
  SPEAKER_00: The program needs direct outbound access.
• 8:32
  SPEAKER_00: Routing through a company-mandated forward proxy isn't supported.
• 8:36
  SPEAKER_00: Check that policy before scheduling a migration.
• 8:39
  SPEAKER_00: Augur's two dated calls remain unchanged.
• 8:42
  SPEAKER_00: Fireworks publicly lists a US-hosted DeepSeek V4.1 flash endpoint by November 30, 2026 at 70% odds.
• 8:52
  SPEAKER_00: Snowflake announces that model as available in Cortex AI by December 31st, 2026 at 55% odds.
• 9:01
  SPEAKER_00: These are testable predictions with public listing criteria, not announcements of availability.
• 9:07
  SPEAKER_00: The written dispatch carries the exact hit or miss rules, and the website tracks the calls so they can be judged later.
• 9:15
  SPEAKER_00: Pilot the coding comparison or the data rules allow it.
• 9:18
  SPEAKER_00: Watch the promised hosting path and the policy commitments.
• 9:22
  SPEAKER_00: Keep the video test bounded by the cost of an approved result.
• 9:27
  SPEAKER_00: The written brief with every source is at augurdispatch.ai.
• 9:31
  SPEAKER_00: That is the Augur Dispatch for Tuesday, September 15th, 2026.
• 9:36
  SPEAKER_00: Thank you for listening.