Latest / The Augur Dispatch / Descript's AI savings: cut the wait before you switch models

Transcript

Descript's AI savings: cut the wait before you switch models · The Augur Dispatch

• 0:06
  SPEAKER_00: EScript says a new AI model used to wait about a week for an engineer before anyone could test it.
• 0:12
  SPEAKER_00: Now the evaluation runs in an hour or two.
• 0:16
  SPEAKER_00: Before you buy a model router, find out how much of your own testing time is spent waiting for someone to become available.
• 0:22
  SPEAKER_00: This is the Augur dispatch for Monday, September 21st, 2026.
• 0:28
  SPEAKER_00: Descript makes video and audio editing software.
• 0:31
  SPEAKER_00: Its underlord agent uses AI to help edit video.
• 0:34
  SPEAKER_00: According to Open Routers case study, Descript's total model spending fell below half its previous level as cheaper models took on more work, partly through more frequent testing.
• 0:45
  SPEAKER_00: That account comes from the vendor selling the connection.
• 0:48
  SPEAKER_00: Descript says quality held against its own test cases, but we don't have an independent audit or enough detail to judge what those tests cover.
• 0:57
  SPEAKER_00: The useful finding is still the wait.
• 1:00
  SPEAKER_00: A cheaper model can't save you anything while its integration request sits untouched on an engineer's calendar.
• 1:08
  SPEAKER_00: DScript had direct connections to OpenAI, Anthropic, and Google.
• 1:12
  SPEAKER_00: Each new model still meant checking how the provider expected requests to work.
• 1:17
  SPEAKER_00: Someone had to do the integration, and someone else had to wait.
• 1:21
  SPEAKER_00: OpenRouter provides one connection to many models.
• 1:24
  SPEAKER_00: DScript built its own testing workflow around that connection, using Anthropic's Claude Tag integration in Slack to start evaluations and propose code changes.
• 1:33
  SPEAKER_00: A human reviews and approves them.
• 1:36
  SPEAKER_00: Alex Mistratov, DScript's head of AI product, describes seeing a model announcement while out walking, asking for an evaluation through Slack and getting results an hour or two later.
• 1:47
  SPEAKER_00: The team now tests models several times a week, and most don't make it into production.
• 1:52
  SPEAKER_00: That last detail matters.
• 1:54
  SPEAKER_00: A useful testing process should make rejection inexpensive too.
• 1:59
  SPEAKER_00: By August, DScript had 13 models in production.
• 2:03
  SPEAKER_00: Open weight models, whose trained parameters can be downloaded, have become part of the mix.
• 2:08
  SPEAKER_00: Auger's interpretation is that faster testing helped the team find cheaper options worth using.
• 2:14
  SPEAKER_00: The case study doesn't isolate how much of the saving came from speed alone.
• 2:18
  SPEAKER_00: OpenRouter's own guidance says to compare costs per completed task.
• 2:23
  SPEAKER_00: A low-price model that needs repeated attempts, or a stronger model to repair its work, can lose its apparent advantage.
• 2:29
  SPEAKER_00: Procurement can approve the cheap model.
• 2:31
  SPEAKER_00: The retries don't attend that meeting.
• 2:34
  SPEAKER_00: Easier switching doesn't remove dependence on suppliers.
• 2:37
  SPEAKER_00: It changes where that dependence sits.
• 2:40
  SPEAKER_00: Mistradov says a provider that isn't on OpenRouter needs an unusually strong offer before he'll integrate it directly.
• 2:46
  SPEAKER_00: That makes the router's catalog a practical filter on which options receive attention.
• 2:51
  SPEAKER_00: Descript keeps a dedicated, open weight deployment on Base 10 and tells OpenRouter to try that host first.
• 2:58
  SPEAKER_00: Other approved hosts provide backup for the same model.
• 3:01
  SPEAKER_00: That's a change of hosting provider, not a surprise change of model when something fails.
• 3:06
  SPEAKER_00: Those backups still have to meet DScript's data policy.
• 3:09
  SPEAKER_00: Easier access isn't a reason to send customer information to an unapproved host.
• 3:14
  SPEAKER_00: For your own team, start with the last model you added.
• 3:18
  SPEAKER_00: Measure the calendar time between its public release and its first evaluation in your product.
• 3:22
  SPEAKER_00: Then separate actual engineering work from time spent waiting in somebody's queue.
• 3:27
  SPEAKER_00: If waiting dominates, a router could help.
• 3:30
  SPEAKER_00: If the slow part is writing useful tests or deciding what acceptable output means, another connection won't solve it.
• 3:37
  SPEAKER_00: The pilot should be small, one existing workflow, one candidate model, and the current model as the comparison.
• 3:44
  SPEAKER_00: Have the engineering lead own it.
• 3:47
  SPEAKER_00: Keep the quality standard unchanged, measure time to first evaluation and cost per completed task, and review the results after two weeks.
• 3:56
  SPEAKER_00: Stop if quality falls or the required data controls can't be maintained.
• 4:01
  SPEAKER_00: Expand only if the team gets useful results sooner without shifting the cost into retries or manual repair.
• 4:07
  SPEAKER_00: Every competitor can buy the same router.
• 4:10
  SPEAKER_00: Your advantage would come from learning faster on work your customers actually need.
• 4:15
  SPEAKER_00: There is a different testing problem after an agent makes a mistake.
• 4:19
  SPEAKER_00: You change the code and run the job again, but the model chooses a different path.
• 4:23
  SPEAKER_00: Now you don't know whether the repair worked or the failure simply didn't recur.
• 4:28
  SPEAKER_00: Microsoft engineers presented Chronicle as a way to record what goes into and comes out of an agent's model and tool boundaries.
• 4:36
  SPEAKER_00: Think of a boundary as the point where the agent asks a model for an answer or asks software to do something.
• 4:42
  SPEAKER_00: The new paper describes replaying selected recorded answers while running the change code live.
• 4:48
  SPEAKER_00: That can turn a recorded failure into a repeatable check on the repair.
• 4:52
  SPEAKER_00: The scope matters.
• 4:54
  SPEAKER_00: Chronicle records the boundary interface, not every internal side effect.
• 4:59
  SPEAKER_00: The evaluated implementation doesn't cover concurrent parallel tool calls, and streaming answers are recorded in assembled form.
• 5:07
  SPEAKER_00: The researchers tested six deliberately constructed short instants with simulated model responses.
• 5:13
  SPEAKER_00: Full replay made no model calls.
• 5:16
  SPEAKER_00: Their targeted tests caught every deliberately broken guard that allowed the recorded unsafe action through, but that doesn't mean they caught every possible code defect.
• 5:26
  SPEAKER_00: For the developer trying to fix a failure, holding those recorded responses steady could remove one source of confusion.
• 5:32
  SPEAKER_00: It's a promising method to investigate, not proof that a production agent is safe.
• 5:37
  SPEAKER_00: Any test that runs a destructive tool live belongs in a sandbox, away from real customer systems.
• 5:44
  SPEAKER_00: Voice systems need tests that change with the market.
• 5:47
  SPEAKER_00: Real-time Venus is a pair of smaller models, one for speech and another for speech with video.
• 5:52
  SPEAKER_00: Each has 9 billion parameters, the learn values inside the model.
• 5:56
  SPEAKER_00: On the author's conversation benchmark, the speech model responded to about three-quarters of interruptions and kept going through background speech about 86% of the time.
• 6:06
  SPEAKER_00: Picture a customer correcting an address while a voice agent is still speaking.
• 6:11
  SPEAKER_00: Stopping for that correction is useful.
• 6:13
  SPEAKER_00: Stopping because someone else talks in the room may not be.
• 6:16
  SPEAKER_00: A natural conversation depends on telling those situations apart.
• 6:20
  SPEAKER_00: The paper compares those behaviors with Gemini 3.1 Live and GPT-4.0.
• 6:25
  SPEAKER_00: Google released Gemini 3.8 Live on September 15th, and OpenAI put GPT-Live 1 into its developer interface on September 10th.
• 6:34
  SPEAKER_00: A buyer should rerun the comparison against those current alternatives.
• 6:39
  SPEAKER_00: The reported scores don't settle that choice, and the evidence reviewed for this brief doesn't establish that the real-time Venus model weights are available to download.
• 6:50
  SPEAKER_00: Keep it on the watch list rather than treating the paper as a deployment recommendation.
• 6:55
  SPEAKER_00: Nate Jones brings the spending question back to the customer.
• 6:59
  SPEAKER_00: He describes a customer asking why products still looked much like last year's if AI was supposed to change so much.
• 7:05
  SPEAKER_00: His own support time study found that gathering a customer's records across separate systems consumed the effort.
• 7:12
  SPEAKER_00: Writing the reply was the cheap part.
• 7:15
  SPEAKER_00: Anyone who has searched several systems before answering one ordinary question will recognize the distinction.
• 7:21
  SPEAKER_00: A faster first sentence doesn't help much when the facts are still hiding in another application.
• 7:26
  SPEAKER_00: For next quarter's budget, inspect that work before buying another writing assistant.
• 7:31
  SPEAKER_00: The aim is to make useful attention affordable for more customers, not simply to produce their holding message sooner.
• 7:38
  SPEAKER_00: Today's auger call is that Scrolled will have a publicly downloadable consumer app listed on the Apple App Store or Google Play by October 31st, 2026.
• 7:49
  SPEAKER_00: The odds are 55%.
• 7:51
  SPEAKER_00: A hit requires a scrolled branded app from the company to be publicly listed and installable by that date, confirmed by the store listing or a company announcement linking to it.
• 8:01
  SPEAKER_00: Otherwise, it's a miss.
• 8:03
  SPEAKER_00: These dated calls make Augur's predictions testable and they're tracked on the website.
• 8:08
  SPEAKER_00: Adopt the measurement of your own testing delay.
• 8:11
  SPEAKER_00: Pilot a router where integration cues are the problem.
• 8:14
  SPEAKER_00: Watch the replay and voice research until the evidence reaches your actual workflow.
• 8:20
  SPEAKER_00: The written brief with every source is at auger dispatch.ai.
• 8:25
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 8:28
  SPEAKER_00: That is the Augur Dispatch for Monday, September 21st, 2026.
• 8:33
  SPEAKER_00: Thank you for listening.