Latest / The Augur Dispatch / Jev's speed claims: test the decisions before switching models

Transcript

Jev's speed claims: test the decisions before switching models · The Augur Dispatch

• 0:06
  SPEAKER_00: A faster model can send the customer to the wrong queue faster, too.
• 0:10
  SPEAKER_00: Before you replace a language model with TypeSafe's new decision model, JEV, test it on cases where your team already knows the right answer.
• 0:18
  SPEAKER_00: The launch numbers make a pilot worth considering.
• 0:21
  SPEAKER_00: They don't tell you whether the change will work for your business.
• 0:25
  SPEAKER_00: This is the Augur Dispatch for Tuesday, September 22nd, 2026.
• 0:30
  SPEAKER_00: Versailles engineer Pranit Sharma told TechCrunch that replacing an open AI model with JEV for classifying commands produced results 5 to 18 times faster with better accuracy.
• 0:42
  SPEAKER_00: That's useful production evidence from a named engineer, although we don't have a published test set or an independent replication of that result.
• 0:51
  SPEAKER_00: Picture a support manager looking at a queue of messages that all need to reach the right department before anyone can help.
• 0:58
  SPEAKER_00: A system that sorts them accurately and cheaply could free up time.
• 1:02
  SPEAKER_00: A system that sorts them confidently but badly just changes who has to apologize.
• 1:15
  SPEAKER_00: The practical question is how often that answer is right, especially on the cases where an error costs more than a little delay.
• 1:24
  SPEAKER_00: TypeSafe does publish comparisons.
• 1:26
  SPEAKER_00: That's important because its founder's opposition to generic public benchmarks shouldn't be confused with an absence of evidence.
• 1:34
  SPEAKER_00: The company compares models across four workflows, including invoice processing and customer service.
• 1:41
  SPEAKER_00: It reports accuracy alongside cost and time, using reference answers derived from two powerful models.
• 1:48
  SPEAKER_00: In plain language, the test asks how closely each model agrees with those reference answers.
• 1:55
  SPEAKER_00: That is useful, but it isn't the same as checking against decisions that people have independently established are correct.
• 2:02
  SPEAKER_00: If the reference models share a mistake, agreement with them won't expose it.
• 2:06
  SPEAKER_00: TypeSafe acknowledges other limits.
• 2:09
  SPEAKER_00: Its own capabilities team created the workflows, so selection bias could be present.
• 2:14
  SPEAKER_00: The language models use a wrapper that produces structured decisions and probabilities.
• 2:18
  SPEAKER_00: That may improve their answers, but it can also make them slower and more expensive than a simpler setup.
• 2:24
  SPEAKER_00: Founder Diogo Almeida argues in his Late in Space interview that public benchmarks can be gamed and that teams should measure models inside the workflows they actually use.
• 2:34
  SPEAKER_00: You don't have to agree with every part of that argument to see why your own test matters.
• 2:39
  SPEAKER_00: There's another distinction worth keeping.
• 2:41
  SPEAKER_00: TypeSafe says Jeb's answers are guaranteed to fit the structure the developer has defined.
• 2:46
  SPEAKER_00: If the permitted choices are billing or technical support, the model can't invent a third department.
• 2:52
  SPEAKER_00: They can still pick the wrong one.
• 2:54
  SPEAKER_00: A properly formatted mistake remains a mistake.
• 2:58
  SPEAKER_00: For the engineering leader, the sensible next step is one small shadow test.
• 3:04
  SPEAKER_00: Let JEV make proposed decisions alongside the existing system without allowing it to change what happens to a customer.
• 3:11
  SPEAKER_00: Choose a low-risk routing task with past cases that the operations team can label reliably.
• 3:17
  SPEAKER_00: Give the engineering owner responsibility for the comparison and have the operations owner set acceptable error limits before looking at the results.
• 3:25
  SPEAKER_00: Agree to review the evidence in one week on September 29th.
• 3:30
  SPEAKER_00: Compare JEV with the model you actually use under comparable conditions.
• 3:34
  SPEAKER_00: Measure how often each gets the answer right, how long it takes, and the total cost once human review and fallback calls are included.
• 3:41
  SPEAKER_00: Count expensive mistakes separately, and overall score can look respectable while hiding the one kind of misroute your team cannot afford.
• 3:49
  SPEAKER_00: Keep the current path available.
• 3:51
  SPEAKER_00: Expand only if JEV meets those quality limits and delivers a useful cost or latency improvement.
• 3:57
  SPEAKER_00: Stop if it mishandles a critical case, breaches the agreed limits, or loses its savings once correction work is included.
• 4:04
  SPEAKER_00: Don't make autonomous safety approvals the first experiment.
• 4:08
  SPEAKER_00: Imagine the finance lead asking why the model bill fell while the review team's hours increased.
• 4:13
  SPEAKER_00: That's a fair question.
• 4:14
  SPEAKER_00: And a cheaper invoice from the vendor isn't the whole answer.
• 4:18
  SPEAKER_00: The advantage, if this works, comes from a better tested workflow and lower operating cost.
• 4:23
  SPEAKER_00: Your competitors can buy the same model.
• 4:25
  SPEAKER_00: They won't automatically have your checked examples, your understanding of which errors matter, or your experience handling exceptions.
• 4:34
  SPEAKER_00: Missing evidence also matters when an agent answers a question about the business itself.
• 4:40
  SPEAKER_00: Databricks's Richard Tomlinson describes an internal test in which an assistant prepared a polished briefing for a product advisory board.
• 4:47
  SPEAKER_00: It said 24 customers were participating.
• 4:50
  SPEAKER_00: When challenged about the source, it admitted it had made up the number.
• 4:54
  SPEAKER_00: Anyone preparing for that meeting would now have a second job, checking the briefing that was supposed to save time.
• 5:00
  SPEAKER_00: A number in a tidy paragraph is still a number somebody may have to defend.
• 5:05
  SPEAKER_00: V7, a separate company featured in an OpenAI customer story, is working on this context problem.
• 5:11
  SPEAKER_00: Its V7 Go platform organizes company information and links facts back to their sources, using several OpenAI models for different parts of the work.
• 5:22
  SPEAKER_00: V7 reports a reduction of roughly 40% in hallucinations on unanswerable questions in one benchmark.
• 5:29
  SPEAKER_00: That result comes from its retrieval-only system compared with the benchmark's official baseline, so it doesn't establish that every complete workflow will reliably decline when a required file is missing.
• 5:41
  SPEAKER_00: For an IT leader, the useful test includes questions the system should answer and questions it cannot answer from the material supplied.
• 5:49
  SPEAKER_00: Check whether the cited source supports the claim, not merely whether a citation appears.
• 5:54
  SPEAKER_00: A footnote can be beautifully formatted and still send you to the wrong place.
• 6:00
  SPEAKER_00: The choice of model brings a different set of constraints.
• 6:03
  SPEAKER_00: Interconnect's Nathan Lambert reports that Chinese labs lead the current open weight comparison, including a substantial lead in downloads and capability benchmarks over American open alternatives.
• 6:15
  SPEAKER_00: Open weights means a team can obtain the trained model parameters rather than relying only on a vendor's hosted service.
• 6:22
  SPEAKER_00: That can matter for deployment control, but it doesn't remove licensing or governance questions.
• 6:27
  SPEAKER_00: The article's estimates of the gap to the closed American frontier are approximate, and an older estimate cited in today's dispatch puts the gap wider.
• 6:36
  SPEAKER_00: These are assessments, not a single settled measurement.
• 6:39
  SPEAKER_00: A business choosing a fallback should therefore test the specific task and review the license and deployment constraints.
• 6:46
  SPEAKER_00: Chinese models deserve consideration where they meet those requirements.
• 6:50
  SPEAKER_00: American alternatives still exist.
• 6:52
  SPEAKER_00: A national ranking can't make the purchasing decision for you.
• 6:57
  SPEAKER_00: Chemistry gives us a more tangible example of why the test's meaning matters.
• 7:02
  SPEAKER_00: A Microsoft-led team with pharmaceutical and academic collaborators has published Retrochimera in Nature.
• 7:09
  SPEAKER_00: The system combines two models to suggest how chemists might make a target molecule, working backward toward the starting materials.
• 7:16
  SPEAKER_00: For 10 challenging targets, experts accepted complete routes for nine, compared with two to five for the component models and the baseline.
• 7:24
  SPEAKER_00: Those are expert judgments about proposed routes.
• 7:27
  SPEAKER_00: They are not nine completed laboratory syntheses, and they don't establish a 90% success rate across chemistry.
• 7:34
  SPEAKER_00: For a synthesis team, the potential benefit is more useful proposals to examine.
• 7:39
  SPEAKER_00: The repository describes a research release that can still produce false suggestions, and it requires independent expert verification before real-world use.
• 7:48
  SPEAKER_00: That supports careful evaluation by qualified chemists.
• 7:52
  SPEAKER_00: It doesn't support taking a route off a screen and treating the experiment as done.
• 7:57
  SPEAKER_00: Keep the human judgment and the laboratory result distinct.
• 8:03
  SPEAKER_00: Augur's recommendation today is to pilot one low-risk decision step with checked answers in the existing system as the baseline.
• 8:12
  SPEAKER_00: Watch for independent JEV evaluations and for customer evidence about how memory systems handle information they don't have.
• 8:20
  SPEAKER_00: Today's two proposed Augur polls are withheld, while their criteria are checked against products and evaluations that already exist.
• 8:27
  SPEAKER_00: No replacement prediction or grade is being issued.
• 8:31
  SPEAKER_00: Calls are meant to make forecasts accountable through dated, testable outcomes with tracking on the website.
• 8:37
  SPEAKER_00: That requires checking the starting point, too.
• 8:40
  SPEAKER_00: The written brief with every source is at augerdispatch.ai.
• 8:45
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 8:48
  SPEAKER_00: That is the Augur Dispatch for Tuesday, September twenty second, twenty twenty six.
• 8:53
  SPEAKER_00: Thank you for listening.