Latest / The Augur Dispatch / AI finished the report. Why are you still working?

Transcript

AI finished the report. Why are you still working? · The Augur Dispatch

• 0:00
  SPEAKER_00: Imagine getting a supplier comparison back almost immediately.
• 0:03
  SPEAKER_00: The prices are neatly lined up, there's a recommendation at the bottom.
• 0:07
  SPEAKER_00: Then you notice one price is from last year, and another buys a different product.
• 0:12
  SPEAKER_00: The AI has finished.
• 0:14
  SPEAKER_00: You haven't.
• 0:15
  SPEAKER_00: Sakana's new research offers a way for a model to improve how it works.
• 0:20
  SPEAKER_00: Using examples, it helps judge itself.
• 0:22
  SPEAKER_00: That's worth attention.
• 0:24
  SPEAKER_00: For the person checking that supplier report, the useful improvement would be fewer mistakes to untangle, not just an earlier arrival in the inbox.
• 0:32
  SPEAKER_00: This is the Augur dispatch for Monday, October 12, 2026.
• 0:37
  SPEAKER_00: Sakana calls the method multi-agent self-supervision, or mass.
• 0:42
  SPEAKER_00: It sounds like a room full of machines.
• 0:44
  SPEAKER_00: In this case, one underlying model takes different roles, doing the work, suggesting a better approach, and judging which attempts to keep.
• 0:53
  SPEAKER_00: The selected examples become training material.
• 0:56
  SPEAKER_00: The model learns from them and tries again.
• 0:58
  SPEAKER_00: That goes beyond asking several assistants to discuss an answer.
• 1:02
  SPEAKER_00: The training changes the model itself.
• 1:05
  SPEAKER_00: The training tasks were machine-generated research exercises, so the experiment doesn't establish how well this transfers to a company's own records.
• 1:14
  SPEAKER_00: The appeal is easy to see.
• 1:16
  SPEAKER_00: If a system can produce useful lessons from its own attempts, improving it may require less hand-prepared training material.
• 1:23
  SPEAKER_00: Sakana's experiment gives that idea something concrete to examine.
• 1:28
  SPEAKER_00: After two rounds of training, Sakana reports better research results, including more benchmark score for each output token.
• 1:35
  SPEAKER_00: Tokens are the chunks of text a model produces.
• 1:38
  SPEAKER_00: Across four research tests, the reported ratio is roughly 1.2 to 1.6 times the starting models.
• 1:45
  SPEAKER_00: There are gains in the answers too.
• 1:48
  SPEAKER_00: On a scientific task benchmark, success rises from about 29% to about 32%.
• 1:54
  SPEAKER_00: Some coding benchmarks go the other way.
• 1:57
  SPEAKER_00: Those results make the choice more interesting than whether to believe in self-improving AI.
• 2:02
  SPEAKER_00: A system might become a better research assistant without becoming a better programmer.
• 2:07
  SPEAKER_00: Buying it for the wrong job could erase the benefit before anyone opens the bill.
• 2:12
  SPEAKER_00: There's also a distinction worth getting right about who marked the work.
• 2:16
  SPEAKER_00: During training, the model judge's candidate approaches itself.
• 2:20
  SPEAKER_00: The paper then reports separate evaluation using other models and public benchmarks with their own scoring methods.
• 2:26
  SPEAKER_00: That corrects the original dispatch's claim that there was no outside evaluation.
• 2:31
  SPEAKER_00: There was.
• 2:32
  SPEAKER_00: Other model judges aren't a human accepting the work, and the author's tests aren't independent replication, but they shouldn't be dismissed as the model simply awarding itself a prize.
• 2:42
  SPEAKER_00: For our supplier comparison, the unanswered question is whether the trained system would make fewer of the mistakes that sent the analyst back to the source documents.
• 2:51
  SPEAKER_00: The paper doesn't establish that result on this imagined business task.
• 2:56
  SPEAKER_00: The way the job is divided could matter as much as the model doing it.
• 3:00
  SPEAKER_00: Suppose the analyst needs information about several suppliers.
• 3:04
  SPEAKER_00: Separate assistants could each investigate a company using the same requirements and the same date for prices.
• 3:11
  SPEAKER_00: That work has room to happen in parallel.
• 3:14
  SPEAKER_00: Now suppose one assistant calculates the costs, another applies a discount, and a third chooses the winner.
• 3:20
  SPEAKER_00: If they pass along an assumption incorrectly, each later step can make the wrong answer look more complete.
• 3:26
  SPEAKER_00: Putting the assistants in a group doesn't resolve that difference.
• 3:30
  SPEAKER_00: Anyone who has tried to reconcile several versions of the same spreadsheet has met the human version of this problem.
• 3:36
  SPEAKER_00: Google Research tested 180 agent configurations.
• 3:40
  SPEAKER_00: Coordination improved performance on a financial reasoning benchmark whose work could be split apart.
• 3:46
  SPEAKER_00: On a sequential planning benchmark, where later steps depended on earlier ones, the tested multi-agent approaches performed worse.
• 3:54
  SPEAKER_00: That was separate research, not a test of Sakana's system, and its results don't predict the saving in a purchasing department.
• 4:01
  SPEAKER_00: It does give us a reason to separate two jobs that often get bundled together, gathering independent evidence and maintaining one consistent chain of reasoning.
• 4:10
  SPEAKER_00: Auger's reading is that the supplier search may be a better place to try coordinated assistance than the final calculation deciding which contract to sign.
• 4:20
  SPEAKER_00: Come back to the analyst with the finished report.
• 4:23
  SPEAKER_00: The first useful change might happen before the model starts writing.
• 4:27
  SPEAKER_00: Give it the terms the comparison must satisfy.
• 4:30
  SPEAKER_00: The same product category, comparable contract periods, and current prices with sources the reviewer can open.
• 4:37
  SPEAKER_00: When a price isn't available, the report should say so rather than fill the space with a plausible substitute.
• 4:44
  SPEAKER_00: That doesn't guarantee a correct answer.
• 4:46
  SPEAKER_00: It gives the person reviewing it a defined job, instead of asking them to discover what the comparison was supposed to mean after the recommendation has already arrived.
• 4:56
  SPEAKER_00: A review process that checks dates and contract terms is different from one that rereads polished paragraphs.
• 5:03
  SPEAKER_00: If the new system saves effort on those checks, it has done something useful, even though a person still signs off.
• 5:09
  SPEAKER_00: It could also help the analyst investigate another viable supplier rather than stop at the familiar short list.
• 5:16
  SPEAKER_00: That's a possible use of time saved, not a result Sakana has demonstrated.
• 5:20
  SPEAKER_00: The value would depend on whether the extra research changes the decision.
• 5:25
  SPEAKER_00: This is where the efficiency number runs out.
• 5:27
  SPEAKER_00: Score per output token doesn't include the entire training bill, tool use, failed attempts, or the analyst's afternoon, nor does a shorter response automatically mean less checking.
• 5:38
  SPEAKER_00: A concise report can still omit the condition that would have changed the purchase.
• 5:43
  SPEAKER_00: Brevity has never stopped a document from creating a meeting.
• 5:47
  SPEAKER_00: A useful comparison would follow the work all the way to acceptance.
• 5:52
  SPEAKER_00: Keep the failed runs and the cost, count the corrections, and record how often the result meets the requirements.
• 5:58
  SPEAKER_00: Otherwise, a cheap answer that nobody can use can make the spreadsheet look better while the work piles up elsewhere.
• 6:05
  SPEAKER_00: The detailed test plan is in the episode notes.
• 6:08
  SPEAKER_00: It starts with 20 recent non-sensitive research tasks and compares the existing process with single-agent and coordinated runs.
• 6:16
  SPEAKER_00: The research or operations lead owns the result, with an experienced reviewer deciding what passes against criteria agreed beforehand.
• 6:24
  SPEAKER_00: The first review is October 26th.
• 6:27
  SPEAKER_00: Stop a task group if acceptance gets worse, a critical error slips through, or the experiment exceeds its approved data scope.
• 6:35
  SPEAKER_00: A small sample can expose a problem.
• 6:37
  SPEAKER_00: Repeat the comparison on fresh tasks before treating a saving as dependable.
• 6:42
  SPEAKER_00: This tests the workflow.
• 6:44
  SPEAKER_00: Reproducing mass would also require its training procedure and a separately measured training cost.
• 6:50
  SPEAKER_00: Sakana already has public workflow and training code, although the repository documents differences from the reported experiments.
• 6:58
  SPEAKER_00: The opportunity for a business is to learn which parts of a recurring job can be delegated reliably.
• 7:04
  SPEAKER_00: That knowledge can remain useful when the next model arrives.
• 7:07
  SPEAKER_00: The model name is likely to change sooner than the need to compare suppliers fairly.
• 7:13
  SPEAKER_00: Augur would pilot the independent research work while keeping the purchasing decision and its checks with people.
• 7:19
  SPEAKER_00: Adopt the full cost measure now.
• 7:21
  SPEAKER_00: Watch whether Mass's gains survive independent reproduction and human review before treating this as evidence for a broader replacement.
• 7:29
  SPEAKER_00: Expansion would make sense if the reviewer could accept more useful work with less total effort, without letting serious errors through.
• 7:37
  SPEAKER_00: A fast draft that still needs reconstructing hasn't earned that change.
• 7:42
  SPEAKER_00: In our supplier example, the analyst should finish knowing which prices can actually be compared and which uncertainties still need a call to the vendor.
• 7:51
  SPEAKER_00: The AI's contribution is worth paying for when it helps the person reach that point sooner.
• 7:56
  SPEAKER_00: The written brief with every source is at augurd dispatch.ai.
• 8:01
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 8:05
  SPEAKER_00: That is the Augur Dispatch for Monday, October 12th, 2026.
• 8:10
  SPEAKER_00: Thank you for listening.