Latest / The Augur Dispatch / AI improves the draft. Is your junior team learning?

Transcript

AI improves the draft. Is your junior team learning? · The Augur Dispatch

• 0:06
  SPEAKER_00: A better document can hide a training problem.
• 0:09
  SPEAKER_00: In a three-month trial with patent lawyers, access to an AI assistant improved drafting, but junior lawyers didn't show an average advantage when they later worked without it.
• 0:19
  SPEAKER_00: If you're buying these tools partly to develop your people, Augur's advice is to test that promise separately.
• 0:26
  SPEAKER_00: Review what someone can explain and correct without the assistant, as well as the document they deliver with it.
• 0:32
  SPEAKER_00: This is the Augur Dispatch for Thursday, October 8, 2026.
• 0:37
  SPEAKER_00: The trial enrolled 133 lawyers at 11 American firms.
• 0:42
  SPEAKER_00: 91 completed the full protocol, including the final task without AI.
• 0:47
  SPEAKER_00: Independent patent experts graded their work without knowing who had received the assistant, so knowing a draft came from the treatment group couldn't influence the grade.
• 0:56
  SPEAKER_00: A beautifully written paragraph can be wrong, and good typography has never signed a liability waiver.
• 1:03
  SPEAKER_00: The assistant improved drafting across experience groups.
• 1:06
  SPEAKER_00: The harder question came later, when lawyers had to find and fix mistakes in an existing patent without using AI.
• 1:14
  SPEAKER_00: Senior lawyers who had received the assistant outperformed senior lawyers in the control group on that final test.
• 1:21
  SPEAKER_00: Juniors showed no average advantage, but their scores were more spread out, with more poor and good results rather than more excellent ones.
• 1:29
  SPEAKER_00: There is an important limit here.
• 1:31
  SPEAKER_00: The researchers didn't test unaided skill at the start, so this isn't a before and after record of each lawyer getting better or worse.
• 1:39
  SPEAKER_00: It compares groups assigned by chance, with the interpretation depending on their starting skills being comparable.
• 1:46
  SPEAKER_00: The findings come from a working paper that hasn't been peer-reviewed, and Google paid the direct costs of the experiment at firms that already did patent work for the company.
• 1:55
  SPEAKER_00: Three months with a small, specialized group cannot tell us what happens across every profession.
• 2:01
  SPEAKER_00: Still, the distinction is useful.
• 2:04
  SPEAKER_00: Earlier research with Boston Consulting Group consultants found that weaker baseline performers gained more while using AI.
• 2:11
  SPEAKER_00: Leveling wasn't just a guess drawn from the average, but better performance with a tool doesn't establish what someone can do after that tool is removed.
• 2:20
  SPEAKER_00: Picture a manager approving a junior colleague's polished draft.
• 2:24
  SPEAKER_00: The work may be perfectly usable.
• 2:27
  SPEAKER_00: The manager still doesn't know whether the colleague can explain a consequential choice or catch the same mistake next week.
• 2:34
  SPEAKER_00: The study doesn't settle why the junior scores spread out.
• 2:38
  SPEAKER_00: It does describe editing habits that deserve attention without turning them into a causal explanation.
• 2:44
  SPEAKER_00: Some juniors worked through the document from the top, spending time on wording before reaching the claims that mattered most.
• 2:50
  SPEAKER_00: Others identified a problem in a comment without actually correcting it.
• 2:55
  SPEAKER_00: The researchers saw those habits among control juniors too, so we can't blame the assistant for creating them.
• 3:01
  SPEAKER_00: For the team lead, that suggests a useful conversation about priorities.
• 3:06
  SPEAKER_00: Ask which error carries the greatest consequence and why, rather than only whether the draft looks finished.
• 3:12
  SPEAKER_00: An explanation can reveal a gap that a clean document won't.
• 3:16
  SPEAKER_00: Stanford's related workplace research adds another complication.
• 3:20
  SPEAKER_00: Its account describes interviews across three companies and nearly two years observing two divisions at a law firm.
• 3:27
  SPEAKER_00: The division that presented AI as a way to enrich people's work used it more and experimented more.
• 3:33
  SPEAKER_00: That division also provided training, mentorship, and time to explore.
• 3:38
  SPEAKER_00: This wasn't a controlled test proving that a nicer slogan changes behavior.
• 3:42
  SPEAKER_00: The support changed too.
• 3:44
  SPEAKER_00: Scheduling a learning lunch and then filling the lunch hour with deadlines would rather miss the point.
• 3:50
  SPEAKER_00: Augur recommends a bounded learning pilot, not taking the assistant away from junior staff.
• 3:56
  SPEAKER_00: Choose one drafting team and make its lead accountable for checking independent judgment alongside output quality.
• 4:03
  SPEAKER_00: Before starting, use a short task without AI and have a reviewer assess the reasoning.
• 4:08
  SPEAKER_00: Repeat comparable tasks during the pilot, using the same rubric and, where possible, reviewers who don't know which stage they're grading.
• 4:17
  SPEAKER_00: Give people protected practice and ask them to explain important edits before accepting them.
• 4:22
  SPEAKER_00: At a four-week review, look at the full spread of scores, especially the weaker results, alongside document quality, rework, and time.
• 4:30
  SPEAKER_00: Keep the normal review of customer-facing work in place throughout.
• 4:34
  SPEAKER_00: If unaided judgment or the ability to catch consequential errors deteriorates, pause expansion and change the support.
• 4:42
  SPEAKER_00: Expand only after repeated checks show stronger reasoning without sacrificing the quality of the finished work.
• 4:49
  SPEAKER_00: That is Augur's proposed test, not a training method already proven by this paper.
• 4:54
  SPEAKER_00: A comparable team or phased rollout would help separate the intervention from changing assignments or reviewers.
• 5:01
  SPEAKER_00: The advantage, if it develops, comes from how the organization teaches and learns.
• 5:06
  SPEAKER_00: The subscription itself is available to competitors.
• 5:09
  SPEAKER_00: A renewal invoice is not a certificate of professional development.
• 5:14
  SPEAKER_00: Averages also need care when the conversation turns from people to model bills.
• 5:19
  SPEAKER_00: Anthropic has released Haiku 5.5, its small model for high-volume tasks, and says it costs around 75% less to run on average.
• 5:29
  SPEAKER_00: The price depends on prompt length.
• 5:31
  SPEAKER_00: Up to 100,000 tokens, the text units the model processes, the published input and output rates are 90% lower than Haiku 4.5.
• 5:40
  SPEAKER_00: Above that prompt length, the reduction is 50%.
• 5:44
  SPEAKER_00: Anthropic's average also accounts for a changed tokenazer that uses slightly more tokens per task.
• 5:50
  SPEAKER_00: The price of each unit and the number of units both matter.
• 5:54
  SPEAKER_00: Imagine the engineer responsible for thousands of short classification requests alongside a few enormous document jobs.
• 6:02
  SPEAKER_00: Applying the headline discount to the whole bill would hide that difference, before even considering retries or changes in answer quality.
• 6:10
  SPEAKER_00: Augur's recommendation is to compare measured cost for accepted results on your actual workload before changing routing.
• 6:17
  SPEAKER_00: The published rates are useful, they don't guarantee the savings on your particular work.
• 6:22
  SPEAKER_00: NVIDIA's competition results call for a similar separation between the headline and the system you might buy.
• 6:28
  SPEAKER_00: Its specialized Nemotron setup scored above the programming Olympiad's gold threshold and the top human score.
• 6:35
  SPEAKER_00: That run was unofficial and unsupervised.
• 6:38
  SPEAKER_00: The mathematics system also cleared gold, and NVIDIA says official Olympiad graders assessed its proofs.
• 6:45
  SPEAKER_00: Don't erase that distinction.
• 6:47
  SPEAKER_00: Neither result, however, tells you what the ordinary downloadable model will do on your company's coding tasks.
• 6:53
  SPEAKER_00: Extra training and software that generated, checked, and refined answers were part of the tested systems.
• 7:00
  SPEAKER_00: There is one dated Augur call today.
• 7:03
  SPEAKER_00: Google will make Gemini 4 Argon available to paid API customers by November 30th, 2026.
• 7:10
  SPEAKER_00: Augur assigns 55% odds.
• 7:12
  SPEAKER_00: A hit requires an official Google announcement or product page offering paying API customers access without a waitlist or partner-only program.
• 7:22
  SPEAKER_00: Access confined to fairwind, government users, or an invitation list counts as a miss.
• 7:27
  SPEAKER_00: Google's current fairwind policy still restricts access to approved partners.
• 7:32
  SPEAKER_00: This call remains open and ungraded.
• 7:34
  SPEAKER_00: Augur calls are dated predictions with explicit tests, tracked on the website so the forecast can be checked against what actually happens.
• 7:43
  SPEAKER_00: Separate from that call, the signals on the clock are follow-up questions, not additional forecasts.
• 7:49
  SPEAKER_00: Watch how Haiku's prompt length pricing translates into real bills and whether Argon access broadens beyond approved partners.
• 7:57
  SPEAKER_00: Microsoft's Agent Lightning trains agents with the same operating software they use in deployment.
• 8:03
  SPEAKER_00: Results on production tasks will tell us more than the framework's compact code alone.
• 8:08
  SPEAKER_00: OpenAI's GPT-6 announcement already includes ChatGPT business and enterprise in the intelligent UI rollout.
• 8:16
  SPEAKER_00: Enterprise access depends on administrator settings, and the update covers chat, not work or codecs.
• 8:22
  SPEAKER_00: Check what your workplace actually enables.
• 8:25
  SPEAKER_00: AI2 has published its Bulmo research in Nature and released models extending the approach beyond its original family.
• 8:32
  SPEAKER_00: These models process the underlying bytes used to represent text.
• 8:36
  SPEAKER_00: Broader adoption by other builders remains something to watch.
• 8:41
  SPEAKER_00: For today's operating decision, pilot a separate check of independent judgment and keep document review intact.
• 8:48
  SPEAKER_00: Watch the claims that still need evidence from your own workloads.
• 8:51
  SPEAKER_00: Don't use a polished output as the only proof that your team is learning.
• 8:56
  SPEAKER_00: What can your junior colleagues now explain or correct without the assistant that they couldn't do before?
• 9:03
  SPEAKER_00: The written brief with every source is at auger dispatch.ai.
• 9:08
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 9:11
  SPEAKER_00: That is the Augur Dispatch for Thursday, October eighth, twenty twenty six.
• 9:16
  SPEAKER_00: Thank you for listening.