Latest / The Augur Dispatch / Who grades your AI grader? Allen AI's 52-point warning

Transcript

Who grades your AI grader? Allen AI's 52-point warning · The Augur Dispatch

• 0:00
  SPEAKER_00: Allen AI changed the instructions given to an AI grader, and its accuracy fell from about 94% to about 41%.
• 0:08
  SPEAKER_00: The research ideas being graded hadn't changed.
• 0:11
  SPEAKER_00: Before you use a model's score to approve another model, you need to know what the grading instructions are doing to that score.
• 0:18
  SPEAKER_00: There's an important qualification here.
• 0:20
  SPEAKER_00: This wasn't simply a polite question becoming a blunt one.
• 0:24
  SPEAKER_00: The researchers removed a cue saying human reviewers had found one of the ideas novel.
• 0:29
  SPEAKER_00: They changed what the grader was told to assume.
• 0:32
  SPEAKER_00: This is the Augur dispatch for Wednesday, October 28th, 2026.
• 0:37
  SPEAKER_00: Augur's view is that the grading setup needs its own evaluation owner.
• 0:42
  SPEAKER_00: A score that looks convincing under one set of instructions hasn't earned the right to settle a release decision on its own.
• 0:49
  SPEAKER_00: Picture a product lead arriving at a release meeting with a reassuring chart.
• 0:53
  SPEAKER_00: The assistant's answers have improved, according to the model that grades them.
• 0:57
  SPEAKER_00: Nobody has changed the customer examples.
• 1:00
  SPEAKER_00: Everyone would like the meeting to end.
• 1:02
  SPEAKER_00: Asking to inspect the graders is unlikely to win the shortest meeting award, but it may be the most useful question in the room.
• 1:10
  SPEAKER_00: The reason teams use these graders is sensible.
• 1:13
  SPEAKER_00: If an assistant writes a customer response, there may be dozens of acceptable answers, and a rule that checks for one exact sentence won't tell you whether the response helped.
• 1:23
  SPEAKER_00: A model can read the meaning, apply a rubric, and return a score.
• 1:27
  SPEAKER_00: A rubric is just the criteria you're asking it to use.
• 1:30
  SPEAKER_00: Human review costs time, so an automated first pass can make testing practical at a scale people couldn't otherwise manage.
• 1:38
  SPEAKER_00: Anthropic has described a simple grading call that returned scores and a pass or fail judgment, and said it matched human judgments better than more elaborate arrangements in its research system.
• 1:50
  SPEAKER_00: That is a reason to take the method seriously.
• 1:53
  SPEAKER_00: It isn't a guarantee that the same approach will work for your assistant.
• 1:57
  SPEAKER_00: Agreement with humans is evidence about the examples and instructions you tested.
• 2:01
  SPEAKER_00: Changing either can change what that agreement means.
• 2:04
  SPEAKER_00: This is where Allen AI's experiment becomes useful.
• 2:08
  SPEAKER_00: Its study asks models to judge whether research ideas are novel, meaning they contribute something new rather than repackage existing work.
• 2:16
  SPEAKER_00: That's a difficult judgment even before you automate it.
• 2:20
  SPEAKER_00: The researchers built a pool of about 150 high-novelty ideas from accepted papers at a major machine learning conference.
• 2:27
  SPEAKER_00: They didn't just assume accepted papers must be original.
• 2:31
  SPEAKER_00: They looked for explicit praise of originality in the reviews and excluded papers where a reviewer disputed it.
• 2:37
  SPEAKER_00: They also collected lower novelty human ideas and produced a separate set of ideas with a basic AI generator.
• 2:44
  SPEAKER_00: Those different populations turned out to matter.
• 2:48
  SPEAKER_00: The striking result came when GPT-5.4 compared strong human ideas with generated ideas.
• 2:55
  SPEAKER_00: One prompt told it that reviewers had found exactly one idea novel.
• 2:59
  SPEAKER_00: Another left that statement out.
• 3:01
  SPEAKER_00: The measured accuracy dropped by more than 50 percentage points, more than half the verdicts changed on the same pairs.
• 3:09
  SPEAKER_00: But the paper says the prompts aren't equivalent, and we shouldn't quietly erase that sentence because it makes the headline less tidy.
• 3:16
  SPEAKER_00: Giving the judge information about prior reviewers can change its expectations.
• 3:21
  SPEAKER_00: This experiment doesn't isolate the effect of swapping one harmless phrase for another.
• 3:26
  SPEAKER_00: There's also uncertainty in the answer key.
• 3:29
  SPEAKER_00: The researchers assumed the basic generators' ideas were less novel than the selected human ideas.
• 3:35
  SPEAKER_00: One expert checked 30 pairs and agreed with that ordering in every case, but most of those were difficult cases for the strongest judge.
• 3:43
  SPEAKER_00: It wasn't a complete independent review of the whole collection.
• 3:47
  SPEAKER_00: Those limits affect how confidently we should interpret the accuracy figures.
• 3:51
  SPEAKER_00: They don't make the changed vertex disappear.
• 3:54
  SPEAKER_00: Two opposing answers to the same comparison can't both be the right answer.
• 3:59
  SPEAKER_00: On human-only pairs, the same prompt change moved GPT-5.4 by about seven points.
• 4:05
  SPEAKER_00: Another model moved by 21.
• 4:07
  SPEAKER_00: The grader that looked steadier on human writing wasn't necessarily steadier when generated ideas entered the comparison.
• 4:14
  SPEAKER_00: For the product lead in our imagined meeting, that changes the follow-up.
• 4:18
  SPEAKER_00: It isn't enough to ask whether the grader once agreed with people.
• 4:22
  SPEAKER_00: Ask whether the examples in that check resemble the output the team is producing now.
• 4:27
  SPEAKER_00: There is a reasonable defense of the grader.
• 4:30
  SPEAKER_00: New instructions can contain relevant information.
• 4:33
  SPEAKER_00: A human reviewer might change a judgment after learning the same thing, and research originality is a long way from checking whether a response includes the required account number.
• 4:43
  SPEAKER_00: Augur wouldn't use this paper to declare every automated score worthless.
• 4:48
  SPEAKER_00: That would repeat the mistake of stretching one measurement beyond the job it actually tested.
• 4:53
  SPEAKER_00: The narrower problem is still consequential.
• 4:56
  SPEAKER_00: If your team changes the instructions, adds a claim about what experts believe, or switches the kind of output being evaluated, the score may stop meaning what you thought it meant.
• 5:07
  SPEAKER_00: Imagine the engineer responsible for improving that score.
• 5:10
  SPEAKER_00: They've been adjusting the assistance prompts and checking each version against the same judge.
• 5:15
  SPEAKER_00: If the judge has an unnoticed preference, the engineer can spend a week satisfying it without making customers any better off.
• 5:23
  SPEAKER_00: The chart may be accurate about the graders' preferences.
• 5:26
  SPEAKER_00: That's an awkward thing to discover after the launch email.
• 5:30
  SPEAKER_00: Other research raises related concerns.
• 5:32
  SPEAKER_00: An audit of vulnerability analysis found that roughly 60% of correct model verdicts came with fabricated or unverifiable claims in their explanations.
• 5:42
  SPEAKER_00: That's security analysis, not this novelty experiment.
• 5:46
  SPEAKER_00: And it doesn't prove why Al NAI's judges change their minds.
• 5:50
  SPEAKER_00: It does explain why a plausible explanation shouldn't end the inspection.
• 5:54
  SPEAKER_00: A correct-looking verdict and a dependable reason for reaching it are separate things to check.
• 6:00
  SPEAKER_00: The practical response is a small comparison with a control.
• 6:04
  SPEAKER_00: Keep a sample of past outputs fixed, including cases the team considers unambiguous.
• 6:09
  SPEAKER_00: Run the original grading prompt again to see how often it changes its mind without an edit.
• 6:15
  SPEAKER_00: Then try two rewrites that another person has checked for the same meaning.
• 6:19
  SPEAKER_00: Keep the criteria and information intact.
• 6:22
  SPEAKER_00: If you want to test a reviewer authority queue, do that separately and label it as a different experiment.
• 6:28
  SPEAKER_00: Compare the changes with the repeated original baseline.
• 6:31
  SPEAKER_00: Have the evaluation owner inspect consequential reversals before the next rollout meeting.
• 6:36
  SPEAKER_00: A few stable examples don't prove the judge is safe, and a reversal doesn't prove every previous score was wrong.
• 6:44
  SPEAKER_00: The useful outcome is knowing which decisions need human attention.
• 6:48
  SPEAKER_00: Don't keep editing the instructions until the result turns green.
• 6:51
  SPEAKER_00: A grading prompt shouldn't have to negotiate its way onto the status slide.
• 6:56
  SPEAKER_00: Use fixed checks when the work permits them.
• 6:59
  SPEAKER_00: Wheaton Labs describes checking coding agent work against known correct spreadsheets, which gives those outputs a reference that doesn't change its opinion about the wording of a question.
• 7:09
  SPEAKER_00: For less definite work, keep the difficult customer examples and the human decisions made about them.
• 7:15
  SPEAKER_00: That history can make your evaluation better suited to your business.
• 7:19
  SPEAKER_00: Buying the same grading model as everyone else won't create that history for you.
• 7:24
  SPEAKER_00: A few signals remain on the clock in the wider written dispatch.
• 7:28
  SPEAKER_00: Anthropic has filed for a proposed NASDAQ listing under ANT.
• 7:32
  SPEAKER_00: A filing isn't a first trade.
• 7:34
  SPEAKER_00: Its new government discount is available for direct purchases, with cloud provider discounts planned in the coming months.
• 7:40
  SPEAKER_00: Watch for disclosed state or city contracts under those terms.
• 7:44
  SPEAKER_00: Allen AI's benchmark and code are available for further testing.
• 7:49
  SPEAKER_00: Evidence on narrower grading tasks would tell us how far this concern extends.
• 7:54
  SPEAKER_00: Google Research reports an offline learning method that retains information in model weights.
• 7:59
  SPEAKER_00: A product appearance would be the next useful signal.
• 8:02
  SPEAKER_00: DeepMind's Genie 4 is already entering a research preview for trusted testers.
• 8:07
  SPEAKER_00: Broader developer access and usable terms remain worth watching.
• 8:11
  SPEAKER_00: The written brief with every source is at augerdispatch.ai.
• 8:16
  SPEAKER_00: Portions of the Augur Dispatch are AI generated.
• 8:19
  SPEAKER_00: That is the Augur Dispatch for Wednesday, October 28th, 2026.
• 8:24
  SPEAKER_00: Thank you for listening.