Latest / The Augur Dispatch / Lean's bogus proof: what an independent AI check must verify

Transcript

Lean's bogus proof: what an independent AI check must verify · The Augur Dispatch

• 0:06
  SPEAKER_00: A proof can get a computer's approval without earning your trust.
• 0:10
  SPEAKER_00: Trail of Bits has demonstrated that with Lean, the software behind two of this month's biggest AI math announcements, the practical response is to check what the approval depends on before you rely on it.
• 0:21
  SPEAKER_00: A second check helps only if it doesn't inherit the same blind spot.
• 0:25
  SPEAKER_00: This is the Augur Dispatch for Thursday, September 10th, 2026.
• 0:30
  SPEAKER_00: A quick disclosure before we begin.
• 0:32
  SPEAKER_00: Portions of the Augur Dispatch Podcast are generated using artificial intelligence.
• 0:37
  SPEAKER_00: The written dispatch contains the full source trail and verification notes.
• 0:42
  SPEAKER_00: The important qualification comes first.
• 0:44
  SPEAKER_00: Trailabits did not break Lean's core logical checker.
• 0:48
  SPEAKER_00: Its demonstration used a feature that trusts the results of compiled code, and a bug, let that code, disagree with Lean's own mathematical definition.
• 0:58
  SPEAKER_00: That's a narrower finding than saying formal verification is broken.
• 1:02
  SPEAKER_00: It's also a useful warning for anyone buying a system on the strength of the words machine check.
• 1:08
  SPEAKER_00: Picture developer preparing a result for review.
• 1:12
  SPEAKER_00: The status says passed.
• 1:14
  SPEAKER_00: The manager asks whether it's verified, and the developer says, yes.
• 1:18
  SPEAKER_00: Nobody has lied, but they may be using the same word for very different levels of assurance.
• 1:25
  SPEAKER_00: Trail of bits says the issue affects stable lean releases through 4.33.1.
• 1:31
  SPEAKER_00: The memory safety fix merged roughly three hours after filing, and the remaining mismatch was fixed five days after the record.
• 1:38
  SPEAKER_00: The patch is incorporated in 4.34.0, release candidate 1.
• 1:44
  SPEAKER_00: Updating matters.
• 1:45
  SPEAKER_00: It doesn't replace checking the proof's assumptions.
• 1:48
  SPEAKER_00: Lean's own guidance separates ordinary checking from the stronger protection needed for potentially malicious proofs, including unreviewed AI submissions.
• 1:58
  SPEAKER_00: It recommends inspecting assumptions and, for high-risk cases, comparing the intended statement and using external checkers in a protected workflow.
• 2:08
  SPEAKER_00: If a proof relies on a native computation that another checker can't validate, accepting that computation again doesn't make the second check independent.
• 2:17
  SPEAKER_00: The team has to address that dependency.
• 2:20
  SPEAKER_00: For an engineering leader, adopt the inventory work now.
• 2:25
  SPEAKER_00: Find results whose acceptance depends on lean, record the version and assumptions, and recheck affected work on a build containing both fixes before calling it verified.
• 2:35
  SPEAKER_00: The operating advantage would come from making review repeatable.
• 2:39
  SPEAKER_00: A team that knows exactly what it accepted can investigate a newly disclosed bug without reopening every project from memory.
• 2:46
  SPEAKER_00: Auger's suggested pilot is one consequential proof workflow owned by the engineering lead.
• 2:52
  SPEAKER_00: Record today's review time and unresolved assumptions.
• 2:55
  SPEAKER_00: Then add statement comparison and a genuinely independent check where supported.
• 2:59
  SPEAKER_00: Review the results in a week.
• 3:01
  SPEAKER_00: Expand only if the intended statement and acceptable assumptions survive both checks with a manageable review burden.
• 3:08
  SPEAKER_00: Stop reliance on any result the team cannot validate.
• 3:13
  SPEAKER_00: OpenAI's claim solution to the Navier-Stokes problem needs another distinction.
• 3:18
  SPEAKER_00: These equations describe fluid motion, and the company says its internal system resolved a long-standing question about whether a particular kind of smooth flow can break down.
• 3:28
  SPEAKER_00: It published a mathematical write-up and a lean formalization.
• 3:32
  SPEAKER_00: The supplied reporting had not established mathematical acceptance as of September 9th.
• 3:38
  SPEAKER_00: This episode does not resolve that question, and the lean exploit is not evidence that OpenAI used the trick.
• 3:45
  SPEAKER_00: There is also a dispute about how the work began.
• 3:48
  SPEAKER_00: NYU mathematician Tristan Buckmaster alleges OpenAI learned about his progress before it was public.
• 3:55
  SPEAKER_00: OpenAI says it heard rumors that prompted its effort, but did not see the researcher's work before publication.
• 4:01
  SPEAKER_00: It also says it cannot rule out de-identified product usage data helping improve its models.
• 4:07
  SPEAKER_00: That is the company's response, not an independent finding about what happened.
• 4:12
  SPEAKER_00: Mathematical correctness and research provenance require different evidence.
• 4:16
  SPEAKER_00: A proof file cannot settle who knew what, and a dispute about priority does not by itself show that a mathematical argument is wrong.
• 4:24
  SPEAKER_00: For buyers, this remains a watch item.
• 4:26
  SPEAKER_00: Don't turn an unresolved research announcement into a capability guarantee in a contract.
• 4:33
  SPEAKER_00: There is a more constructive example of checking in AI2's account of Goodfire's work on Olmo.
• 4:39
  SPEAKER_00: Goodfire traced part of a safety regression back to individual training examples and predicted behavioral changes before a full training run.
• 4:48
  SPEAKER_00: The training data contained pairs of answers labeled preferred and rejected.
• 4:53
  SPEAKER_00: Some preferred answers encouraged behavior the researchers wanted the model to refuse.
• 4:58
  SPEAKER_00: AI2 publishes those examples, along with intermediate models and training recipes.
• 5:03
  SPEAKER_00: That let Goodfire investigate specific causes and test targeted changes, rather than guess from the finished model alone.
• 5:11
  SPEAKER_00: Give the engineer who owns a model update that starts answering requests it used to refuse.
• 5:16
  SPEAKER_00: With the training record available, there's a concrete place to investigate.
• 5:20
  SPEAKER_00: Without it, the engineer may spend days changing prompts around a problem introduced elsewhere.
• 5:25
  SPEAKER_00: These are the company's reported results without outside replication established in this brief.
• 5:30
  SPEAKER_00: They still give procurement teams a useful question about access to the training process, beyond whether the finished model's weights are downloadable.
• 5:38
  SPEAKER_00: A downloadable model is helpful, it isn't necessarily a maintenance manual.
• 5:45
  SPEAKER_00: Mistral's energy sector migration makes the same point through older software.
• 5:50
  SPEAKER_00: The starting reservoir simulator had no test suite and no central documentation.
• 5:55
  SPEAKER_00: Mistral says its first sprint migrated 40,000 of 300,000 lines from Fortran 77 into C.
• 6:03
  SPEAKER_00: Crucially, the team built a comparison hardness before migrating.
• 6:07
  SPEAKER_00: It captured outputs from the old program, including intermediate values selected by reservoir engineers, and compared them with the new implementation.
• 6:15
  SPEAKER_00: Matching the old program doesn't prove the underlying physics is correct.
• 6:19
  SPEAKER_00: It provides evidence that the translation preserved the behavior being compared.
• 6:24
  SPEAKER_00: That is a useful, bounded claim.
• 6:26
  SPEAKER_00: Picture the reservoir engineer reviewing an unfamiliar replacement.
• 6:30
  SPEAKER_00: A large count of translated lines doesn't answer whether an important pressure calculation still agrees.
• 6:35
  SPEAKER_00: A comparison at the relevant checkpoint gives that conversation somewhere to start.
• 6:40
  SPEAKER_00: Mistral also reports that human supervised workflows worked better for this project than fully autonomous migration.
• 6:47
  SPEAKER_00: For a leader pricing a modernization contract, fund the comparison setup and specialist review explicitly.
• 6:54
  SPEAKER_00: The source figures remain Mistral's account, not independently replicated results.
• 7:00
  SPEAKER_00: OpenAI's commercial offering is moving to.
• 7:04
  SPEAKER_00: Astra reached Snowflake's Cortex AI and private preview yesterday.
• 7:08
  SPEAKER_00: It is already available directly through OpenAI, including its API, so Snowflake's preview status should not be confused with the model's overall availability.
• 7:18
  SPEAKER_00: OpenAI says it is training a more capable internal model.
• 7:22
  SPEAKER_00: Government testing reportedly requested no changes to Astra, but that doesn't establish the absence of every outside constraint.
• 7:30
  SPEAKER_00: Augur's procurement judgment is to keep routing switchable and avoid commitments that assume today's model will remain the best available option.
• 7:39
  SPEAKER_00: Today's Augur call puts a 35% probability on OpenAI making a model it identifies as newer than GPT-6 Astra, generally available through its API by December 31st, 2026.
• 7:53
  SPEAKER_00: A qualifying public announcement or model page makes it a hit.
• 7:57
  SPEAKER_00: No such listing by the deadline makes it a miss.
• 8:01
  SPEAKER_00: These dated calls make Auger's judgments testable.
• 8:04
  SPEAKER_00: The website tracks them so readers can judge the record, not just the confidence of the wording.
• 8:09
  SPEAKER_00: Adopt the verification inventory now.
• 8:12
  SPEAKER_00: Pilot stronger checking on one consequential workflow.
• 8:15
  SPEAKER_00: Watch the unresolved math claims without treating them as settled science.
• 8:19
  SPEAKER_00: The written daily dispatch contains the full source trail, the verification limits, and the items Augur is watching next.
• 8:26
  SPEAKER_00: That is the Augur Dispatch for Thursday, September tenth, twenty twenty six.
• 8:31
  SPEAKER_00: Thank you for listening.