Latest / DX Today | No-Hype Podcast & News About AI & DX / OpenAI Dropped 722 Math Papers From a Model Nobody Can Test

Transcript

OpenAI Dropped 722 Math Papers From a Model Nobody Can Test · DX Today | No-Hype Podcast & News About AI & DX

• 0:00
  SPEAKER_01: Welcome to the DX Today Podcast, your daily deep dive into the AI ecosystem.
• 0:04
  SPEAKER_01: I'm Chris, and joining me as always is Laura.
• 0:07
  SPEAKER_00: Hi, Chris, and I have to tell you, today's story is one I have been genuinely buzzing about since Tuesday night.
• 0:13
  SPEAKER_00: Because it sits right where artificial intelligence collides with the oldest and proudest discipline in all of science, which is pure mathematics.
• 0:21
  SPEAKER_01: Okay, you have my attention because when you say pure mathematics, I picture chalkboards and lonely geniuses, not data centers.
• 0:28
  SPEAKER_01: So set the scene for us and tell me exactly what landed this week.
• 0:32
  SPEAKER_00: So on Tuesday, October 6th, OpenAI posted 722 mathematics manuscripts to a public GitHub repository, all of them produced by an internal frontier model that the company has not released and has not even named publicly.
• 0:46
  SPEAKER_01: 722 papers in a single afternoon is a staggering number, but I want to make sure we are honest about it, because a manuscript is not the same thing as a solved problem, right?
• 0:55
  SPEAKER_00: Exactly right.
• 0:56
  SPEAKER_00: And that distinction matters a lot here.
• 0:58
  SPEAKER_00: The company groups those manuscripts into 372 families of results, where a family might bundle a main theorem together with companion arguments, consequences, or alternative proofs of the same idea.
• 1:11
  SPEAKER_01: So the headline number roughly halves the moment you read the fine print, which is still enormous.
• 1:16
  SPEAKER_01: But let's talk about how the model actually got these results because I suspect the process is the real story.
• 1:21
  SPEAKER_00: According to the repository, the model was handed roughly 4,000 problems during an internal evaluation, and the company then published the outputs it judged to be significant.
• 1:31
  SPEAKER_00: So we are seeing the winners, not the misses, or the dead ends.
• 1:35
  SPEAKER_01: That's a crucial detail because 4,000 attempts and a few hundred families of results tells me there was a huge amount of filtering, and we don't really know what the failure pile looks like or how big it was.
• 1:46
  SPEAKER_00: And here is the number that made my jaw drop.
• 1:48
  SPEAKER_00: The company says the average result used about three hours of Chat GPT Pro thinking time.
• 1:53
  SPEAKER_00: And a spokesperson told Scientific American that almost every paper came from a single prompt given to a single agent.
• 2:00
  SPEAKER_01: Three hours and one prompt for a research result that might take a graduate student a year.
• 2:05
  SPEAKER_01: Sounds almost too clean.
• 2:06
  SPEAKER_01: So how does that compare with what we saw from the same model just last month?
• 2:10
  SPEAKER_00: That comparison is fascinating.
• 2:12
  SPEAKER_00: Last month, the same model produced a claim proof related to the Navy Air Stokes equations.
• 2:17
  SPEAKER_00: And that effort used 10,000 coordinating agents working for 88 hours.
• 2:21
  SPEAKER_00: So the jump to one agent and one prompt is dramatic.
• 2:24
  SPEAKER_01: I'll play devil's advocate here, because a dramatic efficiency jump is exactly the kind of claim that deserves suspicion.
• 2:31
  SPEAKER_01: And if I were a working mathematician, I would want to know who checked all of this before it went public.
• 2:36
  SPEAKER_00: You and a lot of mathematicians are thinking the same thing.
• 2:39
  SPEAKER_00: Only 162 of the 722 papers, roughly 22%, have a main result formalized in Lean, which is software that mechanically checks every logical step of a proof.
• 2:50
  SPEAKER_01: Let's slow down on lean for a second, because many listeners have never heard of it, and I want them to understand why a computer-checked proof is treated as such a gold standard in this debate.
• 3:00
  SPEAKER_00: Think of Lean as an incredibly pedantic referee.
• 3:02
  SPEAKER_00: You write your proof in a precise formal language, and the software refuses to accept it unless every single step follows from the rules.
• 3:10
  SPEAKER_00: So if Lean says yes, the logic is airtight relative to the statement you wrote down.
• 3:14
  SPEAKER_01: Relative to the statement you wrote down is doing a lot of work in that sentence.
• 3:19
  SPEAKER_01: So I'm guessing there's a catch hiding there that people outside the field would completely miss.
• 3:24
  SPEAKER_00: There absolutely is a catch.
• 3:26
  SPEAKER_00: A passing lean check proves the argument follows from the formal statement, but it does not prove that statement faithfully matches the original problem.
• 3:34
  SPEAKER_00: And it says nothing about whether the result is new, important, or already known.
• 3:39
  SPEAKER_01: So that even the most verified 22% still needs humans to confirm the right question was answered.
• 3:44
  SPEAKER_01: And the other 78% is essentially resting on trust in the company and the model.
• 3:48
  SPEAKER_00: And to their credit, the company says so directly.
• 3:51
  SPEAKER_00: Its own materials state that some of the unformalized results could have issues, and it promises to fix problems quickly and add more formal proofs as they become available over the coming months.
• 4:02
  SPEAKER_01: I appreciate that candor, but promising to fix things later is a strange way to publish science.
• 4:07
  SPEAKER_01: So which researchers are pushing back hardest?
• 4:09
  SPEAKER_01: And what exactly are they saying out loud?
• 4:12
  SPEAKER_00: Tristan Buckmaster, a mathematician at New York University who was personally working on the Navy or Stokes problem before this model tackled it, told the New York Times, quote, I don't think they've done their sort of due diligence at all, end quote.
• 4:25
  SPEAKER_01: That's a remarkably blunt statement from someone with real skin in the game.
• 4:29
  SPEAKER_01: And I imagine it stings when a machine sweeps into the exact corner of mathematics you spent years building your career around.
• 4:37
  SPEAKER_00: It surely does.
• 4:38
  SPEAKER_00: And Andrew Sutherland at MIT gave what might become the slogan of this whole episode.
• 4:43
  SPEAKER_00: He told Scientific American that we should treat single agent claims as unverified, and his exact words were, we should ask for receipts.
• 4:51
  SPEAKER_01: Ask for receipts is such a perfect phrase because it captures the core tension, which is that the results are public, but the model, the prompts, and the full reasoning that produced them are still locked inside the company.
• 5:03
  SPEAKER_00: That's precisely the problem the Institute for Advanced Studies saw coming.
• 5:07
  SPEAKER_00: Its advisory group on mathematics and artificial intelligence published guidelines on September 29th asking labs to stop testing hard problems on private models and to disclose prompts, time, and compute for every result.
• 5:21
  SPEAKER_01: Wait, so a respected independent body published those guidelines one week before this release, and the company says it drew on that advice.
• 5:28
  SPEAKER_01: So did it actually follow the guidelines or just cite them?
• 5:32
  SPEAKER_00: That's the awkward part.
• 5:33
  SPEAKER_00: The company published average compute figures and 10 abridged reasoning summaries, but no prompts, and the model is still private.
• 5:41
  SPEAKER_00: On Tuesday night, the advisory group said its advice did not endorse the results or the way they were obtained.
• 5:47
  SPEAKER_01: That sounds like a polite but very pointed public distancing.
• 5:50
  SPEAKER_01: And I'm curious whether the advisory group said anything that captures how the broader field should feel about all of this.
• 5:56
  SPEAKER_00: It did, and I love this line.
• 5:58
  SPEAKER_00: The group said, quote, this release is the beginning, not the completion of the process of human understanding, end quote.
• 6:05
  SPEAKER_00: And it warned that private lab models risk creating a two-tier system where labs outrun everyone else.
• 6:11
  SPEAKER_01: The two-tier warning is the part that worries me most, because science depends on anyone being able to reproduce a result.
• 6:17
  SPEAKER_01: And here only one company on Earth can rerun the experiment that produced these papers.
• 6:21
  SPEAKER_00: And reproducibility is not just a philosophical point.
• 6:25
  SPEAKER_00: One researcher, Keith Adler, noticed that the GitHub repository has its issue tracker turned off and has never accepted a pull request.
• 6:32
  SPEAKER_00: So the public cannot even formally submit a correction or a new formal proof there.
• 6:37
  SPEAKER_01: That's a wild design choice for something framed as a gift to the research community because GitHub's entire culture is built around outsiders filing issues and suggesting fixes, and switching that off sends a mixed signal.
• 6:50
  SPEAKER_00: To be fair, the company says it is exploring other community-hosted options, and it plans to fund workshops, conferences, and special programs devoted to understanding major results produced by AI, so it does seem aware the release needs a human follow-up.
• 7:04
  SPEAKER_01: Let's give the optimists a real hearing now, because I don't want this episode to sound like a pylon.
• 7:09
  SPEAKER_01: And surely some mathematicians looked at these papers and felt genuine excitement rather than suspicion.
• 7:14
  SPEAKER_00: Absolutely.
• 7:15
  SPEAKER_00: Abishek Saha called it a very big day for mathematics.
• 7:19
  SPEAKER_00: While noting most results are exceptional advances within existing research programs rather than paradigm shifts.
• 7:25
  SPEAKER_00: And Dimitri Rybin described one proof about coloring the plane as, in his words, totally unbelievable alien math.
• 7:32
  SPEAKER_01: Alien math is such an evocative phrase.
• 7:34
  SPEAKER_01: So can you explain that coloring problem in plain English?
• 7:38
  SPEAKER_01: Because I think listeners deserve at least one concrete example of what this model actually proved.
• 7:43
  SPEAKER_00: Sure, imagine coloring every point on an infinite flat plane so that no two points exactly one unit apart share a color.
• 7:51
  SPEAKER_00: The question is the minimum number of colors needed, and a manuscript argues that, under certain technical measurability conditions, you need at least six.
• 8:00
  SPEAKER_01: That's a lovely puzzle.
• 8:01
  SPEAKER_01: Simple enough for a kid to understand, and yet hard enough that experts have chewed on it for decades, which is exactly why a surprising machine proof gets people talking.
• 8:09
  SPEAKER_00: And the surprise is the point.
• 8:10
  SPEAKER_00: Rybin highlighted an unexpected step involving what are called weakly measurable colorings, the kind of move a human might never think to try, which is why he called it alien rather than simply clever or efficient.
• 8:23
  SPEAKER_01: What about the really big names?
• 8:25
  SPEAKER_01: I've seen headlines throwing around the Riemann Zeta function and even the Hodge conjecture, which sound like the mathematical equivalent of landing on Mars.
• 8:34
  SPEAKER_00: Careful framing matters here.
• 8:36
  SPEAKER_00: Two results did not follow the standard procedure.
• 8:38
  SPEAKER_00: One establishes a zero-free region for the Riemann Zeta function and a human edited that write up for readability.
• 8:45
  SPEAKER_00: The other claims a proof of the Hodge conjecture for a special class called CM Abelian varieties.
• 8:51
  SPEAKER_01: So neither of those is a claimed solution to the famous million-dollar millennium prize problems themselves, but rather results in the neighborhood of those giants, which is impressive, but very different from what a breathless headline implies.
• 9:03
  SPEAKER_00: Exactly.
• 9:04
  SPEAKER_00: And some coverage blurred that line badly.
• 9:06
  SPEAKER_00: A special case or a partial region can still be a meaningful contribution, but it is a far cry from the full Riemann hypothesis or the full Hodge conjecture, and experts will spend months judging the actual significance.
• 9:19
  SPEAKER_01: I want to bring in the company's own explanation for why it ran this experiment at all.
• 9:23
  SPEAKER_01: Because publishing hundreds of papers seems like an unusual move for a company that mostly sells chatbots and developer tools.
• 9:30
  SPEAKER_00: Dan Roberts, a research lead at OpenAI, told the New York Times that testing internal models on hard problems helps the team build better tools, and that the proofs were essentially a byproduct of that internal evaluation work rather than the primary goal.
• 9:45
  SPEAKER_01: Calling hundreds of research papers a byproduct is either refreshingly honest or a little bit of a flex.
• 9:51
  SPEAKER_01: And I genuinely can't decide which, but it does show how central mathematics has become as a benchmark for reasoning models.
• 9:58
  SPEAKER_00: And there's a competitive backdrop too.
• 10:00
  SPEAKER_00: Last month, Anthropic published a lean verified formalization of Ferame's last theorem, roughly 13 million lines on GitHub.
• 10:08
  SPEAKER_00: But that project formalized the proof Andrew Wiles completed decades ago rather than claiming brand new results.
• 10:14
  SPEAKER_01: That's a really interesting contrast in philosophy, because one lab is using AI to make known mathematics bulletproof, while the other is using AI to generate new claims at a speed the verification machinery simply cannot match yet.
• 10:27
  SPEAKER_00: And that mismatch is the bottleneck nobody solved this week.
• 10:30
  SPEAKER_00: Human referees typically take months to review a single serious paper.
• 10:34
  SPEAKER_00: So 722 manuscripts arriving at once could overwhelm the small communities of specialists qualified to evaluate them.
• 10:42
  SPEAKER_01: So for the listener running an engineering team or an AI program inside a company, what's the practical lesson here?
• 10:47
  SPEAKER_01: Because most of us are never going to referee a paper about abelian varieties.
• 10:52
  SPEAKER_00: The lesson is that generation is now cheap and verification is the scarce resource.
• 10:57
  SPEAKER_00: Whether it is code, contracts, or theorems, the winners will be the teams that invest in automated checking, like lean for proofs or tests the suites for software, and keep humans accountable for meaning.
• 11:09
  SPEAKER_01: That's a sharp takeaway, and I'd add one more, which is that transparency is becoming a competitive asset because results nobody can reproduce will always carry an asterisk, no matter how impressive they look on day one.
• 11:22
  SPEAKER_00: Agreed, and Daniel Litt at the University of Toronto made a related point, arguing there's no good reason to keep answers like these secret.
• 11:30
  SPEAKER_00: So releasing them is better than hoarding them, even if the release process itself is messy and incomplete.
• 11:36
  SPEAKER_01: So where do you land overall, Laura?
• 11:38
  SPEAKER_01: Is this a historic moment for mathematics, or a very sophisticated public relations event wrapped in a GitHub repository full of impressive looking papers?
• 11:47
  SPEAKER_00: Honestly, I think it is both at once.
• 11:50
  SPEAKER_00: The mathematics may be real and even beautiful, but until the model, the prompts, and independent replication arrive, the right posture is exactly what Andrew Sutherland said.
• 11:59
  SPEAKER_00: Curious, excited, and firmly asking for receipts.
• 12:02
  SPEAKER_01: That's all for today's episode of the DX Today podcast.
• 12:05
  SPEAKER_01: Thanks for listening, and we'll see you next time.