Latest / The Information Bottleneck / What’s Left for Mathematicians? | Daniel Litt

Transcript

What’s Left for Mathematicians? | Daniel Litt · The Information Bottleneck

• Ravid Shwartz-Ziv: Hi everyone and welcome back to the Information Button podcast. And today we have the pleasure to host Daniel Leed, he's a professor at the University of Toronto. Hi Daniel, thank you for joining us. Daniel Litt: Hey. Yeah, it's my pleasure. Ravid Shwartz-Ziv: So today we're going to talk mostly about AI and math. You are professor that you you worked on math for for all your h academic life. so I I want to start like maybe at the beginning, right? Or like maybe like a year ago. You had this bet that AI wouldn't write top tier math papers by twenty thirteen. Daniel Litt: So this would yeah, so it was March twenty twenty five, so I guess about a year and a half ago, and I bet Teme Busroglu that or Tema Busroglu that so the specific the precise bet was that there would not be paper autonomously produced by AI for a budget of a hundred thousand dollars that in j my judgment could have been published in the Annals of Mathematics, which is like one of the top math journals, in twenty twenty five. So I think actually, strictly speaking, the bet has not resolved. Like right now the models cannot literally do this. I think there are some results they've proven more or less autonomously, which could get in the annals. But like they can't actually can't write a paper. like the the papers just suck. Like you would not be able to get them published. But yeah, I mean I I decided to concede the bet because like the whole point was you know, of course the you can you can try to you can try to l you know analyze technically the bet, but clearly like I'd morally lost the bet. Like they're they're doing interesting math. Ravid Shwartz-Ziv: So what what has changed, right? Like what actually like now let's say AI models can do that they didn't do like a year ago or a year and a half ago. Daniel Litt: Yeah. Yeah, it's kind of interesting. I mean I think like my expectation at the time was that like there were like a lot of things you needed to write like a doogle high quality mathematics. And it seemed to me at the time that the models were like getting good at some of those things and not getting good at others. so this was I guess March twenty twenty five was maybe around when O three came out and the models were like start starting to be able to do like technical things like sub somewhat reliably. but like, you know, kind of just didn't really have a good understanding of of a lot of basic objects. and they were like very uneven within different parts of math, or like between different parts of math. What I think I was just kind of wrong about at the time was that like I thought you needed like all the skills that people have in order to do high quality math. And like instead of I think what's happened is the models have become like really superhuman at some things, like they like know really know everything in the literature pretty well, for example. And they're actually like still way worse than us at some other things. but they're they're able to kind of substitute some of that, like like the skills where they're superhuman for some of the skills where they're not so good. so yeah, I think where I kind of was wrong and like I've you know, I've kind of learned a lot about what it takes to do high quality math is just like, you know, just by being very, very superhuman at some some kind of bundle of skills, which is like a relatively small, I think, portion of the bundle of skills that people use when they do math. You can still actually produce pretty high quality. Ravid Shwartz-Ziv: So w let's try Allen Roush: And Ravid Shwartz-Ziv: to like decompose it. Like what models are really good in in math Daniel Litt: Mm-hmm. Ravid Shwartz-Ziv: and what they're like not so good? And like and do you think they are like getting improved like also in these areas? Daniel Litt: yeah, I think they're definitely okay, so like there's like a bunch of things I think I do when I do when I do math. Like, okay, so first of all like I have some knowledge of what people know and like basic techniques or whatever. I know a bunch of examples. So they're the models are like super, super good. Like they they really just know everything. not necessarily at a super high level, but you know, like they're kind of more from definitely very familiar with yeah. Ravid Shwartz-Ziv: So if l l let's l just a second. So like if I'm taking like existing like math paper and I'm telling like the model to I don't know, reproduce it or like to prove the thing that's like already in the in the paper, it will it will do it. Daniel Litt: they'll just even like know what paper you're talking about and like be like, well that's this this result and s I mean it they it won't even have to reproduce it. Like it's just very, very familiar with the the full literature. yeah but yeah and you know if you take well you can ask about like papers that are not available like can it prove a lemma in one of my papers in progress? We can talk about that, but I think that's kind of a different different question. okay so that's one thing, like just like broad knowledge. another thing is just like you know, reasoning ability. it's actually not clear to me how the models compare to humans and reasoning ability. my guess is like probably pretty comparable, but you know, right now Yeah, so like you're trying to you're trying Ravid Shwartz-Ziv: W what does it mean? W what you a what does it actually mean? Daniel Litt: to get from A to B and you try to you try to figure out like a path, you know, some some some steps you need to take. So you maybe you need to approve some intermediate results or something on the way, and you need to like reason to figure out what those results are and like how to prove them. so the models are like pretty good at this, but it's like not an even comparison, right? So like when I try to go from A to B, I do some like huge amount of reasoning to find like find what the intermediate statements I might need are. And like 99% of the time, a bunch of those are in the literature. Right. So I'm doing a huge amount of reasoning that like ends up being going into some literature search. And the models don't have to do that because they know the literature super well. So Yeah, so I think that you know, there's like when humans do mathematics, they a lot a huge amount of expertise and sorry, a huge amount of reasoning is spent like rediscovering things that people already know. so yeah. Then there's a bunch of other sort of more fuzzy skills, like you know, a a lot of math is like philosophy. Like often when I'm like trying to figure out what might be true, there's like an analogy I have between two two situations. And there I've just like never seen the models do that. I'm not saying they can, but like You know, maybe they're like signs of this happening at some very low level. So there's yeah, some kind of more fuzzy like creativity skills that like I think, you know, we don't really see from the models in other domains either so much and those are also relevant to math. I mean I think one issue is like the tasks we ask the models to do in math are like typically of the form like here's a conjecture, which is like some precise statement, like prove that conjecture. That's like not most of what mathematicians do. Like of course, like that's kind of the thing we measure, like it's like a high prestige activity or whatever, like you're you know, working and working and then you like prove some old old conjecture and like you get a nice paper out of it. But like most of what I do is like more like I have some kind of fuzzy you know some object I would like to understand and I'm like just even trying to figure out what's true about it. And that I just don't know how to measure, but like when I ask the models to do it you know, of course it's hard to to say precisely whether the output is good or bad, but it's like I don't like the output typically. Like it it my my judgment Ravid Shwartz-Ziv: But what Daniel Litt: is that it's bad. Ravid Shwartz-Ziv: but l let's try to like to try to quantify it. Like what you actually don't like it. Is it is it not true because like the I don't know the derivation are not true? Is it not like you you don't like it because like it's too verbals? Like do like what exactly or like maybe it sounds like two AI generated Daniel Litt: Like they don't don't seem to be getting at getting at anything fundamental. Like it's you know, maybe I wanna you know, I've tried to say like, okay, well here's some phenomenon I know about in situation A. Like I don't know, but I'll just I'll say some words to give you a prec precise example. So the behavior of the cohomology of a family of degenerating varieties, some object I like to think. Okay, and I would like to understand what's happening in situation B. So instead of replacing cohomology with some other invariants, the fundamental sort of something I'd love to think about. And so okay, you say like, I would like to find a statement analogous to such and such, and like, you know, the models will produce some output and it's like just like clearly a bad answer to the question and they should know that. like it doesn't actually get at anything fundamental. They'll like make to some assumptions that are way way too way too strong to be interesting. Like they'll try to say something true, maybe, but at at the cost of like, you know, like sometimes I don't know would be a better answer. Allen Roush: So so you you've talked about the jagged frontier and the tendency of models to be, you know, ultra superhuman in one place and maybe not so much in other places. it appears further, at least in my opinion, that the rate of AI development progress almost seems like it's accelerating, you know, at least locally, I would argue certainly since twenty twenty-two. What do you think is the end game of all of this? Like, do you think that most theoretical mathematicians? mathematics that a human could even conceive of will be solved relatively soon. And do you further you know think that these apparently superhuman or math systems will create basically miracles? Like what I'm trying to understand is is like is it mostly just going to be theoretical math that gets solved or are we going to get room temperature, superconductors and all the other cool stuff that we would hope from solving all of math? Daniel Litt: Okay, so let me just say so for mostly what I've been saying in the last five minutes is like a description of current capabilities. My expectation is that the models will get robustly superhuman at math and like at intellectual labor in general. It's you know, I think even in the sort of the bundle of tasks I've described where they're like not that good, I think there's some weak evidence that they're getting like a little bit better. And, you know, as you say, the kind of the historical trends have been that if if you see them getting a little bit better at something, then probably they'll get a lot better soon. okay and and my guess is that that's like just the case across like all intellectual labor, although like to be clear this is like a pretty low certainty prediction. Like I you know, I think it's more likely than not, but you know, I wouldn't bet the house on it. Okay, so yeah, will we see miracles? Like will we solve math? Like just no, I think. there's just no no evidence for that, and I think that there's like very good Reasons to think that like even just basic mathematical questions a person might ask can get arbitrarily hard. so, you know, the models will get to some level, but it will b still I my expectation is it will be very easy to find questions that they make essentially no progress on. And even questions of the form like, does this particular polynomial have an integer solution? Like those th that kind of question can get arbitrarily hard. Ravid Shwartz-Ziv: But do you think like models will be also good in suggesting these questions? Daniel Litt: Eventually, yeah. I I that's some it's an area where they're relatively weak now, but again I think there's like signs of improvement. Like right now if you just like tried to a run the models in a loop. Or you said like try to ask some good questions and then resolve them, like you'd basically get garbage. Like they would try to solve a bunch of open conjectures and then like once they get stuck on whatever list of open conjectures they find, they would try to s they would start generating questions and like the questions they generate would be like pretty low quality in in like my judgment or the judgment of most mathematicians. but yeah, they're getting clearly getting better at asking questions and so like yeah, I expect them to get good at it eventually. What's a little bit relevant to the future of mathematics is a human endeavor is like will not just like will they get good at asking questions, but like will they ask the same questions I would ask? And that's much less clear, like that that's you know, I have my own personal preferences, I'm not Chat GPT, like probably we keep asking different questions and it's like very easy for me at least to generate interesting questions. so it's sort of hard for me to imagine a world where they're just like gonna have already asked everything I would ask. even if they get good at asking questions. Ravid Shwartz-Ziv: But okay, so maybe it's a good time like to talk about like the rule of mathematicians, right? Like in the sense that like okay, so if if like AI can ask the question and like and and and answer these questions, why we actually care about like human mathematician in the sense that okay, so like you can you can I agree that like it may be that like you will ask like other questions, right? But like why we why we care about it? Like why your questions like are more important than other questions? Daniel Litt: Well, they're not necessarily more important. I think they might be more important to me. But yeah, of course it's very reasonable to a to ask like why do we want human expertise in anything? like why do we want, you know, the quantum chromodynamics is as you know, that's some fundamental question about the universe, which thus far has zero applications, but nonetheless maybe we think we want people to know about it. Maybe let me give you another example. so telescopes are superhuman at looking at the stars, and astronomers are like not inventing stars, right? They're like discovering something that exists. but nonetheless, like maybe we want people to know about like the structure of the universe or something like that. Okay, so now let's imagine a world where we have superhuman AI, so it's like asking questions that in my judgment are good, and then sometimes resolving those questions. So like the output of that activity is like a lot of PDFs or something, right? okay, who cares, right? Like w w why do we want more PDFs is the question I would ask you. Like my hope is that we would want those PDFs because someone i can read them, and someone is reading them. and so like You know, okay. Presumably there some of the PDFs will have applications. Those might be important for other reasons and like well, I also think we probably if we're kind of gonna take a result produced by an AI, that at least right now we'd have no guarantees that it's like behaving well and and and sort of make it instantiated in the real world, like probably we want people to understand that. But even for like just an answer to some fundamental question in number theory or algebraic geometry or whatever, like about, you know, some basic concepts of shapes and numbers, like my view is that these outputs are worthless if no one can understand them. So like we want a reservoir of human expertise, like just so we can actually engage with like whatever great results are being produced. Ravid Shwartz-Ziv: And do do you think like like some people said that like at the end we want to that mathematicians or like math in general like generate tools, right? That like the the let's say the final goal is not only like to to to to prove something, but also like to generate like more tools that we can use them in in other like fields or like other domains. Do you agree with that? Daniel Litt: Yeah. Yeah, so in in in fact I would argue that the fundamental goal of mathematics is not really about proof only. like okay, so one thing we definitely want to know is like what is true, right? So you know, we ask some question and then like we want to know the answer for sure, and it's like really exciting to find the answer. but like actually the questions we ask are like usually operationalizations of something that is a bit more fuzzy. Right. So like the the actual thing we want is like understand like what numbers like what numbers are or what shapes are or how they behave. And then like we try to ask that question in like a really precise way and then we prove it to ground what we're doing in truth. But like the the usually the question is like meant to measure like how much do we understand this object. And a lot of the time you kind of only know if the answer has really helped you understand it post hoc. Right. So like for example, Let me yeah, like look the Riemann hypothesis or something. It's like a famous conjecture number theory. And like let's imagine we had an alien come and give or whatever, AI come and give us some kind of zero knowledge proof of the Riemann hypothesis. Okay, so this would give us like a yes or no. But like the Riemann hypothesis is also true for a reason. And like we wanna know the reason. So okay, so sometimes a proof will tell you the reason and sometimes it won't. I mean so of course for the Riemann hypothesis it's we don't we don't have a proof, so we don't know whether what a proof will tell us. but yeah, i it it's actually kind of common to have some situation where you like think That knowing the answer is going to like definitely improve your situation, your understanding of the situation, and then you get the answer and you're like, well, that didn't really help that much. And then you have to find a new question, or you know, maybe find a different proof often of the same question. and that's like a lot of what we do is like, or at a lot of what I do is like actually, you know, po you know, you get the answer and then you're not done. Like there's now a a thousand more questions that you have to to ask and like, you know, even if an AI produces a proof, like, okay, well Right now the expositions are are very bad and you work have to work really hard to understand the proof. Even if it produces a really beautiful proof that's perfectly exposited, you usually still have to work really hard to understand it. Like there's you know, it's maybe you guys have read math papers and it turns out that's like a hard thing to do, even if you're a professional. even if they're like beautifully written. And then like you're just not done, right? Like so maybe your your understanding has increased a small amount. and then okay, well met then you keep keep working to understand it. maybe look like let me give you an example. So I think the first interesting result in my view that the the AI systems proved autonomously was this resolution to the unit distance conjecture. so Urdish had some conjecture about like, you know, what's you know, i if you if you arrange endpoints in the plane, what's the maximal number of pairs that you can arrange to have unit distance one apart from one another. And so he conjectured it was like n to the one plus of one log over log log log n or some okay, some very, very slowly increasing function. And then the models slightly improved that. so they they found some sequence of constructions where you got n to the one plus epsilon, okay, for some very small value of epsilon. okay, we don't actually know what the truth is, right? Like the even once we have this great new result, which has been very generative and led to a lot of other kind of interesting constructions, like we still actually what we have is an upper bound of n to the four thirds and a lower bound of n to the, you know, one point two or something. Okay, so like here we've very slightly increased our understanding and now okay, we're almost back where we started. Like there's just this huge gap between what we know and what we wanna know. Ravid Shwartz-Ziv: do do you think like there is this like you just you mentioned like bounds or like there is this like I don't know claim that like if you actually if AI will be so good so we can actually like make our bounds so so tight, right, that like they will be actually like be practical in real life. do you agree with that? Daniel Litt: I I mean Okay, there's a couple things I kind of where I yeah, where I'm skeptical that we we see the kind of improvements you're talking about. Okay, like I I definitely think AI systems will produce results that are useful. So like the that's maybe the weak version of your question and that I I think absolutely will happen. Okay, so there's Two th two places where I think I I kind of generally have disagreement with this view. So one is like as Alan asked about miracles. I we might see some very impressive technology. I don't think we'll see like I I don't think there's good evidence we'll see like miracles. Like it's definitely not the case that like any question you ask, the models will be able to answer. okay, so there's two reasons I think that so one is this fact I mentioned that like, you know, just basic questions can get arbitrarily hard. And this is like that's not just like an article of faith. Like if you just try to try to make sense of what hardness means, there's just a pretty easy counting argument that that will tell you that like yeah, you should be able to find extremely hard questions very easily. There might be some reason like interesting questions are systematically easier, but empirically like that doesn't really seem to be the case. okay, then the other question is just like let's observe how progress happens in mathematics but also outside. I think what's really happening most of the time is like we have some body of knowledge and then like people are slowly pushing that outward. And then every once in a while, like opportunistically, you kind of have some new result that lets you like whatever resolve some basic question or like find some new opp understanding. But it's not like you know, if you had v you know, if you asked von Neumann every question, you know, without him knowing anything, he would be able to answer it. Like he's still sort of using the existing body of knowledge and like maybe pushing it out epsilon in each direction. And then okay, if we look at what the AI systems are doing now, they're really doing the same thing. So like even with the very, very best results they're producing, like you usually Usually see that they're using very recent work of typically of human mathematicians. So we saw this with Nader Stokes, we saw this with arguably the unit distance problem. So there was this quite recent work of Hajir, Mara, Ramakrishnan being used. We saw this with like this production of like construction of a non-Sophic group. So like we really see that AI systems, at least right now, are acting in this kind of same opportunistic way that like human mathematicians have acted. they're they're sort of you know still just pushing the boundary a little bit. and that, you know, so maybe we'll start pushing the boundary a little bit very quickly and then we'll see some rapid progress. That's not so clear to me. that's like kind of what my expectation is. But like I don't think we're gonna end up in some world where like you just ask any question which, you know, would it requires you to take, you know, a thousand brilliant leaps of insight and then the Ravid Shwartz-Ziv: Mm-hmm. Daniel Litt: bottom can just get there. Like just math is some hugely high dimensional space. You just can't I I I don't you know, i it it's very hard to hard to find your way around a thousand dimensional or million dimensional space. Allen Roush: So so I I have some questions that are hopefully going to go more towards the math side and and so I and again I my math background is the basically minimum needed for a computer science degree, right? But I'm fascinated Daniel Litt: That's Allen Roush: by intuitionism and mathematical constructivism and basically people who don't regard the law of excluded middle as okay, right? Like because frankly my own personal experience is I always thought that that it was obvious to people That the law of excluded middle did not match direct experience. But of course, apparently this is a common thing in computer science departments and not math departments. Anyway, what I'm curious is how do you think AI systems like today interact now with these mathematical epistemology debates? I think today my understanding is that most mathematicians view intuitionism Ravid Shwartz-Ziv: Yeah. Allen Roush: and constructivism as valuable, but still kind of heterogeneous. paradox and not very well maybe Daniel Ravid Shwartz-Ziv: Maybe maybe explain first, Ellen, like what does these terms Allen Roush: should explain it because he'll probably do it more correctly. Daniel Litt: I well yeah, I I put well I let me just say in terms of culture within the math community, there are people who worry about foundations and like basic questions of like not like logic like you're asking. The vast, vast majority of mathematicians just don't worry about this stuff at all. Like in in principle, right, so Allen Roush: huh. Daniel Litt: maybe let me kind of say like how math is supposed to work and how it actually works. So like in principle the story you hear of mathematics is that there's like some sequence of axioms. So most people think about like the Zermelo Frankel set theory axioms, maybe plus the axiom of choice. And then when you prove something, you're supposed to start at those axioms and apply some deduction rules until you get to the theorem you're proving. Okay, and then there's various sets of axioms, there's various like allowed deduction rules. So you mentioned the law of the excluded middle. That's that's a rule you might accept or not, and if you don't accept it, you might be able to prove fewer things. Okay, and then people like sometimes argue, you know, which set of axioms should we start with. you can there's a a cool p I'm not an expert in set theory, but like for there's this really cool paper by Penelope Maddie called Believing the Axioms where like she tries to explain like why she thinks some collection of axioms like i is actually a reflection of truth in some way. Okay, in practice, this is not how mathematics works. Like, if you asked the the the modal working mathematician, list the axioms of Zermella Frankel set theory, they will not be able to do it. Okay, and and to be clear, I am such a person, I cannot list for you right now the axioms of Zermelo-Frankel set set theory. and in fact, like most results you would want to prove are not really dependent on the axioms. Like for any sort of reasonable collection of axioms that people actually think about, you'll be proving the same results. And so so my view of this is that like this sort of idea of like form like kind of formal mathematics, kind start from axioms and apply them. Deduction rules. That's sort of a way of like trying to ground ourselves in truth. Like like we're supposed to in principle be able to do this. But actually we're doing something that's like kind of closer to physics. So like I'm trying to understand like what is a shape. And I have some intuitive notion of what if what is a shape. And like, you know, then one can ask questions about that intuitive notion and I'm trying to like probe those questions in some way. And I need some way to stay like make sure I'm not bullshitting. So like I still try to prove my things are correct and like try to live in some world where I can, you know, persuade myself that those proofs in principle could be formalized in whatever set theory or some other foundations. But like this is just it's just like a way of grounding ourselves in truth. It has nothing like the day to day impact of like which axioms we choose and which whatever, whether you're an intuit intuitionist or constructivist or or you know, or whether you accept the law of excluded middle, like basically is irrelevant to most mathematical practice. Allen Roush: I I do wanna just quickly note that the the type that I do care about's infinite set theory a lot in my personal reading, and I claim that those axioms are especially important there. But anyway, that that's Daniel Litt: Yeah. Can can you say more? that's good. Allen Roush: No, not honestly not really. Ma mainly Ravid Shwartz-Ziv: Ha ha ha. Allen Roush: on things like I guess the the one thing would be on Cantor's diagonalization argument and if you look into the whole debates about like can math be formalized, Hilbert and Bauer and all that, that's Daniel Litt: Probably. Allen Roush: where they started complaining and Cantor being a particularly controversial figure. That's what I'm thinking of when I cite all these, but again, you're the expertist. Daniel Litt: Yeah. Yeah, so there was definitely a like a big debate about this in the early twentieth century. yeah, I think even then it was like probably not super relevant to like the modal mathematician who was like whatever trying to understand differential equations or something like that. But yeah, I mean, you know, there have been many foundational crises. I think now this intercepts a little bit with like existing mathematical practice and and AI because you know, it's now in some cases practical to actually formalize one's arguments. So you can write take a take your paper and ask an AI model to formalize it in in most people now are using lean, which is some kind of it's called a proof assistant, so it's a way of like formally checking your proof is correct. And there's other other compet competing languages like EGDA and Coke and so on. But yeah, so so these are not it's not literally the same thing. so for example, there's no proof that lean is sound, and in fact many people there have been many errors found in it recently and patched. So like you can ask, well, how much of a guarantee is a formalization to begin with? I think it's like pretty good evidence, but not not a perfect guarantee of correctness. But yeah, I mean this is sort of interesting for practice now because we have some cases where the models have like produced some argument with an accompanying lean formalization. so that's like very, very good evidence that the argument is correct that we get before anyone actually understands it. so it's kind of a an interesting sociological development. Ravid Shwartz-Ziv: So I want to talk a bit about like how you actually like evaluate or check this proof, right? So if if human like in the past, right, like when humans actually like wrote this these proofs, so pr we we we assume that humans can go over them and check them, right? sometimes, right? Like this was our assumption at least. Daniel Litt: Yeah. Ravid Shwartz-Ziv: but like today, right, like So tell me like now if the model like generate very long proof, do you have any way to to check it? Daniel Litt: Yeah, so okay, maybe let me just say what the pre existing situation was. So it's very hard to check papers written by either human or AI. The primary evidence previously we had that a paper was correct was that a human wrote it, and then historically the author has been responsible for correctness. And this works like pretty well. So I would say like basically every paper that a human has ever written has lots of errors in it. Typically those errors are pretty minor and like they tend to shake out in the wash, like in the sense that you know, if a result actually gets used, it's it's it's checked and like sort of stress tested in various ways. Like a wrong result is like super useful 'cause the do you know about the principle of the explosion? So i it like a false statement implies everything. so It's in some sense it's kind of we expect false statements to get caught. and I think, you know, I've found lots of wrong papers in my my career, like not not my own papers, but other people's papers. Ravid Shwartz-Ziv: Yeah. Daniel Litt: this has been very you know, it's very annoying. Like, you know, you try to prove something and you're like, we we try find some useful statement in the literature and it turns out it's it's false. so yeah, so so that was the situation until recently. Okay, so the models now can produce Very long arguments. sometimes they're right. they're not always right. I think they're qu relatively reliable now. So if the models produce a relatively short argument, so let's say, you know, five to ten pages, and you ask them, is it correct? It's like quite likely correct. they're not perfect at this yet. so I have seen even with the latest models, kind of sort of significant errors. the the way those errors often arise is that they will rule you know the models will rely on a wrong statement in existing literature. That's that's kind of a typical typical kind of error. right. So then you know you can ask well what if there's a lean formalization or whatever some some kind of formalization. so often the I mean I think those are much more reliable. there's a lot of Uncertainty in general about how to understand them. So for example, it might be I I think a lean argument or a formal argument is very good evidence that the statement as it has been formalized is true. it is quite hard to tell sometimes that the formalization of a statement actually agrees with the informal statement you it's supposed to it's supposed to capture. so for example, if someone were to write down a formal statement of the Hodge conjecture in Lean, it would probably be many thousands of lines at least, and it would be very hard to check that it's kind of actually faithfully formalizing the Hotch conjecture. And I'm not sure I would believe it if the model produced a a formal statement of the Hodge conjecture. Like I'm pretty sure that that conjecture has not been formalized in Lean by by human authors. And yeah, it may it you know something well I guess we'll get there when we get there. but okay Ravid Shwartz-Ziv: But but at least like like in in my perspective like the the the major the big problem is like the intent. Like in the past like we assume that like humans again most of them right like most of the time like they have they they are coming like with good intent like they they don't cheat right and they don't like try Daniel Litt: Yeah. Ravid Shwartz-Ziv: to yeah beside some of them but But like with models we don't have a this actually. Like it can be so I I I I just like someone on on Twitter I just mentioned like there there are for example like some models will find kernels bugs in Lin, right? Lin Daniel Litt: Yeah. Ravid Shwartz-Ziv: is like a a language, programming language and so there are some bugs inside it and you can and models, right? They can try to ch if they now you give them I don't know. twenty four hours like over a week to to to prove something and they try again and again and again, at the end Daniel Litt: Yeah, they'll Ravid Shwartz-Ziv: they don't care. They will try to find and like if there is a bug and they can cheat and and prove this with a bug on the kernel in the kernel, they will do it. Daniel Litt: Yeah, so this is definitely right. So yeah, so first of all, yeah, the models will definitely try to cheat in informal proofs. Like if you if you ask them to prove something, they will often kind of like yes at a key point. If if they can't, you know, sorry, this is I should say this really only happens at the edge of capabilities. So if they can prove something, like they definitely will do it. Like they they they won't lie just for fun. But yeah, so as you say, if you run them for twenty-four hours on goal mode or something, they'll they may try to trick you. At the end, so either in informal proof or in Laten, right? So they might, you know, try to exploit some kind of kernel bug. I think there are examples where maybe this has been seen, although it's kind of unusual. Or they might try to misformalize a statement or something like this. so yeah, this happens but it's not as bad as you might think. And the reason why is we can just like ask them, did you formalize this faithfully? And like usually they'll say no. I mean like there there's no the models don't coordinate with each other to trick you. So Ravid Shwartz-Ziv: do but do you think that this is something that like there are a lot of discussion about alignment these days, right? And Daniel Litt: Yeah. Ravid Shwartz-Ziv: safety. and like what will happen when the model will be too smart and like will cheat and hide their chain of thought and all these things, right? do you think it will happen if it will happen also in math? Daniel Litt: I mean I we definitely see examples of it, but all yeah, all I'm saying like this, you know, the models definitely are do not behave in the same kind of good faith that like we hope human mathematicians do. but because they, you know, different instances of the model don't coordinate with each other, it's sort of not such a big deal, like as long as you actually actually ask them to to check their work. So yeah, this is actually like a real failure mode we see. Like we there are people who are like trying to go through the whole literature and like just produce a paper on every conjecture. And like a lot when someone does this, like a lot of those outputs are wrong even though the models are really reliable and it's just because they're like trying to elicit some output and like don't care care if it's true. Like I think this is an example of human misalignment. and like if you take those those outputs and just put them back in the model and go, is this correct? Like usually the model will know they're not. So not always. The models don't don't always find their own errors, but it's it's you know you can really do a lot to kind of reduce this kind of failure mode if you're just, you know have, you know, even the tiniest amount of due diligence, which not everyone using the models to do math. Ravid Shwartz-Ziv: So so you don't think we need a new ways to to review and check papers? Daniel Litt: we we definitely do. I mean but the the main issue is like not correctness. Like I think that we you know, for ex formalization, you know, is not practical always. Like, you know, this whatever formalization from Oslas theorem maybe costs somewhere between three hundred thousand and a million dollars and like I don't have that kind of money to burn on on formalizing math, but like in a year probably it'll be much cheaper. And I think that will provide like really good evidence of correctness. Like way, way better than we've had in the past, you know, So just comparing, you know, my guess is that like basically every paper written at least two years ago has lots of minor errors in it and some of them have lots of major errors. And you know, that's something that's gonna get a lot better. Where we need we're gonna have to think about how we do review is just like we also want to like learn something from the outputs, bes yeah, besides binary yes or no, and like probably there's gonna be in the next few years more papers produced, like more mathematical PDFs produced than have been produced in the entire history of like human mathematics. And so like right now our existing systems are definitely not robust to that. And so, you know, we want we'll want people to engage with some portion of that and figuring out which portion and how to get people to engage with it is like I think a pretty serious problem the the profession has to has to to figure out. Allen Roush: So how how big of a problem is there with slop in the sense of like repetitive overused phraseology possibly leading to diversity collapse and the kinds of problems that systems would work on, the c ways that they prove results, Ravid Shwartz-Ziv: Mm-hmm. Allen Roush: the the results themselves, et cetera? Like do you think that humans, even well after AI systems are super super human at math, will retain value as diversity generators? Daniel Litt: I think that's a possibility. yeah, so I think We definitely see signs of this now. So there are multiple examples now of of different teams putting out like proofs of the same result due to whatever ChatGPT 5.6 or ChatGPT six or Claude, like more or less simultaneously. Like the the models seem to be just doing the same things, basically. So even like different models, so ChatGPT and Claude, like the collection of of results they can prove seems to be like basically the same. So I know if you remember but Ravid Shwartz-Ziv: Why? D do you have a guess why? Daniel Litt: it's a good question. and partly it's like a lot of the you know, there's certain things they're very good at, like the you know, the the main thing the models are really, really good at is like taking a bunch of known techniques and then applying them in a technically strong way. And this is something where like a I think a lot of conjectures are accessible this way. That's like a good portion of what human mathematicians do. But you know, sometimes you need to develop a new technique. I'm not saying the models can't do that. Like, I think there are examples of that too, but they're way, way weaker at that. And so, you know, the the sort of the strengths the models have just like play to like just this having this super broad knowledge and like willingness to grind and not getting tired. And so the results they're proving are like the ones where those skills are really useful and like where the where the skills they're kind of bad at are are not as useful. but and also what I'm describing is like kind of a feature of human mathematics also, right? Like the models are being used by human mathematicians or people at the labs who might be mathematicians and like so there's some collection of people who are like trawling through some collection of conjectures and not adding anything new and just kind of pressing the button. And you know those those groups of people are proving the same conjectures 'cause just they're doing the same thing and not adding any kind of as you said, cognitive diversity. yeah, so right now, you know I'm working on a bunch of questions where like I'm pretty sure I'm the only person who's ever even thought to ask these questions. So I you know, let alone the model asking asking the questions. And so like I'm not really worried about getting scooped or, you know, duplicated work for this reason. and yeah, I think Yeah, this is and one possibility is that the models end up like really, really superhuman, but having some kind of idiosyncratic preferences that means we continue to add diversity. I I worry about this a little bit, like it's not clear to me how we measure the value add there, which is sort of a problem for, you know, the allocative aspects of the profession, like giving people jobs or grants or whatever. And so it's not clear to me, you know, if we can't measure these inputs, it's not clear to me how we preserve them long term. But yeah, I think it's a possibility. you know, my expectation is that the models will get pretty good at doing diverse very diverse things. Like I I'm not super concerned about this. I think that but yeah, it's something you know I think we should try to measure it and try to understand whether Ravid Shwartz-Ziv: So h how do you see like how do you use models like in your day to day life, right? In your own your Daniel Litt: Yeah. Ravid Shwartz-Ziv: domain? Like are you trying like how how this process looks like? Daniel Litt: Yeah, so okay, so I I do not typically just like as a matter of kind of I'm not interested in just like getting the models to like one-shot some famous conjecture. Like that's not in my view a good use of my time. so mostly like I have some collection of questions I'm studying. And right now I kind of treat the models as like kind of a research notebook, so I'll just like discuss my thoughts and questions with them. and there have been yeah, so for actually a long time the models like way, way longer than like way after the models got like good at math, which was maybe January or something, they were really not so useful for this because a lot of the time these questions were not really precise. so I think I just had the first a first example of a model really answering kind of a not quite precise question I had. with a sort of high quality answer with a GPT six like two weeks ago. and it was like still just kind of a partial answer. so, you know, that's why I I say, you know, I think we do see signs of the models getting better at at some of these more fuzzy skills too. But it's really quite recent. so yeah, mostly I would say where I actually get value out of them is like kind of literature search and just learning. So like I don't know, last summer I needed to learn something about hyperkiller manifolds and like I learned a lot about that from O three, which was which was fun because it was like you know, O three was not reliable. Like unlike the current models that would make lots of varieties and but you it was still kind of fun to have have an entity to to talk to about it. Yeah, these days I'm learning about like micro local sheep theory from from the models and also of course reading papers. so you know, I I think mostly I'm trying to take advantage of the fact that the models have these strengths of like basically knowing everything. Like you can just ask them questions about known things and they're you know, they don't have very good theory of mind, so they don't get great explanations, but they give some kind of explanation. you know, of course, if I have some question that's like precisely stated, and I have no idea what to do, I will sometimes ask the model just to see what it what comes out. you know. to be honest, like most of the time if I'm working in my area of expertise I have like a pretty good sense of whether a problem is accessible to known techniques. so this is you know what often happens is like I'll be like, you know, pretty much know what to do and then I also does the the model does this work and then it will think through it and say either yes or no. There have been a couple yeah, usually if I ask just some kind of like moonshot question, like like always, you know, can you prove such and such, is this true? Like the typical success is it just gives like an opp like a stupid counterexample that I missed. And if it's some kind of deep thing and, you know, usually isn't isn't super useful yet. Well, you know, who knows. But yeah, so I I kind of just treat the models as a research notebook, I would say, and then, you know, there's some amount of reproducting and sometimes they say useful things, but Yeah, it's not it's quite different from the sort of like one shotting Navier Stokes stuff that that seems to be getting a lot of a lot of media attention. Ravid Shwartz-Ziv: And will you be happy or or sad if like models will be so good that you can just like zero shot on these on like your like moon shoot questions? Daniel Litt: Yeah, so right. So I think you mean like you mean precise I I think the answer is I would be happy for any version of this question. Like I I re you know one of the things that like one of the reasons I'm a mathematician is that like I really I have all these questions and I really hate not knowing the answers and like I wanna know the answers. I don't think the model giving me an answer is like the end of my job. Like I still have to understand it. and that's kind of really hard to read math. Like it's like almost as hard to understand written math as it is to discover it to begin with. so that's like kind of the you know that to me that's kind Ravid Shwartz-Ziv: Mm-hmm. Daniel Litt: of the situation. I think as long as the models are superhuman only at answering precisely stated questions, which seems to be where we're trending in for the near term, they're basically like automating, I don't know, a tenth of my job or something like that. And that doesn't that sounds like purely good. I do wonder a little bit what the job will look like once they start getting better at these sort of more vague questions. I think I would still enjoy it, but I I'm you know, we'll we'll have to you know, if that happens or when that happens we'll have to think a little bit more about what the what the job will really look like, you know. you know, personally a big part of what I like about the job is like you know, interacting with other mathematicians and like discussing stuff at the board. And so, you know, I think there are a lot of people concerned that the profession will become less fun. It's a very I love math. It's a really fun profession. and I think that's like a reasonable concern. definitely things will change and it's not obviously that that like the the fun will will remain the same or increase. but you know, I think it's at least possible that that it stays fun or even gets way more fun. We'll see. Allen Roush: Do we do you do you think that the prestige or value associated with solving a novel problem as a result of utilizing AI will will go down? because I'm and and if so, how do human academics in math or really any other field separate themselves? I mean, we humans apparently do believe in meritocracy, even though the person who coined that word was doing so in a negative way, but since we we have implicitly do think that it's good, then how do we maintain it? Daniel Litt: So okay, I think prestige and credit are dead. Like there's just no way they survive what's coming. I think that in fact, not only will solving a problem with AI be less prestigious, solving a problem at all will be less prestigious because we won't know whether you did it. okay, nonetheless, as you say, like we still would like to allocate you know, to the extent that we have scarcity, we need to allocate scarce resources, and like to some extent we we probably want to do that according to some notion of merit. So what will we do? I'm there's some kind of some kind of meritorious things you can do that I think just are intrinsically impossible to automate, like human understanding, the models can't understand something for you. at least, you know, until they put a chip in your head or something like that. I'm sorry, to be clear, I think this is very unlikely to to happen. but yeah, so what we just we'll try to do, I think, is find operationalizations of like the forms of merit that are not automatable. So I wrote this essay beginning for mathematics where I suggested like kind of no longer evaluating written text basically, just because the models will be producing an unbelievable amount of it and we won't know what people produced and what the models produced. Instead of primarily evaluating things like a thesis defense or, you know, seminars or or sustained conversation or like things like where you build a research community. so those are all things like where you can't you can't have a computer do it for you. and I think that's kind of where we'll end up going. it's a little unfortunate that some of these things are like arguably gonna be a bit more subjective than our existing criteria, but you know. Even in math, which is like arguably this like super objective field where we're like worried about just the you know, did you prove a theorem? There's still this subjective judgment, like was it interesting? Did it actually like answer the question that the thing you proved was meant to answer, that kind of thing. So in some ways it's not so different from Ravid Shwartz-Ziv: So but but do you think Daniel Litt: Directly up here. Ravid Shwartz-Ziv: like do you think you can I don't know separate like distinguish between now a PhD that worked like six months on a problem but didn't solve it and a PhD that worked and like and then solve it with with the AI or like a PhD that worked six months on a problem and solve it by by himself? Like d d do do you see what I mean? Because like now when you want to you go to Daniel Litt: I think that me can Ravid Shwartz-Ziv: evaluate him, like now you need to talk with him and and how you like how you want to to do it. Daniel Litt: Yeah, I think that we won't know, obviously, whether someone used AI. I think that's just not you know, you just can't tell. I Ravid Shwartz-Ziv: Do you think it's important? Daniel Litt: n I'm sorry, it's important in the sense that we would like to know how, you know, how good someone is at understanding something. But also we have like face reality and so like it is not an it is not a good idea to try to do impossible things. Like so yeah. we we cannot know whether someone used AI. What I do think is that like In practice, so suppose you like click a button and have the model write your PhD thesis, right? Like, so then what does the defense look like? Probably it looks pretty bad. so you know, I think in practice doing things like where like like trying to directly instead of trying to directly evaluate the written work, like you try to evaluate the person who produced it, it probably incentivizes people to like at least engage more deeply with the math. even if kind of fundamental ideas are coming from AI. And to be clear, like I think it's sort of totally fine if if AI does a lot of important mathematics. Like it's even great. Like we also would like to incentivize like high quality outputs. Like we want we have all these basic questions we want resolved. I think what we just don't want is someone, you know, is someone playing the slot machine. So like you know, there's this goal of like incentivizing high quality outputs, like that we definitely want. And we also want to maintain human expertise. And those I think if we kind of stick with the current incentive structure of mathematics, those two goals are intention, but they don't have to be. Like we just have to redesign incentives a little bit. Allen Roush: So, do you do you then think that all of the so called extinction rhetoric and concerns about loss of control and autonomous hacking and all of the recent developments, what's what what do you think of all this? Like do you think it's justified? are the pauses good, etc., to the extent that you're willing to talk about any of this? Daniel Litt: Yeah, I'm I feel very unsure about the future. I I I've made wrong a lot of wrong predictions about AI capabilities in the past. it seems pretty clear to me that like cybersecurity stuff is like a real concern, for example. Like that probably there will be a lot of hacking in the next year or so as open models get really good at it. I'm a little skeptical of like AI killing everyone. you know, i it seems to me like there's a lot of efforts to Align AI and control it and like, you know, what happens if we do nothing is very different from what happens maybe if those efforts continue. my personal worry is mostly about like social upheaval. so we've already seen like how the math community is reacting to sort of highly capable models. I would expect a comparable reaction in all professions. It's sort of you know, that seems to me like y you know, the math community is not particularly rowdy, but you know, I I think we can expect, you know, kind of a lot of social upheaval and that that I'm a personally a little bit more worried about than like the models turning us all into Grey Goo. But I don't mean to dis dismiss that worry. That's just something that I I think we'll probably see some more mere warning shots before the models kill everyone and then Ravid Shwartz-Ziv: But d do you think that it will happen in other fields because like there is this argument that math like and the end is like maybe like the perfect framework for for models, right? Everything is verifiable, everything is like in in the simulation, right? Like you don't have you you don't interact with the real world. so you can Daniel Litt: Yeah. Ravid Shwartz-Ziv: run just like spin up I don't know, thousand of agents that will work together and will generate a lot of proofs and a lot of things together and we'll validate it and but the real world is not like that. Like we talked with some people that I don't know design drugs or things like that. and they're saying look like we have like coding agents are good in in some parts of the of the process but like at the end we have the real world and we need to to test it and we need to we can't accelerate like the time. Daniel Litt: Yeah. So this is definitely something people say. I think it's probably wrong. No, Ravid Shwartz-Ziv: Do you agree? Daniel Litt: I think it's probably wrong. so what is true is that like the RL loop in math is fast. that's that's clearly right. What is not true is that math is like uniquely verifiable. so what I was pointing out is it's actually very hard to tell if in like an in all proof is correct. That's like a difficult task that's like not You know, you can try to formalize it. That's not as I understand of what the models what what the Frontier Labs do in order to improve math capabilities. And in fact, like until quite recently, like they were were, you know, it was basically impossible to formalize a lot of kind of mathematical statements. Like it was just not something the models could do. so for example, I don't think there's a a formalization of lots of basic statements in algebraic geometry. Not saying it's impossible now, but just that it's expensive. so what that means is that actually the the the frontier models were able to get very, very good at informal math reasoning, which is much less verifiable. So I think we should understand this as a sign that the models are gonna get really good at a lot of less verifiable tasks. you know, what the rate of improvement will be is another question, but I yeah, I I think that We should understand this as like coming to at least all knowledge where Ravid Shwartz-Ziv: So so why why math is like the first domain that had such acute success, right? Like even in code, Daniel Litt: Yeah. Ravid Shwartz-Ziv: it like we thought that like, like yeah, we you can write and you can run and you can test and everything. Models are good, but they are not solving like Daniel Litt: Yeah. Ravid Shwartz-Ziv: end to end problems as we see in math. Daniel Litt: They are well they sort of are, right? Like, you know, you can you can ask it to to to rewrite FFmpeg from scratch and they it can do this. What I think is actually what happened is the models have gotten about as good, like very, very good at the same sliver of tasks in math as they have in other areas. And it just happened to be the case that that was those that was like the sliver of tasks that math used as a a metric. so like basically math got good hearted. So, you know, like in fact as I was saying, like, you know, there's this huge bundle of tasks I do, like ninety percent of my job doesn't really feel particularly close to being automated right now, but okay, the the part of it that we use to like hire people and give grants definitely is a lot closer to being automated. So I think this is just something about the structure of math as a profession rather than like the structure of math itself. you know. So yeah, it's just, you know, physics, for example, doesn't have these big open problem lists that are like highly prestigious. Like their open problems are not even kind of precisely stated usually. my sense from talking to physics physicists is that the models are like really, really good at some things in physics and not really good at other things in physics. But the things they're good at aren't like the things that would win you a Nobel Prize. whereas in math, you know, they're now very good at things that we would like give you an award for. Allen Roush: Do you so so then do do you think that we'll continue even five or ten years from now where it would still be a jagged frontier and even a few individual capabilities humans will still beat them in? Or do you think that they'll catch up even to the point of being kind of near human on even the worst things? Daniel Litt: So I would say I expect the models to become broadly superhuman. not just within math. just because we see you know, I think like you can kind of try to try to evaluate the monfuzzy skills and we do seem to be seeing some slow improvement. I don't have a particular time like I've kind of been burned before making precise predictions on timescale. So let me maybe let me not do it now. But it it seems plausible to me at least. What I do think is possible is that there will be like a bunch of kind of fuzzy skills where we like just don't know if the models are better at them than us. Like just because like some stuff is really hard to measure, and that's also the stuff that's hard to hill climb. Right? So what seems kind of plausible to me is like we end up in this world where the models like everything you can measure seem to be super, super good. And then we're just like not sure, like can they do these sort fuzzy tasks? Like how do you tell even, like if someone can do a fuzzy task? And so okay, in practice, what does that look like in math? Maybe it looks like the models are like proving all kinds of amazing theorems, They're doing really, really high quality work, and then every once in a while, like the models can't prove some theorem and a person does. or you know, there's some other task the models seem not to be able to do just intermittently, and a person sometimes can do it. again, you know, even if this happens, like maybe that person knows it's happening, but like we don't know for sure because how do we know that they didn't cheat and use a model? So yeah, I I I think that's kind of that's sort of to me the most plausible future where the models don't become robustly superhuman. Like there's just a bunch of hard to measure stuff that that they're they're not so good at, but we don't really know for sure. That seems a little bit less likely to me than them actually becoming robustly superhuman. Ravid Shwartz-Ziv: And and why do you think like all the the big labs actually like put so much effort on math? Like it was it Daniel Litt: Yeah, okay. So I first of all I I think they're not actually. I for a long time I think they were. and then I think now they're Ravid Shwartz-Ziv: Okay. Daniel Litt: you know maybe they I guess they've spent fifteen million dollars trying to prove neighbor stokes. the reason they did that is because they heard a rumor the other lab was doing it. Right? It's not that Ravid Shwartz-Ziv: But why why you think like d do you think like it's I don't know, it's some like scientific curiosity? Do you think like it's a PR thing? Do you think like they actually like make their models good in other things because of of math? Daniel Litt: Okay, yeah, so okay. In practice, I think what's happening is three things. So like Sam Altman definitely, like, I don't think he cares about math. I mean, I I haven't talked to him, but I would be surprised if, you know, like if it's a big deal to him. They probably for the C suite it's about PR. what is true is that historically, the several companies have tried to Get the models to get good at math as a way of developing techniques that they hoped would generalize to other areas. So for example, for a long time the models were hill climbing on a benchmark based on the high school math contest, the AME, AIME. and my understanding is that a lot of the techniques used to climb that benchmark ended up generalizing and like kind of being important for reasoning models and so on. So it's like a test bed where you can, you know, you can test out your research ideas and and maybe get relatively quick feedback that they're that they're Four sort of very verifiable math problems like the Amy problems are are things with numerical answers. You don't need a check a proof. Okay, so I I think that historically that was kind of the the one of the main reasons the labs were working on math. My understanding from talking to people and to be clear, this is just a rumor, I don't really have any inside information, is that that hasn't been the case for some time. Like they're the labs are not investing a huge amount just on math. I think the reason they're doing math now is like the labs have hired a lot of mathematicians and the people those people like math and like asking math questions to the models. So I yeah, I that that's that third reason is my favorite reason. That's also why I do math. Ravid Shwartz-Ziv: And d do you think like now it's a good PR? It's still a good PR? Because like there were all these like rumors that now like after the the backlash they got from previous publication now they don't want to publish some of these results. Daniel Litt: Yeah. So so proofing a millennium problem is clearly good PR for the labs, even if some people complain about it. there is a question, you know, okay, so my favorite c open conjecture is the growth in decats P curvature conjecture. I think if the labs released approve of that, I would be like pretty excited about it. And the total number of people who would be excited about it in the world is like seven. And then probably like a lot of, you know, maybe okay, maybe a hundred or something. And then a lot of people would be like, you know, kind of complain the the you know, some some bad behavior on the part of the labs or norm violation or something like this. So yeah, for something like that it's not clear to to me that the labs benefit in terms of PR. because it's like, you know, the median math conjecture is like pretty obscure. but, you know, I think they are trying to convince investors that the models are really smart. and you know, it does seem like proving conjectures has that effect sometimes, rightly or not. so I don't know. I I I think your your question, you know, the part of your question is the implication that it might be bad PR because there are like some questions around academic norms and like how you should put around a result. Certainly, you know, a proliferation of slop PDFs is like kind of obnoxious in some ways. I think that what's really the case is like n there's no agreement on what the norms should be. And so like there's a big argument in the community about this. my personal view is that I I I I don't think the existence of a proof, even if it's slop, can hurt you. I think the month the companies should just release everything. you know, one one To be honest, I think a big part of like why there people are seeing complaints is that like you know, academics when they prove a result, they don't usually do a press release. and that's like kind of gauche. so you know, one way the companies could avoid a lot of blowback is just like not make a big deal out of it. I think that would be that would you know, that would go some way to help. You know, one thing is when the the when OpenAI released this proof of unit distance problem you a bunch of mathematicians, including me, were asked to like r like write a good version of it. And I think that went a long way also to you know, basically there was no criticism of that release because it was sort of, you know, the community was involved in some way. and one of the basic problems here is that there are these like race dynamics between open AI and anthropic, which means they have to rush. And so they're not doing that. they're not kind of like, you know like the the For them, the main thing is to be first and so they put out some slot. and you know I for you know, my personal view is I don't really care if they put out a bit bad PDF. Like that that's fine, we'll figure it out. what I do find worrying is the race dynamics. Like, you know, we see here that they they know what they should be doing. Like they did put the unit distance conjecture, they did it right. And then when there's race dynamics, they decide like, okay, well, I guess we know we should do this, but we're not gonna do it. So the the kind of whatever principle they know they should follow is not being adhered to because they have to beat the other company. And no, Anthropic is doing this too, to be clear. It's not just opening eye. so I I'm kind of worried about that because I think there are other more important principles at at stake here than like the sanctity of academic publishing, which frankly I don't care about. and I'm worried that those principles will also be thrown under the bus, you know, because of these race dynamics. Ravid Shwartz-Ziv: And Allen Roush: Mm. so so do you think then that that the the pause or slowdown that the Frontier Labs is doing is is good or bad? Like and and in particular, do you think that the negative PR that is being created or has been created will continue to lead to stronger, I would call them Luddite style movements that further radicalize the general population against AI and possibly act very counterintuitive or counterproductively for Daniel Litt: Yeah, okay. So first of all, do I think the pause is good? I think if it's real it's good, sure. you know, there's clearly some you know, alignment issues that it would be good to solve. my cynical expectation is that it will last until Deep Seek releases a g a good new model or no you know I I'm yeah, I'm a little cynical about the labs' behavior. I think the yeah, it's good to publicly, you know, that both the labs have agreed to to pause. Like that's a good signal, but you know, let's let's wait and see how it goes before applauding them. in terms of like public movements, you know, I do think that we're going to see a lot of Yeah, I don't I don't want to say La Dike movement. I think mm so I think people are very reasonably concerned about their livelihoods. about, you know, even let's just say in the math com community, I think it's very reasonable for PhD students to be concerned about, you know, what what jobs will look like. It's useful for it's very reasonable for mathematicians to to say like this isn't what I signed up for, like I like these certain aspects of the jobs, I don't want job, I don't want them to go away. like that seems to me like I I think that there's gonna be a lot of great stuff to come out of the technology, but it's also fine for people to mourn it. I do think that there is a dangerous trend I'm observing where you know, we have some value. Like I wanna know the answer to some questions, right? And then the models can do it, and then we say, like, well I didn't really wanna know the answer. I wanted some, you know, more mystical understanding or something like that and like, okay, now the models can help us understand something and then say, well I didn't really want to understand something. It's actually important that it is human artisanal. like I I think there's some kind of worrying trend I see where we take all our values and like once the models can satisfy those values we throw them away and we say they weren't our values to begin with. it's some kind of some moral version of moving the goal posts. And I I worry that we end up in a place where we don't have any values. All we all we say is that like all we care about is humans doing something. Maybe that's what you mean by a by a Luddite movement. and personally I think we should try to hold on to our values and and you know, if the models can help us do amazing mathematics or whatever, produce you know, whatever cool thing we care about, like we should be excited about that. Ravid Shwartz-Ziv: And what about you? Do you think like do do you think about going to one of these labs to to have infinite amount of compute? Asking models all Daniel Litt: do I think about it. Ravid Shwartz-Ziv: day they your questions? Daniel Litt: You know okay, I would be lying if I say the thought that said the thought never crossed my mind. Like it would be cool to be like the guy who, you know, is is in some way involved in whatever result in the Ruben hypothesis or something, if that's something the models will do. that said, like, to be honest, the appeal of being the guy who presses the button is like really not that's not like I don't Like I wanna know the answer. it doesn't have you know, if someone else presses the button, fine. Like, you know. We all did the s if if what you did is press the button, I would say we did the same amount, to solve the problem. Ravid Shwartz-Ziv: But but you but but you will get more money also. Daniel Litt: That's true. Yeah. Well let me put it this way if if I wanted to if I if that was highly motivating for me, I would not be a math professor. Ravid Shwartz-Ziv: okay, I think like so so there are a lot of questions from like on Twitter and LinkedIn, I will choose some of them. so one question, so if insight and understanding is to become more viable in the future, is there a way to formalize it and make it explicit, in lean proof of something like that? Daniel Litt: Yeah, I don't think so. I mean in my view, understanding is something that happens in your head. I don't think a piece of paper can understand something. Or whatever. you know, maybe the model weights in some ways understand you know, the the some understanding the model weights. I think like we have some evidence of that, like insofar as, you know, proving a conjecture measures understanding. but you know When we talk about understanding already, like okay, like Jean Pierre Serre or like Maxim Khansevich like understands a lot of things I don't. And that's like not satisfying to me. Like I also wanna understand them. So yeah, I don't I don't think it's something that it makes sense to formalize. Like it's something like the formalization of understanding cannot be understanding 'cause that's something that happens in your head. Like you can't you can't extract it and put it somewhere else. So yeah, I yeah, I I I see no way one could reasonably formalize this. Yeah. Ravid Shwartz-Ziv: And and and what about like I don't know intelligence. Do you think there is a good way to to measure or like to to formalize this one? like we we talk like it's also it's also related like to to another question that someone asked that like Basically said that yeah, like in LLM, right, we can think about it as like mathematical object, right? Like a huge function Daniel Litt: Mm-hmm, mm-hmm. Ravid Shwartz-Ziv: and build out of like linear algebra optimization, probability. but but somehow like we like and and we have kind of like intelligent. from these systems right like it can be debatable what does it actually mean but like it kind of like it emerged from these these objects but no one actually or like most of the mathematician mathematicians don't talk about it or like don't try to to analyze what does it actually mean right so what do you think about it Daniel Litt: Right. you know, I mean you know, what you're what we're really doing here is we're taking some English words like understanding or intelligence and we're trying to find some precise operationalization of them. But like they're not precisely defined objects, right? So like any like literally any operationalization will not be intelligence or understanding or something. It's just like the the those words don't have precise meanings. So that means they're just impossible to you know, you can't if you make them precise you've defined a different concept. Okay, of course people try to understand intelligence. Like one way you might un operationalize it is just like, you know, can't you know, you say A is more intelligent than B if for every, you know, whatever puzzle or math problem or whatever, B can solve, A can also do it, and they can also solve more. That's like a fine operationalization. I don't think it's like what people mean when they say the word intelligence. it's so you know, I I you know, just at the level of like good epistemology, I think it's kind of good to distinguish our imprecise terms, which really like they mean something. Like the notion of understanding is like a very useful concept, even if it's hard to operationalize, from whatever we use to measure it. so yeah, that that's my take on that. I like I think it's kind of I'm you know, I'm a mathematician, maybe it's like disorienting t for t to to hear me go go to bat for imprecision, but like I think it's kind of like philosophically important to be willing to use imprecise term imprecise terms and like try to come come up with some shared understanding of them that isn't like precisely measurable. Like I I think if you kind of try to formalize everything and try to operationalize everything, like you lose something. And that's something that happens in math too, right? Like we were talking earlier about like sometimes a conjecture is supposed to measure understanding and it like ends up not. Like you're allowed to, you know, you have your operationalization of something like whatever, proving famous open conjectures, that's like some kind of operationalization of intelligence. And we're allowed to come back and say, like, well, you know, that's really cool. That did something we wanted. It didn't do everything we wanted. Like post hoc, you're allowed to evaluate like was that operationalization perfect? Ravid Shwartz-Ziv: Okay and final question. how do you recommend I teach my math to my thirteen years old? I think problem solving competitions are still valuable. but there is like this like change that we that we already talked about. Daniel Litt: Yeah. Ravid Shwartz-Ziv: So yeah, what is your your take for teaching math? Daniel Litt: Yeah, so so I've I have the same question, I guess. I I have a three year old who, you know, I'm I I'm teaching her how to read and she can she can add single digit numbers, but you know we're we're yeah, so you know I'm not a child psychologist. my understanding is that like generally speaking, screen time is is something you should keep minimal for a while. My personal view is that a lot of the skills that we care about now will still be valuable. like it will still be very important to know how to think, even if L LMs are very good at it. Like probably you will not have a rich inner life if you allow you know the models to do your thinking for you. the primary difficulty I think that you know I face as a edu I teach undergrads I I face as an educator and I think people, you know, much earlier in the pipeline also face as an educator is that any evaluation that does not happen in person can now be you know, delegated to an LLM and the the models are very, very good at, you know, like whatever solving problem sets. so that's you know, there's actually some like incentive to do that, right? Like you get a better grade, maybe if you if you delegate your work to an L Lm. You know, as an individual I don't know how much you can do. Like you can, you know, not not you know, discourage your kid from spending too much time in front of the computer. I think like primarily this is up to our institutions. Like, you know, for example, now I don't grade homework for correctness anymore. I only grade in class exams or oral exams. so my view is this incentivizes people not to cheat on their homework because you well you don't get a better grade if you do and then if you do i i i if you have in fact delegated all your thing to the L you're gonna bomb the exams. you know, you can do versions of this in the home too, I guess. I also think institutions should probably make it easier to not use L LMs in some ways. So for example, like one idea I've been playing around with is the idea of like a math lab where you know, you know, and when you do take a physics course or a chemistry course you have lab time where you go in and you like actually use the equipment. What I would suggest is a, you know, a math lab where you go into the room and you actually use your pencil and paper and brain. and you know, maybe there's no internet activity in this room. you know, you still need some kind of social structures to make people actually work rather than goofing off. but yeah, I think, you know, a big part of the purpose of college and and and elementary education is just like having a structure that gets people to work. and, you know, for highly motivated students they don't need that structure. And this, you know, this is like the golden age, like Any question they have, they can get an answer from the LM and that's amazing. For less motivated students, we need that structure just to get them to, you know, think at all. I think we'll be able to find it. But yeah, you know, right right now I I'm sorry I don't have great advice for for the present question, but I think, you know, it's important just think about Ravid Shwartz-Ziv: No, that's good. Daniel Litt: how to how to make sure they're actually doing Ravid Shwartz-Ziv: Okay, one final question whether like when you think we'll have the answer if p equals NP. w if and when Daniel Litt: you know that's a good question. I I don't know. I'm not a I you know, I I quite like the question P equals NP. I I would say w one day a year I I actually I try to think and think sit sit back and think, do I have an idea about p P versus NP? 'Cause it's something where I do think it's possible that there's some kind of basic idea that would be would be useful. but you know, when you would try to ask like when will problem X be solved, there's a big w open open question which is how hard is X and that we have no information about. so yeah, I have no idea. Ravid Shwartz-Ziv: D do you think it would be in the the next five years? Daniel Litt: So okay, my this is going to be unpopular with some of your viewers. My personal view is that it's quite possible that P versus NP is not that hard. like okay, there's been no progress, but it could be it could be one missing idea away. I think there are other questions in in math that I'm more familiar with where I have the sense like you have to defeat these three or these five obstructions to get there. and I don't know yeah, I don't know if there are five obstructions or a hundred. Obstructions on the way to P versus NP. It seems to me it could just be one. with the Riemann hypothesis, I think there are several, for example, so I would be a little bit surprised to see a proof of the Riemann hypothesis really soon. That said, like, you know, historically, when you look at the kind of the time between a big idea that's relevant and a problem getting solved, it's been kind of short, like for Mozl's theorem, like maybe the last, even just between The time between the Tanayama Shimura conject Ushimura Tanayama conjecture and the proof from Osless theorem is not that long. It's like fifteen, twenty years. So it could be that, you know, if there in these cases, if there is just like one big idea missing, then maybe it's soon. But yeah, hard to say anybody. Ravid Shwartz-Ziv: Okay. Okay. we're out of time. Do you have anything else that you want to add? Daniel Litt: No, this was was a lot of fun. Yeah, thank guys for having me on. Allen Roush: It was a pleasure. Ravid Shwartz-Ziv: Thank you so much. It's really pleasure. Thank you so much. Daniel Litt: All right. I look forward to it. All right. Bye bye.