Latest / Cheeky Pint / OpenAI cofounder Greg Brockman on the scaling hypothesis and refactoring as a killer AI use case

Transcript

OpenAI cofounder Greg Brockman on the scaling hypothesis and refactoring as a killer AI use case · Cheeky Pint

• John Collison: Okay. Well, I'll just dive straight into all the questions I have, which is a lot. Greg Brockman: All right. John Collison: If you were not working in AI, how could one have known that something was about to start working? People were telling you that AI was the future in the 1970s, in the 1980s, in the 1990s. And then, very quickly, in the late 2010s, everything started happening. Greg Brockman: Well, I was someone who was not in the field, and so I remember very much what it was like. 2013, 2014, it felt like every day on Hacker News there would be a new “deep learning for X” article, and I remember being like, "What is deep learning?" I knew, like, one person in the field, and I asked them to introduce me to more people in the field, and I just kept getting introduced to a bunch of my smartest friends from college. Now, if you actually look at the work that was being done… 2012, basically, image recognition, for the first time, you could solve with a neural net much better than anything else. It just blew all these traditional computer vision approaches out of the water. It's like this learned system is able to outperform 40 years worth of, "Let's write down all the rules and try to handcraft the algorithm for the task." It's very easy to then be like, "Okay, well, this approach, sure it works for computer vision, but it's never going to work for machine translation." In 2014, suddenly, you're getting great results in machine translation. I think that this pattern was applied in subfield after subfield. John Collison: One thing I've been wondering about is, so many different things are finally working at the same time. We have LLMs, which are obviously amazing, but then we also separately have image models really working. We also have text-to-speech and speech-to-text working way better than they were before. What's the common factor behind everything starting to work at the same time? Greg Brockman: Well, it's deep learning, right? I think deep learning is the core— John Collison: We’ve had deep learning for a long time. Why didn't deep learning work in the 1980s? Greg Brockman: If you look at the number of orders of magnitude of compute that we've gone through from 1940 to today, I mean, it's just astounding. John Collison: You think they're all explained by compute scale-ups applied to the right algorithms? Greg Brockman: Of course, the type of algorithm changes, and some of those results aren't even deep learning. But I think that fundamentally, it is about compute, and you need an algorithm that is scalable that can actually absorb that compute. John Collison: Was OpenAI the first company to take the scaling hypothesis really seriously? Greg Brockman: I think that claiming the first is always difficult, but I think that it is clear that we succeeded much more wildly sooner than anyone else. I think that we had real conviction behind what we needed to do. Some people think that OpenAI set out to prove the scale hypothesis, whereas it was almost the other way around. The scale hypothesis is what we observed as the thing that was working for us. We really saw it for the first time, actually, during our Dota 2 project. We started out with 16 cores to train a little agent on… Jakub and Szymon, who were leading the project from an ML perspective, on their desktop. Then they scaled to 32 cores. It felt like every week I'd come back to the office and they'd scaled it by another 2x, and we had 2x performance. It was just so clear, you just need to keep going. Where does this thing peter out? It just never did. John Collison: Founders get too much credit, because you have an initial product that's a pretty reasonable idea. Then you listen to the customers, and you follow what's working. So you were saying that was kind of OpenAI with the scaling hypothesis, where you started trying to make Dota AIs work, and you noticed that adding more compute worked really well, and you said, "Where else will just throwing more compute at it yield benefits"? Greg Brockman: I think that's to the first order correct. One thing that distinguishes OpenAI from the typical startup is we did everything in reverse. It's like, you're supposed to have a problem to solve. No one cares about the technology. John Collison: Form the entity up front. Greg Brockman: Exactly. Yes. And for us, we really chased the technology without any idea of how it would be applied. A lot of pursuing the technology really is, you have to let reality hit you cold, hard in the face. There's just no other way to achieve results. You can't will it into existence. You can't convince people that this is the thing. You have to actually make the system work. We have to just figure out what is the right frontier, what are the problems, what are the things that are on the edge of working, and to really double down on those. John Collison: What else did you take away from the Dota work? How else did it… Because you could, say, have just started with LLMs, and we could have skipped that period in the wilderness. But it sounds like it was somewhat formative for the OpenAI organization. Greg Brockman: I think Dota had many lessons, one of which actually was a management lesson for me. I remember when we started out the project, I tried to set a list of milestones. It's like, "Okay. This date, we're going to beat this player. This date, we're going to beat this player." John Collison: That didn’t work? Greg Brockman: It did not work at all. I remember our first milestone came and… John Collison: Went. Greg Brockman: Exactly. And so you realize that you cannot control the outcome, right? You cannot set outcome-based milestones. What you can do is you can control the inputs of, "We're going to try these experiments by this date. We're going to implement this feature by this date." That is what actually worked. I remember, it was one of those things that was just like, a story that I could not have written any better if we'd intended to. We beat our in-house best player. Then we were playing against a semi-pro, and he was just trouncing us, trouncing us. Then, suddenly we're starting to get pretty good. So we showed up at the international tournament blind. First day, we had three players that we played against. We went 3-0, 3-0, and then 2-1. We're like, "Oh no. We lost. What happened?" It turned out that this pro that we're playing against, he had used an item we'd never trained against. We were like, "Oh no. We're totally going to be hosed." So what did we do? Well, we just need to change the training. And so people stayed up all night to get this done. They added this extra item in there. 4 am, they finally get the job running. That Wednesday, we're supposed to play against the number two and the number one player in the world. And our semi-pro plays against it, and he's like, "This bot is totally broken," and we're like, "Oh no. We clearly had a bug. Something terrible has happened." He was like, "Look, it's taking all this damage it doesn't need to. I'm going to go kill it." He goes in to kill it, he loses. He was like, "That was weird." He realized that what had happened was it had learned a baiting strategy. And then we realized, "Well, we have a super-bot, but it's so bad at the beginning because it's trying to do the baiting. So what if we just stitch the two bots we have together?" And then that bot was just undefeatable, and we played against this number one player and won. To me, this is the story of how deep learning works, right? It's like, you can't control where you're going to go. You can control everything that goes in. You can put these metrics in, these measurements, and you can have the evaluations, and being able to gauge where you're at is almost as important as being able to make the forward progress. But if you get all those elements right, then you can do true magic. John Collison: You're also describing something that works really well for an organization, where it was motivating to stay up all night. If the prize was impossibly far away, it wouldn't have been as motivating. But the fact that there was a near-term reward function, and you were able to show concrete progress… Greg Brockman: I think so. I mean, I think some of my favorite engineering stories have the same character. I remember you and I staying up all night to get our ISO 8583 integration done. John Collison: There's something about staying up all night for critical projects that actually have important history in all startups. I'm glad to hear the tradition is alive and well at OpenAI. So, Dota has fallen. Chess has fallen. Go has fallen. We've passed the Turing test, I think by anyone's measure. People comment on how there was little fanfare when we did, but we seem to have pretty clearly done so. What's a good new Turing test? Greg Brockman: Well, I'll tell you two things. One is that if you look at the strict version of the Turing test, I would actually claim we haven't done it yet. No one's really gone that extra mile to say, "Can we actually have an AI that is fully indistinguishable from a human?" It's not clear if it's even a good task, right? But I think that the right question, to your point is, well, what is the milestone that we should be chasing in terms of capability? I remember talking to one of our board members in 2018, and he said, "Look, we're all excited about near-term AGI, but it just doesn't feel like it's on track." I asked, "Well, what do you mean?" He said, "In a world with near-term AGI, you would expect massive economic value to be delivered by AI already, and where is it?" In 2018, I think that was a very fair criticism. Clearly, that's starting to change now. John Collison: It feels like one thing that may really change the AI market is personalization. Up to quite recently, when you asked ChatGPT a question, it was like walking into a shop off the street. They've never met you before, they know nothing about you, whatever. That's obviously not ideal for this close part of your digital life. I'm curious how you're thinking about personalization from a product point of view. Because it feels to me like the most meaningful change since the chat interface two and a half years ago. Greg Brockman: I mean, I think it's absolutely critical, and I think it is very rightly considered to be a next frontier. I am someone who always, when I Google something, I go into Incognito Mode, because I don't even want my computer to remember that history. I always used to go for temporary chats on ChatGPT. But now my usage has totally reversed. I want ChatGPT to remember everything. I want it to remember all of my interactions, because it's useful. John Collison: Okay. So you guys figured out, from a product point of view, how to make the memory actually work better. Greg Brockman: It’s the product point of view, but also, really, the research point of view. John Collison: And I presume there's a flip-flop between the product and research where, when you find something that's useful from a product point of view, then the product people say, "I'm just a product person, you researchers go actually make this good." And then that kicks off more research. Is that how it works? Greg Brockman: To some extent, that's a failure mode in our mind. I think that we really don't want to have that kind of silo. We really want to blur the lines and have people cross-collaborate. And so it's very different mindsets from how you would traditionally build a product versus how you do research. Part of what had happened, actually, was that we had GPT-3, we knew we needed to build a product in order to be able to continue to raise funding, and we were like, "Well, what product do we build?" We wrote down a list of, like, a hundred different products. We could do a medical thing. Then you're like, "Okay, well, now we have to sell to hospitals, we're going to have to hire doctors." You realize you give up on the G in AGI, right? You're going to go for a specific thing. Someone had the idea of saying, "Well, why don't we just make an API and let people figure it out?" And, again, this is totally backwards from how you're supposed to do a startup, right? You're supposed to have a problem, and we had no idea what the problem was. And so— John Collison: Yeah, yeah, yeah. We're going to back into the problem. Greg Brockman: This actually felt like this was probably the hardest project that I've ever done, because it felt totally doomed, right? It's like, I know every instinct, every builder instinct of mine says— John Collison: Did it actually feel doomed, or was it— Greg Brockman: Oh, it felt totally doomed. John Collison: It wasn't just open-ended or something? Greg Brockman: No, it felt doomed. John Collison: But you were still doing it. Greg Brockman: Yeah. I mean, it's like, at some point, if you have… There was definitely no other path. There was no other path. It was the only shot we had. I remember someone also saying, "I can't imagine anyone paying for samples from this model," and I was like, "You might be right." John Collison: I'm still trying to imagine it myself. Greg Brockman: Yes. It was just not clear, were we above threshold or below threshold? We showed it to people, and people were interested. But that’s very different from people being like, "I will build my company on top of this." John Collison: So what was the first use case to get any traction? Greg Brockman: AI Dungeon. John Collison: What was that again? Greg Brockman: There you go. AI Dungeon was a text-based adventure game. John Collison: Oh, sure, yeah. Okay. But that was real revenue, that was non-zero revenue? Greg Brockman: It was enough. Yes. And, in fact, I believe they were our first paying user. John Collison: And that gets you confused, where you're like, "Ah, clearly the future of OpenAI is gaming," you know? Greg Brockman: I know, right? Well, it’s back to our roots. John Collison: Exactly. Greg Brockman: It's interesting, too, because we had dreamed of all of these applications, like medicine and all these things, and you start with the gaming application. But we could see signs of life on so many other things. I think in many ways, GPT-3 was the world's best demo machine, right? When we released the API, people were coming up with all these cool things you could do, but making them reliable was so hard. It really wasn't until the next generation of GPT-4, until we started to figure out how to do post-training well, that then you were actually able to build real businesses on top of these things. John Collison: Bill Gates was saying recently that GPT-4 was the best demo he'd ever seen since Xerox PARC. You know this quote? Greg Brockman: Yes. He said it to me the night that he saw it. John Collison: That's high praise. I want to touch the medicine thing because you mentioned it. Like you said, your family has personal stories, you've talked about getting very valuable diagnostic help. We, ourselves, actually, it's much more minor in our family, but we managed to fix a cat thanks to debugging it with an LLM. I think that that’s an interesting example, because so many people that I know have had some kind of experience like this, and maybe it's because you actually don't get that much time from a doctor. Are there other examples like this medicine application where you're seeing a lot of success, that many people have similar stories, but we just hear less about? Greg Brockman: Yeah. I think it's a great question. By the way, medicine is an example of one where I kind of thought it was going to be one of the last domains that we'd successfully be able to add value in, but it turns out that the bar is so low, you just need to exceed WebMD. I think that we have seen other areas that are a real common theme. One that's very interesting right now is the life coach, life advice kind of application, where you just talk to your AI. John Collison: That's actually really taking off? Greg Brockman: Sometimes way too good. That's right. I think that there are things like… Education is another area that just clearly is really having an impact. There are studies coming out now that actually show that people are able to learn better through the use of these tools. John Collison: That's to be expected, right? It is the Bloom 2 sigma effect in a product. Greg Brockman: Yes. That, for example, is why Sal Kahn started Khan Academy, to think about if you can give personalized tutoring to everyone. We showed him GPT-4, and he's like, "This is the thing. We need to become a GPT-4 app." I think that there are these really amazing applications that are affecting everyone's daily lives. Obviously, programming is another one. People are seeing it all across the board in professional contexts.. So I think that we're heading to a world where, just like if you want to do productive work and you don't have access to a computer, you're going to be hampered, similarly, not having access to AI, it's heading in the same direction. John Collison: Speaking of not having access to AI, I will posit that these days, it feels like AI product development is mostly OS-limited. Is that how you feel? Are we stuck at the moment? Greg Brockman: I do feel a little of the stuckage. But not to worry, it is overcomeable. But yeah, I think it is true. Two years ago, we released plug-ins in ChatGPT. Do you remember those? John Collison: Yeah. Greg Brockman: That was trying to make it so anyone could write apps that then ChatGPT could access. The models were just not that good, right? Then we limited it to three plug-ins at a time, you could have only so many functions and stuff, and it just wasn't that reliable. Now we're in a world where MCP is really taking off as a way to hook up your AI to different tools, and very much trying to take that same type of idea and really make it work. Now the world that we're in is very similar, where there are certain interfaces we don't have—being able to access your phone and all those APIs. And there's a question of, is the model above threshold to actually use them or not? My observation has been that basically, I think that there is maybe a lag of six months of different interfaces that are hard to access, but once we have a model that's good enough, we will find a way. People will find a way. I think that we're in a world where I have every expectation that we will get the future that has been promised. It’s just going to take some work. John Collison: I feel like there are many moments where I'm using my phone, and I want a single button where it’s just like, "ChatGPT, what do you think of this? I need your comment. I need your fact check. I need your explanation." Something like that. You take a screenshot and you go into ChatGPT, and you click “upload photo.” It feels very 1993 versus the button on my phone that just says, "Hey, ChatGPT, what do you think about this?" Obviously, you guys are not empowered to go build that. That's what I mean by it feels somehow like we're a little operating system-limited. Greg Brockman: I definitely get it. But I'll say, I think that there are two dimensions. This is how I've been thinking about things since we released the API back in 2020. There's capability and convenience. What you're referring to is the convenience, right? It's pretty inconvenient to do the screenshot and paste it. But the thing is, if the capability is good enough, you are willing to accept any sort of inconvenience. It's like, if by taking the screenshot and showing it to ChatGPT, it could give you amazing insight, it could tell you how to build Stripe in some way, and it takes you like a month to do it, you have to crawl to the top of some mountain, you'll do it, right? The convenience will not stop you. So the point that I'm trying to make is that if the capability is high enough, people will start doing a specific flow. John Collison: They'll discover the use cases, and the convenience will just catch up. Greg Brockman: And on the convenience, there's so much pressure. There's pressure on the phone manufacturer. There's pressure on us. There's pressure on everyone in order to bring down the— John Collison: So I just need to be patient, and it'll be great in three years time. Greg Brockman: Yes. And really lean it in and use the AI. John Collison: A criticism people like to levy of AI is, “Yeah, it's great and handy and all, but it hasn't come up with a single novel advance in mathematics or science.” Greg Brockman: Have you? John Collison: Well, you could have if you'd become a mathematician. But humanity has, if we're keeping the score. What do you make of that criticism? Greg Brockman: Just wait. John Collison: Okay. So you think, like, take one of the Millennium Prizes. Do you think we plausibly will see that? Greg Brockman: I think for sure. I mean, there's no question. John Collison: Two years, five years, 10 years? Greg Brockman: I think that is the question. It's just timing. John Collison: That is my question. Greg Brockman: I mean, I would put two to five years as the right number. I think ultimately this comes back to the question of benchmarks. Actually, being able to solve the Millennium Problem is a pretty high bar. John Collison: Yeah. Greg Brockman: Pretty high bar. Once you can do that, there's so many other things that will definitely be possible. I think that we're starting to see the leading edges of this. And to me, if we look at our definition of AGI… We recently started talking about this framework of thinking about levels of AGI, starting from chatbots, to reasoners, to agents, to innovators, to organizations. Five levels. We're basically somewhere in level three right now. Level four, this innovator, that's going to be different. I recently posted some pictures of our visit to Abilene, Texas, where we're building these big data centers together with our partner, Oracle. Imagine taking that whole data center and just thinking hard about one problem. Imagine it just thinking about how to solve a Millennium Problem or how to cure a specific kind of cancer. Maybe it needs access to some apparatus. Maybe it needs access to robotic wet labs. Maybe it needs access to different tools in the world. But that level of computational power coupled with the ability to experiment and learn from your ideas, that is going to be something the world has never seen. John Collison: So yet again, we just haven't put a respectable amount of compute on these problems compared to what we will be doing. Greg Brockman: Yeah, we're still on these tiny little computers. John Collison: So that actually gets to, in terms of these scaling laws, do they eventually run out because we just run out of compute? Or do we eventually get to the point where we're inventing new kinds of nuclear energy, and that is what unlocks the next level? A lot of energy that comes online now is for data centers, which was not true when you guys started training GPT-2. Isn't that the upcoming bottleneck? Greg Brockman: I mean, as it should be, right? It really should be that it's energy manufactured into intelligence, and that's your only bottleneck. John Collison: But I'm saying that'll be quite a plateau compared to the exponential growth we've seen over the past few years. Greg Brockman: Well, I think… John Collison: Unless things really change in terms of permitting and plans for building and everything like that. Greg Brockman: Yes. This is, I think, the core, right? If you look at every trend in this field, there are these exponentials, these S-curves that sum up to exponentials. John Collison: Sure. But these exponentials were mostly existing in tech, Silicon Valley space, where it was pretty easy to have exponential growth. It's pretty hard in permitting and real estate and damming rivers and building nuclear power plants. It's harder to have exponential growth there. Greg Brockman: Well, let's see how fusion pans out. John Collison: Yeah, okay. But even fusion, most industry observers would say it's still five years away. And so where does the next five years of power growth come from? Greg Brockman: I think that it is very possible that we end up bottlenecked on energy, and that's actually one reason that we've been spending a lot of time really trying to advocate for the fact that we just need far more power. My observation of the market is that ultimately, the capitalist markets do provide. I think there's this absolute tsunami of demand that is coming our way, but I feel some confidence that, again, when there's enough pressure, when there's enough clarity of, this is the bottleneck… And it's not just for any company, right? It's really for national competitiveness. You look at other countries that are building huge amounts of power, far more than we are. I think that actually, for America to remain competitive, there's just no choice but to build it. John Collison: We’ve got to figure out power. Greg Brockman: We do. John Collison: Speaking of bottlenecks, everyone was talking about the data wall in 2023. I think this is an interesting thing, where no one is talking about the data wall anymore, and yet it doesn't feel like AI progress has slowed down. What happened there? But is it just test-time compute? Is it, like, people were wrong about the data wall and where it presented a bottleneck? Is there actually still a data wall, but it's two years away? Greg Brockman: It's basically all of these things. It truly is. It's like, you keep changing the paradigm. That is the real core of the Kurzweil view of the world, is that, fine, this one way of doing things taps out, and if you just look at that one way of doing things, you feel hopeless. You feel like this is it. But somehow, you will find a new S-curve. I think that's what's happened. For example, synthetic data. For example, reinforcement learning. If you think about the RL paradigm. Fundamentally, that's a data production mechanism. It's just that the AI happens to be training on its own data, and then you learn on it very rapidly, and then you learn on that. Each of these has taken us much further. I think there are lots of algorithmic ideas, lots of techniques, lots of ways of even using the existing data better. I think that fundamentally, the S-curves continue, and if you zoom out, it all looks smooth and uninterrupted. John Collison: It's kind of like chip miniaturization, where each generation, people are like, "Okay, well, that's the smallest you can possibly make a chip." Do you know what I mean? Greg Brockman: That’s it. Yes. John Collison: "We're done with miniaturization." And somehow, we figure out a way. Greg Brockman: Now, one difference with chips is I think at the end of the day, there is some limit. John Collison: Right. But we've never been that close to that. Greg Brockman: Right. Yes. John Collison: Where does AI coding go? In particular, vibe coding is all the rage right now. It's kind of the term of 2025. It's sort of working. It's very impressive. No one is really fully letting AI software engineers run end-to-end in production. I'm just curious, what are your one- to two-year predictions on what happens with AI coding? Greg Brockman: Well, my general observation is that once something kind of works in this field, the next gen is going to be great. I think that's where we are right now for AI coding. I think what we're going to see is AIs taking more and more of the drudgery, more of this pain, more of the parts that are not very fun for humans. Now, one thing that's very interesting is that I think that so far, the vibe coding has actually taken a lot of code that is actually quite fun and left behind the review and the deployment, these things that are not fun at all. I'm hopeful that we're actually going to be able to make a lot of progress on these other areas as well. But fundamentally, we should really end up with a full AI coworker. I think it really will be anything you want to create, you can be the manager, and you can have this team of software engineering agents. Now, the thing that I think will be very interesting to see is, is there a world where the AI becomes the manager, and it gives you ideas, and gives you some tasks to do? That's something that, again, is just totally backwards in terms of how we think about it. But are there ways in which you can actually have outcomes for companies and actually have people whose jobs become much more meaningful because they have an AI who really deeply understands them in the same way that your AI doctor really deeply understands all of your needs? John Collison: But isn't part of the common thread that we're talking about here, often places where AI tools underperform, it's because they're trying to do something generally. Like, voice recognition is not that good because it's trying to recognize all voices as opposed to trying to recognize my voice in particular. Similarly, with AI coding, they work well in places where you need no context at all and we're single-shotting an app based on publicly available libraries. In places where you have to understand a million-line code base, they haven't fully figured out how to do a good job of that. Is that a fair parallel to draw between all these challenges? Greg Brockman: Well, I think there are two things in there. One is that I think this is already changing, right? So if you look at something like Codex, it's actually great at operating in a big codebase. I ask it for where functionality is implemented, and it's better than I am at finding it. Which is kind of a wild fact. It's super cool to see it grepping around and just going and exploring. Actually, this is one thing that we really shot for with Codex, was to build a tool for software engineers who are not necessarily vibe coding. It's not about building a new app from scratch, which is a cool demo, but that's not actually how most software gets written. Actually, I think maybe the killer enterprise feature is refactors. It's like, rewriting your COBOL app or changing your— John Collison: When Facebook did HipHop to do static PHP. Greg Brockman: Exactly. If you think about it, the amount of deep, sophisticated thought that is required to accomplish a refactor is actually not that high. There's a lot of mechanical work, and it’s just the sheer volume of it that's hard. That's an AI-shaped problem for sure. I think we're going to see a lot more productivity on all sorts of AI tasks as a result. Now, we are in a world where… You said a second thing, which is maybe you need to narrow down more. I think that the way these models work is you actually do want one model that knows more and more things, and you want it to have some personalization to you, but the fact that they have this one base model that knows everything is actually a very useful starting point. So I do think that you're going to see a world where we'll have more and more capable base models, and figuring out how you really connect it to all of your organization's code, and context, and history. John Collison: How does OpenAI decide what products to do? I'm just curious how you think about when to develop specific products or when you think, "Oh, you can trivially do that with ChatGPT, and that's good enough." Greg Brockman: Yeah. It's a really tough question. It's something we really struggle with, and I think that over time, when we… I guess, actually, when we first launched ChatGPT, we were left with this, "Well, we're an enterprise business and we're a consumer business," and that seems terrifying as a startup. I remember talking to one of my board members who said, "It just feels like what you have is an unfocused strategy at first, because you're just doing all these different things. But if you think about it, maybe an analogy is to a company like Disney, where you make one core asset, like The Little Mermaid. Then you productize it in all these different ways.” Think about The Little Mermaid the ride, the lunchbox, the T-shirt. I think that we have some element of that. We have a core model, and then we have a question of, "Well, what are the applications that this can add a lot of value to quickly, with a small amount of additional work?" So I think the question of what areas to go into are how far does it take us off the general path, the return on how important is this domain area, especially for achieving the bigger goal, how much synergy is there across it with respect to other things we work on? Coding is one where there's very clear synergy, very clear ROI, because if we can speed ourselves up, that's something that accelerates everything. John Collison: How is being from North Dakota? How has it shaped you? Greg Brockman: North Dakota was an amazing place to grow up. John Collison: Was it actually? Come on. Greg Brockman: It actually— John Collison: I've been there. Greg Brockman: And it was great. It was incredibly safe. Our doors didn't even have working locks. It was that kind of place. I had a lot of freedom academically. Sixth grade, my dad taught me some algebra. Seventh grade was the first time they split you into advanced math, so I was going to be taking pre-algebra. My mom took me to go see the teacher and we asked, "Can he skip?" The teacher looked at us very condescendingly and said, "Every parent believes that their child is special. I can guarantee your son will be plenty challenged in my class." After a month of me sitting in the back just playing games on my calculator, and she'd call on me randomly to try to trip me up, and I'd just look at the board and be like, "2x,” she said, "Okay, fair enough, your son has nothing to learn in this class." So they moved me into eighth-grade algebra. But then eighth grade rolled around and I had no more math left in my middle school. John Collison: That's right. You went to college, right? Greg Brockman: I did, in high school, start going to University of North Dakota, and took a bunch of classes there. But also, I was connected to a lot of people who were the top math kids in the country through things like math camp and the math competitions. John Collison: So you're saying the social scene was not too distracting in North Dakota? Greg Brockman: Not too distracting. But it was definitely fun. John Collison: Last question. Do you remember, we were going to Camp YC in 2017, and I asked you how far AGI was away, and you said, “Two or three years”? Greg Brockman: Did I say that? John Collison: You did. Greg Brockman: I don't see the recording. John Collison: I'm just trying to think, was I right? Were you right? How should we grade that? Because we didn't get AGI, but we didn't not get AGI either. I'm curious if you have any reflections from your own AGI prediction journey. Greg Brockman: I think that we are… I will say, I think that AI is surprising. I think that that is the single most consistent theme. The thing we were picturing, we got something different. But we got something better, more magical, something that is more helpful. I'm actually quite happy with that. Now, predicting where you go, again, it's really hard to manage the outputs here. One goal of OpenAI that we have successfully achieved is every year to have at least one result that just feels like a step function better than anything before. I think as long as— John Collison: You know it when you see it, kind of? You just want to have one really awesome AI-feeling thing each year? Greg Brockman: That kind of thing. John Collison: I like that. That's a good way to tie it back, which is the way we grade that prediction is that you've stopped setting metrics based on outputs. Greg Brockman: Yeah, exactly. But it does feel we're getting really close to something pretty magical. John Collison: I agree. Thank you. Greg Brockman: Thank you.