Latest / The Pragmatic Engineer / Building Codex with Tibo Sottiaux

Transcript

Building Codex with Tibo Sottiaux · The Pragmatic Engineer

0:00
SPEAKER_00: Codex is one of the most popular AI coding harnesses today, but how did it all start?
0:05
SPEAKER_00: Many of you will know today's guest,
0:06
SPEAKER_00: Thibault, from his generous and pretty frequent Codex usage resets.
0:10
SPEAKER_00: He was also there when Codex as a product started and has led the broader Codex team since.
0:14
SPEAKER_00: Today we cover how Codex started and why it was built in Rust and made open source.
0:19
SPEAKER_00: Coderies are changing inside the Codex team and OpenAI.
0:23
SPEAKER_00: What it means when maintenance and re-architecting are getting ridiculously cheap.
0:26
SPEAKER_00: What the merge of Codex into ChatGPT looked like and
0:29
SPEAKER_00: the many underappreciated engineering challenges of this project.
0:32
SPEAKER_00: If you want to understand how teams inside of OpenAI plan,
0:35
SPEAKER_00: review and ship software, this episode is for you.
0:38
SPEAKER_00: This episode is presented by TurboBuffer,
0:40
SPEAKER_00: a ridiculously scalable,
0:42
SPEAKER_00: fast and cheap hybrid search engine built on top of object storage by
0:45
SPEAKER_00: an engineering team that I've really gotten to like after spending time with them.
0:48
SPEAKER_00: TurboBuffer is the tool that companies like Entropic, Notion,
0:52
SPEAKER_00: Cognition,
0:52
SPEAKER_00: and Harvey all use to connect their AI products to massive amounts
0:55
SPEAKER_00: of unstructured data.
0:57
SPEAKER_00: When I've talked with engineers who use TurboBuffer,
0:59
SPEAKER_00: the theme that always comes up is reliability and performance at scale.
1:03
SPEAKER_00: The reasons for this have everything to do with TurboPuffer's architecture.
1:07
SPEAKER_00: TurboPuffer uses only object storage for state and NVMe SSEs with memory cache for compute.
1:13
SPEAKER_00: Data in TurboPuffer is organized into namespaces.
1:16
SPEAKER_00: You can think of a namespace as a database table or a search index or
1:20
SPEAKER_00: an S3 prefix depending on the world you come from.
1:23
SPEAKER_00: When a namespace is not being queried,
1:25
SPEAKER_00: it stays on cheap object storage with no associated compute cost.
1:29
SPEAKER_00: When a namespace is active,
1:30
SPEAKER_00: TurboPuffer pulls it up into hot catching tiers, so queries are very fast.
1:35
SPEAKER_00: This design fundamentally makes it effortless to scale to hundreds of millions of namespaces.
1:40
SPEAKER_00: If you're building a multi-tenant AI product,
1:42
SPEAKER_00: every user and their agent can have their own dedicated search index
1:45
SPEAKER_00: without any overhead.
1:47
SPEAKER_00: And each namespace can hold hundreds of millions of documents
1:50
SPEAKER_00: without any special configuration.
1:52
SPEAKER_00: You can scale TurboPuffer virtually without limit,
1:54
SPEAKER_00: and the performance, reliability, and operating model all stay the same.
1:58
SPEAKER_00: If you need to connect AI to lots of data, TurboPuffer should be your first choice.
2:02
SPEAKER_00: Check it out at turbopuffer.com slash pragmatic.
2:06
SPEAKER_00: Thibaut, welcome to the podcast.
2:07
SPEAKER_00: So good to have you here.
2:08
SPEAKER_00: Thank you for having me.
2:09
SPEAKER_00: It's so good to see you again.
2:11
SPEAKER_00: It's good to do this last time we did it in person.
2:13
SPEAKER_00: Now we're doing our video.
2:15
SPEAKER_00: First, I wanted to...
2:16
SPEAKER_00: Ask you, how did you get into tech?
2:19
SPEAKER_00: When did you first know that you want to work with computers?
2:23
SPEAKER_01: That's a good question.
2:24
SPEAKER_01: It was a long, long time ago.
2:29
SPEAKER_01: My parents actually decided to move out of Brussels,
2:33
SPEAKER_01: where I was born,
2:34
SPEAKER_01: and just thought it was great to just buy a small house and refurbish it,
2:41
SPEAKER_01: but it was in the middle of a village with not much going on.
2:46
SPEAKER_01: I think there was like roughly 200 people living there.
2:50
SPEAKER_01: Not many that I felt like I wanted to talk to or, you know, could make friends with.
2:55
SPEAKER_01: And so I kind of got stuck.
2:57
SPEAKER_01: This is like very early, like eight, eight years old.
3:00
SPEAKER_01: I kind of got stuck as like,
3:01
SPEAKER_01: you know,
3:01
SPEAKER_01: computers and, you know, it's like early days of like the for me, the Internet.
3:05
SPEAKER_01: And, you know, that was my way to learn about things.
3:08
SPEAKER_01: And so just the rest is just like, you know, came from that.
3:12
SPEAKER_01: So like I owe it to my parents to,
3:14
SPEAKER_01: you know,
3:14
SPEAKER_01: have moved into the middle of nowhere and then,
3:17
SPEAKER_01: you know, I had no choice but to get interested in computers.
3:19
SPEAKER_00: Once you you finished high school, like you went on and went to university, right?
3:23
SPEAKER_00: Actually studying it properly.
3:25
SPEAKER_01: Yes, I studied mathematics, applied mathematics at university.
3:30
SPEAKER_01: I went there quite, quite early.
3:33
SPEAKER_01: And so I graduated earlier as well.
3:36
SPEAKER_01: Like I thought for a long time that I would actually not make it.
3:40
SPEAKER_01: And then I would drop out.
3:42
SPEAKER_01: I had small companies and small consulting business while I was studying.
3:50
SPEAKER_01: I was working for banks.
3:52
SPEAKER_01: I was very interested in supply chain and applied mathematics problems.
3:57
SPEAKER_01: And I was selling that and learning a lot through that.
4:00
SPEAKER_01: Eventually ended up in the startup world in Belgium.
4:04
SPEAKER_01: I did that for a little while and then moved to London to work initially at Google
4:09
SPEAKER_01: and then DeepMind and then now moved to be here at OpenAI.
4:13
SPEAKER_01: This is California.
4:14
SPEAKER_01: I love the California weather.
4:15
SPEAKER_01: We can talk about that.
4:17
SPEAKER_00: It's been very good.
4:18
SPEAKER_00: Right after university, you founded a startup, right?
4:22
SPEAKER_00: You had the startup bug in you or the entrepreneurial bug.
4:26
SPEAKER_01: Yeah, so this startup was all about pharmaceutical supply chain,
4:32
SPEAKER_01: looking at the supply chain for clinical trials and like try to optimize
4:36
SPEAKER_01: and decide like, hey, you know, should you produce more medicine?
4:39
SPEAKER_01: Where should you send it?
4:40
SPEAKER_01: Where should dispatch it?
4:40
SPEAKER_01: How do you avoid waste?
4:42
SPEAKER_01: And through that, making clinical trials more efficient.
4:46
SPEAKER_01: And this was using traditional non-ML techniques,
4:51
SPEAKER_01: more optimization, solving, Monte Carlo simulations, these kinds of things.
4:56
SPEAKER_01: Stochastic multi-stage optimization problem, really.
4:59
SPEAKER_01: And we also applied it on steel industry,
5:02
SPEAKER_01: and we applied it to electrical grid as well in Europe.
5:05
SPEAKER_01: It was like anything that sort of had the shape of like an optimization problem,
5:09
SPEAKER_01: we would sort of like get interested in.
5:10
SPEAKER_01: And, you know, to this day, like this, this, this company still exists.
5:13
SPEAKER_01: And I think they do some of the most interesting work still,
5:16
SPEAKER_01: but it's changing a lot, you know, with modern AI, for sure.
5:20
SPEAKER_00: But it's interesting because you kind of said like, oh, yeah, that wasn't ML.
5:23
SPEAKER_00: It was just a traditional stuff.
5:24
SPEAKER_00: And then you go into like Monte Carlo simulation and optimization and this algorithm.
5:28
SPEAKER_00: I get a sense that you kind of just went deep, right?
5:31
SPEAKER_00: It was like, OK, like, here's a problem space.
5:33
SPEAKER_00: Like, how can I use mathematics,
5:35
SPEAKER_00: stuff that I learned, stuff that I didn't learn to just go deeper and deeper?
5:38
SPEAKER_00: Do I sense that correctly?
5:40
SPEAKER_01: Yeah, that's why I was obsessed with applied mathematics is just really this idea
5:44
SPEAKER_01: of you have theoretical mathematics or you have theoretical science and physics
5:48
SPEAKER_01: and like there you just you do it because.
5:52
SPEAKER_01: There's something to be discovered and something beautiful about it.
5:55
SPEAKER_01: It's all about patterns and pushing the frontier,
5:57
SPEAKER_01: but you don't necessarily always know how you're going to apply it.
6:00
SPEAKER_01: And then there's the real world, right?
6:03
SPEAKER_01: There's all these cool problems that just lie around.
6:05
SPEAKER_01: And I was very interested in seeing how can I make the world better?
6:09
SPEAKER_01: And so how do I apply sophisticated mathematics to just optimize the world around me?
6:15
SPEAKER_01: And that was a lot of the thesis behind that startup.
6:18
SPEAKER_00: Yeah.
6:18
SPEAKER_00: And then after a startup, you ended up at Google and first at Google London.
6:22
SPEAKER_00: It was in 2015.
6:23
SPEAKER_00: And I remember 2015,
6:24
SPEAKER_00: Google was a really,
6:25
SPEAKER_00: really competitive place to get into,
6:26
SPEAKER_00: like maybe as competitive as OpenAI is today in terms of the industry or in terms
6:30
SPEAKER_00: of prestige.
6:32
SPEAKER_00: You worked on Maps initially, and then you moved over to DeepMind.
6:35
SPEAKER_00: Can you talk a little bit about what you worked on and then why did you move on from
6:40
SPEAKER_00: an already really interesting space that you clearly loved,
6:42
SPEAKER_00: you know, like the optimization, logistics and all these things?
6:45
SPEAKER_01: Yes, I didn't start on Google Maps.
6:48
SPEAKER_01: I started on a project that was meant to make the web faster and meant
6:55
SPEAKER_01: to make websites faster, especially on mobile.
6:59
SPEAKER_01: At the time,
6:59
SPEAKER_01: Google So like seeing the transition from desktop to mobile and like more
7:05
SPEAKER_01: and more traffic going to like, you know, mobile phones.
7:07
SPEAKER_01: And so like wanted to get ahead of that.
7:09
SPEAKER_01: So funded like a number of initiatives and projects.
7:13
SPEAKER_01: I was working on one of them.
7:14
SPEAKER_01: This was like really,
7:16
SPEAKER_01: really fun because it was a small group within actually the ads organization.
7:21
SPEAKER_01: It was meant to sort of like,
7:22
SPEAKER_01: you know,
7:22
SPEAKER_01: offset the loss for the ad revenue loss because of this shift of traffic to mobile.
7:28
SPEAKER_01: And worked on it for roughly two years.
7:31
SPEAKER_01: And then it was canceled.
7:33
SPEAKER_01: And although it was like the most fun I've had on,
7:35
SPEAKER_01: you know,
7:36
SPEAKER_01: solving hard technical challenges,
7:38
SPEAKER_01: I learned a lot from not having product market fit,
7:40
SPEAKER_01: not having the right users, not having the right feedback.
7:45
SPEAKER_01: And then, you know, one day, it's just like this VP flew in from California.
7:52
SPEAKER_01: And then it was just like, oh, yeah, it's like, you know, we're canceling this project.
7:56
SPEAKER_01: You know, unfortunately, you only have, you know, hundreds of users.
7:59
SPEAKER_01: And this is clearly not Google scale.
8:01
SPEAKER_01: And then it's unbelievable.
8:03
SPEAKER_01: But people were surprised.
8:05
SPEAKER_01: So there's a lesson there that I carry with me,
8:09
SPEAKER_01: of course,
8:10
SPEAKER_01: is just always question,
8:11
SPEAKER_01: always go to,
8:13
SPEAKER_01: always deeply think about the impact that you're having,
8:16
SPEAKER_01: but also the importance of the overall project that you're contributing.
8:20
SPEAKER_01: And then I moved into Google Maps.
8:22
SPEAKER_01: Google Maps was super fun, worked on reviews.
8:24
SPEAKER_01: And then after roughly a year, I couldn't ignore DeepMind.
8:27
SPEAKER_01: It was just...
8:28
SPEAKER_01: It was this special place headquartered in London.
8:32
SPEAKER_01: So many great things were happening.
8:34
SPEAKER_01: This was like really the early days, you know, with rumblings of things like AlphaGo.
8:39
SPEAKER_01: And they just seemed to be doing extraordinary things and,
8:42
SPEAKER_01: you know, just really tackling the very, very hardest problems that you can tackle.
8:45
SPEAKER_01: And like with my background, I was obviously drawn to that.
8:48
SPEAKER_01: I started there.
8:49
SPEAKER_01: It's like I worked on a lot of the research infrastructure, research tooling.
8:53
SPEAKER_01: This is a theme that I carried on for almost a decade.
8:56
SPEAKER_01: And it's like this is very much also
8:57
SPEAKER_01: the like how I approach things is how can I build tooling and products
9:02
SPEAKER_01: that help make others more efficient and bring a lot of utility to them.
9:06
SPEAKER_01: Initially, I was doing this for research.
9:08
SPEAKER_01: And then like over time,
9:10
SPEAKER_01: you know,
9:11
SPEAKER_01: I got like into thinking about things in a much more like more general and general
9:15
SPEAKER_01: and general way, you know, eventually like, you know, ending up where I am now.
9:19
SPEAKER_00: And a fun story that you recently shared on X as well is how you were part of
9:23
SPEAKER_00: the team that built this internal Googlebot that was,
9:27
SPEAKER_00: if you want to say, similar to ChatGPT, but a year before ChatGPT.
9:33
SPEAKER_00: Can you talk about that?
9:35
SPEAKER_00: That is a new story.
9:36
SPEAKER_00: I haven't heard it before.
9:37
SPEAKER_01: This was part of DeepMind.
9:38
SPEAKER_01: There were multiple efforts as well.
9:39
SPEAKER_01: There was Brain as well that was separate at the time.
9:42
SPEAKER_01: They had their own efforts on large language models,
9:44
SPEAKER_01: but it was definitely something that was being explored.
9:47
SPEAKER_01: It was not the main thrust of DeepMind.
9:49
SPEAKER_01: DeepMind was very much worried and busy thinking about brand challenges and games
9:55
SPEAKER_01: and thinking about RL.
9:58
SPEAKER_01: Not in the language sense.
10:00
SPEAKER_01: And so there was this group that was pushing on large language models and thinking
10:06
SPEAKER_01: about, what if large text corpuses are everything?
10:10
SPEAKER_01: What if you just pushed language to its maximum and you just scaled language models?
10:16
SPEAKER_01: Would that be enough to get to general intelligence?
10:19
SPEAKER_01: That was the hot debate at the time.
10:21
SPEAKER_01: And then one group decided to Just really push on that.
10:25
SPEAKER_01: And then it felt really natural.
10:27
SPEAKER_01: As I was building tooling with others for research,
10:31
SPEAKER_01: obviously you're like, what can we do with this model?
10:34
SPEAKER_01: How do we present it to the researcher?
10:36
SPEAKER_01: How can they debug the inputs?
10:39
SPEAKER_01: And eventually you sort of like end up with, you know, like a chat system.
10:42
SPEAKER_01: So we built that internally.
10:43
SPEAKER_01: We had a lot of fun.
10:45
SPEAKER_01: Initially,
10:45
SPEAKER_01: the models were like,
10:46
SPEAKER_01: you know,
10:46
SPEAKER_01: kind of like almost like a little bit absurd, like,
10:48
SPEAKER_01: you know,
10:48
SPEAKER_01: not very coherent,
10:50
SPEAKER_01: not super useful, but it was a lot of fun to sort of like thinker with them.
10:55
SPEAKER_01: That caught up like,
10:56
SPEAKER_01: you know,
10:56
SPEAKER_01: like wildfires, like,
10:57
SPEAKER_01: you know,
10:57
SPEAKER_01: this application is just sort of like,
10:58
SPEAKER_01: you know, everyone was kind of like sharing little conversations within DeepMind.
11:04
SPEAKER_01: It felt more than like a research project or like a research project.
11:08
SPEAKER_01: A project for researchers.
11:09
SPEAKER_01: And so then there was this desire over time to launch it as an external product.
11:13
SPEAKER_01: But DeepMint was just not set up.
11:15
SPEAKER_01: There was the right way to launch products at Google.
11:18
SPEAKER_01: There was the whole machinery of how you do that.
11:22
SPEAKER_01: The last production stock,
11:24
SPEAKER_01: obviously very,
11:25
SPEAKER_01: very optimized over the years to do things well,
11:28
SPEAKER_01: but also very, very hard as an environment to truly innovate.
11:32
SPEAKER_00: And then I wanted to ask what made you, you know, look around or maybe even consider open AI.
11:37
SPEAKER_00: You partially answered this question.
11:40
SPEAKER_00: Just putting myself back into your shoes,
11:41
SPEAKER_00: like you're,
11:42
SPEAKER_00: you know,
11:43
SPEAKER_00: if it's 2024 or 2023,
11:45
SPEAKER_00: you're inside of Google who are publishing amazing papers,
11:48
SPEAKER_00: doing really good research.
11:49
SPEAKER_00: You're doing super fun stuff, right?
11:50
SPEAKER_00: Pushing the limits of what's been done before.
11:53
SPEAKER_00: It's inside a company where you already moved.
11:55
SPEAKER_00: You know,
11:56
SPEAKER_00: for people who are feeling kind of comfortable or good about
11:59
SPEAKER_00: where they are right now,
11:59
SPEAKER_00: which I imagine you must have been, like what made you still explore?
12:03
SPEAKER_00: All right, like what else might be there?
12:06
SPEAKER_01: Yeah, I was very comfortable.
12:07
SPEAKER_01: It's a good place.
12:12
SPEAKER_01: Really, I had a desire to meet great people, but also join a mission that I truly believed in.
12:20
SPEAKER_01: And that I felt like the people were true to the mission and cared deeply
12:25
SPEAKER_01: about impacting the world in a deeply positive way, but also in a direct way.
12:31
SPEAKER_01: Not being like, oh, yeah, it's just like, no, we just do this work over here.
12:34
SPEAKER_01: And then it's like, it's the job of someone else to figure out how to make this useful.
12:39
SPEAKER_01: I wanted to join a group where all the parameters were considered together,
12:44
SPEAKER_01: where research and product were really co-designing.
12:47
SPEAKER_01: OpenAI was just crushing it.
12:49
SPEAKER_00: I thought ChatGPT
12:50
SPEAKER_01: was taking off.
12:53
SPEAKER_01: I met a couple of people from OpenAI, and then I was like, wait, what?
12:56
SPEAKER_01: You only have 20 people working on ChatGPT?
13:01
SPEAKER_01: That is an insanely small number.
13:03
SPEAKER_01: That must be extremely empowering.
13:05
SPEAKER_01: How does that work?
13:06
SPEAKER_01: How do you manage to maintain a product with that level of scale and with that level
13:13
SPEAKER_01: of autonomy with only 20 engineers?
13:15
SPEAKER_01: And, you know, I kind of dug and dug and dug.
13:17
SPEAKER_01: And it's like,
13:17
SPEAKER_01: it was just an amazing group of people,
13:20
SPEAKER_01: amazing mission, you know, super talented, super driven.
13:22
SPEAKER_01: And like, it was, it was like drew me in.
13:25
SPEAKER_01: And then I joined pre-reasoning efforts immediately, like typical OpenAI fashions.
13:31
SPEAKER_01: Like I joined,
13:31
SPEAKER_01: it was like,
13:32
SPEAKER_01: oh yeah,
13:32
SPEAKER_01: you know,
13:33
SPEAKER_01: like there's this thing going on, like,
13:34
SPEAKER_01: you know,
13:35
SPEAKER_01: we're going to launch reasoning models, like, you know, it's like a new paradigm.
13:38
SPEAKER_01: And then, you know, start sprinting on that.
13:39
SPEAKER_01: And like, you know, like a month later, Like the company launched O1, O1 Preview.
13:45
SPEAKER_01: And that was exhilarating to be part of.
13:47
SPEAKER_01: I wanted to be part of like a place that moves fast,
13:50
SPEAKER_01: cares about impact,
13:51
SPEAKER_01: would be in tune with the world and, you know, just really listen.
13:56
SPEAKER_01: And sort of like,
13:56
SPEAKER_01: that's also to me, like,
13:58
SPEAKER_01: you know,
13:58
SPEAKER_01: what I've carried with me,
13:59
SPEAKER_01: like when building Codex,
14:01
SPEAKER_01: when building products,
14:02
SPEAKER_01: it's like having a community,
14:04
SPEAKER_01: listen to the community, just really focus on like a really intense feedback loop.
14:09
SPEAKER_01: And then building something that is just like,
14:12
SPEAKER_01: you know,
14:12
SPEAKER_01: you just really want to care about it and like,
14:14
SPEAKER_01: you know, care about the utility of it that it provides to the world.
14:17
SPEAKER_00: And then, of course, you start to work pretty quickly on Codex.
14:20
SPEAKER_00: So you joined in 2024.
14:22
SPEAKER_00: Can you take us back what the thinking back there when you joined was about AI
14:27
SPEAKER_00: or LLMs and code?
14:29
SPEAKER_00: I know there was this ASWE effort back then.
14:34
SPEAKER_00: We talked about it in the deep dive as well that we did in The Pragmatic Engineer,
14:37
SPEAKER_00: the Autonomous Software Engineer.
14:39
SPEAKER_00: ASWE.
14:40
SPEAKER_01: Yeah, that's what it was pronounced internally.
14:43
SPEAKER_01: We don't...
14:45
SPEAKER_01: Have a nice re-effort anymore.
14:47
SPEAKER_01: It's Codex.
14:48
SPEAKER_01: But really for me, I joined, I started building infrastructure for research.
14:54
SPEAKER_01: A lot of what I did before was large-scale data storage,
14:59
SPEAKER_01: analysis, and then tools to understand training runs.
15:03
SPEAKER_01: I did a lot of different things over my years,
15:06
SPEAKER_01: but it was always about building for others and making them faster
15:11
SPEAKER_01: and just really caring about fundamentally doing that well.
15:15
SPEAKER_01: And then through tooling and infrastructure, making new things possible.
15:18
SPEAKER_01: And so when I joined OpenAI, it was like with the same idea.
15:22
SPEAKER_01: And then with one preview and like,
15:25
SPEAKER_01: you know,
15:26
SPEAKER_01: some of the later models,
15:27
SPEAKER_01: it was very clear that we had to use the models themselves to help us go faster.
15:32
SPEAKER_01: And so I just really got obsessed with this idea of what were the limitations?
15:36
SPEAKER_01: How are we going to use those models for research itself?
15:40
SPEAKER_01: So we got together with other folks in research.
15:43
SPEAKER_01: We started training models.
15:45
SPEAKER_01: We started building little agents.
15:47
SPEAKER_01: Those were truly the precursor to Codex.
15:50
SPEAKER_01: And this was like,
15:51
SPEAKER_01: we were training internal models to be very proficient on the Python code base
15:57
SPEAKER_01: of OpenAI,
15:58
SPEAKER_01: and then very proficient with having good taste in architecture,
16:04
SPEAKER_01: good taste in code style.
16:07
SPEAKER_01: It was Python only.
16:09
SPEAKER_01: And then the idea was like,
16:10
SPEAKER_01: you know,
16:10
SPEAKER_01: we would sort of like use that to build infrastructure very quickly and,
16:14
SPEAKER_01: you know, help researchers code faster as well.
16:16
SPEAKER_01: And then, you know, and then we would move faster.
16:18
SPEAKER_01: And then over time,
16:19
SPEAKER_01: when you just kind of push that and simplify it to its score,
16:22
SPEAKER_01: you're making a lot of,
16:24
SPEAKER_01: you know,
16:24
SPEAKER_01: we found that we could make a lot of progress very quickly and
16:27
SPEAKER_01: then learn very quickly.
16:28
SPEAKER_01: And then Greg and Sam are, you know, people with immensely supportive.
16:34
SPEAKER_01: And also Greg was very adamant that,
16:37
SPEAKER_01: you know,
16:37
SPEAKER_01: we would,
16:38
SPEAKER_01: We would not just focus on ourselves,
16:40
SPEAKER_01: but we would also focus on benefiting the world.
16:43
SPEAKER_01: And so he just sort of encouraged that we would be thinking about this not just as
16:48
SPEAKER_01: a tool for OpenAI itself,
16:49
SPEAKER_01: but also as something that we would actually make into a product.
16:52
SPEAKER_01: And this is when We merged this research effort with this A3 effort,
16:58
SPEAKER_01: and we started building one thing.
17:00
SPEAKER_01: And then that led to a sprint,
17:01
SPEAKER_01: which was like the initial cloud codex that we launched,
17:04
SPEAKER_01: which didn't really have PMF because it was like a little bit too high friction.
17:08
SPEAKER_01: And we also launched the codex CLI, and we continued to push.
17:12
SPEAKER_01: But it was always this idea of, hey, how do we get models to really help here?
17:17
SPEAKER_00: You mentioned that first you started to build this model to train on the Python code
17:21
SPEAKER_00: and actually help build it in for better.
17:24
SPEAKER_00: But then you made this interesting decision where for Codex, you built it in Rust.
17:29
SPEAKER_00: And at the time, the model was not on distribution for Rust, right?
17:33
SPEAKER_00: It wasn't as good as Rust and it was in Python or TypeScript.
17:36
SPEAKER_00: Why did you make that kind of a decision?
17:39
SPEAKER_00: Was it kind of like,
17:40
SPEAKER_00: did you expect that it will catch up or you figure
17:42
SPEAKER_00: that performance is more important?
17:44
SPEAKER_00: Because it was very counterintuitive.
17:45
SPEAKER_00: Most of the other harnesses built were actually not built in Rust.
17:49
SPEAKER_00: They were built on distribution of TypeScript or Python or something else.
17:52
SPEAKER_01: Yes.
17:53
SPEAKER_01: From first principles,
17:57
SPEAKER_01: very early on,
17:58
SPEAKER_01: we were thinking about the product interface and the agent as different things.
18:05
SPEAKER_01: It was very important to build the core of the agent in a way that was robust,
18:14
SPEAKER_01: that was secure as well, that was engineered for efficiency and skill.
18:22
SPEAKER_01: Having worked through projects over the years that go from, hey,
18:27
SPEAKER_01: this is a fun thing to, hey,
18:29
SPEAKER_01: we need to scale this to the scale of the largest data center,
18:35
SPEAKER_01: decisions early on really turned out to be quite important.
18:40
SPEAKER_01: As long as you don't sacrifice too much of the velocity.
18:42
SPEAKER_01: And so it's a tradeoff, but we had very prolific and amazing Rust developers.
18:47
SPEAKER_01: Our internal models were not bad at Rust.
18:50
SPEAKER_01: And then you get a lot of validation as well at compile time.
18:53
SPEAKER_01: It's like statically verified and all these things.
18:56
SPEAKER_01: And that is great for agents too.
18:57
SPEAKER_01: So it turns out,
18:59
SPEAKER_01: you know,
18:59
SPEAKER_01: it was quite clear that,
19:01
SPEAKER_01: you know,
19:01
SPEAKER_01: Rust as a language would actually be quite good for agents fairly quickly
19:06
SPEAKER_01: if we decided to put some effort into it.
19:08
SPEAKER_01: But primarily we were focused on correctness and we were focused on efficiency as well.
19:12
SPEAKER_00: Interesting.
19:13
SPEAKER_00: So you're saying,
19:14
SPEAKER_00: you know,
19:14
SPEAKER_00: it's worth,
19:15
SPEAKER_00: in your case, it was worth thinking ahead of where you want this thing to be.
19:19
SPEAKER_00: And for example, think like a language choice.
19:20
SPEAKER_00: Obviously,
19:21
SPEAKER_00: with agents,
19:21
SPEAKER_00: you can rewrite a bunch of stuff and easier than in the past,
19:24
SPEAKER_00: but it's still like you can save yourself reworking by putting in the right,
19:29
SPEAKER_00: I guess,
19:30
SPEAKER_00: scaffolding or, well, you know, the baseline of what you're building on, right?
19:35
SPEAKER_01: I think we could have been successful if we had written it in TypeScript
19:39
SPEAKER_01: or maybe even Python and then it would have been fine and
19:41
SPEAKER_01: then we would have rewritten it at some point.
19:43
SPEAKER_01: But having a very clean separation between the agent itself,
19:49
SPEAKER_01: which can exist irrespective of the product, It was a very important principle.
19:54
SPEAKER_01: And if you write everything in the same code base,
19:57
SPEAKER_01: in the same language,
19:57
SPEAKER_01: it's like inevitably you're going to be a little bit sloppy and you're going
20:02
SPEAKER_01: to intertwine things more than you should.
20:04
SPEAKER_01: And then it's going to prevent further innovation after that.
20:08
SPEAKER_01: And so that was that was very important.
20:09
SPEAKER_01: Like the Rust boundary, in a sense, like was very useful for that.
20:12
SPEAKER_00: One interesting decision that you made,
20:14
SPEAKER_00: which is unique across all of the major labs is There was something really
20:44
SPEAKER_01: Cool about the idea of having the code open source
20:48
SPEAKER_01: because fundamentally what you're billing is you're billing a coding agent.
20:53
SPEAKER_01: And so we were sort of like thinking about, well,
20:56
SPEAKER_01: if you have that,
20:57
SPEAKER_01: you know,
20:57
SPEAKER_01: you're obviously going to point it at itself and now maybe You know,
21:02
SPEAKER_01: you can build a community of contributors that use it to improve it.
21:06
SPEAKER_01: And then, you know, you can learn a lot from that.
21:08
SPEAKER_01: Also, it felt at the time is like,
21:10
SPEAKER_01: you know,
21:10
SPEAKER_01: very clear to us that if we were going to be successful,
21:14
SPEAKER_01: open source itself would change and the role of code itself would change.
21:18
SPEAKER_01: And so being part of that community seemed important instead of divorced from it.
21:22
SPEAKER_01: I think, you know, it's hard to solve problems if you don't sort of like witness them yourself.
21:28
SPEAKER_01: And then the other thing was just,
21:30
SPEAKER_01: it still feels like early, but it was very early at the time.
21:33
SPEAKER_01: It felt like we would have some ideas for how to solve things well.
21:39
SPEAKER_01: And we were co-designing these with the training and the research.
21:45
SPEAKER_01: And it's all about expressing the capabilities of the model in the most flexible and
21:50
SPEAKER_01: the best way.
21:51
SPEAKER_01: But also, we didn't have all the answers.
21:54
SPEAKER_01: Sort of being very open about, hey, this is what a good harness looks like.
21:59
SPEAKER_01: This is how we think about it.
22:01
SPEAKER_01: We did like a couple of like very technical like deep dives and blog posts
22:05
SPEAKER_01: and we talked about it a lot.
22:06
SPEAKER_01: And we thought, you know, hey, it's just like the world is vast out there.
22:10
SPEAKER_01: Just like,
22:10
SPEAKER_01: you know,
22:10
SPEAKER_01: crazy smart people is like,
22:11
SPEAKER_01: you know, we're going to get inspired by other open source project as well.
22:16
SPEAKER_01: And so let's just.
22:18
SPEAKER_01: Make this a level playing field and sort of like encourage a lot of tinkering
22:24
SPEAKER_01: and exploration at this stage.
22:26
SPEAKER_00: Now, this has been now,
22:27
SPEAKER_00: you know,
22:27
SPEAKER_00: like a year later,
22:29
SPEAKER_00: a year and a half later, which is a very long time right now in this AI timeframe.
22:33
SPEAKER_00: But looking back or taking the experience,
22:37
SPEAKER_00: What are the benefits you've seen,
22:38
SPEAKER_00: the kind of engineering benefits,
22:40
SPEAKER_00: the engineering teams benefits from being open source?
22:42
SPEAKER_00: And just honestly, what are things that are kind of hard about being open source, right?
22:46
SPEAKER_00: Like there must be downsides, like just trying to get an honest take on both sides.
22:51
SPEAKER_01: Yeah, there are definitely downsides.
22:54
SPEAKER_01: It comes at a cost, right?
22:57
SPEAKER_01: The benefits are, it's awesome to build in the open.
23:00
SPEAKER_01: It's awesome to have like a small repo as well.
23:04
SPEAKER_01: Like whenever we hire someone,
23:06
SPEAKER_01: If someone joined the Codex team,
23:08
SPEAKER_01: they've seen the repo before, they've looked at PRs.
23:12
SPEAKER_00: Onboarding is done.
23:13
SPEAKER_01: Yeah, it's done.
23:14
SPEAKER_01: Onboarding is just like you use Codex to look at the repo with you
23:18
SPEAKER_01: and you ask some questions,
23:19
SPEAKER_01: but it's not a secret issue that you can get productive right away.
23:23
SPEAKER_01: We get a lot of good contributions, although we get a tsunami of random stuff as well.
23:29
SPEAKER_00: Obviously, you and everyone else.
23:31
SPEAKER_00: Right?
23:31
SPEAKER_00: Open source is changing.
23:32
SPEAKER_00: I think this is one of the examples.
23:34
SPEAKER_01: That's right.
23:35
SPEAKER_01: And then to me,
23:37
SPEAKER_01: and to a lot of the team,
23:38
SPEAKER_01: it just brings a lot of energy to just be part of the community
23:41
SPEAKER_01: and be directly contributing.
23:43
SPEAKER_01: Not just saying that we care about the community,
23:45
SPEAKER_01: but actually doing things that, you know, you can see it's costing us effort, right?
23:49
SPEAKER_01: We don't have to do it.
23:51
SPEAKER_01: The downsides are it's separate from the rest of our code.
23:56
SPEAKER_01: So sometimes we have to draw artificial boundaries and work across multiple repos.
24:01
SPEAKER_01: When we're working on something particularly exciting and we're building it in
24:06
SPEAKER_01: the open,
24:07
SPEAKER_01: then at times we find that others copy it before we have the time to release it.
24:13
SPEAKER_01: And it's like, It's just a little bit sad, but also it's like it's part of the game.
24:19
SPEAKER_01: You know,
24:19
SPEAKER_01: it's like you're building an open and it's like,
24:21
SPEAKER_01: you know,
24:22
SPEAKER_01: that's sort of like the contract that you signed is like, you know, you can copy it.
24:25
SPEAKER_01: We have a very permissive license as well,
24:27
SPEAKER_01: but it does sting a little bit when you're working on something and you're like,
24:31
SPEAKER_01: you know.
24:31
SPEAKER_01: And then the third thing is just like everyone else is like,
24:34
SPEAKER_01: you know,
24:35
SPEAKER_01: we are overwhelmed with,
24:36
SPEAKER_01: you know,
24:36
SPEAKER_01: random contributions and, you know, we have to deal with that additional tax.
24:41
SPEAKER_01: But then that pushes us to also try and solve for it, which I think is good.
24:46
SPEAKER_00: And on top of the open source,
24:47
SPEAKER_00: one thing that surprised me about Codex,
24:49
SPEAKER_00: and I didn't even know about it until recently, it's not tied to the OpenAI models.
24:54
SPEAKER_00: You can use other models with Codex.
24:58
SPEAKER_00: You know,
24:58
SPEAKER_00: like putting myself in a vendor's shoe,
25:01
SPEAKER_00: it might not be very obvious because, again,
25:04
SPEAKER_00: all the other brands that I look at when they do a CLI,
25:06
SPEAKER_00: it's kind of use it with our models.
25:08
SPEAKER_00: Again, what made you decide to be this permissive about using or allowing
25:14
SPEAKER_00: to use your harness with other models?
25:18
SPEAKER_01: It felt quite natural.
25:20
SPEAKER_01: If you are part of this community and building an excellent coding harness,
25:29
SPEAKER_01: why would you couple it to your model?
25:32
SPEAKER_01: That felt quite disappointing to make that decision.
25:35
SPEAKER_01: So it didn't feel right.
25:37
SPEAKER_01: And in general,
25:38
SPEAKER_01: it's like I think,
25:39
SPEAKER_01: you know,
25:39
SPEAKER_01: it's like I kind of try to make decisions that I'm like,
25:42
SPEAKER_01: yes, you know, it's just like I can just sort of like explain it.
25:45
SPEAKER_01: You know, it is correct.
25:47
SPEAKER_01: It's the same reasoning with, you know, it is open source in the first place.
25:50
SPEAKER_01: It would have been trivial for anyone to fork it and then add support for another thing.
25:56
SPEAKER_01: But then you're just encouraging people to just like, you know, go and use that fork.
26:00
SPEAKER_01: And then not suddenly you have overhead.
26:02
SPEAKER_01: And the only reason you have a fork is because,
26:04
SPEAKER_01: you know,
26:04
SPEAKER_01: you wanted to change like 10 lines of code to add support
26:07
SPEAKER_01: for another model provider.
26:09
SPEAKER_01: That feels very silly.
26:10
SPEAKER_01: So why not just support it in the first place?
26:13
SPEAKER_01: The other thing is we benefit a lot from being able to just give optionality.
26:18
SPEAKER_01: So maybe today you love using OpenAI models and you're super productive with them.
26:27
SPEAKER_01: But tomorrow there's a new model that comes out.
26:29
SPEAKER_01: You want to try that.
26:31
SPEAKER_01: Why force you to go and completely change your setup just to try a new model?
26:35
SPEAKER_01: And then we benefit from the feedback that we didn't get,
26:38
SPEAKER_01: which is maybe there's something that you liked about that model.
26:40
SPEAKER_01: Maybe it actually didn't work well.
26:43
SPEAKER_01: It's sort of like being nice to our users and to the community feels like
26:48
SPEAKER_01: the right thing to do here.
26:50
SPEAKER_01: And then we also try other models, right?
26:54
SPEAKER_01: So we try them in the same harness and it's just all good.
26:59
SPEAKER_01: And then this is also often like this optionality is very important to companies
27:04
SPEAKER_01: that we work with.
27:06
SPEAKER_01: This is something that we absolutely lean into.
27:09
SPEAKER_00: This last point,
27:09
SPEAKER_00: I think,
27:10
SPEAKER_00: you know,
27:10
SPEAKER_00: as any serious company,
27:11
SPEAKER_00: you want to have optionality and you want to use a tool that gives you
27:14
SPEAKER_00: that optionality.
27:16
SPEAKER_00: But I kind of appreciate it because I feel to me like it's kind of honest, like,
27:18
SPEAKER_00: look,
27:19
SPEAKER_00: like it forces the whole company to be to compete the best in everywhere in
27:24
SPEAKER_00: the model layer and the harness layer with open source with with chooseable models.
27:27
SPEAKER_00: And it kind of like doesn't doesn't allow you to like kick back and say like,
27:30
SPEAKER_00: all right, we're done.
27:31
SPEAKER_00: We can we can we can hang back for a little bit for now.
27:33
SPEAKER_01: Yeah, I want us to win users by having,
27:36
SPEAKER_01: you know, the best models, the most efficient models, the best product.
27:40
SPEAKER_01: And then, you know, if we do all of these things, it's like we're going to have a good time.
27:43
SPEAKER_01: If we sort of like force you to use the product because,
27:45
SPEAKER_01: you know,
27:46
SPEAKER_01: this one thing,
27:46
SPEAKER_01: it's just like,
27:47
SPEAKER_01: then I don't think that will attract,
27:49
SPEAKER_01: you know, the very best people to work on this product either.
27:52
SPEAKER_01: And it's like, you know, it's like we're doing our best work here.
27:55
SPEAKER_01: We care a lot about the experience.
27:57
SPEAKER_01: It should feel delightful,
27:58
SPEAKER_01: you know,
27:59
SPEAKER_01: like it doesn't,
28:00
SPEAKER_01: irrespective of the model that powers it, should feel delightful.
28:03
SPEAKER_00: I love the idea of winning based on merit, not based on lock-in.
28:06
SPEAKER_00: And this is a perfect time to mention our season sponsor,
28:09
SPEAKER_00: Entire, who also play by the same rules.
28:11
SPEAKER_00: Like it or not, Git is becoming a bottleneck for modern agent-heavy software development.
28:16
SPEAKER_00: Devs are creating more code with agents.
28:18
SPEAKER_00: Agents are pushing more code,
28:19
SPEAKER_00: many devs are running more parallel agents, these are pushing even more code.
28:23
SPEAKER_00: GitHub is clearly struggling to keep up and has frequent outages.
28:27
SPEAKER_00: So, what's the solution?
28:28
SPEAKER_00: Entire was founded by GitHub's last CEO,
28:31
SPEAKER_00: Thomas Dumka, and he rebuilt GitHub hosting for the agentic era from scratch.
28:35
SPEAKER_00: Entire was built to be very fast,
28:36
SPEAKER_00: and to have your repos regionally close to you to reduce latency,
28:40
SPEAKER_00: allowing for fleets of agents to push in parallel.
28:43
SPEAKER_00: Some numbers they published.
28:44
SPEAKER_00: Entire can handle 418 pushes per second.
28:47
SPEAKER_00: That's up to 89 times faster than every competitor on the market.
28:52
SPEAKER_00: When GitHub is down, you can still keep working and you don't even need to migrate from GitHub.
28:56
SPEAKER_00: You just sign up to Entire and the platform mirrors your repo.
28:59
SPEAKER_00: And one more neat thing.
29:01
SPEAKER_00: Have you ever wondered what prompt resulted in this specific code being generated?
29:05
SPEAKER_00: I find that the prompt and conversation with the agent carries more information than
29:09
SPEAKER_00: the PR itself, at least for me.
29:11
SPEAKER_00: Entire captures all the prompt history with your agent right in the repo,
29:15
SPEAKER_00: easy to check back, and has a pretty innovative UI to show all of this.
29:18
SPEAKER_00: If you're looking for Git hosting that works even when GitHub is down,
29:21
SPEAKER_00: head to entire.io slash pragmatic,
29:23
SPEAKER_00: install the CLI, and mirror your repo with a click.
29:26
SPEAKER_00: I've already done it.
29:28
SPEAKER_00: Oh, and did I mention that it works with any agent and it's open source?
29:32
SPEAKER_00: I'd also like to mention our season sponsor, Antisysys.
29:35
SPEAKER_00: Thibault talked about how the experience of the software you use should feel delightful.
29:39
SPEAKER_00: Delightful includes no annoying bugs.
29:41
SPEAKER_00: But when you're using agents to write your code, how do you avoid shipping bugs?
29:45
SPEAKER_00: Reviewing every line of code is becoming a challenge with the amount of code
29:48
SPEAKER_00: that agents generate, which is why Antisysys goes well beyond code review.
29:53
SPEAKER_00: Antisysys runs your whole system in a hostile simulation.
29:56
SPEAKER_00: This simulation includes both targeted testing and fast testing.
30:00
SPEAKER_00: By running this simulation, it finds every bug before your users do.
30:04
SPEAKER_00: And because the simulation is fully deterministic,
30:07
SPEAKER_00: it doesn't only find bugs,
30:08
SPEAKER_00: it gives you a perfect reproduction of every issue,
30:11
SPEAKER_00: which makes it much easier to fix issues.
30:13
SPEAKER_00: The first thing I thought when I heard about antithesis is
30:16
SPEAKER_00: that automated bug discovery
30:17
SPEAKER_00: and fully deterministic testing sounds like science fiction,
30:20
SPEAKER_00: but it's actually hardcore engineering under the hood.
30:23
SPEAKER_00: Janestreet,
30:24
SPEAKER_00: Fly.io and the etcd community ship agent written code with full confidence
30:28
SPEAKER_00: because they know it's been verified by Antistisys.
30:31
SPEAKER_00: To see more case studies and details head to antistisys.com slash pragmatic.
30:36
SPEAKER_00: And with this let's get back to Thibault and why competition between tools is great.
30:41
SPEAKER_00: Yeah, and I think as an engineer,
30:42
SPEAKER_00: I always see that whenever there's competition as someone who's using tools,
30:46
SPEAKER_00: it's always amazing.
30:47
SPEAKER_00: I remember when Microsoft had with JetBrains,
30:50
SPEAKER_00: the IDE wars,
30:52
SPEAKER_00: and then there's the clouds battling with each other with all the features.
30:56
SPEAKER_00: And now, of course, we have the harnesses, we have the models.
30:58
SPEAKER_00: And as a user, it's great because now we have more choice.
31:01
SPEAKER_00: They just develop faster.
31:03
SPEAKER_00: I guess our voice gets heard a bit better.
31:05
SPEAKER_00: So it's great to hear.
31:07
SPEAKER_00: Speaking of the harness,
31:08
SPEAKER_00: can you tell me how it works today in the sense of like,
31:10
SPEAKER_00: when I start a codex task, does it run always on my machine?
31:18
SPEAKER_00: Does it choose the cloud?
31:20
SPEAKER_00: Does it use a sandbox?
31:21
SPEAKER_00: And how do I control this or know this?
31:25
SPEAKER_00: Or how much should I know about this as an engineer?
31:27
SPEAKER_01: Yes.
31:28
SPEAKER_01: So by default, it runs sandboxed.
31:36
SPEAKER_01: If there is a command that should run with additional permissions outside of
31:41
SPEAKER_01: the sandbox, it will ask you as a user for permission.
31:46
SPEAKER_01: But every tool execution happens within the sandbox by default,
31:51
SPEAKER_01: and it runs entirely on the machine, your local machine.
31:57
SPEAKER_01: This has been the case for more than a year now.
32:01
SPEAKER_01: But it is something that is evolving and shifting,
32:04
SPEAKER_01: where you can select to run this in the cloud,
32:08
SPEAKER_01: which then runs in a managed VM,
32:11
SPEAKER_01: where it's the same VM that you get through ChatGPT work.
32:16
SPEAKER_01: And you can inspect it, but it runs in a Cata container.
32:20
SPEAKER_01: It's a secure environment.
32:22
SPEAKER_01: And so everything runs inside of that.
32:26
SPEAKER_01: VM and it doesn't run on your machine.
32:27
SPEAKER_01: And then the only thing that happens in your machine is like your input and then
32:32
SPEAKER_01: the streaming back of the output.
32:34
SPEAKER_01: And so that obviously then is like much nicer on your CPU and your machine
32:39
SPEAKER_01: and you can scale much, much, much more.
32:42
SPEAKER_01: And this is just a step.
32:45
SPEAKER_01: It's going to be much more seamless in the future to use cloud machines and
32:51
SPEAKER_01: then maybe have a combination of partial execution on your laptop,
32:55
SPEAKER_01: partial execution on cloud machines.
32:59
SPEAKER_01: And really the thing that we're thinking about that is very natural is
33:03
SPEAKER_01: as models just get better and more capable,
33:07
SPEAKER_01: They can leverage so much more compute and many more resources than are available
33:13
SPEAKER_01: on your local machine.
33:15
SPEAKER_01: And so it would be a constraint at some point to just limit execution on your local machine.
33:20
SPEAKER_00: But one thing that is great about it running locally,
33:23
SPEAKER_00: and I think the reason I love it when it runs locally,
33:26
SPEAKER_00: of course,
33:26
SPEAKER_00: it's a pain because if I'm doing some work,
33:28
SPEAKER_00: it's like, you know, I have several agents, it's eating CPU.
33:31
SPEAKER_00: If I want to close my laptop, I cannot kind of leave it like half open.
33:36
SPEAKER_00: Right.
33:36
SPEAKER_00: When I was in one of the offices of an AI company,
33:39
SPEAKER_00: I had it half open and they're like, are you running agents?
33:41
SPEAKER_00: I'm like, yeah, I have one running.
33:43
SPEAKER_00: I get it.
33:43
SPEAKER_00: But the reason I do it because I have my local tools.
33:46
SPEAKER_00: I have my local Postgres database.
33:49
SPEAKER_00: I have my this and that.
33:51
SPEAKER_00: How are you thinking about the cloud is amazing,
33:53
SPEAKER_00: but it doesn't have this setup or it's just a pain to set it up.
33:56
SPEAKER_00: Are you thinking or are you experimenting with You know,
34:00
SPEAKER_00: making the setups and I'm kind of reminded of a topic that we talked about pre-A,
34:04
SPEAKER_00: which is cloud development environments and like 2022, 23, they're hot.
34:08
SPEAKER_00: And then we talked about AI more.
34:10
SPEAKER_01: But I think outside
34:12
SPEAKER_01: of large tech companies like cloud dev boxes really never took off because there's
34:17
SPEAKER_01: a very big upfront cost.
34:20
SPEAKER_01: And then you need to pay like a maintenance cost as well.
34:22
SPEAKER_01: And you just, you know, don't benefit from it as like a solo developer or like a small team.
34:27
SPEAKER_01: With the level of capabilities that we have in agents now,
34:30
SPEAKER_01: it's like the setup almost is free, right?
34:32
SPEAKER_01: So like this,
34:33
SPEAKER_01: the setup cost and this maintenance cost is like,
34:36
SPEAKER_01: if your agent is capable of doing it, you know, it should just do it for you.
34:39
SPEAKER_01: So for example,
34:39
SPEAKER_01: if you're saying like, hey,
34:40
SPEAKER_01: you know,
34:41
SPEAKER_01: I have like,
34:41
SPEAKER_01: I have my local, um, SQLite, or I have a local server and MCPs and whatnot.
34:48
SPEAKER_01: It's like,
34:49
SPEAKER_01: how hard is it to actually configure exactly the same setup and keep it in sync on
34:54
SPEAKER_01: a cloud dev box?
34:56
SPEAKER_01: Well, maybe it's not that hard if the model just does it for you.
34:59
SPEAKER_01: And so I think we're going to see a resurgence of You know,
35:02
SPEAKER_01: fully cloud orchestrated machines, which then frees you from your laptop, right?
35:08
SPEAKER_01: It's like one thing that we've seen a ton of success with
35:11
SPEAKER_01: with ChatGPT work is like it's just available on your mobile.
35:14
SPEAKER_01: I start my day just dictating a bunch of tasks into it next to the coffee and it just does it.
35:20
SPEAKER_01: It has access to my calendar.
35:21
SPEAKER_01: It has access to my email.
35:23
SPEAKER_01: It has access to Slack.
35:24
SPEAKER_01: And it's just so awesome to just be able to walk around and,
35:27
SPEAKER_01: you know, get stuff done without having to, you know, carry my laptop everywhere.
35:32
SPEAKER_01: And I think it's the same thing that,
35:34
SPEAKER_01: you know,
35:35
SPEAKER_01: we ship like Codex Remote where,
35:36
SPEAKER_01: you know,
35:36
SPEAKER_01: execution is like still happening on your laptop,
35:38
SPEAKER_01: but it would be wonderful if, you know, you didn't have to keep your laptop open.
35:43
SPEAKER_00: Can you tell me a bit on how, in the past, how did you improve Codex?
35:47
SPEAKER_00: Because I remember when I first used Codex,
35:50
SPEAKER_00: this was one of the early versions,
35:52
SPEAKER_00: you know, like you could talk to it, it did stuff.
35:54
SPEAKER_00: But for example, I said like, all right, make this change.
35:57
SPEAKER_00: And it did that change.
35:58
SPEAKER_00: And I had unit tests and it didn't run it.
36:00
SPEAKER_00: And then later,
36:01
SPEAKER_00: a few months later,
36:02
SPEAKER_00: I don't know exactly when, it just started to run it automatically.
36:05
SPEAKER_00: Were these things,
36:05
SPEAKER_00: did you improve the, you know, the script that runs, you know, the instructions?
36:09
SPEAKER_00: I'm not sure how exactly you call the...
36:12
SPEAKER_00: The bootstrapping script or whatever that is, is it improving the model?
36:16
SPEAKER_00: As a dev,
36:17
SPEAKER_00: how can I imagine you making each version better between the harness and then
36:21
SPEAKER_00: between the model?
36:22
SPEAKER_00: And what's the connection between the two?
36:25
SPEAKER_01: Yeah, this is a good question.
36:27
SPEAKER_01: So the harness, in a sense, is always a little bit ahead of the model.
36:33
SPEAKER_00: Oh, really?
36:34
SPEAKER_00: How so?
36:36
SPEAKER_01: What I mean by that is that you have the model,
36:39
SPEAKER_01: it's capable of certain things,
36:40
SPEAKER_01: but then you set it up with a couple of crutches so that it can actually do
36:47
SPEAKER_01: the thing to a level of reliability and in a way that is efficient and also with
36:56
SPEAKER_01: the behavior that you expect as a user.
36:59
SPEAKER_01: And so that's the role of the harness,
37:01
SPEAKER_01: provide guardrails,
37:03
SPEAKER_01: safety, make it more efficient, make it more steerable, controllable.
37:07
SPEAKER_01: And then the harness usually is also responsible for what we call
37:12
SPEAKER_01: the developer message, which is infected in the context at the start of each turn.
37:26
SPEAKER_01: A
37:29
SPEAKER_01: lot of what you have is the result of the harness and the model.
37:34
SPEAKER_01: Initially, like maybe you're like, oh, it doesn't run tests.
37:37
SPEAKER_01: So, you know, you have to remind it to run tests.
37:39
SPEAKER_01: And then,
37:39
SPEAKER_01: you know,
37:39
SPEAKER_01: we train a battle model that is just You know,
37:43
SPEAKER_01: capable of like better reflecting on what is it that you really want when you ask
37:48
SPEAKER_01: for something and then, you know, you don't actually have to tell it anymore.
37:52
SPEAKER_01: So over time,
37:53
SPEAKER_01: what we see is like the system,
37:56
SPEAKER_01: the developer message shrinks and then the harness also shrinks.
37:59
SPEAKER_00: Inside of the Codex team, do you have specific goals?
38:02
SPEAKER_00: Do you say like,
38:03
SPEAKER_00: all right,
38:03
SPEAKER_00: right now the Codex as a hardness and model combined is not very good at this,
38:10
SPEAKER_00: or it's kind of doing silly mistakes right here, or how can I imagine that?
38:14
SPEAKER_00: As the engineering team, how you're working on the next version of Codex.
38:19
SPEAKER_00: Because the thing that I don't really get as a dev is like,
38:22
SPEAKER_00: okay, there's a model, which to me is this magical thing, which will get better.
38:25
SPEAKER_00: Of course,
38:26
SPEAKER_00: I'm sure you have some feedback channels,
38:27
SPEAKER_00: but you also have the hardness, which is the tools that you're building.
38:30
SPEAKER_00: Like that's probably what the team is responsible for.
38:32
SPEAKER_00: How do you set even your goals, right?
38:33
SPEAKER_00: Like in traditional software,
38:34
SPEAKER_00: you'd be like,
38:34
SPEAKER_00: we will build this feature and you build that feature because you know how to do it.
38:37
SPEAKER_00: But it feels a bit more fuzzy to me, this development process.
38:40
SPEAKER_01: Yeah, it is.
38:41
SPEAKER_01: And it's why we co-design most things.
38:44
SPEAKER_01: And it's a process where it's a collaboration between research and
38:49
SPEAKER_01: the engineering team, like primarily building the core agent harness.
38:55
SPEAKER_01: It's always a question of like, okay,
38:57
SPEAKER_01: we see today that,
38:59
SPEAKER_01: you know, we are very good at this, but we're not very good at this.
39:02
SPEAKER_01: And, you know,
39:03
SPEAKER_01: we have a desire to do like another thing because it would be
39:06
SPEAKER_01: a very cool product feature.
39:07
SPEAKER_01: And then, you know, we always sort of like look at it.
39:09
SPEAKER_01: It's like, okay, should this be like a harness change or should this be a model change?
39:14
SPEAKER_01: And if it's a model change, like how soon can we have it?
39:16
SPEAKER_01: Can we have it in a month?
39:17
SPEAKER_01: Can we have it in, you know, three months, six months?
39:19
SPEAKER_01: And we sort of like work through that.
39:21
SPEAKER_01: And then depending on how soon we can just fix it in the model at which level
39:26
SPEAKER_01: of training,
39:27
SPEAKER_01: then we might decide to not even do something in the harness at all and just wait
39:32
SPEAKER_01: for the model to solve it.
39:34
SPEAKER_01: It's agents all the way, right?
39:36
SPEAKER_01: Agents to analyze a lot of the feedback,
39:38
SPEAKER_01: to come up with themes,
39:40
SPEAKER_01: to just help us have these conversations and decide on priorities.
39:44
SPEAKER_01: But we analyze it across all of coding.
39:47
SPEAKER_01: We analyze it across all of the other domains,
39:50
SPEAKER_01: like finance,
39:51
SPEAKER_01: comps, marketing, all the things where our users are using these agents nowadays.
39:56
SPEAKER_01: And there's subcategories within those.
39:58
SPEAKER_01: And then we roughly know how well we perform.
40:01
SPEAKER_01: And then we're always pushing the frontier.
40:03
SPEAKER_01: And there's a thing that is interesting is like,
40:05
SPEAKER_01: as we make,
40:06
SPEAKER_01: you know,
40:06
SPEAKER_01: as our pre-training model gets better,
40:08
SPEAKER_01: as we make the overall model better, like the whole thing lifts up.
40:11
SPEAKER_01: But then there are sometimes things that we pay a little bit more attention to.
40:14
SPEAKER_00: You mentioned, you analyze agents all the way.
40:17
SPEAKER_00: Can we talk about the software development lifecycle on Codex in the sense of whenever...
40:24
SPEAKER_00: A new engineer joins a team, any team, it's like, okay, how are things done here?
40:28
SPEAKER_00: And, you know,
40:29
SPEAKER_00: back pre-AI,
40:30
SPEAKER_00: it would have been you joined a company like Uber or Google,
40:33
SPEAKER_00: and they would tell you,
40:33
SPEAKER_00: like, cool, the way it works is we have an idea or the PM has an idea.
40:37
SPEAKER_00: We make a plan.
40:38
SPEAKER_00: We get together.
40:39
SPEAKER_00: We do some estimations.
40:40
SPEAKER_00: We break up the work.
40:41
SPEAKER_00: We code the work.
40:42
SPEAKER_00: We do tests.
40:42
SPEAKER_00: We do code reviews.
40:44
SPEAKER_00: We release.
40:44
SPEAKER_00: We do feature flags.
40:45
SPEAKER_00: And then, you know, we're on call.
40:48
SPEAKER_00: You know, that's how it used to be.
40:49
SPEAKER_00: When someone joins the Codex team, you know, they...
40:53
SPEAKER_00: They've been contributing to the open source part.
40:55
SPEAKER_00: But what do you tell them?
40:56
SPEAKER_00: How do things get done here if they're like a total newbie?
41:00
SPEAKER_01: I introduce them to great people.
41:02
SPEAKER_01: And then the thing that they hear the most about when they have a question is like,
41:06
SPEAKER_01: have you asked Codex?
41:08
SPEAKER_01: And Codex is just by default at OpenAI is plugged into everything.
41:14
SPEAKER_01: So it has access to Slack, it has access to all the documents, access to all the code.
41:19
SPEAKER_01: And it still surprises new starters that you can basically ask it anything.
41:25
SPEAKER_01: And it will very often just come up with a really good response.
41:30
SPEAKER_01: And so the easiest way to understand the state of a project or who's working
41:35
SPEAKER_01: on something or why a decision was made is Codex knows about it all internally.
41:40
SPEAKER_01: And so you just use all of that.
41:43
SPEAKER_01: We do a lot of work in, you know, for that reason, we do a lot of work in public channels.
41:48
SPEAKER_01: We open up documents with like, you know, fairly broad permissions.
41:52
SPEAKER_01: And so that, you know, everyone has access to this information as well.
41:55
SPEAKER_01: So that, you know, your agent can go through things and like, you know, reason through things.
41:58
SPEAKER_01: And then,
41:59
SPEAKER_01: you know,
41:59
SPEAKER_01: we have a couple of other things that are just really very helpful
42:02
SPEAKER_01: for team productivity and team collaboration that we haven't released yet,
42:07
SPEAKER_01: but are going to come, like some of it at Dev Day.
42:10
SPEAKER_01: All of that just sort of like makes you very grounded and in tune with the rest of the team.
42:15
SPEAKER_01: And allows you to just very,
42:16
SPEAKER_01: very quickly understand the state of things and produce things yourself.
42:20
SPEAKER_01: The general recommendation is just like, hey,
42:23
SPEAKER_01: care about the user,
42:24
SPEAKER_01: care about the coherence of the product,
42:27
SPEAKER_01: care about the models and where they're going.
42:29
SPEAKER_01: If you're doing something and you're building this 10,000 lines of code crutch
42:34
SPEAKER_01: to work around the model flaws, you're probably doing the wrong thing.
42:37
SPEAKER_01: So we have a set of principles,
42:39
SPEAKER_01: but it's just really So like a team culture and ethos at this point.
42:44
SPEAKER_01: And, you know,
42:45
SPEAKER_01: it's just very much like carries on,
42:47
SPEAKER_01: you know,
42:47
SPEAKER_01: when people join,
42:48
SPEAKER_01: it's just like through the rest of the team,
42:50
SPEAKER_01: just like, you know, sort of like teaching the ropes.
42:52
SPEAKER_00: And then when I have an idea, I think it's a good idea.
42:55
SPEAKER_00: I talk it through a codex.
42:57
SPEAKER_00: Maybe I talk with it with some of my colleagues, like,
42:58
SPEAKER_00: here's a cool new feature I'm going to build as my first contribution
43:02
SPEAKER_00: or first major contribution to codex.
43:05
SPEAKER_00: How do I go about that?
43:06
SPEAKER_00: Obviously, I code it down with Codex.
43:07
SPEAKER_00: I obviously test it and make sure that it works.
43:10
SPEAKER_00: From there on, what's the process?
43:12
SPEAKER_00: Do you still have the concept of code review or AI code review,
43:16
SPEAKER_00: a verification of rolling out, a verifying of stage rolled out?
43:19
SPEAKER_00: You know, the thing is because Codex itself, it goes out to millions of people.
43:23
SPEAKER_00: Like I just crossed a big 20 million active user mark.
43:27
SPEAKER_00: But if it's ChatGPT, then it also goes out to like even a lot bigger number of people.
43:32
SPEAKER_01: Yes, but it's surprisingly like a similar process,
43:36
SPEAKER_01: whether you ship on Codex or ChatGPT,
43:38
SPEAKER_01: even though ChatGPT goes out to like a billion active users and growing,
43:43
SPEAKER_01: you can ship a PR,
43:47
SPEAKER_01: you can make a change and get it shipped like the next day or like even
43:50
SPEAKER_01: the same day, and it just goes out to a billion users and it's fine.
43:53
SPEAKER_01: We just really instill a sense of ownership and care.
43:59
SPEAKER_01: So you're like, people are very empowered to make changes, even large changes.
44:02
SPEAKER_01: The general thing that is being asked is like sort of like evidence that it's going
44:06
SPEAKER_01: to be well received,
44:07
SPEAKER_01: evidence that is like a worthy addition,
44:10
SPEAKER_01: evidence that,
44:11
SPEAKER_01: you know,
44:11
SPEAKER_01: it's like it is worth maintaining over time,
44:13
SPEAKER_01: but also like the cost of maintenance is like just really has,
44:17
SPEAKER_01: you know, gotten done significantly as well.
44:19
SPEAKER_01: We think about these things slightly differently than, say, two years ago or three years ago.
44:24
SPEAKER_01: The other thing as well is we automate as much as possible.
44:26
SPEAKER_01: So a lot of the process of code review and deploys and catching regressions,
44:32
SPEAKER_01: all of that is pretty much automated.
44:35
SPEAKER_01: And so you get to just focus on It's really the idea and how it's going to help our users.
44:42
SPEAKER_01: And you care about the coherence of it all and the overall power of the agent
44:47
SPEAKER_01: and making things better.
44:48
SPEAKER_01: And we have a long, long list of things that we aspire to do and haven't gotten to yet.
44:54
SPEAKER_01: And then there's the north star direction, which is...
44:58
SPEAKER_01: A delightful,
44:59
SPEAKER_01: simple-to-use personal AGI that knows everything about you,
45:04
SPEAKER_01: that it needs to know,
45:05
SPEAKER_01: has access to the right resources,
45:07
SPEAKER_01: can take sometimes risky actions on your behalf,
45:09
SPEAKER_01: but then you get the push notification and then you can verify that.
45:13
SPEAKER_01: And it's a thing that you deeply understand as a user.
45:16
SPEAKER_01: But also it knows about your schedule.
45:18
SPEAKER_01: It knows about your goals.
45:19
SPEAKER_01: It can be proactive and it should be like extremely natural.
45:21
SPEAKER_01: It should be something that you can control through like natural language,
45:25
SPEAKER_01: voice, you know, like maybe it should understand, you know, your emotions.
45:28
SPEAKER_01: Like if it has like a camera feed, it should be the most natural thing on earth.
45:31
SPEAKER_01: It's like it should not be like a thing with 10, you know, different buttons and configurations.
45:36
SPEAKER_01: It's like.
45:36
SPEAKER_01: AGI should be simple
45:38
SPEAKER_00: to use.
45:39
SPEAKER_00: You kind of mentioned just briefly the review, the code review, but I wanted to go back to it.
45:44
SPEAKER_00: You worked at Google on a product used by hundreds of millions,
45:48
SPEAKER_00: which is Google Maps,
45:49
SPEAKER_00: and Google is very well known for their culture of Very strict code reviews.
45:53
SPEAKER_00: They have, I think, two layers of code reviews.
45:55
SPEAKER_00: There's a language correctness review.
45:57
SPEAKER_00: And they've taken, I think they've really perfected it across the industry for a long time.
46:01
SPEAKER_00: And they do believe that it works and they use it.
46:04
SPEAKER_00: How do you think that part is changing specifically the human review?
46:07
SPEAKER_00: Because for a very long time,
46:09
SPEAKER_00: until maybe a year or two ago,
46:11
SPEAKER_00: I would have said your code review has all these benefits.
46:14
SPEAKER_00: Knowledge sharing,
46:15
SPEAKER_00: the second pair of eyes,
46:16
SPEAKER_00: removing the bus factor,
46:17
SPEAKER_00: because now someone else understands them when that person is out that they can jump
46:20
SPEAKER_00: in.
46:21
SPEAKER_00: Conversations are happening about architecture, not just the code.
46:25
SPEAKER_00: But now there's, you know, there's a lot more code.
46:30
SPEAKER_00: And what was the value of code review?
46:33
SPEAKER_00: What, in what cases,
46:35
SPEAKER_00: and so on your team,
46:36
SPEAKER_00: because you guys are so ahead of this,
46:39
SPEAKER_00: where do you see humans still being or developers being involved in the review stage
46:44
SPEAKER_00: of valuable?
46:44
SPEAKER_00: And where is it fine?
46:47
SPEAKER_00: Did you find it fine to hand it off to an agent?
46:51
SPEAKER_01: Yeah, the role of code review is changing.
46:54
SPEAKER_01: One of the early projects that I did on Codex was working with research
46:58
SPEAKER_01: on developing a code review model that was going to be to a level
47:06
SPEAKER_01: where it can spot mistakes in logic and reasoning to a degree
47:11
SPEAKER_01: where it would require humans potentially multiple hours to capture the same level
47:18
SPEAKER_01: of mistake because It requires really digging three,
47:21
SPEAKER_01: four levels deep into the dependencies and understand that maybe
47:25
SPEAKER_01: the documentation actually was wrong and the implementation
47:27
SPEAKER_01: of this third-party dependency is different from what you expected and
47:30
SPEAKER_01: so therefore your invariants are not upheld.
47:33
SPEAKER_01: And these things,
47:34
SPEAKER_01: it's just like,
47:35
SPEAKER_01: unless you're an expert in that library,
47:36
SPEAKER_01: you wouldn't know and therefore you have a bug.
47:39
SPEAKER_01: And so we developed these code review models and we released them and now they're
47:44
SPEAKER_01: the same level of capability and ability to Spot these mistakes
47:49
SPEAKER_01: by doing deep verification are just part of the mainline models.
47:53
SPEAKER_01: When we benchmark them, it's like they're superhuman in code review.
47:56
SPEAKER_01: And this is not just true for correctness.
47:58
SPEAKER_01: This is also true for security,
48:00
SPEAKER_01: for example, where they're capable of reasoning across Very, very complex things.
48:04
SPEAKER_01: And then coming up with, hey,
48:06
SPEAKER_01: you have a critical security vulnerability here,
48:08
SPEAKER_01: which is now mandatory across all of OpenAI pull requests.
48:12
SPEAKER_01: We block pull requests from merging if we flag them with a security issue.
48:17
SPEAKER_01: And this is like all automatic.
48:19
SPEAKER_01: And the role of code review now is like, I think it was always about correctness.
48:23
SPEAKER_01: It was always about, you know, ensuring that things work.
48:25
SPEAKER_01: But it was also sort of like a little ritual for information exchange and,
48:29
SPEAKER_01: you know,
48:29
SPEAKER_01: bringing people on the same page and like,
48:31
SPEAKER_01: you know, encouraging like a discussion, which ideally would have happened before.
48:35
SPEAKER_01: But sometimes it just only happens like around the code because once it merged,
48:38
SPEAKER_01: it just actually runs in production and it's doing stuff.
48:40
SPEAKER_01: And then you have to maintain it.
48:42
SPEAKER_01: So there's this social aspect to it as well.
48:45
SPEAKER_01: I think all of it is changing.
48:46
SPEAKER_01: The correctness, the cybersecurity, I think that will be automated.
48:51
SPEAKER_01: Really what we see and I see is there's this sort of really discussion around
48:56
SPEAKER_01: the intent that takes place around the pull request.
48:59
SPEAKER_01: It's like, what are you even trying to do?
49:02
SPEAKER_01: And is that the right thing to attempt to do?
49:04
SPEAKER_01: I think you can have that discussion outside of the pull request.
49:06
SPEAKER_01: It doesn't have to be around code.
49:08
SPEAKER_00: So maybe this helps crystallize where a discussion needs to happen versus
49:13
SPEAKER_00: where we did it because maybe we didn't have the type of tooling
49:17
SPEAKER_00: that we have right now.
49:18
SPEAKER_01: Yeah, I think this is going to change.
49:20
SPEAKER_01: And it was like a forcing function because you have to have that discussion
49:23
SPEAKER_01: or it's good to have that discussion before you merge it
49:25
SPEAKER_01: and it becomes production code.
49:27
SPEAKER_01: But I think there are other ways to have these discussions
49:31
SPEAKER_01: and design things together and make sure that the intent is good.
49:35
SPEAKER_01: And then the code doesn't matter as much.
49:37
SPEAKER_00: And it's interesting because when I think back of all my code reviews,
49:40
SPEAKER_00: like, of course, I have like memories where like, it was great.
49:42
SPEAKER_00: We had a good discussion or I learned something really interesting.
49:46
SPEAKER_00: But a bunch of times, honestly, it was such a pain in the ass.
49:49
SPEAKER_00: Like, I was trying to get my stuff.
49:51
SPEAKER_00: You're pinged.
49:52
SPEAKER_00: Hey, could you review my code?
49:54
SPEAKER_00: And like, no, right now I'm busy.
49:55
SPEAKER_00: No, I really need this to unblock me.
49:57
SPEAKER_00: And then you context switch.
49:58
SPEAKER_00: And then I feel it's always been like...
50:01
SPEAKER_00: Good and bad, right?
50:02
SPEAKER_00: So I feel whatever we do,
50:04
SPEAKER_00: there will be always upsides and downsides, but now they're just moving.
50:08
SPEAKER_00: So I guess one upside is,
50:10
SPEAKER_00: as an engineer,
50:11
SPEAKER_00: you might have to not give your attention to just kind of basic stuff
50:15
SPEAKER_00: that doesn't need your input, per se.
50:17
SPEAKER_01: Yes, it saves time.
50:19
SPEAKER_01: And progressively what we're going to see is also like you have an agreement on,
50:25
SPEAKER_01: you know, the box and the overall contract of what it's supposed to do.
50:29
SPEAKER_01: And then, you know, what is inside the box, as long as you have like strict guarantees.
50:33
SPEAKER_01: It's like,
50:39
SPEAKER_01: what happens inside a box is, you know, it could be literally anything.
50:42
SPEAKER_01: You just, like, don't really need to care.
50:44
SPEAKER_01: And, like, really what you need to agree on is, like, The cost of maintenance has gone down.
51:09
SPEAKER_00: Maintenance is always such a hot topic whenever we build something inside
51:13
SPEAKER_00: of all these companies like Google, Uber,
51:14
SPEAKER_00: even startups, like building was the fun part, but then maintenance was the painful.
51:19
SPEAKER_00: And that's when we learned like, okay, it was not worth building it, et cetera.
51:23
SPEAKER_00: Inside of Codex and OpenAI,
51:25
SPEAKER_00: what do you see maintenance becoming cheaper changing in terms of instead
51:30
SPEAKER_00: of what you're building,
51:31
SPEAKER_00: what the ambition is, the, I guess, custom tooling, those kinds of things?
51:35
SPEAKER_01: Maintenance is really like sort of like a tower So,
51:38
SPEAKER_01: you know,
51:38
SPEAKER_01: it's like,
51:39
SPEAKER_01: okay, you have, you have this third party dependencies, like you need to upgrade
51:57
SPEAKER_01: You can fully automate this.
51:59
SPEAKER_01: If you have good changelog and the code is well documented and
52:03
SPEAKER_01: the model can just reason through it,
52:06
SPEAKER_01: it can just blast through your code base,
52:08
SPEAKER_01: do it in a couple of hours,
52:10
SPEAKER_01: and previously you would have like Sort of punted on it because it's not
52:13
SPEAKER_01: the most fun thing to do.
52:15
SPEAKER_01: But it's actually really important for your business.
52:17
SPEAKER_01: So it's really important for your project, especially for security vulnerabilities.
52:21
SPEAKER_01: You want to stay up to date, right?
52:22
SPEAKER_01: You want to apply all these patches.
52:26
SPEAKER_01: I think that's just going to be fully automated.
52:28
SPEAKER_01: So a large part of like maintenance, it just kind of comes for free.
52:31
SPEAKER_01: Right.
52:32
SPEAKER_01: And then I think it's awesome to also think about before, like,
52:36
SPEAKER_01: you know,
52:37
SPEAKER_01: when you wanted to just completely react,
52:39
SPEAKER_01: you have to do like a new architecture because you're trying to make space for like
52:44
SPEAKER_01: a new,
52:44
SPEAKER_01: you know,
52:45
SPEAKER_01: different kind of tradeoffs or you have a new understanding of like the workload
52:48
SPEAKER_01: or something.
52:49
SPEAKER_01: You're trying to fit a new feature
52:50
SPEAKER_01: and like suddenly you realize like your current system is just very limiting
52:53
SPEAKER_01: and you need to completely re-architecture it.
52:55
SPEAKER_01: That was like a really, really costly endeavor, right?
52:57
SPEAKER_01: So, you know, sometimes like multiple years.
53:00
SPEAKER_01: And I think this is also like super, super accelerated now.
53:03
SPEAKER_01: So like the cost of mistakes, I would say, is going down.
53:09
SPEAKER_01: But then at the same time,
53:10
SPEAKER_01: the good old rules,
53:12
SPEAKER_01: I would say, of software engineering, having good abstractions really help.
53:16
SPEAKER_01: Going back to having the box with invariants,
53:19
SPEAKER_01: if you draw the right shape,
53:21
SPEAKER_01: you're going to be able to change things much more quickly within the box
53:24
SPEAKER_01: and not affect the rest of the services or the rest of your infrastructure.
53:29
SPEAKER_01: And I think it's important to design for very quick iteration and change.
53:33
SPEAKER_00: I remember when I talked with Peter Steinberger,
53:35
SPEAKER_00: that was before he joined OpenAI, but about OpenClaw and how he thinks about it.
53:40
SPEAKER_00: He told me that he doesn't read the code, but he kept thinking about it.
53:43
SPEAKER_00: I could see that he's holding the architecture in his head
53:46
SPEAKER_00: and he was telling me how he re-architects a lot and he thinks about how
53:49
SPEAKER_00: to make it modular,
53:50
SPEAKER_00: how to allow 100 contributors to each build their thing without stepping
53:55
SPEAKER_00: on each other's toes.
53:56
SPEAKER_00: So I'm hearing what you're saying that this...
54:00
SPEAKER_00: This care,
54:01
SPEAKER_00: this planning,
54:02
SPEAKER_00: this structuring has become maybe just a lot more important to like,
54:07
SPEAKER_00: which was something back in the day,
54:09
SPEAKER_00: you know,
54:09
SPEAKER_00: it was like the architect or the staff engineer
54:12
SPEAKER_00: or experienced folks were doing this thing and other engineers around them were kind
54:16
SPEAKER_00: of building this, you know, smaller parts.
54:17
SPEAKER_00: It sounds like now all engineers need to be aware of when you're building your software, right?
54:22
SPEAKER_00: And plan for it.
54:23
SPEAKER_01: Yeah, and the GPT models are getting better and better at this as well,
54:27
SPEAKER_01: thinking about long-term maintenance and good architecture,
54:30
SPEAKER_01: and this is a natural next step, right?
54:33
SPEAKER_01: It's not just about code quality in the sense of, oh, is this code clean within this file?
54:37
SPEAKER_01: But is the architecture actually correct to reduce maintenance burden over time
54:42
SPEAKER_01: and make space for future product or feature extensions or changes?
54:47
SPEAKER_01: And just really this act of engineering over time,
54:49
SPEAKER_01: that's Kind of like something that models are starting to become capable of thinking
54:54
SPEAKER_01: about very well.
54:54
SPEAKER_01: I think it's just kind of fascinating to understand that the software
55:00
SPEAKER_01: that we're building is just going through the lifecycle much, much faster.
55:03
SPEAKER_01: You know,
55:03
SPEAKER_01: before you had,
55:04
SPEAKER_01: you know,
55:05
SPEAKER_01: you were scaling,
55:05
SPEAKER_01: you were starting it,
55:06
SPEAKER_01: you know,
55:06
SPEAKER_01: maybe as like a small team of,
55:08
SPEAKER_01: you know,
55:08
SPEAKER_01: yourself, maybe a couple of engineers, and then you would add engineers like slowly.
55:13
SPEAKER_01: And then,
55:13
SPEAKER_01: you know,
55:13
SPEAKER_01: maybe after a year,
55:14
SPEAKER_01: you know,
55:15
SPEAKER_01: it's like if it's very,
55:15
SPEAKER_01: very successful, you would have 50 engineers on it or like 100 engineers on it.
55:20
SPEAKER_01: You would have time to see it coming.
55:21
SPEAKER_01: You would have time to see, like, you know, the humans on board.
55:24
SPEAKER_01: And, you know, you can think about the documentation, all of that stuff.
55:27
SPEAKER_01: But now it's just sort of like that explosion of like,
55:30
SPEAKER_01: you know, suddenly you have like 100 agents contributing to this thing.
55:33
SPEAKER_01: It's like, you know, that can happen like, you know, in a weekend.
55:36
SPEAKER_01: And so, you know,
55:37
SPEAKER_01: you're just going through it at, you know, major, major speeds compared to before.
55:41
SPEAKER_00: Okay, but how do you and the folks at OpenAI deal with it?
55:45
SPEAKER_00: Does it not mess with your mind?
55:48
SPEAKER_00: You know what I mean in the sense of you've been in this business
55:50
SPEAKER_00: for quite some time now, like decades or well over.
55:55
SPEAKER_00: And there was a pace that we kind of got used to.
55:57
SPEAKER_00: And obviously it's now a lot faster.
55:59
SPEAKER_00: But how do you get your head around the fact that A, it's faster.
56:03
SPEAKER_00: B, the stuff that you've been doing a year ago,
56:07
SPEAKER_00: right now you're not doing because now the model is good at it.
56:10
SPEAKER_00: And You know, like how do you kind of reconcile that?
56:12
SPEAKER_00: Because I'm sure there's stuff that you've been really good at related to software
56:15
SPEAKER_00: that now you can hand off to the agent.
56:17
SPEAKER_00: Do you not get a little bit of sting?
56:18
SPEAKER_00: You know,
56:19
SPEAKER_00: we talked about it stinging for your features to be implemented open source,
56:22
SPEAKER_00: but it can also sting that I've been really good at, like,
56:24
SPEAKER_00: I don't know,
56:25
SPEAKER_00: refactoring or right now it might be architecture,
56:29
SPEAKER_00: but maybe the model will be really good at that.
56:30
SPEAKER_00: And now I'm like, oh,
56:31
SPEAKER_00: OK, damn, like I'm glad, but also like it would have been nice for me to do that.
56:36
SPEAKER_01: Yeah, I think there's like a craft aspect to it,
56:39
SPEAKER_01: which occasionally I still, you know, pull up an editor and like write some code.
56:44
SPEAKER_01: And it's just like it feels nice.
56:47
SPEAKER_01: And it's sort of like I have fond memories of like late nights sitting in Vim and,
56:54
SPEAKER_01: you know,
56:54
SPEAKER_01: just like cranking it out,
56:56
SPEAKER_01: you know,
56:57
SPEAKER_01: drinking Coke Zero and yeah,
57:02
SPEAKER_01: just not having to think about anything else other than like the problem in front
57:04
SPEAKER_01: of me.
57:05
SPEAKER_01: But really, I think it's all about being in the flow and solving problems.
57:12
SPEAKER_01: And what I find is folks here and also everyone I talk to is just adapting very quickly.
57:18
SPEAKER_01: And I think if you have a mindset where it's all about code as a tool
57:24
SPEAKER_01: to solve problems, and you can solve so many more problems.
57:27
SPEAKER_01: It's like before you wanted to benchmark something and you weren't quite sure
57:31
SPEAKER_01: where you were going to net out.
57:32
SPEAKER_01: It's like, You can just do it.
57:34
SPEAKER_01: It's going to take you no more than 30 seconds to launch something in the background
57:39
SPEAKER_01: and get proper numbers and be able to do a better tradeoff.
57:42
SPEAKER_01: It should make you a better engineer if you just really care about the outcome and
57:49
SPEAKER_01: the system working well.
57:50
SPEAKER_01: And so what it allows us to do at OpenAI,
57:52
SPEAKER_01: it allows us to run our inference much more efficiently.
57:56
SPEAKER_01: It allows us to get much more effective compute and deploy that to the world.
58:02
SPEAKER_01: Everyone's just very focused on that and solving important problems at the speed
58:06
SPEAKER_01: that was not possible before.
58:09
SPEAKER_01: I haven't yet encountered someone who's like, oh, that's not good.
58:13
SPEAKER_01: That's not fun.
58:14
SPEAKER_00: Do I understand correctly that it sounds like if you have ambitious problems,
58:18
SPEAKER_00: if you have way more problems than what you can solve today or tomorrow or
58:21
SPEAKER_00: the next week,
58:22
SPEAKER_00: sounds like this is not really a problem because
58:24
SPEAKER_00: when you get more efficient somewhere,
58:26
SPEAKER_00: you keep going, which is a lot of startups, right?
58:29
SPEAKER_00: Like startups are always way more ambitious than what they're able to do.
58:33
SPEAKER_01: We're not out of problems for sure, right?
58:36
SPEAKER_01: And I don't think we will be for a while.
58:39
SPEAKER_01: We have a long,
58:40
SPEAKER_01: long road ahead of us in terms of mathematical breakthroughs,
58:44
SPEAKER_01: scientific breakthroughs,
58:45
SPEAKER_01: making the world a better place,
58:47
SPEAKER_01: just really building for humans and solving the most important problems
58:52
SPEAKER_01: that everyone is facing and just doing it in a deeply human way.
58:57
SPEAKER_01: That's what we're here for.
58:59
SPEAKER_01: Also, just going back to coding and these late nights,
59:02
SPEAKER_01: I think there's like It's also like maybe like a glamorous version of it.
59:06
SPEAKER_01: It's just like I also had very a lot of late nights where I was trying to refactor something.
59:11
SPEAKER_01: And, you know,
59:12
SPEAKER_01: it's just like it would be like three hours deep into the refactor and
59:14
SPEAKER_01: then realize like, actually, this is a dead end and I must restart from scratch.
59:20
SPEAKER_01: And it was like very frustrating.
59:22
SPEAKER_01: And so it's like there was like there are like these very, very fun times.
59:25
SPEAKER_01: There's also the time where it's like it doesn't compile and you're just like,
59:30
SPEAKER_01: Why is this not compiling yet?
59:32
SPEAKER_00: I'm sure you had the time where you go later.
59:35
SPEAKER_00: It's now super late.
59:36
SPEAKER_00: You need to go to bed because you need to get some sleep.
59:39
SPEAKER_00: And then you can't really sleep.
59:40
SPEAKER_00: And you have this thing where you have some task that is halfway and it upsets you.
59:47
SPEAKER_00: Sometimes I remember dreaming about the code as well.
59:50
SPEAKER_00: And I guess one thing I don't really have these days when I'm working on my software
59:55
SPEAKER_00: for my business,
59:57
SPEAKER_00: I don't really have something that is halfway because I can just do this and
1:00:01
SPEAKER_00: then I can leave it at a state where it's kind of like,
1:00:03
SPEAKER_00: you know,
1:00:03
SPEAKER_00: done, either finished or it's either working or I have proof that it's failed.
1:00:07
SPEAKER_00: But it's interesting because, you know, everything's sped up, right?
1:00:09
SPEAKER_01: Yeah, maybe.
1:00:11
SPEAKER_01: Like I do have like what a lot of people do and I do myself is like,
1:00:15
SPEAKER_01: you know,
1:00:15
SPEAKER_01: I have like sometimes like bigger questions that I'm asking myself like and I,
1:00:19
SPEAKER_01: you know,
1:00:19
SPEAKER_01: from conversations I've had during the day or like I haven't yet,
1:00:22
SPEAKER_01: you know, just like had the time to just look into it.
1:00:25
SPEAKER_01: And so I, you know, I will send off codex to just like look at it overnight.
1:00:30
SPEAKER_01: And then I'm very excited to then wake up and look at the results.
1:00:33
SPEAKER_01: And so, you know, it's always like an exciting morning.
1:00:36
SPEAKER_00: Well, I feel there's an art to doing long running tasks.
1:00:39
SPEAKER_00: And of course, you can use the slash goal, which will go and run.
1:00:42
SPEAKER_00: You know, that's also something that was recently added like a few months ago, right?
1:00:46
SPEAKER_00: The slash goal command to codex.
1:00:47
SPEAKER_01: Yeah.
1:00:48
SPEAKER_01: And back to, you know, maybe like the harness is a crutch, right?
1:00:51
SPEAKER_01: Is a slash goal was like necessary to allow, like,
1:00:57
SPEAKER_01: you know,
1:00:57
SPEAKER_01: to keep the model like on track on like a singular goal for a very long period
1:01:03
SPEAKER_01: of time.
1:01:03
SPEAKER_01: And it's like it allows the model to literally run for days or weeks if it's like
1:01:07
SPEAKER_01: a really, really hard problem.
1:01:09
SPEAKER_01: But with the new generation of models,
1:01:11
SPEAKER_01: like what we're seeing is like, you know, you don't need slash goal anymore.
1:01:14
SPEAKER_01: You don't need a harness around it.
1:01:15
SPEAKER_01: You can just tell the model like,
1:01:16
SPEAKER_01: you know, hey, Go and work for a week and, you know, I will actually do it.
1:01:20
SPEAKER_00: Speaking of hard problems and the fact that you're not out of them,
1:01:23
SPEAKER_00: one of the interesting things that you've shipped from the outside,
1:01:27
SPEAKER_00: I would say it was,
1:01:29
SPEAKER_00: you know,
1:01:29
SPEAKER_00: as an engineer,
1:01:30
SPEAKER_00: it was moderately interesting,
1:01:32
SPEAKER_00: is what you call the merge, which is codecs appeared inside of ChatGPT.
1:01:37
SPEAKER_00: And the reason I say that as engineers,
1:01:39
SPEAKER_00: it's kind of moderately interesting because we've been using codecs like,
1:01:42
SPEAKER_00: yeah, it's there, you can now open it.
1:01:44
SPEAKER_00: In the ChatGPT app, great.
1:01:45
SPEAKER_00: I just went there and I just immediately went to Codex
1:01:47
SPEAKER_00: because I don't really use ChatGPT in the app per se.
1:01:51
SPEAKER_00: But I talked with folks at OpenAI and people on your team
1:01:55
SPEAKER_00: and they were telling me there was a lot of preparation going on,
1:01:59
SPEAKER_00: a lot of engineering challenges.
1:02:00
SPEAKER_00: Can you give a sense of how big this project was,
1:02:03
SPEAKER_00: what you needed to do and why was it difficult to pull off and how did Codex
1:02:09
SPEAKER_00: and other tools help you get it done in ways that would have been hard before?
1:02:14
SPEAKER_00: Since you've launched the merge,
1:02:16
SPEAKER_00: the numbers that you keep sharing of how many people use codecs,
1:02:19
SPEAKER_00: it's like, it's going up way faster than before.
1:02:22
SPEAKER_00: So I assume there's a big scale problem you've solved here.
1:02:25
SPEAKER_01: A lot of things were challenging with the merge is, first of all, Completely different stacks.
1:02:34
SPEAKER_00: ChatGPT is fully
1:02:37
SPEAKER_01: managed, cloud-based.
1:02:40
SPEAKER_01: You run everything on our systems.
1:02:43
SPEAKER_01: We store things.
1:02:44
SPEAKER_01: Traditional way of building things, built for scale, built for efficiency.
1:02:52
SPEAKER_01: Codex, fully local.
1:02:54
SPEAKER_01: And so the merge is just really like,
1:02:56
SPEAKER_01: how do you get the same The same benefits and the same capabilities
1:03:01
SPEAKER_01: from this local coding agent and then build a product around it and build it in
1:03:05
SPEAKER_01: a way where it can benefit like a much,
1:03:07
SPEAKER_01: much broader pool of people,
1:03:10
SPEAKER_01: which is also why all of us joined OpenAI is like to benefit like this very,
1:03:16
SPEAKER_01: very broad population across the world.
1:03:19
SPEAKER_01: And so it was like a very exciting journey of like figuring out like how do we build
1:03:24
SPEAKER_01: a cloud version of this that in essence is capable of like very,
1:03:28
SPEAKER_01: very much the same things.
1:03:30
SPEAKER_01: But it's also built in a way where,
1:03:32
SPEAKER_01: you know,
1:03:32
SPEAKER_01: we can serve it through like tens and hundreds of millions of users in a way
1:03:37
SPEAKER_01: that is still like,
1:03:38
SPEAKER_01: you know, efficient so that we can include it all the way into the plus plan.
1:03:42
SPEAKER_01: ChallengeGPT work is essentially running the full codex harness
1:03:50
SPEAKER_01: together with a cloud computer.
1:03:52
SPEAKER_01: It's a very powerful machine, actually.
1:03:55
SPEAKER_01: People have picked up on it and showed what you can do.
1:03:59
SPEAKER_01: If you are creative with the prompt, you can get ChallengeGPT to train another model in there.
1:04:06
SPEAKER_01: Wow.
1:04:08
SPEAKER_01: There are some really wild things.
1:04:10
SPEAKER_01: You can get it to install Blender and do 3D modeling.
1:04:15
SPEAKER_01: It's very permissive.
1:04:16
SPEAKER_01: It has internet access.
1:04:18
SPEAKER_01: It's a powerful machine.
1:04:19
SPEAKER_01: And then Codex just works on it.
1:04:21
SPEAKER_01: And this is what we shipped through Casualty Work.
1:04:24
SPEAKER_01: So a lot of system challenges, the team did it very quickly.
1:04:28
SPEAKER_01: Obviously,
1:04:29
SPEAKER_01: Codex helped to make it more efficient,
1:04:32
SPEAKER_01: look at and build a lot of the infrastructure.
1:04:36
SPEAKER_01: And then,
1:04:36
SPEAKER_01: you know,
1:04:37
SPEAKER_01: help resolve a lot of the little differences as well that,
1:04:40
SPEAKER_01: you know, had been occurring between Codex and Chattapiti.
1:04:44
SPEAKER_01: Like merging plugins, architecture, you know, merging library.
1:04:48
SPEAKER_01: And like sort of like really,
1:04:49
SPEAKER_01: really working towards like unified system, which is really the goal.
1:04:52
SPEAKER_01: It's like you shouldn't feel like,
1:04:53
SPEAKER_01: you know,
1:04:53
SPEAKER_01: you can do something in Codex that you can't do in Chattapiti or vice versa.
1:04:57
SPEAKER_01: Like what we're trying to build is like one unified product that gives you access to
1:05:01
SPEAKER_01: the same intelligence, but in the way that you want to use it.
1:05:04
SPEAKER_01: And so it was very fun as well because Codex throughout the whole journey also acted
1:05:09
SPEAKER_01: as a journalist to sort of like document all the steps and the debates and
1:05:13
SPEAKER_01: the discussions that the teams were having.
1:05:15
SPEAKER_01: And it was very animated debate,
1:05:17
SPEAKER_01: you know,
1:05:17
SPEAKER_01: of how we should do it and how we should name the thing and,
1:05:20
SPEAKER_01: you know, when to introduce it in what way and like what to merge into what.
1:05:24
SPEAKER_01: There were like many different permutations considered.
1:05:27
SPEAKER_01: And so there's like a very fun journalistic element to it where we have
1:05:31
SPEAKER_01: a full recounting that Codex did over time.
1:05:34
SPEAKER_01: And yeah,
1:05:36
SPEAKER_01: it's just,
1:05:37
SPEAKER_01: it's kind of become known as well as like the toggle arc of OpenAI where,
1:05:42
SPEAKER_01: you know,
1:05:42
SPEAKER_01: we introduced like the work toggle,
1:05:45
SPEAKER_01: which there was also a lot of debate around of like,
1:05:47
SPEAKER_01: you know, whether this was like the right thing.
1:05:48
SPEAKER_01: And then, you know, it's just like, we kind of grew to just really like it.
1:05:52
SPEAKER_01: But over time, We're going to merge things further.
1:05:55
SPEAKER_01: So it's like we're really headed into the direction of like full unification.
1:05:59
SPEAKER_01: And, you know,
1:06:00
SPEAKER_01: we kind of view this as like a temporary state where,
1:06:04
SPEAKER_01: you know,
1:06:04
SPEAKER_01: you have like, you have better, stronger capabilities when you're in work mode.
1:06:09
SPEAKER_01: But over time,
1:06:10
SPEAKER_01: we're bringing this,
1:06:11
SPEAKER_01: you know, all the way to like, you know, everyone that uses ChancePG.
1:06:15
SPEAKER_00: And how do you personally use Codex?
1:06:18
SPEAKER_00: Like, what's your working setup in terms of agents,
1:06:21
SPEAKER_00: in terms of tasks, in terms of what you manage with it?
1:06:25
SPEAKER_00: And related to this, I asked Peter Steinberger what I should ask about you.
1:06:29
SPEAKER_00: And he said, like,
1:06:30
SPEAKER_00: you need to ask him,
1:06:32
SPEAKER_00: how do you deal with the fact that you're involved with all these projects?
1:06:37
SPEAKER_00: Your calendar is like Tetris, but usually you show up pretty cheerful.
1:06:42
SPEAKER_01: My calendar is...
1:06:43
SPEAKER_01: Fine.
1:06:47
SPEAKER_01: And it's just,
1:06:49
SPEAKER_01: I am capable of doing so many more things nowadays because I have
1:06:54
SPEAKER_01: a technology like Codex and I actually shifted a lot of like my work
1:06:59
SPEAKER_01: on mobile using ChatGP3 work.
1:07:03
SPEAKER_01: Where whenever I have something that I want to take note of, I just like fire that off.
1:07:09
SPEAKER_01: I use dictation a lot.
1:07:11
SPEAKER_01: Whenever I have a question,
1:07:12
SPEAKER_01: instead of like writing it down to look into later or delegating to someone,
1:07:16
SPEAKER_01: I just like fire it off in touch of your work and I get like a report.
1:07:20
SPEAKER_01: It has like a whole bunch of like custom skills and Custom instructions
1:07:26
SPEAKER_01: where it's now like very,
1:07:27
SPEAKER_01: very tailored to like,
1:07:28
SPEAKER_01: you know,
1:07:28
SPEAKER_01: produce the kinds of reports and slide decks and code explorations,
1:07:32
SPEAKER_01: you know, in the style that I can consume effectively.
1:07:36
SPEAKER_01: And so every time I'm like between meetings or like,
1:07:37
SPEAKER_01: you know,
1:07:38
SPEAKER_01: you'll kind of like see me like, you know, I was just like dictating to my phone.
1:07:41
SPEAKER_01: As I said before, it's just like, We do a lot of work in public channels.
1:07:45
SPEAKER_01: We have a lot in Slack.
1:07:47
SPEAKER_01: We have a lot in Notion and Google Docs as well.
1:07:51
SPEAKER_01: And so there's pretty much no question really that I feel I cannot ask
1:07:54
SPEAKER_01: that Codex will be able to do at least a first pass of thinking through,
1:07:58
SPEAKER_01: whether it is public sentiment on a feature,
1:08:02
SPEAKER_01: looking at production logs for You know,
1:08:05
SPEAKER_01: how much usage we have on a certain thing,
1:08:07
SPEAKER_01: making a list of things that we should deprecate
1:08:09
SPEAKER_01: because they're not getting traction, understanding what a certain team is up to.
1:08:13
SPEAKER_01: It's like any question I have, I can get an answer to, you know, within 30 minutes.
1:08:17
SPEAKER_01: And so that's how I use it.
1:08:18
SPEAKER_01: I use it for everything.
1:08:19
SPEAKER_01: It's like my personal agent in like all the ways.
1:08:24
SPEAKER_01: And oftentimes on weekends as well,
1:08:27
SPEAKER_01: I do some like code explorations or like I built some prototypes and I have fun,
1:08:31
SPEAKER_01: like sort of like imagining the future of the product in some ways.
1:08:35
SPEAKER_01: And I do that with others on the teams.
1:08:37
SPEAKER_01: It's not always the same team.
1:08:39
SPEAKER_01: And it's just like in one day,
1:08:41
SPEAKER_01: I can build things that I sort of like, I had it in my system, right?
1:08:45
SPEAKER_01: It's like, I woke up one day, I'm just like, we should explore what it means to build this.
1:08:50
SPEAKER_01: And then I can just sort of express all of that and get like something in front
1:08:54
SPEAKER_01: of people in a day so that they can think through it and criticize it
1:08:58
SPEAKER_01: and hopefully get inspired by it.
1:08:59
SPEAKER_01: It's like,
1:09:00
SPEAKER_01: by no means,
1:09:01
SPEAKER_01: you know,
1:09:01
SPEAKER_01: we need to ship it,
1:09:02
SPEAKER_01: but it's more like, okay,
1:09:03
SPEAKER_01: I flush it out of my system and then,
1:09:05
SPEAKER_01: you know, I go on and like, you know, do other things.
1:09:07
SPEAKER_01: So it's just like so, I don't know, it's such a magical time and it's like so empowering.
1:09:12
SPEAKER_00: And as closing,
1:09:13
SPEAKER_00: what would your advice be for a software engineer such as AI engineer,
1:09:16
SPEAKER_00: someone who builds software,
1:09:18
SPEAKER_00: who would want to get the skill set and the experience to have the opportunity
1:09:23
SPEAKER_00: to work at a place like the Codex team, like OpenAI or like an AI startup.
1:09:27
SPEAKER_00: So like, you know, just become this really great builder with these tools.
1:09:33
SPEAKER_00: Because the question that comes up is often like, should I start with the theory?
1:09:37
SPEAKER_00: How important are the basics?
1:09:38
SPEAKER_00: Should I just get really good at using the tools?
1:09:40
SPEAKER_01: Yeah.
1:09:41
SPEAKER_01: I think there are two things that are important is a deep,
1:09:45
SPEAKER_01: deep curiosity for how things work and an ability to train yourself
1:09:50
SPEAKER_01: to understand things very quickly.
1:09:51
SPEAKER_01: And so it is the case that things will continue to change, but people that do experience it.
1:09:56
SPEAKER_01: I think a lot of people that work internally well at OpenAI are people that are able
1:10:00
SPEAKER_01: to grok a system quickly and also dive into a new code base and make sense of it.
1:10:09
SPEAKER_01: But obviously, all of that has helped with agents nowadays.
1:10:14
SPEAKER_01: There's so much information that you need to absorb and being able to understand
1:10:17
SPEAKER_01: and reason through it.
1:10:18
SPEAKER_01: A lot of that is asking good questions.
1:10:20
SPEAKER_01: Really about, you know, how do things work?
1:10:22
SPEAKER_01: And just like going into like the five whys,
1:10:24
SPEAKER_01: which I think, you know, you can just kind of keep digging and digging and digging.
1:10:27
SPEAKER_01: And, you know, you're learning very, very fast through that.
1:10:30
SPEAKER_01: The other thing is being in tune with the community or,
1:10:34
SPEAKER_01: you know, the people that you're trying to solve a problem for.
1:10:36
SPEAKER_01: It's like not everything is like solving a direct problem.
1:10:39
SPEAKER_01: Sometimes you're solving a problem that will be useful,
1:10:41
SPEAKER_01: you know,
1:10:41
SPEAKER_01: to like another group of people in the pursuit of like solving a problem for humans.
1:10:45
SPEAKER_01: But just being crisp about it.
1:10:47
SPEAKER_01: The taste or the needs or the requirements and being able to think clearly and like,
1:10:52
SPEAKER_01: you know, exercising through this clarity of thought feels really important to me.
1:10:55
SPEAKER_01: Like if you can't explain what you're trying to achieve, you can't explain your intent.
1:10:59
SPEAKER_01: If you don't have a tie to a community,
1:11:01
SPEAKER_01: if you don't have the taste, it's going to be much
1:11:04
SPEAKER_00: harder to do great work.
1:11:06
SPEAKER_00: Awesome, Thibaut.
1:11:07
SPEAKER_00: Well, thanks a bunch for this conversation.
1:11:09
SPEAKER_00: This was awesome.
1:11:09
SPEAKER_00: Thanks for having me.
1:11:11
SPEAKER_00: I've always wanted to get together with Thibaut and I'm glad that we finally made it happen.
1:11:15
SPEAKER_00: I appreciated how Tebow talked about not just the upsides of open source,
1:11:19
SPEAKER_00: but also the downsides.
1:11:20
SPEAKER_00: Most notably,
1:11:21
SPEAKER_00: how competitors can copy features you are just working on in the open right now and
1:11:25
SPEAKER_00: then ship it right before release and just how much this stings.
1:11:29
SPEAKER_00: Plus, you get a lot of low quality contributions that you still need to somehow deal with.
1:11:33
SPEAKER_00: Another interesting one was Tebow saying how the harness is always a step ahead of the model.
1:11:38
SPEAKER_00: From the inside,
1:11:38
SPEAKER_00: the Codex team see their job as building clutches for the model with the harness,
1:11:43
SPEAKER_00: the tools, and the setup instruction.
1:11:45
SPEAKER_00: And then the next version of the model will be trained to need fewer of these clutches.
1:11:49
SPEAKER_00: I'll be honest,
1:11:50
SPEAKER_00: as a dev,
1:11:50
SPEAKER_00: this sounds a little demotivating that the The stuff I build,
1:11:54
SPEAKER_00: in the next version of the model, it'll just know and we can get rid of it.
1:11:58
SPEAKER_00: Plus, I do suspect that it's not just about building these clutches,
1:12:02
SPEAKER_00: but also building tools that models will use,
1:12:05
SPEAKER_00: and it's not like the next version of the model will reinvent an MCP protocol
1:12:09
SPEAKER_00: or scales or plugins, at least I hope not.
1:12:11
SPEAKER_00: I also enjoyed hearing what the merge, merging ChatGPC and Codex, looked like from the inside.
1:12:17
SPEAKER_00: It was merging a previously fully local coding agent, Codex, into a managed cloud-based stack.
1:12:23
SPEAKER_00: And doing it efficient enough so that it can be included
1:12:25
SPEAKER_00: in OpenAI's $20 per month plan when $20 is not all that much in terms
1:12:30
SPEAKER_00: of compute purchase.
1:12:31
SPEAKER_00: It was pretty amusing to hear how Kodak itself acted as a journalist of
1:12:35
SPEAKER_00: the whole project as it was present in all the Slack conversations and all
1:12:38
SPEAKER_00: the documents and so it could capture all the important debates and decisions.
1:12:42
SPEAKER_00: I'm not going to lie,
1:12:43
SPEAKER_00: this part felt a little bit of a Big Brother feel to it,
1:12:47
SPEAKER_00: where the AI is always watching,
1:12:48
SPEAKER_00: but it could well become the new normal in startups in the future.
1:12:51
SPEAKER_00: I have not yet decided how I feel about this.
1:12:54
SPEAKER_00: And finally, I appreciated Thibault's advice for engineers to success.
1:12:57
SPEAKER_00: Be curious, understand symptoms quickly, and be in tune with the group you are building for.
1:13:03
SPEAKER_00: It's reassuring to hear from Tebow as well how much the fundamentals still matter.
1:13:07
SPEAKER_00: Do check out the show notes below for deep dives on how Codex,
1:13:10
SPEAKER_00: Clockcode, and Cursor were built, and other related topics.
1:13:13
SPEAKER_00: If you like what you heard, please hit a rating on a podcast player that you're using.
1:13:17
SPEAKER_00: It means a lot to me and to the show.
1:13:19
SPEAKER_00: Thanks, and I'll see you in the next one.