Latest / What's Your Problem? / The AI Pioneer Developing New Kinds of Medicine

Transcript

The AI Pioneer Developing New Kinds of Medicine · What's Your Problem?

• 0:15
  Speaker 1: Pushkin.
• 0:20
  Speaker 2: If I were going to pick one paper from the
• 0:22
  Speaker 2: past decade that had the biggest impact on the world,
• 0:25
  Speaker 2: I would choose one called Attention Is All You Need,
• 0:28
  Speaker 2: published in twenty seventeen. That paper basically invented transformer models.
• 0:35
  Speaker 2: You've almost certainly used a transformer model if you have
• 0:39
  Speaker 2: used chat GPT or Gemini or Claude or deep Seek.
• 0:42
  Speaker 2: In fact, the tea in chat GPT stands for a transformer,
• 0:48
  Speaker 2: and transformer models have turned out to be wildly useful,
• 0:51
  Speaker 2: not just at generating language, but also at everything from
• 0:54
  Speaker 2: generating images to predicting what proteins will look like.
• 0:58
  Speaker 1: In fact, transformers.
• 1:00
  Speaker 2: Are so ubiquitous and so powerful that it's easy to
• 1:03
  Speaker 2: forget that some guy just thought them up.
• 1:06
  Speaker 1: But in fact, some guy did.
• 1:08
  Speaker 2: Just think up transform and I'm talking to him today
• 1:11
  Speaker 2: on the show. I'm Jacob Goldstein and this is What's
• 1:19
  Speaker 2: Your Problem, the show where I talk to people who
• 1:21
  Speaker 2: are trying to make technological progress. My guest today is
• 1:25
  Speaker 2: Yakub Uskolai. And just to be clear, Yakub was one
• 1:28
  Speaker 2: of several co authors on that transformer paper, and on
• 1:31
  Speaker 2: top of that, lots of other researchers were working on
• 1:34
  Speaker 2: related things at the same time, so a lot of
• 1:36
  Speaker 2: people were working on this, but the key idea did
• 1:39
  Speaker 2: seem to come from Yakub. Today, Yakub is the CEO
• 1:44
  Speaker 2: of Inceptive. That's a company that he co founded to
• 1:47
  Speaker 2: use AI to develop new kinds of medicine, and the
• 1:51
  Speaker 2: company is particularly focused on RNA. We talked about his
• 1:54
  Speaker 2: work at Inceptive in the second part of our conversation.
• 1:58
  Speaker 2: In the first part, we talked about his work on
• 2:00
  Speaker 2: transformer models. At the time he started working on the
• 2:04
  Speaker 2: idea for transformers, this is around a decade ago now,
• 2:07
  Speaker 2: there were a couple of big problems with existing language models.
• 2:11
  Speaker 2: For one thing, they were slow. They were in fact
• 2:13
  Speaker 2: so slow that they could not even keep up with
• 2:16
  Speaker 2: all the new training data that was becoming available. A
• 2:20
  Speaker 2: second problem, they struggled with what are called long range dependencies.
• 2:25
  Speaker 2: Basically in language, that's relationships between words that are far
• 2:29
  Speaker 2: apart from each other in a sentence. So to start,
• 2:32
  Speaker 2: I asked Yakab for an example we could use to
• 2:34
  Speaker 2: discuss these problems and also how he came up with
• 2:37
  Speaker 2: his big idea for how to solve them. So, pick
• 2:41
  Speaker 2: a sentence that's going to be a good object lesson
• 2:43
  Speaker 2: for us.
• 2:44
  Speaker 1: Okay, so we could have the frog didn't cross the
• 2:47
  Speaker 1: road because it was too tired. Okay, so we got
• 2:50
  Speaker 1: our sentence. Yep.
• 2:52
  Speaker 2: How would the sort of big, powerful but slow to
• 2:55
  Speaker 2: train algorithm in twenty fifteen.
• 2:59
  Speaker 1: Have processed that sentence? So basically it would have walked
• 3:02
  Speaker 1: through that sentence word by word, and so it would
• 3:06
  Speaker 1: walk through the sentence left to right. The frog did
• 3:12
  Speaker 1: not cross the road because it was too tired.
• 3:15
  Speaker 2: Which is logical, which is how I would think a
• 3:17
  Speaker 2: system would work.
• 3:18
  Speaker 1: It's more or less how we read, right, it's how
• 3:21
  Speaker 1: we read, but it's not necessarily how we understand. Uh huh.
• 3:25
  Speaker 1: That is actually one of the integral I would say
• 3:29
  Speaker 1: for what we then how we then went about trying
• 3:31
  Speaker 1: to speak us all up?
• 3:32
  Speaker 2: Well, I love that. I want you to say more
• 3:34
  Speaker 2: about it. When you say it's not how we understand,
• 3:37
  Speaker 2: what do you mean?
• 3:38
  Speaker 1: So? On one hand, right linearity of time forces us
• 3:43
  Speaker 1: to almost always feel that we're communicating language in order
• 3:48
  Speaker 1: and just linearly. It actually turns out that that's not
• 3:53
  Speaker 1: really how we read, not even in terms of our secades,
• 3:56
  Speaker 1: in terms of our em movements. We actually do jump
• 3:59
  Speaker 1: back and forth quite a bit while reading, and if
• 4:01
  Speaker 1: you look at conversations, you also have highly nonlinear elements
• 4:06
  Speaker 1: where there's repetition, there's reference, there's basically different flavors of interruption.
• 4:11
  Speaker 1: But sure, by and large right, we would say we
• 4:14
  Speaker 1: certainly right them left to right right. So if you
• 4:17
  Speaker 1: write a proper text, you don't write it as you
• 4:20
  Speaker 1: would read it, and you also don't write it as
• 4:22
  Speaker 1: you would talk about it. You do write it in
• 4:24
  Speaker 1: one linear order. Now, as we read this and as
• 4:28
  Speaker 1: we understand this, we actually form groups of words that
• 4:34
  Speaker 1: then form meaning. Right. So an example of that is
• 4:38
  Speaker 1: you know adjective noun, right, it's or say, in this
• 4:42
  Speaker 1: case an article noun, it's not a frog, it's the frog. Right.
• 4:45
  Speaker 1: We could have also said it's the green frog or
• 4:49
  Speaker 1: the lazy frog.
• 4:50
  Speaker 2: Right. Language has a structure, right, and there things can
• 4:54
  Speaker 2: modify other things, and things can modify the modifiers exactly exactly.
• 4:58
  Speaker 1: But the interesting thing now is that structure in as
• 5:01
  Speaker 1: a as a tree structured clean hierarchy, only tells you
• 5:05
  Speaker 1: half the story. There's so many exceptions where statistical dependencies,
• 5:10
  Speaker 1: where modification actually happens at a distance.
• 5:14
  Speaker 2: So okay, So just to bring this back to your
• 5:16
  Speaker 2: sample sentence, The frog didn't cross the road because it
• 5:19
  Speaker 2: was too tired. That word it is actually quite far
• 5:22
  Speaker 2: from the word frog. And if you're an AI going
• 5:25
  Speaker 2: from left to right, you may well get confused there, right,
• 5:28
  Speaker 2: You may think it refers to road instead of to frog.
• 5:34
  Speaker 2: So this is one of the problems you were trying
• 5:37
  Speaker 2: to solve. And then the other one you were mentioning before,
• 5:39
  Speaker 2: which is these models were just slow because after each word,
• 5:43
  Speaker 2: the model just recalculates what everything means, and that just
• 5:47
  Speaker 2: takes a long time.
• 5:48
  Speaker 1: They can't go fast enough exactly. It takes a long time,
• 5:51
  Speaker 1: and it doesn't play to the strengths of the computers,
• 5:55
  Speaker 1: of the accelerators that we're using there.
• 5:57
  Speaker 2: And when you say accelerators, I know Google has their
• 5:59
  Speaker 2: own chips, but basically we mean GPUs.
• 6:02
  Speaker 1: Now right, we mean GPUs, We mean.
• 6:04
  Speaker 2: The chips that Nvidia sells. What is the nature of.
• 6:08
  Speaker 1: Those particular ships. Yeah, So the nature of those particular
• 6:10
  Speaker 1: chips is that instead of doing a broad variety of
• 6:16
  Speaker 1: complex computations in sequence, they are incredibly good. They excel
• 6:23
  Speaker 1: at performing many, many, many simple computations in parallel. And
• 6:27
  Speaker 1: so what this hierarchical or semi hierrachical nature of language
• 6:32
  Speaker 1: enables you to do is instead of having, so to speak,
• 6:38
  Speaker 1: one place where you read the current word, you could
• 6:42
  Speaker 1: now imagine you actually read every You look at everything
• 6:46
  Speaker 1: at the same time, and you apply many simple operations
• 6:52
  Speaker 1: at the same time to each position in your sentence.
• 6:55
  Speaker 2: Huh So this is the big idea, I just want
• 6:58
  Speaker 2: to because this is it, right, this is the breakthrough happening. Yes,
• 7:02
  Speaker 2: it's basically, what if instead of reading the sentence one
• 7:05
  Speaker 2: word at a time from left to right, we read
• 7:08
  Speaker 2: the whole thing all at once.
• 7:10
  Speaker 1: All at once. And now the problem is clearly something's
• 7:13
  Speaker 1: got to give, right, so there's no fore lunch in
• 7:17
  Speaker 1: that sense. You have to now simplify what you can
• 7:20
  Speaker 1: do at every position when you do this all in parallel,
• 7:24
  Speaker 1: but you can now afford to do this a bunch
• 7:26
  Speaker 1: of times after another and revise it over time or
• 7:29
  Speaker 1: over these steps. And so instead of walking through the
• 7:32
  Speaker 1: sentence from beginning to end, whether an average sentence has
• 7:35
  Speaker 1: like twenty words or so average sentence in pros, instead
• 7:39
  Speaker 1: of walking those twenty positions, what you're doing is you're
• 7:41
  Speaker 1: looking at every word at the same time, but in
• 7:46
  Speaker 1: a simpler way. But now you can do that maybe
• 7:49
  Speaker 1: five or six times, revising your understanding, and that turns
• 7:53
  Speaker 1: out is faster, way faster on GPUs and because of
• 7:59
  Speaker 1: this hierarchical nature of language, it's also better.
• 8:02
  Speaker 2: So you have this idea, and as I read the
• 8:06
  Speaker 2: little note on the paper, it was in fact your idea.
• 8:08
  Speaker 2: I know you were working with a t but the
• 8:09
  Speaker 2: paper credits you with the idea. So let's let's take
• 8:12
  Speaker 2: this idea, this basic idea of look at the whole
• 8:15
  Speaker 2: input sentence all at once, yep, a few times, and
• 8:19
  Speaker 2: apply it to our frog sentence. Give me, give me
• 8:22
  Speaker 2: that frog sentence again.
• 8:23
  Speaker 1: The frog did not cross the road because it was
• 8:26
  Speaker 1: too tired. Good.
• 8:28
  Speaker 2: Tired is good because that's unambiguous.
• 8:30
  Speaker 1: Hot could be either one. It could be the road
• 8:31
  Speaker 1: or the frog, right, Hot could be hot could be
• 8:33
  Speaker 1: the one exactly is in fact hot could either could
• 8:36
  Speaker 1: actually either one and non referential and non referential because
• 8:40
  Speaker 1: it was too hot outside.
• 8:41
  Speaker 2: Outside it could be any of three things, the weather,
• 8:44
  Speaker 2: or the frog or the road exactly. I love that
• 8:47
  Speaker 2: tired solves the problem. So your model, this new way
• 8:52
  Speaker 2: of doing things, how does it parse that sentence, what
• 8:57
  Speaker 2: does it do?
• 8:58
  Speaker 1: So basically, let's look at the word it and look
• 9:02
  Speaker 1: at it in every single step of these you know,
• 9:05
  Speaker 1: say a handful of times repeated operation. Imagine you're looking
• 9:10
  Speaker 1: at this word it, that's the one that you are
• 9:12
  Speaker 1: now trying to understand better, and you now compare it
• 9:16
  Speaker 1: to every other word in the sense. Okay, so you
• 9:18
  Speaker 1: compare it to the to frog that did not cross
• 9:22
  Speaker 1: the road because two and tired, there was two and
• 9:28
  Speaker 1: tire and initially in the first past. Already a very
• 9:36
  Speaker 1: simple insight the model can fairly easily learn is that
• 9:40
  Speaker 1: it could be strongly informed by frog, by road, by nothing,
• 9:50
  Speaker 1: but not so by two or by the or maybe
• 9:54
  Speaker 1: only to a certain extent by us. But if you
• 9:56
  Speaker 1: want to know more about what it denotes, then it
• 10:00
  Speaker 1: could be, you know, it could be informed by by
• 10:03
  Speaker 1: all of these.
• 10:04
  Speaker 2: And just to be clear, that sort of understanding arises
• 10:08
  Speaker 2: because it has trained in.
• 10:09
  Speaker 1: This way on lots of data.
• 10:11
  Speaker 2: It's encountering a new sentence after reading lots of other
• 10:15
  Speaker 2: sentences with lots of pronouns with different possible antecedents.
• 10:19
  Speaker 1: Yeah, exactly, exactly. So Now the interesting thing is that
• 10:23
  Speaker 1: which of the two it actually refers to, doesn't depend
• 10:30
  Speaker 1: on only on what those other two words are. And
• 10:33
  Speaker 1: this is why you need these subsequent steps because so
• 10:36
  Speaker 1: let's talk with the first step. So what now happens
• 10:39
  Speaker 1: is that, say the model identifies frog and road could
• 10:44
  Speaker 1: have a lot to do with the word it. So
• 10:47
  Speaker 1: now you basically copy some information from both frog and
• 10:51
  Speaker 1: road over to it, and you don't just copy it,
• 10:55
  Speaker 1: you kind of transform it also on the way, but
• 10:59
  Speaker 1: you refine your understanding of it. And this is all learned,
• 11:02
  Speaker 1: does not given by rules or you know, in any
• 11:06
  Speaker 1: way pre specifying.
• 11:07
  Speaker 2: Right, just by training on loge, just by training this emergency,
• 11:11
  Speaker 2: and so that sort of the meaning of it after
• 11:13
  Speaker 2: this first step is kind of influenced by both frog
• 11:17
  Speaker 2: and road.
• 11:18
  Speaker 1: Yes, both frog and road. Okay, so now we repeat
• 11:23
  Speaker 1: this operation again and we now know that it is
• 11:27
  Speaker 1: unsure or the model basically now has this kind of superposition. Right,
• 11:30
  Speaker 1: it could be road, it could be frog. But now
• 11:34
  Speaker 1: in the next step it also looks at tired, and
• 11:36
  Speaker 1: somehow the model has learned that when it means something inanimate,
• 11:41
  Speaker 1: that tired is not the thing. And so maybe in
• 11:46
  Speaker 1: context of tired, it is more likely to refer to frog,
• 11:50
  Speaker 1: and now you know, well, it is more likely and
• 11:54
  Speaker 1: now maybe the model has figured out already, maybe needs
• 11:57
  Speaker 1: a bit more, a few more iterations that it is
• 12:00
  Speaker 1: most likely to refer to frog because of the presence
• 12:04
  Speaker 1: of tired. So it has solved the problem. But it
• 12:07
  Speaker 1: has solved the problem.
• 12:08
  Speaker 2: So you do, you have this idea, you try it out.
• 12:12
  Speaker 2: There's a detail that you mentioned that's kind of fun,
• 12:15
  Speaker 2: and we kind of skipped it, but you mentioned that
• 12:17
  Speaker 2: another one of the co authors, who has also gone
• 12:20
  Speaker 2: on to do very big things, was about to leave
• 12:22
  Speaker 2: Google when you sort of want to test this idea,
• 12:25
  Speaker 2: and and that fact that he was about to leave
• 12:27
  Speaker 2: Google was actually important to the history of this idea.
• 12:29
  Speaker 1: Tell me about that it was important. So this Ilia Plususian,
• 12:34
  Speaker 1: he was at the time that this started to gain
• 12:39
  Speaker 1: any kind of speed, Elia was managing a good chunk
• 12:44
  Speaker 1: of my organization. And the moment he really made the
• 12:47
  Speaker 1: decision to leave the company, he had to wait ultimately
• 12:51
  Speaker 1: for his co for his co founder, and for them
• 12:55
  Speaker 1: to then actually get going together in earnest and so
• 12:58
  Speaker 1: he had a few months where he knew and I
• 13:00
  Speaker 1: also knew that he was about to leave and where
• 13:04
  Speaker 1: you know, the right thing would of course be to
• 13:06
  Speaker 1: transition his team to another manager, which we did immediately,
• 13:11
  Speaker 1: but where you then suddenly was in a position of
• 13:14
  Speaker 1: having nothing to lose and yet quite some time left
• 13:18
  Speaker 1: to play with Google's resources and do cool stuff with interesting,
• 13:21
  Speaker 1: interesting people. And and so that's one of those moments
• 13:25
  Speaker 1: where suddenly your appetite for risk as a researcher just spikes, right, huh,
• 13:32
  Speaker 1: because you have, for for a few more months, you
• 13:34
  Speaker 1: have these resources at your disposal, you've transitioned your responsibilities.
• 13:38
  Speaker 1: At that stage, you're just like, Okay, let's try this
• 13:41
  Speaker 1: crazy shit and and and it's and that's literally in
• 13:46
  Speaker 1: so many ways, was was one of the integral catalysts
• 13:50
  Speaker 1: because that also enabled, right, this kind of mindset of
• 13:55
  Speaker 1: we're going for this now, whatever the reason. It still
• 13:58
  Speaker 1: you know affects other people. And so there were others
• 14:02
  Speaker 1: who joined that collaboration really really early on, who I
• 14:06
  Speaker 1: feel were much more excited a result, much more likely
• 14:10
  Speaker 1: to really work on this and to really give it
• 14:12
  Speaker 1: there all because of his you know, nothing left to lose,
• 14:17
  Speaker 1: I'm going to go for this attitude at this.
• 14:19
  Speaker 2: Point, Right, was there a moment when you realized it worked.
• 14:23
  Speaker 1: There were actually a few moments. And it's interesting because
• 14:29
  Speaker 1: on one hand, right, it's a very gradual thing, right,
• 14:32
  Speaker 1: And initially, actually it took us many months to get
• 14:35
  Speaker 1: to the point where we saw significant first signs of
• 14:39
  Speaker 1: life of this not just being a curiosity but really
• 14:41
  Speaker 1: being something that would end up being competitive. So there
• 14:45
  Speaker 1: certainly was a moment when that started. There was another
• 14:48
  Speaker 1: moment when we for the for the first time had
• 14:51
  Speaker 1: one machine translation challenge, one language pair of the W
• 14:57
  Speaker 1: and T task as it's called, where our score, our
• 15:00
  Speaker 1: model performed better than any other single model. The point
• 15:05
  Speaker 1: in time when I think all of us realized this
• 15:07
  Speaker 1: is special was when we not only had the best
• 15:12
  Speaker 1: one in one of these tasks, but in multiple and
• 15:17
  Speaker 1: we didn't just have the best number. We also at
• 15:19
  Speaker 1: that point were able to establish that we've gotten there
• 15:22
  Speaker 1: with about ten times less energy or training compute spend.
• 15:27
  Speaker 2: Wow, So you do one tenth the work and you
• 15:30
  Speaker 2: get a better result.
• 15:31
  Speaker 1: One tenth the work and you get a better result
• 15:33
  Speaker 1: not just across one specific challenge, but across multiple including
• 15:37
  Speaker 1: the hardest or of one of the harder ones. Right.
• 15:40
  Speaker 1: And then at that stage we were still improving rapidly,
• 15:46
  Speaker 1: and then you realize, okay, this is for real. There's
• 15:51
  Speaker 1: because there right, It wasn't like we it wasn't that
• 15:53
  Speaker 1: we had to squeeze those last little bits and pieces
• 15:56
  Speaker 1: of gain out of it. It was still improving fairly rapidly,
• 16:00
  Speaker 1: to the point where actually, by the time we actually
• 16:03
  Speaker 1: published the paper, we again reduced the computer requirements, not
• 16:08
  Speaker 1: quite by an entire order of magnitude, but almost right,
• 16:11
  Speaker 1: so it still was getting faster and better at a
• 16:14
  Speaker 1: pretty rapid rate. Wow, so we had in the paper
• 16:17
  Speaker 1: we had some results that were those roughly ten x
• 16:19
  Speaker 1: faster on eighthpus and what we demonstrated in terms of
• 16:23
  Speaker 1: quality on those eight GPUs by the time we actually
• 16:26
  Speaker 1: published the paper properly we were able to do with one.
• 16:29
  Speaker 2: GPU, one GPU meaning one chip of the kind that
• 16:32
  Speaker 2: people by one hundred thousand of now to build a
• 16:35
  Speaker 2: data center exactly. So the paper actually at the end
• 16:39
  Speaker 2: mentions other possible uses beyond language for this technology. It
• 16:45
  Speaker 2: mentions images, audio, and video, I think explicitly. How much
• 16:51
  Speaker 2: were you thinking about that at the time. Was that
• 16:52
  Speaker 2: just like an afterthought or were you like, hey, wait
• 16:55
  Speaker 2: a minute, it's not just language.
• 16:57
  Speaker 1: By the time it was actually published at a conference,
• 16:59
  Speaker 1: not just the preprint. By December, we had initial models
• 17:04
  Speaker 1: on other modalities on generating images. We had the first
• 17:07
  Speaker 1: the first at the stay. At that time they were
• 17:10
  Speaker 1: not performing that well yet, but you know, they were
• 17:12
  Speaker 1: rapidly getting better. We had the first prototypes actually of
• 17:16
  Speaker 1: models working on genomic data, working on protein structure. That's
• 17:20
  Speaker 1: good for shadow good for shadowing exactly. But then we
• 17:24
  Speaker 1: ended up for a variety of reasons, we ended up
• 17:27
  Speaker 1: at first focusing on applications in computer vision.
• 17:31
  Speaker 2: The paper comes out, you know, you're working on these
• 17:33
  Speaker 2: other applications, you're presenting the paper, it's published in various forms.
• 17:38
  Speaker 1: What's the response like. It was interesting because the response
• 17:43
  Speaker 1: built in deep learning AI circles basically between the pre
• 17:49
  Speaker 1: print that I think came out and I want to
• 17:52
  Speaker 1: say June twenty seventeen, and then the actually actual publication,
• 17:56
  Speaker 1: to the extent that by the time the poster session
• 17:59
  Speaker 1: happened at the conference, there was quite a crowd at
• 18:03
  Speaker 1: the poster so we had to be shoved out of
• 18:07
  Speaker 1: the out of the hall in which the poster session happened.
• 18:10
  Speaker 1: About security and had very hors voices by the end
• 18:14
  Speaker 1: of the evening, you guys were like the Beatles of
• 18:18
  Speaker 1: the AI conference. I wouldn't say that because we weren't
• 18:23
  Speaker 1: the Beatles, because it was really it was still very specific.
• 18:26
  Speaker 2: You were more that you were more of the cool
• 18:27
  Speaker 2: hipster band. You were the hipster.
• 18:29
  Speaker 1: Band, certainly more the cool hipster band. But it was
• 18:32
  Speaker 1: an interesting experience because there were some folks and including
• 18:35
  Speaker 1: some greats in the field, who came by and said, Wow,
• 18:39
  Speaker 1: this is this is cool.
• 18:40
  Speaker 2: What has happened since has been wild.
• 18:44
  Speaker 1: It seems wild to say the least. Yes, Is it
• 18:48
  Speaker 1: surprising to you? Of course, many aspects are surprising. For sure.
• 18:53
  Speaker 1: We definitely saw pretty early on already back in twenty eighteen,
• 18:57
  Speaker 1: twenty nineteen, that something really exciting was happening here. Now
• 19:05
  Speaker 1: I'm still surprised by with the advent of chat GPT,
• 19:10
  Speaker 1: something that didn't go way beyond those language models that
• 19:15
  Speaker 1: we had already seen a few years before, was suddenly
• 19:20
  Speaker 1: the world's fastest growing consumer product.
• 19:23
  Speaker 2: Ever, right, I think ever?
• 19:25
  Speaker 1: Ever? Yes?
• 19:27
  Speaker 2: And by the way, GBT stands for generative pre transformer, right,
• 19:31
  Speaker 2: transformer is your word, that's right? So there's an interesting
• 19:36
  Speaker 2: I don't know, business side to this right, which is,
• 19:40
  Speaker 2: you were working for Google when you came up with this.
• 19:42
  Speaker 2: Google presumably owned the idea, had intellectual property around.
• 19:47
  Speaker 1: The idea has filed many a patent.
• 19:50
  Speaker 2: Was it just a choice Google made to let everybody
• 19:52
  Speaker 2: use it? Like when you see the fastest growing consumer
• 19:56
  Speaker 2: product in this year of the world not only built
• 19:58
  Speaker 2: on this idea, but using the name like and it's
• 20:02
  Speaker 2: a different company that was five years later.
• 20:04
  Speaker 1: Five years later.
• 20:04
  Speaker 2: But a patent's good for more than five years? Is
• 20:07
  Speaker 2: that a choice?
• 20:08
  Speaker 1: Is that a stret dig choice? What's going on there?
• 20:11
  Speaker 1: So the choice to do it in the first place,
• 20:15
  Speaker 1: to publish it in the first place, is really based
• 20:19
  Speaker 1: on and and rooted in a deep conviction of Google
• 20:23
  Speaker 1: at the time, And I'm actually pretty sure it still
• 20:26
  Speaker 1: is the case that it is. Actually these developments are
• 20:31
  Speaker 1: the tide that floats all votes, that lifts.
• 20:33
  Speaker 2: All votes, like a belief in progress, a belief in progress,
• 20:38
  Speaker 2: a good old fashioned Now.
• 20:40
  Speaker 1: It's also the case that at the time, organizationally, that
• 20:45
  Speaker 1: specific research arm was unusually separated from the product organizations.
• 20:51
  Speaker 1: And the reason why Brain or in general, the deep
• 20:56
  Speaker 1: learning groups were more separated was in part historical, namely
• 21:02
  Speaker 1: that when they started out there were no applications and
• 21:05
  Speaker 1: the technology was not ready for being applied, and so
• 21:09
  Speaker 1: it's completely understandable and just you know a consequence of
• 21:14
  Speaker 1: organic developments that when this technology suddenly is on the
• 21:20
  Speaker 1: cusp of being incredibly impactful, you're probably still under utilizing
• 21:24
  Speaker 1: it internally and potentially also not yet treating it in
• 21:29
  Speaker 1: the same way as you would have maybe otherwise treated
• 21:32
  Speaker 1: previous trade secrets.
• 21:34
  Speaker 2: For example, as it feels like this out their research project,
• 21:39
  Speaker 2: not like what's going to be this consumer.
• 21:42
  Speaker 1: Product exactly exactly, And to be fair, it took Open
• 21:47
  Speaker 1: a Eye in this case a fair amount of time
• 21:50
  Speaker 1: and to then turn this into this product, and most
• 21:54
  Speaker 1: of that time it also from their vantage point, wasn't
• 21:57
  Speaker 1: a product. Right. So up until all the way through
• 22:01
  Speaker 1: chat REPT, Open Eye have published all of their GPT developments,
• 22:07
  Speaker 1: maybe not all, but you know, their large fraction of
• 22:10
  Speaker 1: their work on this.
• 22:11
  Speaker 2: Yeah, they're early models.
• 22:12
  Speaker 1: The whole models were open exactly. They were more true
• 22:15
  Speaker 1: to their name really also believing in the same thing.
• 22:19
  Speaker 1: And it was only really after chat GPT and after
• 22:22
  Speaker 1: this to them also surprise to a certain extent success,
• 22:27
  Speaker 1: that they started to become more closed as well when
• 22:31
  Speaker 1: it comes to scientific developments in this past. You'll be
• 22:37
  Speaker 1: back in just a minute. Let's talk about your company.
• 22:55
  Speaker 1: When'd you decide to start Inceptive? The decision took a
• 22:58
  Speaker 1: while and was influenced by events that happened over the
• 23:03
  Speaker 1: course of about three months two to three months in
• 23:07
  Speaker 1: late twenty twenty, starting with the birth of my first child.
• 23:13
  Speaker 1: So when am I was born, two things happened. Number one,
• 23:17
  Speaker 1: witnessing a pregnancy and a birth during a pandemic where
• 23:21
  Speaker 1: there's a pathogen that's rapidly spreading, and so all of
• 23:24
  Speaker 1: that was a pretty daunting experience, and everything went great,
• 23:30
  Speaker 1: But having this new human in my arms also really
• 23:35
  Speaker 1: made me question if I couldn't more directly affect people's
• 23:41
  Speaker 1: lives positively with my work. And so I was at
• 23:46
  Speaker 1: the time quite confident that indirectly it would have effect
• 23:49
  Speaker 1: also on things like medicine, biology, etc. But I was wondering,
• 23:53
  Speaker 1: couldn't this happen more directly if I focused more on it.
• 23:58
  Speaker 1: The next thing that happened was that alpha fold two
• 24:01
  Speaker 1: results at CAST fourteen were published. CAST fourteen is this
• 24:05
  Speaker 1: biannual challenge for protein structure prediction and some other related problems.
• 24:09
  Speaker 1: This is the protein folding problem, and this is the
• 24:11
  Speaker 1: protein folding problem exactly.
• 24:13
  Speaker 2: The machine learning solving the protein folding problem, which had
• 24:15
  Speaker 2: been a problem for decades given us chain of amino
• 24:18
  Speaker 2: acids predict the three D structure of approach precisely, and
• 24:22
  Speaker 2: humans failed and machine learning succeeded.
• 24:24
  Speaker 1: Just amazing. Yes, it's a great example. Humans failed despite
• 24:29
  Speaker 1: the fact that we actually understand the physics fundamentally, but
• 24:33
  Speaker 1: we still couldn't create models that were good enough using
• 24:37
  Speaker 1: our conceptual understanding of the processes involve.
• 24:39
  Speaker 2: You would think an algorithm would work on that one, right,
• 24:42
  Speaker 2: You would just think an old school set of rules,
• 24:45
  Speaker 2: like we know what the molecules look like, we know
• 24:48
  Speaker 2: the laws of physics. It's amazing that we couldn't predict
• 24:51
  Speaker 2: it that way. Right. All you want to know is
• 24:53
  Speaker 2: what shape is the protein going to be? You know
• 24:55
  Speaker 2: all of the constituent parts, you know every atom in it,
• 24:57
  Speaker 2: and you still couldn't predict it with a set of rules,
• 25:00
  Speaker 2: but AI machine learning could.
• 25:03
  Speaker 1: Amazing, Yes, and it is amazing. Actually, when you put
• 25:07
  Speaker 1: it like this, it's important to point out that and
• 25:09
  Speaker 1: when we say we understand it, we make massive oversimplifying
• 25:12
  Speaker 1: assumptions because we ignore all the other players that are
• 25:16
  Speaker 1: present when a protein folds. We ignore a lot of
• 25:19
  Speaker 1: the kinetics of it because we say we know the structure,
• 25:22
  Speaker 1: but the truth is, we don't know all the wiggling
• 25:25
  Speaker 1: and all the shenanigans that happen on the way there, right,
• 25:27
  Speaker 1: and we don't know about uh, you know, chaperone proteins
• 25:31
  Speaker 1: that are there to influence the folding. We don't know
• 25:34
  Speaker 1: around all sorts of other I'm doing the physics one.
• 25:36
  Speaker 2: I'm doing the assume a frictionless plane version of protein precisely.
• 25:40
  Speaker 1: Precisely, precisely. And the beauty is that deep learning doesn't
• 25:43
  Speaker 1: need to make this assumption. AI doesn't need to make
• 25:45
  Speaker 1: this assumption. AI it just looks at data, and it
• 25:48
  Speaker 1: can look at more data than any human or even
• 25:51
  Speaker 1: humanity eventually could look at together. It's such a good
• 25:55
  Speaker 1: example problem to demonstrate that these models are ready for
• 25:59
  Speaker 1: prime time in this field and ready for lots of applications,
• 26:02
  Speaker 1: not just one or two, but men sold, and so
• 26:04
  Speaker 1: that happens, so sold exactly. And then the third thing
• 26:08
  Speaker 1: was that the COVID mRNA vaccines came out with astonishing
• 26:14
  Speaker 1: ninety plus percent out of.
• 26:17
  Speaker 2: The gate that they were still so underraty. Under the
• 26:21
  Speaker 2: beginning of the pandemic, people were like, it'll be two
• 26:24
  Speaker 2: or three years, and if there's sixty percent effective, that'll be.
• 26:27
  Speaker 1: Great, exactly exactly, And so everybody forgets. Everybody forgets it.
• 26:30
  Speaker 1: And when you look at it, this is a molecule
• 26:33
  Speaker 1: family that was for you know, most of the time
• 26:35
  Speaker 1: that we've known about it since the sixties, I suppose
• 26:38
  Speaker 1: we've treated it like a neglected stepchild of molecular biology,
• 26:43
  Speaker 1: because you're talking about marine in general. In general.
• 26:47
  Speaker 2: Everybody loves DNA, right, DNA.
• 26:50
  Speaker 3: Everybody loves DNA movie star, Yeah, exactly, exactly, even though
• 26:53
  Speaker 3: now looking back, DNA is merely you know, the place
• 26:57
  Speaker 3: where life takes its notes, maybe the hard drive and
• 27:00
  Speaker 3: the memory.
• 27:01
  Speaker 1: It's the book, right, it's the book. So but but
• 27:04
  Speaker 1: at the end of the day, it was this molecule
• 27:07
  Speaker 1: family that was about to save, you know, depending on them,
• 27:10
  Speaker 1: tens of millions of lives and in rapid time. So
• 27:14
  Speaker 1: all these things hold, but we have no training data
• 27:16
  Speaker 1: to apply anything like alpha fold to this specific molecule family,
• 27:21
  Speaker 1: no training data to speak of. We had two hundred
• 27:24
  Speaker 1: thousand known protein structures at the time, I believe, maybe optimistically,
• 27:28
  Speaker 1: we had maybe twelve hundred known RNA structures. And on
• 27:32
  Speaker 1: top of that, it was also fairly clear that for
• 27:34
  Speaker 1: RNA going directly to function would be much much more important,
• 27:38
  Speaker 1: because it's in a certain sense a less strongly structured molecule,
• 27:41
  Speaker 1: and other aspects of the molecule might play a bigger role.
• 27:45
  Speaker 1: And then on top of that, the attention that generative
• 27:49
  Speaker 1: AI was receiving overall, also now in the field of
• 27:53
  Speaker 1: pharma or of medicine, was building, And so I ended
• 27:58
  Speaker 1: up finding myself in a conversation where very I would
• 28:02
  Speaker 1: say wise longtime mentor of mine pointed out that, you know,
• 28:08
  Speaker 1: maybe ten years from now or so, somebody could tell
• 28:11
  Speaker 1: my daughter that there was this perfect storm where this
• 28:14
  Speaker 1: MACLE molecule with no training data was about to save
• 28:17
  Speaker 1: the world and could do so much more in the
• 28:20
  Speaker 1: direction of positively impacting people's lives. We didn't have training data,
• 28:24
  Speaker 1: would be very expensive to create it, but using the
• 28:27
  Speaker 1: technology that I've been or technologies that I'd been working
• 28:30
  Speaker 1: on for the last I don't know, ten plus years,
• 28:32
  Speaker 1: and the ability because of the attention that people were
• 28:36
  Speaker 1: now giving to AI in this field the ability to
• 28:40
  Speaker 1: raise quite a bit of money. I, in that position,
• 28:43
  Speaker 1: chose to stay back at my cushy dream job in
• 28:47
  Speaker 1: big tech and not actually take this opportunity to really
• 28:53
  Speaker 1: positively impact people's lives, And that idea was not one
• 28:56
  Speaker 1: I was willing to entertain.
• 28:59
  Speaker 2: You couldn't just coast it out at Google and let
• 29:01
  Speaker 2: somebody else go figure out RNA.
• 29:03
  Speaker 1: Yeah, and it's not just RNA. I think RNA is
• 29:06
  Speaker 1: a great starting point at the end of the day,
• 29:08
  Speaker 1: but building models that learn from first of all, all
• 29:14
  Speaker 1: the publicly available data that we can possibly get our
• 29:17
  Speaker 1: hands on, but also from data that we can reasonably
• 29:19
  Speaker 1: effectively create in our own lab. How to design molecules
• 29:24
  Speaker 1: for specific functions is something that now is within reach
• 29:28
  Speaker 1: and that will in the next years, in the years
• 29:32
  Speaker 1: to come, have completely transformational impact on how we even
• 29:36
  Speaker 1: think about what medicines are. That any opportunity to speed
• 29:41
  Speaker 1: this up, to make this happen, even just a day
• 29:44
  Speaker 1: sooner than it could have otherwise happened, is incredibly valuable
• 29:48
  Speaker 1: in my opinion.
• 29:49
  Speaker 2: As you're talking about this idea that the absence of
• 29:52
  Speaker 2: training data is kind of seems to be at the
• 29:55
  Speaker 2: center of it, right, It seems to be the core
• 29:57
  Speaker 2: yeah problem, which makes sense, right, Like the reason language
• 30:01
  Speaker 2: works so well is basically because of the Internet. I know,
• 30:03
  Speaker 2: now we're going beyond it, but like it just happened
• 30:05
  Speaker 2: to be that there was this incredibly giant set of
• 30:08
  Speaker 2: natural life language that became available. We don't have anything
• 30:12
  Speaker 2: like that for RNA, so are you. I mean, it's
• 30:14
  Speaker 2: kind of step one at inceptive creating the data. Is
• 30:19
  Speaker 2: that kind of what's happening?
• 30:22
  Speaker 1: So step one that inceptive is learning to use all
• 30:25
  Speaker 1: the data or was I think we've made a lot
• 30:27
  Speaker 1: of focus in that direction, learning to use all the
• 30:28
  Speaker 1: data that is available already and identify what other data
• 30:33
  Speaker 1: we're missing, and then see how far we can get
• 30:36
  Speaker 1: with just the publicly available data and at the same
• 30:39
  Speaker 1: time scale up generating our own data. And it turns
• 30:42
  Speaker 1: out that actually, because of the nature of evolution, because
• 30:46
  Speaker 1: of how evolution isn't actually incentivized to really explore the
• 30:51
  Speaker 1: entire space of possibilities. It is almost always given that
• 30:58
  Speaker 1: if you are trying to design exceptional molecules, especially ones
• 31:02
  Speaker 1: that are not say, you know, natural formats, you are
• 31:08
  Speaker 1: basically gearing need to need novel training in it.
• 31:11
  Speaker 2: Yeah, basically you're saying you build RNAs that don't exist
• 31:15
  Speaker 2: in the world that have therapeutic uses, and there's no
• 31:17
  Speaker 2: kind of definitionally no training.
• 31:19
  Speaker 1: Yes, that exist. The funny thing is we have a
• 31:21
  Speaker 1: few of them, and so we have existence proofs of
• 31:25
  Speaker 1: OURNA molecules, for example, RNA viruses that actually exhibit incredibly
• 31:32
  Speaker 1: complex different functions in ourselves, that do all sorts of
• 31:38
  Speaker 1: things that we don't usually like. But if we could
• 31:40
  Speaker 1: use those, you know, for good, If we could use those,
• 31:44
  Speaker 1: you know, in ways that would actually be aimed at
• 31:48
  Speaker 1: fighting disease rather than creating them, those kinds of functions,
• 31:51
  Speaker 1: even just a small subset of them, would really transform
• 31:55
  Speaker 1: medicine already. And so we know it's possible. What are
• 31:58
  Speaker 1: you dreaming of when you say that, what are you
• 31:59
  Speaker 1: thinking of? Specific? Okay, So, for example, right, one estimate
• 32:03
  Speaker 1: is that in order for COVID to infect you, you
• 32:07
  Speaker 1: would need potentially as few as five COVID genomes inside
• 32:13
  Speaker 1: your organism that's already in five five viral particles. Five
• 32:16
  Speaker 1: viral particles. Yeah, you inhale those, you wouldn't have to
• 32:21
  Speaker 1: inject it you wouldn't even have to swallow it, you
• 32:24
  Speaker 1: inhale them.
• 32:25
  Speaker 2: If we could have a medicine that worked as well
• 32:27
  Speaker 2: as a disease is a version of your.
• 32:29
  Speaker 1: Truth, exactly exactly so at the end of the day, right,
• 32:32
  Speaker 1: this medicine is able to spread in your body only
• 32:36
  Speaker 1: into certain types of organs and tissues and cells. It
• 32:40
  Speaker 1: does certain things there that are really quite complex, right,
• 32:43
  Speaker 1: changing the cells behavior again not usually in this case
• 32:46
  Speaker 1: in favorable ways, but still in ways that wouldn't have
• 32:50
  Speaker 1: to be modified that much in order to potentially be
• 32:53
  Speaker 1: exactly what you would need for complex multifactorial medicine. And
• 32:56
  Speaker 1: if you could make all of that happen by just
• 32:58
  Speaker 1: inhaling five of those molecules, then again, that would completely
• 33:02
  Speaker 1: change how you think about medicine. Right, you have viruses
• 33:05
  Speaker 1: that aren't immediately active, but that are inactive for long
• 33:09
  Speaker 1: periods of time in your organism, and only under certain conditions,
• 33:13
  Speaker 1: say under certain immune conditions, really start being reactivated. Why
• 33:19
  Speaker 1: can't we have medicines that work in a similar way
• 33:23
  Speaker 1: where you actually not only in a vaccination sense, but
• 33:26
  Speaker 1: where you take a medicine for a genetic predisposition for
• 33:29
  Speaker 1: a certain disease that you are able to take a
• 33:31
  Speaker 1: metic design of medicine that you can take and that
• 33:33
  Speaker 1: waits until the disease actually starts to develop, and only
• 33:36
  Speaker 1: then and only where that disease then starts developed, becomes
• 33:39
  Speaker 1: active and actually affects it and potentially also then alarms
• 33:43
  Speaker 1: the doctor through a blunt test.
• 33:45
  Speaker 2: Like for cancer cells or something. So you have some
• 33:48
  Speaker 2: kind of prophylactic medicine in your body and it is
• 33:51
  Speaker 2: encoded in such a way that it just hangs out there,
• 33:55
  Speaker 2: like herpes, to take a pathological example for example, and
• 33:58
  Speaker 2: only in certain settings does it do anything. And those
• 34:02
  Speaker 2: settings are if you see a cancer cell, destroy it,
• 34:05
  Speaker 2: otherwise just it there precisely.
• 34:07
  Speaker 1: And if you can design those also in ways where
• 34:09
  Speaker 1: you can just make them all go away. When you know,
• 34:12
  Speaker 1: you take a say a completely harmless small molecule, and
• 34:15
  Speaker 1: that's again entirely feasible.
• 34:17
  Speaker 2: Sure, So, I mean you're dreaming big. These are wonderful
• 34:21
  Speaker 2: big you know, science fiction andy dreams that I hope
• 34:23
  Speaker 2: you figure them out. On a practical level. What's happening
• 34:27
  Speaker 2: at the company right now? How many people work there,
• 34:29
  Speaker 2: what are they doing, and what are they figured out
• 34:30
  Speaker 2: so far?
• 34:31
  Speaker 1: We're round forty. What we're doing is really exactly what
• 34:35
  Speaker 1: we just talked about. We're basically scaling data generation experiments
• 34:40
  Speaker 1: in our lab that allow us to assess a variety
• 34:44
  Speaker 1: of different functions of different mostly RNA molecules actually mostly
• 34:50
  Speaker 1: m RNA molecules at the moment, that are relevant to
• 34:55
  Speaker 1: a pretty broad variety of different diseases. And so this
• 34:58
  Speaker 1: ranges from things like infectious disease vaccines to sell therapies
• 35:03
  Speaker 1: that can be applied in oncology or an auto or
• 35:06
  Speaker 1: against autoimmune disease. We have mRNAs that we hope will
• 35:12
  Speaker 1: eventually be effective in enzyme replacement as enzyme replacement therapies
• 35:16
  Speaker 1: for families of a large family of rare diseases, and
• 35:20
  Speaker 1: the list goes on. And so we're creating this or
• 35:23
  Speaker 1: growing this training data set that eventually, on top of
• 35:30
  Speaker 1: foundation and models that we pre trained on all publicly
• 35:33
  Speaker 1: available data, allow us to tune those foundation models towards
• 35:39
  Speaker 1: designing exceptional molecules for exactly those applications and many more
• 35:44
  Speaker 1: sharing similar properties.
• 35:45
  Speaker 2: So you basically build new mr and a model molecules
• 35:50
  Speaker 2: and test them, and then you give that data to
• 35:52
  Speaker 2: your model and presumably it tells you what to build next,
• 35:56
  Speaker 2: or it helps you figure out what to build next.
• 35:58
  Speaker 2: It's sort of a loop in that way.
• 35:59
  Speaker 1: The models are definitely one interesting source for proposals if
• 36:03
  Speaker 1: you wish for what to synthesize and test next, they're
• 36:07
  Speaker 1: not the only such source, so we basically also explore
• 36:11
  Speaker 1: kind of and maybe less guided or heuristically guided ways,
• 36:15
  Speaker 1: but exactly so in some of the cases, it's really
• 36:18
  Speaker 1: quite iterative. For some of those functions and for some
• 36:21
  Speaker 1: of those modalities and diseases or disease targets, we're actually
• 36:25
  Speaker 1: already at a point where our models can spit out
• 36:29
  Speaker 1: entirely novel molecules that really are unlike anything they've ever
• 36:33
  Speaker 1: seen or we've ever seen in nature, that very consistently
• 36:38
  Speaker 1: perform quite favorably compared to pretty strong baselines by incumbents
• 36:43
  Speaker 1: in the field.
• 36:44
  Speaker 2: When you say perform quite favorably compared to baselines by
• 36:48
  Speaker 2: incumbents in the field, and does that on some level
• 36:50
  Speaker 2: mean better than what experts would think.
• 36:54
  Speaker 1: Up, better than what experts can think up, and also
• 36:56
  Speaker 1: better than more traditional machine learning tools can easily produce.
• 37:01
  Speaker 2: It's like that famous moment in the Go match when
• 37:03
  Speaker 2: alpha go made some move that like no human being
• 37:07
  Speaker 2: ever would have thought of.
• 37:09
  Speaker 1: Yes, so I would say we've long passed the move
• 37:14
  Speaker 1: thirty seven in the sense that our understanding of the
• 37:18
  Speaker 1: underlying biological phenomena is so incomplete that for most of
• 37:23
  Speaker 1: the things that we're able to design for, we don't
• 37:26
  Speaker 1: really understand why they happen.
• 37:28
  Speaker 2: Huh, when you say weed, you mean at inceptive or
• 37:31
  Speaker 2: do you mean just medicine in general?
• 37:33
  Speaker 1: I would say just medicine in general.
• 37:35
  Speaker 2: Okay, So Inceptive is doing this very kind of high
• 37:39
  Speaker 2: level work, right, I mean building what will hopefully be
• 37:42
  Speaker 2: the foundation. What's the right amount of time in the
• 37:46
  Speaker 2: future to ask about when will we know if it works?
• 37:48
  Speaker 2: You think five years?
• 37:50
  Speaker 1: So the general idea of using genitive AI and similar
• 37:55
  Speaker 1: techniques to generate therapeutics, there are some things in clinical
• 38:02
  Speaker 1: trials that were largely designed with AI. As far as
• 38:07
  Speaker 1: I know, we're still maybe now we have the first
• 38:11
  Speaker 1: trials just now starting for molecules that were truly entirely
• 38:15
  Speaker 1: designed by A.
• 38:17
  Speaker 2: As opposed to sort of selected from a library.
• 38:20
  Speaker 1: Or selected, influenced, exactly selected, adjusted to you, and tweaked,
• 38:25
  Speaker 1: et cetera. Right, So that's really still only happening just now,
• 38:29
  Speaker 1: but we will see I believe, the first success or
• 38:33
  Speaker 1: a first success of such molecules, certainly within the next
• 38:37
  Speaker 1: five years.
• 38:38
  Speaker 2: What about more narrowly, the project at inceptive.
• 38:41
  Speaker 1: It's a similar timeframe. We should be able to get
• 38:44
  Speaker 1: molecules into the clinic in the next few years, certainly
• 38:49
  Speaker 1: in the next handful of years. Now. These will not
• 38:51
  Speaker 1: be molecules with where the objective that we used in
• 38:57
  Speaker 1: their design is you know, even remotely as complex or
• 39:02
  Speaker 1: you know, kind of the different functions that we're designing
• 39:05
  Speaker 1: for are are not going to be even remotely as
• 39:09
  Speaker 1: diverse as say what you would find because we used
• 39:12
  Speaker 1: this example earlier in ourna virus. These will really be
• 39:15
  Speaker 1: more simpler. Those will be molecules that don't do things
• 39:21
  Speaker 1: that we couldn't possibly have done before, but that do
• 39:24
  Speaker 1: them much better in ways that are more accessible, in
• 39:28
  Speaker 1: ways that come with less side effects.
• 39:30
  Speaker 2: What biotech largely is is they make protein drugs. And
• 39:34
  Speaker 2: so if you could make an mRNA drug where you
• 39:37
  Speaker 2: put the m RNA into the body and the body
• 39:39
  Speaker 2: makes the protein, it wouldn't be some crazy sleeper cell
• 39:42
  Speaker 2: that sits in your body for twenty years or whatever,
• 39:44
  Speaker 2: but it might be a more practical alternative to today's biotech drugs.
• 39:49
  Speaker 1: Absolutely.
• 39:50
  Speaker 2: So you've had a kind of crash course in biology
• 39:53
  Speaker 2: in the last few years, yes, And I'm curious, like,
• 39:56
  Speaker 2: what is what is something that has been particularly compelling
• 40:00
  Speaker 2: or surprising or interesting to you that you have learned
• 40:02
  Speaker 2: about biology.
• 40:03
  Speaker 1: They're countless things. The biggest one, or the red thread
• 40:08
  Speaker 1: across many of them is really just how effective life
• 40:16
  Speaker 1: is at finding solutions to problems that on one hand
• 40:22
  Speaker 1: are incredibly robust, surprisingly robust, and on the other hand,
• 40:28
  Speaker 1: are so different from how we would design solutions to
• 40:34
  Speaker 1: similar problems.
• 40:36
  Speaker 2: Uh huh.
• 40:37
  Speaker 1: That really this comes back to this idea that we
• 40:40
  Speaker 1: might just not be particularly well equipped in terms of
• 40:43
  Speaker 1: cognitive capabilities to understand biology that basically, you know we
• 40:49
  Speaker 1: are we would never think to do it this way,
• 40:53
  Speaker 1: and how we think to do it is oftentimes much
• 40:57
  Speaker 1: more brittle.
• 40:58
  Speaker 2: Uh huh. Brittle is an interesting world, less, less resilient,
• 41:02
  Speaker 2: less able to persist under different.
• 41:03
  Speaker 1: Conditions, exactly exactly. I mean, you know, we still haven't
• 41:06
  Speaker 1: built machines that can fix themselves, for one.
• 41:09
  Speaker 2: Which is fundamentally the miracle of being a human being.
• 41:12
  Speaker 1: Just fundamentally exactly, exactly exactly and so and of course
• 41:17
  Speaker 1: this is true across the scales, right from from you know,
• 41:21
  Speaker 1: single cells all the way to complex organisms like ourselves
• 41:25
  Speaker 1: and and really just how many also very different kinds
• 41:33
  Speaker 1: of solutions life has found and or and or constantly
• 41:37
  Speaker 1: is finding. Uh. And you see this all over the place,
• 41:40
  Speaker 1: and it's both daunting, humbling, but also incredibly inspiring when
• 41:47
  Speaker 1: it comes to applying AI in this area, because again
• 41:51
  Speaker 1: I think that at least so far, it's the best
• 41:54
  Speaker 1: tool and maybe actually the only tool we have so
• 41:58
  Speaker 1: far in face of this kind of complexity. Really design
• 42:02
  Speaker 1: interventions that medicines that go way beyond what we were
• 42:07
  Speaker 1: able to do or are able to do, just based
• 42:09
  Speaker 1: on our own conceptual understanding.
• 42:14
  Speaker 2: We'll be back in a minute with the lightning round.
• 42:18
  Speaker 2: M hm, let's finish for the lightning round. As an
• 42:33
  Speaker 2: inventor of the Transformer model, are there particular possible uses
• 42:38
  Speaker 2: of it that worry you flash make you sad?
• 42:42
  Speaker 1: I am quite concerned about the p doom doomerism, whatever
• 42:48
  Speaker 1: you want to call it, existential fear instilling rhetoric that
• 42:53
  Speaker 1: is in some cases actually also promoted by people by
• 42:58
  Speaker 1: entities in the space.
• 43:00
  Speaker 2: So just to be clear, you're you're not worried about
• 43:02
  Speaker 2: the existential risk. You're worried about people talking.
• 43:05
  Speaker 1: I'm worried about the about the existential risk being inflated
• 43:10
  Speaker 1: or the perception being inflated to the extent that we
• 43:17
  Speaker 1: actually don't look enough at some of the much more
• 43:21
  Speaker 1: concrete and much more immediate risks. Right. I'm not going
• 43:24
  Speaker 1: to say that the existential risk is zero. That would
• 43:27
  Speaker 1: be silly.
• 43:27
  Speaker 2: What is a concrete an immediate risk that is you
• 43:31
  Speaker 2: think under.
• 43:32
  Speaker 1: Discuss these large scale models are such defective tools in
• 43:37
  Speaker 1: manipulating people in large numbers already today, and it's happening
• 43:42
  Speaker 1: everywhere for many, many different purposes by in some cases
• 43:47
  Speaker 1: benevolent and in many cases malevolent actors that I really
• 43:53
  Speaker 1: firmly believe we need to look much more at things
• 43:56
  Speaker 1: like enabling cryptographic certification of human generated content, because doing
• 44:03
  Speaker 1: that with the machine generated content is not going to work.
• 44:05
  Speaker 1: But we definitely can cryptographically certify human generated content as.
• 44:09
  Speaker 2: Such basically watermarking or something some way to say this
• 44:13
  Speaker 2: a human made this.
• 44:14
  Speaker 1: Exactly what would you be.
• 44:15
  Speaker 2: Working on if you were not working in biology on
• 44:19
  Speaker 2: drug development?
• 44:20
  Speaker 1: Education using using artificial intelligence to democratize access to education.
• 44:26
  Speaker 2: What have you seen that has been impressive or compelling
• 44:30
  Speaker 2: to you in that regard?
• 44:31
  Speaker 1: There are lots of little examples so far and really countless.
• 44:36
  Speaker 1: It's what's happening at the con Academy. There are many
• 44:39
  Speaker 1: examples of AI applied to education problems in places like China,
• 44:44
  Speaker 1: for example. You have a bunch of very compelling examples
• 44:47
  Speaker 1: in fiction. A book I really like, like a named
• 44:50
  Speaker 1: Neil Stephenson, The Diamond Age or a Young Ladies Illustrated
• 44:54
  Speaker 1: primer that I recommend if you.
• 44:56
  Speaker 2: Just everybody in AI talks about that, Well now they do.
• 44:59
  Speaker 1: Yeah, it's yeah, well.
• 45:01
  Speaker 2: Now they do.
• 45:01
  Speaker 1: You liked it before?
• 45:02
  Speaker 2: It was cool?
• 45:03
  Speaker 1: I'm sure at one point I thought it was really
• 45:05
  Speaker 1: really important and sure that Neil students know is that
• 45:09
  Speaker 1: we are about to be able to build the primary
• 45:13
  Speaker 1: and so I ended up having coffee with him to
• 45:16
  Speaker 1: tell him, oh, that's great. So at the end of
• 45:19
  Speaker 1: the day, maybe the biggest inspiration there is my daughter.
• 45:25
  Speaker 1: She's four and a half now, and I think she
• 45:28
  Speaker 1: could today read. She can read read okay, but she
• 45:34
  Speaker 1: could read, you know, grade school level if she had
• 45:38
  Speaker 1: access to you know, an AI tutor teaching her how
• 45:41
  Speaker 1: to read?
• 45:41
  Speaker 2: Does your daughter use AI use you know, AI chat
• 45:45
  Speaker 2: butts not directly without me, But we've.
• 45:49
  Speaker 1: Actually used chat GPT to implement an AI reading tutor
• 45:55
  Speaker 1: that works reasonably well. I mean we basically, you know,
• 45:58
  Speaker 1: kind of as I call it now, vibe coding, vibe coded.
• 46:02
  Speaker 1: And I wasn't there for all of it. Took some time,
• 46:04
  Speaker 1: but she was there for some of it. Oh, you
• 46:06
  Speaker 1: vibe coded it with her? Yeah, well, I mean she was,
• 46:09
  Speaker 1: she was there. You know, she witnessed a good chunk
• 46:12
  Speaker 1: of it, Yes, although she was more interested in the
• 46:14
  Speaker 1: image generation parts. But yeah, we have a sketch of
• 46:16
  Speaker 1: one that she quite enjoys. So that's kind of like
• 46:19
  Speaker 1: the extent of her at the sage using I directly.
• 46:30
  Speaker 1: Yakabust is the CEO and.
• 46:32
  Speaker 2: Co founder of Inceptive and the co author of the
• 46:36
  Speaker 2: paper Attention Is All You Need. Just a quick note,
• 46:40
  Speaker 2: This is our last episode before a break of a
• 46:43
  Speaker 2: couple of weeks, and then we'll be back with more episodes.
• 46:46
  Speaker 2: Please email us at problem at Pushkin dot fm. We
• 46:50
  Speaker 2: are always looking for new guests for the show. Today's
• 46:53
  Speaker 2: show was produced by Trinamanino and Gabriel Hunter Chang. It
• 46:57
  Speaker 2: was edited by Alexander Garretton and engineered by Sarah muguerrett