Latest / What's Your Problem? / Predicting Human Health with AI

Transcript

Predicting Human Health with AI · What's Your Problem?

0:15
Speaker 1: Pushkin. Imagine something that is sort of like chat GPT,
0:24
Speaker 1: but for the human body. Chat GPT looks at a
0:28
Speaker 1: sentence and predicts what words are likely to come next.
0:31
Speaker 1: This thing would look at a human body and predict
0:34
Speaker 1: what diseases are likely to come next. The body is
0:38
Speaker 1: wildly complex and unpredictable. This seems like a very, very
0:41
Speaker 1: hard problem, but it is a problem people are working on,
0:45
Speaker 1: and at least in some circumstances, they're figuring out how
0:49
Speaker 1: to make predictions that are truly useful. I'm Jacob Goldstein,
0:57
Speaker 1: and this is What's Your Problem, the show where I
0:59
Speaker 1: talk to people who are trying to make technological progress.
1:03
Speaker 1: My guest today is Charles Fisher, co founder and CEO
1:06
Speaker 1: of Unlearned. Charles' problem is how do you build an
1:10
Speaker 1: AI model that can predict human health. Charles and his
1:14
Speaker 1: colleagues have built a predictive model of human health that's
1:17
Speaker 1: already being used in clinical trials for new drugs and
1:21
Speaker 1: new medical devices. But we started out talking about the
1:24
Speaker 1: big picture, about the very idea of trying to predict
1:27
Speaker 1: what's going to happen to a human body.
1:32
Speaker 2: It's funny when I talk about trying to quantify biology
1:36
Speaker 2: and make it predictable. I often get hit with this
1:40
Speaker 2: critique that biology isn't physics. Biology is complex, biology is
1:45
Speaker 2: not physics. We're not going to be able to do that.
1:47
Speaker 1: Let's deterministic.
1:50
Speaker 2: Right, So for physics, for two thousand years, right, people
1:54
Speaker 2: started working on physics in ancient Greece. And for two
1:58
Speaker 2: thousand years, physics wasn't physics. Physics was unpredictable. Physics was
2:03
Speaker 2: too complex to understand until something was invented. And that
2:07
Speaker 2: thing was calculus.
2:10
Speaker 1: Until new right.
2:10
Speaker 2: Yeah, So once calculus was invented, all of the sudden,
2:15
Speaker 2: we had a new language. In this language, this new
2:18
Speaker 2: kind of mathematics allowed us to really easily describe lots
2:21
Speaker 2: of physical phenomena. And so now physics has become this
2:25
Speaker 2: thing that's very predictable and well understood. And that's what
2:28
Speaker 2: we've been waiting for in biology. We've been waiting for
2:31
Speaker 2: a new tool, a new language, a new mathematics that
2:34
Speaker 2: will allow us to understand these complex systems. And that's
2:37
Speaker 2: really what I think these new tools are.
2:39
Speaker 1: So I think so your hope, your hope is that
2:41
Speaker 1: machine learning generative AI will do for medicine biology. What
2:48
Speaker 1: Calculus did for physics.
2:50
Speaker 2: Exactly. That is big, big, It's exactly what I hope.
2:53
Speaker 2: That's exactly what I hope.
2:55
Speaker 1: So okay, so this is your hope. You're starting this
2:58
Speaker 1: company to test your hypothesis. Uh, what do you do?
3:04
Speaker 2: What do you mean? What do I do? What I
3:06
Speaker 2: do on day one? Or like, what are we doing? No?
3:08
Speaker 1: No, no, We're back to twenty seventeen. You have this
3:12
Speaker 1: big up in the sky, two thousand year, thirty thousand
3:16
Speaker 1: foot idea. But you got to make a thing that
3:18
Speaker 1: somebody is going to pay you for that will hopefully
3:20
Speaker 1: use AI in medicine in some way. So what do
3:22
Speaker 1: you do?
3:23
Speaker 2: So we didn't know what would work, so we focused
3:27
Speaker 2: on two different problems at the time. So one problem is,
3:33
Speaker 2: let's imagine we're going to have a bunch of data
3:36
Speaker 2: from some maybe a big large collection of patients. We're
3:40
Speaker 2: gonna have this data all over time, so the symptoms
3:43
Speaker 2: that a patient might have every week, four year or
3:47
Speaker 2: something like that. And our goal is to be able
3:49
Speaker 2: to create a simulator of a patient's future health. So,
3:53
Speaker 2: given what I know about a patient in the past,
3:56
Speaker 2: can I simulate what will happen to them in the future.
3:59
Speaker 1: And presumably that is sort of probabilistic. I mean, what
4:03
Speaker 1: we know about health, Like you can say there's an
4:05
Speaker 1: x percent chance that in why years this person will
4:08
Speaker 1: have a heart attack something like that.
4:10
Speaker 2: Exactly. Yeah, we want to yes, because so many things
4:13
Speaker 2: are undetermined in that you know, maybe yeah, exactly.
4:16
Speaker 1: Right, and it's just the nature of the world, right,
4:19
Speaker 1: one hundred percent.
4:20
Speaker 2: Yeah.
4:21
Speaker 1: So okay, so you have this idea of basically where
4:24
Speaker 1: chat GBT, which didn't exist yet, but predicts the next
4:28
Speaker 1: word with some probability you want to predict the next
4:31
Speaker 1: health outcome.
4:32
Speaker 2: For exactly that is the big idea. Yeah, So that
4:34
Speaker 2: that was one of them. The other that was not
4:36
Speaker 2: the only one that was the one that is what
4:38
Speaker 2: we do. The one that we didn't do is we
4:40
Speaker 2: were interested as well potentially so that's at a very
4:43
Speaker 2: macroscopic scale, that's at the scale of the person, whereas
4:47
Speaker 2: the other thing we were interested in was potentially could
4:49
Speaker 2: we go at the micro scale and look at what's
4:52
Speaker 2: happening inside individual cells. We were interested in this at
4:55
Speaker 2: the beginning. Basically, the way we figured this out is
4:58
Speaker 2: we signed a few deals with farmer companies to try
5:01
Speaker 2: these things, and we found found that the technology worked
5:06
Speaker 2: really well in this simulating health outcomes, and it didn't
5:11
Speaker 2: work very well when it comes down to simulating what's
5:13
Speaker 2: inside the cell. And I think this comes down to data,
5:16
Speaker 2: which is that we get a ton of data on
5:18
Speaker 2: human health outcomes, like literally every time you go to
5:21
Speaker 2: the doctor, there's data there on your health outcomes. But
5:24
Speaker 2: the data from the things inside the cell, there is
5:28
Speaker 2: a lot of it, but it's much more difficult to
5:31
Speaker 2: work with. So I think that was a lot of
5:34
Speaker 2: what drove us in this direction is really the focus
5:37
Speaker 2: on what we think we have the data to solve
5:39
Speaker 2: these kinds of problems.
5:40
Speaker 1: So, Okay, you go in the direction of simulating health
5:44
Speaker 1: outcomes for patients, and in particular, sort of where you
5:48
Speaker 1: get to is working with companies that are running clinical trials.
5:52
Speaker 1: And I know eventually you get to a point where
5:54
Speaker 1: companies can use your model, use your software to run
5:57
Speaker 1: clinical trials with fewer patients. So just tell me about that,
6:02
Speaker 1: arc tell me how you get there.
6:04
Speaker 2: Clinical trials are, well, they're super tick forever, and they're
6:08
Speaker 2: really really expensive. Something might take like five years and
6:10
Speaker 2: cost one hundred million dollars to run a clinical trial. Yeah,
6:14
Speaker 2: in the way that these are hundreds or thousands of patients, right, oh,
6:18
Speaker 2: thousands of patients typically, right, Yeah, And typically half of
6:22
Speaker 2: the patients in a clinical trial are receiving a PLACBO.
6:25
Speaker 2: So you're going to randomly assign half to receive an
6:27
Speaker 2: experimental treatment have to receive a PLACBO. And the reason
6:30
Speaker 2: is that every clinical trial is ultimately just doing a comparison.
6:34
Speaker 2: You're comparing how a patient responds to the new treatment
6:36
Speaker 2: to how they respond if they don't get that treatment.
6:38
Speaker 1: And let me just give a shout out to the
6:40
Speaker 1: randomized controlled trial as like a really beautiful construct, right,
6:45
Speaker 1: not that old? Not that old. I learned that a
6:48
Speaker 1: ring for this interview, like less than one hundred years old, amazingly.
6:52
Speaker 1: But it's a perfect way to assess not perfect, it's
6:56
Speaker 1: a very very good way to assess causality. It's really elegant.
7:00
Speaker 2: It is an elegant idea. But if you're a patient,
7:04
Speaker 2: why are you participating a clinical trial at all? What's
7:07
Speaker 2: the number one reason people participate in clinic trials. They
7:09
Speaker 2: participate in clinical trials because they want access to this
7:12
Speaker 2: experimental treatment that you can't get any other way. That's
7:14
Speaker 2: the number one reason why patients are participating in clinical trials.
7:17
Speaker 2: Number one, Now they.
7:19
Speaker 1: Don't they don't want to be randomized to the placebo.
7:21
Speaker 2: No, no, no, they don't.
7:23
Speaker 1: I can certainly understand that it is the case, right
7:27
Speaker 1: that most trials fail, meaning the drug is not helping
7:33
Speaker 1: you and possibly hurting you, meaning on average, you're better
7:36
Speaker 1: off being in the placebo arm Like that is true, right.
7:39
Speaker 2: Yea, there's a principle of equipoise. But that's an academic
7:42
Speaker 2: Ivory tower principle.
7:44
Speaker 1: I mean, it also is true. Just sue, that's fine, that's.
7:48
Speaker 2: Fine, but in the end, that's like, in the end,
7:52
Speaker 2: patients choose not to participate in clinical trials because they
7:56
Speaker 2: don't want to get a placebo. Patients drop out of
8:00
Speaker 2: clinical trials when they think they are getting a placbo.
8:03
Speaker 2: Those are also true. Number one reason those things happen.
8:07
Speaker 2: Are those reasons? Fair?
8:08
Speaker 1: Okay?
8:08
Speaker 2: Right? So, And in fact, twenty percent of clinical trials
8:12
Speaker 2: failed not because the drug didn't work, but because they
8:15
Speaker 2: just couldn't find enough people to participate, okay. And what
8:19
Speaker 2: we realized though, is that there was a way for
8:24
Speaker 2: us not to try to replace the randomized control trial,
8:29
Speaker 2: but to make it better, and that what we are
8:31
Speaker 2: doing is we could take what we call digital twins
8:35
Speaker 2: of the patients, so these are these simulations of their
8:38
Speaker 2: of their future outcomes, and that we could incorporate those
8:42
Speaker 2: data into our cts directly randomized control trials. We call
8:48
Speaker 2: it just kind of like a reimagining of our cts.
8:51
Speaker 2: It's it's you're going to have a RCT that is
8:55
Speaker 2: more accurate, that is has requires fewer patients, and as
9:01
Speaker 2: a result, you get a lot of the benefits of
9:03
Speaker 2: faster trials of things that are better for the patients.
9:06
Speaker 2: We can talk about that in a minute, but you
9:09
Speaker 2: keep all of the same scientific rigger.
9:12
Speaker 1: So specifically, okay, that's a good like big picture. Specifically,
9:18
Speaker 1: how does it.
9:19
Speaker 2: Work right now? We build one model per disease. So,
9:25
Speaker 2: for example, we have a model for patients with Alzheimer's disease.
9:28
Speaker 2: We have a separate model for patients with als, we
9:31
Speaker 2: have a separate model for multiple scleroses, et cetera.
9:33
Speaker 1: Let's pick one model and talk about it. What's the
9:36
Speaker 1: one that's farthest along, Which is the one that works
9:38
Speaker 1: the best?
9:39
Speaker 2: Yeah, So our Alzheimer's disease model is that was our
9:41
Speaker 2: first one that we've published scientific papers on and things
9:44
Speaker 2: like this, so that ones our most well known.
9:47
Speaker 1: Okay, so you're setting out to build a model that
9:51
Speaker 1: will predict whether what's going to happen, presumably to a
9:55
Speaker 1: patient who has the early stages of Alzheimer's disease, How
9:58
Speaker 1: will their disease progress? A hard thing to know in
10:00
Speaker 1: the real world. How do you build that? What do
10:04
Speaker 1: you do?
10:05
Speaker 2: So the first thing is that you need data to
10:07
Speaker 2: learn from. Yeah, it's kind of obvious. So our first
10:11
Speaker 2: step was like, oh, we say, okay, we want to
10:14
Speaker 2: have data sets where we get a ton of information
10:16
Speaker 2: about each patient. What's that mean? That means that any
10:19
Speaker 2: individual time, I want to have a lot of different
10:22
Speaker 2: different measurements made on that patient at each time.
10:26
Speaker 1: So alsumably you want to have a lot of moments
10:29
Speaker 1: when lots of information exactly.
10:31
Speaker 2: You also want to have lots of.
10:32
Speaker 1: Lots of times over a long period of time, over
10:34
Speaker 1: a long period.
10:35
Speaker 2: Yeah, and so you know these are going to be
10:37
Speaker 2: for Alzheimer's. You're looking at a bunch of things related
10:39
Speaker 2: to the patient's cognitive performance on different assessments. Just also
10:45
Speaker 2: there's things about just their daily life. How are they
10:48
Speaker 2: able to function in their daily life. There's things related
10:51
Speaker 2: to their caregivers actually, like how does their caregiver rate
10:56
Speaker 2: their behavior? Brain imaging, blood tests, all that kind of information.
11:00
Speaker 2: You want to have as much of it about each patient.
11:02
Speaker 2: You want to have it as many times as possible. Sure,
11:05
Speaker 2: and we'll try to get that for you know, like
11:07
Speaker 2: fifty thousand people. And that's the kind of data set
11:11
Speaker 2: that we that we're starting with.
11:13
Speaker 1: And like, is there one repository that when you get that,
11:16
Speaker 1: you're like jackpot or what.
11:19
Speaker 2: No, we we have to aggregate data from lots and
11:23
Speaker 2: lots of different places to be able to build a
11:25
Speaker 2: big enough data set.
11:26
Speaker 1: Okay, so now you got the data, what do you
11:29
Speaker 1: do next?
11:30
Speaker 2: Then we got to train a model to to to
11:33
Speaker 2: be able to learn from those data how to simulate things.
11:37
Speaker 2: And now actually what we do.
11:38
Speaker 1: In particular in this case, how to predict, given some
11:42
Speaker 1: set of inputs for a patient, what's going to happen
11:45
Speaker 1: next exactly?
11:46
Speaker 2: And so this does look you were using that analogy
11:49
Speaker 2: of like a language model predicts the next word. So
11:52
Speaker 2: given these words I've seen before, predicts the next word.
11:55
Speaker 2: And that's that is similar to how our models and
11:57
Speaker 2: these diseases work. So we're going to say, given I've
11:59
Speaker 2: observed these things in the past about a patient, what
12:03
Speaker 2: will happen to them next? That is is very analogous
12:06
Speaker 2: to kind of what we're doing.
12:08
Speaker 1: It's okay, so you build the model, how does it work?
12:11
Speaker 1: How does it work in a clinical trial, specifically so
12:14
Speaker 1: that you know the people running the trial can can
12:17
Speaker 1: do it with fewer patients.
12:18
Speaker 2: Sure. So in a typical case, we're involved at the
12:25
Speaker 2: beginning of the clinical trial in the design of the protocol. Okay,
12:30
Speaker 2: So there's a question of how many patients should you
12:34
Speaker 2: randomize to your control group, how many patients do you
12:37
Speaker 2: need overall, and how many should be in the treatment,
12:40
Speaker 2: how many should be in.
12:40
Speaker 1: The control It's not always fifty to fifty.
12:43
Speaker 2: It's not always fifty to fifty in our studies. Our
12:46
Speaker 2: typical goal is to try to minimize the number of
12:49
Speaker 2: people that you need to put in the control group. Okay,
12:52
Speaker 2: And so we're involved in doing helping to do that
12:56
Speaker 2: calculation to say, here's how big your trial should be.
12:58
Speaker 2: And so then as patients enroll in the study, we
13:03
Speaker 2: take data from their first visit before they receive whatever
13:08
Speaker 2: new treatment they're going to receive and we take those data,
13:13
Speaker 2: we input them into our pre trained model. So I
13:15
Speaker 2: like to think about you know, CHATCHBTU give it a
13:17
Speaker 2: prompt and it gives us output. Same thing. We take
13:20
Speaker 2: the data from the patient, we prompt the model and
13:22
Speaker 2: it outputs their predictions for what will happen.
13:24
Speaker 1: In the And to be clear, you do that for
13:26
Speaker 1: all of the patients in both arms the treatments.
13:29
Speaker 2: Yes, yeah, and we don't know, right, it's blinded blind
13:32
Speaker 2: it's you, it's blinded to us. We don't know what. Yeah,
13:35
Speaker 2: So we do that for one hundred percent of the
13:37
Speaker 2: patients and then we give those data to the customer,
13:42
Speaker 2: to the farmer company.
13:44
Speaker 1: So then what happens next? What happens next?
13:46
Speaker 2: We wait around for a while. Yeah. And then when
13:50
Speaker 2: the study is actually completed, right, and they they they
13:53
Speaker 2: do unblind the data. We have to help to to
13:57
Speaker 2: say here's how you now can incorporate these these predicted
14:01
Speaker 2: outcomes into the analysis.
14:02
Speaker 1: Like so this is this is it. Now We're at
14:04
Speaker 1: the moment now when the thing you have built is useful.
14:07
Speaker 1: So so now it's it's they have done the study,
14:11
Speaker 1: they have the outcomes for the real human beings and
14:14
Speaker 1: they have the predicted outcomes from your model. How is
14:17
Speaker 1: your system? How's your model useful?
14:20
Speaker 2: So the very first thing that we're basically going to
14:22
Speaker 2: do is what I'm going to say, We're going to
14:24
Speaker 2: recalibrate our model. Recalibrate and you're going to figure out
14:29
Speaker 2: a relationship between your predicted outcomes and your observed outcomes
14:33
Speaker 2: for the patients who really received the placebo, for.
14:36
Speaker 1: The patients in the placebo group, And basically you're going
14:39
Speaker 1: to see how you did how do we do.
14:40
Speaker 2: Yes, and in particularly going to find out not just
14:43
Speaker 2: it's not like a measure of was it good or bad,
14:45
Speaker 2: You're going to find out exactly how are they related?
14:48
Speaker 2: And then you can take that information in adjust your predictions.
14:53
Speaker 2: Okay for everybody. So you can say, let's imagine that
14:57
Speaker 2: I find out, well, on average, I'm i underestimating how
15:03
Speaker 2: much a patient would progress by one point per year.
15:05
Speaker 2: I'm on average underestimating it. Well, then I'll go through
15:08
Speaker 2: and I'll take my prediction and I'll be like, well
15:10
Speaker 2: add one point, add one point forer you. And then
15:13
Speaker 2: now you have said, okay, well, now I've taken the
15:15
Speaker 2: model and I've been able to do it in such
15:18
Speaker 2: a way where I've fixed these mistakes by looking at
15:21
Speaker 2: the actual patients who got place ebo, And now I'm
15:23
Speaker 2: going to apply that model to the patient and the
15:25
Speaker 2: treatment group, and I'm going to look at Now, I
15:29
Speaker 2: just look at that difference between the patients and the
15:31
Speaker 2: treatment group and their predictions from the model, and I
15:33
Speaker 2: average that and I get an estimate for the treatment effect.
15:36
Speaker 2: Now that is described in a two stage procedure. That
15:40
Speaker 2: it's not actually a two stage procedure. It's one mathematical
15:43
Speaker 2: analysis that you do it. But the thing that's really
15:48
Speaker 2: I think quite amazing actually is that this has a
15:53
Speaker 2: bunch of mathematical guarantees to it. We can actually prove
15:58
Speaker 2: that the estimate that you get for how effective the
16:01
Speaker 2: treatment is is still unbiased. So it's not an overestimate,
16:06
Speaker 2: it's not under ustan, it's on average correct. Can prove
16:10
Speaker 2: that if you compute a P value from the analysis
16:12
Speaker 2: like you would typically do, that it has exactly the
16:15
Speaker 2: right properties as it does out of a regular RCT.
16:17
Speaker 1: P value is roughly the probability that the funding was
16:20
Speaker 1: a fluke.
16:22
Speaker 2: Ye right, Yeah. If you compute an arabar the arabar
16:25
Speaker 2: you get from our analysis the air bar you would
16:27
Speaker 2: get from a normal there. They all have exactly identical statistics.
16:31
Speaker 1: This is not intuitive, but but you're saying, the mathematical
16:35
Speaker 1: fact is that it works. Yes, And just to be clear,
16:40
Speaker 1: what this allows you or the people running the trial
16:42
Speaker 1: to do is to enroll fewer people in the placebo
16:46
Speaker 1: arm not none, but fewer than they otherwise would have
16:49
Speaker 1: had to get the same amount of statistical power. Right,
16:52
Speaker 1: that is the bottom line thing that you are delivering. Yes,
16:55
Speaker 1: that's correct, And it's something like a quarter or a
16:58
Speaker 1: third less, is that right? Yeah?
17:00
Speaker 2: So it depends on how accurate our models are. The
17:04
Speaker 2: more accurate the model is, the fewer patients you need
17:06
Speaker 2: in your placebo group. Sure so typically right now, yet
17:10
Speaker 2: somewhere between like a quarter, like fifty percent. It depends
17:13
Speaker 2: on the specific details.
17:15
Speaker 1: So tell me what is the effect of that at
17:19
Speaker 1: a macro scale? What does it mean to say a
17:21
Speaker 1: drug company can get the same statistical power by enrolling
17:26
Speaker 1: twenty five percent fewer people in their study, specifically in
17:30
Speaker 1: the placeboar.
17:31
Speaker 2: Well, I think that there are two things. First is
17:35
Speaker 2: I think people don't always understand how expensive clinical trials
17:39
Speaker 2: are you know, companies are paying one hundred sometimes two
17:43
Speaker 2: hundred thousand dollars per patient in one of their clinical trials,
17:46
Speaker 2: So finding and enrolling and monitoring a patient for all
17:49
Speaker 2: that time is very, very expensive. It also just takes
17:52
Speaker 2: a long time to find people who are willing to participate.
17:55
Speaker 2: And so if you're talking about a large phase three trial,
17:58
Speaker 2: reducing the size of the control group by twenty five percent,
18:02
Speaker 2: that might mean like one hundred fewer patients that you
18:04
Speaker 2: need to actually recruit and enroll in your study, and
18:07
Speaker 2: that that could be like a year. But you know,
18:09
Speaker 2: so you can save six months to a year off
18:12
Speaker 2: of your total clinical trial timeline. That means a lot, right,
18:16
Speaker 2: but both for patients. If the drug is actually successful,
18:19
Speaker 2: that's a year faster it gets to market. And you know,
18:24
Speaker 2: for the farmer company, that's office a big value proposition
18:27
Speaker 2: being able to get the drug to market a year faster.
18:35
Speaker 1: In a minute, moving from clinical trials to individual patients,
18:47
Speaker 1: now back to the show. What is the what's the
18:53
Speaker 1: big picture? Where are you trying to get to and
18:55
Speaker 1: you know, in the medium termament in the long term, So.
19:02
Speaker 2: The ability to understand what a person's health outcome is
19:06
Speaker 2: going to be under different scenarios. This is I think
19:09
Speaker 2: what's really important. Is it not just hey, given that
19:12
Speaker 2: they would get a placebo, what's going to happen to
19:14
Speaker 2: the health outcomes? That's nice for clinical trials, but we
19:16
Speaker 2: want to know, hey, there's ten different treatment options for
19:19
Speaker 2: this patient, and if I were to give them each
19:22
Speaker 2: one of these different treatment options, what would their health
19:24
Speaker 2: outcomes look like in those different scenarios.
19:27
Speaker 1: So there you're also moving out of the clinical trial
19:30
Speaker 1: into the realm of like a doctor seeing a patient.
19:32
Speaker 1: Let's just be very clear, like that that's a huge leap,
19:36
Speaker 1: and like that's what you're talking about.
19:37
Speaker 2: I think that there's a really good pathway to being
19:42
Speaker 2: able to build these things and make them useful for
19:47
Speaker 2: problems that are at the individual patient level.
19:50
Speaker 1: And is the narrow way to think about it, Like
19:53
Speaker 1: before you get to the magical computer that can predict
19:56
Speaker 1: everything for everybody, that you get to a very very
19:59
Speaker 1: good model that can predict for individuals in certain circumstances
20:04
Speaker 1: a certain set of outcomes. So, for example, you might
20:06
Speaker 1: have a very very good Alzheimer's model for certain patients
20:10
Speaker 1: at a certain stage of disease. This model is very
20:13
Speaker 1: powerful at the level of the individual. Is that the
20:15
Speaker 1: way to think about it, Yeah, the way I'll tell you.
20:18
Speaker 2: The way I think about it. I think that the
20:20
Speaker 2: most important thing that models can do, which actually things
20:23
Speaker 2: like a chat ept are not good at, is that
20:26
Speaker 2: they can give you really well calibrated estimates of their
20:33
Speaker 2: own confidence. That's the most important thing that a model
20:37
Speaker 2: can do, because, like we said earlier, health is stochastic.
20:43
Speaker 2: There are all kinds of things that happens fundamentally exactly right.
20:50
Speaker 2: And so you know, we're going to make a prediction
20:53
Speaker 2: about somebody in the future, and sometimes we're going to
20:56
Speaker 2: be really confident in that prediction and then it's actionable,
20:59
Speaker 2: but sometimes you're not. It's not you're not confident, and
21:02
Speaker 2: maybe it's not actionable because you're really unconfident. And the
21:06
Speaker 2: most we're never going to get to the point that's
21:08
Speaker 2: going to say, hey, you're going to have a heart
21:10
Speaker 2: attack on July seventeenth of twenty thirty seven. It's like,
21:15
Speaker 2: it's never going to be like that detail. But the
21:17
Speaker 2: point question is can you believe the model's estimates of
21:22
Speaker 2: its own confidence? And if you can, then you when
21:25
Speaker 2: it is confident, you can act on it, and when
21:27
Speaker 2: it's not confident, you can do other things. And that's
21:29
Speaker 2: the that's so it's actually a really key technical thing,
21:32
Speaker 2: and we know what we need to work on.
21:34
Speaker 1: If I were going to answer pomorphis it, I'd be like,
21:36
Speaker 1: it's like a it's like a humility. It's like an
21:38
Speaker 1: epistemic humility, Like it knows what it doesn't know.
21:41
Speaker 2: It knows what it doesn't know, and it will tell
21:43
Speaker 2: you like I, yeah, here's my prediction. But yeah, exactly
21:49
Speaker 2: So if you can get it to that point where
21:51
Speaker 2: we were, where it's well calibrated that way, then they
21:54
Speaker 2: become really really useful for a whole bunch of things.
21:59
Speaker 2: And it's not going to say.
21:59
Speaker 1: Become probably useful if they can have a relatively high
22:03
Speaker 1: degree of certainty about at least some things, right yea, just.
22:06
Speaker 2: Like yeah, it's not very course, yeah, but exactly so.
22:11
Speaker 2: I think that that's the most important thing for these
22:16
Speaker 2: applications of AI in medicine is to have models that
22:19
Speaker 2: are going to be able to do that effectively.
22:22
Speaker 1: If everything goes well, what problem will you be trying
22:25
Speaker 1: to solve in five years?
22:28
Speaker 2: In five years, I hope that we are rolling out
22:31
Speaker 2: something that is a model for everything. That's what we
22:35
Speaker 2: want to be rolling out, not this one disease at
22:37
Speaker 2: a time thing, but one model for all disease. And
22:39
Speaker 2: the reason why I really want to do this is
22:43
Speaker 2: because if it's one model per disease, I need a
22:46
Speaker 2: ton of data on that disease, a ton. So we
22:48
Speaker 2: can work on these areas like Alzheimer's where I can
22:50
Speaker 2: get data from fifty thousand patients, But how do I
22:53
Speaker 2: work on the disease where I have fifty patients fifty
22:56
Speaker 2: patients in the world who have this rare disease. Those
22:58
Speaker 2: are really really important things. And the only way that
23:02
Speaker 2: we're going to be able to do that is to
23:03
Speaker 2: unlock a new kind of capability in our models to
23:06
Speaker 2: learn from a handful of examples. And so this is
23:11
Speaker 2: this is to me, the next frontier for our work
23:15
Speaker 2: is figuring out how we can do that and then
23:18
Speaker 2: bring that to market, because it opens up the ability
23:21
Speaker 2: to work on rare diseases that are really really important
23:24
Speaker 2: market very difficult to develop drugs for. And it's and
23:29
Speaker 2: again I'm I'm you know, as a scientist, I'm drawn
23:31
Speaker 2: to the technical challenges. Those are the things that.
23:34
Speaker 1: It seems so hard, right, I mean, it seems like
23:37
Speaker 1: this really basic insight about genitive models is that like
23:44
Speaker 1: gigantic amounts of data feeding. You know, for a language model,
23:47
Speaker 1: you feed it the whole internet is the way to
23:49
Speaker 1: get it to understand how language works. And so how
23:53
Speaker 1: how can you do something for fifty people?
23:55
Speaker 2: Like?
23:55
Speaker 1: How how that in five years?
23:58
Speaker 2: Yeah, it's really hard. How the analogy is actually perfect? Okay,
24:02
Speaker 2: if you want to build what we've learned is that
24:05
Speaker 2: if you want to build a really amazing language model
24:08
Speaker 2: that's really specific to some domain, so you only want
24:13
Speaker 2: a language model that's really good at biophysics, it knows
24:16
Speaker 2: biophysics really well. Would you be better off training a
24:19
Speaker 2: model trying to find as much biophysics as you can
24:22
Speaker 2: and training it on that or just training a model
24:24
Speaker 2: on the entire init And what we've learned is much
24:27
Speaker 2: better to train a model on the entire init that
24:29
Speaker 2: there's a lot of things that transfer from one domain
24:34
Speaker 2: to another. And so what we can do now is
24:36
Speaker 2: say we train the model on the whole Internet, and
24:38
Speaker 2: we have one biophysics paper, and we give it that
24:42
Speaker 2: one or two papers on the background of all of
24:46
Speaker 2: the knowledge from everywhere else, and that's much better than
24:49
Speaker 2: trying to get lots and lots of biophysics papers. So
24:51
Speaker 2: the analogy works perfectly in the exact same direction. That's
24:54
Speaker 2: the whole point. We want to be able to take
24:56
Speaker 2: all of the world's Imagine taking a model that has
24:59
Speaker 2: all of the world's health data and putting all of
25:03
Speaker 2: that into one So what seen everything and it can
25:05
Speaker 2: now draw analogies between because there's a lot of things
25:08
Speaker 2: you think about, like Parkinson's and Alzheimer's, they have a
25:11
Speaker 2: lot of similarities, Huntington's a lot of similarities. So why
25:14
Speaker 2: aren't we drawing kind of information or knowledge from one
25:18
Speaker 2: disease area and using it to inform another because they
25:20
Speaker 2: are similar. And so I'm allowing a model to have
25:24
Speaker 2: access to all of the data and figure out how
25:26
Speaker 2: to do it. I think is the right path forward.
25:29
Speaker 2: So is that.
25:32
Speaker 1: Wildly capital intensive? Like what do you actually do to
25:36
Speaker 1: do that? You just get all the health data about
25:38
Speaker 1: all the people you can and say to the machine
25:40
Speaker 1: figure it out, Like what do you do?
25:43
Speaker 2: Yeah? Yes, I mean the first step for us is
25:48
Speaker 2: you need to get a lot of data. The biggest
25:50
Speaker 2: thing is that we need to figure out a way
25:54
Speaker 2: to have the model map all of those data to
25:58
Speaker 2: the same representation.
26:00
Speaker 1: What does that mean, map all of those data to
26:02
Speaker 1: the same representation.
26:04
Speaker 2: So let's imagine that there is some unobservable state of
26:10
Speaker 2: a person which just describes their health. We can't actually
26:13
Speaker 2: observe it directly. It's we don't exactly know what it is,
26:16
Speaker 2: but we can make these measurements of it that tell
26:19
Speaker 2: us something about that underlying state. So I can measure BMI,
26:23
Speaker 2: I can measure heart rate, I can measure all the
26:25
Speaker 2: I can measure all of these different things. And what
26:27
Speaker 2: we want to be able to do is instead of
26:30
Speaker 2: working in the world of measurements, which is where we
26:32
Speaker 2: currently work, we want to be able to work at
26:34
Speaker 2: that underlying unobservable state because if you can, if you
26:37
Speaker 2: can see that, if you could reach through into that
26:40
Speaker 2: underlying state, you can answer any question about any.
26:43
Speaker 1: Patient's health, like like like a number like this one
26:46
Speaker 1: state that is just like one.
26:48
Speaker 2: High dimension the high dimension, right right, Well, okay, yeah,
26:56
Speaker 2: so I mean yeah, but basically talking about is there
26:58
Speaker 2: some vector, some really high dimensional space where we're able
27:01
Speaker 2: to take all diseases and look at them how they're
27:04
Speaker 2: related to each other in this really high dimensional space.
27:07
Speaker 2: That is the way language models work. That's exactly how I.
27:10
Speaker 1: Love And that's intense, Like, that's pretty far out right.
27:15
Speaker 1: Doesn't that feel far out too?
27:17
Speaker 2: I would say, talk like a hippie, But I if
27:20
Speaker 2: I describe this to a machine learning researcher, they're like,
27:23
Speaker 2: that sounds exactly like what you should do. So it
27:26
Speaker 2: doesn't seem far out to me. It seems it seems
27:29
Speaker 2: very clear that that's the direction that we should be
27:31
Speaker 2: taking things.
27:32
Speaker 1: And does five years seem like a like you might
27:34
Speaker 1: actually do it in five years.
27:36
Speaker 2: Yeah, we were hoping to be able to have a
27:40
Speaker 2: version next year. That's a pan neuroscience model, so we're
27:43
Speaker 2: starting with all. So we're starting with start with something
27:47
Speaker 2: more attractable, build a more tractable thing. So right now
27:50
Speaker 2: we're working on a neuroscience model. So we're hoping, I
27:53
Speaker 2: mean which to be totally. This might not work. This
27:56
Speaker 2: is a research idea, right, so it may work, it
27:58
Speaker 2: might not work. But that's you ask where I would
28:00
Speaker 2: hope to be. That's where I hope to be is
28:02
Speaker 2: that we're able to solve those those problems.
28:08
Speaker 1: So we'll be back in a minute. With the Lightning Round,
28:09
Speaker 1: including what Charles learned when he worked as an ice
28:12
Speaker 1: hockey wrap back to the show. I'm going to finish
28:22
Speaker 1: with the Lightning Round. Will just be a few more minutes.
28:24
Speaker 2: Okay.
28:26
Speaker 1: As the name suggests, I've heard you say that you
28:30
Speaker 1: read academic preprints, which is basically studies that are about
28:34
Speaker 1: to be published, that you read them every day. What's
28:37
Speaker 1: one you read recently that you found particularly interesting?
28:41
Speaker 2: Recently? There have been a number of papers that I've
28:44
Speaker 2: been reading around different ways of so training, the kind
28:50
Speaker 2: of neural networks that we use. All of them use
28:54
Speaker 2: a particular algorithm that people call ADAM. It's been used
28:58
Speaker 2: for a really long time, like everyone uses it now,
29:02
Speaker 2: and it has I don't know, it has some problems.
29:06
Speaker 2: There's a paper that was just really recently on a
29:08
Speaker 2: new algorithm people call LION. I don't know what it
29:10
Speaker 2: stands for. L I O N stands for something. And
29:13
Speaker 2: this was a discovered So they used a machine learning
29:17
Speaker 2: out a reinforcement learning algorithm to discover a new kind
29:20
Speaker 2: of optimizer.
29:22
Speaker 1: So if this works, if Lion is better than ADAM,
29:26
Speaker 1: will it be like machine learning figuring out a better
29:29
Speaker 1: way to build machine learning. Is that what's happening here?
29:32
Speaker 2: Yeah, that's what people are working on exactly.
29:34
Speaker 1: This is like the takeoff. This is like the moment
29:36
Speaker 1: when GPT five builds GBT six or whatever.
29:39
Speaker 2: I think the claim is it's like five percent better something.
29:42
Speaker 2: It's not. It's not.
29:44
Speaker 1: Yes, Lion couldn't find the.
29:46
Speaker 2: Time another thing yet Yeah. So yeah, that was a
29:49
Speaker 2: paper I read really recently.
29:52
Speaker 1: If you couldn't work in AI, what field would you
29:54
Speaker 1: work in?
29:58
Speaker 2: If I couldn't work in AI. Uh, I guess I
30:03
Speaker 2: would probably try to work in energy, maybe tim a
30:09
Speaker 2: change something related to that.
30:12
Speaker 1: I think seeing bummed at the prospect of not being
30:14
Speaker 1: able to work in AI, I appreciate that. I don't
30:16
Speaker 1: want to make it.
30:17
Speaker 2: I'm very bummed. Yeah, you know, I think it's the
30:20
Speaker 2: most exciting thing that's happened on Earth since the Industrial Revolution.
30:25
Speaker 2: So it's a new industrial revolution. Yeah.
30:28
Speaker 1: Weirdly, you used to work at a virtual reality hardware company.
30:33
Speaker 1: I feel like VR is always about to break through,
30:37
Speaker 1: you know, like Apple just had this big announcement, had
30:39
Speaker 1: a Facebook did a while ago, but yet it never
30:42
Speaker 1: quite happens. Why not, Like, why are we not doing
30:45
Speaker 1: this interview in the metaverse.
30:48
Speaker 2: So I only worked at that company for a few months.
30:52
Speaker 2: I spent my whole career working in biophysics. I moved
30:56
Speaker 2: to Pfiser. I was working at Pfiser, and then I
30:59
Speaker 2: got im just like, I'm gonna try something totally different,
31:02
Speaker 2: and I went and tried this work at the VR company.
31:05
Speaker 2: I was interested in that because of the underlying technical
31:08
Speaker 2: problems research that I had to do, not because I
31:11
Speaker 2: was drawn to the product. I have only ever used
31:15
Speaker 2: a virtual reality headset twice my entire life. Once was
31:20
Speaker 2: in the interview for that job, and once was testing
31:23
Speaker 2: something while I was working at that job. I'm not
31:26
Speaker 2: interested in it, so you want to know I was
31:29
Speaker 2: interested in the engineering. So you want to know why
31:31
Speaker 2: I don't think it's taken off. Is because most people
31:34
Speaker 2: don't have a compelling reason to use it. Neither do I. Yeah,
31:38
Speaker 2: what'd you learn working as an ice hockey referee? Ice
31:41
Speaker 2: hockey referee? Oh, that was like my super super young job.
31:47
Speaker 2: I would say that I learned it's best not to
31:50
Speaker 2: call penalties on little children. That's what I learned. You know,
31:57
Speaker 2: people would just like like run into each other and
31:59
Speaker 2: they'd fall down. You're like, is that a penalty. Was
32:01
Speaker 2: it on purpose? Not on purpose? If you call a penalty,
32:04
Speaker 2: the parents are going to be real upset at you.
32:05
Speaker 2: So you just just let them play.
32:07
Speaker 1: Good early experiments, cost benefit analysis.
32:10
Speaker 2: Just let them play.
32:17
Speaker 1: Charles Fisher is the co founder and CEO of Umler.
32:21
Speaker 1: Today's show was edited by Sarah Nis, produced by Gabriel
32:24
Speaker 1: Hunter Chang and Edith Russlo, and engineered by Amanda k Wong.
32:30
Speaker 1: I'm Jacob Goldstein. One last note, the show is going
32:33
Speaker 1: to be off for the next several weeks and we'll
32:35
Speaker 1: be back with new episodes in August. Have a rad summer.