Latest / Code & Cure / #43- AI Hype Vs Real-World Medicine

Transcript

#43- AI Hype Vs Real-World Medicine · Code & Cure

0:00
SPEAKER_00: Can you believe this?
0:02
SPEAKER_00: AI outperformed doctors in a Harvard trial of emergency triage diagnoses.
0:09
SPEAKER_01: What hello and welcome back to Code and Cure, the podcast where we discuss decoding health in the age of AI.
0:25
SPEAKER_01: My name is Vasant Sarathy.
0:27
SPEAKER_01: I'm an AI researcher and cognitive scientist.
0:30
SPEAKER_01: And I'm here with Laura Hagopian.
0:32
SPEAKER_00: I'm an emergency medicine physician.
0:34
SPEAKER_00: Can you believe it?
0:35
SPEAKER_00: I I can't believe it, actually.
0:36
SPEAKER_00: I feel like it's um maybe a little bit overhyped.
0:39
SPEAKER_01: Yeah, I think the point of this podcast was don't believe it.
0:42
SPEAKER_00: Be a little bit careful when you see a headline that's like, whoa, oh my gosh.
0:48
SPEAKER_01: Yeah.
0:48
SPEAKER_01: And I think this is an important topic.
0:51
SPEAKER_01: There's, you know, hype with anything, right?
0:52
SPEAKER_01: Any new technology, any new product feels magical at the beginning.
0:56
SPEAKER_01: People are excited.
0:56
SPEAKER_01: There's a whole novelty aspect to it.
0:58
SPEAKER_01: Um, but this one has stuck around for a little bit and it's influencing things that are potentially critical and important.
1:07
SPEAKER_01: And, you know, it's not just a fun little toy that people are excited about.
1:11
SPEAKER_00: Um and for me, the headlines hit kind of close to home because I'm like, oh, this is my job.
1:15
SPEAKER_00: Is AI better than me at my job?
1:17
SPEAKER_00: But I do think there's this piece of hey, this the scientific paper came out.
1:22
SPEAKER_00: It was really interesting.
1:23
SPEAKER_00: And I'm glad that we're gonna dig into it um later today.
1:27
SPEAKER_00: But there's also this component of, okay, you know, NPR, Harvard magazine, all of these places are writing about this article and trying to simplify it and also trying to create a headline that you're gonna click on.
1:41
SPEAKER_00: Yeah.
1:42
SPEAKER_00: Yeah.
1:42
SPEAKER_00: And sometimes people just read the headlines.
1:44
SPEAKER_00: I do that.
1:44
SPEAKER_00: Sometimes I'm like, oh, that's a cool headline.
1:46
SPEAKER_00: I don't need any more information.
1:48
SPEAKER_00: But for this one, I was like, oh my gosh, I'm not sure I believe this headline.
1:52
SPEAKER_00: Let's read the paper.
1:53
SPEAKER_01: Yeah.
1:53
SPEAKER_01: And you know, I think what's going to be interesting for us as we talk through this is to think about all the players involved, what the incentives are for different people, what drives people to write these different types of, you know, uh different types of explanations of the science that that that's that's uh that's there.
2:11
SPEAKER_01: I think it's I think it's fascinating and potentially a little bit scary, but hopefully, you know, we can figure this out.
2:17
SPEAKER_00: Um so when we were prepping for this podcast, you were like throwing out terms, Vassant, like, oh, the the AI summer or the winter.
2:25
SPEAKER_00: And I was like, what what are you talking about?
2:27
SPEAKER_00: But like this has happened before, right?
2:29
SPEAKER_00: Yeah.
2:30
SPEAKER_00: And so I'd love to hear what what is the summer and winter, other than like the regular seasons, and and what has happened before.
2:37
SPEAKER_01: Yeah.
2:37
SPEAKER_01: So AI, so I want to do a little bit of a history lesson.
2:40
SPEAKER_01: So AI has a very interesting history.
2:41
SPEAKER_01: The term AI was coined a long time ago, but um in the in the 50s, and there was there's always been a sense that we're almost there, that we have solved all of AI.
2:51
SPEAKER_01: And that's always been the case.
2:53
SPEAKER_01: So there was a there was a group of researchers back then who uh were the leading AI researchers, and they held a a conference at Dartmouth um where they decided uh this is a great example, and this is very famous.
3:05
SPEAKER_01: Most AI people already know this, but for those who are not, I'm not an AI person.
3:09
SPEAKER_00: You're gonna have to tell me.
3:10
SPEAKER_01: So it's called the Dartmouth Conference because it's really interesting because during that conference, all the top AI researchers who are who were the best minds to think about the challenges of AI, decided that um computer vision was an intern project for the summer.
3:26
SPEAKER_01: For as a summer intern project to solve computer vision.
3:28
SPEAKER_01: And I say computer vision, I mean computers being able to see things and make sense of it.
3:33
SPEAKER_01: And they really, honest to God, believed that that was a summer intern project.
3:38
SPEAKER_00: Like simple.
3:39
SPEAKER_00: They thought that that was a simple thing.
3:41
SPEAKER_01: Simple enough to do it quickly.
3:42
SPEAKER_01: I mean, not literally summer intern, maybe, but but but quick enough.
3:45
SPEAKER_01: Now that was in the that was a 70 years ago.
3:49
SPEAKER_01: And computer vision is an entire field of research with thousands upon tens of thousands upon researchers working, still working hard to figure out how we can get computers to see the way we do.
4:02
SPEAKER_01: And we've we have achieved great progress and we have all kinds of really cool tools.
4:06
SPEAKER_01: But again, back then, that's what those scientists, those people actually believed was a solvable problem.
4:12
SPEAKER_01: Now you can fast forward this to the 1980s.
4:14
SPEAKER_01: Uh similar sort of thing happened when the perceptron was invented.
4:18
SPEAKER_01: Um is the precursor to the neural networks that we all know now.
4:24
SPEAKER_01: Um, but it's a type of technology that kind of set the stage for what um uh would be a future neural network.
4:31
SPEAKER_01: In 1985, there was a New York Times article that the headline of which was Electronic Brain Teaches Itself, where they claimed that the Navy had demonstrated uh this perceptron and that it was the first non-living mechanism able to perceive, recognize, and identify its surroundings without human training or control.
4:51
SPEAKER_01: This is from the article.
4:52
SPEAKER_01: I'm reading this from the article.
4:53
SPEAKER_01: And even the scientist, uh Dr.
4:55
SPEAKER_01: Rosenblatt, um, you know, came out and said, we just one more step of development, a difficult step, he said, is needed for the device to hear speech in one language and instantly translate it to speech or writing in another language.
5:08
SPEAKER_00: Okay, that did not like happen in 1986 for all our listeners.
5:11
SPEAKER_01: Right, right, right, exactly, exactly.
5:14
SPEAKER_01: So um this notion of AI hype is is is been there always, and there's been repeated rounds of this.
5:20
SPEAKER_01: So um AI researchers like to call this in terms of seasons, an AI summer is when the AI hype is at its best.
5:27
SPEAKER_01: All the money and funding for AI research is pouring in.
5:30
SPEAKER_01: Research is able to work on things, people are talking about it, there's New York Times articles about it and such.
5:36
SPEAKER_01: And in the past, there's been multiple AI summers, and those have been followed by AI winters.
5:42
SPEAKER_00: So we're like in an AI summer right now, right?
5:44
SPEAKER_00: I would think so.
5:45
SPEAKER_00: Like, oh, everyone's using LLMs, it's in the news all the time, tons of stuff is getting funded, including by the government, etc.
5:52
SPEAKER_01: Yes.
5:52
SPEAKER_01: And um, but those AI summers in the past have been followed by AI winters.
5:58
SPEAKER_01: And AI winters coming.
6:00
SPEAKER_01: Winter is coming, right?
6:02
SPEAKER_01: Um, AI winters are basically when the world realized that the AI system can't do what everyone's been promising it can do.
6:10
SPEAKER_01: And given the state of current technol current technology then, that it wouldn't be able to do that, we would need a big big change.
6:16
SPEAKER_01: And so then the funding quickly dried up, all these promises went away, and AI research just dropped dramatically.
6:23
SPEAKER_01: Uh, and then, you know, over time it would sort of pick back up again and proceed.
6:27
SPEAKER_00: And the spring and fall seasons do exactly.
6:29
SPEAKER_01: I suppose, I suppose, but we just went from summer to winter, you know, and so let's just let's just it does happen pretty suddenly, and people realize those promises are not, uh can't be fulfilled.
6:40
SPEAKER_00: And so we go from like, hey, this is magic, it solves all the problems, to like, hey, it actually didn't do it didn't do what we thought it was doing.
6:48
SPEAKER_00: Yeah, we thought it could do.
6:50
SPEAKER_01: Now, those times that it happened before are potentially different, right?
6:54
SPEAKER_01: There was no internet in a lot of these previous times.
6:56
SPEAKER_01: There was not spread of information the way it is now.
6:58
SPEAKER_01: So the hype hype uh mechanisms are different and the and the follow-throughs are different, and the funding cycles are all different.
7:05
SPEAKER_01: So I'm not saying that a some winter is going to follow this summer.
7:09
SPEAKER_01: Um, but I will say one thing, which is people are arguing about it.
7:12
SPEAKER_01: And there is an issue, which is um, you know, a recent study came out about open AI, which said hundreds of billions of dollars have been spent by open AI and they haven't yet returned a profit.
7:21
SPEAKER_01: And this is ChatGPT.
7:23
SPEAKER_01: Right.
7:24
SPEAKER_01: And it wasn't enough that they have hundreds of millions of users, they hadn't reached a billion users, and that wasn't enough.
7:29
SPEAKER_01: And AI is still um adopted by a very small percent of the world.
7:34
SPEAKER_01: And AI, although it's got all this excitement and potential value, um there it people have yet to see full returns in terms of its actual value.
7:43
SPEAKER_01: There's still questions of trustworthy.
7:45
SPEAKER_01: We've talked about this in the podcast.
7:46
SPEAKER_01: There's questions of trustworthiness, there's questions of consistency, there's hallucinations, all these things that AI systems do are preventing people from fully taking it on and engaging with it.
7:57
SPEAKER_01: That said, that doesn't mean that we're gonna have an AI winter.
8:00
SPEAKER_01: It just means it could mean this time around that it's you know, it's less of a bubble popping and more of just like a slow transition from you know seeing AI as this magical silver silver bullet that's gonna solve everything to a more narrow tool that can do one or two good things, uh, one or two things good, right?
8:15
SPEAKER_01: Really well.
8:16
SPEAKER_01: So that's kind of uh you know that could happen.
8:19
SPEAKER_01: But the point is there's still the hype cycles, and we're in one right now.
8:22
SPEAKER_01: There's a massive hype cycle right now that's happening, telling us everything about what AI can do.
8:27
SPEAKER_01: And the article that we're gonna talk about today does that to some degree.
8:30
SPEAKER_00: Uh, not the article it's like the news articles about the article did that, right?
8:34
SPEAKER_00: It's like the article is very interesting and it actually called it, they do a very good job in the article about calling out some of the limitations.
8:43
SPEAKER_00: But when you look at the news headlines about it, it's like, oh, in real world tests, an AI model did better than ER doctors at diagnosing patients.
8:52
SPEAKER_00: And AI outperforms doctors.
8:54
SPEAKER_00: That's what a new Harvard study shows.
8:56
SPEAKER_00: And so when you just see the headlines, you're like, oh, like, do I even need to go to the doctor anymore?
9:01
SPEAKER_00: Can I just can I just go on to Chat GPT and ask?
9:04
SPEAKER_00: And and part of this was that they were kind of testing out a new model.
9:08
SPEAKER_00: And the new model, you know, generally did do better than older models.
9:13
SPEAKER_00: Um, but I think that what the study actually said is a is a bit different than what the headlines said it said.
9:22
SPEAKER_01: Yeah, exactly.
9:23
SPEAKER_01: So maybe let's start with the actual study, right?
9:25
SPEAKER_01: What did what did the actual study do?
9:27
SPEAKER_00: Well, in the study, they compared human experts and AI, you know, uh LLMs, large language models, and basically looked at how they each performed.
9:41
SPEAKER_00: And they used a set of two internal medicine doctors for the comparison.
9:44
SPEAKER_00: And they were looking at like how how good of a job did it do at diagnosing what a patient had at triage, um at the level of the ER physician, and at the level of admission.
9:56
SPEAKER_00: And this was a subset of patients who were kind of on the sicker side and all got admitted to the hospital.
10:00
SPEAKER_01: So this is both diagnostic diagnostic test tasks and management tasks that required some some sort of thinking and reasoning.
10:07
SPEAKER_00: Absolutely required clinical reasoning to do.
10:10
SPEAKER_00: And so they used a couple different, you know, LLM models, and then they had two internal medicine doctors um kind of see what they could figure out.
10:21
SPEAKER_00: And as these cases went along, from triage to seeing the ER doctor to admission to the hospital, more information becomes available, right?
10:30
SPEAKER_00: More tests are done, et cetera.
10:31
SPEAKER_00: So to try to figure out, okay, what could be what are the list of things that could be going on, and they narrowed that down to just five.
10:37
SPEAKER_00: You're only allowed to get five, and then like what's your top diagnosis?
10:40
SPEAKER_00: Um, and so in certain cases, you know, they were like, hey, the the model does better.
10:47
SPEAKER_00: And in other cases, they were like, hey, the model kind of did not outperform.
10:51
SPEAKER_00: It did about the same as prior models, especially when it came to, you know, uh diagnoses that you really don't want to miss, or like like landmark cases that it was given.
11:02
SPEAKER_01: So I guess the question I have is just on the process, um, is that I mean, you have experiences being an ER doctor, and that experience is that tracking well here?
11:13
SPEAKER_01: Because it seems like they were having people to diagnose and then they require them to only give back fat.
11:18
SPEAKER_01: I mean, what those some those seem kind of arbitrary.
11:20
SPEAKER_01: I don't know how closely they tie to can you talk a little bit about that?
11:24
SPEAKER_00: Yeah.
11:24
SPEAKER_00: I mean, I think that's one of the problems that I have with this paper because as an ER doctor, I wouldn't limit myself to five.
11:32
SPEAKER_00: So we would come up with differential diagnoses, right?
11:35
SPEAKER_00: And so what that means is like, hey, here's a list of what things this could be.
11:40
SPEAKER_00: So say you showed up to my ER with chest pain.
11:43
SPEAKER_00: Okay.
11:44
SPEAKER_00: Okay.
11:45
SPEAKER_00: Your chest pain could be a heart attack.
11:47
SPEAKER_00: It could be a blood clot in your lung, it could be like something terrible, or maybe you worked out too hard and you strained your chest wall muscle, or maybe you have acid reflux.
11:56
SPEAKER_00: And I could go on and on and on and make a giant list of things, some of which are life-threatening, like potentially like a trauma with a popped lung or something like that.
12:06
SPEAKER_00: And some of them are much more benign, like acid reflux.
12:10
SPEAKER_00: And my goal as an ER doctor was at first especially to triage just to figure out hey, what are the dangerous things it could be?
12:19
SPEAKER_00: And and those would often be higher up on my list because I want to rule them out.
12:22
SPEAKER_00: It doesn't mean that they're more likely necessarily, but it means that, like, hey, you're you showed up to the ER, you think you're having an emergency.
12:31
SPEAKER_00: How can I rule out or how can I check if you have this really bad thing?
12:35
SPEAKER_00: Yeah, yeah.
12:36
SPEAKER_00: Like, do you have risk factors for a blood clot?
12:38
SPEAKER_00: Do you, you know, do you have a history of hypertension, hyperlipidemia, diabetes, family history of heart disease?
12:44
SPEAKER_00: That would make me think that you could be having a heart attack.
12:47
SPEAKER_00: So, so those are the things that end up kind of top of my list because I want to rule those out.
12:52
SPEAKER_01: Yeah.
12:53
SPEAKER_00: And then I'm figuring out, okay, like how sick is this person?
12:56
SPEAKER_00: If your vital signs are abnormal, I'm gonna want to see you sooner and figure out what's going on.
13:01
SPEAKER_00: Like, oh, you know, could you be having an aortic dissection?
13:05
SPEAKER_00: Versus, are things looking more benign?
13:08
SPEAKER_00: And so part of this is like, hey, do we think you need to be admitted?
13:12
SPEAKER_00: Do you need to be seen quickly?
13:13
SPEAKER_00: Can you wait in the waiting room for a couple hours?
13:16
SPEAKER_00: Um, you know, do you need to go to the ICU?
13:18
SPEAKER_00: Like, do you need a surgery?
13:20
SPEAKER_00: So, so a lot of the ER doctor's job is not necessarily honing in on a single diagnosis or even five diagnoses right away, right?
13:28
SPEAKER_00: But figuring out, hey, is this person sick or not sick?
13:31
SPEAKER_00: What tests do we want to run on them to figure out what's going on?
13:35
SPEAKER_00: Right.
13:35
SPEAKER_00: And like what sort of management do we want to do for them in this moment and what's their sort of disposition going to be?
13:40
SPEAKER_00: Are they gonna kind of stay?
13:42
SPEAKER_00: Are they gonna go home?
13:43
SPEAKER_00: Do we not know yet?
13:44
SPEAKER_00: Et cetera.
13:45
SPEAKER_00: And I think that's a really important piece here because they were saying, oh, well, it's the AI models are better at ER triage at coming up with the diagnosis.
13:56
SPEAKER_00: And like my argument would be like, that's not the goal of ER triage.
13:59
SPEAKER_00: You're not supposed to come up with a single diagnosis or even just five there.
14:03
SPEAKER_00: Yeah.
14:03
SPEAKER_00: You're supposed to figure out, hey, what are we gonna do next for this patient?
14:08
SPEAKER_00: Should we be worried about them or not?
14:10
SPEAKER_00: Where do we think they're going?
14:11
SPEAKER_00: What tests do we want to run?
14:13
SPEAKER_00: And oftentimes, and this was part of my training, it will be skewed toward the more dangerous stuff that you want to rule out, even if it's less likely, because you don't want to miss that.
14:21
SPEAKER_00: You don't want to, you don't want to send a heart attack home.
14:24
SPEAKER_01: Right, right.
14:24
SPEAKER_01: So your top five might be a different top five just from from what the study would would do.
14:29
SPEAKER_01: And so that that makes sense because you're focused on a different goal.
14:32
SPEAKER_01: Uh, and your goal being is ruling out things, bad things, but also your goal being what to do next.
14:37
SPEAKER_00: Yeah, exactly.
14:38
SPEAKER_00: And and they did use information from the health record um when they were asking both the the internal medicine physicians, they didn't actually use ER doctors um for this piece.
14:50
SPEAKER_00: Uh, when they were asking the internal medicine physicians and they were asking the AI AI models, they took clinical data from the health record.
14:57
SPEAKER_00: So there's some interesting pieces of this because they only took text-based stuff.
15:02
SPEAKER_00: Yes.
15:03
SPEAKER_00: Right?
15:03
SPEAKER_00: So when you're in the ER, like you get to Oh, you get to see the person.
15:06
SPEAKER_00: See the person.
15:07
SPEAKER_00: Like you talk to them.
15:08
SPEAKER_00: Yeah, yeah.
15:09
SPEAKER_00: You get to hear how they're acting, right?
15:11
SPEAKER_00: If someone's like in a ton of pain, they might be groaning and and that's not maybe there.
15:16
SPEAKER_00: Or you would get visual information like imaging studies.
15:19
SPEAKER_00: Um, you know, this I I joke around about this all the time, but there are certain smells that you get in the ER where you're like, oh, like I know that person's having a GI bleed.
15:27
SPEAKER_00: I can smell it.
15:27
SPEAKER_00: Yeah.
15:28
SPEAKER_00: Um, and so there's there's limited information that's available.
15:32
SPEAKER_00: And the information that's available, someone had to like put into the chart.
15:36
SPEAKER_00: Right.
15:37
SPEAKER_00: Right?
15:37
SPEAKER_00: So, like someone had to say, okay, uh, everyone gets vital signs.
15:41
SPEAKER_00: We all know someone's blood pressure, their heart rate, their temperature.
15:43
SPEAKER_00: That's done at triage.
15:45
SPEAKER_00: But like someone had to say, Oh, I'm gonna order these labs.
15:49
SPEAKER_00: Or here's the history of the present illness.
15:52
SPEAKER_00: A 56-year-old man with a history of diabetes, hypertension, and hyperlipidemia came in with chest pain.
15:57
SPEAKER_00: It's located on the left side of the chest.
15:59
SPEAKER_00: It's a pressure-like sensation, it's worse with exertion.
16:02
SPEAKER_00: It radiates into the right shoulder and is associated with shortness of breath.
16:05
SPEAKER_00: Yes.
16:06
SPEAKER_00: That information had to be put in, somebody had to ask the patient those questions.
16:10
SPEAKER_00: They may or may not have volunteered that on their own, right?
16:13
SPEAKER_00: Um, and then someone had to say, okay, that is the relevant information to put it into the chart.
16:17
SPEAKER_01: The problem of relevance is huge because there's a whole host of other things that are potentially irrelevant that have been smartly excluded uh by somebody, right?
16:25
SPEAKER_01: And any intelligent system needs to account for that, needs to account for the fact that things have been left out and only irrelevant and important things have been focused on.
16:33
SPEAKER_00: Yeah, exactly.
16:34
SPEAKER_01: That's actually also a historical um uh AI problem.
16:38
SPEAKER_01: It's called a relevance problem, is deciding what we need to take into account and what we don't.
16:42
SPEAKER_01: So the reasoning step is almost the easy step because now you already have all the relevant information.
16:47
SPEAKER_01: And now, given this part, let's do some analysis to figure out which how we compose that relevant information into something useful afterwards, right?
16:55
SPEAKER_00: And I would argue that like your clinician is reasoning as they ask those questions of the patient or decide what tests to order.
17:02
SPEAKER_00: If I am ordering a troponin or a D-dimer, or decide to order an EKG and a chest x-ray for someone with chest pain, lab tests, whatever, there's reasoning that went into that.
17:14
SPEAKER_00: This there's a decision tree in my head about what I'm going to do next.
17:18
SPEAKER_00: And so that is a really important piece of what was the AI did not do that as part of the study.
17:26
SPEAKER_01: Yeah.
17:26
SPEAKER_00: So it's important just to call that out that it's like taking the information that was basically curated already.
17:33
SPEAKER_00: Yes.
17:33
SPEAKER_00: And using that.
17:34
SPEAKER_01: Yeah.
17:35
SPEAKER_01: And just to be clear, uh the paper, so this is an interesting piece, right?
17:39
SPEAKER_01: The paper itself has a whole section on limitations.
17:42
SPEAKER_01: They talk about it being only text.
17:44
SPEAKER_01: They talk about it only having certain other aspects of it.
17:47
SPEAKER_01: I think um, you know, what models they used and and what, you know, limited uh well, how the tasks are limited in whatever fashion.
17:53
SPEAKER_01: So they list out a bunch of limitations.
17:56
SPEAKER_00: Um and they actually say, hey, this is just a proof of concept.
17:59
SPEAKER_00: Probably in part because they only compared it to two internal medicine physicians.
18:03
SPEAKER_00: Right, right.
18:04
SPEAKER_00: So there's like proof of concept, yeah.
18:06
SPEAKER_00: This worked and it was pretty limited.
18:09
SPEAKER_00: Yes.
18:09
SPEAKER_00: And like I agree with that assessment.
18:11
SPEAKER_00: I think it's like a really cool thing that the the AI and the AI models have improved, and what they found was very interesting.
18:21
SPEAKER_00: Yeah.
18:21
SPEAKER_00: And it was very limited at the same time.
18:23
SPEAKER_01: And I want to say the paper, the motivation for the paper was to contrast the the whole body of work right now and the news cycles about how AI systems are passing medical exams.
18:32
SPEAKER_01: And their point was to say, yeah, yeah, that's all good, well and good, but we have this way of measuring uh the the, you know, from the 50s, we have this approach to measuring, you know, doing evaluations of of doctors and so on.
18:45
SPEAKER_01: Why not we use that with LLMs?
18:46
SPEAKER_01: So that's kind of where the framing of the paper started, right?
18:49
SPEAKER_00: Which totally makes sense.
18:50
SPEAKER_00: It's like your ability to score well on a licensing exam and your ability to do things with a like in the patient setting are like two very different things.
18:59
SPEAKER_00: Yes.
19:00
SPEAKER_00: And so it's it's like a very appropriate thing to have studied.
19:04
SPEAKER_00: The output of it was very interesting and lim and limited, which they do call out.
19:09
SPEAKER_00: I think the main issue is that when it went into news headlines, that's where the hype came in.
19:14
SPEAKER_00: And it was like, oh my gosh, AI per outperforms doctors in Harvard trial of emergency triage diagnoses.
19:21
SPEAKER_00: In fact, NPR, I think, actually ended up changing their headline after getting some feedback from physicians, like, hey, this actually they didn't compare it to ER doctors.
19:30
SPEAKER_00: Yeah.
19:31
SPEAKER_00: They compared it to in two internal medicine physicians who are who do different things.
19:36
SPEAKER_01: Yeah.
19:36
SPEAKER_01: And what was your this was your original um Yeah, the original said it did better than ER doctors.
19:42
SPEAKER_00: So they had to correct it because what the compare the comparison was to actually to internal medicine physicians who have a different job than the ER, right?
19:51
SPEAKER_00: Yeah.
19:51
SPEAKER_00: And they're not, for example, trying to diagnose people with the limited information that you get at triage.
19:58
SPEAKER_00: That's not their job.
19:58
SPEAKER_00: Right.
19:59
SPEAKER_00: And so you need to make sure that you're comparing the right, the right job.
20:04
SPEAKER_01: Right.
20:04
SPEAKER_01: Right.
20:05
SPEAKER_01: Right.
20:06
SPEAKER_01: Um, but yeah, this is an issue, right?
20:08
SPEAKER_01: Across the board, across in science.
20:10
SPEAKER_01: I mean, you have a bunch of researchers doing their best to do the right research.
20:14
SPEAKER_01: So their the study is good, their uh maybe the results are solid, maybe statistically everything works out, the math works out, um, maybe their design, experimental design is still good and they lay out all the limitations.
20:25
SPEAKER_01: They're being fully honest.
20:26
SPEAKER_01: Um, we saw in the example of the Perceptron in the 1980s that even scientists can sometimes be excited about their work to the extent that they're they're they're seeing the future, but they don't have a sense for how long things will take, and they get excited.
20:37
SPEAKER_01: So there are some exaggerations of claims made even at that level.
20:40
SPEAKER_01: But nowadays, it's much more common to see papers that are uh more realistic about what they are actually proposing and having a whole bunch of caveats in them.
20:48
SPEAKER_01: But what ends up happening is um researchers are incentivized to publish and to get more citations, which means promote their publications.
20:57
SPEAKER_01: So they are going to simplify it themselves.
20:59
SPEAKER_01: And you'll see this a lot on Twitter, for example, or X, where people are posting their own articles or articles by their students and summarizing the main findings in two or three bullets, which often ignore all the limitations because that's not part of the messaging, because they need people to read these papers and cite them and so on.
21:16
SPEAKER_01: So that's one incentive that that's but that's driving them.
21:19
SPEAKER_01: Then you have the university, for example, that the or the institution that the researchers are working with.
21:25
SPEAKER_01: And now they have these researchers with a paper at a top journal or a top conference, and they want to say to the world, hey, we are people who do high-quality research.
21:33
SPEAKER_01: And they have a bunch of science communicators who write those news releases, press releases from the official press releases from the university.
21:41
SPEAKER_01: And those oftentimes are again simplified for a larger audience, often for donors, often for other uh national governmental institutions that are going to provide them funding in the future.
21:50
SPEAKER_01: And so they're gonna say, talk about the article.
21:53
SPEAKER_01: They might list some caveats, but now it's already shortened and simplified.
21:56
SPEAKER_01: So it's gonna you know already have some aspects of um how.
22:00
SPEAKER_01: Hype in there because they want people to actually get excited by this.
22:03
SPEAKER_01: And then you have uh news outlets, uh, NPR, New York Times, um, whatever else, that are picking up on those press releases.
22:10
SPEAKER_01: And they also have science communicators who are also trying to do their job, but have are now twice removed from the original research itself.
22:18
SPEAKER_01: And they may or may not have had conversations with the researchers.
22:20
SPEAKER_01: I I don't want to suggest that they are their journalism is lacking or something.
22:24
SPEAKER_01: But I want to say that in the effort of getting their articles out, they're going to say that talk about the promise and the excitement of this technology.
22:33
SPEAKER_01: Or on the flip side, talk about all the bad things that can happen, right?
22:35
SPEAKER_01: So there's also that.
22:36
SPEAKER_01: There are papers written about all the bad things, and they're sort of doomsday papers.
22:40
SPEAKER_01: And they had the same problem on the flip side where they're saying all the negative things and not identifying what actually was promising.
22:46
SPEAKER_01: So now you have this, and so you have this chain of exaggerations that happen.
22:50
SPEAKER_01: And by the time it gets to, you know, like a Facebook feed, you're reading a very different version without any of the caveats and highly overgeneralized uh scientific research.
23:00
SPEAKER_00: And I do think there's a component of, hey, this headline is something we want to get people to click on.
23:05
SPEAKER_00: So even if the limitations, for example, are called out in the article itself, it's like your headline is meant to get you to read the story.
23:14
SPEAKER_00: Now, some people just read the headline and move on, right?
23:16
SPEAKER_00: But it's meant to be a little bit of clickbait.
23:18
SPEAKER_00: And when you think about posting, having things posted on social media, no one wants to read a whole long paper.
23:24
SPEAKER_00: They want to get, hey, what did this paper say?
23:27
SPEAKER_00: And like the paper did say some of the things that are in the articles or in the little, you know, bites of information, but it's not the whole story.
23:37
SPEAKER_00: And it's hard because like you and I went back and we read the whole article and we were like, hey, this is this is where it got overhyped and how and why.
23:46
SPEAKER_00: But I think it's really hard when you're looking at headlines like this to be like, hey, what does it actually say versus what is the hype?
23:54
SPEAKER_00: And you you may need experts to like actually interpret that for you.
23:59
SPEAKER_00: And I probably I would in other realms.
24:02
SPEAKER_00: Yep.
24:03
SPEAKER_00: And so I think that makes it really difficult because it's like, hey, this is trusted information, this is science, this is a scientific reporter who's putting it out there.
24:13
SPEAKER_00: And it is exciting, but I think it has that excitement has to be tamed a little bit when you understand what the limitations are.
24:20
SPEAKER_01: And I'll take it one step further.
24:22
SPEAKER_01: I mean, if you're, you know, all of this information, for example, the original paper, the news press release from the university and other news sources are all public now and being used to train our LLMs.
24:32
SPEAKER_01: So going forward, if you have a chat bot and you start asking questions about this, they're already diluted with not just the original paper, but all the other um exaggerations that come with it.
24:40
SPEAKER_01: So now are you getting the right answer when that when you ask it uh that that sort of question?
24:45
SPEAKER_01: Are you getting the appropriate caveats?
24:47
SPEAKER_01: Um, I I don't know.
24:48
SPEAKER_01: I think that those are open questions.
24:50
SPEAKER_01: And so you you now have a data set that is that is now diluted at some level with exaggerations.
24:56
SPEAKER_01: Um again, it's even if you have actors along the way that are not, you know, uh trying to make things worse or trying to be intentionally um, you know, bad in any way, you still have this thing because of the existing incentives of the system, right?
25:10
SPEAKER_01: The way the system is set up.
25:11
SPEAKER_00: Yeah, we all want something that's like exciting and fun and interesting and new.
25:15
SPEAKER_00: And here's a great use case, and that's very AI summer of us, I guess.
25:20
SPEAKER_00: Like, but I I think the limitations in this study are really important, and the idea that it was a proof of concept more than anything else.
25:31
SPEAKER_00: And so that has to be taken into account at the same time, or else we're gonna have that AI winter coming, right?
25:38
SPEAKER_01: Yeah.
25:38
SPEAKER_01: So, you know, when you see a news story on Facebook or something, you want to just ask yourself for a second, like, is this really true?
25:45
SPEAKER_01: Um, but I mean, you know, that and before sharing it, right?
25:48
SPEAKER_01: Because that's it's worth thinking about that for a second, um, especially the ones that seem too good to be true.
25:54
SPEAKER_00: Yeah, I think this speaks, I mean, we've talked about misinformation on the podcast before, and this is that same sort of concept where it's like, okay, before you share something, what what are experts saying about this?
26:08
SPEAKER_00: And how can we do some fact checking?
26:11
SPEAKER_01: Yeah.
26:12
SPEAKER_00: Um, and this is an example where it's the story has truth to it, but there are limitations that really did need to be called out because it wasn't meant to prove that AI is better than ER doctors.
26:26
SPEAKER_00: That's not what it was set out setting out to do.
26:28
SPEAKER_00: And but the headlines make you think otherwise.
26:30
SPEAKER_00: So uh I do also want to call out that some of the headlines have been changed since doctors wrote in and said, hey, if that's not what the study says, yes, you've got to change the headline, and and some of the news outlets have.
26:42
SPEAKER_00: So I think that's that's a good thing.
26:44
SPEAKER_00: That's a move in the right direction.
26:46
SPEAKER_00: Um, but I think we all have to be careful about this hype cycle, and we're gonna see it continuing on.
26:51
SPEAKER_00: Yep.
26:52
SPEAKER_00: All right.
26:53
SPEAKER_00: Well, thank you for joining us.
26:55
SPEAKER_00: We will see you next time on Code and Cure.
26:57
unknown: Thank you.