Latest / Code & Cure / #65 - Can care managers trust AI summaries?

Transcript

#65 - Can care managers trust AI summaries? · Code & Cure

0:00
SPEAKER_01: A patient's story is scattered across dozens of data points.
0:04
SPEAKER_01: What if AI could pull those signals together and give care managers a clearer picture of what's happening in a person's life and where intervention might actually help?
0:14
SPEAKER_01: Basically, can AI turn data overload into person-centered care?
0:28
SPEAKER_00: Hello and welcome back to Code and Cure, where we discuss decoding health in the age of AI.
0:34
SPEAKER_00: My name is Vassant Sarathi.
0:35
SPEAKER_00: I'm an AI researcher and cognitive scientist.
0:38
SPEAKER_00: And I'm here as always with Laura Hagopian.
0:41
SPEAKER_01: I'm an emergency medicine physician and I work in digital health.
0:44
SPEAKER_01: And I was particularly excited about this paper because it feels like a great real-world application of AI.
0:53
SPEAKER_01: But I know when I showed it to you, you were like, I'm a little bit hesitant.
0:56
SPEAKER_01: No, no, no.
0:57
SPEAKER_01: No, no, no.
0:57
SPEAKER_00: I think, I think, well, not not hesitant necessarily.
1:01
SPEAKER_00: I think for me, I am probably the most excited about this episode that I've ever been because this is directly in line with some of my own personal research work.
1:11
SPEAKER_00: My my my my work as a AI researcher is in this space.
1:16
SPEAKER_00: And I am deeply aware of all of the pitfalls and the positives of uh work in this space.
1:23
SPEAKER_00: And so it gets me very excited to talk about it.
1:26
SPEAKER_00: And I am literally sitting on the edge of my seat here, pretty, you know, psyched about this.
1:31
SPEAKER_00: So let's go.
1:32
SPEAKER_01: Awesome.
1:32
SPEAKER_01: Yeah.
1:33
SPEAKER_01: So what they did in this study is they basically said, hey, we have all of this data from all these different sources on patients.
1:42
SPEAKER_01: And um, it could be quantitative and structured, or it could be qualitative data.
1:48
SPEAKER_01: It could be something that's just like written in a note somewhere randomly.
1:52
SPEAKER_01: It could be from 10 years ago or one year ago or yesterday.
1:56
SPEAKER_01: There's all this different data stored in different places at different times and in in different ways.
2:03
SPEAKER_01: And the whole concept is hey, if you have a care manager, they need to like really understand where a patient is in the moment so they can intervene and help in a very targeted way with whatever clinical needs they may have, but also whatever social needs they might have, whether it's um issues with housing or food insecurity or uh loneliness or transportation or or whatnot.
2:28
SPEAKER_01: And so the whole concept here is like, hey, can we take this large volume of data, which might take a care manager hours, days, I don't even know how long to kind of sift through before they talk to a patient.
2:39
SPEAKER_01: And can we get a quicker summary so we can intervene on the things that really matter for that patient?
2:45
SPEAKER_00: Right, right, right.
2:46
SPEAKER_00: And the technical name for this subfield is multi-document summarization, which is the idea that you provide lots of different types of documents and you get a summary from it, right?
2:56
SPEAKER_00: And if you think about it, we use AI for that all the time.
2:59
SPEAKER_00: I mean, people who are familiar with using AI tools like ChatGPT, you can think about the times you update you upload a PDF or maybe three or four PDFs or something, and you ask for a summary, right?
3:10
SPEAKER_00: That is a very common task that we ask of AI these days.
3:14
SPEAKER_01: I actually I I mean, I guess I would say one of the places I see it the most is when I do like a Google search now, where I'm like, hey, what's the answer to this question?
3:21
SPEAKER_01: And it pops up with a summary.
3:23
SPEAKER_01: But I will tell you that that summary is not always correct.
3:26
SPEAKER_00: Well, yes.
3:27
SPEAKER_00: I mean, I think everyone's aware of that, right?
3:29
SPEAKER_00: I was talking to someone else recently who was working in the legal space, and they um were able to, you know, they opened up Adobe Reader or something for opening up a legal brief that they were or case that they were reading, and it automatically provided a summary of that document.
3:46
SPEAKER_00: And they were the legal experts reading it.
3:47
SPEAKER_00: So when they read the summary, the summary was all wrong, all kinds of issues, right?
3:51
SPEAKER_00: But the popping up of the summary is a like automated feature now is part of Adobe's reader.
3:56
SPEAKER_01: Yeah.
3:57
SPEAKER_00: And um, which, you know, in some ways can be really helpful.
4:00
SPEAKER_00: But if it's wrong, then it's not helpful at all.
4:03
SPEAKER_00: It's hurtful, right?
4:03
SPEAKER_00: And so I think that that's there's a challenge there, and we'll get to that in a second.
4:07
SPEAKER_00: But I think the uh what I like about this paper is the underlying challenge and that the approach that they've taken to incorporate all of these other systemic challenges that goes into care, all of the sociocultural, all of the stuff that's outside of the core medical things, right, that are also important.
4:27
SPEAKER_01: Absolutely.
4:28
SPEAKER_01: And I think those things can impact your health more than anything else, right?
4:33
SPEAKER_01: Like if you um, for example, are food insecure, then you're not gonna be managing your diabetes.
4:39
SPEAKER_01: You're eating whatever you can get your hands hands on, for example.
4:42
SPEAKER_01: Yeah.
4:43
SPEAKER_01: And that may be more, you know, processed foods or things that have um, you know, the glycemic index that you don't want, all that kind of stuff may be true.
4:51
SPEAKER_01: And so it's like if you want to intervene on the whole person, it's not just, you know, the medical diagnoses that they carry or the behavioral health diagnoses that they carry.
5:00
SPEAKER_01: It's like the integrated, you know, whole person view where you understand not only their medical and behavioral health concerns, but what challenges they may have accessing care and what social concerns they may have as well.
5:14
SPEAKER_01: Right.
5:14
SPEAKER_01: And that's when you can sort of treat the whole person.
5:16
SPEAKER_01: But then that's like, okay, well, now there's data coming in from all these different places.
5:20
SPEAKER_01: And the data changes over time, right?
5:22
SPEAKER_01: Someone may have had housing insecurity and they no longer have it and now they have it again.
5:26
SPEAKER_01: Right.
5:26
SPEAKER_01: So the timelines I think are interesting.
5:29
SPEAKER_01: And there's it's so dynamic that it's like, well, we need to know what's happening now.
5:35
SPEAKER_01: And the fact that so much of it is not structured data.
5:38
SPEAKER_01: Yeah.
5:39
SPEAKER_01: Like it might just be, oh, I talked to someone on the phone and this is what they note wrote in a note.
5:43
SPEAKER_01: Yes.
5:44
SPEAKER_01: Rather than they checked off a box that said, you know, patient has food insecurity.
5:48
SPEAKER_01: Right.
5:48
SPEAKER_01: That was a year ago.
5:49
SPEAKER_01: And now someone wrote in a note, hey, they actually have been doing well, and they um have access to SNAP benefits or whatever it is.
5:56
SPEAKER_01: So like tracking that change over time in different data sources, some of which could be structured, checkbox, and some of which could be unstructured, just like a note that someone happened to write.
6:08
SPEAKER_00: So I think this is what gets people so excited is the promise of large language models being able to do something like this.
6:15
SPEAKER_00: I think that's what people are.
6:16
SPEAKER_00: I I think there is a set sort of general in our psyche is that, oh my God, we have these amazing new AI systems that we can just dump a bunch of text to and they will give out something coherent that either summarizes it, helps us answer questions about it, or links things in it, whatever, right?
6:33
SPEAKER_00: I mean, this is at the very core of why we're so excited about these systems and applying it in different settings.
6:38
SPEAKER_01: Right.
6:39
SPEAKER_01: Because in this case, you can say, okay, well, now I have a summary of this patient.
6:42
SPEAKER_01: Now I know where to intervene.
6:44
SPEAKER_01: Yes.
6:44
SPEAKER_01: Right?
6:45
SPEAKER_01: Oh, like they don't have access to care.
6:47
SPEAKER_01: That's a place I need to intervene before I can work on um, you know, lowering their A1C value with them or whatever it is.
6:55
SPEAKER_01: And so what they did here was they did they kind of iterated over time and they they had this cross-functional team, which I think was really cool, where they had, you know, um engineers and product managers, and they involved the um, you know, the care team um as well.
7:11
SPEAKER_01: And they said, okay, well, we're gonna start off by drafting like a very simple prompt.
7:16
SPEAKER_01: We're gonna prompt the AI to take information from the notes um and and and basically summarize it and do it as as simply as possible.
7:26
SPEAKER_01: And then they saw what the output was and they said, okay, well, this isn't quite what we want as the output.
7:33
SPEAKER_01: Let's refine, let's refine this prompt so that we can get it sort of um clearly summarized in the areas that we think is are most important.
7:43
SPEAKER_01: Yeah.
7:44
SPEAKER_01: Let's add like a formatting structure so things are bold, there are headings so that it's like easier to skim and read.
7:50
SPEAKER_01: Um, and let's make sure that we have all the information that we want to be included on every patient.
7:56
SPEAKER_01: And then they said, okay, let's send this to case managers and let's have the case managers react to it and see if we can make it even better.
8:05
SPEAKER_01: And so they and so they over time collected feedback from the case managers and kept refining and updating the prompt to incorporate those changes.
8:15
SPEAKER_01: Um, and then went back to the to the case managers and said, Hey, is this working?
8:19
SPEAKER_01: And and what they were able to do over time by working with the case managers is say, Hey, we understand what topics you want.
8:27
SPEAKER_01: We're gonna update those.
8:29
SPEAKER_01: We're gonna make sure that there's like a summary at the top that hadn't initially been included.
8:33
SPEAKER_01: We're gonna um change the formatting so it's easier for you to use.
8:37
SPEAKER_01: And so over time, they had, you know, not just the headings and focused topic areas, but like a high-level summary paragraph and a list of what the patient's goals might be, etc.
8:48
SPEAKER_01: And so it just showed that there was like a lot of cross-functional work and they iterated on the prompt a lot to get what they wanted out of it.
8:56
SPEAKER_01: And then, and this is the part where I'm interested to hear what you think, Vasanth, is that they said, okay, we have this ready.
9:05
SPEAKER_01: Now we need to evaluate it.
9:07
SPEAKER_01: Now we need to check how accurate it is.
9:09
SPEAKER_01: Because of course, if you're intervening on stuff that's like not correct, then you're not, you know, you're not, you're not carrying out what you're trying to do.
9:17
SPEAKER_00: Yeah.
9:17
SPEAKER_01: And so they conducted some evaluations to check for accuracy here.
9:23
SPEAKER_00: Yeah.
9:23
SPEAKER_00: Yeah.
9:23
SPEAKER_00: I mean, there's so much to unpack, right?
9:25
SPEAKER_00: So my like my mind is sort of teeming here.
9:28
SPEAKER_00: I think there's so much to unpack.
9:29
SPEAKER_00: And even before we get to the evaluations question, I think I want to take a step back and give people a sense for what's actually happening in these LLMs or these AI systems when they're doing this.
9:42
SPEAKER_00: Because I think that provides a little bit more kind of foundation to move forward.
9:46
SPEAKER_00: So I think we've talked about this before, but it's worth repeating.
9:49
SPEAKER_00: Um, you know, the modern day AI breakthroughs are all based on a particular kind of neural network called a transformer.
9:57
SPEAKER_00: But the big breakthroughs are termed, I mean, the the architecture is called a large language model or an LLM.
10:04
SPEAKER_00: And at the very core of it, you don't have to worry about how that works exactly, but at the very core of it, it's trained on all the internet data and it's trained to do one thing really well, which is predict the next word.
10:14
SPEAKER_00: So you give it a bunch of text and it predicts the next word, and you take that word, put it back to the text that you had before, and then predict the next word.
10:20
SPEAKER_00: So over the sequence, or you know, when you do that over and over again, it gives you sentences.
10:25
SPEAKER_00: And what we've learned is if you train these models on very large amounts of data, that the sentences that come out are actually kind of coherent, grammatically correct, and interesting.
10:35
SPEAKER_00: And as people have worked more and more with these large language models, they've begun to realize that, hey, you can just not only get the next sentence out, but you can sort of train it to answer questions.
10:45
SPEAKER_00: So you can say things like, hey, summarize this for me, and then give it a block of text, and then it produces a summary.
10:51
SPEAKER_00: And how exactly it does that is still a bit of a mystery to most AI researchers.
10:56
SPEAKER_00: Uh, we don't really know how it is that it's run learned these skills of reasoning and uh understanding what you want and understanding what a summary even is to produce the right, to produce what seems like the right thing.
11:07
SPEAKER_01: Magic.
11:07
SPEAKER_01: It's magic, clearly.
11:08
SPEAKER_01: It is a bit of magic.
11:09
SPEAKER_00: So we have are in that world right now where we don't fully understand how these systems are producing what they're what they're producing.
11:15
SPEAKER_00: But they are producing uh, you know, so you give it the input, and that's often called a prompt.
11:20
SPEAKER_00: Uh, some it's also uh another term is um a prompt plus all the other text you provide as input is also called context.
11:26
SPEAKER_00: So you give it the context, and then what it what you know it produces some kind of output, right?
11:31
SPEAKER_00: And some kind of response.
11:32
SPEAKER_00: That's what oftentimes people will use for a word.
11:34
SPEAKER_00: So you get the prompt input and and the context and you get a response from it.
11:39
SPEAKER_00: And uh what the problem, of course, is when you have lots and lots and lots of documents, you the context, uh, the amount of context grows, and that's called the context window.
11:48
SPEAKER_00: It's pretty large.
11:49
SPEAKER_00: And early language models couldn't handle that as well.
11:51
SPEAKER_00: But now more and more language models, that new ones like the OpenAI models and so on, are actually set up so they can handle very large context windows, that is, words like a million words or whatever, right?
12:02
SPEAKER_00: Something ridiculous.
12:03
SPEAKER_00: Um, and so they can handle very big context windows.
12:05
SPEAKER_00: You can provide a lot of input and it provides a lot of it provides output uh that makes sense.
12:10
SPEAKER_00: There are challenges with this.
12:11
SPEAKER_00: These large language models are known to be not so great in dealing with context.
12:16
SPEAKER_00: Now, if you just asked it a question about a patient without providing any context, it's going to give you an answer, and that answer uh will be based on whatever it's learned, right?
12:26
SPEAKER_00: But it's it's got no relevance to you because it's got nothing to do with the patient, right?
12:30
SPEAKER_00: All the knowledge is just internal to it.
12:31
SPEAKER_00: So you provide the knowledge by adding to that context.
12:34
SPEAKER_00: You provide all the documents and you provide all the background in the context.
12:38
SPEAKER_00: That's one way to do it.
12:39
SPEAKER_00: Um, another way to do it is to say, hey, I have all this context stored separately in this database.
12:44
SPEAKER_00: You language model decide when you need to look it up.
12:47
unknown: Uh-huh.
12:48
SPEAKER_00: Pull up what you need, and then answer my question.
12:51
SPEAKER_00: So that's what uh is popularly called rag systems or retrieval augmented generation systems, where you ask the LLM a question.
12:59
SPEAKER_00: It doesn't just use what it knows from random Reddit in internet pre- uh training that it did a long time ago.
13:05
SPEAKER_00: Instead, it looks up a database that you provided it and then pulls whatever pieces of snippets are relevant into its own context and then answers the question.
13:15
SPEAKER_01: And in that case, you're like the LLM is taking it from the spot you want it to.
13:20
SPEAKER_01: Like the information, I mean, I mean that's the question is like, does that make it kind of more accurate, I guess?
13:26
SPEAKER_00: That's right.
13:26
SPEAKER_00: The purpose of the RAG systems is to make it more accurate, right?
13:30
SPEAKER_00: And to deal with uh changes of uh data over time.
13:33
SPEAKER_00: Like new information comes in.
13:35
SPEAKER_00: The LLM that was originally trained wasn't trained on this new information, but it's still smart enough to be able to handle data.
13:43
SPEAKER_00: So you just keep updating the database with new information and use the same LLM to read off of that database.
13:49
SPEAKER_00: Does that make sense?
13:49
SPEAKER_01: Yeah, that totally makes sense.
13:51
SPEAKER_00: Yeah, so I I think that the this is important to understand is that they developed these RAC systems for the purposes of improving accuracy because you don't want it just making up stuff, uh, which by the way, the the jargon word is hallucinations.
14:03
SPEAKER_00: You don't want it hallucinating um or fabricating information.
14:07
SPEAKER_01: You want it to actually look up the correct answer and give it to you, and be grounded in the data that is appropriate in this specific use case for this specific patient, for example.
14:16
SPEAKER_00: Now, there are it's a big subfield, um, and uh that kind of retrieval augmented generation.
14:22
SPEAKER_00: There's been a lot of innovations.
14:23
SPEAKER_00: Um, some of the challenges involve do you even know?
14:27
SPEAKER_00: I mean, why would the LLM use the database and not its own internal knowledge?
14:31
SPEAKER_00: There's been studies about that because sometimes they don't look it up.
14:34
SPEAKER_00: The LLMs don't look it up, they just make it up anyways, right?
14:37
SPEAKER_00: There's no control over that.
14:38
SPEAKER_00: You could say, you could prompt the LLM to say, hey, make sure to look it up.
14:42
SPEAKER_00: But again, these prompts are just inputs that are text and the just they just you know result in outputs being more text.
14:48
SPEAKER_00: It's not like you're forcing the LLM to follow a specific procedure, right?
14:52
SPEAKER_00: Even if you tell it the specific procedure, it's unlikely to exactly follow it all the time.
14:56
SPEAKER_01: So Which actually happened in this study.
14:59
SPEAKER_01: They had to like adjust the prompt that's to say, like, oh, don't infer, don't speculate, don't create information that's not explicitly stated in the provided data.
15:08
SPEAKER_01: Yes.
15:08
SPEAKER_01: So clearly this was happening to them in this study where there were issues where hallucinations and stuff were coming through.
15:15
SPEAKER_01: Um, if there was like an incorrect diagnosis or an incorrect conclusion that came through, they had to say, okay, well, how can we can how can we fix the prompt?
15:24
SPEAKER_01: And is that, I mean, I guess my question is, is that enough to fix the prompt?
15:28
SPEAKER_00: I don't know.
15:29
SPEAKER_00: Notice what just happened there, right?
15:30
SPEAKER_00: One issue is that you had the human identify the problem and say, oh, wait a minute, this wasn't right.
15:35
SPEAKER_00: Let me go fix the prompt to deal with this.
15:38
SPEAKER_00: And to me, at some level, without seeming too negative, it seems like that's a little bit of a band-aid, right?
15:43
SPEAKER_00: Because you could have a new error mode that you hadn't seen before pop up later, and you have to hope that there's a human there noticing that error mode and then fixing the prompt.
15:53
SPEAKER_00: But there's no guarantee that the error mode is gonna be the same, right?
15:56
SPEAKER_01: I can also imagine over time, if you like start to trust a tool like this and you use it all the time, you're not like checking every single location.
16:04
SPEAKER_01: Right to that.
16:05
SPEAKER_01: Yes.
16:05
SPEAKER_01: And so you reach the point where you're like sort of have this, oh, it's automated.
16:09
SPEAKER_01: I have this automation bias.
16:10
SPEAKER_01: I'm just gonna like proceed as if this is correct.
16:14
SPEAKER_01: Yes.
16:14
SPEAKER_01: And then, you know, over time, maybe different error modes pop up that you never even identify.
16:19
SPEAKER_00: It cannot guarantee accuracy, right?
16:21
SPEAKER_00: So there's that problem already.
16:23
SPEAKER_00: But there's another problem, which is you bring in these pieces, large bodies of text, even if you use a quote rag system and allow the LLM to pull in what's relevant.
16:32
SPEAKER_00: If that context window, that input to the LLM is very large, it has issues like recency bias.
16:37
SPEAKER_00: That is, it remembers or only focuses on things that are at the end and not at the beginning of the context.
16:41
SPEAKER_00: It forgets things along the way.
16:42
SPEAKER_00: There's issues like that too that pop up that can cause, even when you pull the right snippet to produce the right summary from that snippet could still be could still have issues with it.
16:51
SPEAKER_01: So basically, like if it would be harder for a human to summarize, then it's harder for an LLM to summarize too.
16:57
SPEAKER_01: If you have too much context, it's gonna have an issue like figuring out, hey, what's the most relevant stuff to pull up?
17:04
SPEAKER_00: Yes.
17:04
SPEAKER_00: And there's another set of issues which comes up, which I think people don't talk about enough because hallucinations or fabrications or making up stuff is not the only problem.
17:14
SPEAKER_00: Even if you had a perfectly correct response, you could be omitting things.
17:18
SPEAKER_00: The LLM could be leaving out things that are actually relevant, that are actually important, and that are actually, you know, uh required in the final summary.
17:28
SPEAKER_00: And even worse so, the LLM could be removing caveats that were there in the original document.
17:33
SPEAKER_00: Because the original documents might have said something and then caveat it with something else, might have said, you know, this is true, except when it's sunny or something, right?
17:40
SPEAKER_00: And if you just say this is true as the output, then it's going to read if you have somebody reviewing it, they're going to say this is correct because it is saying that from the sources, but it's leaving out a big caveat that the except when it's sunny part, and that's a problem too, because you need the caveat to really understand what's being said.
17:58
SPEAKER_01: I think omissions are really difficult as a human to identify.
18:02
SPEAKER_01: And this is the problem that I've had when I've used um like the the uh AI to write my notes when I'm working at like an urgent care clinic is I sometimes I'll see, oh, it omitted.
18:13
SPEAKER_01: Like I asked these red flag questions and the note doesn't say any of those things.
18:17
SPEAKER_01: Sometimes I'll identify it, but I wonder how often I don't.
18:20
SPEAKER_01: Because it's easy to see when something is like wrong, blatantly wrong, like written in front of me.
18:25
SPEAKER_01: Um, you know, like a wrong medication or a wrong dosage or whatever.
18:30
SPEAKER_01: Oh, I know the patient didn't say that because their their mother said that, or whatever it is.
18:34
SPEAKER_01: But when it's omitted, it's like you have to like really be scouring to figure that out and have your brain like totally turned on.
18:42
SPEAKER_01: And I wonder how often I I miss an omission.
18:45
SPEAKER_00: And you do.
18:45
SPEAKER_00: And in fact, I've done studies as of this past few months on this, on human subjects, to study exactly what mode of errors that are easier to track and easier to capture and what ones are not.
18:55
SPEAKER_00: And omissions are exceedingly difficult.
18:57
SPEAKER_00: And it's not surprising.
18:58
SPEAKER_00: If you think about it, um, I am also a lawyer by training, and one of the steps to becoming a lawyer is to do this exam called the LSAT, which is the exam that's, you know, kind of helps you get into law school.
19:10
SPEAKER_00: And some of the questions on those are having to do with like deep critical reason thinking and reasoning, which doesn't come naturally to most people.
19:18
SPEAKER_00: Like if I give you a paragraph of text and say, hey, what are the assumptions that this person is making when they're writing this text?
19:23
SPEAKER_00: It is really hard because the assumptions are not written.
19:26
SPEAKER_00: They're all implicit and they all, you know, once I tell you what the assumption is, it's pretty straightforward to understand it.
19:33
SPEAKER_00: But it's really hard to pick it off.
19:35
SPEAKER_00: And if you are a human and you start to see good responses from these LLMs, you'll start to realize that if it omits things, you're not gonna catch it.
19:43
SPEAKER_00: You're you have automation bias on.
19:44
SPEAKER_00: You're just gonna you're gonna say, okay, this makes sense, this makes sense, nothing weird, nothing weird.
19:48
SPEAKER_00: I don't see any hallucinated claims because it's not seeing random stuff.
19:52
SPEAKER_00: But uh, I don't see any, I should say, I don't see any obviously hallucinated claims.
19:56
SPEAKER_00: If it's subtle and it makes small differences, it's very hard to spot.
20:00
SPEAKER_01: Yeah.
20:00
SPEAKER_00: Um, and it's also dependent on your own expertise and your own expert knowledge.
20:05
SPEAKER_00: And so, um, but if it's got omitted things, you don't even know to look for them.
20:10
SPEAKER_00: How do you know that it's you know, document B, somewhere uh you know, off the 500 documents hasn't has a caveat that you've left out.
20:17
SPEAKER_00: But it turns out that everything in the case turns on that one caveat, right?
20:21
SPEAKER_01: Right.
20:22
SPEAKER_01: And I think it gets harder and harder with a larger volume of documentation because you're like, well, how are you how are you verifying this?
20:29
SPEAKER_01: Yes.
20:30
SPEAKER_01: So I think we should talk a little bit about how they tried to verify hey, are these summaries accurate?
20:36
SPEAKER_00: Yes.
20:36
SPEAKER_00: Yes.
20:37
SPEAKER_00: So uh the space of what I call multi-document summarization has its own little community and has its own set of evaluation metrics, which are constantly, you know, being sort of evaluated in a sense.
20:49
SPEAKER_00: People in the research space are looking at new ways to measure this, right?
20:52
SPEAKER_00: And there are many ways to measure this.
20:54
SPEAKER_00: There's data sets out there that have uh what they call gold summaries, that is the correct summaries.
20:58
SPEAKER_00: So they have a set of documents and they have humans who have created those quote correct summaries, which is, you know, you can argue about whether or not those humans had the right summaries in the first place.
21:07
SPEAKER_00: Um and you gotta start somewhere, though.
21:09
SPEAKER_00: Summaries are also but summaries are also task dependent.
21:12
SPEAKER_00: So some summaries are better for some certain tasks, and maybe you want a different summary for a different task.
21:17
SPEAKER_01: Sure.
21:17
SPEAKER_01: I mean, in this case, we're talking about um whether care management providers are getting an appropriate summary.
21:24
SPEAKER_01: And so, you know, that's gonna include things like housing, financial concerns, food concerns, in addition to clinical, for example.
21:32
SPEAKER_01: If a provider was looking for a summary, maybe they're more interested in in medical episodes and primary care access and that kind of stuff.
21:40
SPEAKER_01: So, yes, totally context dependent, what type of summary yeah.
21:43
SPEAKER_00: So um the evaluation metrics are things like faithfulness and what faith, for example, faithfulness is something like it given my summary, is my summary faithful to my sources?
21:54
SPEAKER_00: Right.
21:54
SPEAKER_00: And the way they measure that is by looking at whether or not every single um sort of aspect.
22:00
SPEAKER_00: Of the summary is was present in the sources, right?
22:03
SPEAKER_00: Was supported by the sources.
22:04
SPEAKER_00: That means it's faithful.
22:05
SPEAKER_00: But notice it leaves out omissions, right?
22:07
SPEAKER_00: Because you can be fully supported and leave out the caveat, right?
22:11
SPEAKER_00: And and correct.
22:12
SPEAKER_01: Like was everything there?
22:14
SPEAKER_01: And so the answer is like, yeah, sure.
22:15
SPEAKER_01: Everything was there.
22:16
SPEAKER_01: Yes.
22:17
SPEAKER_01: Maybe that means there's not a hallucination, but it doesn't mean that there's not an omission.
22:21
SPEAKER_00: Or a piece of counter evidence, something that comp directly objects to think to the thing you're saying.
22:26
SPEAKER_01: Also, the thing that you're looking at, like if we talk about um, I don't know, food insecurity, that may change over time.
22:34
SPEAKER_01: Right.
22:35
SPEAKER_01: And so it could be, it could have been faithful two years ago to what was said in the chart, but maybe it's not faithful to what was said two weeks ago in the chart.
22:45
SPEAKER_01: Yes.
22:45
SPEAKER_01: So that's an interesting one where it's like, hey, there it's dynamic.
22:49
SPEAKER_01: Yes.
22:49
SPEAKER_01: And understanding the timeline is very important when a clinician is looking at it.
22:55
SPEAKER_01: I'm not sure how the AI looks at it, but that's another area where it could be particularly challenging.
23:02
SPEAKER_01: And you could say that it was faithful because it was faithful to something two years ago, but really what's more important is what happened two weeks ago.
23:09
SPEAKER_00: Yeah.
23:10
SPEAKER_00: And the way they actually measure this, there's different ways of measuring whether something in the summary is faithful, has been faithful to the sources, is by saying, um, let's just, let's just look at what's called entailment.
23:21
SPEAKER_00: That means if this thing in the source is true, then the thing in the summary must be true.
23:25
SPEAKER_00: It's kind of logical in nature.
23:27
SPEAKER_00: Um, and that's again a measure that people use.
23:30
SPEAKER_00: But the actual implementation of these metrics is, I think, the problem, is the challenge.
23:35
SPEAKER_00: Because at the end of the day, these metrics are either measured by a neural network that's been trained to do entailment checks, to do checks between if this is true, then is this true?
23:45
SPEAKER_01: Um that's like an uh a large language model as a judge, judging and no, that's that's option number two.
23:51
SPEAKER_00: So this entailment check is just a neural network that's being trained to do one specific thing.
23:55
SPEAKER_00: It's not a large language model.
23:56
SPEAKER_01: Okay.
23:56
SPEAKER_00: But it's trained to do like if I give if if I give it two sentences, a premise and a hypothesis, if the premise is true, tell me if that if the hypothesis is true or contradicting.
24:06
SPEAKER_00: So they do that.
24:06
SPEAKER_00: That's a very common way of measuring whether or not um a particular, you know if the source is true, is the summary true, right?
24:14
SPEAKER_00: Or a sentence in the source is true, then a sentence and a summary is true.
24:17
SPEAKER_00: So you can do the entailment checks, but the actual implementation is the problem.
24:21
SPEAKER_00: Because if you actually look at the models that are being used, they have very basic issues.
24:25
SPEAKER_00: For example, they have trouble handle handling negations.
24:28
SPEAKER_00: So if I had a knot in there somewhere, you know, it's sunny outside, it's not sunny outside, then those look very similar in many ways.
24:36
SPEAKER_00: And even the entailment models struggle a little bit with that, which they should not, but they they do.
24:40
SPEAKER_01: Yeah, but it's a it's a one-word difference.
24:42
SPEAKER_01: So you're like, wow, 90% of them are the same.
24:45
SPEAKER_01: It's just this 10% that's different.
24:47
SPEAKER_00: Yes.
24:47
SPEAKER_00: Oh, that looks accurate enough.
24:49
SPEAKER_00: Exactly.
24:49
SPEAKER_00: So that's one type of accurate accuracy metric.
24:52
SPEAKER_00: The other more common accuracy accuracy metric across the board has been LLM as a judge, which is to idea that you use a different LLM.
25:00
SPEAKER_01: So not the same one that you use to create the summary, but a different one.
25:04
SPEAKER_00: A different one.
25:05
SPEAKER_00: Um use a different LLM in order and you give it some prompts about how to evaluate it and you do the same thing.
25:10
SPEAKER_00: You ask it, is it consistent?
25:12
SPEAKER_00: Is the sources, you know, and and the summaries consistent?
25:15
SPEAKER_01: And just explain um, if you might, if you may, about why you use a different LLM for this.
25:20
SPEAKER_00: Yeah, I mean, I think the idea is whatever biases exist in the first LLM that created the summary, you don't want the same uh LLM biases to translate into the judge.
25:32
SPEAKER_00: Because it's almost like you're it's like circular.
25:35
SPEAKER_00: You're evaluating yourself.
25:36
SPEAKER_00: You're gonna give yourself the same score as you would the first time because you're the one who created it.
25:40
SPEAKER_00: Right.
25:41
SPEAKER_00: I mean, understandably, you're giving the LLM a different task.
25:45
SPEAKER_00: You're saying, hey, evaluate this with a with a critical eye or whatever, right?
25:49
SPEAKER_00: And presumably the answer would be maybe different, but whatever implicit biases that exist in that LLM, you don't want that to translate over.
25:56
SPEAKER_00: Um, and so that's why you use a different LLM.
25:58
SPEAKER_00: Now, there can be arguments about whether or not that actually works because ultimately a lot of these LLMs are trained in the same data, same internet data, right?
26:07
SPEAKER_00: Right.
26:07
SPEAKER_00: So it's some, but then the architectures are all different.
26:09
SPEAKER_00: So like Anthropics Claude model and the OpenAI's model are all slightly different in different ways to make them better suited, have kind of their own little personalities in a sense.
26:18
SPEAKER_00: So it is going to give you different answers, and there's no nothing wrong in using a in fact, I think it's good practice to use a different LLM to evaluate it.
26:27
SPEAKER_00: But it's still an LLM looking at the same set of words.
26:30
SPEAKER_00: And unless you prompt it the way one should, in the perfect way possible, you're not going to get at the same set of issues.
26:37
SPEAKER_00: And you're you're just sort of transferring the evaluation problem to another LLM.
26:41
SPEAKER_00: And how do you know that one's right?
26:43
SPEAKER_00: So, you know, that's I think there's a there's an issue there.
26:46
SPEAKER_01: Yeah.
26:46
SPEAKER_01: And I think they did have a bunch of hurdles that they were trying to overcome as they as they worked through, hey, how can we get a better summary?
26:55
SPEAKER_01: Yeah.
26:55
SPEAKER_01: And they walked through, you know, some of the things they did to try to resolve these.
26:59
SPEAKER_01: And we talked about how they adjusted their prompt to say, like, don't, don't, don't make a speculation or um don't say something unless it's explicitly in the data.
27:10
SPEAKER_01: Because it was just like making inferences or making things, like it was hallucinating, right?
27:14
SPEAKER_01: Um, but they also found, and I uh this does not surprise me that that it was having trouble with comp complex timelines.
27:22
SPEAKER_01: This happened, then that happened, then this happened, or whatnot.
27:26
SPEAKER_01: So it's like, you know, it's conflating events or um just like messing up the details that were coming in.
27:33
SPEAKER_01: And so again, they tried to like adjust a prompt to help fix this.
27:38
SPEAKER_01: Oh, make sure you review all the dates, include the dates uh when available with any claim that was made.
27:44
SPEAKER_01: But they said, you know, this is not, this is not something that they were able to like totally fix.
27:49
SPEAKER_01: This is something that kind of remained an issue.
27:52
SPEAKER_01: Um, and that they are continuing to try to sort of figure out how we can fix it.
27:59
SPEAKER_01: And part of what they're doing is trying to fix the data that goes into it, too.
28:03
SPEAKER_01: Yeah.
28:03
SPEAKER_01: Is hey, can we make the data more structured or can we uh format it better, or can we store it in a better way, or can we make sure that the updates uh truly look chronological to the LLM?
28:16
SPEAKER_01: And can we make this, you know, can we if you have like worse data going in, then it's obviously like harder for the LLM to process.
28:23
SPEAKER_01: So can we make the data look better?
28:25
SPEAKER_01: Yeah, can we improve the prompt to try to get at this?
28:28
SPEAKER_01: I I think this is a really difficult problem to try to solve.
28:33
SPEAKER_00: Yeah.
28:33
SPEAKER_00: And you know, honestly, this is a problem that is pretty deep in the sense that the mo the error modes that can happen are themselves on an unknown, the space of that.
28:42
SPEAKER_00: Like we talked about hallucinations and fabrications, we talked about omissions, but there's others.
28:46
SPEAKER_00: There's distortion, which is the idea that you might have a claim that says, uh, you know, this incident was really bad, but the hallucination or the hallucination, sorry, the LLM claim says this incident was catastrophic.
28:58
SPEAKER_00: It's a small word change, it's in the right direction, but it just inflated the the the degree to which something was said.
29:05
SPEAKER_00: So that's a distortion in a sense.
29:07
SPEAKER_00: And that's that happens all the time with hallucinations with the with the LLMs because they are, you know, they're set up in a certain way that makes them say things in a particular direction.
29:16
SPEAKER_00: Um there's also framing.
29:18
SPEAKER_00: So when you have a summary, uh the summary is not just simply quoting text from the docs, right?
29:23
SPEAKER_00: From the source docs.
29:24
SPEAKER_00: It's summarizing kind of abstractively.
29:26
SPEAKER_00: It's saying, you know, it's kind of um it's giving you not the specific things that happened, you know, it says it says our higher level stuff.
29:34
SPEAKER_00: So for example, if a if a document said something like, you know, the patient went for a walk at 8 a.m., the summary might say the patient went for a walk in the morning.
29:42
SPEAKER_00: Right?
29:42
SPEAKER_00: That's an abstraction of the specific 8 a.m.
29:45
SPEAKER_00: aspect.
29:45
SPEAKER_00: It's not wrong.
29:47
SPEAKER_00: In fact, I would say that's fine as a summary in that in that instance.
29:50
SPEAKER_01: But it could be wrong in other instances.
29:52
SPEAKER_00: Yeah, but as and you're right.
29:53
SPEAKER_00: And and in fact, as you're doing this summarization, you are going to be slightly different from your sources.
29:58
SPEAKER_00: And I think that makes evaluation harder because it's okay to deviate from the sources, but only only in a certain way.
30:05
SPEAKER_00: And we as humans don't fully understand what that means.
30:08
SPEAKER_00: Only in the relevant way.
30:09
SPEAKER_00: Maybe the fact of 8 a.m.
30:10
SPEAKER_00: walks is not relevant at all.
30:11
SPEAKER_00: In which case, it's okay to say you just went went for a walk in the morning, because that's not a relevant you know, fact.
30:17
SPEAKER_00: Um, but it's important enough that you put it in there.
30:20
SPEAKER_00: So there's that.
30:20
SPEAKER_00: So there's distortions, there's omissions, there's also attribution errors.
30:24
SPEAKER_00: Like so-and-so said X, you know, like John said X, um, LLMs are known to change that to somebody else's name.
30:33
SPEAKER_00: And so you how do you how do you know what the source at which it's misattribution is another issue.
30:38
SPEAKER_00: So you don't know who whether the output is correct.
30:40
SPEAKER_00: So there needs to be evaluations along those lines.
30:43
SPEAKER_00: And then you have other range of issues, which is um if you have um, you know, and this doesn't, I don't know, pop up as much maybe in the medical space, but it this sort of issue pops up in other spaces where you're reading a news report and they're reporting about what somebody else said.
30:56
SPEAKER_00: You know, the the the the New York Times story might say sources said blah, blah, blah, then that's a different attribution.
31:03
SPEAKER_00: And that gets really complex very quickly.
31:06
SPEAKER_00: My point is there is a whole the way we speak and the way we use language has so many nuances that getting to the right summary is actually turns out to be a really challenging task.
31:16
SPEAKER_00: And what it means for a right summary itself is a challenging task.
31:19
SPEAKER_00: And I think these guys are doing it systematically, going through the prompt and building out the system.
31:24
SPEAKER_01: For sure.
31:24
SPEAKER_01: I mean, I think this is like absolutely a great use case.
31:28
SPEAKER_01: And if you think about what healthcare data looks like, a lot of it is in in weird formats because it might be in the medical record, which was created to bill insurance.
31:38
SPEAKER_01: Yeah.
31:38
SPEAKER_01: It might be in, you know, phone notes from a discussion with the patient, it could be in lots of different locations.
31:45
SPEAKER_01: It could be sometimes places have like multiple different record systems.
31:49
SPEAKER_01: That's really common.
31:50
SPEAKER_01: And so it's like, well, how are you if if I'm a human going through, I'm like clicking through this system and that system, and then I'm reading these notes, and then I'm going and looking at the labs and the structured data, and I'm trying to synthesize all that in my mind, and then a thousand clicks later, like you can imagine how long that takes someone to try to figure out and synthesize.
32:09
SPEAKER_01: And so this seems like a great use case as long as it's like reason reasonably accurate, is I guess that's the problem.
32:17
SPEAKER_01: It's like it needs to be accurate if you're gonna be helping the patient.
32:22
SPEAKER_00: Right.
32:22
SPEAKER_00: Accurate, relevant, the right, right level of concision, chronologically, all of that, right?
32:28
SPEAKER_00: And um, and an appropriately responsive to a query you might ask.
32:32
SPEAKER_00: Maybe you're not asking just for summary, but you're saying, did the patient do XYZ?
32:37
SPEAKER_00: And that's also available in the records, but the answer needs to be uh has to have the correct reasoning patterns.
32:43
SPEAKER_00: Uh a lot of my work right now has to do with trying to figure out how can we incorporate structured reasoning patterns so that the outputs that are produced are at least based, not just um not we don't just rely on the LLM's own internal process, but we kind of structure the process so that we get something that's can be can be tracked and we have some kind of auditing capability, right?
33:03
SPEAKER_00: At the end of the day, if the LLM gives you wrong answers, you kind of want to know why, so that you can try to fix it.
33:08
SPEAKER_00: Um it's really hard to do that with today's LLMs because they're giant black boxes that we have no idea how they work internally.
33:15
SPEAKER_00: Um so hopefully going forward, we'll have better systems that can sort of scaffold that LLM so that at least we can start to trust the output more.
33:23
SPEAKER_01: Yeah, I think that's it.
33:24
SPEAKER_01: It's like we need to have really good guardrails on something like this because this has great clinical utility.
33:30
SPEAKER_01: And in fact, in this paper, they were talking about how they make 300 summaries every weekday.
33:34
SPEAKER_01: And that, you know, most people said, hey, this this decreases the time that I spend trying to review a chart.
33:40
SPEAKER_01: Yeah.
33:40
SPEAKER_01: This is useful.
33:41
SPEAKER_01: Um, you know, the most people said they didn't see factually incorrect stuff, but some people did.
33:48
SPEAKER_01: And as you were mentioning, just because you don't see it does doesn't mean it isn't there, especially when it comes to like an omission or a caveat.
33:55
SPEAKER_01: So that that I take with like sort of a grain of salt.
33:58
SPEAKER_01: However, I can imagine something like this kind of speeding up how quickly you can understand who a patient is, what they're going through, what their concerns might be, what their goals might be, and then say, okay, well, these are the next steps that we're gonna take.
34:12
SPEAKER_01: And so you spend the time working with the patient on their interventions that are gonna help them in the long run, whether they're, you know, social or other, other.
34:21
SPEAKER_01: Um, and then you spend less time sort of in that chart review mode, which can just take forever.
34:27
SPEAKER_00: Yeah, yeah, yeah.
34:28
SPEAKER_00: And and you know, you as a human might be, I mean, the often argument the argument is made is that the human is also makes you know omissions and errors and hallucinations and all that stuff.
34:38
SPEAKER_00: That I mean, maybe not hallucinations.
34:40
SPEAKER_01: I was gonna say, I'm not sure about the hallucinations as much, but I'm uh yes, when you're reviewing tons and tons of material, admitting it things.
34:46
SPEAKER_01: You might miss things, right?
34:47
SPEAKER_01: Or you might see something chronologically and not realize that X has has happened already and then why at why time it was fixed or whatever.
34:56
SPEAKER_01: Like, sure, we're not um, yeah, we we are error prone too, right?
35:00
SPEAKER_01: Like, yeah, but in a different way.
35:02
SPEAKER_00: But I was gonna say, I want people to realize that the reason this seems so exciting is because we can dump a ton of text to the LLM and it produces what seems like a coherent summary.
35:13
SPEAKER_00: That is very exciting and it almost seems like a silver bullet to all our problems.
35:17
SPEAKER_00: And it what I'm trying to state is that it's not, and there are issues, some of them much more subtle than others, that surface at times that you have no way to predict.
35:25
SPEAKER_00: And that makes this super challenging as um uh a tool to use because it's kind of random.
35:30
SPEAKER_00: It can do things, it can do amazing things, but it can also do things that you can't control exactly.
35:36
SPEAKER_00: And and that makes, I think, using these tools, one has to be very careful in how they're deployed, how they're used, and then how they're evaluated, right?
35:44
SPEAKER_00: We talked about that as well.
35:45
SPEAKER_00: Um, I'm excited for this direction.
35:47
SPEAKER_01: Yeah, I know.
35:48
SPEAKER_01: That's what I was gonna say.
35:49
SPEAKER_01: Like, oh, well, so what do we do in this case?
35:51
SPEAKER_01: Because I do feel like it's a really great use case.
35:55
SPEAKER_01: I mean, I think there's some component you were talking about, how it's a little bit of a band-aid because you have different data and different sources and you're trying to like synthesize all of it and it's coming across in dynamic and nuanced ways, and some of it's qualitative and some of it's quantitative.
36:09
SPEAKER_01: And so this is like, oh, well, can we just dump it all into the LLM and it will just tell us?
36:14
SPEAKER_01: So I do think there's like a little bit of a band-aid component here where it's like, well, maybe if the data were structured better and stored better, it would be easier for anyone to look at.
36:24
SPEAKER_01: Yes.
36:24
SPEAKER_01: But I also think this is this is like a really cool and very practical opportunity when you think about a population as a whole, is like, hey, how can we intervene not just on like 15 people, but 500 people because we can get the summaries faster, which means we can spend our human time actually talking to the patient and figuring out, hey, like we need to get you housing or hey, like we need to get you access to a new primary care provider, whatever it is.
36:49
SPEAKER_00: Yes.
36:50
SPEAKER_00: And I think the way to get better at this, and I think they're doing some of that, is in fact having humans not just look at the output and see if it's correct, but humans when they when summaries are generated, whether it's generated by a human or an LM, to ask why one summary works and why another doesn't.
37:05
SPEAKER_00: Like, what about the summary is correct?
37:08
SPEAKER_00: What about it feels more correct than maybe changing a word here or changing something else?
37:12
SPEAKER_00: Like, why is this the right kind of summary?
37:14
SPEAKER_00: Understand the level of detail, then you'll better understand what the fabrications and omissions are, what the distortions are.
37:20
SPEAKER_00: And understanding that then can guide you in your prompt development because then um you can write better prompts for these LLMs.
37:26
SPEAKER_00: But I think the baseline issue there is that the human has to do a bit of the hard work in really digging in and internalizing the data.
37:33
SPEAKER_00: And I say this because you mentioned this before.
37:35
SPEAKER_00: Humans and LLMs have different ways of thinking.
37:38
SPEAKER_00: When we are given a patient's history or any piece of set of documents that we have to make sense of, our mental process is building world models.
37:46
SPEAKER_00: That means we imagine the patient, we imagine, you know, if it says the patient went for a walk at 8 a.m., we immediately have a visual of that whole setup, what it means to go for a walk at 8 a.m.
37:56
SPEAKER_00: We immediately understand the patient woke up before 8 a.m., obviously.
38:00
SPEAKER_00: We understand things that follow from that.
38:02
SPEAKER_00: They went for the walk, went for a walk, so they must have taken X number of steps.
38:05
SPEAKER_00: We know that, right?
38:06
SPEAKER_00: We know all kinds of information that they're active that morning.
38:09
SPEAKER_00: We things we imply, things we presuppose, all of that is captured in our mental model of the person.
38:14
SPEAKER_00: So as we read more text, we incorporate that, and all of that has to make sense for us all the time.
38:19
SPEAKER_00: That is not how LLMs think.
38:21
SPEAKER_00: They're not sitting there thinking, hey, is this the pay what's this patient about?
38:25
SPEAKER_00: What are they doing?
38:26
SPEAKER_00: Does this all make sense?
38:27
SPEAKER_00: They're not building that kind of model in their head.
38:29
SPEAKER_00: So that's why I think we need a human, at least at these initial stages of development, to still look at some of this data and understand it and understand why the summary works and why it doesn't.
38:38
SPEAKER_00: And once we're able to really synthesize that into the prompt, then we're gonna get better output from the LLMs, I think.
38:44
SPEAKER_01: That's great.
38:45
SPEAKER_01: I mean, I'm so hopeful of this as a use case because I do think um it would let us spend the person hours really intervening on the patients where it matters the most.
38:55
SPEAKER_01: Yes.
38:55
SPEAKER_01: And so having a summarization tool that is accurate and trustworthy is definitely something worth digging into.
39:02
SPEAKER_01: And I think you're right.
39:03
SPEAKER_01: The the more we work on figuring out, hey, what is a human, what would a human do in this situation?
39:09
SPEAKER_01: How is a human thinking this through?
39:11
SPEAKER_01: How can we improve that the accuracy?
39:14
SPEAKER_01: LLM as a judge is one component of it, but there's lots of different components that need to go in, the better we're gonna get at this problem.
39:22
SPEAKER_00: Absolutely.
39:23
SPEAKER_01: All right.
39:23
SPEAKER_01: Well, thanks for joining us.
39:25
SPEAKER_01: I think we'll end here and we'll see you next time on Code and Cure.
39:28
SPEAKER_00: Bye for now,