Latest / Code & Cure / #20 - Google Translate Walked Into An ER And Got A Reality Check

Transcript

#20 - Google Translate Walked Into An ER And Got A Reality Check · Code & Cure

0:00
SPEAKER_02: Your father is being discharged after surgery.
0:03
SPEAKER_02: He doesn't speak any English.
0:05
SPEAKER_02: Would you trust an AI system to translate his care instructions?
0:18
SPEAKER_00: Hello and welcome to Coding Cure, where we decode health in the age of AI.
0:22
SPEAKER_00: My name is Vasant Sarathi, and I'm an AI researcher and cognitive scientist.
0:26
SPEAKER_00: And I'm here with Laura Hagopian.
0:28
SPEAKER_02: I'm an emergency medicine physician and I worked in digital health.
0:32
SPEAKER_02: Awesome.
0:32
SPEAKER_02: So we're going to talk about machine translation today.
0:36
SPEAKER_02: Yeah, and specifically of discharge instructions, which is something I used all the time when I worked in the emergency department.
0:42
SPEAKER_00: Yeah.
0:43
SPEAKER_00: So I I mean, uh not being in the medical field myself, what are discharge instructions and where does translation, where do translations fit in?
0:51
SPEAKER_02: Yeah, sure.
0:52
SPEAKER_02: I mean, uh maybe it's easiest to run through an example.
0:55
SPEAKER_02: Like say you come to the emergency department with an asthma flare, right?
0:59
SPEAKER_02: And now we've treated your asthma flare, you're doing better, and it's time for you to go home.
1:05
SPEAKER_02: Now we have to say, hey, here are here's what, here's what's next.
1:09
SPEAKER_02: And um give basic information about an asthma flare, but also some personalized information, like, hey, here's this new medication you're gonna be taking and you're gonna be on it for five days.
1:18
SPEAKER_02: And here's what you should do if um your symptoms get worse.
1:22
SPEAKER_02: And here's when you need to see your primary care provider again.
1:26
SPEAKER_02: And so there's some sort of generic stuff, but then there's some stuff that's very specific to like your use case.
1:32
SPEAKER_02: And if someone doesn't speak English, what you do is you either have translated instructions, which we had in some languages.
1:40
SPEAKER_02: A printed copy.
1:40
SPEAKER_02: Like a printed copy that you're so you hand it to someone and you go over it with them too, right?
1:45
SPEAKER_02: Like you're gonna give them the instructions verbally, you're gonna give them a packet so that, you know, it's like I can tell you it's impossible to remember all the stuff someone someone says to you.
1:54
SPEAKER_02: So you want to be able to like go back and review it.
1:56
SPEAKER_02: Um, but you can only do that if it's in like the language that you understand.
2:00
SPEAKER_02: Oh, yes, right.
2:02
SPEAKER_02: And so it's it's great when those discharge instructions are are translated.
2:06
SPEAKER_02: And sometimes that generic information, like say general information about asthma, that might be available in Spanish and other languages, right?
2:14
SPEAKER_02: But then the specific information about your medication change and your follow-up and et cetera, that doesn't necessarily get translated.
2:20
SPEAKER_02: And so you often get an interpreter either in person or over the phone or over video to go over those instructions with the patient.
2:28
SPEAKER_02: And then you like hope that they remember them because what you've given them is in English, right there.
2:32
SPEAKER_02: Like that's that not all of it ends up getting translated.
2:35
SPEAKER_00: And they're not necessarily annotating it as you're talking, you know, uh giving them this personalized stuff, right?
2:41
SPEAKER_02: Yeah, they're trying to like absorb it and and take it all in.
2:44
SPEAKER_02: And um, it's this is why the written discharge instructions can be so useful, but of course, like very difficult if they're not in a language that you can review them in.
2:54
SPEAKER_01: Yeah.
2:54
SPEAKER_02: And and by the way, like I worked in inner city emergency departments.
2:58
SPEAKER_02: I I I used interpreters all the time.
3:01
SPEAKER_00: Right.
3:02
SPEAKER_02: Um, there were even times, like occasionally, where especially on the night shift, if you had a rare language, you couldn't find you couldn't find an interpreter even over the phone.
3:10
SPEAKER_02: But most of the most of the time we had them um either in person, and if not in person, you could get a video or a phone interpreter.
3:17
SPEAKER_00: Okay.
3:18
SPEAKER_00: Okay.
3:18
SPEAKER_00: So okay, so that's the world of discharge instructions.
3:21
SPEAKER_00: And the ideal situation there is to get something very personalized in a language that the patient can understand and follow.
3:28
SPEAKER_00: And um, and that's challenging because of course you need interpreters and all that stuff.
3:32
SPEAKER_00: And so, and also it needs to be accurate, right?
3:34
SPEAKER_00: The translation needs to be actually clinically accurate.
3:38
SPEAKER_02: Mm-hmm.
3:39
SPEAKER_02: And like I said before, when you have generic information, you can have that sort of pre-translated, right?
3:46
SPEAKER_02: Yeah.
3:47
SPEAKER_02: But if it's 2 a.m.
3:48
SPEAKER_02: and you're discharging someone who speaks Khmai, and you're trying to give them very specific instructions about their follow-up or their medications, that's really hard.
4:00
SPEAKER_02: That's that's really hard to get interpreted into text in the middle of the night, just as.
4:04
SPEAKER_00: Right.
4:05
SPEAKER_00: Yeah.
4:05
SPEAKER_00: Right, right, right, right.
4:06
SPEAKER_00: So that's where sort of AI slash machine translation techniques come into play, right?
4:11
SPEAKER_00: And um and the idea the hope is that these machines can do what you know you're asking it to do in this sense.
4:18
SPEAKER_02: Um that it's like time, you know, this this even more timely access at your fingertips, right?
4:24
SPEAKER_02: But you have to make sure it's accurate.
4:25
SPEAKER_02: That's that's the thing.
4:26
SPEAKER_02: It's like if it's not accurate, then you're giving people the wrong information.
4:30
SPEAKER_02: Right, right, right.
4:30
SPEAKER_00: And you're not in a position to know that it's accurate because you can't do the you can't tra I can't vet the translation, not in every language.
4:37
SPEAKER_02: I mean, in this paper that we specifically looked at, they looked at um simplified Chinese, Somali, Spanish, and Vietnamese.
4:44
SPEAKER_02: And it's like, well, I I'm I might be okay in Spanish, but not in the in those other languages.
4:49
SPEAKER_00: Yeah, yeah, yeah, yeah.
4:49
SPEAKER_00: And then that's the whole point, right?
4:51
SPEAKER_00: If you were okay with those other languages, then you could just communicate directly in those other languages to the patient.
4:55
SPEAKER_00: But the problem is you there's a lot more languages than you know.
4:58
SPEAKER_02: And ideally, honestly, you're getting an interpreter though.
5:01
SPEAKER_02: Yeah.
5:01
SPEAKER_02: Like ideally, you're getting an interpreter who has like the medical translation background to be able to do this stuff.
5:08
SPEAKER_00: Yeah, yeah, yeah.
5:10
SPEAKER_00: Yeah.
5:10
SPEAKER_00: And and and and honestly, translation has been machine translation has been a task um in the um technology space for a while now, and people have tried many different techniques to get it to be really good.
5:21
SPEAKER_00: And so that's something that's ongoing research, but it's also been around for decades now.
5:25
SPEAKER_00: People have been doing this stuff.
5:26
SPEAKER_02: So this was interesting to me because what what they did in this paper was they said, hey, we're gonna compare like a professional translation to an AI translation.
5:36
SPEAKER_02: And when I started to look under the cover, I was like, under the hood of this, I was like, oh, what kind of AI translation are they using?
5:42
SPEAKER_02: And they said, Oh, well, we were thinking about two different versions.
5:46
SPEAKER_02: One is large language models, LLMs, which we talk about all the time, right?
5:51
SPEAKER_02: And the second was neural machine translation.
5:54
SPEAKER_02: And they ended up choosing that second option, neural machine translation.
5:58
SPEAKER_02: And I wanted to ask you a little bit about that.
6:02
SPEAKER_02: Why is that different than a large language model?
6:05
SPEAKER_02: And like why would one be chosen over another?
6:09
SPEAKER_02: Like, what's the difference between the two?
6:10
SPEAKER_00: Okay.
6:11
SPEAKER_00: So, in order to answer that question, I'm gonna do a little bit of a history lesson slash like step back towards machine translations in general, and then kind of work my way back towards this.
6:20
SPEAKER_00: So bear with me as I Okay, I will bear with you.
6:22
SPEAKER_02: Yes, because I I want to understand this.
6:24
SPEAKER_02: Yeah.
6:24
SPEAKER_00: Yeah, yeah, yeah.
6:25
SPEAKER_00: So let's talk about the problem of translations in general, right?
6:29
SPEAKER_00: There's a lot of issues.
6:30
SPEAKER_00: It's not just that give a sentence has that has five words, give bring me a cup of water or something like that.
6:36
SPEAKER_00: And um, the translation is exactly word for word, a different word in the different language, right?
6:42
SPEAKER_00: Because different languages have different sentence structures.
6:45
SPEAKER_00: The way something is said is different.
6:47
SPEAKER_00: Maybe they have all kinds of politeness norms that are in place.
6:50
SPEAKER_00: So you can't just say bring me caught me.
6:51
SPEAKER_00: You might have to say please in those languages in a particular way.
6:54
SPEAKER_00: Maybe you're addressing uh there might be gender terms uh in the language.
6:58
SPEAKER_00: Uh, maybe you're addressing an elder and then it's different.
7:01
SPEAKER_00: Then you say it completely differently from how you would say it otherwise.
7:04
SPEAKER_00: So there's all of these little nuances.
7:05
SPEAKER_00: So it used to be the case decades ago that people had rule-based systems to do the translations.
7:10
SPEAKER_00: And of course, those rule-based systems uh didn't work because there were too many exceptions, there were too many rules, and they were too brittle in the sense that you could not get this free-flowing natur translation of the whole sentence in context, how it what it means for that particular subculture accurately.
7:28
SPEAKER_00: So that became really challenging.
7:29
SPEAKER_00: That's when machines, uh, machine learning came into play, where people were like, okay, instead of humans figuring out all the rules that are involved in translating a particular language, let's look at um um let's give it data so that we can have it just figure out all the rules for itself, right?
7:46
SPEAKER_00: But of course, if it's going to figure out all the rules for itself, it's going to need data to do it, you know, to do it um to understand what the patterns are in the translations, right?
7:57
SPEAKER_02: So you're not like saying, oh, this is the pattern.
8:00
SPEAKER_02: Yeah, you're not saying you're not telling it what the pattern is.
8:02
SPEAKER_02: Yeah.
8:03
SPEAKER_02: You're actually letting it kind of figure out those patterns for itself, essentially.
8:08
SPEAKER_02: I mean, I don't I don't know if that's fully reasoning, but like it's able to recognize it by repetition, essentially.
8:13
SPEAKER_00: That's it.
8:14
SPEAKER_00: And and because there's so many patterns, so many rules, it's just important.
8:17
SPEAKER_00: And they vary by language, right?
8:19
SPEAKER_00: They vary by language, and like I said, they have all kinds of nuances that are not captured.
8:22
SPEAKER_00: So assuming your data set has all those nuances captured, right?
8:27
SPEAKER_00: Assuming the data set that you have, which is when I say data set, I mean has a set of English, you know, for example, English sentences and the corresponding same meaning, you know, Spanish sentences or whatever, right?
8:38
SPEAKER_02: You're giving it correct translations.
8:39
SPEAKER_02: So you're training it not on like all the data in the world, you're training it on like here is the correct translation from English to Spanish or English to Somali or whatever.
8:47
SPEAKER_00: Yeah, yeah, yeah.
8:48
SPEAKER_00: And remember, again, we're in the this is I'm I'm in back in history a little bit because I'm talking about machine translations as a field.
8:54
SPEAKER_00: Okay.
8:54
SPEAKER_00: Right.
8:55
SPEAKER_00: And so this is the problem they were trying to solve, and they fed figured that machine learning systems are going to do better at it.
9:00
SPEAKER_00: And there's many different ways to do machine learning.
9:03
SPEAKER_00: There is uh linear type of models, there is all kinds of other models that we've talked about in this podcast before.
9:09
SPEAKER_00: Um, and neural networks is another type of model that also does machine learning.
9:14
SPEAKER_00: Now, neural networks are really good at figuring out really complex patterns.
9:18
SPEAKER_00: If these patterns between um the translation that's sort of inherent in this translation, if it's really complex and non-linear in many ways, then you have um neural networks that are actually really good at being able to handle this.
9:31
SPEAKER_00: And uh during the sort of the boom of neural nets and you know, resurgence, I would say, in the 2000 to 2010 range, uh, and that happened again because um neural networks were around forever, but um there were now graphics and hardware that were um there now that that made it that sort of you know, Nvidia graphics cards and GPUs and those things made it so that you could now use neural networks again because they were very computationally heavy.
9:56
SPEAKER_00: Okay.
9:57
SPEAKER_00: So the hardware kind of caught up, and now people brought neural networks back, and all of a sudden you could do all these different tasks, uh, which are really difficult, and you could use the powerful neural networks behind the scenes and to do neural machine translation.
10:09
SPEAKER_00: Well, to do neural machine translation, but to do various other tasks image classification.
10:13
SPEAKER_00: Is there a cat or a dog in this picture?
10:14
SPEAKER_00: Yeah, yeah.
10:15
SPEAKER_00: Like all these other tasks came up.
10:17
SPEAKER_00: Now, when you think about a neural network, um, you're giving it some input, right?
10:22
SPEAKER_00: And in an image, it's always the same size.
10:25
SPEAKER_00: You're always giving it an image that is you know, how many of a pixels by how many of a pixels?
10:30
SPEAKER_00: And that means there is you know so many million pixels, but that number is fixed.
10:35
SPEAKER_00: And you're asking it to give you an answer that is also fixed.
10:38
SPEAKER_00: Is it a cat, is it a dog, maybe like 12 other things that it could be, right?
10:42
SPEAKER_00: And in those set settings, when you build a neural network out, it's very fixed.
10:46
SPEAKER_00: And so in some sense, you can once you've got those parameters set, you can just give it data, you know, pictures with the correct answers, and then train it.
10:54
SPEAKER_00: Right.
10:55
SPEAKER_00: And that's straightforward.
10:56
SPEAKER_00: Language is a little different.
10:57
SPEAKER_00: Rang language is a form of sequential data where you're giving it words one after the other.
11:01
SPEAKER_00: It's sort of time series when you speak to each other.
11:03
SPEAKER_00: We're not just saying, you know, all the 12 sentences that I'm about to say in one go to you, not necessarily happening quite that way.
11:10
SPEAKER_00: But maybe you are processing that way a little bit.
11:12
SPEAKER_02: Right.
11:13
SPEAKER_02: But like what I say next may depend on what I say said before or what you said before in the conversation.
11:18
SPEAKER_02: Like there is like there's like big contextual elements of this.
11:21
SPEAKER_02: Yeah.
11:22
SPEAKER_00: And so, and also you don't know how long something is going to be that's a meaningful chunk that needs to be analyzed.
11:28
SPEAKER_02: I mean, I could talk forever, right?
11:30
SPEAKER_00: There you go.
11:30
SPEAKER_00: So it's unlike an image in the sense that it's not a fixed length, it's uh it's sort of ongoing, and uh there's sort of a time series time relationship between them.
11:38
SPEAKER_00: In an image, there's no time relationship between the pixels.
11:41
SPEAKER_00: There's like locality, like pixels close to this are maybe similar to other pixels close to this other piece of the uh, you know, image, but that's not that's not that issue is not there.
11:51
SPEAKER_00: So you have this sort of time series notion.
11:53
SPEAKER_00: And so people are trying to solve the sequential and in machine translation, you have that, right?
11:57
SPEAKER_00: You send in words and you want to get out sequence of words, no matter how long the sentence is, right?
12:02
SPEAKER_02: Yeah, and you want those what you say at the end has the knowledge of what was said at the beginning.
12:08
SPEAKER_00: Right, exactly.
12:08
SPEAKER_00: So people started developing new architectures, new ways of organizing the neural networks to enable that.
12:15
SPEAKER_00: And some of those involved keeping track of memory a little bit.
12:18
SPEAKER_00: And one of those um techniques was called recurrent neural networks, and it was a very powerful technique in which you send it one word at a time and it sort of remembered what was said before, and then you'd output, and then you can, you know, it would output one word at a time.
12:31
SPEAKER_00: Um and it was able to do uh, you know, quite a bit, and it was actually quite quite amazing at the time.
12:36
SPEAKER_00: Um, the issues that it had was that it would, um, if you give it a really long sequence, it would forget like the thing that you said first.
12:42
SPEAKER_00: And as a result, the translations were not very good.
12:46
SPEAKER_00: Um at least not by the end.
12:48
SPEAKER_00: Right.
12:48
SPEAKER_00: And so then people came up and said, wait a minute, we need another architecture, which just means another way of arranging the neural network um to produce better results.
12:56
SPEAKER_00: And some of the issues that the neural uh recurrent neural networks had was remembering uh the context of the whole sentence that was coming in.
13:02
SPEAKER_00: Um and so they came up with these various other techniques.
13:04
SPEAKER_00: I I'm gonna use jargony words right now without expanding them only because it doesn't matter as much.
13:09
SPEAKER_00: But just makes you sound smart.
13:10
SPEAKER_00: It makes me sound smart.
13:11
SPEAKER_00: Yeah, buy LSTMs and LSTMs and all of this stuff and encoder decoder networks and such.
13:16
SPEAKER_02: Yep, you do.
13:16
SPEAKER_02: You sound smart.
13:17
SPEAKER_02: There you go.
13:18
SPEAKER_02: Thank you.
13:18
SPEAKER_00: I appreciate that.
13:20
SPEAKER_00: But um, but all of these guys, all of these uh different architectures had the same idea, which was you send in a bunch of words, and then now what you would do is you would capture the meaning of those words in um some kind of numerical representation inside the machine, and then you would use that whole captured meaning to then produce the output.
13:38
SPEAKER_00: So you notice how it's different from a recurrent neural network where you're sending in one word at a time and one word's coming out to now like getting a little bit more, capturing the meaning of that, and then outputting what you need.
13:50
SPEAKER_00: So it's a little bit more, uh, it takes it, it remembers more.
13:53
SPEAKER_00: It does a better job of remembering things uh across longer, you know, inputs.
13:58
SPEAKER_02: Which you, I mean, if the sentence structure is varied from language to language, it makes more a lot of sense that that would like I'm just thinking about an example in Spanish where you say, like, I like coffee, and then when you say it in Spanish, you're saying me gusta, which is like, this is a pleas, like coffee is pleasing to me or something.
14:14
SPEAKER_02: Like, do you know what I mean?
14:15
SPEAKER_02: That the sentence structure is different, the way that you say the words is different.
14:18
SPEAKER_02: It's not exactly parallel.
14:20
SPEAKER_02: You can't, you can't do I like the same way in Spanish and in English.
14:25
SPEAKER_02: Yeah.
14:25
SPEAKER_00: Yes.
14:26
SPEAKER_00: And one, I think one quick sidebar here is that all neural networks, they compress information.
14:32
SPEAKER_00: That is the core idea of intelligence here, which is that you get all of this data that it's learned about, that it's trained on, and it's all compressed.
14:41
SPEAKER_00: And when I say compressed, I mean think of literally a file on your machine, on your computer.
14:46
SPEAKER_00: Now imagine you had um a bunch of data, and the data was, I don't know, 10 megabytes or something, right?
14:53
SPEAKER_00: Huge amount, I mean some amount of data.
14:55
SPEAKER_00: The learned machine learning model, which is also a file in your computer, would be much less, would be like one megabyte.
15:02
SPEAKER_02: So can I can I make a comparison here?
15:04
SPEAKER_02: Yeah.
15:04
SPEAKER_02: I know we talk about large language models all the time, but this this version to me, it sounds like a small language, small language model, right?
15:11
SPEAKER_02: Like you're a language, but you're right.
15:13
SPEAKER_02: Yeah, like we're trading on like small amounts of data, and the data that's going into it is compressed, and it's only about translation.
15:20
SPEAKER_02: Yes.
15:20
SPEAKER_02: It's not about all the other things that you could find in the world.
15:24
SPEAKER_02: Yeah.
15:24
SPEAKER_02: So it's like, yes, that's a good comparison.
15:26
SPEAKER_02: It's a instead of an LLM, it's an SLM.
15:29
SPEAKER_00: Which is a separate thing, by the way.
15:30
SPEAKER_00: You're entering a uh you're opening up Pandora's box of a different set of problems.
15:34
SPEAKER_00: You're welcome.
15:35
SPEAKER_00: But so so for all the people who know what small language models are, that's actually a thing.
15:39
SPEAKER_00: That's actually a thing.
15:40
SPEAKER_00: So this is not what we're talking about.
15:41
SPEAKER_00: But that said, that's okay.
15:43
SPEAKER_00: That's the concept is there.
15:44
SPEAKER_00: There's a c there's a notion that you have a more tightly focused um neural model that that is doing a thing.
15:51
SPEAKER_00: Now I wouldn't call it a language model, it's just a small model, right?
15:53
SPEAKER_00: Just a small focus model.
15:54
SPEAKER_02: Small model that is focused on one goal.
15:57
SPEAKER_02: It's not like you go in and ask it anything, you're asking it to translate.
16:00
SPEAKER_02: And it could be to multiple languages, but like that's what that's what its goal is, and it's not doing other stuff.
16:06
SPEAKER_00: That's right.
16:07
SPEAKER_00: And uh the what's really cool is that you uh so people worked on this problem and said, okay, let me try to, you know, do this compression thing.
16:15
SPEAKER_00: So so all neural networks do compression, right?
16:17
SPEAKER_00: So that's the basic idea is you compress the data so that you get those rules of translation in a compact representation that nobody can understand because it's all just numbers in a machine.
16:27
SPEAKER_00: And so then you're um compressing all of that information and you're not, you know, kind of you're throwing out irrelevant details, but then using that compressed knowledge to then drive what uh the output is going to be.
16:43
SPEAKER_00: So compression is a really important part of any machine learning model because that's what that's the value in it, right?
16:50
SPEAKER_00: And so um now we're in a world where we're able to do that, but there were still issues with this new architecture because it was not able to capture all the context and it was doing a poor job of um um a number of other issues that it had, which was specifically related to this notion called attention, which was this idea that words in a sentence are linked in a uh the meaning of a particular word in a sentence is connected to all the other words in a sentence in a particular way.
17:18
SPEAKER_00: And depending on how you use the word, it would mean different things in different contexts.
17:23
SPEAKER_00: And that was really hard for these systems to do correctly.
17:27
SPEAKER_00: So um in 2017, a landmark paper was released.
17:31
SPEAKER_00: Um, it's called Attention Is All You Need.
17:33
SPEAKER_00: Uh, this was by uh folks at Google, and it was the first paper introducing the transformer architecture.
17:39
SPEAKER_00: And the transformer architecture is what all LLMs right now are based on.
17:44
SPEAKER_02: Um this came the everything came from like trying to translate.
17:48
SPEAKER_00: They were trying to translate, yeah.
17:49
SPEAKER_00: So that's really cool.
17:50
SPEAKER_00: Interesting, right?
17:51
SPEAKER_00: And so 2017, they came out with this paper and they said, look, you know what?
17:54
SPEAKER_00: Instead of sending these words one at a time, and instead of worrying about hoping that those sentences that you send one at a time is compressed correctly, let's just send everything in parallel and have the system just figure out what's important, like which word relates to which word, and just give it a bunch of data and have it figure it out.
18:13
SPEAKER_00: And they changed the architecture quite a bit, and it obviously did really well.
18:18
SPEAKER_00: So the transformers did really well for machine, neural machine translation purposes, and uh, which is wonderful, which is you know a good, you know, was a very big step in neural machine translation.
18:28
SPEAKER_02: And that's what they use in this paper, right?
18:29
SPEAKER_00: They chose to use that over the large language models.
18:32
SPEAKER_00: Yes, and it's a language model in the sense that you're uh you're sending in language and you're identifying a pattern of use of that language.
18:42
SPEAKER_00: And when I say language, I just mean like words, right?
18:44
SPEAKER_00: Right.
18:44
SPEAKER_00: So you're sending in words in whatever language, and you're finding out the inherent pattern in the data set of how those words are used.
18:51
SPEAKER_00: Now, if your data set had other languages and stuff like that, you're gonna get all of those patterns as well.
18:56
SPEAKER_00: And um, the remarkable piece of that whole experience of doing the machine translation and going through the transformer architecture and everything else was the dawn of LLMs and modern AI.
19:08
SPEAKER_00: And that only happened because somebody thought to say, hey, what if we just gave it all of the internet?
19:13
SPEAKER_00: What would happen then?
19:14
SPEAKER_00: And if you throw enough money at this problem and enough resources and all of the internet, you get large language models, and all of a sudden, they not only can they do machine translation, they can do all kinds of other things.
19:26
SPEAKER_00: Because it turns out our language use demonstrates not just how we understand and talk to each other, but also potentially some reasoning mechanisms that are in our brain are also evident in the language.
19:37
SPEAKER_00: So that's why LLMs all of a sudden are more generalized, they're not just doing translations, they're doing other things.
19:43
SPEAKER_02: Oh, so interesting.
19:44
SPEAKER_02: And so are L like I understand that these neural machine translation models, they're trained on multiple languages.
19:50
SPEAKER_02: Are LLMs like trained on lots of languages, or are they mostly trained in English?
19:55
SPEAKER_02: I'm just curious.
19:56
SPEAKER_00: So um, again, it's all of the internet, so all of the internet contains all of the languages that are on the internet.
20:01
SPEAKER_00: Uh, there's a disproportionately high amount of English usage on the internet.
20:06
SPEAKER_00: So obviously, LLMs uh best understand English.
20:10
SPEAKER_00: Um, there are languages like Spanish and German that also have good representations, Hindi, Mandarin, and so on.
20:16
SPEAKER_00: So those are also captured.
20:17
SPEAKER_00: Um, but there are lots of languages that you know scientists call low resource languages that don't have as much representation, and LLMs will do a poorer job at those.
20:27
SPEAKER_00: Now, I do want to say one thing, which is uh, and this paper talks about it, and people meant use this term all the time, neural machine translation.
20:34
SPEAKER_00: And there's a sort of sense that there's a difference between neural machine translation and LLMs, and there is, and like we talked about before, uh Google Translate, right?
20:44
SPEAKER_00: Let's take take the example of Google Translate.
20:46
SPEAKER_00: Versus like Gemini, they're not even verses, just let's look at Google Translate by itself.
20:51
SPEAKER_00: Um, we go on Google.com and we type in translate, and it pops up this little box on the top that says Google Translate, gives you the options of which language to select, and you type in an input and you hit a button and it gives you an output, right?
21:02
SPEAKER_00: Now ask yourself this question Can you uh ask a Google Translate to say, hey, make this more informal?
21:10
SPEAKER_00: No, you can't.
21:11
SPEAKER_00: Can you say, hey, here's a whole body of text?
21:15
SPEAKER_00: Can you give me an accurate translation of everything in the text?
21:19
SPEAKER_00: I mean, you you don't know the answer to that, but it probably won't be able to do that accurately because it doesn't capture all of the context.
21:24
SPEAKER_00: Um, can you say, hey, I like what you said there, but can we also add um something about the you know, some other detail?
21:31
SPEAKER_00: Like if you told that to Google Translate, would it be able to do that?
21:34
SPEAKER_02: No, but but hold on.
21:35
SPEAKER_02: On the other side, on the other hand, like LLMs can hallucinate, and I don't think I've ever seen Google Translate do anything weird, like yeah, yeah.
21:45
SPEAKER_00: That's it, and so that's the distinction, right?
21:47
SPEAKER_00: So you have to think about the use cases.
21:49
SPEAKER_00: If your input to your system is j if the if the job of the system is to do one thing, which is translate, right, you can't all of a sudden ask it all kinds of other questions, right?
22:00
SPEAKER_00: So you can that means what it's expecting as input is some sentence, and what it's expected to produce is another sentence in a different language that captures the same meaning.
22:12
SPEAKER_02: But if you want it to do more, if you want it to you want to be able to tweak it to make it more conversational, or if you want to improve the readability or something, yeah, exactly.
22:21
SPEAKER_02: Now and say, Oh, I need to make sure that this is simplified.
22:24
SPEAKER_02: I can't have the discharge instructions be complex.
22:26
SPEAKER_02: Yes.
22:27
SPEAKER_02: Now you're talking about going into LOM territory.
22:30
SPEAKER_00: Yes.
22:30
SPEAKER_00: But you could, in theory, say, okay, I'll train a separate neural machine translation model to only produce fifth grade level readable text.
22:40
SPEAKER_00: But then I'd have to give it data, input data, what the input might look like, and then output data that is all fifth grade level readable text, and then I train it and then it's going to do that thing.
22:49
SPEAKER_00: But that's a whole training cycle that you have to go through.
22:52
SPEAKER_00: And um, but but LLMs, you don't have to do that because because of the fact that they've absorbed the internet and they understand to some degree um some aspects of human reasoning, they can and and because they have been trained to follow your instructions, which is a separate thing, by the way, they can understand you when you say, Hey, make this fifth grade reading level instead, right?
23:13
SPEAKER_02: Okay, yeah, that makes sense.
23:14
SPEAKER_02: So in the paper we we read, they ended up choosing neural machine translation for a number of reasons.
23:21
SPEAKER_02: But one thing was that they were like concerned about what's the accuracy or could this hallucinate?
23:27
SPEAKER_02: Because when you're translating in real time and then you're giving that to someone as discharge instructions and you don't have someone to QA it, the accuracy ends up being really important.
23:36
SPEAKER_02: I mean, they looked at other things too, right?
23:38
SPEAKER_01: Yeah.
23:38
SPEAKER_02: They looked at like fluency for clarity and flow and vocabulary and grammar.
23:43
SPEAKER_02: They looked at the meaning, like, does it actually say the same thing as the original text was saying?
23:48
SPEAKER_02: Um, they looked at how how accurate or complete or appropriate was.
23:52
SPEAKER_02: And if there were errors, they looked at like how severe is this error and how could it impact someone clinically?
23:58
SPEAKER_02: Um, but I think there's like a very big difference between relying on an AI translation in real time and not having someone Q be able to QA it versus having someone be able to QA a translation that comes out of one of these systems.
24:17
SPEAKER_00: Yeah, and that's fair.
24:18
SPEAKER_00: But I have to say though, the state of the art machine translation still uses transformer models.
24:24
SPEAKER_00: Um, that's what everyone uses.
24:26
SPEAKER_00: And actually, many of the LLMs that are out there right now are quite competitive with the state-of-the-art neural machine translation techniques.
24:33
SPEAKER_00: So even on pure translation tasks where you only input is, you know, when when they when they put head-to-head against the best machine neural machine translation systems, LLMs in terms of accuracy are still gonna be quite quite competitive.
24:44
SPEAKER_00: Now, LLMs use a lot more compute power, right?
24:48
SPEAKER_00: Because they're doing, they're they're meant to do a whole host of things.
24:51
SPEAKER_00: Right.
24:51
SPEAKER_00: They don't have to do just neural machine translation.
24:53
SPEAKER_00: So they're bigger, they're potentially slower, because again, the other ones are smaller models like we talked about before.
25:00
SPEAKER_00: Right.
25:00
SPEAKER_00: So they're gonna be faster, they're gonna have low latency, which means they're gonna be able to produce your answers much quicker, which in real time, if you're thinking about a system that as you're talking translates for you, kind of futuristic in some sense, but there's some there are some um products out there that I've seen that already do that.
25:16
SPEAKER_00: Right.
25:16
SPEAKER_00: My guess is they don't use LLMs because they don't need to.
25:19
SPEAKER_00: They that's not what they're supposed to do.
25:20
SPEAKER_02: That's not the goal.
25:21
SPEAKER_02: And you and the goal needs to be like, oh, let's do this as fast as we can.
25:24
SPEAKER_02: Yes.
25:25
SPEAKER_02: Um, in as accurate a way as possible.
25:27
SPEAKER_02: Right.
25:28
SPEAKER_02: Okay.
25:28
SPEAKER_00: So you have to really think about what the goal is of the task.
25:30
SPEAKER_00: If your task goal requires you to have some kind of um nuance in the way it's written, or there's some styling issues, and there is potentially context involved, potentially rewrites and such, LLMs are going to be what you need.
25:43
SPEAKER_00: But if your goal is very quick translation of incoming text accurately, then you want, you don't, you don't need an LLM.
25:50
SPEAKER_00: It's potentially problematic to use an LLM.
25:53
SPEAKER_02: Right.
25:53
SPEAKER_02: This is so interesting.
25:55
SPEAKER_02: And I I do want to like go through the findings of the paper because we've been we've been talking a lot about like how do machines translate, but the the paper actually looked at, you know, this uh my poorly termed smaller models, like neural machine translation models versus um professionally translated um items and for the discharge summaries.
26:15
SPEAKER_02: And the AI translations were mostly inferior.
26:20
SPEAKER_02: They were like okay for Spanish, um, but did not do well in simplified Chinese, Somali, and Vietnamese.
26:26
SPEAKER_02: But also at the same time, they did not train on that many examples in the other languages.
26:32
SPEAKER_02: They trained on way more examples in Spanish.
26:34
SPEAKER_02: And so I think it's very interesting, having heard what you've said, is like the the more data that you train it on, the more information that it has about this in English is this in Somali, the better job it's going to do.
26:48
SPEAKER_02: And so it had the most information to be trained on in Spanish and it had the least information in Somali.
26:55
SPEAKER_00: Yes.
26:55
SPEAKER_00: And if you think about an LLM, an LLM could provide that extra context, right?
26:59
SPEAKER_00: Potentially culturally relevant information that may be useful to support uh translations in low resource languages.
27:05
SPEAKER_00: So if you think about your Somali interpreter, human interpreter, they grew up in a culture so they're familiar with all the cultural norms, all the politeness norms, all the speaking norms.
27:15
SPEAKER_00: They are pro they're comfortable with the different subcultures that might exist.
27:18
SPEAKER_00: So it's not just enough to be Somali, but maybe you need to be a specific tribe or a specific group of people within that.
27:24
SPEAKER_00: They know that right away.
27:25
SPEAKER_01: Yeah.
27:25
SPEAKER_00: They know how translations work differently in those settings.
27:29
SPEAKER_00: Your pure machine translation might not have that data.
27:32
SPEAKER_00: LLMs also might not have the data, but they might have a little bit more.
27:35
SPEAKER_00: They might know that there is a country called Somalia, that there is these regions, that there are maybe some politeness stuff into it.
27:41
SPEAKER_00: So as long as you are um, you know, you could, in theory, prompt the LLMs by telling them, look at these other factors when providing your translation.
27:49
SPEAKER_00: And so then you're providing potentially more nuanced translations that are closer.
27:53
SPEAKER_00: So there's value in the LLM, is what I'm trying to say, even in those cases.
27:57
SPEAKER_02: Yeah, it was very interesting because you're bringing back memories in the ER where I would actually use an interpreter not just for, hey, can you can you can you say this to the patient?
28:07
SPEAKER_02: But also I would talk to them about what were the cultural norms or like how what's the best way to tell this diagnosis to someone?
28:14
SPEAKER_02: Should it be, you know, should all the family be in the room versus not?
28:17
SPEAKER_02: Like I, you know, when you don't know the cultural norms in an area, it it can definitely affect what your relationship with a patient or family might be and getting their expertise on that, it's something that, you know, it's I I had to ask those questions essentially.
28:32
SPEAKER_00: Right, right, right, right.
28:33
SPEAKER_00: Exactly.
28:34
SPEAKER_00: Um, but yeah, I think that's kind of where the LLM still has value.
28:38
SPEAKER_00: Uh, but yes, hallucination is a huge problem.
28:41
SPEAKER_00: And what if it's just making up these norms, right?
28:43
SPEAKER_00: You have no way to know one way or the other, and having a human in the loop to confirm that it's in fact makes sense, you know, the human's job can be pretty straightforward, right?
28:51
SPEAKER_00: They can look at the translation and immediately know because they will know it immediately whether it actually is like weird or if it's like normal.
28:58
SPEAKER_02: And and honestly, like there's like in the Affordable Care Act, there's some um information about this that like essentially when accuracy is essential, you can't have automated translation of medical documents unless you have some sort of QA on it, right?
29:17
SPEAKER_02: If it's like a critical piece of text that has really important clinical information, you can't just have automated translation without a human review.
29:26
SPEAKER_02: So it's sort of like baked into some of this.
29:28
SPEAKER_02: And I think this concept of having a human in the loop makes a lot of sense.
29:33
SPEAKER_02: It's just how do you implement that at 2 a.m.
29:36
SPEAKER_02: when you're not able to find someone to do that?
29:40
SPEAKER_02: I don't know.
29:40
SPEAKER_02: I don't have I don't have an answer to that question, but I I think this does open up the door to really be able to communicate with patients better and provide them not only verbal but written information about what's next and what they they may need to do in order to prevent them from bouncing back to the ER or needing a hospitalization or whatnot.
29:59
SPEAKER_02: Right.
30:00
SPEAKER_00: And you know, I I we didn't talk about this, but there's another aspect of context, which is the medical context and jargon and specific medical understanding.
30:06
SPEAKER_00: I mean, again, if you're doing a neural machine translation model, you're gonna have to uh factor that in.
30:11
SPEAKER_00: And so it has to be catered to medical discharge, I mean in this instance, discharge salaries, uh how they're written, and there's a style in which they're written, there's a there's a type of words they use.
30:20
SPEAKER_00: The system has to try to understand all of that.
30:23
SPEAKER_00: The LLMs might come in with some of that knowledge, right?
30:25
SPEAKER_00: They already have a lot of uh quote unquote, I mean, medical knowledge in the sense that they might understand some of the jargon.
30:31
SPEAKER_00: And so they might provide a translation that already takes that into account.
30:35
SPEAKER_00: Um, but having the human in the loop again, again, like I said, is going to be critical, um, you know, in a sense, both from the cultural standpoint, but also from the medical standpoint.
30:46
SPEAKER_02: Yeah, yeah.
30:47
SPEAKER_02: Well, I think we can kind of wrap up with this.
30:50
SPEAKER_02: I is this is very this was a very interesting topic to me because um I think there's a lot of promise here, and there's lots of different AI models that can be used to do this, and they have different use cases.
31:01
SPEAKER_02: Um, what you know, I think is like eye-opening.
31:03
SPEAKER_02: We talked about Google Translate versus an LLM.
31:06
SPEAKER_02: And there's definitely benefits to all of them, but also the importance of like being able to prompt it appropriately, being able to fine-tune it, et cetera.
31:14
SPEAKER_02: So I I know this paper didn't find um, you know, great results in terms of machine translation, but I think there's a lot of promise here if we keep pushing the envelope on it.
31:25
SPEAKER_02: Yeah.
31:25
SPEAKER_02: And there's a lot for patients to benefit from too.
31:28
SPEAKER_00: Agreed.
31:29
SPEAKER_02: All right.
31:30
SPEAKER_02: Well, we will see you next time on Code and Cure.
31:32
SPEAKER_02: Thank you for joining us.