Latest / Code & Cure / #53 - Pretty Pictures, Dangerous Mistakes

Transcript

#53 - Pretty Pictures, Dangerous Mistakes · Code & Cure

0:00
SPEAKER_00: AI medical images are fast and free until you factor in the cost of clinical errors and systemic bias.
0:09
SPEAKER_00: Today we're looking at the hidden tacks of easy AI art.
0:22
SPEAKER_01: Hello and welcome back to Code and Cure, the podcast where we discuss decoding health in the age of AI.
0:28
SPEAKER_01: My name is Vasant Sarathi.
0:29
SPEAKER_01: I'm an AI researcher and a cognitive scientist, and I'm here with Laura Hagopian.
0:34
SPEAKER_00: I'm an emergency medicine physician.
0:35
SPEAKER_00: Images.
0:36
SPEAKER_00: Images!
0:37
SPEAKER_01: Yeah.
0:38
SPEAKER_01: Sorry, I didn't have any words to start this off.
0:40
SPEAKER_00: No, that's fine.
0:41
SPEAKER_00: I think I think this ability to produce images like at the snap of your fingers is something that's really cool about AI.
0:50
SPEAKER_00: Yes.
0:51
SPEAKER_00: And could be very useful if used appropriately, especially in terms of medical education, because a lot of what you're doing and you're learning is inherently visual.
1:04
SPEAKER_00: Right?
1:04
SPEAKER_00: Yes.
1:05
SPEAKER_00: Um when you're learning to like examine patients or do a procedure or how to do a specific type of yeah, like suture or splint or something for someone, the anatomy that you're looking at, a lot of that is like very visual.
1:19
SPEAKER_00: Yeah.
1:19
SPEAKER_00: And so oftentimes I'd be like looking at my friends in class or looking at pictures that have been drawn by a famous anatomy guy, Netter.
1:30
SPEAKER_00: You know, you're looking for visual stuff all the time so that you can carry out what needs to be done.
1:36
SPEAKER_00: Yeah, and you can remember it better, right?
1:37
SPEAKER_00: Yeah, for sure.
1:38
SPEAKER_01: The orientation, the angles, the specific part of the body, and all of that stuff, right?
1:43
SPEAKER_01: I mean, there's I know there's names for that for everything, but still it's easier when you actually look at it visually.
1:48
SPEAKER_00: Well, it's like one thing to say, okay, first you're going to, you know, find the acromioclavicular joint, then you're going to palpate.
1:55
SPEAKER_00: You know, so it's it's another thing to actually just like look at a picture of the acromioclavicular joint and be like, here's what the bones look like, here's what it looks like on the outside, uh, here's, you know, here's where you would want to palpate it.
2:07
SPEAKER_00: Here's what an x-ray of it looks like.
2:08
SPEAKER_00: So the visual is like so important as you're learning because you're like putting all those pieces together.
2:14
SPEAKER_00: Okay, the acromioclavicular joint, it can have a separation.
2:18
SPEAKER_00: This is how it often happens.
2:19
SPEAKER_00: Here's what an x-ray view would look like, here's what the exam would look like, etc.
2:23
SPEAKER_00: So you're using, I'm just giving a random example, but but you need the visual to like understand it.
2:30
SPEAKER_00: Yes.
2:30
SPEAKER_00: And so if you could sort of mass produce very custom visuals for medical education, that would be awesome.
2:38
SPEAKER_01: Or have individuals just be able to produce images for things that they're learning about, that would also be useful, right?
2:43
SPEAKER_01: Then they themselves would be able to generate it and not rely on a central source that may or may not have the specific question that they have in mind.
2:50
SPEAKER_00: Yeah.
2:50
SPEAKER_01: Or the angle that they have in mind.
2:52
SPEAKER_00: But not surprisingly, like this runs into problems, right?
2:55
SPEAKER_00: Because uh and I guess we'll I'll have you get into some of the specifics about how these models work, but like not surprisingly, it's not always gonna produce the best images.
3:06
SPEAKER_00: So if you have your learners using an image that's not great, then what are they learning?
3:12
SPEAKER_00: What are we teaching the next generation of medical students or PA students or NP, like what are we teaching people?
3:19
SPEAKER_00: We're teaching them the wrong thing.
3:20
SPEAKER_01: Yeah.
3:21
SPEAKER_01: Yeah.
3:21
SPEAKER_00: And so I think you can get like, yes, it could be very useful, and it could also get very hairy very fast and create more problems, essentially.
3:30
SPEAKER_00: And that's what this paper that we're gonna include in the show notes kind of gets into is that they looked at a bunch of different papers that looked at these AI-produced images for medical education purposes.
3:44
SPEAKER_00: And they said, hey, are these good?
3:46
SPEAKER_00: What are the problems that are coming up?
3:48
SPEAKER_00: Right.
3:48
SPEAKER_00: And they came up with two major issues that kind of compound each other.
3:54
SPEAKER_00: One was an issue about representational bias, and two was an issue about clinical fidelity.
3:59
SPEAKER_00: And I think we can unpack each of these a little bit.
4:02
SPEAKER_00: Yeah, let's do it.
4:02
SPEAKER_00: Yeah, yeah.
4:03
SPEAKER_00: Okay, so the representational bias one is like, hey, who's being depicted in these AI images?
4:09
SPEAKER_01: And where this is headed.
4:10
SPEAKER_00: You know where this is headed.
4:12
SPEAKER_00: You can guess.
4:12
SPEAKER_00: Where is this headed?
4:13
SPEAKER_01: Um, mostly white males.
4:15
SPEAKER_00: Are are the doctors?
4:16
SPEAKER_01: Oh, the doctors.
4:16
SPEAKER_00: Yeah, that's exactly what happened.
4:18
SPEAKER_00: So the biases that the data is trained on are the biases that come out in these images.
4:25
SPEAKER_00: And so in in one of these studies, they said, Oh, okay, we want to find, you know, a special uh ear, nose, and throat specialist.
4:32
SPEAKER_00: They found the clinicians were 88% of the time white, 88% of the time male.
4:37
SPEAKER_00: Right.
4:37
SPEAKER_00: Um, and surgeons, again, predominantly male.
4:41
SPEAKER_00: Um and and they did this in other areas as well, like not just you know, physicians.
4:50
SPEAKER_00: Um, they did it for radiation technologists, and the AI outputs in this case showed more minorities than are present in that population.
5:01
SPEAKER_00: So it's not like it went the other way.
5:03
SPEAKER_01: It went the other way.
5:04
SPEAKER_01: It almost made up for it almost made up for its uh sort of bias by going well, it went too far.
5:10
SPEAKER_00: I mean, too far in either direction isn't great.
5:12
SPEAKER_00: Like you want it to represent who the workforce actually is.
5:15
SPEAKER_00: Yeah.
5:16
SPEAKER_00: Right?
5:16
SPEAKER_00: And it and it doesn't.
5:18
SPEAKER_00: In either of these scenarios, it doesn't.
5:20
SPEAKER_00: And so I think that's, you know, you could shrug off a single image or two images, but if you think about doing this at scale, all you're doing is like cementing those biases.
5:32
SPEAKER_01: Well, I was gonna ask, I was gonna ask, I was like, okay, if if the goal is medical education and you're learning, say, a procedure or something, what does it matter who those people look like in the image?
5:42
SPEAKER_01: It's more important about what they're doing and you know, the other side of it.
5:46
SPEAKER_01: But of course, what I'm missing when I ask that question is the fact that every single time an image is shown to someone, there is a certain stereotype that's being pushed forward.
5:56
SPEAKER_01: There's a certain bias that comes with it, and there's a certain presumption or assumption that somehow that image not only represents that procedure, but represents the type of person who does that procedure.
6:08
SPEAKER_01: And that which in turn represents uh can have an impact on whoever's watching those uh videos.
6:13
SPEAKER_01: So there's a much deeper seated issue here.
6:15
SPEAKER_01: It's not just a matter that we want an equal number of all the races or whatever, it's much more about uh not biasing, you know, not not and reinforcing these uh uh false biases.
6:28
SPEAKER_00: Right, exactly.
6:29
SPEAKER_00: And the thing is that if you see the same stuff over and over again, you start to believe that that is how it is.
6:37
SPEAKER_00: Yes.
6:37
SPEAKER_00: Oh, like I'm a female physician.
6:40
SPEAKER_00: Uh, guess there aren't that many of me because all the pictures are of men, right?
6:43
SPEAKER_00: And so it it can narrow or distort the norms of what you're used to seeing.
6:49
SPEAKER_00: And it doesn't just apply to, you know, providers, it could apply to, you know, like the x-ray text, it could apply to the the patients.
6:58
SPEAKER_00: I was gonna say the type of patients they're showing.
7:01
SPEAKER_00: Yeah.
7:02
SPEAKER_00: Like there was one example where it for cancer survivors, they mostly they were mostly female and they were mostly smiling.
7:10
SPEAKER_01: Like I guess they survived cancer, so they're I see you're showing bias there though.
7:16
SPEAKER_00: Like maybe they're you know, just because they're surviving right now doesn't mean that they're thriving.
7:20
SPEAKER_00: Are they all happy or positive?
7:22
SPEAKER_00: Like um, and and why why are they all in this sort of feminine group?
7:28
SPEAKER_00: It doesn't make sense.
7:31
SPEAKER_00: Um, and so if you see cancer survivor over and over as a female, if you're showing that in medical education, then are people gonna just believe that most cancer happens to females?
7:43
SPEAKER_00: That's not true.
7:44
SPEAKER_01: Right, right.
7:45
SPEAKER_01: But it's it's but it's sort of a subconscious bias that's developed over time, which is a problem because people don't then people don't even know that they have it, right?
7:52
SPEAKER_00: Yes.
7:53
SPEAKER_00: And so that that is that is one major issue.
7:57
SPEAKER_00: And then the second major issue they found, and and these compound each other, were issues in clinical fidelity, meaning, you know, things were like actually wrong in the images.
8:07
SPEAKER_00: Yeah.
8:08
SPEAKER_00: Which is not great.
8:09
SPEAKER_00: Right, right.
8:10
SPEAKER_00: When you're trying to teach people something, whatever the something is, right?
8:14
SPEAKER_00: It's like, okay, the anatomy was actually wrong in some of these images.
8:19
SPEAKER_00: Um you know, the the they weren't, they didn't clinically make sense.
8:25
SPEAKER_00: They could lead to misconceptions.
8:28
SPEAKER_00: Um, sometimes they had like the wrong equipment present in the room.
8:33
SPEAKER_01: Yeah.
8:33
SPEAKER_00: Right.
8:34
SPEAKER_00: Um there were hallucinations, there were errors in terms of labeling, um, there were, you know, issues with how realistic it was.
8:44
SPEAKER_00: And so how can you use that for education if you have all of these issues?
8:51
SPEAKER_00: Yeah, exactly.
8:52
SPEAKER_01: Right.
8:53
SPEAKER_00: Right.
8:54
SPEAKER_00: It doesn't make sense.
8:56
SPEAKER_00: Um, there was one example where they looked at dermatology and they found that almost all the dermatology images it produced were on light skin.
9:05
SPEAKER_00: Wow.
9:05
SPEAKER_00: Well, it doesn't help you with darker colored skin then.
9:10
SPEAKER_01: Doesn't it doesn't actually help you learn um yeah.
9:12
SPEAKER_01: You played around with it a little bit too, right?
9:15
SPEAKER_00: I did play around with it because I was like, oh, I wonder if this I will have the same sort of issues that were coming up in the paper.
9:22
SPEAKER_00: And the paper was a systematic review, so it was looking at all these different papers out there.
9:26
SPEAKER_00: And I sort of was like, yeah, let me narrow in on like one random very specific thing and see how it does and see what I get.
9:33
SPEAKER_01: So what'd you find?
9:34
SPEAKER_00: So I gave it a very specific prompt to generate an image of an orthopedic surgeon placing an ulnar gutter splint on a patient who has a boxer's fracture.
9:44
SPEAKER_00: A boxer's fracture is um a hand fracture.
9:48
SPEAKER_00: Okay.
9:48
SPEAKER_00: It's called a boxer's fracture because it's usually because you like punch somebody else, not quite the right way.
9:53
SPEAKER_00: Um, and so you you break the bone that's kind of like underneath where your pinky is.
9:59
SPEAKER_00: Got it.
10:00
SPEAKER_00: Okay.
10:00
SPEAKER_01: Yeah.
10:00
SPEAKER_00: And so there's a specific kind of splint that you put on for this called an ulnar gutter splint.
10:05
SPEAKER_00: And what it does is it immobilizes that part.
10:09
SPEAKER_00: That part.
10:09
SPEAKER_00: And it, you know, extends down into the forearm and and up, and it kind of covers your pinky and your ring finger, and they're bent in a very specific position.
10:19
SPEAKER_00: And so this is something that like a medical student or you know, a a castech or lots of people need to learn how to do, but you want to learn how to do it correctly because if you do it wrong, then you're not immobilizing the area that's broken.
10:33
SPEAKER_00: It's not gonna heal correctly.
10:35
SPEAKER_00: Right.
10:35
SPEAKER_00: Right.
10:36
SPEAKER_00: Um, so I I ran the I ran it twice, and both times I got a white provider and a white patient.
10:44
SPEAKER_00: Um, the provider was male in the first example, and in the second, you can't actually see the provider.
10:50
SPEAKER_00: Okay, so you just see their hands or something.
10:51
SPEAKER_00: You can just see their hands, exactly.
10:53
SPEAKER_00: Okay.
10:54
SPEAKER_00: Um so definitely could, you know, this is my NF2.
10:58
SPEAKER_00: This is not a systematic review, but in my non-systematic review, I mean, this is what they found in the paper.
11:03
SPEAKER_00: So this is shouldn't surprise us at all.
11:05
SPEAKER_00: Yes.
11:05
SPEAKER_00: Um, that they're showing white male providers.
11:08
SPEAKER_00: But the second thing is, and getting into specifics, is that it actually showed the ulnar gutter split splint wrong both times.
11:16
SPEAKER_00: Yeah.
11:16
SPEAKER_00: And if someone was trying to learn how do I do an ulnar gutter splint, in both of the examples, it's like starting to a little bit cover the pinky finger, but it's not covering the ring finger at all.
11:28
SPEAKER_00: And it's not fully covering the part of the hand where the where it's actually fracture happened, yeah.
11:33
SPEAKER_01: Right.
11:33
SPEAKER_00: So it's it wouldn't actually be immobilizing that.
11:35
SPEAKER_00: And the fingers are actually being held at like the wrong angle.
11:38
SPEAKER_00: The fingers are supposed to be kind of flexed down towards the hand.
11:43
SPEAKER_01: Yeah.
11:43
SPEAKER_00: Um, for the splint to work correctly, and they're not.
11:47
SPEAKER_00: And so if you had a learner placing an ulnar gutter splint based on the images shown here, they would do it wrong.
11:55
SPEAKER_00: They would do it wrong.
11:56
SPEAKER_00: And the patient would potentially have a bad outcome because they're where they're supposed to not be able to move, they would be able to move.
12:04
SPEAKER_00: Right, right.
12:05
SPEAKER_00: That's not how you heal a fracture.
12:06
SPEAKER_01: I was gonna say there's a clear right and wrong answer here.
12:08
SPEAKER_01: It's not, yeah, it's not, it's not a question of debate or interpretation or anything like that.
12:12
SPEAKER_01: It's exactly there's a right way to do it, and there's not a right way to do it.
12:15
SPEAKER_01: A wrong way to do it, that is.
12:16
SPEAKER_00: This is a wrong way to do it.
12:17
SPEAKER_01: Right, right, right.
12:18
SPEAKER_01: Exactly.
12:19
SPEAKER_01: So that's okay.
12:20
SPEAKER_01: So that's super interesting, also.
12:21
SPEAKER_01: Like, why did it get that wrong?
12:23
SPEAKER_01: Given that it knows, like if you asked it, explain to me what an you know, ulnor fracture is, it probably would give you the right answer in understanding what it is.
12:31
SPEAKER_01: Because boxer fracture, yeah.
12:33
SPEAKER_01: Box is fracture.
12:34
SPEAKER_01: Um, uh, yeah, right.
12:35
SPEAKER_00: Or if I said step by step, give me, give me how do I do an ulnar gutter splint.
12:39
SPEAKER_00: Right.
12:40
SPEAKER_00: Maybe it would get that.
12:41
SPEAKER_00: I don't know, maybe it would get that correct.
12:42
SPEAKER_00: But that kind of leads me to toss the question back to you, which is like, how are these images even generated?
12:49
SPEAKER_00: And why would it get something like this wrong?
12:52
SPEAKER_01: Well, there's two uh let me let me start off by saying You like that compound question?
12:55
SPEAKER_00: I'm just gonna just just give me all the answers to what's wrong here.
12:58
SPEAKER_01: Two-part question, actually, 20-part question.
13:00
SPEAKER_01: Um, no, I I think the the difference here is that there's two types of models, right?
13:05
SPEAKER_01: There is the LLMs or large language models that people are familiar with that help that write text.
13:11
SPEAKER_01: This is a chatbots.
13:12
SPEAKER_01: You write a bunch of text and answers back in text.
13:14
SPEAKER_00: That's what I use though.
13:15
SPEAKER_00: Right.
13:15
SPEAKER_00: Okay, I use it to make an image.
13:17
SPEAKER_00: Okay.
13:17
SPEAKER_01: You did, you did.
13:18
SPEAKER_01: But what you're using, but what they have behind you use ChatGPT or Claude or whatever, uh, behind the scenes, that's just not one model.
13:24
SPEAKER_01: It's a system.
13:24
SPEAKER_01: So they've already decided which models to apply when uh for whatever question you ask.
13:29
SPEAKER_01: So it's a more complicated system, but uh at its very core, when you're generating text, that's a different type of model from when you're generating images.
13:38
SPEAKER_01: And generating text, and we've talked about this before, is all about predicting the next word, and it learns patterns of human language and it's able to predict the next word given a pattern, given a specific uh you know, sp sequence of words.
13:49
SPEAKER_00: Right.
13:49
SPEAKER_01: Okay and and what we have discovered is remarkably that uh having learned from all of the words on the internet, it knows some common sense knowledge and it might even know some specialized knowledge and answers your uh questions with some degree of uh displaying that knowledge.
14:04
SPEAKER_01: Um image models are very different.
14:06
SPEAKER_01: And people who've used image models before, you know, you might have used Midjourney for art, you might have used uh back in the day there was like stable diffusion, there were some other models like that, and they work differently.
14:15
SPEAKER_01: They're trained differently because they start with images, but the way they're trained is they're actually trained in a very unique way called diffusion um training, where you start from what they do is they have all of these images um in their training set, and they basically add noise to the images to the point where it's completely noisy.
14:32
SPEAKER_01: And then they try to denoise it, remove the noise, and they train a system to automatically start from a noisy state to a nice clean state.
14:40
SPEAKER_01: And when I say noisy state, I mean like you can't recognize anything in the image, it's just a bunch of noise.
14:44
SPEAKER_01: But they're able to then use um a way to add text to condition that noise.
14:50
SPEAKER_01: That is, if you had an image with um uh a label or a caption, right, as your input to this training model, it learns how to make that noisy, but it also learns how to make that noisy with respect to the text given.
15:01
SPEAKER_01: So in the future, you can start with a piece of noise and add the text like you just did, and then it would generate an image.
15:08
SPEAKER_00: But I didn't have any noise really.
15:10
SPEAKER_00: Like it was just like nothing.
15:11
SPEAKER_00: I had nothing, and I got to do it.
15:12
SPEAKER_00: You don't have to provide the noise.
15:13
SPEAKER_01: My point is the noise is just random noise.
15:15
SPEAKER_01: Because it it it just creates, you know, like black, black and white dots on the screen, kind of.
15:19
SPEAKER_00: So it's not like taking, just check my knowledge here.
15:22
SPEAKER_00: It's not like taking the step by step that you might find on the internet of like the first thing that you want to do for an ulnar gutter splint is X, and the second thing is Y, and the third thing is Z.
15:32
SPEAKER_00: It's like actually It might not.
15:34
SPEAKER_01: That said, there might be things on the internet where there is an explanation of this uh procedure together with images, right?
15:40
SPEAKER_01: That is possible.
15:42
SPEAKER_01: If the training set had taken out those images and attached those captions to those images, then yes, it's possible.
15:48
SPEAKER_01: I don't really know the specifics of the training process.
15:51
SPEAKER_01: Those are often um, you know, built pretty quickly with large amounts of text.
15:55
SPEAKER_01: So they just pull all the images from the internet, they add a lot a lot of text associated with it, and they just clamp it on.
16:00
SPEAKER_01: It's not necessarily like carefully curated or even if it is curated, it's not like carefully designed to make sure that the procedure perfectly matches the images or anything like that.
16:10
SPEAKER_01: So they're not like a cross-check.
16:12
SPEAKER_01: No, especially for these large ones, right?
16:13
SPEAKER_01: These large models.
16:14
SPEAKER_01: Now, I'm not saying there isn't any uh specialized medical education model out there that someone's making that might be doing that, but I'm saying if you take your standard GP chat GPTs and those kinds of uh tools that are more general, they take very large data sets and uh of images and they create these uh these these um basically these diffusion models that you can then just like write into and ask ask it to show you an image of this and that, and it tries to get there as quickly as possible.
16:42
SPEAKER_00: And it looks like really realistic, to be honest with you.
16:45
SPEAKER_00: Like when I was looking at it, I was like, oh, if I didn't know better, I'd be like, well.
16:49
SPEAKER_00: Yeah.
16:50
SPEAKER_00: So that that looks like it did a good job.
16:52
SPEAKER_01: So then the question is, why is it that there are these clinical fidelity issues, for example?
16:57
SPEAKER_01: I mean, to some degree, why is it that there are bias of representation issues?
17:00
SPEAKER_00: I feel like the bias issues is it's like, well, we know that it was trained on biased data because that's what's out there is like you see pictures on the internet, and it's like all white men that are doctors.
17:09
SPEAKER_00: So I feel like I'm I mean, I'm I'm interested, but like less interested in that one.
17:13
SPEAKER_00: I feel like uh we just need better data sets to trade it on.
17:16
SPEAKER_00: Yes, yes, but I'm more interested in like, geez, these clinical fidelity issues, they they're really problematic.
17:22
SPEAKER_00: Like, I don't even know how I don't know why, I don't know how to fix it because I don't know why it's happening.
17:26
SPEAKER_01: Right, right, right.
17:27
SPEAKER_01: I mean, there is a training data mismatch issue even here, because uh these models, like I said before, are trained on web scraped images, right?
17:34
SPEAKER_01: There's large data sets of just web web scraped image text pairs.
17:39
SPEAKER_01: Um, these contain very few uh clinical authentic clinical images, and often stock photos, illustrations are uh these things are often mislabeled, right?
17:49
SPEAKER_00: That is absolutely true.
17:50
SPEAKER_00: Because if you like just Google it, yes, you know, something like you may or may not, you you're gonna get like some random company selling you something that may or may not be the right thing.
18:00
SPEAKER_00: You know what I mean?
18:01
SPEAKER_00: Yeah.
18:01
SPEAKER_01: Yeah, or if it's in front of a brochure, together with some caption that says these are all the procedures we do, and there's some generic image out there, right?
18:10
SPEAKER_01: That is what it's going to associate those image and text combos with, right?
18:13
SPEAKER_01: And so it's going to have a scene that looks kind of stylistically like a medical scene, but may not be anatomically or f physiologically um accurate.
18:22
SPEAKER_01: That's one issue.
18:22
SPEAKER_01: So there's the data issue.
18:24
SPEAKER_00: Okay.
18:24
SPEAKER_00: Yeah.
18:24
SPEAKER_00: And that makes sense to me because there's like sort of, you know, if you type something like Elnar gutter split into Google, it's not going to give you what you get in the medical textbook.
18:34
SPEAKER_00: It's going to give you some stuff that might be right and other stuff that's like just not.
18:39
SPEAKER_01: Yeah, exactly.
18:40
SPEAKER_01: The second issue is that if you look at the way the diffusion models are trained, which is kind of what I described before by adding noise and removing noise and such, it's training it rewards um realism, perceptual realism.
18:54
SPEAKER_01: So the that those are outputs that match kind of the visual look and feel of the of the data.
19:00
SPEAKER_01: So notice that it's about the overall image appearance, not about what the image is about, right?
19:06
SPEAKER_01: So there's no anatomical constraints, there's no ground truth verification, there's nothing that penalizes during the training process, nothing that penalizes the model for coming up with structurally impossible um things.
19:20
SPEAKER_01: Uh if you remember a few years ago, there was all this thing about images having like humans having like eight fingers and that didn't happen here.
19:27
SPEAKER_00: I was like, oh, there's not six fingers in these.
19:30
SPEAKER_01: They've solved that problem uh by removing, you know, having it constrained in different ways, but that was done in response to that happen, that issue happening.
19:38
SPEAKER_01: So it's sort of like It was like a band-aid.
19:41
SPEAKER_00: It was a band-aid, right?
19:50
SPEAKER_00: And you're like totally right.
19:51
SPEAKER_00: When I said generate an image of an orthopedic surgeon placing an on our gutter splint, if you look at the background of my images, there's like a picture of a skeleton, a picture of like in the in the room.
20:04
SPEAKER_00: Oh, yeah.
20:04
SPEAKER_00: There's like a on in the background, like hanging on the wall.
20:07
SPEAKER_00: There's a picture of a skeleton, like a skeleton, there's a picture of like the muscles of the body, there's an ace wrap, you know, you like feel like you're you're actually in, you know, yes, the orthopedic office.
20:17
SPEAKER_01: That's it.
20:18
SPEAKER_00: And so they've created this sort of situation where you feel like, oh yeah, that looks like a that that looks like a surgeon, maybe biased, but hey, that looks like a surgeon.
20:27
SPEAKER_00: They're sitting on a medical bed, the the decorations in the room look correct.
20:31
SPEAKER_00: Like this looks good.
20:33
SPEAKER_01: Yes, exactly.
20:35
SPEAKER_01: So that's another set of issues, is because of fundamentally the training doesn't constrain it in any way, shape, or form to be accurate, to be correct, anatomically correct, or whatever, right?
20:46
SPEAKER_01: Um, the other thing you mentioned was like sometimes the there's like text in there too that doesn't make sense or is not correct.
20:52
SPEAKER_01: That also happens.
20:52
SPEAKER_01: Text rendering in these things are known to be pretty bad.
20:56
SPEAKER_01: Um diffusion, because if you think about it again, diffusion models are generating images, and so texts are just like collections of pixels, right?
21:02
SPEAKER_01: They're not symbols.
21:04
SPEAKER_01: Like an LLM, a piece of text is a symbol.
21:06
SPEAKER_01: It's not a just an it's not just a pixel on the screen.
21:09
SPEAKER_01: And so there's issues with that, and people often have that's a big challenge in the research in the diffusion uh research community to figure that out to make text more realistic.
21:19
SPEAKER_01: Like again, you've seen I don't know if people have seen this, but they've there's definitely, you know, when you generate images and ask it to put some text on it, sometimes it's ridiculous.
21:26
SPEAKER_01: Oh, I've definitely seen that.
21:27
SPEAKER_00: It's like not even real letters.
21:28
SPEAKER_00: I'm trying to zoom in on mine right now to be like, did this did it happen with mine?
21:31
SPEAKER_00: But I can't I can't tell.
21:32
SPEAKER_00: Like I have text in mine, um, but it's it's like so far in the background that I can't tell what it actually says.
21:39
SPEAKER_01: Yeah, no, exactly.
21:40
SPEAKER_01: Um, and so that's one more issue.
21:43
SPEAKER_01: And so the other problem now is with a lot of neural networks, and this is true of diffusion models too.
21:49
SPEAKER_01: The idea is to take something that's really complex and compress it into a what's called a smaller, more tightened representation.
21:55
SPEAKER_01: And when you compress information, right, you at least In this case, tend to lose a lot of the details.
22:02
SPEAKER_01: So what ends up happening is you have um all the fine structures, the small things, the precise spatial relationships, the landmarks, all of this stuff gets smoothed out across the data because you're compressing all of the data across all the different places into this like one representation.
22:20
SPEAKER_00: But it doesn't like know what's important then.
22:22
SPEAKER_00: Like it's just get getting rid of stuff, and you're like, maybe it was an important thing.
22:27
SPEAKER_00: Like in the example that I did, it's like, hey, you need your fourth and fifth digits at 70 to 90 degrees of flexion, and like maybe just got rid of that because the two images I generated didn't have that.
22:38
SPEAKER_00: Yep.
22:38
SPEAKER_00: So then now you've done your splint wrong.
22:40
SPEAKER_01: Yeah, right.
22:41
SPEAKER_01: So so there's that issue too.
22:43
SPEAKER_01: All the finer details are often lost, you know, in addition to because of the nature of the compression itself.
22:48
SPEAKER_01: And and finally, I think there's one issue which is independent of the model, it has to do with how when we prompt something, we can tend to underspecify what we actually want.
22:58
SPEAKER_01: And so you can't encode every single exact anatomical detail in a real reference that a real reference would demand.
23:05
SPEAKER_01: Like if you had to write out precisely every I mean, this would be an interesting experiment to do, but like if you had to lay out precisely, not just ask for this, but like lay out precisely what you mean by everything, provide all the anatomical details that need to be correct, would it then produce the correct answer?
23:21
SPEAKER_01: And that's still unclear given what I've said so far, but oftentimes you get bad images because the prompts are underspecified, right?
23:29
SPEAKER_01: And and in your head, you're like, of course it's not underspecified.
23:32
SPEAKER_01: It's telling I'm telling it what kind of issue I'm having, and it has the world knowledge to know what you know a boxer's fracture is.
23:40
SPEAKER_01: So it should be able to pull out what's relevant and then figure out all the details after that.
23:43
SPEAKER_01: But it's not doing that, right?
23:44
SPEAKER_01: It's not figuring out figuring out things, it's pattern matching.
23:47
SPEAKER_01: So, in that sense, unless you give it all of the text you need to describe everything that you can possibly need, it's going to be really hard for it, you know, to do.
23:58
SPEAKER_01: And I think that that's the other challenge.
24:00
SPEAKER_01: That's a sort of a separate challenge, right?
24:02
SPEAKER_01: Independent of the model itself.
24:04
SPEAKER_00: Yeah, this is interesting because um one of the solutions they actually propose in this paper is hey, like when you create an image, then you should also be saving the prompt text too.
24:15
SPEAKER_00: So you can go back and like look at it and figure out if you need to tweak it in the future, when is it appropriate, when is it not?
24:23
SPEAKER_00: Um, because yeah, I can say ulnar gutter splint, but I can also go into like a ton of detail about how the splint material goes from the DIP joint of the fifth finger along the ulnar side of the fifth finger wrist and the forearm, and then it folds into a U shape, uh, you know, around the the fourth and fifth fingers, dorsally and volarly.
24:43
SPEAKER_00: Like I can go into that much detail if I if I need to, if I want to.
24:49
SPEAKER_00: It's like I could pretend that I'm like teaching someone who doesn't know what this is at all, right?
24:54
SPEAKER_00: Right.
24:55
SPEAKER_00: But I I think you're right.
24:57
SPEAKER_00: Like I have this expectation that an ulnar gutter splint is a thing.
25:01
SPEAKER_01: Yeah.
25:02
SPEAKER_00: There's text about how to do the thing uh all over the place.
25:05
SPEAKER_00: So why can't I just do the thing?
25:06
SPEAKER_01: Yes, yes, yes.
25:08
SPEAKER_01: Yeah, and and you know, that's the expectation.
25:10
SPEAKER_01: So it is interesting and it is very prompt sensitive, right?
25:12
SPEAKER_01: It depends on the on the on the on that text that you provide as an input.
25:16
SPEAKER_01: So I think there is more, a lot more research that needs to be done, both from the model itself, the data, the training, and all of these different steps, and in understanding how to use these models to get what you want.
25:27
SPEAKER_01: I mean, if you were just creating art, then it doesn't matter because you're experimenting and you're playfully trying different things.
25:32
SPEAKER_01: And sometimes you might get surprises that you don't care about.
25:34
SPEAKER_01: It's just a visual thing.
25:36
SPEAKER_01: Whereas if you want accuracy and specific things in an image, then you're gonna have to ask for it.
25:40
SPEAKER_01: And then you get into all these other issues that I talked about, which is even if you ask for it, it is not um capable right now to provide, give you all of the spatial constraints and be all precise about everything, right?
25:52
SPEAKER_01: That's already still a challenge.
25:53
SPEAKER_01: Um, to you know, you'll say, I want the patient standing in the left of the screen with your arms up 45 degrees.
25:59
SPEAKER_01: You can provide all that information, it's not gonna do that.
26:01
SPEAKER_01: Right.
26:01
SPEAKER_00: And I might have to say now, oh, I want the patient to be female and African American, and I want the doctor to be female.
26:08
SPEAKER_00: And like, do I have to go through all of that to try to make it not be biased too?
26:13
SPEAKER_00: It just feels like a lot of a lot of extra work for something that may or may not actually pan out.
26:20
SPEAKER_01: Yeah, exactly.
26:21
SPEAKER_01: Exactly.
26:22
SPEAKER_01: So I I think this world of image generation is uh has so much potential, right?
26:27
SPEAKER_01: But we obviously live in a world where it has downsides like this and deep fakes and other things that people have have expressed a lot of you know issues with.
26:35
SPEAKER_01: And I I really don't think they're ready for medical education, clearly.
26:39
SPEAKER_01: Maybe for a brochure, front page of a brochure or something, you can have the sort of feel that you're in a doctor's room or something.
26:45
SPEAKER_01: But beyond that, and even then you have to be worried about bias representation.
26:48
SPEAKER_01: Uh, but beyond that, for fidelity, we're not anywhere close.
26:52
SPEAKER_00: Yeah, and the fact that you can just ask, you know, Gemini or whatever to create these images for you, a medical student could be doing that.
26:59
SPEAKER_00: But the truth of the matter is they're not necessarily getting correct information.
27:03
SPEAKER_00: And so without expert review, these things shouldn't shouldn't be live.
27:08
SPEAKER_00: Like they shouldn't be going out there.
27:09
SPEAKER_00: People should not be learning f from them.
27:12
SPEAKER_00: Right, exactly.
27:13
SPEAKER_00: And you know, I think that has to be part of medical education now is you can't just like type something into an image generator and expect it to be correct.
27:21
SPEAKER_01: Yes.
27:22
SPEAKER_00: It's it's not going to be.
27:24
SPEAKER_00: At least not in my case, and in not in a lot of the cases that were reviewed in this paper.
27:30
SPEAKER_01: Right, right, exactly.
27:32
SPEAKER_00: So, I mean, I guess it makes sense why the limitations exist based on your explanation of these diffusion models and the real world application is like, hey, this is not ready for prime time yet, and we we get why that is true on multiple levels.
27:48
SPEAKER_01: Yeah, exactly.
27:50
SPEAKER_00: All right, well, thank you for joining us.
27:51
SPEAKER_00: We'll see you next time on Code and Cure.
27:53
SPEAKER_00: See you later.