Latest / Code & Cure / #21 - The Rural Reality Check for AI

Transcript

#21 - The Rural Reality Check for AI · Code & Cure

• 0:00
  SPEAKER_00: The fate of AI in rural medicine will be shaped not in sterile server farms, but at kitchen tables, over backyard fences, and on front porches.
• 0:11
  SPEAKER_00: The very places where social support has always done its quiet, steadfast work.
• 0:26
  SPEAKER_01: Hello and welcome to Code and Cure, where we discuss decoding health in the age of AI.
• 0:31
  SPEAKER_01: My name is Vassant Sarathy.
• 0:33
  SPEAKER_01: I'm an AI researcher and cognitive scientist, and I'm here with Laura Hagopian.
• 0:37
  SPEAKER_00: I'm an emergency medicine physician and I work in digital health.
• 0:41
  SPEAKER_00: And I just read a quote from the end of this amazing article about bringing AI-enabled care to rural America.
• 0:48
  SPEAKER_01: Yeah, no, that's uh I I also read their article and I thought it was very interesting.
• 0:53
  SPEAKER_01: And, you know, I think in this podcast, we've been over the past several weeks, we've been talking about all of these really cool things that AI can do uh with helping physicians and hospitals, you know, ambiently listening, listening to the physician notes and making, you know, creating discharge summaries that are understandable to patients.
• 1:11
  SPEAKER_01: Documentation, yeah.
• 1:12
  SPEAKER_01: All kinds of documentation stuff and conversational agents that help with um dealing with patients, but also dealing with doctors directly.
• 1:20
  SPEAKER_01: Um and then all of these other AI tools behind the scenes that help with you know improving drug repurposing and other you know pharmaceutical type applications.
• 1:28
  SPEAKER_00: Yeah, predictive algorithms, uh triage tools, all of that kind of stuff.
• 1:33
  SPEAKER_01: Yeah, and at the center of all of that is um, you know the this AI system that we're talking about that seems so cool.
• 1:41
  SPEAKER_01: So to me, you know, it comes naturally to say, of course, you know, you have a cool, and we sorry, one more thing we also talked about was the um how AI can be used to help manage all this data, right, that we're collecting and all these wearables and all that stuff.
• 1:56
  SPEAKER_01: But you know, on the on the flip side, I think what we missed and what this article kind of focuses on is the idea that humans collecting all this data and reviewing all this data are living all over the country, right?
• 2:09
  SPEAKER_01: I have my Apple Watch and it's collecting my data and it's giving me re my my information.
• 2:14
  SPEAKER_01: And maybe, you know, in an ideal world, it's sending that information maybe to the doctor and it's keeping and the and my caregivers are keeping track of it.
• 2:21
  SPEAKER_01: And all this is wonderful, right?
• 2:23
  SPEAKER_01: But I, you know, I'm not the only person who's here and there's people living all over the country, right?
• 2:30
  SPEAKER_01: And that's what I love about this article because it talks about rural America.
• 2:34
  SPEAKER_00: Yeah, exactly.
• 2:35
  SPEAKER_00: We're not all living in suburban or urban environments, people are living in rural environments, and we talk a lot about data sets and what data these LLMs are trained on.
• 2:47
  SPEAKER_00: And it's like, well, are they trained differently on rural data?
• 2:51
  SPEAKER_01: Yeah, and at first glance, it seems like, okay, what does that matter, right?
• 2:54
  SPEAKER_01: I mean, everyone has people have phones everywhere, people have smart watches everywhere across the country.
• 3:00
  SPEAKER_01: They're being sold, there's Apple stores everywhere.
• 3:02
  SPEAKER_01: You know, like people can drive to things that are not that far, but what is the actual difference?
• 3:06
  SPEAKER_01: And as it turns out, there is quite a bit of difference, right?
• 3:09
  SPEAKER_01: I mean, it's um and speaking to the specifically to the training data point I think you make, which I think is particularly interesting.
• 3:17
  SPEAKER_01: Again, we've talked about training data as being sort of English versus low resource languages in other parts of the world.
• 3:22
  SPEAKER_01: And we always think of rural as being some rural country that, you know, has no internet and you know has that they speak a very weird language or an unknown language.
• 3:33
  SPEAKER_01: And as a result, there's not enough training data that um that the LLMs learn on to understand and speak to those people, right?
• 3:40
  SPEAKER_01: That's what we think of.
• 3:41
  SPEAKER_01: We don't think of English speaking um people here in our country.
• 3:45
  SPEAKER_01: And so Yeah, but they're here.
• 3:46
  SPEAKER_00: I mean, rural rural America is here.
• 3:48
  SPEAKER_01: Yes, and and it and they speak differently too.
• 3:51
  SPEAKER_01: People speak in uh people in urban areas speak differently from people in rural areas.
• 3:55
  SPEAKER_01: Um there's different cultural norms that are in play uh that cause them to say things uh you know directly versus indirectly, maybe more politely in some places.
• 4:04
  SPEAKER_00: Or I think in rural areas, certainly, like there's a lot of terms where you know they'll sort of just like deal with the pain.
• 4:10
  SPEAKER_00: They don't want to be a bother or they'll make do.
• 4:12
  SPEAKER_00: And so they won't say I'm in 10 out of 10 pain.
• 4:15
  SPEAKER_00: Oh, it hurts, but I'll make do.
• 4:17
  SPEAKER_00: Right.
• 4:18
  SPEAKER_01: And that's you know, not a language constraint, that's still English, but that's a it's clearly a resource constraint for the language models because they might not have enough training data.
• 4:28
  SPEAKER_01: Because again, it's not like people are constantly on the machines typing in Reddit where a lot of the straight data comes from, um, and using all the social media that's out there potentially in the same way in all parts of the country.
• 4:41
  SPEAKER_00: Or if it's trained on like all the data throughout the country and doesn't differentiate between rural, suburban, and urban, that could potentially be a problem too.
• 4:50
  SPEAKER_01: Yes, that's true.
• 4:51
  SPEAKER_01: It's just like it's all awash, right?
• 4:52
  SPEAKER_01: It doesn't really think about that aspect.
• 4:54
  SPEAKER_01: And so those things end up being a big deal, right?
• 4:58
  SPEAKER_01: That's kind of what I learned from reading this article.
• 5:00
  SPEAKER_00: Yeah, and the other thing that you just kind of skimmed over when we were talking about this is like people may not actually have access to broadband internet.
• 5:12
  SPEAKER_00: So like you're using your Apple Watch, and maybe you get an alert for, I don't know, like your heartbeat is irregular or the noise level is very high or whatever it is, right?
• 5:26
  SPEAKER_00: That alert has to go through cell service.
• 5:31
  SPEAKER_00: At least some of these do.
• 5:32
  SPEAKER_00: Yes.
• 5:33
  SPEAKER_00: Some of these alerts have to go through cell service.
• 5:35
  SPEAKER_00: Um, and so if the cell service is down or you don't have broadband or whatnot, then the alerts aren't delivered, right?
• 5:45
  SPEAKER_01: Yeah, exactly.
• 5:46
  SPEAKER_00: And so this article, they talk about how older adults in rural zip codes are 1.6 times more likely to lack in-home internet service than their urban peers.
• 5:56
  SPEAKER_00: I mean, I think we take it for granted that everyone can just, you know, log on, listen to our podcast anytime, go on their computer, type on Reddit, but like people don't, not everyone has access to that stuff.
• 6:07
  SPEAKER_00: Right.
• 6:08
  SPEAKER_00: And so if you're trying to do remote monitoring, for example, and you're relying on that in the moment service to be available, like maybe it's not available.
• 6:19
  SPEAKER_00: Yeah.
• 6:19
  SPEAKER_00: And then how do you trust a system where, you know, it's designed to alert you and the alert doesn't come through?
• 6:26
  SPEAKER_01: Yeah, then you don't trust it.
• 6:27
  SPEAKER_01: And sometimes the alerts come through, sometimes they don't.
• 6:30
  SPEAKER_01: And then another level there is the alerts might come through, but because it's been trained on bad data or or incorrect data, that the it might be false positives, right?
• 6:39
  SPEAKER_01: And that also is sort of, you know, it's crying wolf, and they're they're also missing, and then people start to ignore that and start to, you know, lose trust and lose confidence in these systems again.
• 6:49
  SPEAKER_01: And, you know, that defeats the whole purpose because we still need, I mean, maybe we need to take a step back and think about do we need all these AI things all the time everywhere, right?
• 6:58
  SPEAKER_00: But some people might, I mean, some people might benefit from those things, right?
• 7:02
  SPEAKER_00: If they're used correctly.
• 7:04
  SPEAKER_00: Like you can imagine um, you know, someone who has a heart condition maybe needing to monitor their weight more frequently and uh they step on the scale and they've gained weight.
• 7:16
  SPEAKER_00: And then you're like, well, what do you do about that?
• 7:18
  SPEAKER_00: Now they're like maybe they're two hours from their cardiologist.
• 7:21
  SPEAKER_00: Yes.
• 7:21
  SPEAKER_00: This is a great place for digital health to come in and say, hey, like there's, you know, there's some alert set up that's noticed a weight gain.
• 7:30
  SPEAKER_00: What does it tell the member, that patient, though?
• 7:32
  SPEAKER_00: Does it tell the patient like, hey, you've gained weight?
• 7:36
  SPEAKER_00: Does it tell the patient, hey, you should call your doctor?
• 7:40
  SPEAKER_00: Does it alert, you know, a nurse to call the patient and say, hey, how you doing?
• 7:44
  SPEAKER_00: How's your breathing?
• 7:45
  SPEAKER_00: We've noticed you gained a couple of pounds.
• 7:47
  SPEAKER_00: We're worried that might be fluid.
• 7:49
  SPEAKER_00: Like what, what, how is how is the human involved there?
• 7:53
  SPEAKER_00: Because that that's one of the things that's very clear to me from reading this article is the importance of having humans, not just in the loop, right?
• 8:03
  SPEAKER_00: But actually like involved in this system.
• 8:06
  SPEAKER_00: And it doesn't necessarily have to be the provider, that the doctor, like it could be the neighbor, or it could be a provider like a nurse or a social worker.
• 8:18
  SPEAKER_00: Um, it's like who's gonna be there in person to help deliver on whatever data is important, who's gonna help figure that out.
• 8:28
  SPEAKER_01: Yeah.
• 8:28
  SPEAKER_01: And you know, I I also found this interesting because the places that need all this technology the most are potentially these rural areas, because they lack, you know, enough nurses.
• 8:39
  SPEAKER_01: They lack um, you know, there's a lot of scattered um, you know, healthcare coverage.
• 8:44
  SPEAKER_01: And like you just talked about, things could be two, three hours away, right?
• 8:47
  SPEAKER_01: That's exactly where technology can be helpful, but that's what the technology is least prepared for right now, which is ironic in some ways, because you want the technology to serve a human purpose, and right now it's serving another purpose, but that's the purpose it needs to serve.
• 9:04
  SPEAKER_01: And so we need better training data.
• 9:06
  SPEAKER_01: We need the system to understand the populations where it's supposed to and can help the most.
• 9:11
  SPEAKER_00: It's really interesting because there aren't enough clinicians in rural areas, it's worse than suburban and urban areas.
• 9:19
  SPEAKER_00: There's not enough there either.
• 9:20
  SPEAKER_00: Yeah, but if you have some sort of monitoring tools, maybe your rural nurses can actually go and see more patients, right?
• 9:31
  SPEAKER_00: Yes.
• 9:31
  SPEAKER_00: And so it could like this technology could help enable that.
• 9:35
  SPEAKER_01: Yes.
• 9:35
  SPEAKER_01: I mean, it's a support tool in that sense, right?
• 9:37
  SPEAKER_01: It's a caregiver support tool.
• 9:38
  SPEAKER_01: And I think that that's definitely one useful feature, a characteristic of it.
• 9:42
  SPEAKER_01: Um, but again, we have to fix all the issues, right?
• 9:45
  SPEAKER_01: The broadband, and that's a core infrastructure issue, providing more access to the internet or finding clever ways to work with, you know, scarce uh resources.
• 9:55
  SPEAKER_01: But we need people working on those problems, is kind of what I'm getting at.
• 9:59
  SPEAKER_00: Oh, for sure.
• 10:00
  SPEAKER_00: We absolutely need people working on those problems because that's like the core of how you actually intervene here, right?
• 10:08
  SPEAKER_00: If you're saying, oh, hey, like we're gonna do some predictive monitoring, let's let's figure out if someone is um, you know, doing worse and we need to intervene.
• 10:18
  SPEAKER_00: If you don't have the broadband connected, that alarm may never trigger, right?
• 10:25
  SPEAKER_01: Yeah.
• 10:25
  SPEAKER_00: Or if the data, like we've talked about, is maybe not trained.
• 10:29
  SPEAKER_00: It's maybe the system isn't trained on the right data, maybe it's alarming all the time.
• 10:33
  SPEAKER_00: And then what do you do with that?
• 10:34
  SPEAKER_00: You just like ignore it.
• 10:36
  SPEAKER_00: I ignore it if I get too many alarms that say the same thing over and over again.
• 10:41
  SPEAKER_00: And so this is why it's like, well, we need the training on the raw data, we need the broadband, and then we need the human to like go and do the intervention.
• 10:49
  SPEAKER_01: Right.
• 10:49
  SPEAKER_01: And that human connection is still necessary, especially given that we're not quite there with a lot of the AI language interaction and cultural norm-specific capabilities.
• 10:57
  SPEAKER_01: But you know, I I really liked also this article talked about, you know, the the sort of the different use cases of AI and the particular challenges that you would face in uh in a rural environment.
• 11:08
  SPEAKER_01: And we can just sort of go down that list because I think that was really great.
• 11:11
  SPEAKER_00: Yeah, I think I just I think I just covered the first one, the predictive monitoring one.
• 11:14
  SPEAKER_01: Yeah, you did.
• 11:15
  SPEAKER_01: And really that's about you know early detection.
• 11:17
  SPEAKER_01: Uh, but you know, it it it you can pair alerts with caregivers and do those kinds of things, but of course it's it's risky if the systems don't work well and then you get you know alarm fatigue.
• 11:29
  SPEAKER_01: Um, and as a result, you might get delayed responses.
• 11:32
  SPEAKER_01: And so that there's a fine balance there again, goes back to using the right training data and all of that to get the right predictions.
• 11:39
  SPEAKER_01: Uh then the second use case was triage tools.
• 11:42
  SPEAKER_01: Um, and and here again, you're um you have to train on rural uh communications.
• 11:49
  SPEAKER_00: Um Yeah, I mean the idea here is you're you might get like a ton of patient messages in and you're like, well, which which ones are the most important?
• 11:57
  SPEAKER_00: Maybe I have like 50 messages and I have time to go through five of them.
• 12:01
  SPEAKER_00: Right.
• 12:01
  SPEAKER_00: Which are the five urgent ones?
• 12:03
  SPEAKER_00: And can we use an AI system to help figure that out?
• 12:07
  SPEAKER_00: Of course, this is where the language and cultural concerns come in.
• 12:11
  SPEAKER_00: Oh, I, you know, um, my stomach's really hurting, but I don't want to be a I don't want to be a bother to you.
• 12:17
  SPEAKER_00: Um, you know, I just figured I'd check in.
• 12:20
  SPEAKER_00: Does that set off an alarm bell in an urban area versus a rural area?
• 12:25
  SPEAKER_00: The I I don't want to be a bother thing, you know, does that actually mean someone is or isn't in pain?
• 12:31
  SPEAKER_01: Yeah.
• 12:31
  SPEAKER_00: And so this is a kind of data that, you know, you could miss.
• 12:36
  SPEAKER_00: You could miss when something is urgent because the culture is different in a rural location.
• 12:42
  SPEAKER_00: And if it's not trained on those patterns, um, and you don't understand like how stoic some of these people can be, then you're you're missing out.
• 12:53
  SPEAKER_01: Yeah.
• 12:54
  SPEAKER_01: And and maybe in that case, you can't, you shouldn't be using AI as a triage tool.
• 12:58
  SPEAKER_01: That's not necessarily you or you train it on the rural data.
• 13:02
  SPEAKER_01: If you have it, right, exactly.
• 13:04
  SPEAKER_01: And and understanding when an AI tool is ready for deployment and when it isn't, I think is really key.
• 13:09
  SPEAKER_01: Because we see a lot of hype right now about every AI tool that comes out, how it's going to change the world and it's gonna be great.
• 13:14
  SPEAKER_01: But these types of issues matter, and deployment in a real environment totally matters if you're in fact gonna build trust and have it have people use it.
• 13:23
  SPEAKER_00: It's like you have to have people in that specific community sort of involved in the development, which is hard to scale.
• 13:30
  SPEAKER_00: But also, like you're you're right.
• 13:32
  SPEAKER_00: You you can't have these tools where they're not trained on the right data.
• 13:37
  SPEAKER_01: That's right.
• 13:38
  SPEAKER_01: Um, the AI can also be used um as conversational agents, like chat agents.
• 13:43
  SPEAKER_01: Um, and the idea would be to help reduce uh loneliness and generally support mental health.
• 13:48
  SPEAKER_01: And you know, that's already kind of happening right now with Chat GPT and so on.
• 13:52
  SPEAKER_01: That are, you know, you can talk to Chat GPT and people share a lot of concerns and such.
• 13:56
  SPEAKER_01: But again, there could be again, it's trained on the same you know, data.
• 14:00
  SPEAKER_01: And so you not on local norms, not on local norms, and so you have to be very careful with this.
• 14:06
  SPEAKER_01: And you know, uh, it's not just local norms, but also age-specific norms that could come into play as well, which are also influenced by local cultures.
• 14:14
  SPEAKER_00: This is an interesting one to me, because you know, in the article they talk a lot about how there's uh you know a lot of social support and how we people in rural environments they rely on their their neighbors.
• 14:26
  SPEAKER_00: Um, and so this is an interesting one because it's like, well, maybe they don't always have that.
• 14:30
  SPEAKER_00: And it would be nice for them to have someone to talk to as long as they don't get too too attached to this chatbot.
• 14:38
  SPEAKER_01: Right, exactly.
• 14:39
  SPEAKER_01: Um, and finally they had uh, and this we talked about a little bit already, which is the caregiver support tools.
• 14:45
  SPEAKER_01: So the AI use as a uh to assist families with uh decision making and and and also with reminders, right?
• 14:51
  SPEAKER_01: And so it's sort of a helper for the caregiver.
• 14:54
  SPEAKER_00: Yeah, like did you, you know, did you remind mom to take her meds or like maybe you get mom's weight and see if you know her heart failure medication might need to be tweaked so you can call her cardiologist.
• 15:06
  SPEAKER_00: So I think that again, it has to be sort of tailored to this subpopulation and the norms inside of this subpopulation.
• 15:15
  SPEAKER_01: Yeah, and you know, and again, you have an issue where you begin to over-rely on these automation tools, but then your trust and everything else is kind of conditioned on that.
• 15:24
  SPEAKER_01: So it if you start seeing things that are just, you know, spurious uh alarms and reminders, then you're gonna start ignoring all of them, right?
• 15:32
  SPEAKER_01: And so um and some of them are gonna be more important than others.
• 15:36
  SPEAKER_01: And so again, it goes back to all the same issues we just talked about, which is a set of problems that are unique to rural um uh use cases.
• 15:43
  SPEAKER_01: However, uh, that's where technology can really help out.
• 15:46
  SPEAKER_01: And I'm really hoping that people um, you know, develop technologies and and do things in in those communities because they're the ones who need it the most.
• 15:55
  SPEAKER_00: Yeah.
• 15:56
  SPEAKER_00: And another thing they brought up in this article was like, hey, if you're gonna use these tools, then the insights you give back to the patient has to be something that's like actionable.
• 16:07
  SPEAKER_00: It's not so helpful to say, oh, your heart rate is irregular.
• 16:10
  SPEAKER_00: It's like, well, what do I do with that?
• 16:12
  SPEAKER_00: Like, do I call my provider?
• 16:14
  SPEAKER_00: Do I like try to make an appointment?
• 16:16
  SPEAKER_00: Do I drive two hours away?
• 16:17
  SPEAKER_00: Do I need to call an ambulance?
• 16:18
  SPEAKER_00: Like, what's what what do I do with that information?
• 16:21
  SPEAKER_00: Yeah.
• 16:22
  SPEAKER_00: And um and like who does that information go to?
• 16:26
  SPEAKER_00: Does it automatically ping to someone else?
• 16:28
  SPEAKER_00: Is it just staying with me and I have to decide what to do?
• 16:32
  SPEAKER_00: So it's it's very, very interesting.
• 16:36
  SPEAKER_00: One of the things that I think is key about this article is we talk about human being in the loop all the time for all of these AI tools.
• 16:45
  SPEAKER_01: As a way to check, make sure AI is doing the right thing.
• 16:47
  SPEAKER_00: Yeah, but I think there's more to it than that here, where it's like the the system needs to be really reinforced by humans.
• 16:58
  SPEAKER_00: So they're not just like monitoring the AI side of things, right?
• 17:02
  SPEAKER_00: This is like a tool that humans are using, whether they're the patients, the family members, the care providers.
• 17:11
  SPEAKER_00: This is something to sort of like supplement and amplify what is already being done.
• 17:17
  SPEAKER_00: So now you have your nurse who maybe they could see 10 patients in a day because they're going around and doing home visits.
• 17:24
  SPEAKER_00: Maybe now they can check in with a few more over the phone because they've gotten alerts about weight gain or irregular heart rate or number of steps taken or whatever it is.
• 17:34
  SPEAKER_00: I think that's one of the keys here is like, hey, this is trust is really important for these communities, and trust is built with humans.
• 17:45
  SPEAKER_00: And maybe trust can be built with AI too, right?
• 17:48
  SPEAKER_00: Yeah.
• 17:48
  SPEAKER_00: Sometimes we overtrust these systems and then all of a sudden they mess up and then we don't trust them at all anymore.
• 17:54
  SPEAKER_00: Right.
• 17:55
  SPEAKER_00: And this is where the human element becomes so important, and our reliance on relationships becomes so important.
• 18:03
  SPEAKER_01: Yeah.
• 18:03
  SPEAKER_01: And you know, just sort of concluding on that point, which is the the human in the loop idea.
• 18:08
  SPEAKER_01: The human in the loop idea is often introduced in context with an AI automating automating a certain workflow, completely automating a certain thing, right?
• 18:17
  SPEAKER_01: Or completely automating a certain job function.
• 18:19
  SPEAKER_01: And I think what we're saying here is that that is not the point here.
• 18:22
  SPEAKER_01: The point here is not to automate anybody, the point here is to support people who really need um additional sort of scaling help.
• 18:32
  SPEAKER_01: Um, you know, there's not enough nurses.
• 18:34
  SPEAKER_01: We need the nurses that exist there to really be able to see more patients and be more effective with their own uh um communities.
• 18:41
  SPEAKER_01: And so that's a different use case.
• 18:44
  SPEAKER_01: And it's not about AI automating anything, it's about AI being, like you said, used as a tool for that person, uh, for that nurse or whoever to really help more people.
• 18:54
  SPEAKER_00: And and the relationship can't go away, right?
• 18:56
  SPEAKER_00: The relationship with the other human, that's the most important thing.
• 18:58
  SPEAKER_01: Yeah.
• 18:59
  SPEAKER_00: And that's what's gonna get uh people the care that they need.
• 19:02
  SPEAKER_01: It's just making their lives easier and more efficient.
• 19:05
  SPEAKER_01: Um, and seeing when, you know, seeing the the the right patients at the right time, you know, when they need it, right?
• 19:10
  SPEAKER_01: And so that's really where the AI tools can really come in and help out.
• 19:14
  SPEAKER_00: Yeah.
• 19:15
  SPEAKER_00: Well, um, I guess we need to solve this problem of unreliable broadband.
• 19:20
  SPEAKER_00: Yes.
• 19:21
  SPEAKER_00: And that's something that's you know it's a big issue.
• 19:26
  SPEAKER_00: Yeah.
• 19:26
  SPEAKER_00: If we want rural health to go digital, we need broadband, yes, and we need training data for this community.
• 19:36
  SPEAKER_00: And I think that's what we can walk away from to with today.
• 19:40
  SPEAKER_01: Yeah.
• 19:41
  SPEAKER_00: All right.
• 19:42
  SPEAKER_00: Well, thanks for joining us on Code and Cure.
• 19:44
  SPEAKER_00: We'll see you next time.
• 19:46
  unknown: Bye bye.