Latest / Code & Cure / #4 - From Florence Nightingale to AI: Revolutionizing Outbreak Surveillance

Transcript

#4 - From Florence Nightingale to AI: Revolutionizing Outbreak Surveillance · Code & Cure

• 0:01
  Speaker 1: What do Florence Nightingale and 21st century AI have in common?
• 0:05
  Speaker 1: More than you might think.
• 0:17
  Speaker 2: Welcome.
• 0:17
  Speaker 2: My name is Vasant Sarathy and I'm here with Laura.
• 0:20
  Speaker 1: Hagopian.
• 0:21
  Speaker 2: Hi Laura.
• 0:23
  Speaker 1: Yeah, I'm excited about today's topic.
• 0:24
  Speaker 1: We're here to talk about early detection of disease outbreaks with AI.
• 0:30
  Speaker 1: Oh, that's cool, yeah, but first I think we should talk a little bit about Florence Nightingale.
• 0:35
  Speaker 2: I guess that was the hook.
• 0:38
  Speaker 1: When I say her name, what do you think about?
• 0:40
  Speaker 2: I think that she was the first nurse and she did some amazing things during the Crimean War, I think and yeah, I just I think of her as a nurse.
• 0:51
  Speaker 1: Yeah, I mean she was an amazing nurse, but did you also know that she was an amazing statistician?
• 0:56
  Speaker 2: I mean, no, but I kind of do because we're doing this podcast.
• 1:01
  Speaker 1: I know, just like, come on roll with me here, okay, no, I didn't know, thank you, thank you, okay.
• 1:09
  Speaker 1: Um, well, one of the amazing things that she did was she started to notice trends in the deaths that she saw and she started to categorize them and she started to map them out.
• 1:21
  Speaker 1: So, basically, what she did was, during the war, she noticed, okay, there are a lot more people dying of kind of potentially preventable disease than of battle wounds.
• 1:34
  Speaker 1: And so, month by month, what she did was she marched out, okay, how many deaths are there due to battle wounds?
• 1:40
  Speaker 1: How many deaths are there due to disease, and how many deaths are there due to disease and how many deaths are from other reasons, and she categorized those in different colors.
• 1:48
  Speaker 1: You know red, black and blue.
• 1:50
  Speaker 1: And she found that the blue was the largest.
• 1:53
  Speaker 1: Guess what the blue was Not battle wounds.
• 1:56
  Speaker 1: It was not battle wounds, it was not, it was disease.
• 2:01
  Speaker 1: Wow, and like by a large order of magnitude.
• 2:06
  Speaker 2: Which must have blown people's minds back then, because obviously the natural assumption is you got hurt and injured and you died from that right, and so that's what everything must have pointed towards, and so this must have been completely counterintuitive.
• 2:19
  Speaker 1: Right, and so that was one of the first ways that she was able to use statistics to advocate for.
• 2:27
  Speaker 1: Oh gosh, we need a sanitary commission to come in and intervene.
• 2:31
  Speaker 1: Right, we need better sanitation, better hand washing, better cleaning, better nutrition, all of this stuff.
• 2:39
  Speaker 1: And when that happened, guess what happened to the blue wedges?
• 2:44
  Speaker 2: Oh, became smaller.
• 2:46
  Speaker 1: Yeah, exactly Like almost down to almost zero, right.
• 2:51
  Speaker 1: And so once she tracked this and brought in this Royal Sanitation Commission to fix the problem, after synthesizing her data, she was able to basically move forward, create an action plan and create goals, and then you could see, with the way that she tracked it, gosh, this worked.
• 3:14
  Speaker 2: That's incredible, Right.
• 3:15
  Speaker 2: I mean just just, it's so many, so many levels.
• 3:18
  Speaker 2: That's incredible, right.
• 3:19
  Speaker 2: I mean she figured out that nutrition, ventilation, shelter, those things matter more.
• 3:23
  Speaker 2: Sanitation and sanitation matter and matter more, and she was able to then translate that knowledge that she acquired from gathering all the data to something that was an action plan that she could act on quickly, and it actually resulted in incredible results.
• 3:38
  Speaker 2: I think that's amazing.
• 3:39
  Speaker 1: Yeah, so what does this have to do with AI?
• 3:42
  Speaker 1: Well, I think this concept of taking lots of data and synthesizing it in order to create an action plan is something that AI can help with Not as much the action plan necessarily.
• 3:58
  Speaker 1: That's where you might need public health officials to come and step in.
• 4:00
  Speaker 1: But I do think that she manually noticed, she noticed this and then she manually aggregated the data and plotted it out and charted it, but I do think AI could step in here.
• 4:12
  Speaker 1: So I wanted to talk a little bit about and today we're going to talk about disease outbreaks like how can AI help us kind of figure out early warning signs that a disease outbreak might be coming?
• 4:24
  Speaker 1: And that's traditionally, you know, not been done with AI.
• 4:33
  Speaker 1: As a physician, I remember reporting on certain diseases, or my hospital would, or my laboratory, would you know, notifying the public health authorities.
• 4:40
  Speaker 1: Hey, like I saw a case of gonorrhea, or here are a couple of cases of salmonella, and that's how you end up ultimately reading about it on the news is, you know, physicians or providers or public health officials have noticed a trend.
• 4:54
  Speaker 1: Lyme disease is another one where you know that's something that I would get a notice of.
• 5:00
  Speaker 1: The lab is going to report on this automatically for you.
• 5:02
  Speaker 1: Flu is something we report on too, although it's usually like a number of cases and not individual people, and so traditionally we've relied on sources like me who you know.
• 5:18
  Speaker 1: It might be a little bit slow, right.
• 5:21
  Speaker 2: Yeah, I mean, I think there's that.
• 5:22
  Speaker 2: But I think I want to take a quick step back right now and talk about these sorts of early warning systems.
• 5:28
  Speaker 2: And you mentioned disease outbreaks and that's great, but there's also this notion of potentially new diseases that come about, and again you want the early warning signs to those are going to be even harder, I would imagine, because then you don't know what you're looking for, but you just start to see this sort of common set of symptoms or something like that.
• 5:46
  Speaker 2: And so those are also, I would argue, part of this bucket of a group of early warning systems that I believe AI can in fact help.
• 5:56
  Speaker 2: And, like you mentioned, those approaches, the more formal methods of reporting and such, are potentially slow in their recognition of these outbreaks, but they also are labor intensive, oh for sure, and they're somewhat passive in some sense, and the labor intensive part requires people to some people to actively do something and report these things.
• 6:21
  Speaker 2: But also, you know, keeping track of all of these different pieces is really hard, and and they're potentially not as good because they're not using as many data sources that could possibly give you signs and hints.
• 6:31
  Speaker 2: And I mean, all of this is speaking to the same thing, which is that we now live in a world where there is all forms of data and data and new forms of data are constantly being developed, right, I mean?
• 6:43
  Speaker 2: So you're collecting so much information about people's conversations, people's behavior, people's movements, physiological statuses of people and all of these things globally, which, then, is very powerful.
• 6:57
  Speaker 2: But, of course, now, if it was labor-intensive before, it's really labor-intensive now to actually piece through all of that, which is where, you know, computational methods and AI can really jump in and be helpful.
• 7:09
  Speaker 1: Yeah, it's like all of this data is at your fingertips, but it's almost too much data for it is too much data for like a human to sort through and you think about you're doing this kind of at scale at one level, but also you really want to understand what's happening in your local, in your location.
• 7:28
  Speaker 2: Yeah, we talked about this before and I really thought that was an interesting insight, which is you both want sort of the global level things that are happening, but practically speaking for a healthcare provider or for a human being living in a particular location, they care about what's happening around them, and so you need both the local and sort of the really hyper focused local and the overall global at the same time, and I thought that was very interesting.
• 7:50
  Speaker 2: We talked about this a bit earlier and that's a hard problem too.
• 7:53
  Speaker 2: But again, potentially the quote unquote data has all the answers right.
• 7:58
  Speaker 1: Right, if you can, you know, sift through it all, synthesize it all and translate it all.
• 8:04
  Speaker 1: Right, If you're talking at the global level?
• 8:06
  Speaker 1: Now you're talking about things in so many different languages and, again, that's hard for a human to parse through.
• 8:13
  Speaker 2: Yeah, so can we kind of talk through potentially what this world of data looks like.
• 8:20
  Speaker 2: So it's one thing to say we have formal, the old methods where you see somebody coming in, you have some set of symptoms or the CDC tells you about something that's happening.
• 8:29
  Speaker 1: Right, I might get an email from the Department of Public Health.
• 8:32
  Speaker 1: Hey, we've seen a number of cases of this.
• 8:34
  Speaker 1: Please report if you see any.
• 8:36
  Speaker 1: We do track things like wastewater too.
• 8:39
  Speaker 1: I mean, people probably remember COVID wastewater numbers coming out and seeing peaks there, and that would sometimes come ahead of the actual cases that were reported too.
• 8:49
  Speaker 1: So there's definitely a bunch of different ways that we traditionally go about doing it, but it doesn't involve always things like oh, let's synthesize news articles, let's look at what people are saying on social media.
• 9:04
  Speaker 1: Oh, let's look at what people are searching for.
• 9:07
  Speaker 1: What are their search queries about?
• 9:08
  Speaker 1: You know what does their wearable data say?
• 9:12
  Speaker 1: Or what's happening from a meteorological standpoint.
• 9:15
  Speaker 1: You know what's the temperature, what's the rainfall, what's the humidity and how could that impact early disease spread.
• 9:22
  Speaker 1: So there are so many different sources of data out there that go beyond what we think of traditionally as what we might use for surveillance.
• 9:32
  Speaker 2: Yeah, no, that's incredible, and what we've seen from our reading of this is that there are systems out there that have been out there, I would say, for quite a bit now, like some of them have been out there for 15, 20 years now that have been tracking using data and using whatever the state of the art computational methods are to do this right.
• 9:51
  Speaker 2: Should we talk about some of those?
• 9:53
  Speaker 1: Yeah, I mean there's a bunch of them, but HealthMap, ProMedMail, are just a couple of examples.
• 10:01
  Speaker 1: There are a bunch of these out there Blue Dot, et cetera and so I think I started playing around with Health Map just as an example to kind of figure out how does this work?
• 10:13
  Speaker 1: How user-friendly is this?
• 10:14
  Speaker 1: I mean, you can go on and look at it too.
• 10:16
  Speaker 1: It's not behind any sort of wall.
• 10:17
  Speaker 1: And I was curious to see, like, what does it say about?
• 10:21
  Speaker 1: We're in Massachusetts right now?
• 10:22
  Speaker 1: What does it say about where we live?
• 10:24
  Speaker 1: And there was a couple of red dots on Massachusetts.
• 10:28
  Speaker 1: I clicked on all of them and it said oh, you know, watch out, there's West Nile.
• 10:32
  Speaker 1: There's all these news reports of West Nile being detected in mosquitoes in a variety of towns in Massachusetts and being found in potentially some animals as well, and so well, I'm going to make sure that I put on bug spray when I go outside.
• 10:45
  Speaker 1: But it was interesting to see, like what is it synthesizing, what is it looking at and how does it do that at the local level?
• 10:55
  Speaker 1: But if you zoom out, there were dots all over the map all over the world that show a similar thing in a different location.
• 11:02
  Speaker 1: So I thought that was that was really cool to see.
• 11:04
  Speaker 2: Was the red dot an indication of the number of cases or what was it representing?
• 11:09
  Speaker 2: What was the redness?
• 11:10
  Speaker 2: Do you remember that?
• 11:10
  Speaker 2: Or do you know if Health Map also allows you to give you more details about?
• 11:16
  Speaker 2: I know that it aggregates a lot of information, but I guess the question really I'm wondering, is it's actionable to you?
• 11:24
  Speaker 2: You see the red dot and you're like you know it's like a pollen count, right, so you know right away that the pollen count is high.
• 11:29
  Speaker 2: Is it similar to that?
• 11:29
  Speaker 2: I, like you, know it's like a pollen counter, right, so you know right away that the pollen count is high.
• 11:31
  Speaker 2: Is it similar to that?
• 11:32
  Speaker 2: I have no idea.
• 11:33
  Speaker 1: There was a dot and I clicked on the dot to see what it said.
• 11:36
  Speaker 2: Yeah, no, but I mean it's still useful, right, Because you still know.
• 11:38
  Speaker 2: I mean it's still useful because it's aggregating all this information, and West Nile is a thing right now in Massachusetts, so that makes sense, right, it's timely.
• 11:55
  Speaker 2: It linked the news articles that were in my area with dates on them and it said this is what you're looking out for, right, right, right, right, yeah.
• 12:00
  Speaker 2: So I mean, I think that that is an example of an AI maybe not AI necessarily, but a technology use case of aggregating lots of data and maybe we can talk a little bit about that, right, about how exactly it's doing that.
• 12:10
  Speaker 1: That's what I'm curious about.
• 12:11
  Speaker 1: It's like how does something like this work?
• 12:12
  Speaker 1: It's so cool.
• 12:13
  Speaker 2: Yeah, I mean one piece of technology that's been around for a while that actually led to the AI revolution that we have now with large language models, was natural language processing.
• 12:22
  Speaker 2: Natural language processing has been around forever.
• 12:24
  Speaker 2: It's just using various algorithms and computer techniques to read text and figure out patterns from those pieces of text.
• 12:35
  Speaker 2: And our current generation of large language models is in some sense, a natural language processor, because it actually takes in text, it's learned patterns from that piece of text to detect the next token, and it does that.
• 12:50
  Speaker 2: So it's a form of natural language processing.
• 12:52
  Speaker 2: But NLP, as it's called, is a pretty broad field and a very actively researched field as well, and some of this for HealthMap, for instance, their tool goes back to 2007, 2008.
• 13:05
  Speaker 2: And there's lots of tools like that.
• 13:08
  Speaker 2: At the time, who did a lot of NLP work and what they did was they would take a piece of text and they would start breaking it down.
• 13:14
  Speaker 2: And one task that machine learning folks like to do.
• 13:18
  Speaker 2: In fact, now, if you take a machine learning course, that's often like the first task is something called classification, where you're given a body of text and you're told to classify it as one category versus another.
• 13:30
  Speaker 2: For example, you get an email and it's either classified as spam or not spam.
• 13:36
  Speaker 2: So that's an example and the idea is that the NLP system is able to look at that piece of text and figure it out.
• 13:44
  Speaker 1: Wait, is that how my spam filter works?
• 13:46
  Speaker 2: I mean, that's a very old-school spam filter.
• 13:48
  Speaker 2: There are more sophisticated systems now, but that's essentially spam or not spam, and the way these NLP systems are trained is that you give them lots of examples of text that looks like spam and text that doesn't.
• 14:01
  Speaker 2: Now, obviously, that's a lot harder if the text starts to look really good and you're not sure if it's spam or not, and these things get pushed and tested and expanded in that manner.
• 14:09
  Speaker 2: But that's just one example.
• 14:10
  Speaker 2: That might not be enough.
• 14:12
  Speaker 2: Like it might not be enough to look at a social media post and decide, you know, like spam or not spam.
• 14:18
  Speaker 2: You can't just decide if it's disease outbreak or not disease outbreak, right?
• 14:21
  Speaker 2: I mean that's too coarse grained at some level.
• 14:24
  Speaker 2: So what you might need in these cases is something a little bit more narrow, a little bit more focused and, again within the NLP space, there is a classification that you can do, not just on the entire body of text but potentially on words in the text.
• 14:40
  Speaker 2: You can identify looking at a piece of text what location are they talking about?
• 14:45
  Speaker 2: What disease are they talking about?
• 14:46
  Speaker 2: Do they mention the number of cases?
• 14:48
  Speaker 2: Disease are they talking about?
• 14:51
  Speaker 2: Do they mention the number of cases, right.
• 14:52
  Speaker 2: So that's an example where it's not the whole text you're after, but you're really trying to figure out.
• 14:55
  Speaker 2: Okay, they're talking about this location and this number of cases and this disease and that information is useful because we can then use that later for something else to put together in our system.
• 15:05
  Speaker 1: Okay, so I'm going to play this back to you, so like if there were a news article that said hey, there were 17 new cases of pertussis that were detected in the greater Boston area.
• 15:19
  Speaker 1: Now you're telling me it's going to parse out things like the number of cases and the fact that it was pertussis and the location of being Greater Boston.
• 15:29
  Speaker 2: All those things are sort of parsed out from that text Exactly, and it's a topic or a task, I would say in the NLP world, and it's often known as named entity recognition, which is you're identifying pieces of text that have a certain type of type associated with them.
• 15:45
  Speaker 2: So like Pertussis or Boston, for example, is a location type.
• 15:50
  Speaker 2: So it would identify all the things in the text that are location type and Boston is one of them and then that way, be able to extract useful data from it.
• 16:00
  Speaker 2: Now you can take it and that's you know, in general.
• 16:02
  Speaker 2: Entity classification, named entity recognition all of these talk about entities that are things that are being talked about in the text Boston, pertussis, those things are entities.
• 16:13
  Speaker 2: And then there's more sophisticated things you can do.
• 16:15
  Speaker 2: You can do what's called entity and span classification, where you're not looking at the whole piece of text but you're looking at a span of words that is a contiguous sequence of words or a phrase, if you want to think about it that way, and that could mean something, right?
• 16:28
  Speaker 2: So maybe you're looking at just a phrase and it's not the whole thing you care about.
• 16:31
  Speaker 2: There's lots of other things being talked about, but you only care about a certain piece, so you scan that.
• 16:35
  Speaker 2: Then there's other relationships that you can extract.
• 16:38
  Speaker 2: You can extract causal relationships, right.
• 16:40
  Speaker 2: That might be suggested by the text, right?
• 16:44
  Speaker 2: So maybe a person flew from one country to another and they somebody they talked about it on social media about how they travel from here to here, so then you might be able to draw connections that the person was there before Now the person is here now and then build that up.
• 17:00
  Speaker 2: And you know, we talked about knowledge graphs before you could imagine, you could extract, look at these pieces of text and extract out graphs that represent the whole world and all the movements of these people right out graphs that represent the whole world and all the movements of these people right, which is, again, potentially very useful.
• 17:18
  Speaker 2: One really cool thing that Health Map did, in addition to looking at all of these social media posts and extracting useful things, is then overlay it on a map.
• 17:21
  Speaker 2: And that's super interesting too, because social media posts not only contain text but they contain metadata, and this metadata has all kinds of other information GPS, location, timing, all these other things that you don't necessarily have to see when you read the text, but it's in there and it's very useful if you want to know where in the world this was posted from and if you know where in the world it was posted from and the timing and such, that can help you correlate timelines for the disease progression, that can help you figure out spatially, geospatially, where in the world this was happening, and that's what you know.
• 17:57
  Speaker 2: Those are the kinds of things that HealthMap is doing behind the scenes to potentially do that overlay that we saw before.
• 18:06
  Speaker 1: Yeah, it's so interesting because it's like there's so much information out there so it is able to figure out like basically like what's important and where is it happening.
• 18:17
  Speaker 1: And I saw also that it's able to use Google Translate too.
• 18:22
  Speaker 1: So, you can get information in different languages and it's still able to utilize and aggregate that.
• 18:29
  Speaker 2: Yeah, and in fact, one thing.
• 18:31
  Speaker 2: So HealthMap is from the era of NLP.
• 18:34
  Speaker 2: That was before a lot of the modern day LLM work that we have seen in the AI world, and so you could imagine applying these modern techniques and enhancing systems like Health Map quite a bit, because not only now, when you extract out pieces of text and you're doing this business of entity classification and classification of text and such, it's often very contextual, so like something being said is often only understood in the meaning in which it was said and the it's often very contextual, so like something being said is often only understood in the meaning in which it was said, in the context in which it was said Maybe there was other posts that were related, and so you understand it from that context and modern day LLMs are very good at understanding context and so they would potentially be better at doing these kinds of tasks because they would be able to figure out that you know, bank here means financial institution, not the side of a river, and so, like they're able to figure out those kinds of differences and so that I think you know you can easily imagine LLMs being used at various points in this early warning system to extract out interesting features from pieces of text, to identify what's being said there.
• 19:43
  Speaker 2: And as we develop new factors, maybe we care about not just position and disease, but maybe we care about other things.
• 19:47
  Speaker 2: We can start to introduce those as well and start to get more and more useful pieces of data.
• 19:53
  Speaker 1: The challenge always is separating the noise from that's literally what I was just going to ask.
• 19:57
  Speaker 1: How do you know what's really the signal, the right stuff, the stuff that HealthMap or these other applications want to pull in, versus stuff that's irrelevant or wrong or whatever?
• 20:09
  Speaker 1: How do you differentiate the signal from the noise?
• 20:14
  Speaker 2: I mean, you don't to some degree, and people, even some of the best tools, make mistakes, and there's a certain degree of accuracy and there's a certain you know, there's this notion of false positives and false negatives and things like that.
• 20:26
  Speaker 1: But do they try Like do these tools try?
• 20:28
  Speaker 1: They must try.
• 20:29
  Speaker 2: Try.
• 20:29
  Speaker 2: What do you mean by try?
• 20:30
  Speaker 1: Try to separate the signal from the noise.
• 20:32
  Speaker 2: Yeah, yeah, yeah.
• 20:33
  Speaker 1: The right stuff that they want to synthesize and aggregate from.
• 20:36
  Speaker 1: You know stuff that may be wrong, especially when you talk about.
• 20:40
  Speaker 1: I mean news articles, hopefully, are well researched, but other sources of information like social media, it may just be one person saying one thing and you don't know how accurate that could, or couldn't be.
• 20:53
  Speaker 2: So that is a great question and it's not clear to me what the tools exactly are doing.
• 20:58
  Speaker 2: But it seems like you could have a tool that is purely passive, in the sense that it sees all the data, it does all the classification that you ask it to do, figures out all the positions, you know, the locations and the disease and all that stuff, and then it does some processing and then makes a conclusion as to whether or not there's a.
• 21:15
  Speaker 2: You know it should be a red dot or a green dot, right?
• 21:17
  Speaker 2: That's kind of roughly the pipeline, but of course some of this may need to be active.
• 21:25
  Speaker 2: If a human were in that loop there, they might say, hey, wait a minute, you're basing your red dot conclusion on this social media post, but that's not what it's really saying.
• 21:36
  Speaker 2: That media post is really talking about this other thing and we need to go check that other thing.
• 21:40
  Speaker 2: So there's more active back and forth.
• 21:41
  Speaker 2: You need to go back and look at things and come back and verify all the pieces of data.
• 21:47
  Speaker 2: I'm not 100% sure to what degree these systems are doing that, but then again, with modern AI systems, especially AI agents that can go out and gather more information or answer a particular specific, targeted question that's come up because of the way they've set up their intelligence.
• 22:05
  Speaker 2: It's possible that you could have those kinds of systems that are better at actually working through why something's an epidemic risk versus something's not.
• 22:13
  Speaker 2: But again, you don't want these systems to be end-to-end.
• 22:16
  Speaker 2: You don't want one bunch of data going in and then all processing happening behind the scenes and then one output.
• 22:22
  Speaker 2: The problem with that is, if all the processing is happening in one system, then you have a bigger sense for error.
• 22:29
  Speaker 2: There's a possibility that the system doesn't fully understand the complexity of the task involved, whereas if you have humans designing these things and extracting out useful pieces of information or breaking up that big system into smaller submodules, then you have potentially, I would argue, a better, more accurate system.
• 22:46
  Speaker 1: Well, yeah, and then, of course, when you have this like alert oh, this epidemic might be happening, or this is an early warning sign of a disease you really do need the public health officials not only to step in and verify, like you were talking about, but to actually intervene.
• 23:02
  Speaker 1: It's like, if this is happening, what do we do about it and how do we go about it?
• 23:12
  Speaker 2: Yeah.
• 23:12
  Speaker 1: Have we seen any good successes with HealthMap?
• 23:13
  Speaker 1: I don't.
• 23:14
  Speaker 1: I'm trying to remember, yeah, some of these applications we have seen successes with.
• 23:16
  Speaker 1: I mean there was applications we have seen successes with.
• 23:20
  Speaker 1: I mean there was there was, interestingly, an outbreak of, like a vaping related lung disease.
• 23:23
  Speaker 1: So this is not something that you know is like a bacteria or virus or something, but they were able to identify that In 2009, the influenza H1N1 outbreak it was involved in, and some of these actually were able to detect early signs of COVID, like HealthMap and Blue Dot.
• 23:42
  Speaker 1: So it is really interesting to see that they do work, they could work.
• 23:48
  Speaker 1: But there have also been examples of AI misfiring too.
• 23:53
  Speaker 1: One example of that is Google Flu Trends.
• 23:56
  Speaker 1: I don't know if you have heard of that, yeah, yeah.
• 23:57
  Speaker 1: One example of that is google flu trends.
• 23:59
  Speaker 1: I don't know if you have heard of that.
• 24:01
  Speaker 1: It's been shut down.
• 24:02
  Speaker 1: Um, probably like 10 years ago it got shut down, so, um, but it was live for a while and at first it was pretty accurate.
• 24:12
  Speaker 1: There were some papers where it was citing like a 97 accuracy in terms of detecting flu trends, but later on it ran into a couple of missteps.
• 24:24
  Speaker 1: One was that it really did not detect this 2009 influenza outbreak and then the second was that it started to really overestimate how much flu was out there and that was sort of on the tail end and that's kind of when it ended up closing down around 2011 to 2013.
• 24:48
  Speaker 1: It really started to overestimate and part of the reason for this was that was the way that it was basically like I'm not gonna do this justice, but the way that it was basically like I'm not going to do this justice but the way that it was programmed, where people were searching for flu and cough and flu-like symptoms in their searches and there were a lot of news articles about the flu, right, but that did not necessarily mean that there were rising flu cases.
• 25:19
  Speaker 2: Right, so that is super fascinating.
• 25:21
  Speaker 1: Right, and so in my mind it's like well, just because someone's searching for it or because there's a news article about it doesn't mean that there is necessarily an outbreak or there are necessarily a ton of cases happening at the moment.
• 25:35
  Speaker 1: But I can imagine how, you know, an NLP model could think I'm using the word think could think that it's like can't really think, but you need to retrain that sort of model in that case.
• 25:52
  Speaker 2: Yeah, I think that oh, that's you know.
• 25:54
  Speaker 2: It points to a really important piece, which is the there's good data and bad data, and there is.
• 26:01
  Speaker 2: While these social media posts give us a lot of interesting insights and potential, you know, directions and even early warnings, they're also, you know, susceptible to this, and it's not like this was a, you know, was a bad performance on the model itself.
• 26:19
  Speaker 2: It was a situation where there was new data that did not seem to, should not have been considered right, should have been ignored, and knowing what to ignore and what to take into account and why some data points are more important than others is an open challenge.
• 26:33
  Speaker 2: I think that's a really difficult problem to solve and I'm sure there's lots of people working on it, but that's a very difficult technical AI problem to work on.
• 26:41
  Speaker 1: Yeah, and even though it was great at first, it was clear that as the news changed and the searches changed, we needed to keep learning, have the model, keep iterating and maybe bring the human back into the loop to do that, right, yeah, absolutely.
• 26:58
  Speaker 1: Maybe bring the human back into the loop to do that, right, yeah, absolutely.
• 26:59
  Speaker 1: But I do think, from some of these examples and from what we've talked about, ai is excellent at identifying these early signals rapidly from vast amounts of noisy data, and that's when you need to bring in the human and say, okay, let's verify that this data is correct and let's figure out how are we going to intervene on this vaping induced lung injury or on this, you know, flu outbreak, or on this potentially new disease Right, absolutely.
• 27:32
  Speaker 1: So I want to leave us with a couple of take homes before we say goodbye.
• 27:35
  Speaker 1: One is that AI can generate early warning signs for disease outbreaks, and it can do this using large quantities of unfiltered data, and it can do this both at the global level and at the hyperlocal level, and it allows us to be more and more proactive about our public health approach here.
• 27:56
  Speaker 1: And then my second take home is that it's not enough, right, we also need humans to be involved in this process in order to verify is this actually happening Do we see this pattern and then, secondarily, to intervene if it is right.
• 28:16
  Speaker 1: So we see the early warning signal and we need to do something about it to stop the outbreak, and this is where a tool like this could be just amazingly helpful.
• 28:26
  Speaker 2: Great On that note wear your bug spray.
• 28:29
  Speaker 2: Yeah, that's right.
• 28:30
  Speaker 2: Thank you for joining us.