Latest / The Alerting Authority / How Do We Know If an Emergency Alert Actually Works? | The Science of Warning Messages

Transcript

How Do We Know If an Emergency Alert Actually Works? | The Science of Warning Messages · The Alerting Authority

• 0:09
  SPEAKER_01: And we welcome you to another episode of the Alert Authority.
• 0:13
  SPEAKER_01: And as always, we encourage you to subscribe, follow, listen, and most importantly, participate in these podcasts.
• 0:19
  SPEAKER_01: We want your questions, your concerns, your ideas, your problems, your pain points, and your success stories so we as alerting authorities can better do our jobs and make our community safer.
• 0:32
  SPEAKER_02: And we appreciate all the work that we've been able to do in the Alerting Authority.
• 0:36
  SPEAKER_02: And we've gone out and done training.
• 0:37
  SPEAKER_02: And so we want to talk about this aspect of it for the sponsorship because we figure it's better to find out ahead of time what you need work or help with and that what we can help you on.
• 0:49
  SPEAKER_02: The emergency itself is not the time to discover your weakness.
• 0:53
  SPEAKER_02: So at the Alerting Authority, we help organizations strengthen their alerting programs through practical training, governance and policy reviews, message design, exercises, and real-world guidance that prepares people to make good decisions under pressure.
• 1:07
  SPEAKER_02: When every second counts, confidence should come from competence, not just crossed fingers.
• 1:14
  SPEAKER_02: So let us know and happy to work with you.
• 1:18
  SPEAKER_01: Yeah, we've had a lot of fun going out and training in different parts of the country.
• 1:24
  SPEAKER_01: Last week we were in Franklin County, Ohio, where we got participants from all across the state, including the Ohio State University, as part of our training group.
• 1:36
  SPEAKER_01: And we've spent time in Texas and San Mateo and just a lot of great places to go visit.
• 1:43
  SPEAKER_01: And we would love to come and visit yours in the future.
• 1:47
  SPEAKER_01: So today we have a very special treat.
• 1:54
  SPEAKER_01: And I say it's a special treat because we're gonna get super geeky.
• 1:58
  SPEAKER_01: We're gonna go deep into research and research methods because uh I have invited, Eddie and I have invited Dr.
• 2:08
  SPEAKER_01: Hugh Walpole to join us today.
• 2:11
  SPEAKER_01: Um, Hugh is a consulting research scientist for the Warren Room.
• 2:16
  SPEAKER_01: He previously worked with me at the University of Albany as one of my research scientists in the emergency and risk communication message testing lab.
• 2:25
  SPEAKER_01: Long name.
• 2:26
  SPEAKER_01: But the really important part about that is that he was my methods person.
• 2:31
  SPEAKER_01: He is a brilliant scholar, and we've asked him very nicely to bring the language down from postdoc to accessible communication.
• 2:44
  SPEAKER_01: So we're gonna do that today, uh, but we're gonna talk in depth about research methods.
• 2:48
  SPEAKER_01: I'm really excited.
• 2:49
  SPEAKER_01: So here's a little bit more about Hugh.
• 2:51
  SPEAKER_01: Um, his research focuses on how people perceive risk and respond to risk communication associated with a wide variety of natural and technical technological hazards and psychological measurement of risk-related concepts.
• 3:05
  SPEAKER_01: Um, Hugh received his PhD in environmental social sciences from the Ohio State University.
• 3:12
  SPEAKER_01: So welcome, Hugh.
• 3:14
  SPEAKER_00: Oh, I'm really thrilled to be here.
• 3:16
  SPEAKER_02: Thanks for having me.
• 3:17
  SPEAKER_01: Yeah.
• 3:18
  SPEAKER_02: Can I can I throw in just a little plug here for anybody who's that practitioner who's just like, oh man, it's gonna get too geeky.
• 3:26
  SPEAKER_02: I'm not gonna understand.
• 3:27
  SPEAKER_02: Stand down.
• 3:28
  SPEAKER_02: All right.
• 3:29
  SPEAKER_02: You are gonna get stuff out of this.
• 3:30
  SPEAKER_02: And it's super important.
• 3:32
  SPEAKER_02: And we talk about this.
• 3:33
  SPEAKER_02: I I don't just need you to understand to if I say do it and you do it, right?
• 3:39
  SPEAKER_02: I need you to understand the why and how did you get there?
• 3:43
  SPEAKER_02: And for me, I've been an investigator for a very long time.
• 3:45
  SPEAKER_02: Whenever anybody tells me, hey, this is what you should do, I'm naturally inclined to say, why?
• 3:51
  SPEAKER_02: How did you get that?
• 3:52
  SPEAKER_02: I mean, convince me.
• 3:54
  SPEAKER_02: And that's what we want to really do with this episode.
• 3:57
  SPEAKER_02: And and and again, I'm gonna say, do I say doctor?
• 4:01
  SPEAKER_02: Because now it's doctor, doctor, and not a doctor.
• 4:04
  SPEAKER_00: You can't possibly.
• 4:06
  SPEAKER_02: But it's I I really want to invite everyone who's listening, think, think about these things that are gonna get talked about because when we're telling you follow the research and and everything else, it's because of how the data was collected, how it was analyzed, the findings that came from from it.
• 4:24
  SPEAKER_02: And that's why you're gonna get that like peek behind the curtain today.
• 4:28
  SPEAKER_02: So I'm gonna do my best to make sure that that Hugh does it in a way that I can understand.
• 4:34
  SPEAKER_02: And if I can understand it, I have a very high level of confidence that all of you can too.
• 4:39
  SPEAKER_02: So, all right, we'll keep going.
• 4:42
  SPEAKER_01: They're gonna expect you to speak in another language when you start talking, Hugh.
• 4:47
  SPEAKER_01: Well, this is not the case.
• 4:49
  SPEAKER_01: All right, let's kick it off.
• 4:50
  SPEAKER_01: Um, so I read your bio.
• 4:52
  SPEAKER_01: Can you introduce yourself a little bit better?
• 4:54
  SPEAKER_01: Like, how did how did you become interested in risk and risk perception and quantifying responses to risk communication?
• 5:01
  SPEAKER_01: Like, how'd you get into it?
• 5:03
  SPEAKER_00: Yeah, um, so I'm Hugh Walpole.
• 5:06
  SPEAKER_00: Um, I got started in this uh during graduate school at the Ohio State University.
• 5:12
  SPEAKER_00: Thank you for getting that right.
• 5:13
  SPEAKER_00: Um I want to I want to start by just expressing the enormous amounts of of fear about being described as a methods expert, because I always think of myself as a person who knows just enough to be dangerous, um, which in this context might be great.
• 5:29
  SPEAKER_00: Um communication researchers are legendary for being the worst communicators around.
• 5:35
  SPEAKER_00: Um and so hopefully knowing just enough to be dangerous is is kind of the the space that I need to be in for this.
• 5:42
  SPEAKER_00: Um But yeah, so I got started uh in graduate school.
• 5:46
  SPEAKER_00: I was studying environmental social sciences at the School of Environment and Natural Resources in Franklin County at the Ohio State University.
• 5:52
  SPEAKER_00: Um and uh my advisor, Dr.
• 5:54
  SPEAKER_00: Robin Wilson, um, and I got a lot of questions from practitioners every time we went to speak at conferences or um working with community partners or other people because we would talk a lot about how people think and feel about risks from hazards is really important.
• 6:08
  SPEAKER_00: Uh, and people would ask us, ooh, hey, we want to do a survey to kind of figure out how people feel about the hazards that uh we deal with as an organization.
• 6:16
  SPEAKER_00: And we'd really like to ask them questions about how they think about those hazards.
• 6:20
  SPEAKER_00: What questions should we ask them?
• 6:21
  SPEAKER_00: And we said, uh, uh-oh, well, I don't know.
• 6:26
  SPEAKER_00: Um there's there's a lot of research on different kinds of things and bringing those together.
• 6:31
  SPEAKER_00: And for every hazard, there's a different way of asking about how people think and feel about that hazard or the associated behaviors.
• 6:38
  SPEAKER_00: And so for my dissertation work, um, I focused a lot on trying to bring together a lot of those different threads, trying to figure out how we can talk about uh risks of different kinds, whether that's a health risk or a natural hazard or technological hazard, and think about the unifying concepts, the things that are present in people's perceptions for all of those different things, and tie those all together so that when we're doing this kind of research, we can compare apples to apples.
• 7:05
  SPEAKER_00: Um, that was kind of the biggest challenge that we had when we were trying to give people advice on how to measure that stuff.
• 7:11
  SPEAKER_00: So largely I got into that kind of by accident.
• 7:14
  SPEAKER_00: It was uh tangential to a lot of the work that I was doing at the time on uh wildfire evacuation decision making.
• 7:20
  SPEAKER_00: So thinking about how people make a decision to either stay or leave uh in the event that they've got a wildfire threat in their property, which is what I was doing for my master's research, and ran into all sorts of measurement problems while I was conducting that research.
• 7:34
  SPEAKER_00: So got really excited about trying to figure out how to do that better.
• 7:37
  SPEAKER_00: Um and it's kind of led me on a meandering path to where I am today.
• 7:43
  SPEAKER_01: I think your path is really fun and exciting because I like to geek out about research too.
• 7:48
  SPEAKER_01: And um, thinking about how different people's perceptions are, how they vary in response to different kinds of hazards.
• 7:57
  SPEAKER_01: The first thing that comes to mind is like the idea of hazards that are dreaded more than others, you know, the the classical stuff coming out of Fischoff and and Slovak and thinking about you know, the things that are that you can't see, you can't feel, you can't, you don't know the long-term impacts.
• 8:13
  SPEAKER_01: And those are the ones that are most scary to people.
• 8:17
  SPEAKER_01: Yeah.
• 8:17
  SPEAKER_01: Um, yeah, I love thinking about how do we measure that versus something that's like the everyday common hazard of being exposed to like, I don't know, some chemical in your home.
• 8:28
  SPEAKER_00: Right.
• 8:28
  SPEAKER_00: Yeah, I had a colleague who did a lot of work on radon in Ohio, and it's a thing that people don't think about a lot, but um does kind of pose a risk that's sort of more everyday.
• 8:37
  SPEAKER_00: Uh, and kind of thinking about those kinds of things is just fascinating.
• 8:42
  SPEAKER_01: Yeah.
• 8:43
  SPEAKER_02: Well, I didn't meet you when we were talking about radon, but I did meet you when we were working on the FEMA sponsored project of the message design dashboard, right?
• 8:53
  SPEAKER_02: And I got to work with you on that team.
• 8:56
  SPEAKER_02: Um, you were obviously a much bigger player on that team.
• 8:59
  SPEAKER_02: Um, but what's really fun is one of the areas of your responsibility that I understand, right, is you were able to help think through all the different ways to measure public responses to emergency messages, which I I would love everybody to understand, it's because this hadn't really ever been done.
• 9:18
  SPEAKER_02: Like there, I mean, there that we have research on on messages, on design, on on a lot of different things.
• 9:26
  SPEAKER_02: But when we get into what was looked into, this was all, I mean, groundbreaking stuff.
• 9:33
  SPEAKER_02: And as a practitioner, something that I I wanted the information because it was gonna help me be better.
• 9:40
  SPEAKER_02: But help take us back to school just a little bit, because we're gonna talk about like the basic units of of measurement in social sciences.
• 9:50
  SPEAKER_02: Because uh what I don't want to do is to just say, well, we did this, you know, or you know, whatever, whatever maybe, because I know what's coming up.
• 9:59
  SPEAKER_02: Can you talk about those different types of measurements?
• 10:03
  SPEAKER_02: Um, because I think we'll again talk more about them, but there's a couple.
• 10:07
  SPEAKER_02: Can you go for it?
• 10:10
  SPEAKER_02: Yeah.
• 10:11
  SPEAKER_02: I'm fumbling here, folks, but because I'm I want to be like, explain to me the difference because you have qualitative, you have quantitative, but then there's there's even more.
• 10:21
  SPEAKER_00: Yeah, so I I think the qualitative-quantitative divide is is the best place to start because those are ways that we categorize a lot of the different methods that we use.
• 10:29
  SPEAKER_00: So a lot of the methods that we use get really, they feel really complicated and they're really specific to certain things, but they all fall into these two kind of broad categories.
• 10:37
  SPEAKER_00: You have qualitative methods, you have quantitative methods, and then I'm gonna complicate it a little bit more because there's some hybrids too that kind of bridge that gap.
• 10:44
  SPEAKER_00: But if you think about qualitative data and the methods that are associated with them, um, mostly we're just thinking about data that people convey to us in words.
• 10:53
  SPEAKER_00: They're talking about a concept, having a conversation, an interview, a focus group, something exactly like this, right?
• 10:58
  SPEAKER_00: You ask me something about uh what kinds of different uh data types or measurement types there are in the social sciences.
• 11:06
  SPEAKER_00: I'm giving you qualitative data right now on what kinds of things are are in there.
• 11:11
  SPEAKER_00: So if you were social scientifically inclined, you could go in and kind of pull apart the things that I'm saying to try to pull out individual concepts about um, in this case, research methods.
• 11:21
  SPEAKER_00: And those would be qualitative data.
• 11:23
  SPEAKER_00: Um usually when I think about qualitative data, uh, I'm thinking about answering questions about what and why.
• 11:31
  SPEAKER_00: So I'm really interested in what kinds of things people are thinking about, what's important in the kind of emergency management, emergency communication context.
• 11:41
  SPEAKER_00: I might be thinking about what kinds of information people need or think that they need when they get a piece of emergency communication.
• 11:46
  SPEAKER_00: I might think about what kinds of behaviors they want to engage in when they get a message of different kinds.
• 11:52
  SPEAKER_00: And so thinking about what kinds of things are important is something that qualitative research is really, really useful for.
• 12:00
  SPEAKER_00: And I think it's particularly true, and there's a lot of qualitative researchers out there who might say it's true all the time.
• 12:06
  SPEAKER_00: But I think the the most important time in my work when we use qualitative research is when we don't know what we don't know.
• 12:15
  SPEAKER_00: Um, we don't have a great sense of the lay of the land.
• 12:18
  SPEAKER_00: And so our best option when we want to know something about something that we don't know a lot about is to just ask people.
• 12:24
  SPEAKER_00: Ask the people who are um particularly affected or particularly knowledgeable about that topic.
• 12:31
  SPEAKER_00: So um we get really complicated in a lot of ways, but at the end of the day, psychological measurement, measurement in general just comes down to if you want to know the answer to something, ask uh and people will tell you.
• 12:43
  SPEAKER_00: So um if we think about it in general terms, like an example, if I want to know uh what a population thinks about a tsunami risk, right?
• 12:52
  SPEAKER_00: I might say, okay, what are the what are the consequences that you think about the most when you're being threatened with a tsunami?
• 12:59
  SPEAKER_00: And people might say, like, oh, I'm thinking about the the giant wave that's coming that's that's gonna do damage.
• 13:04
  SPEAKER_00: And I say, okay, yeah, that's that's great.
• 13:06
  SPEAKER_00: What else kind of comes to mind?
• 13:07
  SPEAKER_00: And they would be like, well, I'm really worried about the the wave bringing a bunch of man-eating sharks up onto the shore uh that are then gonna, you know, legit me.
• 13:15
  SPEAKER_02: Yeah.
• 13:15
  SPEAKER_00: Right?
• 13:16
  SPEAKER_00: And so as a as a researcher, maybe as a tsunami expert, maybe as uh uh you know, someone who's working on a designing a project like this, that might not be the first consequence that comes to your mind.
• 13:28
  SPEAKER_00: And you might not think to ask about are you worried about the sharks that will come out of a tsunami when it happens?
• 13:34
  SPEAKER_00: Uh to be clear, I'm pretty sure that that's low on the list of priorities when you have to worry about uh a tsunami risk.
• 13:39
  SPEAKER_00: But again, I'm not a tsunami expert, so tsunami experts at me on that.
• 13:43
  SPEAKER_00: Um but if we didn't do the qualitative research that we did uh at the start of this imaginary project, then we wouldn't know that that's a risk that people are really concerned about.
• 13:52
  SPEAKER_00: And as we proceed with our research or with our practice, it's really useful to know what those people are perceiving and thinking about as those primary consequences that they're dealing with.
• 14:04
  SPEAKER_00: So if we think about those qualitative uh bits of data as just words or concepts that people will convey to us in an interview or in a focus group, which uh is a lot like an interview setting.
• 14:15
  SPEAKER_00: It's just there's more people involved.
• 14:17
  SPEAKER_00: So you would be interviewing a group of people at the same time.
• 14:20
  SPEAKER_00: You can get some crazy stuff out of those.
• 14:23
  SPEAKER_00: Um, it really does help kind of break the ice, get some people in some social dynamics to think about it.
• 14:29
  SPEAKER_00: And you get some really different, rich kind of data when you can talk to a group of people all at once.
• 14:34
  SPEAKER_00: Um it's it's really fun stuff, but uh I'll move on.
• 14:38
  SPEAKER_00: So if we think about kind of the other big division where I spend most of my time um and where a lot of the methods tend to get complicated in the legendary ways where people are like, I don't understand any of this, um, is in the quantitative space.
• 14:52
  SPEAKER_00: So when we think about quantitative data, we're thinking about data that's stored in the form of numbers, right?
• 15:00
  SPEAKER_00: Um most often in the social sciences, uh, when you're talking about quantitative data, what you're looking at are scaled responses to items on a survey or in an experiment, which we'll talk about in a little bit.
• 15:15
  SPEAKER_00: But um, I might, for instance, ask you, um, to what extent do you think that the learning authority is the best podcast on the internet?
• 15:25
  SPEAKER_01: Oh, it's a 10.
• 15:26
  SPEAKER_00: 10 strongly disagree to five strongly agree.
• 15:28
  SPEAKER_00: So you would say a 10, and I would say great.
• 15:31
  unknown: Yeah.
• 15:31
  SPEAKER_00: So you would say strongly agree, I would say perfect.
• 15:34
  SPEAKER_00: That's I I think that's that's that's great.
• 15:36
  SPEAKER_00: So I will put down a five for you on that.
• 15:38
  SPEAKER_00: If I ask a hundred people that question, then because everybody's answers are in the form of numbers, I can combine those numbers in ways that's really hard to do with qualitative data.
• 15:50
  SPEAKER_00: I can take those numbers and I can take an average.
• 15:52
  SPEAKER_00: I can say, like, ah, the alerting authority among the hundred people that we asked, the average was a 4.7.
• 15:57
  SPEAKER_00: Everybody agrees to strongly agrees that's the best podcast on the internet.
• 16:01
  SPEAKER_00: And that gives me some really unique capabilities in terms of making comparisons between uh different people, different groups of people, uh, or different, for instance, podcasts, right?
• 16:12
  SPEAKER_00: I can say, like, ah, the alerting authority got a 4.7 on average out of my population, but something like Radio Lab or something was like a 4.2.
• 16:20
  SPEAKER_00: Oh, look at that.
• 16:21
  SPEAKER_00: Among my population, on average, people think the alerting authority is the best podcast on the internet more than they think Radio Lab is the best podcast on the internet.
• 16:28
  SPEAKER_00: Um so it gives us those kind of unique capabilities to then go further and use some statistical analyses to make a comparison between the averages on uh one set of questions versus another and tell us which one of these is really bigger, um, which one of these is kind of uh a significant difference between those groups.
• 16:53
  SPEAKER_00: And we'll talk about significant differences a lot when we talk about quantitative research.
• 16:56
  SPEAKER_00: And I can go into more of that uh if people care.
• 16:59
  SPEAKER_00: But largely it's just a statistical way of saying like this difference is here, but is it meaningful?
• 17:06
  SPEAKER_00: Um so generally speaking, when I think about quantitative uh results or or data or research, we're answering a different kind of question than we normally do for qualitative research.
• 17:18
  SPEAKER_00: So rather than asking what or why questions for the most part, quantitative research is really, really good at answering how many and how much.
• 17:28
  SPEAKER_00: So we want to get a sense of how important a belief is in a community.
• 17:32
  SPEAKER_00: We want to get a sense of how many people evacuate.
• 17:36
  SPEAKER_00: We want to get a sense of whether or not people think that a wildfire is riskier than the flood that comes after it.
• 17:43
  SPEAKER_00: Umsing quantitative methods can give us some ability to make some judgments about which ones are being perceived as as more important in that community.
• 17:55
  SPEAKER_00: Um so what I think is really kind of critical about that, thinking in the context of qualitative uh research, is that the quantitative research is most useful when we do qualitative research first.
• 18:09
  SPEAKER_00: I can only ask you questions quantitatively if I know what questions to ask you.
• 18:16
  SPEAKER_00: And so at the start of a process, when you know not very much about what a population thinks, doing that qualitative research to start to get a sense of what matters can give you a really good sense of what to ask about when you're asking how much do those things matter, or what matters more than other things, um, which is what quantitative data is really good for.
• 18:35
  SPEAKER_00: Uh, it can trip you up when you start with quantitative research, which so many of us are tempted to do because there's a lot of fun methods we can use.
• 18:43
  SPEAKER_00: Um, but if we start asking questions before we know the lay of the land and we assume that we know what matters and what people are thinking, we can run into some serious trouble when we're kind of trying to make judgments about why people do sudden things and why they don't, if we haven't asked them the right questions.
• 19:01
  SPEAKER_00: So those qualitative parts of the research process are so important to helping us do that.
• 19:06
  SPEAKER_00: Um if we think about kind of some specifics, I want to give a couple of examples of just what kinds of methods we might use for these kinds of things for.
• 19:14
  SPEAKER_00: Um there's a great hybrid method that we see uh in the research that we do a lot, um, but we see all over the place that's focused on what's called content analysis.
• 19:26
  SPEAKER_00: So content analysis is a really useful way to take a bunch of qualitative data that you've got and ask some quantitative questions about it.
• 19:34
  SPEAKER_00: So a lot of the time uh you've collected a bunch of qualitative data, or there's qualitative data out in the world, um, documents that exist, you can do secondary analysis on things like um uh messages themselves that have gone out in the past, or you can look at uh plans for emergency response and make comparisons between them on that basis.
• 19:53
  SPEAKER_00: But if we want to identify kind of what the most prevalent concepts that go out in messages are, for instance, we need a way to quantify how frequently certain things come up.
• 20:03
  SPEAKER_00: And content analysis is a way for us to label the qualitative information that comes up across those different documents and identify kind of those quantitative characteristics about it.
• 20:15
  SPEAKER_00: How frequently does it come up?
• 20:17
  SPEAKER_00: How important does it seem to those groups?
• 20:19
  SPEAKER_00: And so using a content analysis is really, really useful.
• 20:23
  SPEAKER_00: For instance, the tsunami example, if one person in our 50 interviews brings up shark attacks as a serious tsunami risk, uh we might not think it's terribly important.
• 20:33
  SPEAKER_00: But if it's 30 people, we might think, oh, this is a very important central concept for this population for tsunami risk.
• 20:40
  SPEAKER_00: And so if we're doing quantitative research later, we really need to ask about the sharks.
• 20:45
  SPEAKER_00: Whereas we might not do that if it was just one.
• 20:48
  SPEAKER_00: Um I can't help myself.
• 20:52
  SPEAKER_00: I gotta bring up my favorite thing in the world.
• 20:55
  SPEAKER_00: It's a special case of quantitative kind of research data, and that's an experiment.
• 21:00
  SPEAKER_02: Here we go.
• 21:01
  SPEAKER_00: Here's the you.
• 21:03
  SPEAKER_00: So this that was all to get all of that out of the way.
• 21:05
  SPEAKER_00: I hope you have the next 45 minutes for me to talk about experiments.
• 21:08
  SPEAKER_00: They're great.
• 21:11
  SPEAKER_00: But um most people are familiar with kind of how an experiment works.
• 21:16
  SPEAKER_00: Um, but just to kind of roll it back.
• 21:19
  SPEAKER_00: In case we're not.
• 21:20
  SPEAKER_00: In case we're not.
• 21:22
  SPEAKER_00: The idea behind an experiment is that you uh randomly assign, I'm gonna use some jargon here and then I'm gonna walk it back.
• 21:30
  SPEAKER_00: You randomly assign people to receive one of a variety of different treatments.
• 21:36
  SPEAKER_00: And they use the word treatment because it's generic to any number of things that you could do.
• 21:41
  SPEAKER_00: Um and treatments could be any number of different things, giving people different information, putting them in a different situation, um, having them live in a different place for a period of time.
• 21:52
  SPEAKER_00: That you anything that you can think of that you could systematically vary between people could be a treatment.
• 21:57
  SPEAKER_00: If we're thinking about it for emergency care.
• 22:00
  SPEAKER_00: Communication, most typically, what we're talking about for treatment is giving people a different message.
• 22:05
  SPEAKER_00: So, message that contains different information, is structured a different way, talks about uncertainty differently, what have you.
• 22:12
  SPEAKER_00: There's a kind of a wide variety of different ways that you can vary different things.
• 22:17
  SPEAKER_00: What's really important is that you randomly assign people to one of any number of groups within there that have different treatments.
• 22:27
  SPEAKER_00: And the best thing about an experiment is that because you're randomly assigning people, you get the experiment superpower, which is that because you're choosing what group or what treatment people are in on the basis of just a random number instead of some characteristic of that person, you can attribute any of the differences that you see between those groups exclusively to the differences between the treatments.
• 22:53
  SPEAKER_00: Because there's no logical reason for you to assume that the groups will be different in any way.
• 22:59
  SPEAKER_00: So you're not going to get all of the pessimists in one group and all of the optimists in the other group because they're randomly assigned.
• 23:05
  SPEAKER_00: And so if pessimism or optimism happens to matter for how people respond to your treatment, they should be the same across those groups.
• 23:11
  SPEAKER_00: And that means that the only differences that you see between those groups should be the direct result of the changes that you made in between those two treatments.
• 23:20
  SPEAKER_00: So if you're going to be able to do that, you're limiting your variable, right?
• 23:23
  SPEAKER_00: Exactly.
• 23:24
  SPEAKER_00: Yep.
• 23:24
  SPEAKER_00: You're uh functionally eliminating anything about any of the individuals as a variable at the group level.
• 23:32
  SPEAKER_00: So nothing about that person should, in the aggregate, be affecting the results, the differences between those groups.
• 23:43
  SPEAKER_00: Because if you get an extreme person in one group, they could have just as likely been in the other group.
• 23:49
  SPEAKER_00: And that's true for any of those groups of people.
• 23:51
  SPEAKER_00: So even if you have a lot of kind of extreme individuals, uh, they'll be spread across those two groups randomly.
• 23:59
  SPEAKER_00: And so as long as the differences that you're seeing are big enough, then we can say with certainty that they're attributable to the differences in the treatment rather than differences between the people.
• 24:09
  SPEAKER_00: And that's the real power behind doing experiments, is it lets you be definitive about the thing that's causing the changes that I'm seeing or the differences that I'm seeing, uh, is what I did to the treatments.
• 24:20
  SPEAKER_00: And that in the messaging context is super useful because it lets you learn how to structure your messages differently.
• 24:28
  SPEAKER_00: You can compare messages that are structured in different ways or contain different pieces of information and then use that to help you figure out which ones are more effective at moving the outcomes you're interested in.
• 24:40
  SPEAKER_02: Can I do a little thing of don't try this at home, folks?
• 24:44
  SPEAKER_02: Because I think there's a danger when you don't understand it and then you start labeling things.
• 24:51
  SPEAKER_02: And I we're gonna get into this a little more, but I've actually got to sit at across the table, you know, with Jeanette, with Hugh, and other amazing um scientists and learn how important it is for the experiment to be done in a way to increase its validity, right?
• 25:10
  SPEAKER_02: To make it more short, to make it uh, as an example, if you were to ask 10 of your closest friends, how am I doing at writing an alert and warning, you may get a certain level of response, especially if one of those is your mother, right?
• 25:26
  SPEAKER_02: They may say, You're doing great, kid, rather than you asking what would be, I don't know, a better like group or a representative group of your population, right?
• 25:38
  SPEAKER_02: And so just uh to help, because I I really think sometimes that's what we end up doing as practitioners, is we do these little experiments or we try to seek out information, but we're not we're not doing it the way I think that you're recommending.
• 25:53
  SPEAKER_02: And so some of the information we're getting isn't right, or it's just different, let's say.
• 25:59
  SPEAKER_02: And so that's why I want people to understand there there's a method and a reason behind you talking about experiments like this.
• 26:07
  SPEAKER_00: Yeah, and I'm I'm really glad you brought that up, that idea of representativeness.
• 26:11
  SPEAKER_00: And we struggle with this a lot in terms of trying to figure out what group of people to talk to when we're trying to figure out the answers to any of the questions that we have, whether they're what questions or how much questions.
• 26:23
  SPEAKER_00: Um I think you're spot on in terms of thinking hard about who the people you're talking to are and whether they have a reason to be different from other people and how they might respond.
• 26:34
  SPEAKER_00: Um, so if you ask your colleagues uh if how well you are doing at message writing, um, if your colleagues are are any good to you, uh, they'll tell you you're doing great.
• 26:43
  SPEAKER_00: And that feels really good to hear and it's great stuff.
• 26:46
  SPEAKER_00: But a lot of the time, those people that you're gonna be asking, your friends, aren't the community that you're gonna be talking to when you write those messages.
• 26:53
  SPEAKER_00: So getting into those communities and using some kind of social scientific sampling strategies to try to make sure that we're getting um people who represent the community that you're uh communicating with, that's the best way to get the information um that you need about the quality of your message writing, the needs of your communities, and other things like that.
• 27:12
  SPEAKER_00: Um and Jeanette, you and Eddie, I know do a lot of that kind of work with different practitioners as well.
• 27:18
  SPEAKER_00: So I ask everybody to defer to you guys in terms of thinking about how to collect those kinds of data or work with others to do so.
• 27:26
  SPEAKER_00: It's definitely something that you want to be careful about doing on your own without having someone there to help walk you through some of those pitfalls, um, because it can give you that kind of false sense of confidence that you know uh exactly what people are thinking about what you're doing.
• 27:41
  SPEAKER_00: And that can get you into trouble kind of as we go down with practitioners as much as researchers.
• 27:49
  SPEAKER_01: I think a lot of times we fall into what's the easiest way of collecting data, which tends to be a convenience sample, which is where we send it off to our friends or whoever is willing to do our survey instead of working with a company that can recruit people that you don't know and have no investment in the outcome.
• 28:10
  SPEAKER_01: Um, and asking somebody that you know to talk to you to do that initial formative research, the qualitative, that can help you to identify where the the key issues are when you balance it out across other people who are willing to talk to you.
• 28:27
  SPEAKER_01: But in terms of an experiment, that's where it starts to fall apart.
• 28:31
  SPEAKER_00: So we we like to call that kind of initial talking to your friends.
• 28:35
  SPEAKER_00: We like to call it anecdotal, because it sounds better than just anecdotes, but but functionally it is just people kind of giving you their gut sense.
• 28:42
  SPEAKER_00: And while you can learn some useful things that can help you kind of refine your research in the future, I'm asking a question that people will misunderstand.
• 28:51
  SPEAKER_00: I'm uh not asking questions that I really need to be asking.
• 28:54
  SPEAKER_00: That your friends can be really helpful about giving you that kind of anecdot.
• 28:59
  SPEAKER_00: But it's no substitute for being able to go into those communities and move past that convenience sample to the extent that you can.
• 29:05
  SPEAKER_00: A lot of the times it's really challenging.
• 29:07
  SPEAKER_00: But um there really is no substitute for being able to do that if you want to draw robust conclusions.
• 29:12
  SPEAKER_00: And with the kind of work that we do, um, the kind of space that we're all in, uh, you really want to do the best you can.
• 29:20
  SPEAKER_01: Yeah.
• 29:21
  SPEAKER_01: Yeah.
• 29:22
  SPEAKER_01: See, Hugh, you said that you're not a methods person, but honestly, you just gave us like this great undergraduate lecture on the different kinds of qualitative and quantitative approaches to social science data collection.
• 29:33
  SPEAKER_01: So, like, bravo.
• 29:34
  SPEAKER_01: And I am forever grateful that I have somebody like you who's been on my team because as somebody who was is more qualitatively inclined, having your quantitative mind where you said your most favorite thing in the world is experiments.
• 29:50
  SPEAKER_01: Like you could see why having a partnership is important.
• 29:53
  SPEAKER_01: It's any partnerships are always important in this space, in research space, in emergency management.
• 30:00
  SPEAKER_01: It's the reason that Eddie and I work together.
• 30:03
  SPEAKER_01: He's like the boots on the ground.
• 30:04
  SPEAKER_01: He understands what people are going through.
• 30:06
  SPEAKER_01: And I'm bringing this more academic approach to thinking through alerts and warnings.
• 30:11
  SPEAKER_01: So, you know, it takes a village.
• 30:15
  SPEAKER_01: Um, so okay.
• 30:17
  SPEAKER_01: One of the things that I think you touched on a lot were the different paths.
• 30:23
  SPEAKER_01: But I think that when we think specifically about emergency alerts, there are some strategies that are better for thinking about what motivates people to act than other strategies.
• 30:35
  SPEAKER_01: So, can you talk a little bit?
• 30:36
  SPEAKER_01: We've had so many conversations about this particular topic.
• 30:40
  SPEAKER_01: What do you think?
• 30:41
  SPEAKER_01: What have we talked about?
• 30:43
  SPEAKER_01: Um, what's the most fruit fruitful strategy for learning about motivation to act?
• 30:49
  SPEAKER_00: Um so just thinking back on the conversation we just had, if we're starting fresh, if we're working with a fresh population, we're starting with a hazard that might be novel or something that's not terribly well studied, um, or if we're kind of in a nascent research space, then starting with that qualitative research really, really helps.
• 31:07
  SPEAKER_00: Um, it helps us learn what we don't know, and it gives us a basis for doing more robust quantitative work down the line.
• 31:14
  SPEAKER_00: So for instance, in the emergency space, we might go um talk to a group of people who've experienced a hazard of some kind, uh talk to groups of people who acted, talk to groups of people who didn't act, try to figure out what kinds of things they were thinking.
• 31:29
  SPEAKER_00: So, what kinds of actions were you considering?
• 31:31
  SPEAKER_00: Um, why were you considering those actions over other actions and that kind of thing?
• 31:35
  SPEAKER_00: And that can give us a really strong sense of the kinds of things that people were thinking about, but also the kinds of concepts that motivate the decisions that they make to ultimately take action or not.
• 31:46
  SPEAKER_00: Um so taking those answers that people give you in that qualitative data can help you design the quantitative work in a survey.
• 31:55
  SPEAKER_00: Um, I would leave experiments for later, much as I would love to dive in and do an experiment right away.
• 31:59
  SPEAKER_00: Usually you want to do that a little bit later, a little bit further in the process.
• 32:02
  SPEAKER_00: But design some quantitative work like surveys that can help you answer some of those how much or how many questions, thinking about from what I heard, from that small group of people, the rich qualitative data that I got about what they did and why and what was important, trying to get a sense of what things are the most important and what things are are kind of more general to populations.
• 32:21
  SPEAKER_00: And when you start being able to generalize outside of those tiny groups that you're talking to, that one group of survivors who uh got through a hazard and expand it to uh a larger geographic area, a more diverse group of people, you can start asking those quantitative questions and refining down what is general to more people.
• 32:42
  SPEAKER_00: Um, and once you start doing that, once you start thinking about the core ideas that are motivating people to take action for different hazards, for different parts of uh the US, different parts of the world, being able to drill down into those core concepts is when you start getting into theory.
• 32:59
  SPEAKER_00: Um so when we talk about theory in the social science, we're generally speaking bringing in concepts from dozens of studies where people have done a lot of refining, they've done a lot of qualitative work, they've followed up with a lot of quantitative work, and they're ideally drawing parallels between all of those things to help build a general sense of how we think people respond when they're being faced with pieces of information, pieces of risk communication uh with hazards and other things.
• 33:28
  SPEAKER_00: So that's kind of where we are now in the hazard space.
• 33:32
  SPEAKER_00: We've got a couple of really robust theories.
• 33:34
  SPEAKER_00: Um people differ on the ones that they use depending on what kinds of questions they're focused on.
• 33:40
  SPEAKER_00: So if they're thinking about um kind of more broader sociological questions about like why do people do X, Y, or Z, you can think about a different set of theories than if you're thinking about like, well, when a person is making a decision about whether they do X or Y, how do they make that decision?
• 33:57
  SPEAKER_00: You can use different theories to help you guide how those different kinds of processes come about.
• 34:06
  SPEAKER_00: So where we are right now in the hazard space is we have a number of these relatively robust, sizable theories that give us a sense of what kinds of things motivate people.
• 34:17
  SPEAKER_00: Um one of the things that we use a lot in our work and has a really kind of robust, uh long lineage of research is the warning response model.
• 34:28
  SPEAKER_00: Um so thinking about how you motivate people is about structuring your message around particular contents and uh making sure that people have the information that they need in order to feel motivated.
• 34:43
  SPEAKER_00: And if we want them to be motivated to take some kind of action, the warning response model, based on a lot of research, has suggested that the things that people really need, so they need to be able to understand the information that's in the message, they need to believe that the message is talking about something they need to actually be worried about.
• 35:00
  SPEAKER_00: They need to personalize the risk from that message.
• 35:03
  SPEAKER_00: So they have to feel like it's gonna affect them in some way.
• 35:06
  SPEAKER_00: So not only do I know what the risk is, I believe that the risk is real, but it's also gonna impact me.
• 35:13
  SPEAKER_00: And then finally, you also have to be able to make a decision on the basis of the information that's included in that message.
• 35:19
  SPEAKER_00: And this makes a lot of sense, right?
• 35:21
  SPEAKER_00: Um, if you think in terms of I'm receiving a message, if I don't have any of those or even one of those things, if I don't understand what the message is trying to tell me about the hazard, I don't believe it's true, I don't think it's gonna affect me, or it's so complicated or so information poor that I can't use the information in the message to make a decision, it's not gonna do a lot to motivate me to act.
• 35:45
  SPEAKER_00: So we think a lot about what kinds of things motivate people to take some kind of protective action so that we can make kind of useful comparisons between messages that have different features.
• 35:56
  SPEAKER_00: So I think we'll probably talk a little bit later about message contents and what kinds of things you need to include in messages in order to kind of build some of these motivations.
• 36:05
  SPEAKER_00: But if we're measuring those kinds of motivations and we see that uh one kind of message is helping build those motivations more than another, then we can be fairly certain, at least on the basis of the warning response model as theory, that those kinds of messages are going to be more likely to motivate people to take some action to keep themselves safe.
• 36:28
  SPEAKER_02: I feel like there's a good time to break and tell people again.
• 36:32
  SPEAKER_02: Hugh is amazing, right?
• 36:33
  SPEAKER_02: Jeanette's amazing.
• 36:34
  SPEAKER_02: We have a lot of people in this space.
• 36:36
  SPEAKER_02: If you're struggling to understand any parts of this, A, you can listen to it again, or B, you can reference the things that they're talking about.
• 36:44
  SPEAKER_02: And that's the idea, I think, is every time I hear comments and discussions like this, it doesn't make me think to myself, man, well, I want to replicate it the best that I can.
• 36:56
  SPEAKER_02: I I'm I'm actually saying I want to just use what they're doing because I I feel much better in that space.
• 37:04
  SPEAKER_02: Um so, Hugh, like one of the things that practitioners, emergency managers want to talk about is and and know about is compliance.
• 37:13
  SPEAKER_02: Um now that's not what your research focuses on.
• 37:16
  SPEAKER_02: So I guess the question to you is why are we not talking about compliance?
• 37:22
  SPEAKER_00: Yeah, this is this is a really good question.
• 37:23
  SPEAKER_00: And I think it's really, really important.
• 37:25
  SPEAKER_00: Compliance is such a tempting metric to use.
• 37:28
  SPEAKER_00: I I get it.
• 37:29
  SPEAKER_00: It's it's objective.
• 37:30
  SPEAKER_00: It's one of the only objective senses that we have of like how well did something work?
• 37:36
  SPEAKER_00: Did people do what we told them to do?
• 37:39
  SPEAKER_00: Um so the way that I think about compliance, and and you can correct me if you have a kind of a different interpretation, is did people do the behavior that I included in the message as the thing I ordered them to do?
• 37:50
  SPEAKER_00: If you order a mandatory evacuation, um, a shelter in place, or any number of other behaviors, did they do what they were told to do?
• 37:59
  SPEAKER_00: Um again, it's really tempting to use, but we don't use it in our research for a couple of, I think, really important reasons.
• 38:08
  SPEAKER_00: Um the first is that there's a number of different reasons that people might do things that are different than we ask them to do to keep themselves safe.
• 38:18
  SPEAKER_00: Um I think generally speaking, in the emergency community, in the risk community, we care so much more that people make themselves safe than they did what they were told.
• 38:31
  SPEAKER_00: Um, so there's a number of different behaviors often that people can do when they're faced with a disaster that they can do to keep themselves safe.
• 38:40
  SPEAKER_00: And generally speaking, especially when we're talking about terse messages that uh only allow you, for instance, 360 characters say we are, we don't have room to tell them everything that they could do to make themselves safe.
• 38:51
  SPEAKER_00: We have room for one thing.
• 38:52
  SPEAKER_00: And if we treat that as the only thing that they can do to make themselves safe, treat every other instantiation of that behavior as as a failure of the message, then we miss a lot of really legitimate things that people might do.
• 39:05
  SPEAKER_00: Um so first there's a there's a number of things they could do that might be more optimal than the thing that we're including in the message for them.
• 39:13
  SPEAKER_00: I think generally speaking, when we do an emergency communication, we're doing it over a pretty substantial geographic area.
• 39:20
  SPEAKER_00: And expecting that the behavior that we are trying to get people to do that we're including in guidance in those messages is the optimal behavior for every single person in that geographic area is probably unrealistic.
• 39:32
  SPEAKER_00: There's a lot of different topographic features, distance from the coast, availability of transportation, egress and ingress, those kinds of things can all kind of change what is the best way to make a person safe at that granular level that we can't reach yet with uh with messaging.
• 39:48
  SPEAKER_00: So there could be other things that they um could do to make themselves safer.
• 39:53
  SPEAKER_00: There's also a number of other considerations that might be driving their behavior that would lead them to not comply with what we're telling them to do, but that we really do still want them to do.
• 40:02
  SPEAKER_00: So if you're responsible for children or um elderly relatives, then we really wouldn't like it if you evacuate immediately.
• 40:11
  SPEAKER_00: We'd really like you to go take care of them, pick up your kids, pick up your elderly neighbors, and evacuate after that.
• 40:16
  SPEAKER_00: So thinking about in terms of when people take that action, they're not immediately complying, but they're doing things that we really want them to do and wouldn't want to treat as a message failure.
• 40:28
  SPEAKER_00: They're doing everything that they can in order to meet the spirit of the message and what we're asking them to do.
• 40:34
  SPEAKER_00: Um I think the the kind of final challenge with compliance is I guess taking a step back.
• 40:43
  SPEAKER_00: What we're trying to do in the research space, in the research that we're doing is we're trying to help improve message writing.
• 40:52
  SPEAKER_00: Basically, we want to give people the tools that they can use to write messages that are effective at getting people to make themselves safe.
• 40:59
  SPEAKER_00: And at the end of the day, when you're writing a message, what you can change is the contents of your message, the way that you write it, maybe the time that you send it out.
• 41:07
  SPEAKER_00: But what you can't change is people's behavior directly.
• 41:11
  SPEAKER_00: And so when we're thinking about whether a message is effective, successful, or whether it achieves its goals, if we focus on compliance, we're focusing on a metric that we can't control, that we can't change with the way that we adjust our messages.
• 41:24
  SPEAKER_00: There's a bunch of reasons that have nothing to do with our messages why people may or may not comply.
• 41:30
  SPEAKER_00: If we instead think about those motivations that give people, as far as we know, the best chance of taking some action to make themselves safe, then we're focusing on things that we can control within the message.
• 41:41
  SPEAKER_00: And so we'll be able to treat a message that doesn't motivate action as well as less successful than one that motivates action better.
• 41:51
  SPEAKER_00: And if we make ourselves agnostic to whether or not they take that action or whether or not they do exactly what we wanted them to do rather than something that just keeps them and their family safe, then we're giving ourselves the best chance at figuring out how to incrementally make messages better over time.
• 42:09
  SPEAKER_02: No, I think it makes a lot more sense and it it really avoids that trap because I have admittedly put myself in that trap before where I just look at the compliance piece and and just as you said, it's I'm I'm not giving appropriate um worth to all the different reasons and variables, things that that could have played into that.
• 42:30
  SPEAKER_02: And so my assessment is ultimately just wrong as to whether or not it was you know related or not.
• 42:37
  SPEAKER_02: So no, I I appreciate that.
• 42:39
  SPEAKER_01: Um I think that one of the things that we forget is when we're doing these kinds of message testing experiments, we're having we have somebody sitting at their computer in their home or in their office or on vacation, and they've logged in and they're doing this little test.
• 42:59
  SPEAKER_01: And they're concentrating entirely on, well, maybe hopefully entirely on the task of looking at the message and answering a bunch of questions.
• 43:07
  SPEAKER_01: And maybe they're interrupted briefly, but there's not a tornado that's bearing down on them and it's the middle of the day, but their kid is in school on the other side of town.
• 43:20
  SPEAKER_01: And those are two very different contexts, you know, like when we ask you in a in a non-hazardous situation to tell us what you think about a message versus we ask you in the hazardous situation.
• 43:33
  SPEAKER_01: Like we might tell you to go into your basement into an interior room because there's a tornado coming, and you in an experimental setting, you're like, oh yeah, no problem.
• 43:42
  SPEAKER_01: Of course I'm gonna do that.
• 43:43
  SPEAKER_01: That's the best thing for me.
• 43:44
  SPEAKER_01: Until you're like, wait.
• 43:48
  SPEAKER_01: My dog's outside, my kids on the other side of town.
• 43:51
  SPEAKER_01: I am not gonna think about my safety first.
• 43:54
  SPEAKER_01: I'm thinking about them.
• 43:55
  SPEAKER_01: And that is where I think that we get a little bit trapped on thinking it's.
• 44:00
  SPEAKER_01: It's either compliance or it's not, because that context matters so much.
• 44:05
  SPEAKER_01: And we can we cannot manipulate the context.
• 44:09
  SPEAKER_01: It would be unethical for us to put people into a tornado situation.
• 44:16
  SPEAKER_00: To be clear to everybody watching, no researcher has ever considered doing that.
• 44:20
  SPEAKER_00: Ever.
• 44:20
  SPEAKER_00: Never.
• 44:22
  SPEAKER_01: No, we have no.
• 44:24
  SPEAKER_01: And an institutional review board would shut it down.
• 44:27
  SPEAKER_01: I mean, I've heard of some kind of uh interesting experiments that have been done, but not putting people directly in danger.
• 44:35
  SPEAKER_01: At least not since the IRBs have been set up and they're like, no, we're not doing that thing anymore.
• 44:40
  SPEAKER_00: We had a couple of wild decades there uh uh a little while ago, but uh not in many years.
• 44:46
  SPEAKER_01: Right.
• 44:46
  SPEAKER_01: Right.
• 44:47
  SPEAKER_02: So Or could it be even pretending that there is an abduction so you can test a system with the community without telling them?
• 44:56
  SPEAKER_02: I mean I don't know.
• 44:57
  SPEAKER_02: Jeanette and I have heard of a couple doozies.
• 44:59
  SPEAKER_01: Don't yeah, yeah.
• 45:02
  SPEAKER_01: The last study that I saw published where it was um a surprise or um where people don't really know the context that they're doing something in.
• 45:13
  SPEAKER_01: And what's the name of it when they're doing well, you you're kind of disguising the purpose of the experiment when you're doing it.
• 45:20
  SPEAKER_01: So they put people into a room and they have people doing these activities, and then they set off an emergency alert, and it was for an active shooter.
• 45:29
  SPEAKER_01: But they didn't know that it was for an active shooter, and so they so they were testing people's responses to this active shooter message coming in.
• 45:36
  SPEAKER_01: I'm like, I cannot believe that the IRB approved that.
• 45:40
  SPEAKER_00: Did they?
• 45:41
  SPEAKER_01: Yeah, they published on it.
• 45:44
  SPEAKER_01: So that's not the kind of experiment my my my lab has ever conducted, nor would we ever conduct that kind of experiment.
• 45:51
  SPEAKER_01: It just is, you know, there's so many traumatic incidents out there already that we don't need to put people through trauma to to learn about how they would respond to a message.
• 46:03
  SPEAKER_02: And just to point that out, IRB, Jeanette, that's the governing body, right?
• 46:08
  SPEAKER_01: That's the institutional review board, which manages it that oversees human subjects research to make sure that when a social scientist or any kind of scientist is doing research with human beings or with animals, that you are doing research that is ethical and that you are protecting people who are vulnerable.
• 46:28
  SPEAKER_01: And you're also making sure that your research is equitable so that you're not just targeting one kind of population, but you're including everybody who can be included so we get results can be applied to a lot of different people.
• 46:41
  SPEAKER_01: So yeah, that's an institutional review board.
• 46:44
  SPEAKER_01: And we are bound to them from the design of a survey to the kind of flyer that's posted on social media to recruit people.
• 46:53
  SPEAKER_01: Like everything is reviewed by them so that we have approval before we start our research.
• 46:59
  SPEAKER_00: And I think just to just to insert, I mean, the there's a lot of of red tape there in terms of getting through processes and getting things approved.
• 47:09
  SPEAKER_00: And we're all familiar with the effect that red tape can have on the things that we do.
• 47:12
  SPEAKER_00: But at the end of the day, it's really trying to make sure that the research that we do uh to the extent that it opposes any risk or any harm on the people, and even if that harm is just it took up 15 minutes of their time, um, making sure that it's worth it.
• 47:28
  SPEAKER_00: The kind of potential outcomes from that research, it tells you something that you really need to know that warrants the burden that you're putting on people.
• 47:35
  SPEAKER_00: So for an experiment where you're setting off a fake uh kind of active shooter alert to just see how people respond.
• 47:45
  SPEAKER_00: I I don't even know how sure you would have to be that you're gonna learn something absolutely critical that's gonna help you immediately save lives.
• 47:52
  SPEAKER_00: Just the bar is so high for that kind of trauma.
• 47:55
  SPEAKER_00: I can't imagine any IRB looking at that and saying, like, yeah, sure, that seems like seems like it's worth it.
• 48:04
  SPEAKER_01: Yeah, it w it's pretty amazing.
• 48:08
  SPEAKER_00: I should I should watch what I say because I'll probably run into one of these people one day.
• 48:12
  SPEAKER_01: Uh I think I one it one of them might still be around, but I think another one of them is retired.
• 48:18
  SPEAKER_02: Okay.
• 48:19
  SPEAKER_02: So for those that have joined us in this last episode, um, this is just part one.
• 48:25
  SPEAKER_02: And the reason that we're breaking it down is so that you can consume all of it, right?
• 48:29
  SPEAKER_02: There's a lot of information, um, but I don't want you to be scared of it either.
• 48:34
  SPEAKER_02: Um, again, this is something that I am, and I know I I tease about it, other people will joke with me about it.
• 48:40
  SPEAKER_02: I'm I'm definitely learning about all this, but I'm telling you from the practitioner experience, this has helped me reshape programs um all around the nation because we're no longer relying on just as I said before, that hey, let me ask 10 of my friends if we're doing a good job, right?
• 49:00
  SPEAKER_02: There there is a right way to do this, and we need people who have been trained, who have the experience, and who know how to best review all the data that we're bringing in.
• 49:12
  SPEAKER_02: Because otherwise it's just it's just information on a shelf if we don't know how to actually apply it.
• 49:18
  SPEAKER_02: And then the whole thing that Hugh talked about, Jeanette, and I'll pass it after this, is I really liked it's it's taking information, but but doing it in a way that keeps your your experiment or your your ideas honest so that the information you're pulling from it can actually be considered it's it's true, it's it's real, it's not from your own ideas beforehand, right?
• 49:43
  SPEAKER_02: Um, so I don't know.
• 49:44
  SPEAKER_02: For me that that was important, and I I think you're absolutely right.
• 49:49
  SPEAKER_02: He needs to put his little three-minute spiel on qualitative data, quantitative data on YouTube, because I I think it was very well explained for someone like me.
• 50:02
  SPEAKER_01: Well, it was more than three minutes, but I love how how he brought it to um he made it really clear, I thought.
• 50:11
  SPEAKER_01: And so I truly appreciated that um quick lesson in qualitative and quantitative research.
• 50:17
  SPEAKER_01: But I think what I really want to impress on our listeners is the discussion around uh how we measure effectiveness and that it is the qualitative research, informing the quantitative research and we're not measuring the outcome that people are asking for most frequently, which is how do I get people to do what I tell them to do?
• 50:42
  SPEAKER_01: We're not measuring compliance, we're measuring instead what are the elements that we can adjust in a message to help to increase people's understanding and believing and personalizing so they can make a decision for themselves at the time that they are in danger and they can take the action that's best to put them into safety.
• 51:04
  SPEAKER_01: So um we will come back to this conversation in one of our next episodes because we're gonna continue talking with you.
• 51:13
  SPEAKER_01: We're gonna give people give people a little mind break, listen to this first episode, come back to episode two because we're gonna talk about the most recent publication from my lab that Hugh helped to design, collect the data, do the analysis.
• 51:31
  SPEAKER_01: And we're gonna talk about something called content mapping paper, and you're definitely gonna want to tune in for that.
• 51:38
  SPEAKER_01: So thanks everybody for listening, for bearing with us.
• 51:42
  SPEAKER_01: And back to you, Eddie, to sign us out.
• 51:45
  SPEAKER_02: Well, sign us off.
• 51:46
  SPEAKER_02: If after you're you've listened to this, I would talk to your any if you're in school doing anything, ask for credit.
• 51:53
  SPEAKER_02: Okay.
• 51:53
  SPEAKER_02: Because you literally had two amazing scholars talk to you for you know 45 minutes to an hour.
• 52:00
  SPEAKER_02: You should get some credit for you know going through this.
• 52:03
  SPEAKER_02: It should count as continuing education.
• 52:06
  SPEAKER_02: Um, but thank you for being a part of this, folks.
• 52:09
  SPEAKER_02: And again, there's a reason why we do this and why we talk about it.
• 52:13
  SPEAKER_02: I promise you, it will make you better, it will make your program better and your community safer.
• 52:17
  SPEAKER_02: Um, this is a story that we need to keep talking about, and that's why there's going to be a part two.
• 52:23
  SPEAKER_02: So, this is the alerting authority where every second has a story.
• 52:27
  SPEAKER_02: And thank you for listening to part one of this.