Latest / Paths of Curiosity in an AI World / Q3 - AI in Data, Behavior, and Human Systems with Berkeley Dietvorst, BS, PhD

Transcript

Q3 - AI in Data, Behavior, and Human Systems with Berkeley Dietvorst, BS, PhD · Paths of Curiosity in an AI World

• 0:04
  SPEAKER_01: Welcome to passive curiosity in an AI world where we explore the careers through the lens of AI.
• 0:13
  SPEAKER_01: What if the algorithm was actually right and you still refuse to trust it?
• 0:20
  SPEAKER_01: Dr.
• 0:20
  SPEAKER_01: Berkeley Dietvorst discovered that people abandon algorithms the moment they see them make a mistake, even when those algorithms consistently outperform humans.
• 0:32
  SPEAKER_01: Let's get curious.
• 0:39
  SPEAKER_01: Berkeley Dietvorst is a professor of marketing at the University of Chicago Booth School of Business.
• 0:46
  SPEAKER_01: He coined the term algorithm adversion, and his follow-up research found that giving people just a tiny amount of control over an algorithm's output is often enough to fix it.
• 0:59
  SPEAKER_01: His work in management science finds that people prioritize being right over minimizing mistakes, even when the math says otherwise.
• 1:08
  SPEAKER_01: He holds a BS in economics and a PhD in decision processes, both from the Wharton School at the University of Pennsylvania.
• 1:17
  SPEAKER_01: Dr.
• 1:18
  SPEAKER_01: Diavorst, thank you so much for joining us.
• 1:22
  SPEAKER_00: Absolutely.
• 1:23
  SPEAKER_00: Thank you for having me.
• 1:24
  SPEAKER_01: You coined the term algorithm aversion.
• 1:28
  SPEAKER_01: What is it and why should someone even care?
• 1:32
  SPEAKER_00: That's a good question.
• 1:34
  SPEAKER_00: So algorithm aversion is a tendency that people sometimes have when they're deciding whether or not to use an algorithm to make a prediction.
• 1:44
  SPEAKER_00: So, for example, you might use an algorithm to predict what the weather's going to be, uh how long it takes to drive from point A to point B, etc.
• 1:56
  SPEAKER_00: And algorithm aversion would be a phenomenon where the algorithm is actually a little better at making that prediction than the human alternative.
• 2:09
  SPEAKER_00: But we still choose to use the human instead because the algorithm doesn't feel like it's good enough for to use it.
• 2:18
  SPEAKER_00: It doesn't live up to our standards, and so we choose not to use the algorithm, even though um one could argue that it's actually the better option because it performs better.
• 2:31
  SPEAKER_01: I can't wait to dive deeper into that.
• 2:34
  SPEAKER_01: But first, how would you describe what you do today to your student self?
• 2:41
  SPEAKER_00: That's a tricky one because yeah, I have a very unique kind of career.
• 2:45
  SPEAKER_00: So what I do, uh, I'm a professor.
• 2:49
  SPEAKER_00: So a lot of my work is doing research, and I also teach a little bit.
• 2:54
  SPEAKER_00: Um, by research, I really study human decision making.
• 2:58
  SPEAKER_00: So what I want to try to figure out is how people make decisions from multiple perspectives.
• 3:05
  SPEAKER_00: So I really want to understand from economic perspectives, from decision scientific perspectives, and from psychology how people make decisions, how people arrive at the choices they make.
• 3:17
  SPEAKER_00: And so, broadly speaking, that's what I study.
• 3:20
  SPEAKER_00: And I'm often applying that to consumers making decisions, like buying things, but really broadly just understanding how people choose what they choose.
• 3:32
  SPEAKER_01: Walk us through your path and how you ended up studying when people trust algorithms.
• 3:39
  SPEAKER_00: Yeah, that's a good question.
• 3:41
  SPEAKER_00: So when I was an undergrad, I was actually a finance major first.
• 3:46
  SPEAKER_00: So I thought I was going to go into finance, and what I was really interested in was investing.
• 3:52
  SPEAKER_00: So I've always kind of been interested in prediction.
• 3:56
  SPEAKER_00: What I thought was really cool about investing was that you'd be able to predict which investments are going to go up in value and which are going to go down, and that would help you decide what to invest in, how to make these investments.
• 4:09
  SPEAKER_00: And so I studied that for a while and I really liked it.
• 4:13
  SPEAKER_00: And then just by chance, I took some classes on human decision making.
• 4:18
  SPEAKER_00: And so I started to learn that actually what I'm really interested in is how humans make predictions and how to understand what's going on in our heads and the process by which we all arrive at a prediction.
• 4:32
  SPEAKER_00: And so I kind of made a pivot where instead of trying to make the predictions myself, doing finance and studying investing, I said, you know what, I'm going to try to figure out in general how people make predictions.
• 4:44
  SPEAKER_00: And one of the coolest problems that I ran into early in my PhD program was what about all the tools that people can use to make predictions?
• 4:54
  SPEAKER_00: So there was this really old literature that was interesting, kind of started by people like Paul Mule and Robin Dawes, where they were showing that even really simple statistical tools could often make predictions better than humans.
• 5:12
  SPEAKER_00: So we had all these cases where a really simple statistical tool was better at diagnosing patients than a doctor, or a really simple statistical tool was better than an admissions officer at predicting what student was going to perform really well in a program.
• 5:29
  SPEAKER_00: And it was kind of fascinating because a lot of these tools were just so simple.
• 5:35
  SPEAKER_00: And how could this really simple tool be outperforming people?
• 5:39
  SPEAKER_00: And so that itself was interesting.
• 5:41
  SPEAKER_00: But then what was even more interesting was people's reaction.
• 5:44
  SPEAKER_00: So Paul Meal and Robin Dawes and other people who worked on this literature took this to their colleagues and said, all right, we've done so many studies and these algorithms are better.
• 5:56
  SPEAKER_00: They actually called them mechanical predictors back then, but we would call them algorithms.
• 6:01
  SPEAKER_00: Uh let's start using them.
• 6:03
  SPEAKER_00: They're big performance, they offer big performance advantages.
• 6:06
  SPEAKER_00: That's obviously what we should be doing.
• 6:08
  SPEAKER_00: And all of the colleagues were actually really resistant to it.
• 6:12
  SPEAKER_00: People didn't want to use algorithms to decide what students to admit, and they didn't want to use algorithms to decide what a diagnosis should be, or all those types of things.
• 6:22
  SPEAKER_00: And so that kind of piqued my curiosity because we had these simple tools that performed really well, which in itself is interesting, and then also people didn't really want to use them.
• 6:34
  SPEAKER_00: And so that's kind of where I picked up the thread and started doing research is seeing, okay, well, we know these tools are pretty good, but people don't want to be using them.
• 6:45
  SPEAKER_00: Let's study that psychology of why don't want to be why don't people want to use these prediction tools?
• 6:51
  SPEAKER_00: And how are people making this decision if they're not using the better option?
• 6:56
  SPEAKER_00: And so that's kind of how I got to the what I researched today.
• 7:01
  SPEAKER_01: Could you walk us through the original algorithm aversion experiment, which you asked people to do, and what surprised you the most about the results?
• 7:13
  SPEAKER_00: Yeah, so those original studies were interesting.
• 7:16
  SPEAKER_00: When we actually were designing the studies, we didn't anticipate what would happen.
• 7:21
  SPEAKER_00: We thought we would find something else.
• 7:23
  SPEAKER_00: So essentially, the way those studies worked, they're very simple.
• 7:28
  SPEAKER_00: You're going to make some predictions, and the better you do, the more money you earn.
• 7:33
  SPEAKER_00: So we had participants in this scenario where they should really care about how good their predictions are, because the better their predictions are, the more money they get.
• 7:41
  SPEAKER_00: And people are showing up to earn money, so they should really want to do well and earn that money.
• 7:46
  SPEAKER_00: So the way we set them up is before you make those predictions that count for money, you're gonna go through some practice rounds just to practice.
• 7:55
  SPEAKER_00: And so you get to practice a little bit before it's time to earn the money.
• 7:59
  SPEAKER_00: Now, in that practice, that's where we actually did our experiment and manipulated what people did.
• 8:06
  SPEAKER_00: So some participants just saw an algorithm make predictions and that was their practice.
• 8:12
  SPEAKER_00: Other participants only made the predictions themselves, and that was their practice.
• 8:18
  SPEAKER_00: A third group of participants did both.
• 8:20
  SPEAKER_00: They saw the algorithm perform and they made the prediction themselves.
• 8:24
  SPEAKER_00: And a fourth group of participants just kind of skipped the practice and didn't really do anything.
• 8:30
  SPEAKER_00: Then after that practice round, people made their choice.
• 8:34
  SPEAKER_00: Do you want to make predictions yourself or rely on the algorithm to make the predictions that count for the bonus and are going to earn your money?
• 8:43
  SPEAKER_00: Now, what we thought would happen is people who haven't seen this algorithm perform might not think it does well and might not trust it.
• 8:52
  SPEAKER_00: But once people see that it performs really well, then they'll use it because they'll see that it does pretty well and it earns more money than they do, and so then they'll choose to use it.
• 9:03
  SPEAKER_00: And what we found was almost exactly the opposite, where people who hadn't seen the algorithm perform yet actually often used it.
• 9:11
  SPEAKER_00: They were kind of assuming that it must have this down to a science, it must perform really well.
• 9:16
  SPEAKER_00: And so they expected the algorithm to do really well.
• 9:19
  SPEAKER_00: And so when they hadn't seen it perform, they were kind of using it, saying, I bet it's better than me, I bet it's pretty good.
• 9:26
  SPEAKER_00: On the other hand, people who did see the algorithm perform chose not to use it very often.
• 9:31
  SPEAKER_00: And what happened was even though the algorithm was better than people, it still made tons of mistakes all the time, right?
• 9:39
  SPEAKER_00: So, for example, if it was making 20 predictions, it would only get a couple completely right, and another handful would be close to right, and all the rest would be way off.
• 9:51
  SPEAKER_00: And it turns out that's way better than the participants did themselves, and that would earn more money than they would, but it just didn't look very good.
• 9:59
  SPEAKER_00: And so people, once they saw that's how well the algorithm performed, they just decided I don't want to use that, that doesn't look good enough.
• 10:09
  SPEAKER_01: That is so interesting.
• 10:11
  SPEAKER_01: Giving people just a tiny ability to tweak an algorithm output makes them more willing to use it.
• 10:19
  SPEAKER_01: Like, why do small tweaks matter so much psychologically?
• 10:26
  SPEAKER_00: Yeah, that's a great question.
• 10:27
  SPEAKER_00: This is follow-up research we did after that initial research.
• 10:32
  SPEAKER_00: So, in that initial research, we were finding, yeah, even though these algorithms are better, people aren't using them.
• 10:38
  SPEAKER_00: And so, one of the next questions we had is what are some interventions we can implement to try to get people to use those algorithms more?
• 10:45
  SPEAKER_00: Because they do perform better and they earn more money, so it would make sense for people to use them in some scenarios.
• 10:52
  SPEAKER_00: Um, one of the things we found is really effective, just like you said, is just giving people the ability to tweak the algorithm's output a little bit or overrule some of its predictions.
• 11:03
  SPEAKER_00: So they're kind of injecting some of their own judgment into the process instead of only relying on the algorithm.
• 11:10
  SPEAKER_00: Now, one reason we think that's really effective is because once people have learned that the algorithm makes a lot of mistakes and it's far from perfect, just adopting it wholesale without having any ability to do anything makes you feel like you're adopting the system that's inevitably going to be wrong a lot.
• 11:31
  SPEAKER_00: So you're kind of almost giving in and saying, all right, I'm okay with being wrong a lot.
• 11:36
  SPEAKER_00: And people don't like that.
• 11:38
  SPEAKER_00: People don't like adopting a system where they know a lot of the time it's going to be wrong, and I have to be okay with that.
• 11:46
  SPEAKER_00: And so when you have just a little bit of your own judgment and you can change the predictions, it kind of gives you this feeling of I'm not accepting that I'm going to be wrong every time.
• 11:58
  SPEAKER_00: Maybe if I intervene and maybe if I do my best, I can make it better, and I don't have to be wrong all the time.
• 12:07
  SPEAKER_01: So people look at it like a partnership.
• 12:10
  SPEAKER_01: The algorithm brings its strength, you bring yours, and together you get more than either could alone.
• 12:18
  SPEAKER_00: Yeah, or even it's just it'd be a very similar psychological thing of imagine you try something out, it doesn't go perfect.
• 12:26
  SPEAKER_00: People often then want to try something else or change things a little bit.
• 12:31
  SPEAKER_00: You wouldn't go do that same exact thing because you just learned that it's it isn't perfect.
• 12:37
  SPEAKER_00: So it's kind of natural for people when they try something, it makes a lot of mistakes to say, well, I have to change it because I don't want to accept those mistakes.
• 12:52
  SPEAKER_01: Say I'm a student, and an AI college counselor just told me not to apply to my dream school.
• 12:59
  SPEAKER_01: The algorithm has a better track record than any human counselor.
• 13:03
  SPEAKER_01: What does your research say I should do?
• 13:08
  SPEAKER_00: This is a really interesting question because there's kind of a hidden leap in the question that people often make that's really important to call out.
• 13:17
  SPEAKER_00: So, what I would say is that human, that AI, sorry, the AI counselor might be really way better than humans at predicting your likelihood of getting into your dream school.
• 13:30
  SPEAKER_00: But the decision is yours.
• 13:32
  SPEAKER_00: It can't make a decision, that has to be you.
• 13:35
  SPEAKER_00: So all of my research is about predictive algorithms and us using their predictions.
• 13:43
  SPEAKER_00: But what's really important to remember is that a prediction is just part of a decision.
• 13:47
  SPEAKER_00: So just knowing a prediction doesn't tell us a decision yet.
• 13:52
  SPEAKER_00: We need to combine that with our values and what we want to do.
• 13:57
  SPEAKER_00: So, as an example, imagine that you look up on a weather app that you trust that is really good and it says there's a 20% chance of rain during a concert that you really want to go to.
• 14:11
  SPEAKER_00: Um, and the concert's outside, so you would get rained on if it rained.
• 14:16
  SPEAKER_00: That might be the best prediction we could make.
• 14:18
  SPEAKER_00: That 20% chance of rain might be perfect, but it doesn't say what the decision should be.
• 14:23
  SPEAKER_00: That has to be up to you.
• 14:25
  SPEAKER_00: Only you know how much do you dislike getting wet?
• 14:29
  SPEAKER_00: How much do you care about the concert?
• 14:31
  SPEAKER_00: How much fun would you have if you did get wet, or how disappointed would you be if you stayed home and it didn't rain?
• 14:39
  SPEAKER_00: All of those things are your value and your utility for these different outcomes.
• 14:44
  SPEAKER_00: And so the way we need to make decisions is I think we need to be honest with ourselves when algorithms are better at making predictions.
• 14:52
  SPEAKER_00: So, for example, for weather, I just ask my trusted weather algorithm that I use because it knows better than me how likely it is to rain.
• 15:01
  SPEAKER_00: If I just come up with my own predictions, I'm gonna be way worse off.
• 15:05
  SPEAKER_00: But those predictions are just part of the decision.
• 15:07
  SPEAKER_00: Then I need to integrate that prediction with how I feel about different outcomes.
• 15:12
  SPEAKER_00: So now applying this to your college example, an AI counselor in theory could tell you here's what I believe your probability of getting into your dream school is if you apply.
• 15:24
  SPEAKER_00: But you're the one that needs to decide what to do with that.
• 15:28
  SPEAKER_00: So it depends on a lot of things.
• 15:31
  SPEAKER_00: You know, how costly is it for you to apply?
• 15:33
  SPEAKER_00: Applying to a school takes time.
• 15:36
  SPEAKER_00: I think sometimes it takes money, you have to like submit money with the application.
• 15:40
  SPEAKER_00: So only you know how big a deal that is.
• 15:43
  SPEAKER_00: Also, only you know how much better is your dream school than your second best school.
• 15:49
  SPEAKER_00: Is it 20 times better than your second best school in your mind, or are they almost tied?
• 15:54
  SPEAKER_00: And so you need to use all that information to make a decision with that probability.
• 15:59
  SPEAKER_00: And you can imagine there's some people where even if they only had a 0.01% chance of getting in their dream school, it would still be worth it for them to apply because it would be so great if they got in.
• 16:11
  SPEAKER_00: And they would really regret it if they didn't.
• 16:14
  SPEAKER_00: Maybe someone else, they might say, Well, if I have less than a 20% chance of getting in, I'm not even going to apply.
• 16:21
  SPEAKER_00: Because they just have different values and they feel differently about it.
• 16:25
  SPEAKER_00: So I think this is kind of the mindset we need with these predictive algorithms.
• 16:29
  SPEAKER_00: We need to be kind of open-minded when they're better than us at making predictions, and we need to understand that if we use their predictions, we're going to be better off.
• 16:38
  SPEAKER_00: But the decision still rests with us because we're the only ones that know how good are different outcomes and how happy or sad we'd be in different cases.
• 16:50
  SPEAKER_01: I really like the way you put that hiring, parole, medical diagnosis, domains that feel right for resisting algorithms to most people.
• 17:02
  SPEAKER_01: But is that resistance justified?
• 17:08
  SPEAKER_00: Um I don't know.
• 17:10
  SPEAKER_00: I think I don't take a stance on that.
• 17:13
  SPEAKER_00: Um what I would say is two answers.
• 17:18
  SPEAKER_00: One, going back to what we just talked about with decisions, people might be objecting to these algorithms making decisions, which I think is perfectly fair.
• 17:28
  SPEAKER_00: Right?
• 17:29
  SPEAKER_00: Because to make a decision, like we just talked about, you have to say how good and bad are different outcomes.
• 17:36
  SPEAKER_00: And who's an algorithm to say how good or bad it is to put an innocent person in jail or let someone who's not innocent go free and all that stuff.
• 17:45
  SPEAKER_00: Those are things I think we need to decide as humans.
• 17:48
  SPEAKER_00: And so, just like what we just talked about with the AI counselor or the concert, we need to test these systems and see are they good at making predictions?
• 17:58
  SPEAKER_00: Maybe some of them aren't even good at making predictions.
• 18:00
  SPEAKER_00: Then there's no reason to use it.
• 18:02
  SPEAKER_00: Right?
• 18:02
  SPEAKER_00: If we audit the system and it can't make good predictions, there's no point using it.
• 18:07
  SPEAKER_00: We might as well get rid of it.
• 18:09
  SPEAKER_00: If it can make good predictions, then we still need to ask ourselves, well, what are we going to do with that prediction?
• 18:17
  SPEAKER_00: And that's really a human decision because it depends on how good and bad different outcomes are.
• 18:22
  SPEAKER_00: So I would say, one, in order to use these systems, we should say, prove that you perform well.
• 18:29
  SPEAKER_00: If we don't have any track record and we don't know that they perform well at predicting, we don't have to use them.
• 18:35
  SPEAKER_00: Second, even if they're good at predicting, it doesn't say what the decision should be.
• 18:40
  SPEAKER_00: That's really a human matter.
• 18:42
  SPEAKER_00: And humans should be part of that process saying how you should use that probability to make decisions.
• 18:48
  SPEAKER_00: Right?
• 18:48
  SPEAKER_00: So, for example, no matter how good an algorithm would be at predicting whether someone would commit a crime in the future, my belief is you could never punish someone if they haven't committed the crime yet.
• 19:04
  SPEAKER_00: Right?
• 19:04
  SPEAKER_00: Maybe that's just my morals, and that's what I think is right.
• 19:07
  SPEAKER_00: And so just because an algorithm would be really good at predicting that thing doesn't mean that we have to use it.
• 19:13
  SPEAKER_00: And it doesn't mean that it's right to go take that prediction and do something with it.
• 19:18
  SPEAKER_00: That we really need to think about that as humans.
• 19:22
  SPEAKER_01: So algorithms for predictions and humans for the decisions.
• 19:28
  SPEAKER_00: Algorithms for the predictions and humans for the decisions.
• 19:31
  SPEAKER_00: So, in a lot of cases, the algorithm is just gonna be better than us at making predictions.
• 19:38
  SPEAKER_00: Now we shouldn't take that for granted.
• 19:39
  SPEAKER_00: We should tell the system, you know, prove it, let's let's take it to some data and make sure it actually is good, right?
• 19:46
  SPEAKER_00: Because it is easy to build a really bad algorithm.
• 19:49
  SPEAKER_00: We could build a ridiculous algorithm that incorporates all kinds of stuff that doesn't make sense and it makes really bad predictions.
• 19:56
  SPEAKER_00: So, but uh usually an algorithm that is based on evidence, that uses good data, that someone's done a good job building it, it often will make really good predictions.
• 20:06
  SPEAKER_00: But predictions are just one part of a decision.
• 20:09
  SPEAKER_00: Predictions don't tell us to do what on their own.
• 20:13
  SPEAKER_01: A lot of our listeners have grown up being sorted by algorithms their whole lives.
• 20:18
  SPEAKER_01: Does this exposure change how young people make decisions?
• 20:25
  SPEAKER_00: Um, that's a great question.
• 20:27
  SPEAKER_00: I think first off, that's really hard to study, so I don't think we know.
• 20:33
  SPEAKER_00: One of the big problems is as these changes have been happening, a million other things have been changing as well.
• 20:41
  SPEAKER_00: Right?
• 20:42
  SPEAKER_00: There's a bunch of changes in the world that happen all at once, and this development of these systems is just one of them.
• 20:48
  SPEAKER_00: And so it's really hard to separate out what has been the effect of this change versus all the other changes.
• 20:55
  SPEAKER_00: And so that's just what makes it a really hard question to study academically, because many things changed at once.
• 21:01
  SPEAKER_00: That's a classic case where it's really hard to say what happened.
• 21:06
  SPEAKER_00: Another thing I would say is that the types of systems making predictions have definitely changed and advanced, but it's actually been a really long time that we've had equations making predictions that affect us.
• 21:22
  SPEAKER_00: So, for example, the equations that would give people credit ratings that would decide like, can you get a loan for a house or could you get a credit card?
• 21:31
  SPEAKER_00: Those have been kind of algorithmic predictions for a long time.
• 21:34
  SPEAKER_00: And so, my whole life, that's been how we've made those predictions.
• 21:40
  SPEAKER_00: I think definitely something that's changing is we're using different types of systems and they might be getting more advanced.
• 21:48
  SPEAKER_00: But I think for a while, in many domains, we've been relying on these systems.
• 21:52
  SPEAKER_00: I would guess it does have many effects, but I don't know what they are because it's so hard to study.
• 21:58
  SPEAKER_00: And so I think if you can find clever.
• 22:00
  SPEAKER_00: Ways to study, you know, one individual change, that's a really good research question.
• 22:06
  SPEAKER_00: But it's really hard to say when things evolve over time what the effect is because so many things change at once.
• 22:14
  SPEAKER_01: Your most recent research looks at when people adapt predictive AI in their real life.
• 22:20
  SPEAKER_01: What was the key finding and why does it matter for how AI tools get built?
• 22:28
  SPEAKER_00: Yeah, that's a great question.
• 22:30
  SPEAKER_00: Um a lot of my most research is yeah, building theories about how people adopt these tools.
• 22:37
  SPEAKER_00: One, probably if I had to say what the biggest factor in my theory is, it would be that how predictable the domain is really matters.
• 22:46
  SPEAKER_00: So let me describe what that means.
• 22:49
  SPEAKER_00: There's some domains that are what we call deterministic.
• 22:53
  SPEAKER_00: That means that they're perfectly predictable.
• 22:56
  SPEAKER_00: You, if you were really smart, you could always get it right.
• 23:00
  SPEAKER_00: So this would be something like solving a math problem, right?
• 23:03
  SPEAKER_00: If you're a calculator or you're really good at math, I could give you a bunch of addition problems and you could get them all right.
• 23:10
  SPEAKER_00: In those types of domains, people are really willing to use algorithms because if they're built well, they can kind of always be right.
• 23:19
  SPEAKER_00: There's other domains that I would call random domains.
• 23:22
  SPEAKER_00: Those are domains where no matter how smart you are, you can just you just can't always be right because randomness is determining outcomes and you just can't predict randomness.
• 23:35
  SPEAKER_00: So one good example would be predicting a die roll.
• 23:39
  SPEAKER_00: Right?
• 23:39
  SPEAKER_00: No matter how smart we are, if there's kind of a fair die roll, we just can't predict what the die is going to land on.
• 23:47
  SPEAKER_00: And what that would mean is now take any game that involves die rolls, like think about games you play in Las Vegas or something.
• 23:55
  SPEAKER_00: You can't always be perfect because you can't predict what the die roll is going to be.
• 24:01
  SPEAKER_00: And what I find is that there's a lot of domains like this in life, right?
• 24:06
  SPEAKER_00: So the stock market's pretty random, to some extent, weather's random, lots of things we can't predict them perfectly, even if we try, just because there's some randomness that's baked in, at least with our current knowledge, we don't think it's perfectly predictable, and we can't do it with our skills and our information right now.
• 24:26
  SPEAKER_00: So those are the domains where even the best performing algorithm is going to be wrong a lot because it just can't predict the die rolls, it can't predict all the randomness.
• 24:38
  SPEAKER_00: Now that gets tricky because they're the exact domains where, even though algorithms are often not going to be perfect, they're often better than the humans.
• 24:48
  SPEAKER_00: And that's where people kind of fall into this trap.
• 24:51
  SPEAKER_00: Um, where it could be the case that we see the algorithm perform, it makes the tons of mistakes, and we say, I'm not really sure I want to use that.
• 25:01
  SPEAKER_00: But it turns out it actually performs 20% better than we do.
• 25:05
  SPEAKER_00: And maybe from that perspective, we're missing out by not using it.
• 25:10
  SPEAKER_00: So I would say the biggest factor is that domains really differ in how random they are.
• 25:16
  SPEAKER_00: Lots of domains are pretty deterministic, and we're really comfortable using algorithms in those domains, right?
• 25:23
  SPEAKER_00: So people are actually pretty comfortable using algorithms to do things like navigation now.
• 25:30
  SPEAKER_00: Right?
• 25:30
  SPEAKER_00: Algorithms can almost always get you from point A to point B in your car.
• 25:34
  SPEAKER_00: Uh, every once in a while they'll make a mistake, but you know that in a new town you've never been to, they'll be better than you.
• 25:41
  SPEAKER_00: But in other domains, there's just too much randomness.
• 25:44
  SPEAKER_00: And if I have an algorithm, you know, predicting what stock prices will be in two months, it's gonna be wrong a lot because it's just impossible to get that perfect.
• 25:54
  SPEAKER_00: And those are the domains where we really risk maybe passing up an algorithm that is better than us, even though we've seen it make a lot of mistakes.
• 26:04
  SPEAKER_01: You've now built a research reputation that shapes how companies think about AI.
• 26:10
  SPEAKER_01: What skills did you have to develop first?
• 26:16
  SPEAKER_00: First off, I'm not sure I have that much influence, but I that's nice of you to say.
• 26:20
  SPEAKER_00: I really don't think so.
• 26:21
  SPEAKER_00: But I think the skills I built up first really just kind of the fundamentals.
• 26:27
  SPEAKER_00: So when I was uh in high school, I actually that's when I took my first statistics and economics and psychology class.
• 26:39
  SPEAKER_00: So even in high school, I kind of got introduced to these topics of statistics, economics, and psychology.
• 26:47
  SPEAKER_00: And then I continued that in college and just kind of built up my understanding of those basic fields that I need an understanding of to do the research I do today.
• 26:59
  SPEAKER_00: So I think the most important thing was yeah, recognizing early on what was really interesting to me, and then taking more of those classes and just building up skills, building up knowledge.
• 27:13
  SPEAKER_00: And I think that's often the best way to kind of guide yourself to what you're really interested in.
• 27:19
  SPEAKER_00: If you just do what's interesting and what you're really excited about, you're just gonna keep going on that path and learning the things that you need to learn to do what you find exciting in a lot of cases.
• 27:33
  SPEAKER_01: Decision processing isn't a major most students have heard of.
• 27:37
  SPEAKER_01: For a listener drawn to psychology of decision making, what would you tell them about turning that into a career?
• 27:45
  SPEAKER_00: Yeah, decision processes is a really interesting field because it does integrate a lot of those things I just talked about.
• 27:53
  SPEAKER_00: So, what decision processes does is it looks at decision making from multiple different perspectives.
• 28:02
  SPEAKER_00: So it looks at what we call normative decision making, which means like in theory, kind of the right way to make decisions.
• 28:09
  SPEAKER_00: Now I put quotes around that because there's really no right or wrong way to make decisions, but it's saying, like, from the perspective of economics or from the perspective of some of these theories of how like a rational decision maker would act, what do those theories say the right way to make a decision is?
• 28:25
  SPEAKER_00: And so we try to understand that the normative side of kind of what's the in theory the right way to make decisions.
• 28:32
  SPEAKER_00: Then we also look at descriptive stuff, which is more like from psychology, of how do people actually make decisions?
• 28:39
  SPEAKER_00: And in a lot of cases, there's a contrast between those, right?
• 28:43
  SPEAKER_00: So, kind of the normative way of what we would say, what an economic what an economist would say is the rational way to make a decision is different from the description of how lay people are actually doing it.
• 28:54
  SPEAKER_00: And then we have a third kind of decision making we look at, which is called prescriptive decision making, where we say, what are some interventions we could use to try to get those lay people to look more like the normative people that economics talks about?
• 29:09
  SPEAKER_00: So you have this normative kind of in theory, what decisions should look like, descriptive, how are people actually doing it, and then prescriptive, are there ways that we can get those lay people to look more like the rational decision makers that economics says they might be?
• 29:29
  SPEAKER_00: And so I think to the best start for learning about that field, if it sounds interesting, is just kind of like learning about those core disciplines, right?
• 29:39
  SPEAKER_00: So taking an econ class would be helpful, taking a statistics class would be helpful, taking a psychology class would be helpful.
• 29:47
  SPEAKER_01: How do you see AI changing the field of behavioral science over the next 10 to 20 years?
• 29:54
  SPEAKER_00: That's a really good question.
• 29:56
  SPEAKER_00: I think everyone is asking that question, and no one quite knows the answer.
• 30:00
  SPEAKER_00: Um, I think it's definitely going to change things.
• 30:02
  SPEAKER_00: I don't know exactly how.
• 30:04
  SPEAKER_00: I already see one change that's happening and that it's making things accessible that weren't accessible before.
• 30:12
  SPEAKER_00: So now it's getting more and more the case that if you can think up a study to run, AI can help you build it.
• 30:21
  SPEAKER_00: So we used to be really constrained where if you didn't have programming skills and you're using some survey software, you would kind of only be able to do the things you knew how to program or the things you could do in that survey software.
• 30:37
  SPEAKER_00: Where now with AI, people are able to do all kinds of new things because AI can help them kind of build whatever they can imagine in terms of a study.
• 30:46
  SPEAKER_00: So that's kind of exciting because it means now you can study a lot of things you couldn't study before because there's a tool that can help you figure out how to build the study you'd need to build.
• 30:58
  SPEAKER_00: That's definitely one change I've seen.
• 31:01
  SPEAKER_00: Um I'm guessing it's also going to help people with some technical skills that they might have felt like were out of their reach before, but AI can both be like a really good tutor talking back and forth to understand something and um kind of building those skills.
• 31:20
  SPEAKER_00: So I would predict that those might be two changes of like we just are less constrained in what we can do because we can use these tools to do stuff that we wouldn't have found possible.
• 31:30
  SPEAKER_00: And I'm sure there's gonna be many more changes that I have no clue what they're gonna be.
• 31:36
  SPEAKER_01: What is one concrete skill a student can do in the next year to start building towards that kind of work?
• 31:44
  SPEAKER_00: That's a good question.
• 31:45
  SPEAKER_00: So if you're really interested in this kind of work specifically, um, I already mentioned, you know, take an econ class, take a psychology class, take a statistics class.
• 31:55
  SPEAKER_00: One thing that I would challenge students to do, pick out a couple academic papers that sound interesting and challenge yourself to actually go read like an academic paper.
• 32:07
  SPEAKER_00: So it's actually really different than reading a book where people are summarizing research or reading an article online where they kind of cite research.
• 32:17
  SPEAKER_00: Reading the research itself is really different, and it gives you a window into how the research was done and all of the details, and just seeing that and understanding better how the research is done will be really helpful.
• 32:32
  SPEAKER_00: And you'll also be building a skill of you know learning how to read and think about academic papers.
• 32:37
  SPEAKER_00: And so I think if you're really serious and you really want to see how interested am I in this and what are some skills I could build to get better, actually going to the academic papers themselves and trying to understand and read them would be something that I think would be really valuable.
• 32:56
  SPEAKER_01: What is something your work has required that most people would never guess?
• 33:04
  SPEAKER_00: That's a good question.
• 33:06
  SPEAKER_00: I think one thing I realized early on.
• 33:10
  SPEAKER_00: So the to get where I am, first you do a PhD.
• 33:14
  SPEAKER_00: So you're a PhD student, and then you get to be kind of a professor somewhere, hopefully, if you get a job.
• 33:22
  SPEAKER_00: For both of those positions, there is absolutely nobody telling you what to do every day.
• 33:29
  SPEAKER_00: There's no one monitoring you, there's no one saying, if you want to end up with this paper in a year, here's all the steps you need to take.
• 33:39
  SPEAKER_00: There's no one telling you how to manage your time.
• 33:42
  SPEAKER_00: And so, one big skill you need to learn to be successful is just kind of managing your own time and figuring out how to take some really big goal, like having a paper or finishing a project, and break it down into all these little pieces of progress you need to make along the way, and make sure that every day you're just kind of taking one step forward and making a little bit of progress.
• 34:07
  SPEAKER_00: So it's often really hard and daunting to kind of have this big goal in front of you that you want to accomplish, but not know exactly what the next thing to do is, and not know at first how to manage your time and how to pursue it.
• 34:23
  SPEAKER_00: So, definitely one of the first things you have to learn is how can you make sure you're using your time well and kind of break these big long-term goals down into little steps you can take every day to make progress.
• 34:40
  SPEAKER_01: And finally, you mentioned people should read journals and those types of articles.
• 34:46
  SPEAKER_01: Are there any specific books, podcasts, or other media you would recommend to a student who wants to learn more?
• 34:55
  SPEAKER_00: Yeah, so I think maybe I'll just stick with what I was saying.
• 34:59
  SPEAKER_00: I'm gonna recommend a couple academic papers that I think are interesting, where if you're interested in the kind of questions I was talking about in that stream of research, maybe those papers will be interesting.
• 35:12
  SPEAKER_00: They'll at least be fun to try to read, and might you might learn that it's fascinating, or you might learn that it you don't really like it.
• 35:19
  SPEAKER_00: So, two of the papers that I think are good to start out with that are um on this topic.
• 35:26
  SPEAKER_00: One is called Accepting Air to Make Less Air.
• 35:30
  SPEAKER_00: It's by a scholar who is named Hillel Einhorn.
• 35:35
  SPEAKER_00: It was published in 1986 in the Journal of Personality Assessment.
• 35:39
  SPEAKER_00: And I think it's a really good article because it talks about one of these problems I was talking about, which is some domains are really predictable and deterministic, other domains are really random, and how do we wrestle with that?
• 35:53
  SPEAKER_00: How do we try to make good predictions in these random domains?
• 35:56
  SPEAKER_00: And it's just kind of a thought piece, and I think it'll be really interesting to read if you're interested in these topics.
• 36:03
  SPEAKER_00: Another good paper is called The Robust Beauty of Improper Linear Models and Decision Making.
• 36:09
  SPEAKER_00: That one's by Robin Dawes.
• 36:12
  SPEAKER_00: Uh, that was published in 1979 in American Psychologist, and that's a really cool article that I really liked.
• 36:18
  SPEAKER_00: It was my favorite paper as a PhD student because it's one of those that was showing even these really simple models that we know are imperfect and are just ultra simplistic, often make better predictions than humans do, even experts.
• 36:35
  SPEAKER_00: So it's really cool to kind of see how Dawes built these really simple models, and even though they're so simplistic, they were actually performing better than humans.
• 36:45
  SPEAKER_00: And so I think both of those would be fun things where, like, if you're interested in these questions, challenge yourself to actually read that academic research and see what it's like.
• 36:56
  SPEAKER_01: Well, Dr.
• 36:56
  SPEAKER_01: Diet Forrest, thank you so much for joining us today.
• 37:01
  SPEAKER_00: Absolutely, yeah.
• 37:02
  SPEAKER_00: Thank you.
• 37:02
  SPEAKER_00: It was interesting to talk about some of these and and think through the questions again.
• 37:13
  SPEAKER_01: This has been Tas of Curiosity in an AI world.
• 37:16
  SPEAKER_01: Explore careers through the lens of AI.
• 37:20
  SPEAKER_01: Until then, stay curious.