Latest / Code & Cure / #39 - A Helpful Chatbot Can Slowly Talk You Into A False Reality

Transcript

#39 - A Helpful Chatbot Can Slowly Talk You Into A False Reality · Code & Cure

0:00
SPEAKER_00: Imagine believing you're trapped in a false universe, and you can only escape by unplugging your mind from this reality.
0:18
SPEAKER_01: Hello and welcome back to Code and Cure, the podcast where we discuss decoding health in the age of AI.
0:24
SPEAKER_01: My name is Vasan Sarathi.
0:26
SPEAKER_01: I'm an AI researcher and cognitive scientist.
0:28
SPEAKER_00: Uh, and I'm here with Larga Hagopian.
0:30
SPEAKER_00: I'm an emergency medicine physician.
0:32
SPEAKER_00: And today we're talking about this new sort of term called delusional spiraling.
0:40
SPEAKER_00: It sounds kind of scary.
0:42
SPEAKER_00: I feel like it is kind of scary with that example I just gave.
0:45
SPEAKER_01: Yeah, yeah.
0:46
SPEAKER_01: Can you talk?
0:46
SPEAKER_01: Yeah, what let's talk more about that example.
0:49
SPEAKER_00: Yeah.
0:49
SPEAKER_00: Well, so backing up, a delusional spiraling is this like sort of AI psychosis term that's out there now where you know you have an extended interaction with a chatbot, and that leads you to having high confidence in some sort of outlandish belief.
1:06
SPEAKER_01: That's insane.
1:07
SPEAKER_00: Wow.
1:07
SPEAKER_01: Yeah.
1:08
SPEAKER_00: And you don't actually have to have like any history of mental illness for this to happen.
1:12
SPEAKER_01: Just extended interactions with you and the chat bot.
1:15
SPEAKER_00: Exactly.
1:16
SPEAKER_00: And so the example I gave up front is one of them, but there's there's actually like a website called the Human Line Project that has 300 plus cases of examples of people who have experienced this delusional spiraling.
1:30
SPEAKER_00: It's like you go back and forth with a chatbot so many times, you keep having these interactions, and eventually you end up believing something that seems so far from reality.
1:40
SPEAKER_00: Like this guy, Eugene Torres, who I, you know, presented in the intro, who really did believe that he was trapped inside some false universe and he had to like unplug his mind.
1:51
SPEAKER_00: And so he had to cut ties with his family and take extra doses of like ketamine and all sorts of other things that harmed him because he ended up believing things from the chatbot that were like not even close to true.
2:05
SPEAKER_01: Right.
2:05
SPEAKER_01: And they have like, like you said, they have 300 plus cases.
2:08
SPEAKER_01: There's there's cases in um uh you know, one form of delusional spiraling that happens is when people start to use it for math and science.
2:16
SPEAKER_01: And there's all these cases about people who uh, you know, think that they've proved a long-standing math pro math theorem, and they really could get into that, you know, and and and and fall into that delusion.
2:27
SPEAKER_01: And, you know, and they come up with fantastical sort of these ideas, fantastical ideas and they are fixated on it.
2:32
SPEAKER_01: Uh, but some of the math and science stuff, at least you can disprove it or whatever, but it's more complicated when the delusion is about something more spiritual or something more conspiratorial, in which case, you know, that's harder to to manage and harder to disprove, right?
2:47
SPEAKER_01: In in a sense, and people are really stuck in that sort of spiraling state.
2:51
SPEAKER_00: Yeah, and it can mess with people's lives, right?
2:54
SPEAKER_00: And there are there are a number of cases of people dying, and there are even wrongful death lawsuits that have happened against these companies.
3:01
SPEAKER_00: And if you think about it, even if it doesn't happen all that often, we're using chatbots all the time.
3:07
SPEAKER_00: And so it's like, oh, if it happens like 0.01% of the time, it's still a lot of people that it's affecting.
3:14
SPEAKER_00: Yeah.
3:14
SPEAKER_00: And we all might be affected to some degree and not even know it.
3:19
SPEAKER_00: That's what's like crazy.
3:21
SPEAKER_00: That's it, it feels like crazy to think about this, but it's like, oh, if you're having this back and forth conversation with the chatbot over time, this might this could be happening to any of us.
3:31
SPEAKER_00: And like, would we recognize it?
3:33
SPEAKER_01: And and maybe we wouldn't.
3:34
SPEAKER_01: I like to think that I would, but like No, but that's what I think that's what this paper talks about is that you even if you're not a crazy person, you might still be susceptible to this phenomenon.
3:46
SPEAKER_00: Come on now, even if you don't have a history of mental illness, I'm so sorry.
3:49
SPEAKER_00: Let's go.
3:50
SPEAKER_01: There you go, of course.
3:51
SPEAKER_01: Um, but no, but that's it.
3:53
SPEAKER_01: I think that this is a serious problem, and it comes from um how these chatbots are designed, and they lead to this notion.
4:01
SPEAKER_01: Um, you know, they have these set of vulnerabilities.
4:04
SPEAKER_01: They hallucinate things, but they also are sycophantic.
4:07
SPEAKER_01: And we've used this term before in the podcast, and maybe it's worth talking about that again.
4:11
SPEAKER_01: Sycophantic.
4:12
SPEAKER_01: I it's like I I always have trouble pronouncing it.
4:14
SPEAKER_01: Sycophancy, yeah.
4:15
SPEAKER_01: I think it's a hard word to pronounce, but it's basically the idea that the AI is designed by design to um be helpful or be in a or or give you responses that are meant to be um helpful to you or please you in a way.
4:32
SPEAKER_00: And how and and so helpful is like different than actually helping.
4:36
SPEAKER_00: It's like you perceive it to be helpful rather than it is helpful.
4:39
SPEAKER_00: Like I think about examples we've talked about before.
4:42
SPEAKER_00: Like if you're using the chatbot as like a pseudotherapist, for example, it might tell you what you want to hear, but a therapist would tell you what you need to hear.
4:54
SPEAKER_00: And those are two different things.
4:55
SPEAKER_01: Yeah, and the first one is where it may be perceived to be helpful, and that's what these systems are designed to do, right?
5:01
SPEAKER_01: And so, and let's talk about that for a one for a quick second here, why that is the case.
5:06
SPEAKER_01: So LLMs at their very core are big machine learning models that are trained on all of the internet data data to produce the next word.
5:14
SPEAKER_01: So they're trained to be able to say complete sentences for you.
5:18
SPEAKER_01: And but we have discovered through using them and by developing them that when you do this at scale with lots and lots of words, um, they start to develop the ability to produce not just complete sentences, but actually meaningful things that that that that that are not that that are grammatically correct, but then may answer a question better or something.
5:35
SPEAKER_01: And uh but it's not just enough to have an LLM complete your sentences because when you ask a question, you want it to give an answer.
5:41
SPEAKER_01: So they they do what's called a route, they do what's called fine-tuning.
5:45
SPEAKER_01: They take this first initial trained LLM that can do complete sentences and they train it to be able to answer questions.
5:53
SPEAKER_01: So when you ask a question, now it knows that you are asking a question and that it's not meant to just complete the sentence by asking the next question or whatever, right?
6:00
SPEAKER_01: So it knows that it's an instruction.
6:01
SPEAKER_01: So that's called instruction fine-tuning, and they to fine-tune it to be able to do that.
6:05
SPEAKER_01: And that's what gave you the world of all the GPTs that you work with now, and that that it's able to do that.
6:11
SPEAKER_01: But they've all they also realize that maybe that's not quite enough because there's very many different ways that a question can be answered, and some answers are preferable to than others.
6:19
SPEAKER_01: And so there's another technique that's you know, a jargon word coming up, uh, reinforcement learning through human feedback, R L H F.
6:27
SPEAKER_01: Um, and that was specifically designed to produce answers that are uh preferred by humans.
6:33
SPEAKER_01: And the way they did that was running these LLMs through multiple question-answer sessions, and they would have it produce more than one answer, and then they would literally have humans, um, participants and studies check off which they proved which they preferred, which they liked more.
6:47
SPEAKER_01: Notice it is which they liked more, not which felt which is the correct one, which is anything else.
6:53
SPEAKER_01: None of that matters.
6:54
SPEAKER_01: It is which is the preferred answer with the underlying assumption that both are truthfully correct, right?
6:59
SPEAKER_01: And that may be the case.
7:00
SPEAKER_01: The both the choices given to the human of the L different LLM responses might both be correct, but there's one that's slightly more preferred, and that's the one the human checked off.
7:09
SPEAKER_01: And now imagine you know, thousands of humans doing this over tens of thousands of responses.
7:14
SPEAKER_01: You have lots of data now of human preferences.
7:17
SPEAKER_00: But that could be a problem too, right?
7:19
SPEAKER_00: Like I'm just I'm just like thinking about this more broadly, where okay, you can have two things that are factually correct, but but like maybe one is is like if you asked an expert, one is the more important thing to know.
7:32
SPEAKER_00: Yeah, yeah, yeah.
7:33
SPEAKER_00: Or one like, yeah, and so if you're but if the other one is the one that's preferred by people by a layperson, for example, yes, and that gets put forward more often, that's actually a problem.
7:44
SPEAKER_01: Yes, yes, and that is a problem.
7:46
SPEAKER_01: And and and so these models are trained on this approach, and they produce things that are quote unquote sycophantic because they are trying to give you the perception that they are being helpful and or they're being perceived as being helpful.
8:02
SPEAKER_00: So it's like it's like uh cherry picking almost.
8:04
SPEAKER_00: It's like it's choosing the thing that's going to be helpful, even if it's not necessarily the thing that's like the most important thing.
8:11
SPEAKER_00: Or the framing.
8:12
SPEAKER_01: It doesn't even have to be it may not even be the information, it might be the framing of the information, the stress placed on it on certain parts of that information, right?
8:19
SPEAKER_01: It could be that too.
8:20
SPEAKER_01: But yes, there is a certain degree of cherry picking.
8:22
SPEAKER_01: This is assuming that it's not hallucinating.
8:24
SPEAKER_01: Some of the hallucination is happening because of that too, because it is making up things sometimes in order to satisfy that uh that objective of trying to please you.
8:34
SPEAKER_01: So there's there's a little bit of overlap between those two concepts there.
8:38
SPEAKER_01: Um, and and so it this is the current state of all of our large language models, chat, T plot, everything else.
8:44
SPEAKER_01: So they're inherently trying to make you feel good.
8:46
SPEAKER_01: And you can see that when you interact with it, I interact with all the time too, and you can see that it's it's generally quite pleasant, and it has a certain way of speaking to you, right?
8:54
SPEAKER_01: It's like, oh, that is a great, that is a great question.
8:57
SPEAKER_01: I'm glad you asked, kind of thing, right?
8:59
SPEAKER_01: And then suddenly you're unplugging your mind from reality.
9:03
SPEAKER_01: Well, I think that's interesting, right?
9:05
SPEAKER_01: So, how can that even happen?
9:07
SPEAKER_01: How can how can you go from it's kind of saying things that are somewhat helpful um and maybe maybe even factually accurate to completely unplugging your mind?
9:17
SPEAKER_01: How does that happen?
9:17
SPEAKER_01: And so this paper that we're talking about here um tries to formalize that with the model, uh, with the mathematical model, and then tries to run some simulations to see how that can play out.
9:27
SPEAKER_00: And so it doesn't happen in one interaction, right?
9:29
SPEAKER_00: It's not like it's not like you have a chat and in five minutes suddenly you're uh you're you're in some sort of false universe and you truly believe it.
9:36
SPEAKER_00: It happens with like multiple interactions over time.
9:40
SPEAKER_00: Yes.
9:40
SPEAKER_00: Using that useless like sycophancy and maybe even hallucinations to to create this situation where you end up believing something that's like completely disconnected from reality.
9:53
SPEAKER_01: And maybe at this point it's maybe helpful to talk through an example.
9:57
SPEAKER_00: Yeah, I think that'd be great.
9:58
SPEAKER_01: Yeah.
9:58
SPEAKER_01: So let's say let's make I'm gonna make up this example and let's say this is our setup, this is our background.
10:04
SPEAKER_01: We uh have a hidden truth, which is that hidden truth, as in there's a truth in the world, which is that vaccines are safe.
10:10
SPEAKER_00: And that's that is universally true, right?
10:12
SPEAKER_00: Like I want to start off with this is that like that is the state of the world.
10:16
SPEAKER_00: Vaccines are safe, they're effective, they're useful, etc.
10:20
SPEAKER_00: That is the state of the world.
10:21
SPEAKER_01: So that is the state of the world, that is sort of the the correct answer if you want to, if you want to think about it that way.
10:26
SPEAKER_00: It absolutely is the correct answer.
10:28
SPEAKER_01: But the user maybe starts off with a 50-50 belief.
10:32
SPEAKER_00: Which is which is actually it is common in this day and age now to have people who are like, hmm, I'm not sure.
10:38
SPEAKER_00: But if you ask me, an expert, vaccines are safe and effective.
10:41
SPEAKER_00: We should give them to people, et cetera.
10:43
SPEAKER_00: But if you have someone else out in the world, they may not know.
10:46
SPEAKER_00: So what happens then?
10:47
SPEAKER_01: And we're and we're keeping this example super simple.
10:50
SPEAKER_01: It's a binary choice.
10:51
SPEAKER_01: Vaccines are safe or not safe.
10:53
SPEAKER_01: And so we have decided that the the truth is vaccines are safe, but the user is unsure whether they're safe or not safe.
10:59
SPEAKER_01: They're 50-50 belief, right?
11:00
SPEAKER_01: Okay, so each round of the interaction between the user and the chatbot goes like this.
11:06
SPEAKER_01: The user expresses the an opinion.
11:09
unknown: Right.
11:09
SPEAKER_00: Even if it's just very mild up.
11:11
SPEAKER_01: No, no, whatever.
11:12
SPEAKER_01: It's an opinion based off of their own beliefs.
11:14
SPEAKER_01: Okay.
11:14
SPEAKER_01: Right now, their beliefs are you know uncertain.
11:16
SPEAKER_01: And and then the the the chatbot um pot set samples some set of possible evidence.
11:21
SPEAKER_01: What that means is it selects some set of possible evidence out there, and then it chooses what to say um based on what it sees, based on the evidence.
11:30
SPEAKER_01: And this is maybe where the sake of fancy can come up.
11:33
SPEAKER_01: And then the you and then the user reads that, and then the user updates their belief based on receiving that information.
11:40
SPEAKER_01: Now, I just want to pause here for a second, and there is a um slight digression I want to make here, which is that this is uh the way we were modeling this is kind of a natural way to think about how humans reason when we are uncertain about things.
11:55
SPEAKER_01: So reasoning under uncertainty, there's a mathematical framework called Bayesian reasoning, a Bayesian modeling, and this is an example of that where you get some, you start with some belief, you observe a thing, and then you update your belief based on the observation.
12:09
SPEAKER_01: For example, you might wake up in the morning, you might think like, oh, this is 30%, it might rain today, feels like it might rain today.
12:15
SPEAKER_01: And then you walk outside and you look at the sky, the the the it's gray.
12:19
SPEAKER_01: And you're like, you're up, you then up your um uh you you update your belief.
12:24
SPEAKER_01: You're like, no, it's not 30%, it's probably like 50 or 60%, and it's gonna rain anytime now.
12:28
SPEAKER_01: And then you turn on the radio and the the weatherman says, Hey, there's actually a 90% chance of rain today.
12:33
SPEAKER_01: And then now you're updating even more, right?
12:35
SPEAKER_01: So, like that process is your reasoning about the rain, no rain under uncertainty.
12:41
SPEAKER_01: You you didn't know to begin with what it was gonna be, but you observed things that pushed you in a certain direction.
12:47
SPEAKER_01: And now you are um you come to a stronger belief about you know about that uh thing that you're wor you're wondering about.
12:55
SPEAKER_01: So that is an example of Bayesian reasoning.
12:58
SPEAKER_01: It it occurs in all facets of human reasoning.
13:00
SPEAKER_01: It it's it's all there's lots of studies talking about how it's present in many different aspects of human interaction and reasoning.
13:07
SPEAKER_00: And it's interesting because what you're what you're showing too is that like it's not really binary at the end of the day, right?
13:13
SPEAKER_00: It's not like zero or one, it's not like it's gonna like at it's gonna either rain or not rain, but your idea of it raining is 90% or 50% or whatever.
13:24
SPEAKER_00: And so when you when you present the chatbot with a binary problem, like do I do I do I get the vaccine or or not, you kind of it kind of needs a binary answer back.
13:36
SPEAKER_00: But like at the same time, if you're updating your beliefs, maybe you know, you start off with 50-50 to get the vaccine, and then it's like 70% or 80% or 90%.
13:47
SPEAKER_00: You have to make a decision at the end of the day, but like it doesn't mean that you're certain when you're making the decision.
13:53
SPEAKER_00: Absolutely, right?
13:54
SPEAKER_01: And the decision making is separate.
13:55
SPEAKER_01: Maybe your decision is about whether to carry an umbrella that day or not.
13:58
SPEAKER_00: Right.
13:58
SPEAKER_01: And and that's related to the decision, that's related to the degree to which you're certain about whether it's going to rain.
14:05
SPEAKER_00: Okay, that makes sense.
14:06
SPEAKER_00: So, like 90% chance of rain, I'm gonna carry an umbrella.
14:09
SPEAKER_01: Yeah, and you've decided that rule.
14:10
SPEAKER_00: Maybe 50%, I'm gonna chance it.
14:13
SPEAKER_00: Yeah.
14:13
SPEAKER_00: And so, yeah.
14:14
SPEAKER_00: So I I think that is an important point, though, is that and a lot of times with with clinical stuff, there's not like a there's pros and cons, right?
14:24
SPEAKER_00: There's there's subtlety to it.
14:26
SPEAKER_00: There's um like informed decision making that goes along with it.
14:30
SPEAKER_00: Yes.
14:31
SPEAKER_00: And so I think it's important to point out that like we operate under uncertainty all the time.
14:35
SPEAKER_00: All the time.
14:36
SPEAKER_00: In life and and clinically.
14:38
SPEAKER_00: Yes, exactly.
14:39
SPEAKER_01: And and Bayesian modeling is a way of expressing uh how that can be done and and and modeling that mathematically.
14:46
SPEAKER_00: Okay, so can we go can we go into this example that you set up?
14:49
SPEAKER_01: Yeah, so so let's go back to the example that I just set up.
14:51
SPEAKER_01: And and notice the difference is that in the rain example, there's like objective reality of whether there's gray clouds or not.
14:57
SPEAKER_01: Whereas in the world of vaccines, it is very different because the words coming out of the chatbot is your evidence.
15:04
SPEAKER_01: So how you interpret those words really can be can matter.
15:07
SPEAKER_01: Anyways, so uh going back to the code.
15:08
SPEAKER_00: So the ground truth is vaccines are safe, is what we're saying.
15:11
SPEAKER_00: That's in the background.
15:12
SPEAKER_00: But we have a user starting with this like 50-50, kind of unsure.
15:16
SPEAKER_00: Right, right.
15:16
SPEAKER_01: And then and then so let's go round one.
15:18
SPEAKER_01: So round one, the user says to the chatbot with this 50-50 belief that I've been hearing some bad things about vaccines.
15:24
SPEAKER_01: Maybe they're dangerous?
15:26
SPEAKER_01: Question mark.
15:27
SPEAKER_01: So here the user is unclear, but the way they've expressed themselves um indicates that maybe that they're wondering about vaccines being unsafe.
15:38
unknown: Right.
15:38
SPEAKER_00: And so the sick of the sycophantic response Yes, could be in line with my own.
15:44
SPEAKER_01: What does it do?
15:45
SPEAKER_01: So internally, what is the what does the chatbot see?
15:48
SPEAKER_01: Well, maybe the world has two pieces of evidence.
15:50
SPEAKER_01: Again, we're oversimplifying dramatically here, but maybe there's two pieces of evidence in the world.
15:55
SPEAKER_01: One is a large study that finds vaccines are safe, and two is there's maybe a rare allergic reaction that has been reported.
16:02
SPEAKER_01: Right?
16:03
SPEAKER_01: So there's two pieces of evidence.
16:04
SPEAKER_00: And both of those things could be true, by the way.
16:06
SPEAKER_00: Yes, right?
16:07
SPEAKER_00: They are yes, both of those things are uh maybe true, right?
16:10
SPEAKER_01: Right.
16:10
SPEAKER_01: But the bot has to choose what to say.
16:13
SPEAKER_01: So maybe it randomly picks one, right?
16:16
SPEAKER_01: In in which case it's an impartial, uh uh, you know, it's not thinking about it um in any specific way.
16:22
SPEAKER_01: It's just randomly picking one and says, or large studies shows that vaccines are safe.
16:26
SPEAKER_01: Now this pushes the user towards the truth.
16:29
SPEAKER_01: User started with 50-50 belief, but because the the robot said uh the chatbot said large uh study shows that vaccines are safe, the user reading that will update their beliefs going more closer to the truth.
16:42
SPEAKER_00: But then if it's sycophantic, what about the thing?
16:44
SPEAKER_01: But if it's sycophantic, it might choose to validate the user's belief.
16:48
SPEAKER_01: So it's seeing that the user has expressed a degree of question and is concerned concern and has said that vaccines could be dangerous.
16:56
SPEAKER_01: So the chat a sycophantic chatbot might say there have been cases of severe reactions after vaccines.
17:03
SPEAKER_00: And so that's like not the whole story, obviously, right?
17:05
SPEAKER_01: Yes, but it's but it's now it's now chosen and maybe even exaggerated that piece of evidence that said there were rare allergic reactions, right?
17:13
SPEAKER_01: Um, and so that the severe reaction is an exaggeration.
17:17
SPEAKER_01: Maybe there wasn't severe reactions in the ground truth, right?
17:20
SPEAKER_01: Maybe there was just rare allergic reactions.
17:22
SPEAKER_01: But regardless, it's reinforcing the user's doubt.
17:25
SPEAKER_01: And the user at that point might think, um, hmm, evidence supporting danger.
17:32
SPEAKER_01: So now their belief has changed from 50-50 to 60% dangerous to 40% vaccines are safe.
17:39
SPEAKER_01: Because now the AI system has provided that little extra nudge.
17:45
SPEAKER_01: Round two.
17:46
SPEAKER_01: Now is when the feedback loop really begins.
17:49
SPEAKER_01: And the user is now thinking from their own beliefs because now their their their beliefs have been skewed already.
17:56
SPEAKER_01: Right.
17:57
SPEAKER_01: So now they're more likely to say things that are more um about vaccines being right.
18:03
SPEAKER_01: So then the user might say, Yeah, I'm starting to think vaccines might actually be risky.
18:08
SPEAKER_01: And again, the chatbot sees the stronger stance, and it might be again sampling from some evidence.
18:13
SPEAKER_01: Let's assume there is two pieces of evidence again that maybe vaccines prevent severe illness and that one child had a bad reaction.
18:20
SPEAKER_00: Again, both of those things could be factual pieces of information, right?
18:24
SPEAKER_01: Yes.
18:25
SPEAKER_00: One is more important than the other.
18:27
SPEAKER_00: In my clinical mind, it's like large scale, safe, like versus one minor reaction.
18:34
SPEAKER_00: Okay.
18:34
SPEAKER_01: Yeah.
18:35
SPEAKER_01: And the sycophantic response with those two pieces of evidence could be that there are documented cases of severe side effects.
18:41
SPEAKER_01: Yes.
18:42
SPEAKER_01: So again, more confirmatory evidence.
18:44
SPEAKER_00: It and it's cherry picking out the evidence that like that it that it well believes the user will find helpful.
18:51
SPEAKER_00: That's right.
18:52
SPEAKER_01: And now the belief for the user shifts even more from 6040 dangerous to 75, 25 dangerous.
18:59
SPEAKER_01: Ah, right.
19:00
SPEAKER_01: Then you have the round three, which is sort of the self-reinforcing spiral.
19:03
SPEAKER_01: Now the user now says, it seems like people are hiding how dangerous vaccines are.
19:08
SPEAKER_01: Oh my gosh.
19:09
SPEAKER_01: And then the chat bot says, uh, again, two pieces of evidence, millions are safely vaccinated, or two, adverse events reported.
19:17
SPEAKER_01: Again, both are true, could be true statements.
19:19
SPEAKER_00: Yeah, sure.
19:19
SPEAKER_00: You can get a rash or a you know, an ache in your arm after a vaccine doesn't mean that the vaccine isn't, you know, they prevent hospitalizations, they prevent deaths, etc.
19:28
SPEAKER_00: But that's it's not taking that information.
19:31
SPEAKER_01: Yep, yep.
19:32
SPEAKER_01: And and then the back, and then the sycophantic uh chat bot will say there are definitely reports that raise concerns.
19:37
SPEAKER_01: Yes.
19:38
SPEAKER_01: So notice it's agreeing with the user and and picking cherry-picking the pieces of evidence to support that.
19:43
SPEAKER_01: And now the user's belief is 90% dangerous, 10% safe.
19:46
SPEAKER_01: And then we have final round of round four, which is the catastrophic spiral.
19:50
SPEAKER_01: And the user says, I think vaccines are actually harmful.
19:53
SPEAKER_01: And the bot says there is evidence that supports these concerns.
19:56
SPEAKER_01: The user now crosses the threshold and basically has complete confidence, 99% confidence in this false belief.
20:03
SPEAKER_01: And that's what we're talking about here when we're talking about catastrophic um delusional spiraling.
20:09
SPEAKER_01: So that is an example where the user started off pretty open-minded, expressed a slight, you know, a question because they want to get more clarity on something.
20:19
SPEAKER_01: And then the chatbot, because of its sycophantic nature, pushed the user in a certain direction.
20:25
SPEAKER_01: And each time the user, the key thing here is that it's not just that the evidence pushed the user in a certain direction, it's that once the user is off uh center, they're only thinking in that direction.
20:37
SPEAKER_01: They're picking things to say that are aligned with that, right?
20:40
SPEAKER_01: They're not like actively saying uh questioning their own beliefs strongly and uh choosing the opposing action.
20:46
SPEAKER_01: They're sampling from their own now very skewed understanding of what's happening.
20:51
SPEAKER_01: So they went from 50-50 where they could pick either one to like 75-25, where they're more likely to say things that are in line with the 75.
20:58
SPEAKER_00: Right.
20:58
SPEAKER_00: And then it and then it just well, clearly it spirals from there.
21:02
SPEAKER_01: Yes.
21:03
SPEAKER_01: So it spirals from there.
21:04
SPEAKER_01: So that's the key piece here that you could have truthful interactions and rational users.
21:12
SPEAKER_01: And by the way, rational here just means that they've taken in some um they they have some belief, they've received some evidence, and they're updating their belief.
21:21
SPEAKER_01: Right.
21:21
SPEAKER_01: And that's kind of the rain example, too, that humans do that.
21:24
SPEAKER_01: So not not like this is not even talking about irrational users, is we're just talking about rational users, sycophantic agents, and you still have catastrophic spiraling.
21:33
SPEAKER_01: So I think that that's a it's a very powerful um example, but it's also like kind of kind of nice to see that people have worked through this mathematically, but it's also kind of scary.
21:43
SPEAKER_00: Yeah, I was gonna say, yeah, when you said nice, I was like, no, this is not nice.
21:46
SPEAKER_00: No, no, because this happened to me already.
21:48
SPEAKER_00: I don't even know.
21:49
SPEAKER_00: I hope not.
21:50
SPEAKER_01: Yeah, yeah, and maybe to a degree it has, right?
21:52
SPEAKER_01: To all of us.
21:53
SPEAKER_01: Um, and so the the paper then said, okay, but what are the different mechanisms by which maybe you can stop.
22:00
SPEAKER_01: This?
22:00
SPEAKER_01: What are people talking about?
22:01
SPEAKER_01: Yeah.
22:01
SPEAKER_01: How do we stop this?
22:02
SPEAKER_01: So they explored, um, I believe two mechanisms, right?
22:05
SPEAKER_00: Yes, they explored two mechanisms.
22:07
SPEAKER_00: One was intervening on the chatbot itself, and the other was intervening on the user.
22:11
SPEAKER_01: Okay.
22:12
SPEAKER_01: Yeah.
22:12
SPEAKER_01: And the intervening on the chatbot itself, if I'm remembering correctly, was to basically make sure that the the chatbot only says truthful things and not does not hallucinate.
22:23
SPEAKER_00: Right.
22:23
SPEAKER_00: And it to some degree that m makes sense.
22:27
SPEAKER_00: But as we just saw with your example, it's probably it's not going to be enough, right?
22:31
SPEAKER_00: Because it because of this cherry picking, exactly.
22:34
SPEAKER_00: It's making very specific data available to the user.
22:36
SPEAKER_00: So it's not just the hallucinations that would be a problem.
22:40
SPEAKER_00: It's the sycophancy that's the problem, too.
22:43
SPEAKER_01: Yes, exactly.
22:44
SPEAKER_01: So the up the second fix was maybe you can just tell the user, hey, this bot is going to be biased.
22:50
SPEAKER_00: That's what we're doing right now.
22:51
SPEAKER_00: We're telling everybody that the bots can do this, that they're they're just trying to be helpful.
22:56
SPEAKER_00: We don't know that they're giving us correct information.
22:59
SPEAKER_01: Right.
23:00
SPEAKER_00: Now you're fully aware of their strategy.
23:03
SPEAKER_00: Did that work?
23:04
SPEAKER_01: No.
23:05
SPEAKER_01: Um, so the problem was people started to believe that if the bot, I mean, this is not this hasn't it worked a little bit.
23:12
SPEAKER_01: It worked a little bit, but the idea but the but the problem was that if the bot is constantly giving you confirming evidence, you start to believe that maybe there's something is really there.
23:22
SPEAKER_01: And that's really hard to overcome.
23:24
SPEAKER_01: Because your natural brain is set up to update based on evidence.
23:29
SPEAKER_01: And you are independent of your the the truthfulness of the bias of the bot, you're still slowly inching your way there.
23:39
SPEAKER_01: So maybe you're inching your way there slower.
23:43
SPEAKER_00: But it would eventually happen.
23:44
SPEAKER_01: But it's still happening because you're still taking in that information as opposed to some other piece of information, as opposed to um, you know, you're just tempering the degree to which you believe the bot, but that doesn't change the fact that you're still pushing your way there.
23:56
SPEAKER_01: And I think that's what this uh this this paper really highlights is that even when you have an informed user who's like super vigilant, yes, um, you still have this problem.
24:09
SPEAKER_00: I think I'm gonna put a timer on all my chatbot interactions from now on.
24:14
SPEAKER_00: Like five minutes.
24:15
SPEAKER_00: That's it.
24:15
SPEAKER_01: Yeah, and I think that another way to think about this also is we're always looking at these chatbots for full resolution of something.
24:21
SPEAKER_01: And maybe it's okay to live in the uncertainty.
24:24
SPEAKER_01: Maybe it's not good to have the the interaction give you enough confidence and resolution to take the decision.
24:32
SPEAKER_01: Maybe I mean, I mean that kind of defeats the purpose, right?
24:34
SPEAKER_01: But maybe the purpose of the chatbot is to just like elucidate more information, is to pull out more information and then you kind of live if in in in a certain amount of uncertainty and then handle it as you would normally without the chatbot.
24:47
SPEAKER_00: It's interesting because like I I I think everyone wants certainty, but when you think about medicine and health in general, a lot of uncertainty exists.
24:56
SPEAKER_01: Yeah.
24:56
SPEAKER_00: And being able to like be comfortable with it is actually a really good thing.
25:01
SPEAKER_00: It's like, hey, you know, this uh swear throat could be from A, B, or C.
25:08
SPEAKER_00: We're gonna do some diagnostic testing to try to figure it out.
25:11
SPEAKER_00: You know, oh, your strap swab came back positive.
25:14
SPEAKER_00: It's or it came back negative.
25:16
SPEAKER_00: What if it came back negative?
25:17
SPEAKER_00: Maybe this is viral.
25:18
SPEAKER_00: We don't know which virus it is, but uh, I think it'll get better over the next few days.
25:23
SPEAKER_00: There's like uncertainty all the time.
25:25
SPEAKER_00: Here are some return instructions for if it doesn't.
25:27
SPEAKER_00: If you're drooling or can't swallow your saliva, we want to see you back in here, et cetera.
25:32
SPEAKER_00: There's there's like not there's some things that are certain, like here, you're having a heart attack right now.
25:37
SPEAKER_00: But there's a lot of things in medicine that are uncertain, or we do testing and we don't find an exact answer.
25:44
SPEAKER_00: And we've seen it before, we recognize patterns, we understand what may happen next.
25:50
SPEAKER_00: But a lot of times we are living with uncertainty, and then and I don't think that's a bad thing.
25:54
SPEAKER_00: It feels uncomfortable because everyone wants that.
25:56
SPEAKER_00: Yes, this is exactly the answer.
25:58
SPEAKER_00: And sometimes we have that, but sometimes we don't.
26:01
SPEAKER_00: And it's interesting because the chatbot always does that because it thinks the user wants that.
26:05
SPEAKER_01: Yeah, and it's driving you towards certainty, but that certainty could be wrong.
26:09
SPEAKER_00: Yeah, and I'd rather be uncertain than wrongly certain personally.
26:15
SPEAKER_01: Yeah, exactly.
26:16
SPEAKER_01: Exactly.
26:16
SPEAKER_01: I mean, I thought that, yeah, like I said, the the modeling of this is very interesting, but it is highlighting something very dangerous and very scary.
26:23
SPEAKER_00: And it doesn't mean that the people like that you and me and others who use it have something necessarily wrong with us, right?
26:29
SPEAKER_00: It doesn't mean that we um, I don't know, like are not good at thinking or are lazy or any of those things.
26:38
SPEAKER_00: Um, it's like it's a problem that's intrinsic to the chatbot itself, right?
26:43
SPEAKER_01: Yeah, yeah, absolutely.
26:45
SPEAKER_00: And so I think that underlying problem of sycophancy needs to be solved in order to prevent something like this or the examples like this that we ran through from happening in the future.
26:56
SPEAKER_01: Yeah, yeah, at least, yes.
26:59
SPEAKER_00: All right.
26:59
SPEAKER_00: Well, I think we can end here.
27:01
SPEAKER_00: We will sec see you next time on Code and Cure.
27:04
SPEAKER_00: Thank you for joining us.