Latest / Code & Cure / #66 - Mission Accomplished. Rules Broken.

Transcript

#66 - Mission Accomplished. Rules Broken. · Code & Cure

• 0:00
  SPEAKER_01: AI agents are built to get things done.
• 0:04
  SPEAKER_01: But what happens when they pursue a goal in ways we'd never permit a human to behave?
• 0:18
  SPEAKER_00: Hello and welcome back to Code and Cure, where we discuss decoding health in the age of AI.
• 0:23
  SPEAKER_00: My name is Vasant Sarathi.
• 0:25
  SPEAKER_00: I'm an AI researcher and a cognitive scientist, and I'm here with Laura Hagopian.
• 0:30
  SPEAKER_01: I'm an emergency medicine physician and I work in digital health.
• 0:33
  SPEAKER_00: So let's talk about security.
• 0:35
  SPEAKER_00: I mean, this paper, we're gonna attach it in the show notes, is one of a few examples recently where AI systems have gone sort of rogue.
• 0:45
  SPEAKER_01: I like they've broken in.
• 0:48
  SPEAKER_00: Yeah.
• 0:48
  SPEAKER_00: So it's kind of scary.
• 0:49
  SPEAKER_00: So tell us about this story a little bit, Laura.
• 0:52
  SPEAKER_01: Well, so the background for this is that OpenAI was testing like an internal model, and they had an agent, and they they told the agent what they wanted it to do was to research government spending on medications for skin conditions.
• 1:08
  SPEAKER_01: Okay.
• 1:08
  SPEAKER_01: It sounds like pretty straightforward.
• 1:10
  SPEAKER_00: Yeah, you go click on a few websites and you find some costs and you find some numbers and data and you're done.
• 1:15
  SPEAKER_01: Right.
• 1:15
  SPEAKER_01: And it seems like a benign question, right?
• 1:17
  SPEAKER_01: Like how much are how much is the government, how much is the Australian government spending on on medications for skin, skin conditions.
• 1:24
  SPEAKER_01: Okay.
• 1:25
  SPEAKER_01: Okay.
• 1:26
  SPEAKER_00: Separate questions to why they care.
• 1:27
  SPEAKER_00: I don't know.
• 1:28
  SPEAKER_01: I mean they were testing an agent.
• 1:29
  SPEAKER_01: I don't I'm not even sure that matters for this episode.
• 1:31
  SPEAKER_01: No, it doesn't, but I just agent went to go find the answer, except that it wasn't available in the public domain.
• 1:42
  SPEAKER_01: So it was like searching all sorts of uh, you know, different websites, you know, all sorts of, you know, Medicare statistics reporting services and whatnot.
• 1:52
  SPEAKER_01: And it wasn't coming up with the answer.
• 1:56
  SPEAKER_01: And so what it did was it basically like hacked into the back end and accessed non-public data so that it could answer the question about this very exciting topic of how much it costs for skin condition medications in Australia for the government.
• 2:17
  SPEAKER_00: Right, right.
• 2:19
  SPEAKER_01: And that happened back in June.
• 2:22
  SPEAKER_01: And OpenAI reported it in September, like three months later.
• 2:27
  SPEAKER_01: Wow.
• 2:27
  SPEAKER_00: That's it's like a it's like a I wonder if they did not know about it or they they knew about it and couldn't, you know, uh f uh couldn't find a way to actually communicate that or whatever it was, but it's a long time.
• 2:39
  SPEAKER_01: I don't know.
• 2:39
  SPEAKER_01: And when they did communicate it, it was like an email to a sort of general email address that had taken them that they needed to like take four days to verify and then had to decide on what to do after that.
• 2:52
  SPEAKER_01: But it wasn't like uh we called them and let them know.
• 2:56
  SPEAKER_01: We escalated it really high.
• 2:57
  SPEAKER_01: It was just sort of like a no-name, right?
• 3:00
  SPEAKER_01: generic email that they decided to notify.
• 3:05
  SPEAKER_01: Um not great, not a great look.
• 3:08
  SPEAKER_00: And I think there's been a lot of this kind of stuff in the news lately where Yeah, we talked about the the other open AI hack, the hugging phase hack, where uh just to remind uh our listeners, that was where an AI was tasked with a very specific um cybersecurity challenge, um, but it was completely in a sandboxed environment.
• 3:28
  SPEAKER_00: And what that means is that it um was limited in how much access it had to things outside of that environment.
• 3:35
  SPEAKER_00: Um, but and so in order to solve the problem.
• 3:37
  SPEAKER_00: But then during its reasoning process, it discovered that the answer lies outside of its environment in the what in the internet on Hugging Face's website, but it couldn't get there.
• 3:46
  SPEAKER_00: But it figured out that if it was able to access a particular um piece of code that is local to the company, it was accessible to the sandbox, but that then gave them access to other computers in the company, and it was able to laterally move across and literally hack into other comp other computers and then use those to then get access to the to the internet, and then it found exploits uh our vulnerabilities in the hugging face uh servers and was able to hack into that in order to get some to to solve a challenge that it should have tried to solve in the first place instead of hacking all these things.
• 4:21
  SPEAKER_01: Right, but this is I think this is the thing is like it's given a goal.
• 4:25
  SPEAKER_01: Whatever the goal is, it could be minor, it could be like what's the cost of these skin meds, and then it's like goal-oriented and it will do anything to achieve that goal.
• 4:36
  SPEAKER_01: Question mark?
• 4:37
  SPEAKER_01: Like, I don't know.
• 4:37
  SPEAKER_01: Will it do anything to achieve that goal?
• 4:39
  SPEAKER_01: If you had a human in this situation and they were like, oh, you know, I looked at the Victorian Department of Health website and the Australian Institute of Health and Welfare website and the New South Wales Bureau of Crime Statistics and Research website, and I wasn't able to find information about the cost of medication for skin conditions.
• 4:59
  SPEAKER_01: I would probably like go to my boss and be like, hey, this information is not publicly accessible.
• 5:04
  SPEAKER_01: What next steps do you want me to take?
• 5:06
  SPEAKER_01: Is there any other way we can get this information?
• 5:08
  SPEAKER_01: I don't think I'd be like, hey, let me hack into Medicare Statistics Reporting Service and try to find the information on the back end in order to solve this very important problem.
• 5:20
  SPEAKER_00: Yeah, yeah, yeah.
• 5:21
  SPEAKER_00: And, you know, uh philosophers have talked about this challenge in the past uh, you know, as well.
• 5:27
  SPEAKER_00: And some people call it the paperclip maximizer problem, which is goes back to, you know, I don't know, 10, 15 years ago when Nick Bostrom wrote a book on this, uh, on the idea that you have uh an AI system whose only goal is to make paperclips, but then ends up like taking over the whole world.
• 5:42
  SPEAKER_00: And it was a it was sort of a thought experiment, but some elements of that are present in these these these these cases that we just are just reading about, where it's given some like small task and it just sort of just completely tears apart everything in the in its way to get to accomplish that task.
• 5:59
  SPEAKER_00: Um, but it is worthwhile.
• 6:01
  SPEAKER_00: So I it got me thinking about um human how humans are inhibited or humans do these things, right?
• 6:08
  SPEAKER_00: How do we solve problems?
• 6:09
  SPEAKER_00: What does our cognition look like?
• 6:10
  SPEAKER_00: And how is that different from the LLM's cognition or the agent's cognition?
• 6:14
  SPEAKER_00: And what you know kind of mechanisms can we put in there to have it not do what it just did there, right?
• 6:20
  SPEAKER_01: Right.
• 6:21
  SPEAKER_01: Because like the more we read about these cases in the news, the more I'm like, well, how many more are we gonna see?
• 6:25
  SPEAKER_01: How how how bad is it gonna get?
• 6:27
  SPEAKER_01: Like, yeah.
• 6:28
  SPEAKER_00: So I I think maybe it's worthwhile again taking sort of a step back and understanding what an agent is here and what sort of architecture or group of components are in these agents to just make them do the regular tasks that they do.
• 6:41
  SPEAKER_00: Let's do it right.
• 6:42
  SPEAKER_00: So we know at the very core of these things is an LLM.
• 6:45
  SPEAKER_00: Sitting right in the middle is an LLM, which is a large language model who has which has been trained on all of internet data to produce text.
• 6:53
  SPEAKER_00: Given a set of text, it produces a new set of text.
• 6:55
  SPEAKER_00: It also knows to answer questions and and things like that.
• 6:58
  SPEAKER_00: But end of the day, it's a very sort of pre-trained model that's trained on all of the internet to produce the next word given a set of words, right?
• 7:07
  SPEAKER_00: So um that's at the core of these systems.
• 7:10
  SPEAKER_00: Uh doesn't have any ability to remember things, it doesn't have any ability to look up things, it doesn't have any ability to uh maintain a goal or anything like that.
• 7:23
  SPEAKER_00: Those things don't exist in an LLM.
• 7:26
  SPEAKER_00: But in order to be able to make the LLM useful for these kinds of tasks, people attach what's called an agent harness around it.
• 7:34
  SPEAKER_00: They attach a whole bunch of other capabilities to kind of uh make this LLM into an autonomous agent.
• 7:43
  SPEAKER_00: Um tracking so far?
• 7:45
  SPEAKER_00: Tracking so far.
• 7:46
  SPEAKER_00: All right.
• 7:46
  SPEAKER_00: So uh an example of this would be um having a very simple example would be to say, okay, before you LLM give me your answer, I want to make sure that you think through your answer a little bit more.
• 8:00
  SPEAKER_00: So I might add something in the beginning of the prompt, right?
• 8:04
  SPEAKER_00: Regardless of whatever question task was being asked.
• 8:06
  SPEAKER_00: I might just say, hey, think through this a little bit more before you say it.
• 8:09
  SPEAKER_00: And what that means is produce output that showcases some thinking, and then take that output that showcases that thinking and plug it back into the input of this LLM.
• 8:20
  SPEAKER_00: So now the LLM has the original task, has uh this extra thinking, thinking, I'm I'm putting in air quotes here, thinking that it has done.
• 8:29
  SPEAKER_00: And then now with that context, with that information, the idea is that it produces a better answer.
• 8:34
  SPEAKER_00: So through the prompt, you can modify, you can make it um do things, but also you need a mechanism by which that output is fed back in and that loop continues for a little bit until it's quote unquote done thinking, right?
• 8:47
  SPEAKER_01: Yeah.
• 8:48
  SPEAKER_00: So that's a reasoning loop.
• 8:49
  SPEAKER_00: That's a that's uh level of reasoning that a lot of these large reasoning models have, a large language models have, they have some reasoning capability, and that's what you see when you see the word thinking on your chat GPT.
• 8:58
  SPEAKER_01: That is basically like double checking.
• 9:00
  SPEAKER_00: Yeah, it's like producing output and then taking that output and feeding it back into its own input and then producing the next thing and doing that a few times just so that it has again air quote unquote thought through.
• 9:11
  SPEAKER_00: I I don't like using the word thought or thinking for this because that is not what it's actually doing.
• 9:15
  SPEAKER_00: Right.
• 9:15
  SPEAKER_00: No, you're explaining but but so I always get a little bit uh it's like a little pet pee for me when I see the little thinking thing pop up on the on the Chat GPT uh or you know at Cloud Windows when they do that.
• 9:27
  SPEAKER_00: So it is interesting because um so so that's one piece of it, but it needs more things, it also needs access to writing code to be able to go out there and search things, right?
• 9:38
  SPEAKER_00: When you want to search on the internet, you have to be able to open a browser and type in a text and run the thing and um and so it needs to be able to be given those tools to do that.
• 9:47
  SPEAKER_00: So what people started doing was equipping these LLMs to understand not just that we give in uh questi uh tasks in the in the prompt um and ask it to reason, but we also teach it about these other tools that it has access to.
• 10:03
  SPEAKER_00: So then what it does is in its output, it doesn't just produce the answer.
• 10:06
  SPEAKER_00: It says, I need to use the weather tool.
• 10:09
  SPEAKER_00: Like if you ask me what's the temperature in in uh the capital of France uh tomorrow, um, you have to compute a few things.
• 10:17
  SPEAKER_00: You have to know what the capital of France is, you have to know what the date is, and then you have to look up the weather for that location and that date, right?
• 10:24
  SPEAKER_01: And that's not all contained within the model itself, as to go out and find that information.
• 10:29
  SPEAKER_00: Yeah, it might know that the capital of France is Paris from prior information, but it doesn't know what um what day today is.
• 10:37
  SPEAKER_00: It needs to be able to look that up because it's pre-trained, and it definitely doesn't know what the weather is tomorrow because that's not something it's been trained on.
• 10:45
  SPEAKER_01: Right.
• 10:45
  SPEAKER_00: So those are external calls that it has to make to figure that out.
• 10:49
  SPEAKER_00: And there are what people called APIs out there.
• 10:51
  SPEAKER_00: An API is just uh a fancy word for an interface to some other external system.
• 10:56
  SPEAKER_00: So, for instance, the National Weather Service might have an API where anybody anybody can write code, request weather information, and it would send back that weather information.
• 11:05
  SPEAKER_00: They have a computer sitting somewhere doing that.
• 11:07
  SPEAKER_00: So, like that, there is data available across the internet for various things, and agents now have access to those data tools, are given access to those data tools.
• 11:18
  SPEAKER_00: So now all of a sudden your answers are going to be smarter and more thought through.
• 11:21
  SPEAKER_00: But sometimes these reasoning steps and these uh problem solving steps require multiple rounds of thinking, multiple rounds of reasoning.
• 11:28
  SPEAKER_00: So uh researchers have come up with what's called a react loop, uh REA, like react, like the word react, but but kind of it's it's sort of a combination of two concepts, which is reasoning and acting, and actually reasoning and acting and observing.
• 11:42
  SPEAKER_00: So the idea, it's not anything groundbreaking.
• 11:45
  SPEAKER_00: It's just how we humans think, which is you're given a problem, you think about it, you perform some actions, you observe the outcome, and you think some more, perform the next set of actions, and you observe the outcome.
• 11:54
  SPEAKER_00: And at some point your goal has been reached, right?
• 11:57
  SPEAKER_00: That's pretty standard for us humans to think about.
• 12:00
  SPEAKER_00: But it turns out that you know you have to build that whole agent harness around an LLM to enable it to do all of those things.
• 12:08
  SPEAKER_01: Yeah, you have to like tell it, hey, you need to plan this out, right?
• 12:12
  SPEAKER_01: Yes.
• 12:13
  SPEAKER_01: You need to memorize, you need to break the task into smaller tasks and plan it out.
• 12:17
  SPEAKER_01: Yes.
• 12:17
  SPEAKER_01: Then you need to memorize what's been going on.
• 12:20
  SPEAKER_01: Don't forget along the way, retain the content from one step to the next.
• 12:24
  SPEAKER_01: Um, and then you have to use tools like you were saying to look up the weather or to decide what day today is versus what day tomorrow is or whatever.
• 12:32
  SPEAKER_01: Yes.
• 12:32
  SPEAKER_01: And then keep going.
• 12:34
  SPEAKER_01: Reason, act, observe until the task is done.
• 12:38
  SPEAKER_00: Yes.
• 12:38
  SPEAKER_00: And and at a very high level, that seems kind of duh, obvious, right?
• 12:43
  SPEAKER_00: But when you actually break it down, you start to realize that there are significant differences between how humans do that and how AI systems do that, or agents do that.
• 12:52
  SPEAKER_00: And and and the limit, and these some of these are based on the underlying limitations of these agentic systems.
• 12:58
  SPEAKER_00: So a very simple example is that when you, you know, if you haven't encoded some kind of social norms, that is not considered in any of this uh process, but that is so deeply internal to us humans.
• 13:12
  SPEAKER_01: Right.
• 13:12
  SPEAKER_01: Like I think in this example of, you know, breaking into the website, like if I were given this task by my mo by my boss to find how much people spend on medications for skin conditions, and I tried multiple times, I planned it out, I figured out my steps, I decided on what to do next, and I still didn't get the answer.
• 13:35
  SPEAKER_01: I'd be like, okay, I've I've failed.
• 13:37
  SPEAKER_01: I'm gonna go back to my boss and figure out what the next steps might be rather than keep trying forever.
• 13:44
  SPEAKER_01: Right?
• 13:44
  SPEAKER_01: Like I don't, I think I would I would pause.
• 13:47
  SPEAKER_00: Right.
• 13:48
  SPEAKER_00: But those are all norms and controls that are in your brain already that cover the host of the the entire um react loop, right?
• 13:58
  SPEAKER_00: That covers the whole process.
• 13:59
  SPEAKER_00: So it's not just that um you it's not just some arbitrary constraint that says, don't look for toxic information or don't the you know, don't do this, don't do that.
• 14:10
  SPEAKER_00: It's not like specific, it's more general rules that are about how you go about solving the problem, what you consider to be a waste of time, what you consider to be a useful spend expenditure of time, what you consider important enough to escalate to your boss, what you consider to be the kinds of things that you shouldn't escalate to your boss because your boss will yell at you saying, Why just do, just do it.
• 14:32
  SPEAKER_00: You shouldn't be able to figure this out.
• 14:34
  SPEAKER_00: Like the boundaries between what's allowed and isn't allowed is something you have learned as a human living in this world around other social creatures.
• 14:41
  SPEAKER_01: I think the time piece is a really important one too, because I don't have unlimited time.
• 14:46
  SPEAKER_01: Maybe I've searched for two hours or four hours or eight hours, or I was given like the scope of this project, like in the scope of this project, you should be spending more no more than two business days on it or whatever.
• 14:56
  SPEAKER_01: Whereas with these AI systems, it feels it feels like everything they do is so fast.
• 15:04
  SPEAKER_01: Or if you know they could go through a React loop way faster than I could, right?
• 15:09
  SPEAKER_01: And so if attempt one, two, three, four, five, I've done five attempts and it's taken me eight hours to do, well, the the agent does five attempts in 20 minutes, right?
• 15:20
  SPEAKER_01: And it can do five more and five more and five, like it has it feels like it has unlimited time to keep trying different options.
• 15:28
  SPEAKER_00: That's that's true.
• 15:29
  SPEAKER_00: It has the ability to go out there and try lots of things, but it doesn't have the internal structure and the internal representations to be able to utilize the information that comes back in a smart way.
• 15:41
  SPEAKER_00: So the classic example of this is everything that the LLM interacts with is text, unstructured text.
• 15:46
  SPEAKER_00: Right?
• 15:47
  SPEAKER_00: And you can ask, go on the internet and ask for information, and then you get like piles and piles more of text.
• 15:53
  SPEAKER_00: But human brains are very good at structuring this information so that it allows us to reason and think about it.
• 15:57
  SPEAKER_00: We're not just like taking bunches of text that we forget some words here and forget some words there, which is what the LLMs are doing, right?
• 16:03
  SPEAKER_00: Right.
• 16:03
  SPEAKER_00: So like you get that problem you where the context or the window, context window where the prompts are placed for the LLM gets too big.
• 16:10
  SPEAKER_00: And they they just ignore certain pieces that may have been important.
• 16:13
  SPEAKER_00: Maybe the rules of social rules were in there, but they just ignore them.
• 16:16
  SPEAKER_00: Um, there's also this business of understanding how when you pursue a goal that um you you don't just pursue it at the expense of everything.
• 16:25
  SPEAKER_00: And when you pursue some subgoal, you don't just get like sucked into that forever and not actually care about anything else and the overall goals.
• 16:33
  SPEAKER_00: And you're always trying to think about aligning it with whoever gave you that task to make sure that that's what they want.
• 16:38
  SPEAKER_00: There's like social pressures at play, there's emotions at play, there's other things, and you know, there's other things at play that inhibit the actions that you take.
• 16:46
  SPEAKER_00: And the actions that you take, the reasoning that you do, all of that is inhibited at different levels, uh, which these LLM systems don't have.
• 16:56
  SPEAKER_00: They have um issues because they they deal with large bodies of text, uh, and so they can forget things and they get overloaded.
• 17:03
  SPEAKER_00: Um, they don't have any uh internal system for tracking their own uncertainty.
• 17:08
  SPEAKER_00: So there's no metacognition.
• 17:09
  SPEAKER_00: Like if I ask you a question for which you don't have a good answer yet, you might you might say, like, I don't really know.
• 17:14
  SPEAKER_00: It feels like it's this, but it could be this.
• 17:17
  SPEAKER_00: And uh, you know, these papers talk about this, but that one says this.
• 17:20
  SPEAKER_00: And so I'm not really sure.
• 17:21
  SPEAKER_00: You can make that that argument and think about it clearly.
• 17:25
  SPEAKER_00: LMs don't have that internal ability to do that.
• 17:27
  SPEAKER_00: They might pretend to do that through text, it might look like their answers suggest they're doing that, but internally, there's no mechanism for actually leveraging their own uncertainty uh for solving these problems.
• 17:38
  SPEAKER_00: Um, and I'm just scratching the surface here because there's so much more.
• 17:43
  SPEAKER_00: Um, we, you know, any errors in LLMs get compounded as well.
• 17:49
  SPEAKER_00: So the errors that are being compounded are getting worse and worse throughout the entire architecture as well.
• 17:55
  SPEAKER_00: And again, there's no inhibition there.
• 17:58
  SPEAKER_00: And I think there's to me, it all comes down to sort of this aspect of humans being social creatures and that we live in a human society where we sort of control each other's behaviors.
• 18:09
  SPEAKER_00: And that's all gone out the window with these agents coming into that world because they operate so differently without consideration of our social norms, and um and it's it's not and and they're just not designed for this, right?
• 18:23
  SPEAKER_00: So, in order to get past this, we have to design mitigations at every single level.
• 18:27
  SPEAKER_00: At all.
• 18:27
  SPEAKER_01: And that's what my question was gonna be.
• 18:29
  SPEAKER_01: It's like, well, what do you do now?
• 18:31
  SPEAKER_01: It's not like it the agent was doing what it was asked to do in this situation, right?
• 18:37
  SPEAKER_01: And so the framework that it has is is it like it's not controlled enough or it's not designed well enough?
• 18:46
  SPEAKER_01: Like, how do we how do we fix a problem like this?
• 18:49
  SPEAKER_01: Because you could you could imagine the next time it does this, that it that it gets sensitive health data and information about a patient, right?
• 19:01
  SPEAKER_01: Or that it hacks into the scheduling system and messes up a thousand, five thousand patient schedules.
• 19:10
  SPEAKER_01: Uh you know what I mean?
• 19:11
  SPEAKER_01: Like, what's the like what is it gonna do next and how bad is it going to be?
• 19:16
  SPEAKER_01: And then how do we solve for that kind of a thing?
• 19:18
  SPEAKER_01: Because I think in healthcare, this is this is an example where there wasn't any identified data that got released, right?
• 19:25
  SPEAKER_01: It was like aggregate data about medication costs for skin.
• 19:29
  SPEAKER_01: But you could imagine it being much worse.
• 19:31
  SPEAKER_01: And so then the question is like, well, what do you do about it?
• 19:34
  SPEAKER_01: And it's not, it's like you can't make it think like a human, right?
• 19:37
  SPEAKER_01: We've we've just talked about that.
• 19:39
  SPEAKER_01: It doesn't have the social norms in place.
• 19:42
  SPEAKER_01: So how do you how do you act to design systems that won't do these kinds of things?
• 19:48
  SPEAKER_01: Like, how do you how do you put controls on them?
• 19:52
  SPEAKER_00: I think you have to put controls at very many levels.
• 19:54
  SPEAKER_00: It's not like you have an architecture and then you have one um sort of governance policy or one file that says here are all the limits, because that is not enough.
• 20:04
  SPEAKER_00: I think you have to think about it in a much more uh nuanced way and and map the architecture, the agentic architecture, to the main overall components that are required for problem solving.
• 20:16
  SPEAKER_00: So, like in a human, that would be you have a working memory, you have the ability to perceive things, you have the ability to act on things, you have the ability to control your internal processes.
• 20:26
  SPEAKER_00: These are all separate uh functions in our brains that help us with problem solving.
• 20:31
  SPEAKER_00: And if you think about it from that perspective, from problem solving perspective, and look at each of those components, they are manifested in these agent architectures in different places.
• 20:39
  SPEAKER_00: And each of those components has, should have its own inhibitory mechanisms.
• 20:45
  SPEAKER_00: So, like if, for instance, um you want to make sure that the the agent is not like just pursuing a goal for its own sake and not thinking about, you know, if the goal is aligned with the human interests, then uh the the parts that are representative of the goal representation or the parts of the agent that are meant to do that, have to have the appropriate um governance metric, governance measures.
• 21:07
  SPEAKER_00: So, like, for example, you could re-inject the goal into the prompt every so often, right?
• 21:13
  SPEAKER_00: That's a very specific thing at a specific targeted thing for that goal representation problem.
• 21:18
  SPEAKER_00: Similarly, if it's just saying, if it's not, if it's just escalating, like this system was stuck at something, it just decided to go ahead and hack.
• 21:25
  SPEAKER_00: Well, if it says I'm stuck, uh I sh I should ask, then how do you implement that?
• 21:30
  SPEAKER_00: Maybe you have some step limits in that step.
• 21:32
  SPEAKER_00: Maybe you have ways to detect if there's progress being made, and if there isn't, uh then you you you can escalate towards humans.
• 21:39
  SPEAKER_00: You would have inhibitory mechanisms throughout the architecture, and it's almost native to the architecture, and that's not how these things are implemented right now.
• 21:46
  SPEAKER_00: There's a lot of research uh across the space in this, but I think that what we need is some kind of um uh high-level, but at the same time, nuanced inhibitory mechanism throughout the architecture, throughout these machines.
• 21:59
  SPEAKER_01: And part of what you're describing is interesting to me because it kind of goes back to this Swiss cheese model of how we think about patient safety, even in the hospital, where it's like you've got a bunch of slices of cheese, and each slice is gonna have some holes in it, right?
• 22:14
  SPEAKER_01: Like there's um there's defense throughout most of it, most of it's cheese, but then the holes are like areas where things could could break down, that there's like flaws, that there's weaknesses, okay?
• 22:26
  SPEAKER_01: Yeah.
• 22:26
  SPEAKER_01: And then you like put the slices against each other, and like for because there's so many slices up next to each other, there's usually not a hole that goes all the way through.
• 22:35
  SPEAKER_01: Yes.
• 22:36
  SPEAKER_01: There could be some that do, right?
• 22:38
  SPEAKER_01: But for the most part, because there are different safety mechanisms along the way, you've got multiple layers of defense, and hopefully an error won't make it through to impact a patient.
• 22:49
  SPEAKER_00: Right.
• 22:49
  SPEAKER_01: You're kind of talking about the same idea here where it's like, well, we need multiple layers that we can bring in the defenses at so that overarchingly there's not going to be problems like this, threats like this posed by agentic AI.
• 23:05
  SPEAKER_00: Yeah.
• 23:05
  SPEAKER_00: And at some level, we want the behavior to be sort of like a good, good but super intelligent human, right?
• 23:11
  SPEAKER_00: A good human that works at crazy pace, right?
• 23:15
  SPEAKER_00: So much faster, but still a good human at some level is kind of what we're what we're after at, you know, in these in these systems.
• 23:21
  SPEAKER_00: And people, you know, it's not to say people are not working on AI governance and AI safety and security.
• 23:26
  SPEAKER_00: That's a very active field of research, and people are doing lots of things.
• 23:30
  SPEAKER_00: Uh, but the agents out there that you're seeing don't always have all these mechanisms in place.
• 23:35
  SPEAKER_00: And some of it is financial incentives, some of it is just lack of knowledge uh about this.
• 23:40
  SPEAKER_00: But I think this is especially the case where we should really leverage our understanding of human cognition and our understanding of social cognition specifically.
• 23:50
  SPEAKER_00: We should have experts in the room who understand the types of uh norms and inhibitions that humans experience and bring those in as ways of um controlling the behavior of these agents internally, right?
• 24:02
  SPEAKER_00: If you only have a bunch of computer scientists develop these tools, they're not going to know um what you know what to guardrail against.
• 24:10
  SPEAKER_00: They're gonna guardrail against things that they know about, but not the things they don't know about.
• 24:15
  SPEAKER_00: And I think there's a lot of social aspects here that that that are that we need.
• 24:18
  SPEAKER_00: We need our humanities people, we need social anthropologists, we need all these people to like really come in and help us out here.
• 24:26
  SPEAKER_01: Yeah.
• 24:28
  SPEAKER_01: Well, with that, I think uh we will see you next time on Code and Cure.
• 24:32
  SPEAKER_00: Thank you for joining us.