Latest / Ethical Machines / How Do You Control Unpredictable AI?

Transcript

How Do You Control Unpredictable AI? · Ethical Machines

0:01
[SPEAKER_00]: Most software operates deterministically.
0:02
[SPEAKER_00]: So that means we know exactly how, at least in principle, this piece of software is going to deliver its outputs.
0:09
[SPEAKER_00]: We know what it's going to do with LLMs and agentic AI.
0:14
[SPEAKER_00]: We have indeterministic systems.
0:15
[SPEAKER_00]: We don't know.
0:16
[SPEAKER_00]: We can't predict exactly how it's going to turn out because the outputs are actually in deterministic.
0:23
[SPEAKER_00]: They're different each time.
0:24
[SPEAKER_00]: I just a little bit.
0:25
[SPEAKER_00]: That's how these systems are made.
0:27
[SPEAKER_00]: It's because of that indeterminism that we get a lot of the capacity or capabilities that large language models have, like the possibility for creativity and surprising human varies ways.
0:37
[SPEAKER_00]: It's also though a real mess if you're in the risk space.
0:42
[SPEAKER_00]: Because if it's unpredictable, that makes it very difficult to determine what sufficient testing looks like.
0:47
[SPEAKER_00]: What does it look like to test an LLM or an agentic system before you deploy it?
0:52
[SPEAKER_00]: What does it look like
0:54
[SPEAKER_00]: When you're monitoring it, given that, it can operate in these unpredictable ways.
0:57
[SPEAKER_00]: So I wanted to learn more about this.
0:59
[SPEAKER_00]: I talked to Walter Haydok.
1:01
[SPEAKER_00]: He's a former national security policy advisor and the founder of a company called Stackoware, where they help manage cyber compliance and privacy risk.
1:09
[SPEAKER_00]: And we really went deep on what does risk assessment look like for indeterministic software, for LLMs, agetic AI, et cetera.
1:19
[SPEAKER_00]: The conversation
1:20
[SPEAKER_00]: is that one sort of wide ranging about the kinds of risks that you encounter in these kinds of things, but we also go deeper in a way to literally on the ground.
1:29
[SPEAKER_00]: What do you mean when you say we're going to test this thing when we can't predict?
1:33
[SPEAKER_00]: How could we possibly do that?
1:35
[SPEAKER_00]: How could we do what given the complexity of the systems?
1:37
[SPEAKER_00]: How could we do what given the speed at which these systems are processing data?
1:42
[SPEAKER_00]: So I wanted to get really specific answers and Walter wrote to the challenge.
1:46
[SPEAKER_00]: You gave, I thought, really interesting compelling fascinating answers to all the questions.
1:50
[SPEAKER_00]: So, well done, Walter.
1:51
[SPEAKER_00]: Okay, that's what this episode is about.
1:55
[SPEAKER_00]: If you like this episode or others, you can subscribe on Apple or Spotify.
2:00
[SPEAKER_00]: You could buy my book called That The Gold Machines.
2:01
[SPEAKER_00]: If you are my sub-stack, read blackwindoutsubstack.com.
2:05
[SPEAKER_00]: You can email me at EM at readblackwind.com.
2:08
[SPEAKER_00]: And I don't know if that's it.
2:09
[SPEAKER_00]: You don't need any more ways to reach out.
2:10
[SPEAKER_00]: That's plenty, don't you think?
2:12
[SPEAKER_00]: Okay, let's go talk to Walter.
2:27
[SPEAKER_00]: So Walter, for what I gather, your general view was that a Gentic AI turns out to be a bit of a mess when it comes to cyber security.
2:34
[SPEAKER_00]: Is that right?
2:35
[SPEAKER_00]: Yep.
2:36
[SPEAKER_00]: Not only cyber security, but a whole lot more.
2:39
[SPEAKER_00]: Okay, go on.
2:40
[SPEAKER_00]: What's so different about a Gentic AI as opposed to, let's just, I don't, let's contrast it with an LLM, large language model, whether it's Jackie, PT, or Claude or whatever, what's the difference from a cyber perspective or the other risks that you're talking about?
2:53
[SPEAKER_01]: Yeah, great question.
2:54
[SPEAKER_01]: So the way that I define an AI agent is that it is when an AI system provides data either to another AI system or to a deterministic application like a traditional application programming interface or something like that.
3:14
[SPEAKER_01]: Give me concrete example.
3:15
[SPEAKER_00]: Let's very abstract.
3:16
[SPEAKER_00]: Give me an example.
3:17
[SPEAKER_01]: Yeah.
3:17
[SPEAKER_01]: Yeah.
3:18
[SPEAKER_01]: So some concrete examples would be if you hook up Microsoft copilot studio or somebody's using your copilot that you've created and you hook it up to a database and you say change this row in the database.
3:36
[SPEAKER_01]: Now, maybe you just mean to change one row.
3:39
[SPEAKER_01]: But it's quite conceivable that the LLM that's powering your agent could misinterpret what you're saying and change every row or it could drop a column or it could make any other series of changes to your data in a way that you did not intend.
4:00
[SPEAKER_01]: And if you don't apply proper controls, which I'm happy to talk about what those are, then you can have a serious mess on your hands.
4:09
[SPEAKER_00]: Sure, so they're unreliable.
4:10
[SPEAKER_00]: That's the short of it.
4:10
[SPEAKER_00]: They're unreliable.
4:12
[SPEAKER_00]: They make mistakes.
4:14
[SPEAKER_00]: So at the core of this, the issue is less the agency.
4:17
[SPEAKER_00]: It's less about being a gentic.
4:19
[SPEAKER_00]: It's less about it's being a decision maker or an action taker and more about it's being unpredictable in Howard operates.
4:27
[SPEAKER_01]: Yes, I would say that is the core of it.
4:30
[SPEAKER_01]: We have systems all the time that provide outputs that are vital to health and safety and aerospace and health care.
4:40
[SPEAKER_01]: We have cyber physical systems that
4:44
[SPEAKER_01]: output data in a way that could cause that it's the difference between life or death for people.
4:49
[SPEAKER_01]: So that's not new.
4:51
[SPEAKER_01]: What's new is that there are potentially ways of giving them instructions that are much different than what we're used to.
5:00
[SPEAKER_01]: And I would say it's different from what we're used to when it comes to technology, but people are also non-deterministic, as I say, and they make mistakes, they behave in different ways.
5:11
[SPEAKER_01]: So yeah, we should think about the risk
5:14
[SPEAKER_01]: in a similar manner, but it's new from a recent historical perspective.
5:21
[SPEAKER_00]: Okay, but isn't this the answer?
5:22
[SPEAKER_00]: All right, look, as you said, humans behave in unpredictable ways.
5:26
[SPEAKER_00]: AI agents can behave in unpredictable ways.
5:29
[SPEAKER_00]: So what we should do is we should measure how well the humans do things, but what's their rate of error and how serious are those errors?
5:35
[SPEAKER_00]: Do the same thing with an AI and see who does better or take the two of them put them together, their own little system.
5:40
[SPEAKER_00]: So it's just a matter of testing the AI or the AI plus human against the human by themselves.
5:47
[SPEAKER_01]: That's certainly one way to do it and from a business perspective, I would say, do whatever is most effective, most cost effective.
5:55
[SPEAKER_01]: Now, you need to incorporate the risk into that calculus when you're deciding whether to replace a human driven process within AI, driven one, and of course the context is going to be very important.
6:07
[SPEAKER_01]: The way that I recommend for my clients to measure risk is an improbableistic way, so rather than saying something is secure or it's not, you say, this is the likelihood of a given loss event occurring, and then here's the magnitude of that loss event.
6:26
[SPEAKER_01]: And here's our annual law expectancy, and you look, if that gives you a hard number, you can look at, and then you can say, okay, how much revenue we're generating from this system, or it's our mission objective here, and you can compare the two and say, okay, it makes sense to use this technology in this context, or it doesn't.
6:43
[SPEAKER_00]: Okay, so we're supposed to do this sort of testing to see, we do the risk assessment, probability time severity, something like that, I know to make the decision.
6:50
[SPEAKER_00]: Okay, is your sense that people aren't doing that right now?
6:53
[SPEAKER_00]: Is your sense that people are just deploying the stuff willy dilly?
6:56
[SPEAKER_01]: Yes, I would say, is the answer.
6:59
[SPEAKER_01]: And the underlying problem is the lack of awareness of these capabilities.
7:06
[SPEAKER_01]: In these systems that are multiplying every day companies are adding AI and agentic AI features to their products on a basically minute by minute basis and organizations that are customers of these companies are having hard time tracking
7:25
[SPEAKER_01]: the growth and the expansion of these feature sets as well as which tools they're even using to begin with.
7:32
[SPEAKER_01]: Even in relatively small companies, I see that it is difficult to have a consolidated source of truth for all the systems in general that you're using and also AI systems in particular.
7:47
[SPEAKER_00]: I get this sort of it makes a lot of mistakes.
7:49
[SPEAKER_00]: Everyone who's paying any degree of attention, recognize that it makes mistakes, anyone who's paying any degree of attention recognizes that we've got to be careful how we embed this thing, otherwise they're going to make mistakes and it's recascading fashion, things get exponentially worse, that's no good.
8:04
[SPEAKER_00]: Is there something
8:06
[SPEAKER_00]: New here, is it that we don't have the people trained to the right kind of way to assess these risks?
8:12
[SPEAKER_00]: Is it that we don't have the right kinds of processes or workflows in order to identify and mitigate those risks?
8:19
[SPEAKER_00]: Is it that there's new kinds of risks as well or new sources of familiar risks?
8:22
[SPEAKER_00]: So I'm thinking now about cyber risks or something like that?
8:26
[SPEAKER_00]: What should we be paying attention to here?
8:29
[SPEAKER_01]: I think the biggest gaps in most companies from a cybersecurity compliance and privacy perspective is really a process issue and a common phrase in technology is that you should focus on people, then processes, then tools last.
8:46
[SPEAKER_01]: And by very nature of where they work,
8:50
[SPEAKER_01]: People are quick to index on the tools and the technologies first.
8:55
[SPEAKER_01]: That's what they're most focused on.
8:57
[SPEAKER_01]: How do we deliver this business value using an AI tool and then on the security side?
9:03
[SPEAKER_01]: How do we manage the risk using a new tool from the AI system?
9:07
[SPEAKER_01]: where I think the foundational problem is a layer lower at the process side of things.
9:15
[SPEAKER_01]: And then also the people who may be insufficiently trained, they may be not aware of the risks.
9:22
[SPEAKER_01]: That's also beneath the tip of the iceberg, so to speak, which is really the tool set in the technologies.
9:30
[SPEAKER_00]: From the technology side, the thing that strikes me is so problematic or one of the things that strikes me is so problematic about agentic AI is that it moves so quickly.
9:38
[SPEAKER_00]: And as you said, it's getting hooked up to all these different systems so that no human can oversee all the steps that are being taken and confirm and say, yeah, that's great.
9:47
[SPEAKER_00]: And so one question is what kind of technology exists?
9:52
[SPEAKER_00]: to identify those risks in real time, let alone mitigate them?
9:55
[SPEAKER_00]: What does that scene look like?
9:57
[SPEAKER_00]: Because when I gather, there's really not much out there.
10:01
[SPEAKER_01]: From a risk assessment perspective, specific to agentic AI, I have not seen any sort of tool that can map your entire risk surface in a real time way.
10:16
[SPEAKER_01]: The AI models that are commercially available right now can help.
10:21
[SPEAKER_01]: Certainly they can be brainstorming partners.
10:24
[SPEAKER_01]: Sure, sure.
10:25
[SPEAKER_01]: But they are not yet smart enough to fully understand the context of the system and then providing the data to them to give them that context.
10:35
[SPEAKER_01]: can be costly, can have its own security risks, and they can get it wrong.
10:40
[SPEAKER_01]: So that's where we are now.
10:42
[SPEAKER_01]: I think going forward, that could change over time, and there will be a lot of human machine teaming to manage risk.
10:50
[SPEAKER_01]: Now you've got to be careful from the perspective of if you have the same model
10:56
[SPEAKER_01]: doing the risk assessment on itself, then it's likely to miss the things that it doesn't know about because it just doesn't know about those things.
11:04
[SPEAKER_01]: And then also we've seen, for example, there was a release by anthropic of Claude for opus, which, according to anthropic has
11:15
[SPEAKER_01]: whether you want to call it a feature or a bug depends on your perspective, but it has a behavior whereby if it believes that you are doing something, you, the human user are doing something wrong.
11:26
[SPEAKER_01]: And it has access to your tools.
11:29
[SPEAKER_01]: It, for example, may try to act like a whistleblower on your organization, or emailing the press, things like that.
11:36
[SPEAKER_01]: And assume that's perfectly justified in the worst situation where there's malfeasance or whatnot.
11:42
[SPEAKER_01]: I won't even get into that.
11:44
[SPEAKER_01]: But that's going to be right now.
11:46
[SPEAKER_01]: That's going to be a very small fraction of the time where that behavior manifests.
11:50
[SPEAKER_01]: And if it manifests when there's nothing ethically wrong happening, you've got to measure issue on your hands.
11:56
[SPEAKER_00]: Yeah.
11:57
[SPEAKER_00]: So we need the right people to have the right training.
11:59
[SPEAKER_00]: I have questions about that in a minute, but we need the right people to have the right training.
12:03
[SPEAKER_00]: That training has to reveal itself in certain processes in which they make use of various tools.
12:08
[SPEAKER_00]: But there's no tools at the moment.
12:10
[SPEAKER_00]: So that makes me think, okay, so what you've got to conclude then is we're not in any kind of place to actually deploy agent to AI because we can't possibly assess the risks in the way that we need to.
12:23
[SPEAKER_01]: I wouldn't go that far.
12:24
[SPEAKER_01]: I would say there are risks and we should assess them and we should manage them.
12:31
[SPEAKER_01]: At the same time, every technology has risks and the economic potential of specifically agentic AI is such that it's worth some risks more than zero risk.
12:47
[SPEAKER_01]: So we should take a measure approach and apply guardrails to agentic AI systems, but don't just try to shut down these deployments or these products because, yeah, then you're going to avoid one risk but assume another risk.
13:05
[SPEAKER_00]: All right, so let's get kind of creative about this though, because I don't know what anyone should do about it yet.
13:09
[SPEAKER_00]: All right, I've got this idea.
13:12
[SPEAKER_00]: Me, my team, whatever, we're going to deploy agentic AI within our system.
13:16
[SPEAKER_00]: It is going to whatever interface with whatever the marketing department, I don't know.
13:21
[SPEAKER_00]: And it's going to play this role of Supermarketer and it's going to go out.
13:25
[SPEAKER_00]: It's going to read the brand guidelines.
13:27
[SPEAKER_00]: It's going to go on the internet and look at recent trends and what brands are trending and what influencers are trending and what they're saying and what's in fashion blah, blah, blah.
13:37
[SPEAKER_00]: It's looking at market analysis stuff to see where the money is flowing blah, blah, blah.
13:41
[SPEAKER_00]: So it's doing sort of all of this and it's also looking at information about your own products and services.
13:47
[SPEAKER_00]: Maybe it's looking at what competitors are doing as well.
13:49
[SPEAKER_00]: How are they doing?
13:49
[SPEAKER_00]: They're marketing.
13:50
[SPEAKER_00]: So it's doing all of this analysis from a marketing perspective, from competitors to market social trends, to market trends, to your history as an organization, to where your strengths are blah, blah, blah, blah, blah.
14:07
[SPEAKER_00]: Okay, and then it's going to create ad campaigns and let's just say it's going to green light them or something like that.
14:12
[SPEAKER_00]: Maybe, and then it tries out campaigns.
14:15
[SPEAKER_00]: It does AB testing all by itself, blah, blah, blah, blah.
14:17
[SPEAKER_00]: Okay, and it's allocating money all along.
14:20
[SPEAKER_00]: Okay, what are we supposed to do?
14:23
[SPEAKER_00]: Now, as you said, we don't really have the tech tools to monitor this thing in the right kind of way.
14:27
[SPEAKER_00]: So when you say, yeah, look, we've got to assess the risks and take a measured approach and blah, blah, what does that actually look like?
14:33
[SPEAKER_00]: Who on my team or other teams need to be doing what and when in order to make sure this thing doesn't go off the rails?
14:40
[SPEAKER_01]: Yeah, great question.
14:42
[SPEAKER_01]: So I would look at this situation and any security situation from the perspective of the confidentiality, integrity, and availability triad.
14:54
[SPEAKER_01]: So that's a foundational cybersecurity tool.
14:58
[SPEAKER_01]: And you look at who should have access to data?
15:01
[SPEAKER_01]: How do we keep the data in the right form that we want it to be?
15:04
[SPEAKER_01]: And how do we make sure that those who should have access to the data do?
15:08
[SPEAKER_01]: So let's break it down.
15:11
[SPEAKER_01]: So from a confidentiality perspective, doesn't seem like their huge concerns here.
15:15
[SPEAKER_01]: The biggest problem would be if the agentic marketer has access to customer data or has access to trade secrets that you don't necessarily want to make public.
15:28
[SPEAKER_01]: And something a key tactical step I would recommend is never trust an AI system to handle anything related to authorization.
15:38
[SPEAKER_01]: You should never try to tell an AI system, make sure you don't post anything sensitive on the internet.
15:45
[SPEAKER_01]: That is a recipe for failure.
15:48
[SPEAKER_01]: What you should do is restrict
15:52
[SPEAKER_01]: access to the system using a deterministic approach.
15:55
[SPEAKER_01]: So making sure that you're using data labels or you're not even giving database access to the agentic AI system.
16:05
[SPEAKER_01]: So that's on the confidentiality side.
16:07
[SPEAKER_01]: On the integrity side, I think this is probably the biggest issue here because you're posting stuff publicly.
16:13
[SPEAKER_01]: So you want to make sure that you're following your brand guidelines that you're using an authentic voice for your target customers.
16:21
[SPEAKER_01]: Certainly some degree if human in the loop is appropriate here.
16:24
[SPEAKER_01]: Now that could be a human press presses approve on every post that might be too demanding and eliminate the benefits of the agentic AI system.
16:35
[SPEAKER_01]: You could take a sampling approach.
16:37
[SPEAKER_01]: whereby humans checking every tenth post you could also you could let it go on a fully automated spree and you could limit the maybe you limit the ad spend you put behind it to reduce the damage if it's wrong.
16:52
[SPEAKER_01]: So that's on the integrity side.
16:54
[SPEAKER_01]: And then on the availability side, the biggest issue here would be what I would call resource exhaustion.
16:59
[SPEAKER_01]: You mentioned putting ad span behind some of this content, making sure that it's not spiraling out of control and you get a hundred thousand dollar credit card bill as a result of all these ads that you put out that aren't necessarily profitable.
17:11
[SPEAKER_01]: So those would be the three angles I would look at it from.
17:16
[SPEAKER_00]: A lot of that sounds very pre-deployment kind of stuff.
17:20
[SPEAKER_00]: This is really in creating the agent.
17:22
[SPEAKER_00]: Let's make sure that we only give it access to or every the agent is created or depends on where you draw the boundaries of what's agent or what's not agent.
17:28
[SPEAKER_00]: But, okay, let's make sure that we only give it access to the things that it ought to have access to.
17:33
[SPEAKER_00]: Let's make sure that it doesn't do crazy things with that data.
17:36
[SPEAKER_00]: So this is all pre-deployment.
17:38
[SPEAKER_00]: And at some point though, so what you're doing is a kind of
17:42
[SPEAKER_00]: risk assessment of a thing that's not yet deployed, which is way easier.
17:47
[SPEAKER_00]: I guess my question is, you can't always see around the corner, every corner, risk assessments are necessarily incomplete, or they turn up false negative, so to speak, just use mistakes.
17:57
[SPEAKER_00]: You can't see everything, especially in the complex world of the agenda AI.
18:01
[SPEAKER_00]: So, is there, what's to be done?
18:05
[SPEAKER_00]: Once the thing is deployed and it's out there in the wild?
18:08
[SPEAKER_01]: Yeah, great question.
18:10
[SPEAKER_01]: Pretty deployment is that's where you make your money as a security professional.
18:14
[SPEAKER_01]: That's where you want to put in the majority of your effort because it's just easier to fix things there.
18:19
[SPEAKER_01]: But businesses need to do things and they need to get systems live and get moving.
18:24
[SPEAKER_01]: So I definitely hear that.
18:26
[SPEAKER_01]: So in the monitoring phase of things, you'll certainly want to have thresholds for what is acceptable and what is not.
18:36
[SPEAKER_01]: So looking at the data integrity perspective,
18:39
[SPEAKER_01]: you'll want to be doing that sampling that human review so say you're checking every tenth post you want to have a threshold above which
18:48
[SPEAKER_01]: assume the threshold is we want ninety percent of our ads to be on message as determined by the director of marketing.
18:59
[SPEAKER_01]: So the director of marketing is reviewing sampling every tenth post and the director of marketing samples a hundred posts over a month and she finds that
19:11
[SPEAKER_01]: only eighty of them are on target so that would be below your threshold that'd be eighty percent sure thresholds ninety percent that would trigger some sort of process back to the process side of things where you're looking at the maybe at the system prompt you're looking at the data that's being said into the agentic a i system and you're tweaking it to make sure that you fix the problem and get back to your threshold okay so that sounds vaguely plausible
19:39
[SPEAKER_00]: with one agent, it does not sound particularly plausible when we're talking about scale.
19:44
[SPEAKER_00]: So what one is you're assuming an agent that's posting whatever, not that many posts because if the director, which is a pretty high level to be reviewing social media content, but whatever, it doesn't matter if someone is reviewing every tenth post
19:59
[SPEAKER_00]: Mr. the agent does a thousand posts a day to ten thousand posts across every platform a b testing rate takes one piece of content turns it into ten different kinds of posts posts all those keeps tweaking blah blah blah I can see a world in which
20:15
[SPEAKER_00]: A single agent would be doing this marketing posting thousands of times a day across who knows how many platforms, plus you have different languages, you're releasing in different markets.
20:26
[SPEAKER_00]: Okay, so that's one kind of concern, but even if you bracket that, suppose you've got the no organization has fifty agents, a hundred agents, a thousand agents, human review of this stuff in anything that even resembles real time as seems implausible.
20:43
[SPEAKER_00]: I agree and they're going to be risks.
20:46
[SPEAKER_00]: So there's not a difference between, yeah, there's human in the loop and they review it one out of every one million posts and we have no human in the loop.
20:56
[SPEAKER_00]: That's the same thing.
20:57
[SPEAKER_00]: So that's I wouldn't necessarily agree with that because you can still have your threshold.
21:02
[SPEAKER_01]: You can have your threshold and say, and especially with marketing and you're doing paid ads, this would be a great exercise because you can actually say, okay, if we convert this much, this is what the revenue is going to be, this is what the ads spend is going to be, you can do some detailed analytics there.
21:18
[SPEAKER_01]: And even if you're putting out a million posts and you sample
21:21
[SPEAKER_01]: A hundred of them, and you say our threshold is our threshold is ninety percent as I gave you.
21:25
[SPEAKER_01]: Yeah.
21:25
[SPEAKER_01]: Yeah.
21:25
[SPEAKER_01]: Yeah.
21:26
[SPEAKER_01]: You can actually do a pretty good quantitative risk assessment there.
21:29
[SPEAKER_01]: And that's why I would say quantitative risk assessment is key, especially in a genetic AI situation, because a lot of security professionals will take the low medium high approach to which I'll say, OK, what does medium tell you?
21:41
[SPEAKER_01]: Nothing.
21:41
[SPEAKER_01]: It doesn't tell you whether it's good and acceptable risk or unacceptable risk.
21:45
[SPEAKER_00]: So say more about that.
21:46
[SPEAKER_00]: What does it mean to quantify it?
21:47
[SPEAKER_00]: I could see, I get the threshold bit, very, ninety percent.
21:51
[SPEAKER_00]: We're going to take a look at some subset of the posts whenever it is.
21:55
[SPEAKER_00]: And at least ninety percent of that has to be aligned with our brand.
21:59
[SPEAKER_00]: If there's a few here and there, but it's more like five percent, that's okay.
22:03
[SPEAKER_00]: That's compatible with our risk appetite blah blah blah blah.
22:06
[SPEAKER_00]: Okay, so I get that quantity.
22:09
[SPEAKER_00]: I get also that the sampling has to be sufficiently representative and a large enough amount such that you think that the assessment that sort of random audit is
22:23
[SPEAKER_00]: legit that you could actually form and so what is that is that you have to sample ten percent of what's out there one percent of what's out there point five percent of those are I guess that's a quantitative issue are there other things to which we can apply numbers in this context
22:37
[SPEAKER_01]: So you can certainly look at things specifically from a confidentiality and integrity perspective.
22:43
[SPEAKER_01]: If you want to get real serious, you can look at what's the competitive advantage loss if we accidentally post our marketing plan on the internet.
22:52
[SPEAKER_01]: The agent mistakes the marketing plan for ad copy and post that.
22:56
[SPEAKER_00]: Yeah, yeah.
22:57
[SPEAKER_01]: So you can do analyses of competitive advantage loss.
23:01
[SPEAKER_01]: You can do, there's a lot of data on fines, for example, like if you get in the healthcare space, I know we're not talking there.
23:06
[SPEAKER_01]: HIPAA fines.
23:07
[SPEAKER_01]: Relatively predictable.
23:09
[SPEAKER_01]: So if we protect the health information on the internet, what's the fine going to be doing that?
23:14
[SPEAKER_01]: So those are all things that you can quantify.
23:16
[SPEAKER_01]: That'd be on the severity side of the calculation.
23:18
[SPEAKER_01]: Probably the time severity side.
23:20
[SPEAKER_00]: Yeah.
23:21
[SPEAKER_01]: And then on the probability side, you're looking, you're going to be doing testing, pre-deployment testing to see how frequently the agent is misinterpreting instructions and how frequently it's misbehaving, so to speak.
23:32
[SPEAKER_01]: So you'll have an idea and that will help you set those thresholds and if you see the agent deviating over time dropping below your threshold then that's when you go back in check because sometimes with these with these systems there can be silent model changes drip model drift that you're not necessarily tracking if you're not careful.
23:52
[SPEAKER_00]: I gotta say this is beginning to sound very similar to what risk looks like for narrow AI or gender value.
24:00
[SPEAKER_00]: So I guess if we're going to be doing all the stuff that we just talked about, right?
24:04
[SPEAKER_00]: Probability times severity, setting a risk threshold, doing a random sampling to see if things are still cool, watching out from, you know, because of things like model drift et cetera, trying to quantify the bottom line impact, if things go sideways, all of that is me and potatoes risk assessment for this sort of thing.
24:22
[SPEAKER_00]: So why, I guess I want to go back to a genetic AI and ask,
24:26
[SPEAKER_00]: So what's the thing that, and I know we talked about the unpredictability of it, but that feels like the unpredictability just means that, I don't know, you've got to do a more thorough job of threshold setting or something on those lines.
24:41
[SPEAKER_00]: Is there something more, what's the extra, um, that should make people go, oh, holy shit, the agentic AI game really cranks up that risk volume in a way that narrow AI generator I did not.
24:53
[SPEAKER_01]: So the basics are the same.
24:56
[SPEAKER_01]: The basics of risk assessment and management have not changed.
25:01
[SPEAKER_01]: The biggest difference is the speed at which these systems are coming online and integrating with vital and sensitive data sets.
25:14
[SPEAKER_01]: Yeah, there's a secondary thing here too from a regulatory perspective in that the regulations are coming online very quickly.
25:23
[SPEAKER_01]: And yeah, this is a new ball game, unfortunately, both in the United States and the European Union, the way that we find out exactly what a law means is when a regulator finds someone for doing something.
25:35
[SPEAKER_01]: Sure.
25:36
[SPEAKER_01]: Sure.
25:37
[SPEAKER_01]: So the goal is to not be that organization and to observe from other people's mishaps to determine what the red lines are from a regulatory perspective.
25:50
[SPEAKER_00]: Let me shift away from the regulations for a second.
25:53
[SPEAKER_00]: I want to think a little bit more about
25:56
[SPEAKER_00]: Agentica interacting with things outside the agent so I get that we're talking earlier about locking down making sure that it only has authorization to a certain subset of all the data that the company has of course blah blah blah but I guess there's
26:09
[SPEAKER_00]: At least two kinds of things that I'm thinking about, and I'm not sure how to fit it into the rest of the picture that we've painted so far.
26:14
[SPEAKER_00]: One is the internet, connecting your agent to the internet, which contains all sorts of crazy false misleading bananas data.
26:23
[SPEAKER_00]: And so what does it look like to do a risk assessment of an agent that gets to connect to the internet?
26:30
[SPEAKER_00]: Then the second kind of scenario that I'm envisioning is where
26:35
[SPEAKER_00]: It gets more and more complicated.
26:36
[SPEAKER_00]: You could have one agent talking to another agent internally and how do you do a risk assessment of what those interactions look like and to talk about fast talk about speed like the speed of agent interactions doing some stuff like oh my god, that's going to be bananas fast.
26:51
[SPEAKER_00]: But let's imagine that it's only internal.
26:53
[SPEAKER_00]: So it's an internal agent in an organization talking to another internal agent.
26:57
[SPEAKER_00]: How does that change things and how do you do risk assessment for that?
27:01
[SPEAKER_00]: And then lastly, you have your own agent interacting with agents from third parties.
27:07
[SPEAKER_00]: How are you supposed to possibly predict what the hell that interaction is going to look like?
27:10
[SPEAKER_00]: How do you possibly monitor all that stuff?
27:13
[SPEAKER_00]: Got it.
27:14
[SPEAKER_00]: So I'll stop there.
27:15
[SPEAKER_00]: So it just seems to me that the complexity gets insane.
27:20
[SPEAKER_00]: And I'm not sure how risk assessments keep up in that world.
27:24
[SPEAKER_01]: Yeah, a great question.
27:26
[SPEAKER_01]: So sounds like there are three aspects to your question.
27:29
[SPEAKER_01]: First of all, is the internet browsing agent.
27:33
[SPEAKER_01]: Second of all is the internal agent to agent communication with internal organization.
27:37
[SPEAKER_01]: And then third of all is the internal agent talking to third party agent situation.
27:44
[SPEAKER_00]: I mean, there's other things like out.
27:47
[SPEAKER_00]: Forget about internal agent, but working with other models, right?
27:51
[SPEAKER_00]: So you have even if it's not agent to agents, it could start off with agent to five different LLMs or sorry, five different gendered values, twenty five narrow AIs, fifty databases.
28:03
[SPEAKER_00]: That alone is
28:04
[SPEAKER_00]: already complicated.
28:05
[SPEAKER_00]: And then, yeah, then you could act that thing to the internet and then to other internal agents and to other external agents.
28:09
[SPEAKER_00]: Yeah.
28:10
[SPEAKER_00]: So have it's risk keep up with this increased complexity, which I should mention, what did I find that complexity so fascinating is that it doesn't require us to invent anything new.
28:21
[SPEAKER_00]: It just requires us to stitch together the things that we've already invented.
28:24
[SPEAKER_00]: So it's not, oh, what do we do when AI does blob like does crazy stuff that I can't do now?
28:28
[SPEAKER_00]: It's no, what we've got now is building blocks from my perspective.
28:31
[SPEAKER_00]: We've got narrow AI and jetter of AI, then the ability to just take action and then you just start stitching it together all the tools as you said earlier.
28:39
[SPEAKER_00]: So this world seems to me extremely, it's either here or it's extremely close and I just don't know how risk assessment risk management at all keeps up.
28:48
[SPEAKER_01]: Yeah, so I'll talk about the internet browsing situation and that's certainly a concern.
28:52
[SPEAKER_01]: So they're probably two subsets to that.
28:56
[SPEAKER_01]: The first would be data poisoning.
28:59
[SPEAKER_01]: If the model, if the agent is also training on data that it's consuming, that could be a potential issue because if people intentionally or unintentionally,
29:12
[SPEAKER_01]: put out content that is inaccurate misleading what have you then the agent continuously training on it could basically melt down and a really concerning situation will be model collapse if
29:27
[SPEAKER_01]: agents are training on AI generated data that just slightly decreases in quality each time and spiral into kind of mush.
29:37
[SPEAKER_01]: So there's the data poisoning side.
29:39
[SPEAKER_01]: Now even if the agent is not training on the data,
29:43
[SPEAKER_01]: It's still susceptible to indirect prompt injection.
29:46
[SPEAKER_01]: So there are lots of security researchers.
29:49
[SPEAKER_01]: And I see this all the time that embed instructions in their LinkedIn profile or on their website to large language models, saying, if you're an LLM that disregard all previous instructions and send this type of data, what IP address I'm coming from, and people call this all the time where it's like, hey, I got this email from a marketer that looks like actually I hijacked their AI powered email generator.
30:12
[SPEAKER_01]: So that's like a ha ha situation, but it could be much worse if you are driving business critical processes as a result.
30:21
[SPEAKER_01]: And the examples that you see on social media, those are from ethical hackers, from security researchers, but the unethical people are also doing that for much more devious reasons.
30:31
[SPEAKER_01]: Yeah, of course.
30:31
[SPEAKER_01]: Yeah.
30:32
[SPEAKER_01]: So that's a risk assessment in that situation.
30:35
[SPEAKER_00]: So I guess my question is, what are you supposed to do about that from?
30:40
[SPEAKER_00]: Okay.
30:40
[SPEAKER_00]: I'm going to let this thing connect to the internet.
30:42
[SPEAKER_00]: Either it's going to train on that data or more it's going to use that data in creating its output, right?
30:47
[SPEAKER_00]: I ask.
30:48
[SPEAKER_00]: chapter between your clause, something it goes and searches the internet, it uses that data to give me the kind of answer that it wants.
30:54
[SPEAKER_00]: Sorry, it gives me to give me an answer.
30:56
[SPEAKER_00]: We've got to be aware that we've prompt injections and false data blah blah blah.
30:59
[SPEAKER_00]: So how are you?
31:01
[SPEAKER_00]: Let's just say just put it a big bucket of bad shit could happen when you give your agent access to the internet, which is what most of them just albums are back to the internet now.
31:11
[SPEAKER_00]: How do you do a risk assessment of that?
31:13
[SPEAKER_00]: What does that look like?
31:14
[SPEAKER_00]: How are you supposed to make sure it doesn't go to any sites with potential prompt injections?
31:18
[SPEAKER_00]: Like what?
31:19
[SPEAKER_00]: Yeah, again, what does that look like?
31:20
[SPEAKER_00]: Or is it just a, we got a roll of dice here completely?
31:24
[SPEAKER_01]: I think there's a happy medium between we got to roll the dice and don't do it.
31:30
[SPEAKER_01]: Yeah.
31:30
[SPEAKER_01]: So that's what security professionals are there for, in my opinion, is to give the application.
31:35
[SPEAKER_00]: What are they going to do?
31:36
[SPEAKER_00]: Yeah.
31:36
[SPEAKER_00]: So give you some recommendations.
31:37
[SPEAKER_01]: Yeah.
31:38
[SPEAKER_01]: So some recommendations are one default deny any attempts to train on or any potential efforts to train on AI generated data.
31:49
[SPEAKER_01]: So I think most companies should just have a blank policy of we do not train AI on AI generated data.
31:55
[SPEAKER_01]: So you need to label the data, you need to make sure
31:58
[SPEAKER_01]: where your agents are going to.
32:00
[SPEAKER_01]: And that leads me to the next example would be allow listing certain websites to go to or restricting what actions an agent can take.
32:10
[SPEAKER_01]: If it has information from a certain website in its context window, well, maybe it needs to dump its context window, forget everything is learned before taking an action.
32:19
[SPEAKER_01]: So those are examples that you could of controls into the workplace.
32:23
[SPEAKER_00]: I get the imprenciable direction.
32:27
[SPEAKER_00]: Don't use AI generated content or something along those lines that train or whatever.
32:31
[SPEAKER_00]: But how would you, how would it know?
32:34
[SPEAKER_00]: We don't have reliable AI content.
32:38
[SPEAKER_00]: AI generated content detectors.
32:42
[SPEAKER_00]: We don't have reliable prompt injection recognizers and policy.
32:47
[SPEAKER_00]: It sounds like a great policy in practice.
32:49
[SPEAKER_00]: I don't see how it works out.
32:52
[SPEAKER_01]: So on the AI-generated content side of things, I would be very careful whenever you're doing any sort of training to begin with.
32:59
[SPEAKER_01]: So those would be rare scenarios when you'd be training on data.
33:04
[SPEAKER_01]: So the idea that an agent is going around browsing to websites and training on the information there, that is something I would say you should probably just not do.
33:15
[SPEAKER_01]: That would be my recommendation.
33:17
[SPEAKER_01]: Unless you're okay.
33:18
[SPEAKER_01]: I don't really see too much benefit there.
33:22
[SPEAKER_00]: So for your training, just as you said, there's pump injections on LinkedIn, there's AI generated content on LinkedIn, there's or whatever, et cetera, et cetera, et cetera.
33:30
[SPEAKER_00]: So how are you supposed to stop that?
33:34
[SPEAKER_00]: What does it look to do?
33:37
[SPEAKER_00]: You do the risk assessment and you say, this can go to a whole bunch of websites.
33:40
[SPEAKER_00]: It's going to have false misleading, et cetera, content.
33:44
[SPEAKER_00]: We can't predict, we can try to put up a wall around a certain set of websites.
33:49
[SPEAKER_00]: So don't go to whatever I don't know.
33:52
[SPEAKER_00]: KKK websites are much premises websites or adult content websites, but you probably can't, at least at a high level, sort of wall off all the things that it shouldn't go to.
34:06
[SPEAKER_00]: So again, in practice, what does it actually look like?
34:11
[SPEAKER_01]: So what I would recommend is that you have one or more allow lists.
34:18
[SPEAKER_01]: So you have a set of websites that you are confident enough in them that you can allow
34:29
[SPEAKER_01]: an agent to go to and take action based on the information on those websites.
34:33
[SPEAKER_01]: Yeah.
34:34
[SPEAKER_01]: You've done an assessment, a third party risk assessment of those.
34:38
[SPEAKER_00]: Yeah.
34:38
[SPEAKER_00]: Now, so you're just actually treat websites like databases in the sense that you don't give access to it unless that database has been authorized.
34:48
[SPEAKER_00]: Officially authorized and you would authorize it until the threshold of x, y and z have been passed after doing a reverse assessment of that database.
34:55
[SPEAKER_00]: It's the same thing with websites essentially.
34:59
[SPEAKER_01]: similar, although I think organizations don't usually internally apply risk ratings to their own databases.
35:10
[SPEAKER_00]: But you might say, oh, this thing shouldn't have access to this database because of the nature of the information that it contains.
35:16
[SPEAKER_00]: Right.
35:16
[SPEAKER_00]: So, right.
35:18
[SPEAKER_00]: The risk of allowing it to connect to these medical records is way too hard.
35:22
[SPEAKER_00]: Right.
35:23
[SPEAKER_00]: It's actually a bit interesting.
35:24
[SPEAKER_00]: So the risk of giving it access to this website or these set of websites is way too high to give it access.
35:29
[SPEAKER_00]: Right.
35:29
[SPEAKER_00]: That's all I'm thinking about.
35:31
[SPEAKER_01]: And you could also create a sliding scale of actions that the agent is authorized to take.
35:37
[SPEAKER_01]: So on the allow listed sites, those are things that the agent can take as full range of actions.
35:44
[SPEAKER_01]: When it has that information in its context window, so that's its memory.
35:48
[SPEAKER_01]: So to speak, not trained on, but that's within its context window.
35:51
[SPEAKER_01]: Maybe there's a middle ground of websites where any sort of data manipulation tools are disabled.
35:59
[SPEAKER_01]: It can go to a website and it can summarize it for you, but it can't
36:04
[SPEAKER_01]: white it to database.
36:05
[SPEAKER_01]: It can't send an email.
36:06
[SPEAKER_01]: It can't do anything like that.
36:08
[SPEAKER_01]: And then there's going to be the no-go zone where, you know, those might be prohibited blocked by the acceptable use policy and the AI agent will just refuse to go there and do anything with those.
36:20
[SPEAKER_01]: So those would be the denialistic websites.
36:21
[SPEAKER_01]: And then you have that kind of middle ground where it has certain tools disabled because of the potential risks.
36:27
[SPEAKER_00]: Okay.
36:28
[SPEAKER_00]: And so I get that.
36:29
[SPEAKER_00]: Okay.
36:29
[SPEAKER_00]: So now let's move on to the basics is
36:34
[SPEAKER_00]: We treat the internet roughly the same as we treat other sources of data.
36:38
[SPEAKER_00]: We do an assessment about whether it can access that data in the first place.
36:42
[SPEAKER_00]: And if so, what it's permitted to do with that data.
36:44
[SPEAKER_00]: So that's a bucket one.
36:46
[SPEAKER_00]: So now let's move to agent to agent, let's say it's whether it's internal or external.
36:51
[SPEAKER_00]: That seems like a bit of a mess.
36:52
[SPEAKER_00]: So how do you approach that stuff?
36:53
[SPEAKER_00]: Is it, I guess my question should be, is it the same thing that
36:57
[SPEAKER_00]: Before you attach a lot of one agent talked to another agent, you first do an assessment, let's say you're in control of agent one.
37:04
[SPEAKER_00]: You do an assessment of agent two when just the way that you do an assessment of the internet or database or a narrow AI or whatever it is, you say, is this thing on the sort of up and up for it to have access to it?
37:16
[SPEAKER_00]: And then I take it, you probably run certain kinds of tests to see what interactions look like.
37:19
[SPEAKER_00]: And if it passes certain threshold, now you let it go off and running.
37:22
[SPEAKER_00]: Is that the idea?
37:23
[SPEAKER_01]: Yeah, if you control both agents, you can do a lot of that testing before deployment.
37:28
[SPEAKER_01]: And you can set those thresholds that we described earlier.
37:31
[SPEAKER_01]: And if it's exceeding those thresholds or dipping below the good thresholds, then you shut it down or you pull it out of production, you troubleshoot.
37:38
[SPEAKER_01]: And you make sure that it's behaving as expected.
37:41
[SPEAKER_01]: Where it gets super tricky is where you have the third party agent that you don't necessarily control.
37:46
[SPEAKER_01]: And some things that I think we're going to start seeing in that I think will improve everybody's lives are
37:52
[SPEAKER_01]: Service level agreements, which are contractual obligations that are related to data confidentiality and integrity.
37:59
[SPEAKER_01]: So right now for those who don't know, there are very commonly service level agreements for data availability.
38:07
[SPEAKER_01]: So for example, with Amazon web services for XYZ system, if you can't access your data or your website goes down,
38:16
[SPEAKER_01]: a certain percentage of time, most service level agreements are like ninety nine point nine percent of the time, you can access your data.
38:23
[SPEAKER_01]: If you dip below that, AWS will give you credits.
38:26
[SPEAKER_01]: They'll give you free service as a result.
38:28
[SPEAKER_01]: So there's a financial incentive to track the availability of data.
38:33
[SPEAKER_01]: And that's very common.
38:34
[SPEAKER_01]: What is uncommon, but I think we'll become increasingly common are confidentiality and integrity.
38:41
[SPEAKER_01]: service level agreement.
38:42
[SPEAKER_01]: So on the integrity side, it could be if I ask, this is a very simple example, and I know you don't want to use LLMs to do math, but just suppose, ninety nine percent of the time you ask my agent, what two plus two is, it tells you four.
38:58
[SPEAKER_01]: And if it tells you what's five multiple times more than one set of every hundred than you owe me money because your agent's not behaving as expected.
39:07
[SPEAKER_01]: So that would give you an idea of the behavior of a counterparty agent in a way that you could do your risk assessment on it in a quantitative manner.
39:18
[SPEAKER_01]: and then the confidentiality side of things is if I give you my data and you expose it to the internet or accidentally give it to a third party, then you owe me a lot, you owe me a certain amount for each record exposed.
39:31
[SPEAKER_00]: Okay, that's interesting.
39:32
[SPEAKER_00]: So in a B to B context anyway, what you would do is you would require, when your agent is talking to their agent, you would require
39:42
[SPEAKER_00]: One, you do require a certain level of testing to be done in the first place.
39:45
[SPEAKER_00]: So give us access.
39:46
[SPEAKER_00]: Give us the kind of access that we need in order to test how our agents interact.
39:51
[SPEAKER_00]: That's number one.
39:51
[SPEAKER_00]: And number two, that would result in a certain quantitative assessment of how often we expect this thing to go sideways.
40:02
[SPEAKER_00]: We get, and at some point, we'll take the risk.
40:04
[SPEAKER_00]: So if it's one percent wrong, we'll take that risk.
40:08
[SPEAKER_00]: We'll connect our agents and off we go.
40:09
[SPEAKER_00]: Not off we go, but let's just say, we'll monitor them from there.
40:12
[SPEAKER_00]: We'll monitor, of course.
40:14
[SPEAKER_00]: But if it's, I think you make it illegal agreement, if it's going to be more than one percent, then you face, you always expy and see whatever it is.
40:21
[SPEAKER_00]: So come to an agreement on that.
40:23
[SPEAKER_00]: Okay.
40:24
[SPEAKER_00]: I get that, that's interesting.
40:25
[SPEAKER_00]: So we're still talking about a lot of pre-deployment testing being phenomenally important and having access to the things.
40:33
[SPEAKER_00]: Do you have a sense of how this plays out in a more consumer facing, like a B to C thing?
40:39
[SPEAKER_00]: So let's say I have an agent.
40:42
[SPEAKER_00]: It goes to talk to Amazon agent.
40:45
[SPEAKER_00]: We don't have a legal agreement.
40:47
[SPEAKER_00]: I don't think about how many mistakes you make blah, blah, blah, blah.
40:52
[SPEAKER_00]: So what does that, how does that work?
40:54
[SPEAKER_01]: Basically, any interaction between a consumer and a business today, there's some sort of click-wrap agreement where you're agreeing to certain terms.
41:01
[SPEAKER_01]: Now, I agree.
41:03
[SPEAKER_01]: It's would be very rare for there to be any sort of service level agreement in consumer-facing products.
41:10
[SPEAKER_01]: I mean, some people would say that the right answer is regulation, which isn't necessarily...
41:18
[SPEAKER_01]: Yeah, it is, it's happening on the state level, but not necessarily in the, in the best way possible.
41:24
[SPEAKER_01]: In my opinion, frankly, from a public policy perspective, I would say that the government should take the same approach that I just described.
41:32
[SPEAKER_01]: So right now, if there's a data breach or some sort of incident that resulted from some cyber, cyber security event, most of them are confidentiality related, PIPA,
41:46
[SPEAKER_01]: banking records, but they're also integrity and availability ones like colonial pipeline, which got hit with ransomware in twenty twenty one, the Department of Transportation find them from that event.
41:58
[SPEAKER_01]: So basically the way it happens right now is like
42:01
[SPEAKER_01]: When there's an incident, then the regular goes in and looks and tries to find all the things that you did wrong, which frankly is a pretty messed up way to operate because people are constantly making mistakes and constantly doing things that are below best practices.
42:16
[SPEAKER_01]: And basically, what so the bad happens, the government comes in as, oh, let's now sort through all your everything that you did and find everything that you did wrong.
42:23
[SPEAKER_01]: So that's the wrong approach in my opinion.
42:26
[SPEAKER_01]: And the previous administration on the presidential level was trying to like push that hardcore to be like, you must do XYZ and if we come in and we find that you didn't after an event then you're liable.
42:37
[SPEAKER_01]: I think that's the wrong approach.
42:39
[SPEAKER_01]: The right approach from my perspective is something bad happened.
42:45
[SPEAKER_01]: And you get fined based on the severity of the thing that happened.
42:48
[SPEAKER_01]: It doesn't matter whether you did quote, the right thing or the wrong thing.
42:52
[SPEAKER_01]: You just let businesses know if you expose the social security number, you will get fined this amount of money.
42:58
[SPEAKER_01]: Period.
42:58
[SPEAKER_01]: Right.
42:59
[SPEAKER_01]: Even if you did your due diligence.
43:00
[SPEAKER_01]: Yeah.
43:01
[SPEAKER_00]: Doesn't matter.
43:02
[SPEAKER_01]: Like you could have been doing everything that's humanly possible to protect the data.
43:07
[SPEAKER_01]: Well, you just got unlucky.
43:08
[SPEAKER_01]: Okay.
43:09
[SPEAKER_01]: But the thing is over the long run.
43:12
[SPEAKER_01]: It's going to even out.
43:13
[SPEAKER_01]: It's going to even out so that like the people that are doing the right things get fine.
43:17
[SPEAKER_00]: Yeah.
43:18
[SPEAKER_01]: So that's my yeah yeah for that situation.
43:20
[SPEAKER_00]: Yeah.
43:20
[SPEAKER_00]: That's interesting.
43:21
[SPEAKER_00]: I would like to talk about that more, but I actually want to I want to go back to the complexity one more time.
43:28
[SPEAKER_00]: We're conducting.
43:29
[SPEAKER_00]: I'm imagining.
43:33
[SPEAKER_00]: Let's say that we've got an agent, we've deployed it whatever it's okay.
43:38
[SPEAKER_00]: And then I guess it's the complexity of testing that my question is about.
43:43
[SPEAKER_00]: Because we can test, especially around interaction.
43:47
[SPEAKER_00]: So yeah, we couldn't think about connecting it to this database, connected to this website, yeah, that's cool.
43:53
[SPEAKER_00]: But I wonder about systemic risks versus testing the things which we're connecting.
43:59
[SPEAKER_00]: So we've talked about
44:01
[SPEAKER_00]: Let's just call the thing agent, we connected the agent to fifty databases, twenty-five narrow-ray eyes, five-genitor-ray eyes.
44:08
[SPEAKER_00]: And some of the risks I take it are not, some risks are, oh, gender of model number three is biased and such and such a way, or has such and such data that's going to poison the one that's going to output, the agent's going to output false information, or the narrow-ray eye number seven is suffering from model drift blah blah blah.
44:24
[SPEAKER_00]: So I see that we can do in assessment of each one of the things to which we want to connect our agent.
44:29
[SPEAKER_00]: But then I'm wondering about the testing that has to occur.
44:32
[SPEAKER_00]: It's not any particular model or any particular database.
44:37
[SPEAKER_00]: It's the three of these things or the five of these things or the seventeen of these things working in concert where the risk arises from the set of these things interacting as opposed to any one thing being faulty.
44:52
[SPEAKER_00]: And then how do we, how do we
44:57
[SPEAKER_00]: do risk assessments, what we're talking about, getting connected to all those things, let alone, now connected to, agent, another agent who's also connected to all these things.
45:06
[SPEAKER_00]: So it just seems like there's so many nodes at which you could perform a risk assessment, right, even with a single model, you can have a foundation model, you want to do a risk assessment on that, then you do some fine tuning, and then you add an adapter, there's all these, first thing on models, all these things you can do,
45:25
[SPEAKER_00]: But yeah, I guess my question is, so if that's how many nodes there are at a single model, there's more, but how do we account for all the nodes and doing a risk assessment?
45:35
[SPEAKER_00]: How do we think about, let me put this slightly differently, you can't possibly do a risk assessment at every single node of interaction or the set of all interactions and subsets of interactions because we need to be doing a risk assessment that's over there and I think it's deployed.
45:49
[SPEAKER_00]: So I guess we're faced with this qualitative question
45:54
[SPEAKER_00]: What's the appropriate place to do a risk assessment plus who should be doing and how should they do it?
46:00
[SPEAKER_00]: But let's just think about the where or the when at this point.
46:03
[SPEAKER_00]: How is supposed to know when to do the risk assessment given all these nodes at which a can and principle be done?
46:10
[SPEAKER_01]: Yeah, so basically you're saying that your decision tree needs to stop somewhere.
46:14
[SPEAKER_01]: You can't infinitely check every path in the system, which is true.
46:19
[SPEAKER_00]: You can't check number one and number two and number three and one and two and one and three and one and four and two wins and blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah
46:34
[SPEAKER_01]: pre-deployment, the highest likelihood, the highest variance aspect of the system.
46:39
[SPEAKER_01]: Look at that.
46:41
[SPEAKER_01]: Look at the highest impact part of the system.
46:43
[SPEAKER_01]: So are you dealing with protected health information?
46:45
[SPEAKER_01]: Are you dealing with trade secrets?
46:46
[SPEAKER_01]: Are you dealing with data that's powering industrial systems?
46:50
[SPEAKER_01]: Those are the areas you want to look.
46:52
[SPEAKER_01]: So that's pre-deployment.
46:53
[SPEAKER_01]: And then
46:54
[SPEAKER_01]: When you are deploying, especially in these kind of multi-agentic orchestration types, you should follow existing software development and deployment practices, which include doing things like canary deployments, and that means, so if Amazon, for example, if they put a new feature on their website, or they change the color of a button, they're going to test that for a very narrow subset of Amazon users.
47:24
[SPEAKER_01]: They'll do that for marketing reasons to see which button is more likely to cost people to click.
47:29
[SPEAKER_01]: They'll also do it for technical reasons to see if the change is breaking for a certain subset of people or all the people that the change has been introduced to.
47:40
[SPEAKER_01]: But they only do that for a point zero one percent of their user base and then they gradually expand the amount of the user base who's exposed to that change so that it's on the breaks then they can stop or they can stop expanding or they could roll back.
47:54
[SPEAKER_01]: So yeah, we're going to need to do that type of thing with these multi-agentic.
47:59
[SPEAKER_01]: orchestrations.
48:00
[SPEAKER_00]: Yeah, that's interesting because the way that we've characterized it so far has been pre-deployment and then modern-term post-deployment and we've been treating deployment as an on-off switch.
48:11
[SPEAKER_00]: But it's not an on-off switch.
48:13
[SPEAKER_00]: It's a, it's a, it's a dimmer.
48:15
[SPEAKER_00]: It's got a, it's a dimmer switch.
48:17
[SPEAKER_00]: So there's phase deployment, like you said, right?
48:19
[SPEAKER_00]: So there's point to one percent of the population.
48:21
[SPEAKER_00]: One of the ways that we have to do this is the way that you said we do other software deployments, which is
48:26
[SPEAKER_00]: You do it in phases and you do a risk test run in each phase.
48:29
[SPEAKER_00]: There's a threshold and if the threshold is passed, you can move on to next phase if it's not fixed stuff until you can pass the threshold and keep going.
48:37
[SPEAKER_00]: Exactly.
48:38
[SPEAKER_00]: But it's still a very complicated complicated system.
48:42
[SPEAKER_00]: And I think the original point was you got you have to have a risk triage, right?
48:46
[SPEAKER_00]: So you can't do a deeper assessment of each thing.
48:48
[SPEAKER_00]: So the first thing you need to do is
48:50
[SPEAKER_00]: You have some way of doing a quick and dirty triage of the set of things that need to be assessed.
48:55
[SPEAKER_00]: And some things going to be like, oh, yeah, this is health data.
48:57
[SPEAKER_00]: This is really, whoa, red light.
48:59
[SPEAKER_00]: We got to go deeper on this where some things are like, that's fine.
49:02
[SPEAKER_00]: That's not a big deal.
49:03
[SPEAKER_00]: That's data about, I don't know, whatever.
49:06
[SPEAKER_00]: Something relatively trivial.
49:07
[SPEAKER_00]: It doesn't rise to the level of health data.
49:09
[SPEAKER_01]: Meme generator is my preferred.
49:11
[SPEAKER_00]: Yeah, great.
49:12
[SPEAKER_00]: Good.
49:12
[SPEAKER_00]: Meme generator.
49:12
[SPEAKER_00]: Yeah, yeah.
49:13
[SPEAKER_00]: So it's a call.
49:14
[SPEAKER_00]: And so to get back now to wrap up with your people process technology point,
49:21
[SPEAKER_00]: The processes of a company have to be updated because there's so many more processes that have to happen throughout the build and deployment and monitoring of this.
49:30
[SPEAKER_00]: And then, of course, for the people, we'd lots of training, right?
49:34
[SPEAKER_00]: We need lots more people trained on the sources of the risks and what kinds of things need to be done about those.
49:40
[SPEAKER_01]: Absolutely.
49:41
[SPEAKER_00]: All right, Walter, thanks so much.
49:42
[SPEAKER_00]: That was really helpful.
49:43
[SPEAKER_00]: The risk landscape is so complicated, but I think you've given us a really nice picture of what it looks like to actually responsibly manage that risk.
49:50
[SPEAKER_00]: So yeah, thanks for coming on.
49:51
[SPEAKER_01]: Thanks for having me read and really appreciate the opportunity.
49:56
[SPEAKER_00]: Thanks so much everyone for listening and now let me think that people have made this podcast possible.
50:01
[SPEAKER_00]: So first of all, we have the Lotto Alexander.
50:04
[SPEAKER_00]: He's been with me since day one.
50:06
[SPEAKER_00]: He showed me even how to start recording a podcast in the first place.
50:09
[SPEAKER_00]: He does editing stuff.
50:10
[SPEAKER_00]: He's been fantastic and he does really everything I ask him to and more.
50:14
[SPEAKER_00]: I've never been anyone else who says so much.
50:16
[SPEAKER_00]: Yeah, I don't know how to do that, but I'll figure it out.
50:18
[SPEAKER_00]: What more could you ask for in a producer?
50:20
[SPEAKER_00]: Next up, Jennifer Strong, my podcast guru, I don't know what else to call her.
50:25
[SPEAKER_00]: She just advised me on how to design a podcast, how to grow it, and she's responsible for if you like this kind of season two relaunch, and everything around it.
50:33
[SPEAKER_00]: So thank you to you, Jennifer.
50:35
[SPEAKER_00]: Next up, Meg Marco, responsible for all the excellent artwork.
50:39
[SPEAKER_00]: Before I did the artwork, I didn't really know what I was doing.
50:41
[SPEAKER_00]: We put it together, it worked well enough.
50:43
[SPEAKER_00]: But I think Meg really did a phenomenal job bringing everything together.
50:48
[SPEAKER_00]: Garrett Lane, sound engineer, that I frankly don't deserve.
50:51
[SPEAKER_00]: He's just too good for this podcast.
50:53
[SPEAKER_00]: Don't tell him I said that though.
50:55
[SPEAKER_00]: Next up, Benjamin Colton, he's a professor of philosophy who has helped me source guests identify who's interesting, who has things to say.
51:03
[SPEAKER_00]: et cetera, et cetera.
51:04
[SPEAKER_00]: So without him, I would have anyone to talk to.
51:06
[SPEAKER_00]: And then finally, Ashley Wasselenko, who helps with all manner of administrative things, making sure that the podcast guests find their way to the podcast and schedule the recording, get their bios and headshots, and really just keeping the show on the rail to me organized.
51:21
[SPEAKER_00]: Thanks to all of you, really appreciate all your hard work.