Latest / The Road to Accountable AI / Venkat Siva (Compfly): Governing Agents at the Execution Boundary

Transcript

Venkat Siva (Compfly): Governing Agents at the Execution Boundary · The Road to Accountable AI

0:10
Kevin: Hi, I'm Kevin Werbach, Professor of Legal Studies and
0:13
Business Ethics at the Wharton School of the University of Pennsylvania.
0:16
For decades, I've studied emerging technologies, from
0:18
broadband to blockchain.
0:21
Today, AI is promising to transform our world.
0:25
But AI needs accountability, mechanisms to ensure it's developed and deployed in
0:30
responsible, safe, and trustworthy ways.
0:33
On this podcast, I speak with the experts leading the charge for accountable AI.
0:41
This week in our series on governance of AI agents, I speak with Venkat
0:45
Siva, co-founder and CEO of Comply.
0:48
Traditional security and MLOps systems were built for deterministic workflows,
0:52
but agents make decisions at runtime.
0:54
We discuss what actually distinguishes an agent from other AI systems, why
0:59
traditional access controls fail in the agentic world, how machine-readable
1:03
identity and audit trails for agents might work in practice, the shadow
1:07
agent problem that's keeping CISOs up at night, the security implications
1:11
of the MCP standard for agent access to external tools, and what changes
1:15
when agents start operating robots and other objects in the physical world.
1:21
Venkat, welcome to "The Road to Accountable AI."
1:24
Venkat: Thank you, uh, Kevin, for having me.
1:25
Pleasure to be here.
1:27
Um, so you and your co-founder Anand, I know from being, uh, students at, at
1:32
Penn and Wharton, but how did you come to the problem of agent governance?
1:37
Both
1:37
of us were looking at agentic governance from two different sides, in my opinion,
1:42
but the problem felt very similar.
1:44
Just sort of zooming out, I have spent over 20 years building platform
1:48
products that help enterprises adopt new technology safely and at scale.
1:52
What I saw at Rivian, and what I am frankly seeing across
1:56
the market, is that agents are becoming incredibly easy to build.
2:00
Uh, with today's models, frameworks, and coding tools, almost any engineering team
2:05
can put together a product very fast.
2:07
But hardening these agents for enterprises are a very different problem, right?
2:10
These agents are non-deterministic.
2:13
They can show emergent behavior.
2:14
They can take actions across real business systems, and that's where usually
2:19
platform and security teams get pulled in.
2:22
And from there, if you look at it, it takes several months to get anything out.
2:25
One, because these platform teams are small.
2:27
That means they are tackling all of these different agents coming their way.
2:30
But also, the problem is novel enough that they have to look for controls and
2:34
build them by hand per agent, right?
2:37
Anand was seeing the same sort of shift on his side from an
2:41
audit and security perspective.
2:43
Mm-hmm.
2:43
Uh, traditional controls were largely designed for fixed
2:45
workflows, known permissions, and predictable software behavior.
2:50
They were not designed for software that can decide at runtime what
2:53
tools to call or what context to use or what actions to take.
2:57
That was the key insight for us, and, and we started zooming out.
3:00
Uh, the reason we think collectively we all care about this so much is, is just
3:07
because agents are going to be there everywhere, uh, across organizations.
3:11
So we enlisted this cannot just be another feature, uh, within
3:15
an existing tool or a system.
3:17
Uh, we think agents create a new control problem, uh, and that needs a governance
3:23
layer where decisions-- uh, we need to turn decisions, uh, sort of govern them
3:29
when the decisions turn into actions.
3:31
Uh, that's why we started Comply.
3:33
Kevin: Let me just push you a little bit more on, on that.
3:35
There are plenty of tools now for MLOps and, uh, AI security.
3:42
Why is, uh, why, why don't agents just belong as a feature under
3:46
Venkat: those
3:46
Kevin: platforms?
3:47
Venkat: If you look at most of the current systems, the MLOp system, the security
3:49
system, and all of those things today, most of them are built for a fixed set
3:55
of deterministic set of workflows, right?
3:58
Uh, so we think-- the, the more we think about this, the control plane needs to
4:03
shift to where the decisions are made versus being a static set of steps, right?
4:08
Like that's, that's why we think- Mm-hmm.
4:09
Go ahead ... that these systems cannot continue to sort of be
4:12
enhanced to build something like this.
4:14
You need to think about this in a different wa- different way.
4:16
Yeah.
4:18
Uh, a threshold question is: how do you define what's an agent?
4:21
Good question.
4:22
Uh, agent to me is a system that can take a goal, reason over context, and
4:28
choose from one of several available tools and execute a set of actions,
4:32
let's say with some degree of autonomy.
4:34
The important part, Kevin, to me is it's not whether it has a chat interface.
4:40
Mm-hmm.
4:40
The important part is whether it can act.
4:42
A chatbot that just answers questions may not necessarily be an agent, but a
4:47
system that reads an email, let's say, checks service request, uh, potentially
4:52
updates a ticket or schedules a follow-up- Mm-hmm ... that is agentic.
4:55
Um, I mean, one should also sort of think about like what
4:59
is not agentic or what is- Mm
5:01
outside of this, right?
5:02
Um, uh, my opinion, I think the static model APIs, uh, that potentially
5:07
classifies sentiments or maybe providing recommendations, uh, or maybe a dashboard
5:12
that just summarizes data where that it doesn't act, you have to act.
5:16
Those all still likely needs governance, uh, but they do not have the same kind
5:22
of autonomy risk we are talking about.
5:23
Yeah.
5:24
Speaker 3: Mm-hmm.
5:25
Kevin: Uh, tell us about your company, Comply.
5:28
What specifically is the approach that you're taking to governance of agents?
5:34
Venkat: Absolutely.
5:34
So if you look at the sort of the space, most companies are focusing
5:38
on sort of models or access.
5:41
Mm-hmm.
5:41
What we focus on is how do you govern behavior at the execution boundary, right?
5:45
Like this is exactly where you have- Mm-hmm ... prompts, you have
5:49
memory, you have identity, you have tools, and you have policies.
5:52
All of these sort of come together in one place, and that's also- Where we think
5:56
enterprise harms are actually created.
5:58
Uh, so if you look at a agentic control plane, what does it actually mean
6:02
to have an agentic control plane?
6:03
So in our, in our sort of view, uh, one is measured autonomy.
6:09
A measured autonomy is all about sort of setting the boundaries.
6:12
What actions can this agent take, with what privilege, under what
6:17
approvals, in which context, right?
6:19
Mm-hmm.
6:19
Uh, this includes things like least privilege, let's say restricted
6:22
tool access, some approval gates, constrained outputs.
6:26
The key thing here to sort of understand is least privilege
6:29
alone is not sufficient, right?
6:31
Like, that's because agent- Just for those
6:32
Kevin: who aren't familiar with, with security, what does least privilege mean?
6:35
Venkat: So least privilege means, uh, let's take an example, right?
6:37
Like, uh, let's take an agent that looks for emails or basically
6:43
looks for a trigger from a customer to say, "Show me...
6:45
Can you guys get me a demo for this product?" Right?
6:48
Right.
6:48
Uh, and as agent, what agent could typically do is it could go and
6:51
query Salesforce or some other system to see what this customer is.
6:54
Mm-hmm.
6:54
And then maybe send them an email back saying, thanking them and
6:57
maybe scheduling something, but then maybe CC'ing the account executive.
7:00
So in a least privileged scenario, what you would typically do is you
7:03
basically say, this system or a process, processes gets the exact
7:08
permission that it needs, right?
7:10
Like you don't basically add more permissions than you actually need.
7:12
So typically what you do is in this case, you say, "This system can only
7:16
read stuff from Salesforce-" Mm-hmm.
7:17
uh, and maybe send this email out," kind of thing, right?
7:20
Uh, but in an agentic scenario, one could social engineer this agent and
7:25
make it go and pull data that are outside the realm of what this is.
7:28
Right.
7:28
Like it could potentially go and pull PII data, uh, or some other sensitive
7:32
data and, and it could actually send it to somebody else, right?
7:35
So that's, that's what... So this containment, in our view,
7:40
needs to be semantic, right?
7:41
Like, so we have to sort of shrink the blast radius so that agents is
7:48
sometimes technically allowed to do something, doesn't mean it should be
7:51
allowed to actually do that thing, right?
7:53
Uh, so that- Mm-hmm.
7:54
... is what we call as measured autonomy.
7:56
Second thing is provenance, and the third one is continuous verification.
7:59
Provenance is all about how do you record every key decision
8:02
that this agent is making?
8:03
Uh, how can you audit?
8:05
Uh, what did the agency, how did it decide, uh, uh,
8:08
before it executed something?
8:10
And continuous verification is key because these agents evolve
8:14
or have an emergent behavior.
8:15
Right.
8:16
Uh, so this means you have to monitor, um, drifts, right?
8:21
Like because the, the model one day said something, and then maybe the model
8:24
version changed, and then the agent is behaving slightly differently now.
8:27
Um, so you do need to sort of... Or it could be somebody added a new tool into
8:31
this agent, and that has changed the way this agent thinks about- Mm-hmm.
8:35
how to execute a certain workflow, right?
8:38
So those needs continuous attack test so that we can identify
8:43
these newer exploits or newer sort of drift, uh, for these agents.
8:46
So these three, I wanna say, is a key differentiation- Mm-hmm.
8:49
... from a conflict perspective, and that's the control plane we are building that
8:52
can discover- Validate, secure, and govern agents from code to production.
8:57
Kevin: What, what was the most challenging thing to create?
9:00
Uh,
9:00
Venkat: I, I wanna say I think the, the runtime governance
9:03
is the most difficult part.
9:04
Mm-hmm.
9:04
Because you are making an assessment on what this agent is doing, what's its
9:09
trajectory- Mm-hmm ... uh, and to sort of ensure is this okay for this agent to
9:14
do with the context that it actually has?
9:16
That's the problem we are sort of grappling around.
9:18
We're trying to actually see how do we sort of solve this in the most practical
9:21
way possible, uh, for our customers.
9:23
Mm-hmm.
9:24
Kevin: Um, most organizations are, are building on top of frontier models that
9:29
are coming from third-party major AI labs.
9:33
Uh, and then- Mm-hmm
9:34
you know, the, the agentic tools, uh, are... In some cases, there's
9:38
elements that come from the lab.
9:39
In some cases, they're harnesses.
9:40
They're things, you know, in the enterprise, in deployment.
9:44
Um, how is it possible to effectively provide, uh, governance and security
9:48
of agents in that environment?
9:51
Venkat: So, uh, absolutely.
9:52
So the models themself, I mean, if you look at sort of the, um, uh, the
10:00
models themselves, these labs are spending incredible amount of money in
10:03
trying to secure these agents, right?
10:06
Mm-hmm.
10:06
Uh, but models themself don't truly secure the operational portion of
10:12
how this agent is gonna behave.
10:13
Like, what- Mm-hmm
10:14
permissions does that agent have?
10:16
What data does it have access to?
10:18
Uh, and those kinds of things.
10:20
So those, I wanna say, are very important from an agentic perspective, right?
10:24
Mm-hmm.
10:24
Like, so that's how people should be thinking about sort of model safety
10:27
versus sort of agent safety in that sense.
10:30
Kevin: Can you give an example of a situation where, where the
10:33
model was safe but still some harm came about from the agent?
10:37
Venkat: Let's take automotive as an example, uh, Kevin, right?
10:40
Mm-hmm.
10:40
Uh, imagine, um, a service triage agent, okay, that helps technicians
10:47
diagnose issues and decide next steps.
10:50
Speaker 3: Okay.
10:50
Venkat: Mm-hmm.
10:50
Uh, a customer describes a symptom, and the agent interprets it,
10:54
searches, let's say, service manuals, some past service logs, uh, maybe
10:59
some knowledge base articles, and then recommends what to do next.
11:04
In some cases, it may even, let's say, open a case or
11:08
potentially order a part, okay?
11:10
Now, the model itself may exactly be doing the right thing, uh,
11:15
whatever it was trained to do.
11:16
It may be summarizing it correctly, uh, reasoning correctly, uh,
11:20
following instructions correctly.
11:22
But the enterprise environment could be messy, right?
11:24
It may contain outdated notes, I mean- Mm-hmm ... inventory-driven
11:28
workarounds, uh, or vendor guidances that should not be generalized.
11:31
Uh, or, or worse, there could be malicious content, right?
11:35
Mm-hmm.
11:35
So if somewhere in the knowledge layer there is a note that says, "Whenever
11:39
you see this symptom, always order the part"- The model may treat that as a
11:45
legitimate operational guidance, right?
11:47
Uh, at this point, the model is still behaving well because the model got what
11:51
it got, what the instruction it got.
11:52
Mm-hmm.
11:53
But the agent is now driving the wrong action in the real world.
11:56
Mm-hmm.
11:57
Does it, does it make sense?
11:58
Kevin: Yeah, absolutely.
11:59
Now, and you mentioned malicious content.
12:00
I wanted to ask you about that.
12:01
Prompt injection is obviously a huge issue.
12:05
How is it possible to combat that given that, you know, everything is
12:08
code from the perspective of an agent?
12:10
Oh, no, fantastic.
12:11
So,
12:12
Venkat: uh, honest answer is there is no one magic filter
12:15
for prompt injection, right?
12:16
Like you defend in layers.
12:17
Uh, indirect prompt injection largely happens when malicious instructions
12:22
are hidden into things, right?
12:23
Like it can be hidden in the pages that the agent reads, uh, maybe
12:26
a PDF that it reads, uh, a Slack message or some retrieved documents.
12:30
Uh, I recall OpenAI recently published a, a sort of a blog about these attacks,
12:36
and they, they are looking at this more like a social engineering attack rather
12:40
than just, uh, a prompt engineering.
12:42
And, and what, what I mean by that is it is not one malicious string
12:47
that could potentially deceive.
12:48
That's how injections u-used to be treated in the past, right?
12:52
Like SQL injections are about sort of how do you-- how does one
12:54
sort of trick this particular SQL?
12:56
They're basically saying, let's solve this by restricting actions that
12:59
mutate using control parameters.
13:01
What that means is let the agent do what it does and let
13:06
the agent make an action call.
13:08
And in the action, just make sure you are having a deterministic set
13:12
of checks to say- Mm-hmm ... is this allowed for this particular agent to do?
13:17
I think we can do that, but in not in all cases.
13:20
I, I, I, I think that basically makes this deterministic code very,
13:24
very sort of re-- long and, like, difficult to manage in the long run.
13:28
The agent, there still needs to be a process that needs to check for the
13:32
intent of what this agent is trying to do.
13:34
Uh, so the first rule of thumb I would say is treat external content
13:37
as data, not authority, because that's where these things come in.
13:41
Sanitize memory.
13:42
Memory is going to become a, a big part of what this agent is.
13:45
It stores things persistently for a while.
13:48
A webpage can provide evidence, but it should never be able to
13:51
read that agent's code, right?
13:52
Mm-hmm.
13:52
So practically, sort of what you do is you have trust boundaries.
13:56
Mm-hmm.
13:56
Uh, you tag untrusted content, you restrict decisions that can be
14:00
influenced by those contents, uh, and, and potentially limit tool
14:04
privileges, uh, like OpenAI suggests.
14:06
Mm-hmm.
14:08
Kevin: Uh, you, you mentioned something along these lines early on, but with
14:11
agents, we're not just talking about you give a prompt and you get a response,
14:15
but they're sequences of actions.
14:17
So what does it mean to govern in that kind of environment?
14:23
Venkat: We pro-probably need multiple things, uh, Kevin, to be able to sort
14:26
of govern in such a, such a space.
14:28
Uh, one is we need to be able to sort of understand what agents
14:33
first exist, in my opinion, right?
14:35
Like discovery becomes a big part of this.
14:37
Uh- And being able to sort of identify those agents mean then we know what exists
14:43
out there, uh- Mm-hmm ... and who owns them, what they are pretty much running.
14:46
So that would be the first step.
14:48
Uh, and the second thing is, does this agent have identity?
14:52
Uh- Mm-hmm ... agent's identity becomes an important part.
14:55
We can talk more about this.
14:56
Uh- Mm-hmm.
14:57
But once you actually have an agent's identity, then what you wanna do is to
15:02
sort of ensure that can we capture what this agent is doing, uh, at runtime?
15:09
Uh, and before you deploy anything, in our opinion, you should validate these agents.
15:13
The way you validate that is you actually do a good security and
15:16
safety test on this agent, and those effectively needs to be creative, right?
15:21
Like, you can't just take a static set of data sets and, like, throw it on
15:26
this agent and actually say- Mm-hmm
15:27
"Can we do this?" There's so many benchmarks online these days,
15:31
and benchmarks always trip me because they are-- they need to
15:35
be very specific to an agent.
15:37
Uh- Mm-hmm ... but largely we compare benchmarks, uh, like apple to apple across
15:41
different agents and different models.
15:43
So, so being able to sort of really create specific sort of tests for an agent
15:48
dynamically as the agent's goal changes, as the agent's number of tools available
15:54
changes, uh, and as the agent's sort of actions changes, is very important.
15:58
So being able to sort of test that and being able to provide people a view
16:01
into how this agent performs across the various set of weaknesses, right?
16:05
Like- Mm-hmm ... can, can this allow a direct attack?
16:09
Uh, can this allow, uh, indirect prompt injections?
16:12
Can it allow, uh, multi-turn chaining attacks, right?
16:15
Like- Mm-hmm
16:15
and so forth.
16:16
So we, what we essentially do is we have a way to, for customers to be able to
16:20
see, uh, how these are essentially built.
16:22
And we think that's very important for customers to be able to know where
16:25
the weaknesses for these agents are.
16:26
Mm-hmm.
16:26
And once you know, then you actually figure out, okay, what I need to do
16:29
for this agent to be able to sort of reinforce them, uh, and that becomes
16:33
the, uh, runtime policy, and this is how we essentially start controlling
16:37
what this agent does and how it behaves.
16:38
Yeah.
16:39
Kevin: Got it.
16:40
Ju-just to follow up on something you said, what exactly does
16:43
identity for agents mean?
16:45
And what's, what's the current status, uh, uh, of those kinds
16:50
of systems being in place?
16:52
Venkat: Yeah.
16:53
Um, our view is every enterprise agent needs a distinct, uh, machine-readable
16:58
identity, um, not because identity sort of magically solves governance.
17:03
Identity is an anchor that lets us prove who this is and
17:07
what action did it take, right?
17:08
Mm-hmm.
17:09
Um, at Comply, we believe, um, DID is a good place to start.
17:13
DID is decentralized identifiers.
17:17
Um, these, um, these were specifications, um, uh, that was written.
17:24
And DID is, is an agent's identifier that's cryptographically secure, right?
17:29
Like- Mm-hmm ... it's not just a display name like a HR assistant or
17:32
a, a finance bot, but a verifiable identity that can be resolved- Checked
17:37
and tied to cryptographic keys.
17:39
Once there is identity, the second thing is the agents need verifiable credentials.
17:45
That means at the time of its execution, do we know, is this
17:49
agent allowed to do certain thing?
17:50
Can it access a certain data?
17:52
Is it allowed to operate in the environment that it is, and so forth.
17:55
Uh, uh, and I think third step is agents action need to be signed and
18:00
recorded in a tamper-evident way.
18:01
This is not just about sort of logging something to say this happened,
18:04
but it's all about sort of really cryptographically ensuring that this is
18:08
indeed an action that this agent took.
18:10
And that's what we call as the trust ledger within, within CoverGuard.
18:13
Mm-hmm.
18:14
So, um, so I mean, audit is sort of famously leveraging, uh, screenshots and,
18:20
uh, and sort of loose audit logs today.
18:22
I think in the agentic world that ought to change, uh, and there needs to be
18:26
higher governance questions on sort of how do we sort of know this in a
18:29
cryptographically tamper-evident way.
18:31
And that's, that's, that's a big portion of our focus as well.
18:34
Kevin: Mm-hmm.
18:35
Uh, do organizations even know all of the agents that they have running around?
18:40
With, with, uh, generative AI in general, we have this issue that
18:44
the capability gets pushed out to business users and the CISO may not
18:48
even entirely know what's going on.
18:50
I would assume with agents what we're starting to see now with, with coding
18:53
tools, that's a problem as well.
18:54
So do, do you run into that issue?
18:56
Yeah.
18:56
No, uh, 100%
18:57
Venkat: agreed, right?
18:58
Like, I think, uh, the shadow agents are a big, big risk for companies.
19:01
Mm-hmm.
19:01
And, and, and the interesting challenge with shadow AI is the-- they largely have
19:09
broader access, uh, and no real- Mm-hmm
19:11
guardrails for these agents.
19:13
Um, a good example I would say is OpenClaw.
19:15
Uh, right?
19:16
Mm-hmm.
19:16
Lots of people are dabbling with OpenClaw.
19:18
Uh, I mean, you can connect a ton of tools, including emails, uh,
19:22
uh, browsers, uh, file systems.
19:25
Uh, and, and, and they are not just chatbots, right?
19:28
Like, they can actually do things, uh, across environments.
19:31
Uh, I mean, this means sort of hidden managed agents may quietly, um, have the
19:38
same reach as a powerful employee, right?
19:40
Like, but without any identity, uh, limits or oversight you
19:43
would typically have on a human.
19:45
Um, so I would say CISOs and CDOs are very concerned, right?
19:50
Like about these sort of, uh, shadow agents.
19:52
So discovery becomes an important part, right?
19:54
Uh, uh, I, I remember one of these CISOs, uh, that I was talking to recently was
19:59
talking to me about how some employee just ran OpenClaw in an enterprise setting.
20:05
Mm-hmm.
20:06
And how disaster that could be, right?
20:08
Like, uh, so they, they had to sort of really, uh, warn that employee and also
20:12
sort of ensure everybody's aware that, uh, this is, this is a big credible
20:15
thing that they need to be careful about.
20:17
So most people I know who are running OpenClaw are running in a
20:20
very isolated environment today.
20:21
Mm-hmm.
20:21
It's just not ready for enterprises to sort of, uh, adopt.
20:25
Uh, but, but it is the area that we can't stay away from as well,
20:29
because autonomous agentic system are a thing and they are gonna come.
20:33
We need to sort of pay attention to the space and actually see how do
20:35
we improve the governance, right?
20:36
Like that's a big part, in my opinion.
20:37
Speaker 3: Mm-hmm.
20:38
Venkat: Um,
20:38
Kevin: yeah.
20:38
Yeah.
20:39
How do organizations do that discovery process that you talked
20:41
about to find those shadow agents?
20:43
Venkat: Yeah, 100%.
20:44
I, uh, so agents today come in all form and shapes.
20:48
Mm-hmm.
20:48
They can run in your laptops, they can run in, uh, uh, sort of these model
20:53
providers or agent platforms, uh, and they can also run in, uh, sort of, uh, cloud
20:58
platforms like Kubernetes and- Mm-hmm
20:59
and those kinds of things.
21:00
Uh, being able to sort of, one, look for signatures of what agents,
21:05
uh... Because there is a difference between AI usage and agentic usage.
21:10
Uh, uh, so being able to sort of understand the nuance and the difference,
21:13
I think, is gonna be key to be able to sort of identify what is an agentic
21:16
use versus what's a, a typical sort of AI use, uh, just for some advisory
21:21
or sort of augmented, um, uh, uh, sort of, uh, uh, intellectual, sort of arti-
21:27
uh, sort of AI-based reviews, right?
21:28
Like in that sense.
21:29
Mm-hmm.
21:29
So the way we think about this is, uh, we have a gateway.
21:35
There needs to be something that basically intercepts the request
21:38
that goes from these systems to something like an LLM model.
21:43
That's a good place to sort of hook to sort of understand, okay, what
21:46
kind of request and response goes in and out, and look for signatures that
21:50
tell us that this is agentic use.
21:51
Mm-hmm.
21:52
That's definitely one way to look at this.
21:53
Uh, we're also sort of-- Uh, we also have other ways to sort of, uh, look,
21:57
look for processes in the companies which will essentially force us to identify
22:01
that there is an agentic use, right?
22:02
Mm-hmm.
22:02
Like, identify those and make sure that there are gates there.
22:06
Uh, I mean, this could be... When somebody's asking for a LLM key, I mean,
22:10
that's a great place to start, right?
22:11
Mm-hmm.
22:11
Like, you know that they need an LLM key to be able to
22:14
start doing agentic workflows.
22:16
That could be a great place to start, and so forth.
22:17
So this is how you sort of identify.
22:19
This is structural way to sort of identify this, uh, Kevin.
22:22
Kevin: You talked earlier about, uh, the concept that, uh, you have
22:26
a compliant measured autonomy.
22:27
Mm-hmm.
22:28
Uh, where there's some semantic dimension involved.
22:31
Um, for one thing, I would assume organizations are very different
22:33
in terms of their risk tolerance, uh, of what level of autonomy
22:37
they would be comfortable with.
22:38
Um, but they may not even know how to articulate that.
22:41
So how do you provide the capability so that organizations can figure
22:45
out really what level of autonomy they wanna provide to their agents?
22:51
Venkat: Oh, it's a great question.
22:52
Um, I, I would like to use a phrase called autonomy budget, right?
22:57
Mm-hmm.
22:57
Like, not every action deserves the same level of freedom.
23:01
Um- Enterprises should classify actions by, let's say, a few dimensions, right?
23:06
Mm-hmm.
23:07
Does this agent do activities that are irreversible?
23:11
Does this have financial impact, a regulatory impact, customer impact?
23:15
Um, uh, or does it use operate da-data sensitive, uh,
23:19
sensitive data in this case?
23:21
If an a-if an agent is a low risk and it has reversible things, give
23:26
the agent more autonomy, right?
23:27
Mm-hmm.
23:27
If it's a high risk, reversible, uh, irreversible, um, and it
23:33
has customer-facing capability, it should require approval, uh,
23:36
stronger identity checks, and potentially some circuit breakers.
23:39
Um, a, a practical example, in my opinion, would be, let's say if you
23:43
have a customer serv- a customer support agent, um, I mean, it can-- it should be
23:48
allowed to summarize ticket autonomously.
23:50
It should be able to draft response autonomously, um, and maybe
23:53
even sort of issue maybe a ten dollar refund, uh, autonomously.
23:57
I mean, if the customer is eligible.
23:59
Mm-hmm.
23:59
But a five thousand dollar refund or changing the contract should
24:02
definitely require a human approval.
24:05
Uh, the key is not, like, do human-in-the-loop everywhere, uh,
24:10
otherwise agents will start to look like a very expensive suggestion box.
24:14
Uh, but also too few approvals also makes this agent dangerous.
24:18
So being able to sort of hit that balance, uh, is gonna be key, and that
24:21
you could essentially start to do this with some of the control and policies
24:24
that we talk about for runtime governance.
24:27
Mm-hmm.
24:28
Kevin: What about multi-agent interactions?
24:30
Uh, how to deal with situations where it might be that, that each agent,
24:34
uh, is operating properly, but there's unanticipated consequences or risks that
24:38
come up, and we're going to be in this world, if we're not already, where there's
24:42
fleets of agents interacting with agents.
24:43
So how do you deal with that challenge?
24:46
That's a great question, Kevin.
24:47
So,
24:47
Venkat: um, the way you think about multi-agent is you don't just devise
24:54
governance for agents individually, right?
24:56
Like you govern interactions, handoffs, uh, and, and potentially
25:02
violations that could emerge from that.
25:04
Uh, the, the key here is to just stop thinking that agents are,
25:08
like, governed in isolation, right?
25:11
Um, let's- Take an example, right?
25:13
Like there's a sort of a procurement agent and let's say the AP payment agent.
25:17
Uh, procurement agent updates, uh, sort of vendor record after reading
25:21
some onboarding documents, and accounts payable verifies the vendor
25:24
is active before it pays the invoice.
25:27
Each agent is doing something reasonable on its own, but together
25:31
they could violate segregation of duties if the same workflow effectively
25:34
gets to both of them, right?
25:36
Something that basically says, "Okay, create the agent, uh, modify and
25:39
update the agent, and then like release money." Like that could potentially
25:42
be- Mm-hmm ... uh, dangerous.
25:44
So I, I think one way we think about this is we need to have fleet-level policies
25:49
in addition to agent-level policies.
25:51
Every agent needs to have identity.
25:53
Um, every handoff needs to be authorized.
25:56
Um, so you need policies like no agent or combination of agents can complete
26:01
this workflow without approvals, right?
26:04
Uh- Mm-hmm ... so this helps, especially in a multi-agent scenario,
26:09
um, the segregation of duties that we care about, and that should be
26:12
treated as how we care about today for humans, uh, in, in the- Mm-hmm
26:16
agentic world.
26:17
Mm-hmm.
26:18
Kevin: Uh, there's a number of standards that are developing in the agent world,
26:21
probably the most prominent being m-MCP for connecting to external tools.
26:25
Um, but, you know, how do you see the, the development of these, uh, open standards?
26:32
How do they relate to the work that companies like yours are doing?
26:35
And, you know, where do the, the standards potentially open up new vulnerabilities?
26:40
Venkat: No, um, MC-MCP is a very powerful tool, uh- Mm-hmm ... but it standardizes--
26:45
because it standardizes how agents connect to- Mm-hmm ... tools and context, right?
26:49
Uh, but that effectively, like you said, also creates newer attack surface.
26:54
Uh, the mistake that I see people do typically with an MCP is just
26:59
treating that as just another API.
27:01
Because it's- Mm-hmm.
27:02
Yes, it is an API that's configured as a tool, but LLM is often
27:07
designing, deciding, uh, what and how to call these tools.
27:11
Uh, the tool descriptions, uh, how the server behaviors, um, uh, how the contents
27:17
are processed, uh, and the agentic reasoning is all sort of intertwined.
27:21
Uh, so this creates different risks, in my opinion, um, that of course
27:28
MCP, uh, users should be thinking about tool poisoning, for example.
27:32
Um, uh, supply chain, which is like, what if an MCP server gets infected?
27:36
You pretty much infect the whole agent.
27:38
Uh, COSAI, which is, uh, a community we are part of, um- Mm-hmm ... uh,
27:45
just basically wrote a paper about sort of how many threats are available
27:50
just in the base MCP specification.
27:52
Mm-hmm.
27:52
There are over 40 threats across 12 categories that they've identified,
27:55
including identity privilege escalation, uh, supply chain integrity,
27:58
network security, and so forth.
28:02
I think for enterprises- The first thing is create an MCP inventory.
28:07
Um, just have an allowed list of sort of trust, trusted servers.
28:11
Uh, we need to make sure there is authentication authorization.
28:14
That was not given in MCP, surprisingly, right?
28:16
Mm-hmm.
28:16
Like as you can imagine, I mean, this is a age-old solved problem, but
28:19
it's still one of those, uh, younger ones in this, uh, in this new world.
28:23
Sandboxing execution, being able to sort of really isolate
28:26
execution, uh, when it does.
28:28
I think all of these are important things.
28:30
Um, one thing I would also add, uh, Kevin, is skills.
28:34
Agent skills are becoming another sort of level of- Mm ... MCP servers.
28:39
So what happens is MCP servers are these tools, and then agent servers
28:43
have this higher layer that allows you to sort of define the workflow
28:47
of how this tool can be used.
28:49
Um, I think that's going to be another interesting surface to sort of watch
28:56
as, uh- Mm ... because anybody can-- Because all of these instructions are
28:59
written in text files, um, think about all of these different s- uh, skills
29:04
that will be out there in the company, uh, and sort of really managing that,
29:08
this at scale is going to really have, uh, interesting challenges, right?
29:12
Like I- Mm-hmm ... I think OWASP wrote something called
29:15
as, um, a lethal trifecta.
29:18
Uh, I think when a skill or an agent, or even in this case MCP, have access to,
29:24
let's say, private data, it has access to external communication, and then it ca-
29:29
it gets exposed to some untrusted content.
29:33
Speaker 3: Mm-hmm.
29:33
Venkat: That is a lethal trifecta, right?
29:35
Mm-hmm.
29:35
Like so you do have to assume in those cases that there will be
29:39
exfiltration that's possible, uh, unless you have strong controls.
29:42
So, uh, being able to sort of keep these mind, uh, and sort of ensuring
29:46
that there are specific controls are gonna be so important for enterprises.
29:49
Kevin: Last thing I wanna ask you is, we're, we're running low on time.
29:52
Uh, going forward, what happens when agents are embedded in physical things?
29:58
And I, and I ask in particular because you mentioned you worked
30:00
at Rivian, which builds cars.
30:02
So what does that world look like in terms of the risks of agents?
30:06
Um, and then, you know, can we overcome the kinds of challenges with the harms
30:10
that might come about in that environment?
30:12
Venkat: Absolutely.
30:13
I think digital- failures are often recoverable, right?
30:17
Physical failures have more consequential, uh- Mm-hmm ... and
30:20
irreversible sort of impact.
30:23
Uh, governance should largely shift to pre-action guarantees
30:27
where- Mm-hmm ... the system proves safety before execution, uh, right?
30:32
Like not just explains its action later.
30:34
Um, I mean, imagine a robot with a hazardous chemical in the factory, right?
30:39
Mm-hmm.
30:39
Uh, or a home robot without access to knives.
30:42
Uh, these are all dangerous situations.
30:44
I, I think the lesson that we could take from automotive is they do
30:47
not just rely on intelligent model.
30:49
It's not like every time when a new update comes, it's, it's a bigger, larger model.
30:54
It's about really thinking safety as an entire stack, right?
30:59
You need to have simulations.
31:01
You need validations.
31:02
You need redundancy in the system, fail-safes if something were to go wrong.
31:06
How can you monitor this, uh, with telemetry?
31:09
And sort of really establishing clear boundaries on what this system is
31:13
allowed, uh, to operate where, right?
31:16
Um, I think one way to think about this is, um, governance
31:20
need to be closer to the safety engineering in the physical world.
31:23
Uh, you need to ask questions like, okay, what's the operating envelope?
31:27
Uh, what happens if a sensor disagrees?
31:29
These are all problems- Mm-hmm ... I think from a software world you, you can...
31:34
These are not gonna be catastrophic, right?
31:35
Like y- what is the safety net, for example?
31:38
Okay, can this agent really move fast, uh, the location and the proximity limits that
31:43
it actually has access to and so forth.
31:45
So yeah, I mean, so hallucination are just embarrassing in this world,
31:49
and in the physical world, I mean, if, uh, I mean, for hallucination of
31:53
forklift, you need bumpers, right?
31:54
Like you need to sort of... You need to think about this more holistically than
31:57
just, uh, uh, just a, a software problem.
32:01
Kevin: Absolutely.
32:02
Well, it's an exciting and a scary world that, that we're going into, but one
32:06
where the, the kind of work that you're doing is absolutely going to be critical.
32:09
Venkat, thank you so much.
32:10
Really a pleasure to speak with you.
32:11
Okay.
32:12
That's great.
32:12
Thank you.
32:14
This has been "The Road to Accountable AI." If you like what you're hearing,
32:18
please give us a good review and check out my Substack for more
32:21
insights on AI accountability.
32:23
Thank you for listening.
32:28
If you wanna go deeper on AI governance, trust, and responsibility with me and
32:32
other distinguished faculty of the world's top business school, sign up for the
32:36
next cohort of Wharton's Strategies for Accountable AI online executive education
32:41
program, featuring live interaction with faculty, expert interviews, and
32:45
custom-designed asynchronous content.
32:48
Join fellow business leaders to learn valuable skills you can
32:51
put to work in your organization.
32:53
Visit execed.wharton.upenn.edu/acai for full details.
33:01
I hope to see you
33:01
there.