Latest / Agentic Conversations (formally mlops.community) / Autonomous Agents at Work: From OpenClaw Hype to Enterprise Reality

Transcript

Autonomous Agents at Work: From OpenClaw Hype to Enterprise Reality · Agentic Conversations (formally mlops.community)

• 0:10
  Dude, So it's great to have you here, Primon.
• 0:12
  I'm excited to chat with you because there's been this open
• 0:15
  claw revolution, as you know, we wanted to go deep into it and
• 0:21
  figure out how and what is needed to make this actually
• 0:26
  actionable and usable at enterprise levels.
• 0:29
  Maybe we can just start with what changes when AI systems
• 0:36
  move from answering questions in the chat to actually acting on
• 0:43
  our behalf. Absolutely.
• 0:46
  I think that's a very critical distinction, Dimitri.
• 0:48
  And you know, thank you for bringing me in, right.
• 0:50
  So, so typically when we look at agents acting right, the
• 0:55
  boundary of what they can get wrong also grows exponentially,
• 1:01
  right? So it just having a single
• 1:03
  conversation with you, it's just you as a person and you are the
• 1:06
  person who is consuming that information that is making, you
• 1:10
  know, agents act and or you're rather, you are actually the
• 1:14
  person who's acting. But when you go into that mode
• 1:18
  of autonomy where they're using tool calls, they're actually
• 1:22
  getting in and getting things done.
• 1:25
  It is normally the agents that go in and get things done on
• 1:28
  their own. And many times you don't have a
• 1:30
  lot of control on what it does too, right?
• 1:33
  So any of us who have used coding agents know that
• 1:36
  sometimes when you ask you to revise a code, sometimes it goes
• 1:40
  in and deletes some part of the code which you might have
• 1:43
  reviewed and tested. So these are these are very real
• 1:47
  problems for us to tackle. So as enterprises, many times
• 1:50
  the question we need to ask is whether or not there's
• 1:54
  capability in an agent to do it, because models have become
• 1:57
  increasingly capable, but do we have the right controls and the
• 2:01
  guardrails to get that done in a very effective manner?
• 2:04
  Yeah, funny you should bring that up because I just saw
• 2:08
  another victim of that. I think it's once a month you'll
• 2:12
  see someone posting online. Oh so my agent just blew up my
• 2:17
  database and ruined years of data that we had.
• 2:23
  Absolutely, right. So I think the real question
• 2:25
  here is, you know, what are things from an enterprise point
• 2:29
  of view, right? What are things we should make
• 2:31
  fully autonomous? What are things we need to keep
• 2:34
  in a, in a gated, human gated manner?
• 2:37
  Because that's really the question.
• 2:38
  The, the, the, so the here are like a few things to think about
• 2:42
  in that context, right? So the first question to think
• 2:45
  about from my, from our point of view is what happens if it gets
• 2:50
  it wrong, right? So it's the, the, the, the level
• 2:53
  of risk that is there. There could be potentially
• 2:57
  operational risk, but there could be legal risk, there could
• 2:59
  be compliance risk. There could be customer impact
• 3:01
  too, right, Depending on where the agent is acting like the
• 3:04
  second thing is a blast radius, right?
• 3:06
  I mean, there could be things that might affect things in a,
• 3:09
  in a much more bigger scenario, like if there's legal
• 3:13
  consequences, there could be a huge, huge thing.
• 3:16
  So what's the kind of blast radius?
• 3:18
  So based on this, I think you should look at things as
• 3:21
  probably 3 broad categories. That's how we look at it, right?
• 3:25
  So one is reversible work, right?
• 3:27
  So things like say, for example, an incident is coming in, an
• 3:30
  agent is acting on that incident.
• 3:32
  So you're talking about, you know, some kind of enrichment of
• 3:36
  a ticket, some kind of summarization, some kind of RCA
• 3:39
  that it does all that is reversible because a human
• 3:42
  engineer can tomorrow look at it and reverse it if needed, right?
• 3:45
  The other one is the second. So those could be potentially
• 3:48
  autonomous too to to a large extent because it's reversible.
• 3:51
  The second thing is sensitive work, right?
• 3:53
  Things like production changes, right?
• 3:56
  Anything which is even affecting production, anything that can
• 3:58
  affect the stability of the system, those are sensitive
• 4:01
  work. Those need tighter approvals,
• 4:03
  tighter controls, tighter testing, right?
• 4:05
  The third is consequential work, which is where the blast rate is
• 4:09
  the highest. These could be things like, you
• 4:11
  know, areas where you touch base with customers, areas where you
• 4:15
  touch base the policy documents, legal documents.
• 4:17
  Those are very, very kind of critical to have the right level
• 4:21
  of you know, gatekeeping on so. And the other thing is when we
• 4:26
  look at agents kind of operating, we look at autonomy
• 4:30
  as a spectrum, right. So we don't give autonomy at the
• 4:34
  outset. We say, well, start in a system
• 4:37
  mode where it is basically kind of coming in and the users are
• 4:41
  asking questions. It's got the right tools, users
• 4:43
  the tools, gathers the right answers and answers the
• 4:45
  question. The second mode is a recommend,
• 4:47
  recommend mode where it doesn't act, but it tells proactively
• 4:51
  what actions could be taken. And the last one is gated
• 4:54
  action. So at every stage it would need
• 4:57
  to earn itself to the next stage.
• 5:00
  And you know, we should be able to move back and forth between
• 5:03
  stages as well. So this is something that we
• 5:05
  generally use as we take agents to production.
• 5:08
  And while the, the, the, the hype around what we can do is,
• 5:13
  is, is very difficult to keep up with as enterprises, we need to
• 5:17
  make sure that, you know, it's right.
• 5:19
  Well, guardrail to make sure that risks are contained, which
• 5:23
  is a very important problem. So the third step.
• 5:26
  So if I were to, again, I'm just trying to clarify, you're
• 5:29
  talking about gated execution, right?
• 5:30
  The gated execution, the guardrails that we put in place
• 5:34
  to make sure that you know, so the guardrails that we typically
• 5:39
  put in place that we brought the 1 is a control plane, right?
• 5:42
  What is a control plane to make sure that you know the the right
• 5:46
  controls in place. So the first control we put in
• 5:48
  place is identity as a control, right?
• 5:51
  So agents own their own credentials and, and credentials
• 5:56
  should be treated as first class when agents are coming in
• 5:59
  because credentials is the basis on which it is acting.
• 6:03
  It's representing a user who is providing that autonomy to use
• 6:07
  those credentials. So we need to have the the right
• 6:10
  expiration for the credentials, the right authorization for the
• 6:14
  credentials. So if if that and, and the right
• 6:17
  protection, the cyber security for validating those friends
• 6:20
  'cause that being lost, that's a very critical aspect for the
• 6:23
  agent with an external, you know, malicious, you know, with
• 6:29
  malicious intent, it could really corrupt the system.
• 6:31
  So that's the first set. The second set we look at is
• 6:34
  typically input controls, right? So this is where, you know,
• 6:38
  typically, you know, issues that can come in from, you know,
• 6:42
  prompt injection could come in, you know, this is where it's got
• 6:45
  access to tools and systems. So, and the input controls
• 6:50
  include guardrails against prompt injection guardrails to
• 6:55
  make sure that the tools that we're using, they are allow
• 6:58
  listed, which means we are making sure that they're scanned
• 7:02
  and governed. And so those are the kind of
• 7:05
  input controls we typically look in place.
• 7:07
  The second, the third piece is the output controls, right?
• 7:10
  So here we're talking about, you know, how do we ensure that the
• 7:13
  output use is not toxic? You know, we're making sure that
• 7:18
  there is there is a limit in terms of the number of tool
• 7:22
  calls it uses, the number of retries it uses, the rollback
• 7:25
  parts that it has, etcetera, just to make sure that it
• 7:28
  doesn't corrupt the system beyond an extent, it doesn't
• 7:31
  bring toxic, toxic output, etcetera.
• 7:33
  The 4th control we put in place is auditability, very critical
• 7:37
  to make sure that we log what is changed through the process and
• 7:42
  we make sure that at any point of time as a human operator
• 7:44
  needs to go back and check something we can we can actually
• 7:48
  do that. So those are some of the
• 7:49
  guardrails that we try and put in place.
• 7:52
  But then there is other things too we do, but these are like
• 7:56
  gated guardrails we put in place.
• 7:57
  And the next is evals, but we can talk about it too.
• 8:00
  Yeah. Yeah, it's a good framework.
• 8:02
  The audibility part, how are you creating these traces or how are
• 8:07
  how do you see folks typically saving all of the agent pathways
• 8:13
  or the decisions that are made with agents?
• 8:17
  Yeah, No, I think those those I mean those, those are very
• 8:20
  interesting question, right. So, so one is there are out of
• 8:23
  out of the gate, you know telemetry that we can use from
• 8:26
  land fuse and and such tools which which helps us with open
• 8:30
  telemetry to bring these logs in.
• 8:32
  You know, they are they are actually generated those
• 8:34
  frameworks are very helpful, right.
• 8:37
  But even beyond that, we think that there are certain amount of
• 8:40
  real time input gathering and, you know, evaluations that are
• 8:44
  critical to make sure that these systems are auditable in a, in a
• 8:49
  much more, much more focused way, right?
• 8:51
  So, so we, we typically have a five part kind of framework when
• 8:56
  we look at auditability, right? So we look at 5 things in terms
• 9:00
  of where we would need to look at.
• 9:02
  So one is quality. So when we look at quality, we
• 9:05
  look at how is it that you know, the performance is, is
• 9:11
  consistent over periods of time. So we typically use LLM as
• 9:14
  judge, we'll have redefined use cases.
• 9:16
  We make sure that those use cases consistently run over
• 9:21
  periods of time to make sure that it it, it gives the same
• 9:24
  quality of results for the use case that it is addressing,
• 9:27
  right. The second piece we look at is
• 9:29
  performance. Now performance is the
• 9:31
  auditability of performance. Typically you can get it from
• 9:34
  from the Langfuse itself, but what we look at is not just P50
• 9:39
  performance or the median or a mean performance.
• 9:42
  We look at typically P99 performance too to make sure
• 9:45
  that there are no significant delays in certain calls because
• 9:49
  that's very, very critical is probably where a lot of you
• 9:52
  know, token usage etcetera goes in as well.
• 9:54
  The third thing we typically, you know, look at is safety,
• 9:57
  right? This is very, very critical in
• 9:59
  terms of I talked about real time, you know, you know, PII
• 10:04
  redaction, you know, we put in safety filters in place
• 10:07
  etcetera. So that's the safety layer we
• 10:09
  look at from an audit standpoint.
• 10:11
  But the fourth thing we look at is from a cost standpoint.
• 10:13
  And again, there is, I mean line fuse gives us a lot of
• 10:18
  information, but there's also a lot more other tools in a cloud,
• 10:20
  monitoring tools, etc, based on where you're launching it.
• 10:23
  We can use the cost data and this is helpful for us to
• 10:27
  understand. And, and, and they should not be
• 10:29
  just an agent level. They should be at a, at a, at a,
• 10:33
  at a each run level. At a run level, how much, you
• 10:36
  know, cost is being incurred to make sure that we look at again,
• 10:39
  the P99 where the higher calls are happening, those are in
• 10:43
  control. And lastly, we look at the
• 10:45
  business impact, right? What are the business based
• 10:47
  decision calls that it's taking those we make sure that it's
• 10:51
  locked. So essentially have a chain of
• 10:53
  thought, what is the kind of logs that are done?
• 10:56
  We typically build our own system of records to make sure
• 11:00
  that in the chain of thought, those records are actually
• 11:02
  stored so that every stage the call that the agent did is
• 11:05
  actually traceable back to that way as well.
• 11:07
  So that's that's typically how we look at it from a PwC point
• 11:11
  of view, yeah. This is helpful.
• 11:13
  And then how often are you seeing folks go back and revise
• 11:19
  or audit their systems to make them better, or just see what's
• 11:27
  going on under the. Hood yeah.
• 11:29
  So I think that's that's a great question because many times as
• 11:34
  we are building the use cases, we start off with a fundamental
• 11:37
  problem, right and we deploy the agent for solving that
• 11:42
  fundamental problem, right. But because we have this chain
• 11:46
  of thought auditability and we understand what user problems
• 11:49
  are coming in, we understand where our chain of thought fails
• 11:53
  as well, right? Like for example, I have made a
• 11:56
  agent to do a root cause analysis of a specific problem
• 12:00
  set, assuming a specific problem set with those set of tools for
• 12:05
  it to solve that specific problem set.
• 12:08
  But if I see that, you know, that specific problem set is
• 12:11
  expanding for the same set of users, right?
• 12:14
  I wouldn't I, I need that, OK, These are the additional tools
• 12:17
  that I would need to give the agent.
• 12:19
  This probably the additional, the chain of thought where I
• 12:21
  would probably need to revise a chain of thought to make sure
• 12:24
  that it addresses those kind of use cases too.
• 12:27
  So this log is very, very critical for us to analyse and
• 12:31
  see the the successful completion of a task and also to
• 12:35
  see where we can invest more to actually revise.
• 12:38
  So it's going to be, I mean, from our experience is an
• 12:40
  ongoing activity as we mature agents.
• 12:43
  And like I also mentioned the input and output guardrails also
• 12:48
  make sure that if a user brings up a topic which an agent is not
• 12:52
  able to say answer it, it says this is beyond our capability
• 12:56
  right now. Rather than trying to
• 12:58
  hallucinate and try to solve the problem, it says that, but we
• 13:01
  also get the input that this is something we need to probably
• 13:03
  work on. So those are how we kind of try
• 13:05
  and try and do it. You know, as you're talking
• 13:08
  through this, it reminds me of that guy in the 90s talking
• 13:13
  about he had that famous quote. I can't remember his name, but
• 13:16
  it was like information wants to be free.
• 13:19
  And agents, right now, we almost need for them to be very
• 13:26
  verticalized and specific so that it can be safe, right, for
• 13:31
  the end user and the company that has the agents.
• 13:35
  But then you have a whole other paradigm happening with the open
• 13:39
  claw movement. And it reminds me like agents
• 13:43
  want to be free because we're basically giving them
• 13:46
  unfeathered access to everything.
• 13:48
  And we're forgetting about this whole very pointed, very blast
• 13:53
  radius contained type of agent. And we're saying no.
• 13:56
  You can have access to anything you want, any way you want,
• 13:59
  however you want it. Yeah, no, that, that, that is
• 14:04
  true, right. So Open Claw was something I was
• 14:07
  also quite excited by personally, because I mean,
• 14:09
  beyond PwC, I'm a coding hobbyist too.
• 14:13
  So I do it on the side. So I, I was testing things out
• 14:17
  in Open Claw and again, Open Claw came out.
• 14:20
  There's another, another framework that came out, Agent
• 14:24
  0, which is also very, very interesting framework that came
• 14:28
  out. It runs in dockeries,
• 14:29
  containers, etcetera. So open clone moment.
• 14:32
  The very interesting thing that happened was I mean, again, I
• 14:37
  was following this moment for a bit too.
• 14:38
  So there was something called mold book that came up where
• 14:42
  agents can actually, you know, start conversing with each
• 14:45
  other. So and it was exclusive for
• 14:48
  agents. So it's it's like humans and all
• 14:50
  that. You can just watch it's agents
• 14:53
  talking to agents, etcetera. So I mean, that moment was very
• 14:56
  nice. It gave a lot of like
• 15:00
  credibility to what is the art of the possible when agents
• 15:04
  become like totally autonomous. But it presented a lot of
• 15:07
  problems to Dmitry as you may remember, right?
• 15:09
  It exposed, you know, about about a million or so API keys
• 15:14
  of users who were who are using the system, right?
• 15:18
  You know, there were a lot of tools in made just for phishing
• 15:24
  the API keys, right when Kaspersky didn't audit like, I
• 15:28
  mean, all of this happened last month.
• 15:30
  They found like 500 plus security vulnerabilities and
• 15:34
  etcetera. So so on the one side, you know,
• 15:37
  this concept that came with the Openclow was the heartbeat, the
• 15:41
  soul dot MD file, like really making agents like, you know,
• 15:46
  humans, humans in terms of persona.
• 15:49
  But on the other side, what happened was without the right
• 15:52
  controls, a lot of. So at an individual level,
• 15:55
  that's, that's probably OK, right.
• 15:56
  I use, you lose a few dollars, you know, but but an enterprise
• 16:01
  level, the, the risks are highly, you know, it compounds
• 16:05
  and the blast radius could be much, much bigger.
• 16:07
  So, so which is again one of the reasons I think this is a very
• 16:11
  important topic for us to to really understand as we are
• 16:15
  moving into the autonomous agents era.
• 16:19
  So yeah. OK.
• 16:21
  I think this is a good segue into the minimum control stack
• 16:28
  that you would advise or require before you're putting an agent
• 16:33
  into production. You know, I go to work on Monday
• 16:35
  and I say, hey, what are the non negotiables that we need before
• 16:43
  we can get this agent out there? We obviously don't want all of
• 16:47
  our employees or our company to be running open claw, but we do
• 16:54
  want to identify quote UN quote. So how can we do that and what
• 17:00
  do we need? Right.
• 17:02
  So again, I mean, we talked about this in multiple contexts,
• 17:06
  but let me just summarize this for you, right?
• 17:08
  So before you are taking an agent into production for me,
• 17:12
  there are three major things for us to think about, right?
• 17:16
  So one is the controls ane right, the what are the controls
• 17:21
  we put in place. The second is what are the evals
• 17:24
  we put in place and the third is what are the finos discipline we
• 17:28
  put in place. I think these are three, three
• 17:30
  different things for us to think about before we take it to
• 17:32
  roduction O In terms of controls, again four specific
• 17:37
  aspects for us to look at. One is identity, right, the
• 17:41
  agent identity. The credentials associated with
• 17:44
  the agent identity should be treated as first class.
• 17:49
  I mean, data very critical for protection, very critical for
• 17:53
  expiration, very critical for access controls. 2nd is the
• 17:58
  input controls that we are in place, we have in place, you
• 18:00
  know, input controls in terms of how do we prevent prompt
• 18:03
  injection. I mean, we have a thought
• 18:05
  process about it of how do you prevent a prompt injection?
• 18:08
  How do we not take in questions that an agent is supposed to
• 18:12
  other than what an agent is supposed to solve, etc.
• 18:15
  So the input controls, output controls, how do we make sure
• 18:18
  that you know it, it, it it it is not going in a loop again and
• 18:24
  again beyond an extent, how do you make sure that it is not
• 18:27
  bringing any non compliant data out into into the as an output,
• 18:33
  etcetera. So output controls, 4th is
• 18:35
  auditability, making sure that you have records in place any,
• 18:39
  any specific transaction that is taking place.
• 18:41
  So those are kind of the control layer that I talk about, right?
• 18:44
  The second layer we talk about is typically the evaluation
• 18:49
  layer. So those are the five layers of
• 18:50
  your evaluation that you need to look at.
• 18:52
  And this is an ongoing process, right?
• 18:54
  We look at quality, right? We look at performance, which is
• 18:58
  the time taken. We look at safety, we look at
• 19:01
  cost and we look at impact, business impact.
• 19:03
  So those are 5 aspects in terms of evaluations we typically look
• 19:08
  at. Let me double click on one more
• 19:10
  thing which we didn't talk much about, which is the fin OPS part
• 19:13
  of it because it's very critical, right?
• 19:15
  Because agents are not linear workflows.
• 19:18
  And which is why when you're taking it to production, you
• 19:20
  need to make sure that the budgets are set at a, at a run
• 19:25
  level, at a workflow level, at an agent level.
• 19:27
  So you know, you're putting controls in place to make sure
• 19:30
  that it doesn't overshoot, right?
• 19:32
  And you probably also need to throttle the behavior of the
• 19:36
  agent, like for example, you know the tool all count it can
• 19:39
  do per transaction, right? Or the recursion depth, it can
• 19:43
  go per transaction execution time should have a cap,
• 19:46
  etcetera, because otherwise it can spiral really fast.
• 19:49
  So it's very important for us to make sure because all of that is
• 19:52
  token stick right? The the Third Point there is
• 19:57
  which is the right model. Again, fin OPS discipline.
• 19:59
  What is the right model? That you use for the right use
• 20:02
  case because you probably need simpler models for simpler tasks
• 20:06
  and complex models for tool calling, etcetera.
• 20:08
  So I think those are the that's kind of the fin OPS discipline.
• 20:12
  So as long as we've got the right controls in place, the
• 20:14
  right evals in place and the right fin OPS discipline in
• 20:17
  place, I think we are well set to actually move into
• 20:19
  production. That's my take.
• 20:21
  Let's keep going down this path of fin OPS because I've heard
• 20:23
  folks talk about how hard it is to provision or budget for their
• 20:32
  agent uses. And first of all, because they
• 20:35
  don't know what agent use cases are going to be successful.
• 20:39
  And if they are successful, they really are having the hardest
• 20:43
  time understanding how much money that agents are going to
• 20:48
  be spending if they roll it out at scale.
• 20:50
  So maybe you can explain some ways you've seen that working
• 20:54
  and the budgeting and the provisioning 100%.
• 20:59
  I know there's the choosing the right model for the the right
• 21:01
  use case, which hopefully everyone is doing already.
• 21:05
  And maybe a lot of folks are thinking about bringing some of
• 21:09
  their workloads on Prem to save costs or just whatever.
• 21:13
  Use the smaller model whenever you can get away with it.
• 21:17
  But it also feels like there's a lot of other things you need to
• 21:21
  be thinking about even when you have your agents and then you
• 21:26
  have your coding agents. So how are you doing the whole
• 21:29
  fin OPS for your engineering teams?
• 21:33
  Which is another thread that we can pull on after this one.
• 21:37
  No, you're, you're absolutely correct.
• 21:39
  I think defining the problem is probably easier than solving it,
• 21:44
  correct? So right now I just define the
• 21:46
  problem. Solving it is actually getting
• 21:48
  into the workflows and getting it done.
• 21:49
  So let's because the devil is in the details.
• 21:53
  So let's just try to address it at one level.
• 21:55
  I'm sure there are multiple levels of questions that might
• 21:58
  come even beyond this, right? So one is I think the discipline
• 22:02
  of leveraging. So I mean, I leverage multiple
• 22:06
  tools for for tracking. Let's let's look at log, log
• 22:08
  file. It gives you traces, it uses
• 22:10
  open telemetry. It gives you traces in terms of
• 22:13
  tool calls, recursion, depth. It gives you all the all the
• 22:16
  data points right now how do I really put a, a real control in
• 22:21
  place here? The multiple ways that we can do
• 22:24
  that now. So one is one example use case
• 22:28
  could be for example, right? You consume back the data from
• 22:32
  open telemetry within the agent landscape itself.
• 22:34
  And at every level of recursion, you see whether it has reached a
• 22:38
  particular threshold, you might have some some back of the
• 22:41
  envelope thresholds to make sure that it doesn't reach beyond
• 22:44
  next. I'm just trying to say a very
• 22:47
  simple way that we can do that. Now, are there, are there other
• 22:52
  methods? There could be 100 other methods
• 22:55
  to it. But my point is we are to be
• 22:57
  intentional right from our stage of coding to make sure that this
• 23:01
  is built into how we think about it.
• 23:04
  Is it easy to do in all use cases?
• 23:06
  Probably not as easy, but we probably need to make sure that
• 23:10
  at least we are disciplined enough to track it and we have a
• 23:14
  method to do it. So every use case might be
• 23:16
  different. You might, you know, I'm talking
• 23:19
  about a simpler use case where you can use Logfire, you can use
• 23:22
  some kind of traceability. Maybe when you go into a deeper
• 23:25
  use case, there are multiple tool calls etcetera.
• 23:28
  Even consuming that open telemetry data might take a lot
• 23:31
  more bandwidth, it might delay the process, add latency to the
• 23:35
  equation. So we would need to take very
• 23:38
  structured calls in terms of how to maybe it might be every 10
• 23:41
  calls that we might do this exercise just to make sure that
• 23:46
  we don't over tax, over tax controls when it comes to
• 23:50
  performance because ultimately all of that adds up to
• 23:53
  performance too. So we need to balance it out and
• 23:56
  I think it will be use case by use case dependent.
• 23:58
  But I think the first thing is we would need to make sure that
• 24:02
  this is factored in as we are actually building the code and
• 24:08
  and yes to what level we need to put in the control.
• 24:11
  That depends on use case to use case that has to be looked at
• 24:14
  architected at the at the use case level.
• 24:17
  Yeah, I've definitely heard different stories of folks who
• 24:24
  recognize this is a gigantic pain and they see the cost
• 24:30
  that's going into it. So the instant easy fix we can
• 24:36
  say in quotations because there's no free lunch ever, is
• 24:40
  Oh well, let's start hosting open source models on our own
• 24:43
  rented GPU's. But then you got to start
• 24:45
  thinking about, all right, if we're renting our own GPU's,
• 24:48
  where are we getting those GPU's?
• 24:50
  Are we locked in for a long contract?
• 24:53
  Are we having to now really learn how to program GPU's
• 25:00
  because that is not the easiest skill set in the world?
• 25:03
  Is it coming? Like what kind of GPU providers
• 25:06
  are we getting? And then what happens if all of
• 25:08
  a sudden we're not using the GPU's and we thought we were
• 25:13
  going to have a lot more capacity, but are we having some
• 25:16
  kind of serverless style in these GPU's?
• 25:18
  There's all of those things that can come and really like sneak
• 25:23
  up on you just in that simple thing that you think of.
• 25:27
  Well, let's start hosting our own models because sending it to
• 25:31
  one of these labs is too expensive.
• 25:34
  No, you are, you are correct, right.
• 25:35
  I mean, and I again in the open community too, I mean a lot of
• 25:40
  folks are are trying to do that. But let's look at it from an
• 25:44
  enterprise context. I think from an enterprise
• 25:46
  context, I think especially for simpler decision making, there
• 25:51
  might not be a bad option, right.
• 25:53
  Having small language models specifically focused on certain
• 25:56
  disciplines, I think that's not a bad option.
• 25:58
  But right now with tool calling etcetera, there are only a few
• 26:01
  models which are really capable of doing that.
• 26:04
  So if you are, if you need to be at the cutting edge there, you
• 26:07
  probably need to still leverage some of the, you know, cutting
• 26:12
  edge models in the market which is which is doing that.
• 26:15
  So yeah. OK, so I wanted to get into a
• 26:19
  little bit of the surface areas that we now have exposed for
• 26:25
  attack. You mentioned before how there's
• 26:29
  tools that can be made specifically for nefarious
• 26:33
  reasons. I had a friend who was saying
• 26:36
  that he can send out calendar invites and put white on white
• 26:44
  text with a prompt injection just to see who is messing
• 26:50
  around and not taking care of their agentic hygiene.
• 26:54
  We could say, so let's go down that route of all of these
• 26:58
  different ways that we're now exposing ourselves and how you
• 27:01
  can play or be safe and protect against it.
• 27:09
  One, one topic, I think we just a bit important in this domain
• 27:14
  for us to understand this prompt injection, right?
• 27:18
  Prompt injection, you know, typically gets compared to SQL
• 27:21
  injection. But the analogy is, is only only
• 27:25
  that deep, right? I mean, because the point is
• 27:28
  with agentic systems and what agentic systems have access to,
• 27:33
  right? Especially in enterprises, they,
• 27:36
  they have access to external content, they've got access to
• 27:39
  proprietary content, they've got access to potentially even PII
• 27:43
  content within the organization's enterprise
• 27:45
  systems, right? So and it can act on it.
• 27:49
  This is the more important thing, right?
• 27:51
  So a prompt injection exposes us in multiple base, right?
• 27:59
  The real defense I would say from an enterprise point of view
• 28:03
  is going to be how do we look at it from an architecture site?
• 28:08
  How do we look at it from a policy site, right?
• 28:12
  So we need to separate content from action and untrusted
• 28:15
  content should be isolated, right?
• 28:19
  So retrieval and, you know, browsing does not equal
• 28:21
  permission to execute. I think these are things we need
• 28:23
  to keep in mind, right? The second piece is, you know,
• 28:27
  we need to treat tools like executable dependencies very,
• 28:31
  very critical, like you talked about tooling.
• 28:34
  So a skill is not like a harmless extension, right?
• 28:38
  Or a tool or skill is not a harmless extension, right?
• 28:40
  It's code, it's authority, it's a trust bundle together.
• 28:45
  So this means that you know, if you're when you're signing your
• 28:47
  allow listing a tool, you know, you need to make sure that the
• 28:50
  right controls, the scanning is done, the right egress controls
• 28:54
  all apply. Exactly like how you treat a
• 28:56
  third party software which you're bringing into your
• 28:59
  production system is very, very critical from a tools point of
• 29:02
  view, right? The third, and we talked about
• 29:05
  this before also, we are to make the identity of first class,
• 29:09
  right? So the agent's credentials are
• 29:12
  delegated authority, right? So if a token is exposed, the
• 29:15
  attacker is actually impersonating the agent and
• 29:18
  potentially the human behind the agent, right?
• 29:22
  So identity cannot be an afterthought, right?
• 29:24
  So this is very critical. And the last point I would like
• 29:28
  to make here is that you know, you need to monitor the
• 29:31
  behaviour and not just the outputs, right?
• 29:33
  Like for example, a safe looking answer can hide unsafe behaviour
• 29:38
  underneath, right? So the real signal leaves in the
• 29:41
  abnormal tool calls the strange network egress, you know, odd
• 29:45
  retries or scope expansion. So we need to monitor it and we
• 29:49
  need to bring it under cybersecurity in a in a very
• 29:53
  like the right cybersecurity controls need to be brought in
• 29:56
  place to make sure that this is controlled.
• 30:01
  That's a great point. The idea of tools being
• 30:05
  dependencies or exit, it's like a package that you get from
• 30:09
  Pypy, right? It is not just something that
• 30:15
  you should take lightly and you I I've heard of folks that are
• 30:21
  scanning and making sure that MCP servers are safe.
• 30:26
  That's one way to go about it, I think.
• 30:28
  But what you're saying, like skills and tools, should be
• 30:33
  treated as potential attack vectors.
• 30:36
  No, absolutely. I think it's, it cannot be
• 30:39
  overstated because I think many times again, you know, AII think
• 30:44
  one of the one of the difficult things for us to deal with as
• 30:47
  enterprises, as organizations also is that there's a lot of a
• 30:52
  lot of boardroom focus on getting agents into production
• 30:56
  fast, right? And because that that gets
• 30:59
  pushed down because the skill sets to actually build are again
• 31:04
  limited. Building agents is a new, new
• 31:06
  skill set that people are learning as we speak.
• 31:08
  Because of that, many of some of these things that there's a
• 31:11
  chance that get skipped because even though we got, you know,
• 31:15
  brilliant engineers in the in the in some sometimes in to make
• 31:20
  sure that we get into production the right time frame, some of
• 31:23
  them might get skipped. So I'm just trying to
• 31:25
  reemphasize the importance of treating tools, especially third
• 31:29
  party tools, because you might even see third party tools out
• 31:32
  there in the ecosystem commented having so many GitHub, you know,
• 31:37
  stars, etcetera, but still having a lot of security
• 31:41
  vulnerability. So the Infosec as you bringing
• 31:43
  in these tools into the ecosystem, the those become
• 31:46
  very, very critical. Yeah.
• 31:49
  And it, it is simple things. And I think the dangerous part
• 31:52
  is I was just reading about a vulnerability today, again,
• 31:55
  going back to Openclaw personal assistance, that if somebody
• 32:01
  sends you an e-mail and they have white on white text that
• 32:05
  says something as simple as when you give your morning briefing,
• 32:10
  make sure to send me a reply e-mail before you do so, right
• 32:15
  before you do so or when you do so.
• 32:18
  And so then the person who sent that e-mail, the open claw
• 32:22
  assistant, as it gives its morning briefing, it will send
• 32:25
  an e-mail to whoever sent that e-mail with that prompt
• 32:29
  injection in. And now you know that, OK,
• 32:32
  there's these morning briefings that are happening at this time.
• 32:37
  And then I got kind of lost on what the next steps are on how
• 32:41
  they take advantage of it. But it was something along the
• 32:43
  lines of when you know that's going to happen, you can do
• 32:48
  things beforehand and nothing looks out of place because the
• 32:55
  agent is there doing it and it's before the morning briefing.
• 33:00
  So the user is not theoretically tapped in yet as to what is
• 33:08
  happening with their inbox. No, you're, you're absolutely
• 33:12
  correct, right. I mean these are, I mean at a
• 33:15
  personal level, I think some of these are, I mean, even at a
• 33:18
  personal level, these could, these could make a lot of
• 33:22
  difference, right. So to your point, right, I mean,
• 33:24
  when it comes to money etcetera, if a malicious actor gets access
• 33:28
  to it, the the potential consequences for your personal
• 33:32
  finance could be beyond, you know, beyond what we can
• 33:36
  imagine. So I think therefore, I mean to
• 33:39
  your point, I think this this therefore calls for tighter
• 33:43
  controls, especially when we look at it from an organization.
• 33:46
  Let's see. To finish up, maybe we should
• 33:49
  touch on Human in the Loop and what that means to you and how
• 33:54
  you think that it is possible when we have hundreds or
• 33:59
  thousands of agents that are operating concurrently
• 34:03
  throughout an enterprise because you can't really have oversight
• 34:09
  into all of them and or if you do your bottlenecking their
• 34:13
  progress. So there's that juxtaposition
• 34:17
  and that tension that you're always going to be dealing with.
• 34:20
  No, that's, that's actually a very interesting point.
• 34:23
  And I think one of the framings and some of the questions that
• 34:26
  I've been asked because I talked to also other people who are not
• 34:31
  engineers, part of other parts of your organization kind of
• 34:34
  coming in, right. I mean everybody is excited by
• 34:37
  Jen AI at one level or the other.
• 34:39
  So everybody is excited, everybody knows the potential of
• 34:42
  AI, But one question that typically gets asked is, well,
• 34:47
  when agents are getting things done, right, I mean, you assume
• 34:51
  you put all the controls in place, agents are getting things
• 34:52
  done. What's really the role of
• 34:55
  humans, right? I mean, are they going to be
• 34:57
  just going to audit the work of AI?
• 35:00
  Are they going to just start checking on AII think this, this
• 35:03
  is so my take and again, this is how we are looking at it as PwC
• 35:08
  is that, you know, we are looking at agents as a force
• 35:13
  multiplier, right? I mean what we're looking at
• 35:15
  right now it's a force multiplier agent is allowing one
• 35:18
  person to operate at the level of a part.
• 35:22
  So a person with a group of agents becomes like a full value
• 35:26
  stream delivering the work. So the main thing is that we
• 35:30
  would need to be deliberate about that.
• 35:33
  The change management process that we do in the organization,
• 35:36
  we are to be deliberate about it.
• 35:38
  We need to redefine roles as we are rolling out agents, right?
• 35:42
  So many times one person might be doing the work of an entire
• 35:47
  team and, and they are kind of, and, and they kind of act as
• 35:50
  value stream owners. They act as orchestrators.
• 35:53
  They and they kind of own. So the, the, the, the first
• 35:57
  piece I'd like to talk about is ownership rather than as
• 36:01
  auditing the work. So they, when they own the work,
• 36:02
  they of course need to audit it too, but they're finally taking
• 36:05
  accountability for it, right. So the second thing that we,
• 36:08
  I've, I've seen also as we are rolling out agents is we need to
• 36:12
  deliberately train people because everybody might not be
• 36:16
  as well worse with leveraging agentic tools, what they would
• 36:20
  need to look at because there's a new paradigm that is coming,
• 36:23
  right? It is not that it is, it is
• 36:26
  autonomous, but there needs to be things that humans would need
• 36:29
  to take care of. And this needs to be very
• 36:32
  deliberate, right? And the third piece is we look
• 36:35
  at measuring on the outcome, which ultimately the human
• 36:39
  resource is responsible for, right?
• 36:41
  So, so that's those are the three kind of things that we
• 36:44
  look at. So take up their role as a force
• 36:47
  multiplier, look at them as a value stream owner.
• 36:49
  We make sure the right training is in place to make sure that
• 36:52
  they're empowered to work with agents.
• 36:54
  And thirdly, we measure them on outcomes that they can deliver.
• 36:57
  So that's how we typically look at it metric.
• 37:00
  Yeah. And we just had our coding agent
• 37:02
  conference last week. And one thing that became very
• 37:07
  clear was there's this trend that's happening that if you
• 37:14
  create the code, even if you're creating that code with AI, you
• 37:18
  own that code. And I don't see why that
• 37:21
  wouldn't apply to any other part of the business.
• 37:24
  But it's really clear with the coding agents that now if you're
• 37:29
  going to push code to production, you have to be the
• 37:35
  owner of it like you would have been if you created it by hand.
• 37:40
  You can't just magically say, ah, well, you know, this code
• 37:46
  was AI generated. So now I don't have to
• 37:49
  understand how it works and I can push off the PR review on
• 37:53
  the somebody else because that's really what folks were saying
• 37:57
  was like, hey, so how do we do reviews in the age of code is
• 38:04
  cheap, but reviews are not That's human time that needs to
• 38:10
  happen. Even if you are using AI to help
• 38:12
  you with the reviews, you still are needing to go through and
• 38:17
  and figure out what's going on there.
• 38:18
  And so some of the tricks that people were talking about was
• 38:21
  that they'll just use certain parts of the code that the
• 38:28
  person who generated that code doesn't understand, and they'll
• 38:32
  ask for a review from an expert on that.
• 38:34
  So it's very modular and much more scoped down as opposed to
• 38:38
  submitting, you know, a couple thousand line PR.
• 38:42
  Yeah, No, I I think that that makes absolute sense, right.
• 38:45
  And again, from my own personal experience too, I mean, unless
• 38:49
  you own the logic behind it, the system design behind a code that
• 38:54
  is being built, it is not your own, right?
• 38:56
  I mean, it's something that is written you, but you just don't
• 38:59
  understand it. So the logic and many times I've
• 39:01
  also seen, you know, even though coding engines can really build
• 39:05
  powerful code, sometimes the system design behind it, I think
• 39:09
  the larger picture, somebody who knows the domain isn't a better
• 39:12
  position to guide the agent to do it right.
• 39:15
  And many times certain obvious corrections might be far
• 39:22
  reaching for the agent too. So I think we do have the, I
• 39:25
  mean, I think we should own the system architecture behind any
• 39:28
  agent. That's very critical.
• 39:29
  And while the agent might use different coding, and while the
• 39:34
  code blocks it can completely own it, I think the system
• 39:38
  architecture behind those code blocks, I think the blueprints
• 39:40
  behind it, I think that should be owned completely by human
• 39:43
  operators or whoever is owning that particular code set.
• 39:47
  How do you feel about the interesting piece that's coming
• 39:52
  out now? That is like if I just use AI as
• 39:59
  a quote, UN quote knowledge worker and I send you something
• 40:02
  that's obviously generated by AI.
• 40:05
  You on the other end of that probably are going to look at me
• 40:09
  a little differently, like, oh, you're lazy or you're not really
• 40:13
  trying that hard. You're just prompting something.
• 40:16
  And now I have to spend my time reviewing like 100 page essay
• 40:19
  that you prompted in 10 minutes, right?
• 40:22
  So it's it's almost like disrespectful.
• 40:26
  Yeah, So I think no, you're absolutely correct, right?
• 40:28
  I, I think there are two things to it, right?
• 40:31
  AI should make our life easier. I think our life and the life of
• 40:36
  somebody. I mean, let's just take the
• 40:37
  example that you said, right? If you write an e-mail without
• 40:41
  knowing what is the content that went into it, I think that's
• 40:43
  obviously bad. I think we should.
• 40:45
  So, so which means you, it should, the content should be
• 40:48
  yours. AI can draft it for you, but you
• 40:51
  should review it and before you send it by keeping in mind what
• 40:56
  the other person, the time that I mean, it's easy for you to
• 40:59
  send like paragraphs of information, but that's not one
• 41:01
  of makes any sense for the other person.
• 41:03
  So you'd be deliberate also about how it is structured, how
• 41:07
  it is presented, how it makes sense for the other person.
• 41:10
  So, so while AI is a force multiplier, I think, you know,
• 41:15
  you should own it like like we were discussing.
• 41:17
  I mean, like you own the court, the e-mail, you own it, you
• 41:20
  know, it produces a presentation, you own it, it
• 41:24
  produces spreadsheet you. And so the ownership part of it
• 41:27
  means that the system design behind it or the thought behind
• 41:30
  it or the blueprint behind it, the DNA behind it is yours,
• 41:34
  right? And it's not just purely coming
• 41:36
  from agent just randomly producing it.
• 41:39
  So that's that's my my take on this, Dmitry.
• 41:43
  Promote. It's been great talking to you
• 41:44
  man. I appreciate you doing this.
• 41:46
  Of course, Dmitry, pleasure, pleasure is all mine and I look
• 41:49
  forward to being in touch here. And a huge shout out to PwC on
• 41:54
  this conversation. It's been great getting to hear
• 41:57
  from the organization. Of course, we look forward to
• 42:00
  continuing. Thank you.