Latest / CXOTalk / Enterprise AI: Shadow AI and Agentic Risk - CIO advice
Transcript
- 0:00One of the fundamental tenets of testing is don't change two
- 0:03things at once. Well, we're changing thousands
- 0:05of things at once right now. AI agents are running wild
- 0:09inside the enterprise, and CIO's are on the hook.
- 0:14Anthony Scrofignano is a pH. D data scientist and Tim
- 0:19Crawford advises CIO's at the world's largest companies.
- 0:23The way that we used to think about application development,
- 0:26the way we used to think about testing, the way we used to
- 0:29think about data stewardship and ingestion, the way we used to
- 0:32think about governance, every single one of those things is
- 0:35changing. And then there's new domains
- 0:38that are coming into the sphere, like context engineering and
- 0:42understanding black boxes that are being called through open
- 0:46AIS that are being used by people in the enterprise that
- 0:49you don't even know are using them.
- 0:51So there's a whole new set of challenges for the CIO that we
- 0:55don't even have words for yet. If you go back and look at, for
- 0:58example, like social media, social media was really great
- 1:01because it gave everybody a voice.
- 1:04It lowered the playing field. Everybody had how to make a
- 1:07phone, had a platform in which they could share their
- 1:10perspective. The downside is it gave
- 1:12everybody a voice. And so we're starting to see the
- 1:16same thing play out with AI and especially when you look at not
- 1:20just vibe coding, where someone can vibe code an application
- 1:23that is not a, they're not a developer, they're a, they're a
- 1:26line of business user, but they're able to actually create
- 1:30code that gets deployed within the enterprise OR used within
- 1:33the enterprise all the way through to tooling that.
- 1:38When you look at things like Open Claw, Nemo Claw in Claude
- 1:42Co work and now Amazon Quick and even SA PS Jewel work that they
- 1:47just announced this week, All of those products are starting to
- 1:51enable every user access to AI with varying degrees of
- 1:56guardrails and protections around it.
- 1:59And so for the CIO, this opens up a vastly wider threat
- 2:05paradigm that you have to get your arms around, not just in
- 2:08terms of what's appropriate and what's not, but also in terms of
- 2:12the risk profile. And how do you start to, how do
- 2:15you start to ensure that you're still maintaining the right
- 2:17protections for your organization?
- 2:19One of the tenets of, let's say 10 years ago, that little thing
- 2:24called regression testing, You just did something new and
- 2:27you've proven that the new thing that you built works and you've
- 2:30proven that it integrates into the environment.
- 2:32You've done all the things you need to do.
- 2:34One of the things you have to do is also prove that things that
- 2:37used to work still work. That's a really difficult thing
- 2:41to do when all of these foundation models are changing,
- 2:44when the training data is changing, when the way in which
- 2:48the environment you're relying upon is changing at the same
- 2:52time that you're making changes to how you use it.
- 2:54There's another dynamic in addition to what Anthony
- 2:57mentioned, which is just thinking about identity and
- 3:00credentials. So authorization as well as
- 3:03identity. When you start thinking about
- 3:05agent identity and who's responsible for that, and which
- 3:09agents have access to which credentials.
- 3:11We're talking about this shift from essentially automation
- 3:19managing systems to autonomy. Let's talk about risk.
- 3:25How should CIOs reframe AI risk and also think about the value
- 3:33of AI. How do you balance that?
- 3:35Tim, you want to jump in? We're still figuring this out.
- 3:39You know, this isn't something that we can say, OK?
- 3:42It's just like, it's just like we did with cloud or it's just
- 3:47like we did with the Internet or just like we did with just
- 3:49distributed computing. There's no similar framework in
- 3:53our past that we can use to apply to AI.
- 3:58It's a whole, it's a whole new space.
- 4:00And so one of the challenges for CI OS is kind of working through
- 4:04this process both internally in terms of governance models and
- 4:08how we think about governance, but also with their partners,
- 4:12both their software partners and hardware and services partners
- 4:15as well as their consulting partners.
- 4:18So it's, it's a really complicated question that has to
- 4:23be answered. Now.
- 4:25This is one of the reasons why quite often you see some of
- 4:28these AI projects being much smaller in scope, with the
- 4:32exception of some very sizable outliers.
- 4:36But it's smaller in scope because that's the way that they
- 4:39protect themselves against exposing the blast radius and
- 4:43and potentially open themselves up to undue risk.
- 4:47There are some sort of learned tenets that we have as leaders
- 4:53in this space and they haven't gone away.
- 4:56So governance is still there. The need to have a consistent
- 4:59experience unless you intentionally are changing that
- 5:03experience. Provenance, knowing where you
- 5:06got your answer from, permissible use, knowing that
- 5:09you're allowed to use it for the purpose that you're using it.
- 5:13The difference between a risk, which is something that might
- 5:16happen, and an issue, which is something it is that is
- 5:19happening. These are still there.
- 5:21So there might be a tendency to say, Oh my gosh, and I'm not
- 5:25accusing Tim of saying this, Oh my gosh, LLMS are in the room
- 5:29now and everything is blown up and we need a whole new set of
- 5:31capabilities. I think it's, it's those things
- 5:35and some new things. We don't have to throw away the
- 5:38experience that caused us to all lose hair and get Gray hair.
- 5:42We need to build on that, a new set of capabilities around being
- 5:47able to understand agency, being able to understand
- 5:50authentication and validation in the context of a non human.
- 5:54On the other end of that of the ability to understand the impact
- 5:58of shadow AI and what people might be doing.
- 6:00And we shouldn't necessarily try to stop them because it's like
- 6:04trying to stop oxygen. So there's some new capabilities
- 6:07that we need to name, but those old ones are still there and
- 6:10they will serve as well if we pay attention to them.
- 6:13What about this issue of shadow AI and the fact that AI is
- 6:21proliferating AIAI? The use of AIAI coding, every
- 6:28aspect. The proliferation is very rapid
- 6:32and essentially uncontrollable. You're right, the way that AI is
- 6:38being used has expanded greatly. You know, everything from
- 6:42ChatGPT all the way through to things like Claude Co work.
- 6:47And so it's not just vibe coding code, which has its own set of
- 6:53challenges, but it's also how users are using this technology
- 6:59and the ease of use to use this technology such that now we can
- 7:04give agents. And this kind of gets to
- 7:08Anthony's point about agency. We can give agents our
- 7:11credentials and have these agents do work on our behalf.
- 7:16The challenge with that is that can happen outside of the
- 7:19enterprise walls, meaning the enterprise may not have
- 7:22visibility to it. And so that's part of the
- 7:25concern is that what do these agents have access to?
- 7:29How are they working? And there's a whole downstream
- 7:33piece of this, which is agents have access to more than what
- 7:36the user thinks they have access to because of the way that we
- 7:40structure RBAC systems and protections within the
- 7:43enterprise. As I'm listening to you talk,
- 7:46I'm thinking about earlier in my career when there was a big push
- 7:50to outsource, outsource development, outsource all,
- 7:54outsource everything. If it's not nailed down,
- 7:56outsource it. We had some of these same issues
- 7:59of how do you know that they're doing what we asked them to do?
- 8:02How do you know that we're not giving them access to more than
- 8:06they need or, or or more than we realized?
- 8:10Some of these are not new problems.
- 8:12Obviously, they're happening at a much larger scale now.
- 8:15There are some best practices that are emerging for shadow AI.
- 8:20One of I don't necessarily love them, but I get them and I'll
- 8:24probably come to love them. You know, create a sandbox, put
- 8:27the tools there, put trusted tools there, put trusted data
- 8:30there. Teach people how to use the AI
- 8:33that, that they're trying to vibe code their way through life
- 8:36with so that they get better at, at prompt engineering, they get
- 8:39better at testing. Teach them some of the, the,
- 8:43the, the five biggest things that you can do to make a
- 8:45mistake. So instead of trying to stop
- 8:48this, because I don't think it's stoppable nor do I think we
- 8:51should, maybe we embrace it and find better ways to control is
- 8:57the wrong word. But to walk alongside it and and
- 9:02understand the change that's coming along rather than
- 9:05constantly reacting to it, It's a whack A mole game right now
- 9:08and and that is not a sustainable strategy.
- 9:11Now let's just think about Joe or Mary in the finance
- 9:16department or Joe or Mary in marketing who are coding their
- 9:21own tools and the lawn is growing and the mushrooms are
- 9:27coming up in the lawn and they're using corporate data,
- 9:31which is fantastic. We have as a company, we have so
- 9:35much data, we should share it with everybody.
- 9:38Let me give you an example. A person whom I greatly respect
- 9:44who has been a guest on your show, Steve Daffron once said
- 9:47the most fundamental thing in data science is counting things.
- 9:51And I just love the simplicity of that statement, right?
- 9:54So if we start with counting things, if you just get let
- 9:57people have at it and give them the customer, the CRM list, for
- 10:01example, and they want to count how many customers do we have?
- 10:04And they just count the rows. They're going to grossly over
- 10:07count because there's more than one row per customer, right?
- 10:10So you start by teaching them about relationships.
- 10:12Now, at the end of the day, if there's going to be a reality
- 10:18here that they don't want to be taught, they don't have time,
- 10:21That's why they're vibe coding in the 1st place.
- 10:23They don't want to hear you. It's a lot of wah, wah, wah,
- 10:26wah, wah. And so you have to put some
- 10:28controls in place. And certainly one of the biggest
- 10:30controls I would have in places is how much data are they ulling
- 10:33in? How much data is going out?
- 10:35Where is that data going? The good news is we can count
- 10:38tokens. We can look at things to see who
- 10:41the sort of mega users are. We can put guide rails in place
- 10:45like no PII, you know. I'm not just saying take
- 10:48whatever data you want from the enterprise and do whatever you
- 10:50want with it. Now I understand.
- 10:52So you're going to issue a mandate and everybody's going to
- 10:57do what you want. Nope.
- 10:59I'm going to have controls in that environment, including AI
- 11:02agents, by the way, that watch what's going on and understand.
- 11:08And I'll put understands in quotes because that's
- 11:10anthropomorphizing. But contextualize the biggest
- 11:14risks that we've identified. O let's, let's just take 1/2,
- 11:17right? We want to make sure ersonally
- 11:19identifying information is not somehow leaking out of the
- 11:23enterprise. I can easily put a control in
- 11:25place to watch for that. I can put a control in place to
- 11:30watch for the data that has value, leaving the organization
- 11:33without money that has value coming back into the
- 11:35organization. There are controls that we can
- 11:38put in place, but we have to give people a a.
- 11:42We have to have a way to watch what they're doing and not just
- 11:45hope that they're following the guidelines and the edicts,
- 11:48because we know they won't. Let's take some questions
- 11:51because we have a bunch of questions that have come in.
- 11:53And the first one, I'm going to go over to Twitter X.
- 11:57And this is from Arsalan Khan, who says if we treat AI agents
- 12:03as, quote, almost human, why can't we use existing governance
- 12:09models, apply HR models, for example, to AI agents?
- 12:13Tim, thoughts on this one? They are not like humans.
- 12:17And I think this is one of the one of the challenges.
- 12:20I mean, there's been this conversation for some period of
- 12:23time, you know, and modern is the poster child of this, you
- 12:27know, should we be taking HR policies and applying it to
- 12:31agents because they're just digital agents versus human
- 12:34agents? The answer is no, because it's a
- 12:37different, it's a whole different ball game.
- 12:40So for example, you have everything from just scale and
- 12:44speed as well as complexity that come into play when you get into
- 12:49autonomy and you start talking about complex agents and complex
- 12:53workflows. Now you have this concept of
- 12:56super agents that can go through an orchestration layer and call
- 12:59sub agents and you have to have governance layers at each of
- 13:03these steps as well as understanding.
- 13:06And it's not always for nefarious purposes.
- 13:09That's the other piece that I think we often forget is that
- 13:12it's not just because someone said, you know what, I'm here to
- 13:15screw the corporation and I'm going to do something nefarious
- 13:20or, or just completely go off the reservation.
- 13:23I'm, I've been wronged and this is the way I'm going to write
- 13:26it, but it also could be because agents and AI is imperfect.
- 13:31I mean, let's not forget that we are still learning how AI works.
- 13:36So kind of to one of Anthony's points, I mean, we, we have to
- 13:40be able to have explain ability, especially for audit routines in
- 13:45terms of how AI works. And so we don't necessarily have
- 13:49that in these algorithms today. It's more than just putting
- 13:54policies in place and the guardrails, but we also have to
- 13:58think about how we're setting up these workflows, what the date,
- 14:02what data it has access to, but what it could and couldn't do as
- 14:06well. And that's that's a far broader
- 14:08reach than anyone individual. Let me add to that couple things
- 14:13in my practice that are very interesting in this space. 1 is
- 14:18to have adversarial agents that are trying to break things and
- 14:22they can do that at scale. And that can be very
- 14:24interesting. So the red team concept, Yep.
- 14:27Another thing is ethical checks. So we can look at expect.
- 14:31This is very helpful when you're generating text, when you're
- 14:34generating words, you can look at mean shift and sentiment.
- 14:39You can look at multi modality. Are the leaders leading the
- 14:43group or is the group leading the leaders?
- 14:45Those are very valuable things to look at.
- 14:48Provenance, making sure that there's a way to go back to a
- 14:51fact for any kind of new nouns that you start talking about
- 14:57veracity, looking at misinformation and
- 14:59disinformation. When people repeat things that
- 15:02seem true, they're not necessarily lying.
- 15:04If those things are lies, those lies seem believable, right?
- 15:08So getting at the source of those lies, there are lots of
- 15:12very interesting things that we can do in this space that don't
- 15:16really have names yet, but they have agents, and we can get
- 15:19those agents to do some very clever things if we teach them.
- 15:24This is from Chris Peterson. Totally different topic here
- 15:29now. And he and Chris says
- 15:31historically we could read the release notes and hope to know
- 15:35important changes in software and a new software version.
- 15:39Is there any useful analog in the LLMGPT world?
- 15:45And if not, how do we? And he says this is to Anthony's
- 15:49point, how do we regression test for safety?
- 15:53It's a really important point. The large LLM providers and it's
- 15:57not just the anthropics and open AIS, but when you look at what
- 16:02IBM is doing, when you will look at what Amazon's doing, what
- 16:05Google's doing in terms of building their own, their own
- 16:10models. So whether it's granted from IBM
- 16:12Nova, from AWS, Gemini, from Google, for example, they are
- 16:18putting safety measures in the mix.
- 16:21Is that enough? No, absolutely not because we've
- 16:25already seen examples of where these models have kind of gone
- 16:29off the rails and done some pretty bad things.
- 16:32And so we have to be real cautious about this.
- 16:36And the same goes with the agents too, because we have to
- 16:40understand how these agents work.
- 16:42And that's something that is a new skill that we are just
- 16:46learning about today. The old approach to regression
- 16:49testing. You build a bunch of scripts.
- 16:51You say you know all of these things should still work when
- 16:54we're done. Here's the data that you use
- 16:55with these scripts. When you're done doing whatever
- 16:57it is you were doing, run these scripts with this data and make
- 17:01sure it produces these answers. You can still do that.
- 17:05And to Tim's point, the, the major models at least do tell
- 17:10you when they're changing. They don't tell you what they're
- 17:12changing, but they tell you that you're they're changing.
- 17:16So that's a good time to start doing this.
- 17:19There were some other dimensions in an LLM in an agentic world.
- 17:23I'll say in a Gen. AI agentic world, there are some
- 17:26other dimensions that you need to think about.
- 17:29One of them is misuse. Are people able to, either
- 17:33through prompt injection or stupidity or, you know, kicking
- 17:38the tires, able to get your environment to do something it
- 17:41was never intended to do? And have you just made it easier
- 17:44for them to do that? Another one is misadventure.
- 17:47You can't say hallucination anymore because everybody just
- 17:51gets triggered. But it is absolutely possible
- 17:54that you may have enabled your system to speak for your
- 17:58organization in ways that you never intended.
- 18:01There are other dimensions like that that we can test for.
- 18:05We have to name them. We have to build modalities for
- 18:09testing. Note that I didn't say scripts,
- 18:11because this will not be linear. This is very much a non
- 18:15deterministic kind of thing we're trying to do.
- 18:18But if you say that the first thing we're trying to do is test
- 18:22against misuse, all right, that's a good start.
- 18:24At least now we know what mindset we're in when we're
- 18:27building out regression approaches to misuse.
- 18:31So it's absolutely not a linear process anymore.
- 18:34You can still use that linear process, but you need to build
- 18:37on to it a sort of a non deterministic approach to some
- 18:40of these other ways of messing up with Gen.
- 18:43AI. Tim, this is a a a CIO career
- 18:47question and this is from Preeti Narayan, who says for aspiring
- 18:53CIOs, what capabilities and mindset shifts will matter most
- 18:59in an AI agentic enterprise world?
- 19:03And how difficult will it be for today's CIOs to transition to
- 19:07transition from traditional IT operating models to AI native
- 19:14ways of working? I often talk about this
- 19:17differentiation between the traditional and the
- 19:20transformational CIO, right? The CIO is not a CIO is not a
- 19:23CIO. If you're an aspiring CIO, you
- 19:27need to focus on the behaviors of the transformational or
- 19:31business oriented CIO. In an agentic world, you have to
- 19:37understand your business. And I don't mean just the the
- 19:40superficial layperson's explanation of we're a clothing
- 19:44retailer and we sell shirts and pants and socks and that sort of
- 19:48thing. But you need to understand the
- 19:50underpinnings of how the customer engages with your
- 19:54business. You have to understand the
- 19:56entire workflow of how your business operates end to end.
- 20:01You have to understand some of these dynamics and nuances
- 20:04within how your company operates, not just the cultural
- 20:07and institutional knowledge, but also the industry and where it's
- 20:11going. And so.
- 20:13Part of this answer is that you need to be a business leader
- 20:18first that happens to have responsibility for technology.
- 20:23When we move into this agentic era, it has to understand the
- 20:27business. I mean, if you don't understand
- 20:30your business, it's going to be really hard to truly lean in on
- 20:34these agentic workflows more than just the simple, hey, we've
- 20:38taken a process and we've automated that process.
- 20:41So for the aspiring CIO, first and foremost, understand your
- 20:45business deeply. Second is reimagine your
- 20:48business. Think about how you would do it
- 20:51differently and where technology can play a role.
- 20:53And I think this is where some of the agentic work for flows,
- 20:57especially the more complex ones, become really interesting
- 21:01and surprise, surprise, more valuable to your organization.
- 21:07Anthony, here's a question from Hui Wang, who says, how
- 21:13important do we see the need of approaching a demand to have a
- 21:18very clear and comprehensive blueprint in our solution
- 21:23designs in order to prepare teams to be operational and red
- 21:28team ready? Talk about hand grades, hand
- 21:31grenades. That's a That's a fascinating
- 21:34question and you could probably write a book on the answer, but
- 21:38the, the essence of it is how do you get people who are in this
- 21:42mindset to be ready for this red team concept while everything's
- 21:47changing at the same time? And, and that is a very, very
- 21:52big question. So the first thing I would say
- 21:54is going back to the comments that Tim just made about
- 21:57transformational leadership, one of the essences of, of
- 22:01transformational leadership is not to just be changing things
- 22:04for the sake of changing, to be going from someplace to
- 22:08someplace with overt intentionality and understanding
- 22:12why you're doing it. So if you're bringing red teams
- 22:14into the room, you don't want to just bring red teams in the room
- 22:18and say, all right, see if you can break it.
- 22:20That's a ridiculous mandate. What you want to say is we're
- 22:24worried about 5 things. We're worried about data
- 22:26exfiltration, we're worried about malware injection.
- 22:30We're worried about changes in the whatever those five things
- 22:33are, right? There should be 5.
- 22:35And then the red team is focused on those five things.
- 22:37The non red team, the good guys, know what they're sort of
- 22:41defending against and it's a bit of a fair fight because they're
- 22:45not just coming in and trying to break stuff while you're trying
- 22:47to build stuff and fix stuff. There's got to be some
- 22:50discipline to this. It can't just be a a food fight.
- 22:54What happens when you let it just be a food fight?
- 22:56It's fun, right? People enjoy it and especially
- 22:59the red team enjoys it because they don't have any rules.
- 23:01But it's just so disruptive. And I think we've all worked in
- 23:05large corporate environments where you're just trying to get
- 23:07your job done and emails are coming in that are fake and
- 23:10they're trying to trick you. And there's pen tests going on
- 23:12and you get to a point where you don't trust any e-mail and then
- 23:16you don't respond to a customer because it looked like a pen
- 23:18test. We got to be a lot more
- 23:21thoughtful about this and I think it starts with ways
- 23:23question about how do we get them ready to do this?
- 23:27We've got to teach them what they're doing and not just
- 23:28charge them with breaking stuff. I want to build on this a little
- 23:31bit because there's an additive piece, especially when you start
- 23:35talking about AI, that I think some people are kind of missing
- 23:40along this pathway, which is the automation that's coming soon is
- 23:45not going to be optional. It will be a requirement whether
- 23:49you want it to be or not. And what I mean by that is that
- 23:52the the sheer speed in which we need to operate is increasing so
- 23:57quickly that we just can't put a human in the loop on some
- 24:02things. And so when you think about red
- 24:04teaming, and if you're not familiar with the concepts of
- 24:07blue teaming and green teaming, definitely get your arms around
- 24:11that. But for those that aren't
- 24:13familiar, you know, a red team is the attackers, right?
- 24:16We're simulating those attacks and trying to understand kind of
- 24:20what's happening. The blue team are the folks that
- 24:23are looking to put up the defenses around those attacks.
- 24:28And then the green team says, OK, we found these attacks,
- 24:31these vulnerabilities, how do we fix them?
- 24:33And, and how do we start to ascertain what we have two
- 24:38different ways we could go about this.
- 24:40What's the right way to do it? That's no longer a group of
- 24:45humans doing it, nor is it a group of tools doing it.
- 24:49It has to be a group of agents that are automatically working
- 24:53on your behalf within your constructs and are doing this
- 24:58almost continuously. That's the speed in which we
- 25:02have to get to. And so here's the thing that I
- 25:04would pose to your audience, Michael, is that if you had to
- 25:08automate any said process, whether it's in cybersecurity,
- 25:13whether it's in a business process, if you had to automate
- 25:18that fully, no caveats, no ifs, ands or buts, you had to
- 25:22automate that. How would that change your
- 25:24thinking around all of the other aspects, everything from how you
- 25:29develop the application, how you use the application, how you
- 25:32update the application, like, for example, patching.
- 25:36We're having this conversation with a group of CI OS just this
- 25:38morning. What happens if you have to take
- 25:41patches automatically? You no longer can do testing
- 25:44before you deploy it, because that's that might be a reality
- 25:48for us in the very near term. This is something that that I
- 25:52think people have to start getting their head around is if
- 25:56these automation routines become the norm, meaning we have no
- 26:01choice but to have to do it, how does that change our thinking
- 26:04around other aspects and other decisions and choices we've made
- 26:09within the enterprise? And that's what I would
- 26:11challenge the audience to think about.
- 26:13What could possibly go wrong with automation that you over
- 26:18which you have no choice? We've all been there, right?
- 26:22We've all run organizations. I get it.
- 26:25But we may not have a choice. We may have to try and figure
- 26:30out how to. So for example, thought
- 26:32experiment here, just pick one piece off, pick off the
- 26:36patching. For example, if you had to
- 26:40automate all of your patches because everything is a zero
- 26:44day, how does that change your thinking within the organization
- 26:47and how you operate? I would suggest that there's
- 26:50some of that going on without your knowledge anyway.
- 26:53If you're using anything in the cloud or any API or any external
- 27:00black box, stuff is getting patched.
- 27:03Stuff is changing. It's part of your stack.
- 27:06You're relying upon it, yet you may not even know it changed.
- 27:10And there's that other dark side of, I don't want to name
- 27:15organizations, but there's some very big ones in the news
- 27:18recently where they didn't do a patch for whatever reason and it
- 27:22created some huge vulnerability. And now you're explaining why
- 27:25you didn't do the patch. So you know, the the answer
- 27:29isn't the patch or not the patch.
- 27:31Your your point about the the red team and the and the at
- 27:35least the red team and the blue team being automated clouds of
- 27:39agents. I think it's both right.
- 27:42So there's also orchestration from, you know, Lex Luthor's in
- 27:46those cohorts that are people that that know how to think new
- 27:50thoughts. Otherwise we only learn how to
- 27:52protect ourselves against pre-existing vulnerabilities
- 27:55that we knew about and some of these new things that come up
- 27:59right. I'm amazed at the incredible
- 28:02creativity of bored 14 year olds.
- 28:05Some of the stuff they come up with is just genius in its
- 28:09simplicity. One of the most recent major
- 28:12oops is that hit a big part of the world happened by releasing
- 28:16an empty file and and it caused all kinds of problems because
- 28:20nobody ever thought of that. So yeah, we need people and
- 28:24machines doing this. Let's go to another question and
- 28:31this is from Swami Vaidyanathan. And he says earlier we spoke one
- 28:38of you mentioned social media and he says it was a great
- 28:42analogy with social media where the floodgates opened sort of.
- 28:46But if we don't have a close parallel from history that can
- 28:52be used, that can be used as a model.
- 28:54Do you have a specific path or thought process you can share
- 28:58for CI, OS and teams to approach this?
- 29:02It was my comment that social media gave everyone a voice and
- 29:08that's the upside. And then the downside is social
- 29:11media gave everyone a voice. We have the, we have something
- 29:15similar with AII think the construct here is unlike social
- 29:19media, number one, there's more at risk and the speed in which
- 29:26it's at risk because the social media took time to build an
- 29:29audience and, and have people follow you.
- 29:32And then there's a human in the loop.
- 29:33Now we're talking about agents that can run off and, and
- 29:37potentially do tasks and do a lot of damage before anyone is
- 29:41even to the wiser. It's actually one of the
- 29:44challenges of agents right now is some of these more
- 29:46sophisticated agents are moving at such a speed they're breaking
- 29:50AP is they're breaking websites because they just weren't they
- 29:54were not never designed for that kind of scale and that kind of
- 29:57speed. There are constructs you can use
- 30:02fundamentally to understand kind of how agents work, how agentic
- 30:07workflows work and then step into it.
- 30:10You know, I often say that one of the things you have to think
- 30:14about when it when you're when you're thinking about your
- 30:17organization and how you accelerate your organization in
- 30:21in a gentic era is how do you start with one thing and then
- 30:26move to a second? But with each iteration of this
- 30:28flywheel, you're accelerating the organization.
- 30:32The same thing is true with the gentic workflows is that you
- 30:35need to start small, get your sea legs, but then build upon
- 30:40that and accelerate your knowledge and accelerate your
- 30:42understanding of it. And there is a train.
- 30:46The reality is there is a training aspect of this.
- 30:49We need to educate people on the importance of both the
- 30:52opportunity and the risk so that they can make good business
- 30:56decisions for their organization as well as their role.
- 31:00And that's something that we haven't necessarily stepped into
- 31:03as much today. It's not as much of putting the
- 31:05policies in place and bringing the hammer down, but rather, you
- 31:10know, we've got some intelligent people within the organization.
- 31:13Let's lean into that. Let's help them make good
- 31:16business decisions on our behalf.
- 31:18When we think about making mistakes faster with agents, we
- 31:23also have to keep in mind that the reason we call it Gen.
- 31:26AI is it's generating its own content, and a lot of the
- 31:29content that AI is consuming was produced by other AI.
- 31:33So there's a multiplicative effect of this problem.
- 31:36It's not just making mistakes faster, it's making more chaotic
- 31:40mistakes faster. There's a geometric effect here
- 31:45that we have to keep in mind. We can get out of control at a
- 31:49rate that we cannot recover from at the same rate.
- 31:52And so it's really important to have these conversations about
- 31:56guardrails and red teams and all of that to, to to sort of thwart
- 31:59that kind of multiplicative effect that those are the things
- 32:02that kill us. Let's shift gears and talk about
- 32:07one of the great buzzwords of our time, which is human in the
- 32:12loop. So when is human in the loop
- 32:16meaningful, and when is it theater?
- 32:20If you look at the regulatory response to this right now, if
- 32:24you look at the the, the application and what the
- 32:27application is doing, the more critical the use, the more
- 32:33mandatory it is that you have a human in the loop.
- 32:35That is the general theme of all of the governance that's out
- 32:39there. There's a little bit of blinders
- 32:42in this because I think Tim pointed out earlier, there are
- 32:46things that happened way too fast to have humans in the loop,
- 32:49and that's happening more and more now.
- 32:51And so requiring a human in the loop might make you feel good,
- 32:54but humans can't make decisions fast enough when the environment
- 32:58is changing at certain rates. And so the response to that is
- 33:03sort of audited humans in the loop, like the humans, will look
- 33:05at some percentage of the decisions being made.
- 33:08That's not a sustainable answer either.
- 33:12The reality is that even humans in the loop, all humans, don't
- 33:17have the same reaction when they look at things.
- 33:19So the gold standard, this is called a heuristic frame.
- 33:22The gold standard is to have similarly instructed, similarly
- 33:26incentive people looking at things.
- 33:29Well, you can't get that in the real world because people don't
- 33:32understand things at the same level.
- 33:34They don't have the same background, they don't have the
- 33:36same stake in the game when they're making a decision.
- 33:38Some are trying to get done, some are trying to get done
- 33:40right. So the important thing in this
- 33:43is not to say, well, Gee, Michael, it's really complicated
- 33:45and I wish I could explain it to you, but there is no answer,
- 33:48which is the answer, by the way. The answer is that wherever you
- 33:52are in this human in the loop, be it because of regulatory
- 33:56constraint or because of ethical AI or because of good design
- 34:00principles, be where you are on purpose and constantly re
- 34:04evaluate that in terms of the rate of change in the
- 34:06environment and the elasticity that you have for risk.
- 34:10Tim, given that, what does AI governance actually look like in
- 34:16practice? It's complicated.
- 34:18And if you think it's simple, full stop, you probably don't
- 34:21understand all of the aspects that are needed in complex
- 34:25agentic workflows because you have you have a level of of
- 34:30governance, for example, and I'll just use a very simple
- 34:33example where Michael has access to certain things, Anthony has
- 34:37access to certain things from an agent, Tim has access to certain
- 34:41things. And that's what that's what it
- 34:42presents. And we often talk about our back
- 34:44is the method to say based on what Tim has access to, that's
- 34:48what we're going to present. Well, then the agent governance
- 34:51layer sits above that in terms of what the agent itself has
- 34:54access to be able to do. Above that is a knowledge
- 34:58governance layer, which is how do we start to amass this
- 35:01knowledge and where does it come in and what agents have access
- 35:04to what from a knowledge perspective.
- 35:06So that's just three layers by itself.
- 35:09Oh, and by the way, where does this data that we're going to
- 35:12tap into reside? Because when we start talking
- 35:15about business processes, we're talking about processes that
- 35:19span multiple departments and therefore multiple systems.
- 35:22So we might have some data in service now, we might have some
- 35:25data in Salesforce, we might have some data in SAP.
- 35:30And so each of those systems have their own governance
- 35:33layers. So how do you start to amass and
- 35:37rationalize these different governance models to be able to
- 35:42present the appropriate data to both the agent, but then also to
- 35:47the end user? And this is where things get
- 35:49really complicated because it's not just a really complicated
- 35:53math problem, but there's a, there are a lot more aspects to
- 35:57it because it's not just simply bits on the page of data
- 36:00elements. We have to even look beyond
- 36:03metadata and semantic data and start looking at context around
- 36:07data, which is something of a new concept for us.
- 36:10So how do you start to put all of that together if that already
- 36:15makes your head explode? You're not alone.
- 36:18And so This is why we're starting to work through how do
- 36:21we think about it. But in the meantime, we have to
- 36:24kind of dumb down the the dumb down the architecture a bit
- 36:28because we can't solve that problem today.
- 36:31If you live within a given ecosystem, you have additional
- 36:36opportunities that you don't have when you start working
- 36:39across different systems. But that's also why you have to,
- 36:42you have to kind of wrap your head around, OK, so opportunity
- 36:46risk here, I'd like to be able to do that, but I have to
- 36:49understand how do I start to play these other pieces into the
- 36:53into the equation? One of the first things that I
- 36:55do in my practice is I, I, everybody wants to rush to the
- 37:00tools. You know what, what AI are we
- 37:02going to use? How are we going to generate the
- 37:04code? And I usually try to slow it
- 37:06down and start with what's the problem we're trying to solve?
- 37:09How would we know we solved it? What are the the things that can
- 37:13happen that are sort of the unknown unintended uses of what
- 37:17we're about to do? So I have some fundamental
- 37:19questions that I ask and what if we can't answer those questions?
- 37:23There's a bit of malpractice in taking that next step.
- 37:26So first principles really important.
- 37:29There are certain approaches in, I'll just use LLMS as an
- 37:32example. You can use a rag approach.
- 37:35If you, if you're trying to do something and you know that
- 37:38these documents represent the truth that you understand right
- 37:40now, start here instead of starting with the Internet.
- 37:44You can look at veracity adjudication.
- 37:46You know, how would we know that something is true?
- 37:48How would we know that it doesn't contain, you know, part
- 37:51of the truth to mislead us? So there are definitely mindsets
- 37:55and there are definitely pistemological models, you know,
- 37:59ways of knowing that you can ground yourself in that helped
- 38:03defend against the chaos that Tim is talking about.
- 38:05It's not hopeless, but you do need adults in the room when
- 38:08you're doing this. You can't just throw tools at
- 38:10it. So are there new roles and new
- 38:14organizational structures that CIOs need to be thinking about
- 38:18now or starting to build right now?
- 38:20OK, So what? Are yeah, I mean, the one we
- 38:21keep talking about is context engineers, right?
- 38:24People that understand the way in which the AI is given context
- 38:28of the problem that it's trying to solve.
- 38:29That's definitely an emerging role.
- 38:32You know, we we're talking a lot about red team.
- 38:34That's a a, a role that pre existed, but it's way more
- 38:37important now. Tim, I'm sure you have a few
- 38:39more. Even beyond red teaming, blue
- 38:41teaming, green teaming from an automation standpoint.
- 38:45And then look at the finance piece of this, you know, we
- 38:47haven't even touched on cost. There's a whole talk track
- 38:50around cost and understanding what the and more importantly,
- 38:54understanding the true value, right?
- 38:57Let's be, let's be focused on this.
- 38:59Value is an equation that is value equals opportunity minus
- 39:03cost, right? So it's not just about the cost,
- 39:06it's about what you're able to do for that cost or what you're
- 39:09trying to accomplish. And so there are new frameworks
- 39:13that are starting to come out. Fin OPS is is working on
- 39:18versions of Focus and their framework to be able to
- 39:21accommodate understanding the true AI cost.
- 39:24There are some startups that are just about to announce some of
- 39:28the things that they're working on around this.
- 39:31The TBM council from their perspective kind of that
- 39:34umbrella of all IT understanding value and being able to work
- 39:39down. But this isn't as simple as just
- 39:42tagging resources and saying, OK, well, this resource is this
- 39:45application, this resource is this application.
- 39:48We're talking about agents that let's say the three of us all
- 39:50have access to this agent. How do we build those resources
- 39:53to the appropriate person, right?
- 39:55If this is an autonomous agent and if it's a complex agent
- 39:59where it's an agent that is learning.
- 40:01And it's also discovering other agents, which, oh, by the way,
- 40:04might not be within your organization, might be a third
- 40:07party, and it's able to to build ephemeral agents.
- 40:12There's a whole nother layer of complexity that comes into the
- 40:15mix, right? We've largely been talking about
- 40:18a very simplistic way around agentic AI, but these other
- 40:22pieces are actually in market today.
- 40:24They're just in different forms and it's only a matter of time
- 40:28that we're going to see the mall weave together.
- 40:30Anthony, we are just about out of time.
- 40:33Very quickly, you're going to have the last word.
- 40:35What happens to talent and jobs inside it as a result of all of
- 40:39this? Very quickly please.
- 40:41I don't think AI is going to take all the jobs away.
- 40:43I think that we're going to have lots of new exciting jobs and I
- 40:46think that it's very important that we don't try to just vibe
- 40:48code our way through, you know, evolving our career that we
- 40:51learn these first principles. We learn how things are
- 40:54changing. We learn how to take advantage
- 40:55of it. We start working on new problems
- 40:58that we couldn't solve before because we have this abundance
- 41:00of capability. Don't just look for the easy
- 41:03button. Look for the the next thing that
- 41:05you can do to add you know much more value to the organization
- 41:08in a responsible way. Well, that's pretty good advice
- 41:11and unfortunately, we're out of time.
- 41:15A huge thank you to Tim Crawford and Anthony Scrifignano.
- 41:20Thank you both for being here. I'm very grateful to you.
- 41:23Thank you for having us. And a huge thank you to
- 41:26everybody who watched and especially you folks who asked
- 41:30such excellent questions. Before you go, subscribe to the
- 41:35CXO Talk newsletter. Go to cxotalk.com.
- 41:39Two weeks from today, we're talking with the chief
- 41:42technology officer of Mozilla. So get our newsletter so we can
- 41:47notify you of upcoming shows. And we'll see you again very
- 41:51soon. Everybody, thank you so much.
- 41:52Take care.