Latest / CXOTalk / Governing AI Agents at Scale: Identity, Scope, and Observability (with Glean and Cvent) | CXOTalk #914
Transcript
- 0:00AI agents are making decisions in the enterprise, but who's
- 0:03governing Managing them? Glean's Work AI Institute
- 0:08developed the AWARE framework to tackle agent security,
- 0:12governance and compliance. Pradeep Manakara, CIO of Cevant
- 0:17and Ben Mayrides they are Siso explained.
- 0:21Cevant is a leading SAS hospitality and event software
- 0:24company. About 30 + 1000 customers, 5500
- 0:28employees, over 6000 agents and since about 2023 we're part of
- 0:33the Blackstone portfolio. Ben, when we look at agent
- 0:38governance, agent security, this is a proliferating bundle of
- 0:44issues. What are some of the core
- 0:47challenges from your perspective as a Sousa?
- 0:51Number one is education. The space is moving so rapidly,
- 0:55it's evolving so rapidly and understanding not only what
- 0:58these tools can do for us, but also understanding what they can
- 1:02do to us, I think is incredibly important.
- 1:05And so education awareness from the board level on down, I think
- 1:09is, is is a big challenge. And again, it's just the rapid
- 1:13pace and the eagerness with which organizations might
- 1:16included are adopting these technologies.
- 1:19So that's I think challenge number one.
- 1:22But maybe to get a little bit more technical, I think that
- 1:27when it comes to some of the technical controls, the security
- 1:31architectures that we're all used to are really focused on
- 1:35deterministic systems, right? And as we know, AI is all about
- 1:38non determinism. So the architectures are broken.
- 1:42So when you think about agent identity, for example, agents
- 1:45are planning their reasoning. They're, they're, they're and
- 1:49and they're delegating actions. And our existing IAM controls
- 1:54aren't necessarily architected and built for, for that right
- 1:58now. Same thing for observability,
- 2:01for example, you know, a lot of agent reasoning is opaque and
- 2:06it's difficult to sort of reconstruct those those elements
- 2:09as well. And we need that.
- 2:11We need that for compliance reasons with things like the
- 2:13EUAI Act, but also in the case of a security incident, we need
- 2:17to be able to, you know, restructure and, and understand
- 2:21what actions an agent took and so on.
- 2:23But the existing security controls that we have for
- 2:26observability aren't yet attuned and they don't really integrate
- 2:30very well in an agentic architecture, right.
- 2:33So we need, that's where we need purpose built, you know,
- 2:38governance, we need education and we need these technical
- 2:41controls to mature. Pradeep Ben made a really
- 2:45interesting comment, he said. Agents are valuable for what
- 2:50they can do for us, but also to us.
- 2:57Maybe from a CIO perspective, dig into that for us and to us.
- 3:04Whatever we implement, we need to make sure there's adequate
- 3:06safeguards in place to manage risk.
- 3:09And I rely on Ben, our legal team, our privacy team to kind
- 3:15of help sort of shape that aspect of it, but at the same
- 3:19time move fast. Now your question around what it
- 3:21can do for us, I look at in terms of the value that it can
- 3:25bring us, right? Whether it's taking action,
- 3:27whether it's finding information, whether it's
- 3:29suggesting all those things have been highlighted.
- 3:32I, I, I think we've all seen where agents can do really well
- 3:36and we actually have demonstrated ROI proof of that,
- 3:40the things that you can sort of take to us.
- 3:44And if I necessarily consider a negative, it just means we need
- 3:48to make sure we've got some mitigating controls in place.
- 3:51And that can happen in a variety of ways.
- 3:55One of the challenges go back to your last question we have is
- 3:58there's a lot of vendors, partners, solutions that are out
- 4:01there. You multiply that times the
- 4:04number of employees we have. And now we have a lot of ideas,
- 4:09amazing, great ideas coming to us.
- 4:12And what we've got to figure out is how do we review these
- 4:15things? How do we look at the value of
- 4:16each of these things? And as we try to do this in a
- 4:19meaningful manner, one of the things that we also look for is
- 4:23to make sure that there's financial safeguards as well in
- 4:27place. And then the two things is, you
- 4:30know, I rely on Ben and other teams to make sure we're
- 4:33protecting ourselves. That's how I look at the
- 4:35two-part. You spoke about the various ways
- 4:40of looking at governance, looking at protections.
- 4:44The Work AI Institute's AWARE framework covers 5 governance
- 4:50challenges for agents, intent, context, guardrails, risk
- 4:56scoring and blocking, and ecosystem observability.
- 5:01How do these principles map onto your decision making as you're
- 5:09deciding what to work on, what not to work on, what to hold
- 5:12back, and so forth? It's very relevant, of course to
- 5:17any sort of agentic architecture that we have.
- 5:19So, you know, as we actually have been adopting the platform,
- 5:23it's been taking a look at all right, what is what is an agent
- 5:27actually, what is the identity that it might be assuming or
- 5:31that might be delegated to it. And it's the, it's The Who and
- 5:35and what is taking that that sort of action.
- 5:39We need to actually identify that in advance, like what in
- 5:43addition to what is the scope that that agent is actually
- 5:46going to run in? What are the data connections or
- 5:49the data sources that it's going to run in?
- 5:52And the context is incredibly important because like say, an
- 5:55agent that is maybe going to Salesforce and reading a
- 6:00customer record for the sake of summarizing that record for a
- 6:04salesperson versus an agent that is accessing that record to
- 6:09maybe update it, modify it, or maybe write, you know, summarize
- 6:15that and maybe write some of the sensitive data to another
- 6:18resource. That's a completely different
- 6:20scope, completely different risk profile.
- 6:22So stepping through the aware framework, we're able to
- 6:25actually break that down and then make sure that we have the
- 6:28right kinds of controls in place, right?
- 6:30So it's essentially A comprehensive set of controls
- 6:36and sets of issues you need to look at.
- 6:39In my view, it's a technical structure.
- 6:41It's a technical control structure because you know, the
- 6:45frameworks that are out there are predominantly organizational
- 6:50governance framework. So that when you think about the
- 6:51EUAI Act or the NIST risk Management framework, again,
- 6:56they're great and they have their purpose, but they don't go
- 6:59deep enough and they don't actually tie into sort of the
- 7:03technical controls and the technical considerations that
- 7:06you need to build into an agentic security architecture.
- 7:09And I agree with Ben. So which is why sort of what we
- 7:12look for is try to figure out what's a good pave Rd. of
- 7:16technologies that we can provide to our business.
- 7:20I'm going to separate, I'm going to call it our CTO organization,
- 7:23our product development organization because there we
- 7:26have a, I'm going to use the word a mature framework in place
- 7:29with actual appointed people and their roles and they work within
- 7:33this framework that we put in place on the business side.
- 7:37We see a lot of desire to move fast, right?
- 7:40I see some heads nodding in the room and what we look for is
- 7:44there's a lot of demand that's coming to us to say get us
- 7:48something quickly. So one of the things that where
- 7:51we look to look for is technologies or platforms that
- 7:54have this sort of inherent within that framework or that
- 7:58platform or tool that allows us to then deploy something
- 8:02quickly. Because we are here with Glean.
- 8:04One of the things that we realized with Glean as we kind
- 8:07of did that pilot initially and then rolled it out was that
- 8:11actually US probably about nine months of time candidly
- 8:15speaking, which satisfied a lot of our business users needs.
- 8:20Meaning they didn't want to rely on IT to do things.
- 8:23They wanted to kind of go and experiment and do things on
- 8:26their own. And we as IT and security didn't
- 8:30have a facility to easily enable that within a corporate
- 8:35ecosystem, meaning can I get access to my e-mail, my Slack
- 8:38messages, my box files, Salesforce with that fine grain,
- 8:42security controls and things like that.
- 8:44So by us having the number of platforms that allow our users
- 8:48to achieve that and then training which you brought up
- 8:51earlier, which then also security has sort of blessed the
- 8:56sort of the walls around it saying, hey, if you do it within
- 8:59this space, you don't have to come to us, we know the controls
- 9:03exist, off you go. Actually gave our users a ton of
- 9:09capability to the point where we had AAI council with numerous
- 9:14requests that we would have to somewhat custom do.
- 9:17Those all sort of went off on the wayside.
- 9:19Now there's sort of the next evolution of that that's coming,
- 9:23which is I won't worth long, but.
- 9:25Ben, you mentioned the term training.
- 9:28Pradeep, you said surprisingly to me that culture is part of
- 9:33your role as CIO. How do you make the framework
- 9:39come alive? Which seems to me to be a, a
- 9:44cultural aspect. And it's more than just
- 9:47training, right? It has to be more deeply.
- 9:51All right, So, so talk about that.
- 9:52How do you bring it alive? We are very bullish on adopting
- 9:56responsibly using AI. You know, with with all those
- 9:59other things that you talked about from a governance point of
- 10:01view. And as we're going through that,
- 10:03I'm like we're very bullish on this.
- 10:05However, we haven't baselined our employees.
- 10:08We've maybe we have 15%, maybe I'm being generous and I'll say
- 10:1220 that for whatever reason have to taken training.
- 10:15They've learnt it on their own, they're using it in their
- 10:18lifestyle. But my belief is, you know, you
- 10:20might have 1/3, a third, a third, third that are
- 10:22interested, a third that's like, I don't believe in this. 1/3 is
- 10:24like let's see where this thing goes.
- 10:26But one of the decisions we made was we're going to give
- 10:29mandatory training to everybody in the company, the foundations
- 10:33and illiteracy. And in there security has a
- 10:36piece, legal has a piece, everyone has a piece, piece
- 10:39around the education that kind of talks about some of the
- 10:41pitfalls with AI as well. So again, it's not a one and
- 10:45done, but it sets the sort of guidelines.
- 10:48Actually our CEO was in the first session.
- 10:50Just can I tell you how important this was?
- 10:52So, Ben, it sounds like this is fundamentally a different kind
- 10:55of training. Then we're rolling out a new
- 10:58tool, and here's how you know, Here's how to use the tool that
- 11:02sounds different. It is a big mindset shift.
- 11:04I think as many would probably already understand and and and
- 11:09have experienced themselves. But I think in addition to what
- 11:12Pradeep was just saying in terms of that education and that
- 11:15cultural shift, you also have to have a structured risk process,
- 11:19right? So as ideas, you know, this
- 11:22mindset shift happens and ideas, new ideas are coming up and new
- 11:26proposals are coming up from all over the business.
- 11:29We have to have a structured risk process to be able to take
- 11:32those ideas, evaluate them, yes, across a number of different
- 11:35dimensions, including security. And we have to have clear
- 11:40success criteria, right, for evaluating and building into
- 11:45those projects, building into those deployments and building
- 11:48into the operation of those those AI systems.
- 11:51Also the respect that sort of the this SISO organization earns
- 11:56like it has to be earned that trust.
- 11:58And I think we had a recent example, right, Claude Co work
- 12:02out, right still in sort of the it's not GA, but it's like, hey,
- 12:06it came out, people hear about it.
- 12:08They're like, we want to test it.
- 12:10And and Claude gives you messages when you open it up.
- 12:13Hey, Claude is here, you should try.
- 12:14This now 1000% and they also tell you don't use this in a
- 12:17regulated environment and so on. However.
- 12:20Yeah, read. The fine print.
- 12:21Oh, and by the way, we're not responsible.
- 12:24Yeah. Now at the same time, this is
- 12:26where the CSOCIO we're, I'm like we should have a framework to
- 12:31evaluate that to understand capability, but not at the
- 12:35detriment to the company. Ben and team kind of looked at
- 12:38this and said risk is too high, high and, and, and not just risk
- 12:42is too high as a generic statement.
- 12:44But here are the reasons why well.
- 12:46And it's risk is too high for now, right?
- 12:49Because, you know, there were there, I think a, a challenge
- 12:54that security folks run into is that sometimes maybe we're not
- 12:58looking at the, the business, let's say, is not looking at all
- 13:02of the risk factors. Maybe the, you know,
- 13:04traditionally or up to now, it's been very common for the
- 13:09business to look at model security, right?
- 13:12Where it's about what, what data is that model trained with?
- 13:16What is the way to access that model?
- 13:18Is that model provider actually going to retain my data, for
- 13:21example, and train and use it to train their, their model In an
- 13:26agentic architecture, it's completely different, right?
- 13:29It is about what that agent can do, what it can access, what
- 13:33decisions it's making, what, what actions is delegating, what
- 13:37you know, what users it is impersonating, it's all of those
- 13:40things. And so whoa, let's kind of it's,
- 13:44it's, it's not, no, it's, let's take a look at this.
- 13:48And deliberate about. It let's be deliberate about it.
- 13:51Let's be thoughtful. And again, you know, kind of
- 13:54coming back to the aware framework, what I like about it
- 13:58is that, you know, it sets that clear success criteria for, you
- 14:03know, me and, and him in particular, but also other
- 14:07business leaders that are really keen to adopt these tools.
- 14:10We can say, we'll look, if we can just answer these, let's
- 14:13say, 5 questions related to the aware framework.
- 14:17If we can agree on and we can get to the same answers, it
- 14:21eliminates a lot of the, this guy gives me a hard time all the
- 14:25time about, well, why do you not like it?
- 14:27And I'm just like, just, you know, it's a gut thing.
- 14:31That's not an acceptable answer, right?
- 14:33I mean, you know, I would figure as a therapist.
- 14:36What I tell them about the messaging, it's it's how you
- 14:40message. That same response can.
- 14:42Be interpreted those answers get to a certain level of
- 14:45specificity and we've agreed on the the the rules he he hates it
- 14:48when I I say rules but. Guidelines, Guidelines.
- 14:52Frameworks, requirements perhaps, right.
- 14:57But if we can agree on what the answers of those questions ought
- 15:01to be in advance, gets us to a level of specificity, removes
- 15:06the ambiguity of the, you know, and, and, and that from my
- 15:10perspective, gets me a little bit closer from and out of the,
- 15:14the, the realm of being a gate and maybe just being a part of a
- 15:17business conversation and a business decision.
- 15:20If I can paraphrase or simplify your roles, correct me if I'm
- 15:25wrong. Pradeep's job role as CIO is to
- 15:30move fast your job. Like I said is to make sure he
- 15:35doesn't trip while he's got the scissors.
- 15:37OK, so we can, OK, so we, so we can agree.
- 15:39How does a framework such as aware help manage this really
- 15:45fundamental tension? I I am sure you must have users
- 15:48coming to you and they say we got to do this.
- 15:51You know, we got it now we got to do it.
- 15:53How do you manage the tension? I think it's along the lines of
- 15:56what I was just trying to get at, which is that it gives us AI
- 16:01mentioned a little bit earlier, having a structured risk process
- 16:04and clear success criteria. It gives us a set of questions
- 16:08that if we answer those questions, then we can, you
- 16:11know, when it comes to being able to green light a project in
- 16:15particular, everybody knows what those criteria are in advance.
- 16:19They can build it into the design and to the deployment and
- 16:22the operation of those things, and we can get to yes, a lot
- 16:25faster. Pradeep, when your users come to
- 16:28you and they say this, we need this now, what do you tell them?
- 16:33Similar to the sort of aware framework, we've also actually
- 16:37partnered with our PMO office and actually created an AI sort
- 16:40of flow, which thanks to our partnership with our finance
- 16:44organization says, hey, if you want to go and demo something,
- 16:50by all means, go talk with the salesperson, go see a demo,
- 16:53understand what the capabilities are, love it actually and inform
- 16:56us, right. I don't have to come up with the
- 16:57idea to implement something. We are actually, we'll learn
- 17:00from each other. Now you want to take it a step
- 17:03further means that oh, there's value and, and I love value.
- 17:07So that means if we're going to help add to the top line or take
- 17:10away from the bottom line, however it is that's going to
- 17:12drive value I'm in. So we can do whatever we want,
- 17:15in other words. With our partnership 1000%.
- 17:18So, So what where that comes into is great.
- 17:22Now partnership with the CF organization, you've got to have
- 17:25some sort of ROI if you're talking about some new
- 17:26investment and if you believe there's value, we're going to
- 17:30listen. And I've actually devoted
- 17:32resources specific to some of these AI initiatives so that we
- 17:37don't have those backlog with the rest of what's happening in
- 17:41the business, right, in order to move fast.
- 17:43So that doesn't have to do with security.
- 17:44That's just reorganizing ourselves so that we can be much
- 17:48more nimble to get to these different gates, which is just
- 17:52to understand, hey, is there an ROI?
- 17:54Is there value? Is this thing real?
- 17:56Because there's a ton of vendors and solutions that are out
- 17:59there. So that funnel within the
- 18:02business teams will naturally filter out before it even has to
- 18:05get to Ben or some of our teams. Stage 2 of this is, I love to
- 18:10use the word test and learn. It doesn't mean full blown
- 18:13production. It doesn't mean production data,
- 18:16It doesn't mean integrating it with the rest of our ecosystem,
- 18:19which has higher risk. And, and, and you know, when you
- 18:22look at the checking the boxes on that framework, So we're
- 18:25like, hey, if it's a sandbox, let's create a, a workspace,
- 18:29maybe that's not connected to something to test that out.
- 18:32Let's make sure we have proper criteria.
- 18:34So not to kind of drag on, but what we've done is created
- 18:37different buckets based on the type of proof of concept or
- 18:41technology that you want. And once you get past that, that
- 18:45gives security time, legal time, privacy time to really vet these
- 18:50things based on those frameworks around risk that Ben and team
- 18:53have. Ben, does the shared language
- 18:57that emerges from a framework like AWARE make it easier to
- 19:04work with your counterpart? For sure, right?
- 19:08So again, it's set. And to use that tension it.
- 19:11Well, I don't, I don't know, OK. Let's not go that.
- 19:14Far right when when we're both under pressure or, you know, to,
- 19:19to deliver and to, to help the company turn, transform and
- 19:22adopt and and sort of reimagine itself leveraging AI.
- 19:26That, that that's a lot of pressure, I think for anybody.
- 19:28But yeah, it, it again gives us a that, that I think I like the
- 19:32how you phrase that in the question gives us that shared
- 19:34language where, you know, a user comes to us and says, yes, we've
- 19:38got to adopt this tool now and then I can say, well, look,
- 19:41yeah. So on device inference, that's
- 19:43awesome. Maybe from the standpoint of
- 19:48it's going to maybe slightly reduce the risk of data
- 19:52exfiltration, but there it opens up other cans of worms and I can
- 19:56actually refer to the AWARE framework and say, well, you
- 19:59know, the problem is, is that I'm going to lose potentially
- 20:02some observability, the ability to monitor, the ability to
- 20:06actually sort of reconstruct what might be taking place with
- 20:09an agent's reasoning, for example, if all that stuff is
- 20:13going on like on an end user's laptop, we've got to figure out.
- 20:18Also, not only if we can monitor that and say if there is an
- 20:23action that it that agent might be planning to take and we don't
- 20:28want it to take and we want to intervene before it actually
- 20:31executes anything. You know, we need that
- 20:34observability to be able to enable that, that kind of thing.
- 20:37So again, it, it helps me to just have a sort of a logical
- 20:41way to step through and, you know, along with Pradeep and
- 20:46other technology partners in particular, be able to talk
- 20:49through these issues. And I, I like sort of the
- 20:53independence of it because it's not just Ben thinks it's there a
- 20:56bunch of other smart people out in the industry have given this
- 21:00some thought. And you know that I think as a
- 21:03security practitioner, if there is any in the room or, or you
- 21:07know, online here, sometimes you have to sort of say, we'll look,
- 21:12this is the way that other companies do it and this is the
- 21:15way the industry does it. That that does have, you know,
- 21:18it is a little bit of a credibility.
- 21:21I'm half joking. But if Ben's bot would skip that
- 21:24answer, it would actually people like, oh, OK, the bot AI bot
- 21:27told me it was too risky versus versus Ben.
- 21:30I, I tease. I'm on, I'm on.
- 21:32Camera, so you know. There, there is a little bit of,
- 21:36you know, I call it half joking, but people do look to the
- 21:40responses that AI will generate and actually and Ben's team
- 21:44among other teams have actually created AI bots to kind of do
- 21:48some of that risk assessment, which I think is one of the
- 21:50components of that framework as well.
- 21:52We've just. Spoken about how you overcome
- 21:58the fights what are some of the real challenges when you're
- 22:04working together to make a go no go decision about a model, a
- 22:08tool, what have you when. We're evaluating a go no go
- 22:13decision and and degree like something is a tendency to look
- 22:18at only part of the problem and you want to take as holistic an
- 22:24approach as you can, right. So I mentioned like some people,
- 22:27they, they only know model security, they don't know the
- 22:29agentic security and, and the implications of that.
- 22:34The other thing about go no go decisions is that I think
- 22:38especially given how fast the AI space is moving, how the, the
- 22:45volume at which we and other companies are going to be
- 22:48adopting and building agents and integrating them together into,
- 22:52you know, massive work flows and all that stuff.
- 22:56The the technology is going to evolve, The use cases are going
- 22:58to evolve. The agents are going to change.
- 23:00They're going to be new use cases for them.
- 23:02They're going to be, you know, you know, new delegation change
- 23:06that are going to pop up. So the go no go decision is not
- 23:09a one and done is my point. It's iterative.
- 23:12It's iterative and. It has to be streamlined.
- 23:16And you know, so he and others that are the builders and that
- 23:19are really focused on delivering, they need to
- 23:22understand that they need it. They need to come back to this
- 23:26risk process. And this is where he's like, oh
- 23:29man, but that's good to say. But my job is to make it make
- 23:32sure that I can I can do that risk process at speed and at
- 23:36scale. So you have with.
- 23:37Clear criteria, so. You heard what?
- 23:39He said. I mean, you have to go back.
- 23:40Actually, we've got a. Better path, which is we're
- 23:43going to basically have a catalog that he can.
- 23:45There you go, right. So this is how do you simplify
- 23:48to Ben's point, Ben and legal and privacy have a need to know,
- 23:52hey, what is going on in terms of your organization, your
- 23:56ecosystem with regards to AI. So there is a documentation step
- 24:02that is an important piece. If you especially if you're
- 24:05regulated and things like that, that's going to become more of a
- 24:08requirement, especially is it customer facing, what type of
- 24:11classification it is. So a lot of those elements of
- 24:14that aware framework are actually going in.
- 24:16So one of the things I'm pushing or actually requiring is not at
- 24:22a software level. We already have a software
- 24:24catalog, if it's AI or not or whatnot, but a task level,
- 24:28activity level catalog of what is an agent doing, which the
- 24:33business or tech or whoever can essentially document, which then
- 24:38enables privacy, legal security to query or have an agent go
- 24:42against it. And then based on risk do what
- 24:45they need to. And if it changes, Ben?
- 24:48What's your advice for CIO Cisos to put a framework like AWARE
- 24:53into practice? Consider the.
- 24:55The bigger frameworks, the the more monolithic organizational
- 24:58frameworks that I mentioned and synthesize A framework like
- 25:03AWARE with them, inventory your agents, instrument them for
- 25:08observability and identity. If you can only do and focus on
- 25:13any part of the AWARE framework, it would be focused on those two
- 25:16things first and then go from there.
- 25:18This is great. Pradeep, how do we put this into
- 25:21practice? What should we do?
- 25:23People need to. Partner together.
- 25:25But you also have to simplify the messaging like the business
- 25:28folks want to know. Yes.
- 25:30No. Can I do it?
- 25:31Can I not? Why not?
- 25:33How quickly? So you've got Yeah, we business
- 25:35people can't. Understand a whole lot more than
- 25:37that, and to. Some extent.
- 25:39Nor should they, right? So, so that's the middle ground.
- 25:44I think that's important for CIOs and Csos.
- 25:46And as we work together, how do we help translate, simplify that
- 25:49message? Give them a framework that says
- 25:51here's what you can do. Here's what we need to work
- 25:54together to enable. How is.
- 25:56This agent governance landscape going to evolve over the next
- 26:00year or two. So from a.
- 26:02Compliance perspective. I can see us seeing a control
- 26:06criteria in SoC 2 within like say the next year and a half,
- 26:10two years. We need to be ready for that.
- 26:13I would love. To see a vendors playing in this
- 26:19governance space, so I don't have to work with Ben and team
- 26:22to configure that every single time we have different
- 26:25frameworks, I want an overlay kind of like you do with
- 26:28security that can work across different enterprise grade
- 26:32players. So enterprise grade is going to
- 26:33be important. Great.
- 26:35Do we have? Time for questions at all or
- 26:37when you. Started, you said how?
- 26:38Many agents 2600 we have over 6. Thousand 6000 OK.
- 26:43So when you scaled them starting from 10/20 hundred, what were
- 26:48your scaling challenges as you scaled to 6000?
- 26:51A lot of these now we. Have different platforms where
- 26:54agents exist, which you know, Ben's going to make sure they're
- 26:57all good to run. But a lot of these agents, the
- 27:01sprawl so to speak, happened because of the Glean platform
- 27:04and it enabled them. Now the Glean platform is one of
- 27:08the all the different data sources and things that we have
- 27:10because of the fine grained security controls.
- 27:13We trust it and the users are trained to run.
- 27:18Now, one of the things should we talked a little bit about this
- 27:20is when that started, we knew we were going to have more agents.
- 27:26Not even though we have 6000, how many are actively used?
- 27:28It's probably about 1300, candidly speaking.
- 27:31But what we wanted to do is we need to change that culture.
- 27:34We wanted to get people's behavior to understand the art
- 27:36of the possible. So we encourage people to go
- 27:39ahead and create agents, figure out how your life could be
- 27:42different. So they went on and started
- 27:44creating agents, started tweaking agents, learning from
- 27:47them. It was purposely done.
- 27:49Now, we knew that this was going to be a bit of a mess, but the
- 27:52purpose was different. The purpose this was get people
- 27:55interacting now Meanwhile, we also worked with Glee and said,
- 27:58hey, we need a way to moderate this stuff cuz it's going to be
- 28:00a hot mess. And I call it within a three,
- 28:04four month time the capability came in.
- 28:07We can also see metrics in what's being used, what's not.
- 28:10Now the moderation is in play, right?
- 28:13So part of this is not to say how do we stay one step ahead,
- 28:16but it's what's the goal at this point in time?
- 28:18It's a journey. Part of that journey was get
- 28:21people to create them and understand what's and get them
- 28:23excited. Now, which we're maybe a long
- 28:28winded answer here, but what we're getting now asked for is
- 28:31people want to create workflows. They're like, hey, great, I get
- 28:32the output now I want it to take action and they're very hungry
- 28:37to go to that next stage. So you have.
- 28:39Agents. And then they have kids and they
- 28:42move out to the suburbs and then before you know it you have a
- 28:45gentex sprawl. Yep.
- 28:47Great analogy you. Know the space is so dynamic
- 28:49right now again, you have to have a a good development
- 28:52process a structured and scalable you know tiered risk
- 28:58process as well because I think if you're applying you know
- 29:01governance to the same level of governance and control to every
- 29:05agent or every AI system out there your your governance
- 29:09capabilities are just they're going to crumble right they're
- 29:11not going to scale OK and with. That Ben Mayrides and Pradeep
- 29:16Manakara. Thank you.
- 29:17Thank you. Thank you both.
- 29:19Thank you, Michael. Good on Ben.