Latest / The Road to Accountable AI / Dave Trier (ModelOp): Turning AI Policies into Workflows

Transcript

Dave Trier (ModelOp): Turning AI Policies into Workflows · The Road to Accountable AI

• 0:09
  Kevin: Hi, I'm Kevin Werbach, professor of legal studies and
• 0:11
  business ethics at the Wharton School of the University of Pennsylvania.
• 0:15
  For decades, I've studied emerging technologies, from
• 0:17
  broadband to blockchain.
• 0:20
  Today, AI is promising to transform our world, but AI needs accountability,
• 0:26
  mechanisms to ensure it's developed and deployed in responsible,
• 0:29
  safe, and trustworthy ways.
• 0:32
  On this podcast, I speak with the experts leading the charge for accountable AI.
• 0:40
  Modelop provides AI governance solutions for large enterprises.
• 0:44
  I talked with CEO Dave Trier about what it means to turn corporate
• 0:48
  policies into automated workflows, why pre-approved patterns for low-risk AI
• 0:55
  can speed, deployment, why traditional governance risk and compliance, or
• 0:59
  GRC approaches break down for AI, and what it takes to govern agents all
• 1:04
  the way down to the network layer.
• 1:06
  Here's my conversation with Dave
• 1:11
  Welcome, Dave, and thank you for joining me on "The Road to Accountable AI."
• 1:16
  Dave: Thanks.
• 1:16
  Appreciate you having me, Kevin.
• 1:18
  Why don't we start by talking about what exactly Modelop is and where it fits
• 1:24
  within the larger AI governance stack?
• 1:27
  Sure.
• 1:27
  Be happy to.
• 1:28
  We're an enterprise software company that's focused on large scale AI delivery
• 1:32
  and governance, and we'll talk, I'm sure, quite a bit more about where
• 1:35
  that fits into the overall ecosystem.
• 1:38
  But at the end of the day, we help large enterprises to go and implement AI
• 1:42
  governance just as part of how they do AI delivery, and we have a very powerful
• 1:47
  software platform to enable them to do that at scale What's different about
• 1:52
  doing AI governance for large enterprises?
• 1:55
  Yeah.
• 1:56
  With large, with large enterprises, there is a lot that's involved
• 2:00
  from a policy perspective, right?
• 2:02
  So for large enterprises, it's not just the regulatory policy.
• 2:06
  There's all kinds of corporate policies, as you can appreciate.
• 2:09
  There's corporate policies that are related to, you know, the use of data,
• 2:14
  corporate policies related to security, corporate policies related to the type
• 2:17
  of information that can be exchanged, the type of usages that they may allow
• 2:22
  or disallow from a corporate perspective.
• 2:25
  So at the end of the day, it comes down to how do you help them implement and
• 2:28
  enforce those corporate policies at scale?
• 2:32
  There's a few other minor ones, Kevin, but that's, that's at the, the core
• 2:35
  of it, the key difference when you talk about enterprise AI governance.
• 2:39
  What does that look like in practice, taking those corporate
• 2:42
  policies and translating them?
• 2:43
  Yeah.
• 2:44
  It's, that's, one of the, the big things that you have to do is that
• 2:47
  generally there's corporate policies that are 80, 100-plus pages, right?
• 2:53
  And so how do you translate that into something that is really
• 2:58
  straightforward, but also embedded into how the different business teams
• 3:02
  want to go and develop and use AI?
• 3:05
  So kind of a-- In practice, what happens is that, and this is obviously
• 3:09
  where our software is great at, but what happens is we take those 80
• 3:12
  or 100-page documents, which have a lot of process flows as well, right?
• 3:16
  There's a number of different processes.
• 3:18
  So we take those, and first and foremost, we take the processes
• 3:21
  and turn those into something that is really orchestrated, right?
• 3:26
  So instead of somebody going to have to read the 120-page
• 3:28
  document, rather it's orchestrated.
• 3:31
  So if you're just a developer of an AI solution or you buy a AI
• 3:34
  solution, you just kind of get shepherded through the process.
• 3:37
  You get shepherded through being able to adhere to all the different steps
• 3:41
  that are pertinent in the policy to what you're trying to do with your AI solution.
• 3:46
  So a lot of what we do is help to translate the policy into something
• 3:50
  that is usable, actionable, and obviously scalable, around that
• 3:53
  policy, set of policies and procedures throughout the overall, enterprise.
• 3:58
  How automated is that?
• 4:01
  Yeah, we try to automate as much as possible, right?
• 4:03
  Obviously, we can't automate everything.
• 4:05
  And the purpose of governance and the risk teams and the control partners is
• 4:11
  really to have effective challenge, right?
• 4:12
  So we d- you know, you never want to get rid of the actual effective
• 4:16
  cha- challenge that's done by the different humans involved, right?
• 4:19
  From legal risk, compliance, governance risk, et cetera.
• 4:22
  But we try to automate all of the really just the mundane routine tasks.
• 4:27
  Things as simple as you know, we, we do need to analyze what
• 4:30
  data is being used, and are there any potential PII, PHI in there?
• 4:36
  You know, a lot of times that is done manually, right?
• 4:39
  Another example is that you want to do a risk tiering.
• 4:42
  Very simple concept, but a lot of that's still done with an Excel spreadsheet.
• 4:46
  You go fill it out, and, this Excel macro kicks something out that says,
• 4:49
  "Oh, this is a high risk," or, "This is unacceptable use," in the EU terms.
• 4:53
  So on average, to answer your question, it's, for the, the lower priority,
• 4:58
  or sorry, lower risk, maybe medium risk, you can actually automate
• 5:00
  between 60 and 90% of that process.
• 5:03
  For the high-risk ones, that's where you do want to take a little bit of extra
• 5:06
  time, as you well know, to have humans go and take a, a very in-depth review,
• 5:11
  do their effective challenge around it, identify where there's potential
• 5:15
  findings or validations, conclusions in the banking world, et cetera.
• 5:20
  But so that one you probably, you know, a little bit lower percentage.
• 5:22
  But, like I said, the low-risk ones or one that we're coming up quite a
• 5:25
  bit around, Kevin, a concept that we, we call just pre-approved patterns,
• 5:30
  especially in the GenAI and agentic world.
• 5:32
  If there are, at the end of the day, that people are just creating chatbots, right?
• 5:36
  And so there are certain patterns that you've identified that say, "Okay,
• 5:39
  well, we know this pattern. We don't need to redo the whole risk analysis
• 5:44
  around it because, you know, 90% is the same as the last one," right?
• 5:49
  So that's where we get some real, very nice savings, time savings, accelerations,
• 5:55
  because we say, "You know what?
• 5:56
  90% of this is the same.
• 5:58
  We've got all the evidence, and we've got all the reviews.
• 6:00
  Everything's already been done with that, so we can just focus on the
• 6:02
  10%." So again, just, that's just an area where you actually can help to
• 6:07
  streamline and automate a lot more than, than you would think through
• 6:10
  the use of, of technology like ours.
• 6:13
  Kevin: But presumably, as you said, companies have different policies,
• 6:16
  especially if we're talking about large enterprises, and, and certainly
• 6:19
  some of it is, industry-based, how do you help them decide what are those
• 6:24
  patterns that for them they can automate?
• 6:27
  Dave: Yeah.
• 6:27
  So first off, absolutely on every single enterprise has a slightly
• 6:32
  different policy- Mm-hmm … and sometimes drastically different.
• 6:35
  And, you know, just in general, that's why we really, from our company perspective,
• 6:39
  only focus on the Fortune 500 Global 2000- Mm … is because we really
• 6:42
  designed it so that every company can make slight tweaks, they can configure
• 6:46
  and tailor to their specific policies and procedures as they put forward.
• 6:50
  So that's that, that… On the first bit, I completely agree with you, and
• 6:54
  that's why we've designed it as such.
• 6:55
  Mm-hmm.
• 6:56
  And then on the second one, which you said of, well, how do I then take the learnings
• 6:59
  of, from, say, one customer to the next when they do have a different policy?
• 7:03
  In general, that's where you do start to see some repeatable
• 7:06
  patterns, believe it or not, right?
• 7:08
  It all comes down to the usages.
• 7:10
  What are the target usages?
• 7:11
  What data do they have access to?
• 7:13
  what other tools in the, agentic world are the tools they have access to?
• 7:17
  Is this internal versus customer facing?
• 7:19
  So there are a number of core, questions or inputs that-
• 7:23
  can help to determine what those patterns look like, even if it's
• 7:26
  you're looking at a bank versus wealth management, if you will.
• 7:29
  they get, repeats that are fundamental to determining things like the risk
• 7:33
  rating and how we want to handle it from an ongoing process perspective.
• 7:39
  So let, let's just drill down a little bit more.
• 7:40
  Sure.
• 7:40
  You talked about n- you know, the lower risk tiers where it's possible
• 7:44
  to have a lot of auto-approval.
• 7:46
  Yeah.
• 7:46
  are there consistent factors that would put something in that bucket,
• 7:53
  or does that have to be a bespoke process each time to say what's at
• 7:57
  a low tier for your organization?
• 7:59
  Yeah, so, two parts of that.
• 8:00
  So in terms of the, the tiering, that is something that's often very
• 8:04
  well defined- Mm-hmm … especially for the regulated industry.
• 8:06
  So the- Mm-hmm … the actual risk tiering itself is something
• 8:09
  that is, again, well defined.
• 8:11
  Mm-hmm.
• 8:11
  It's very easy to help to automate based on the inputs, if you will.
• 8:15
  Then the second part of that is then you can further do, like I said,
• 8:18
  those pre-approved patterns- Mm-hmm after the risk tiering's done.
• 8:20
  So let's just say that the risk tier kind of kicks it out as a low risk, if
• 8:23
  you will, and then you can look at a number, to your question, of factors,
• 8:28
  and those factors often come involved of, okay, is this a, a pre-approved LLM?
• 8:33
  So oftentimes that's, you know, the, the, the core team technology and
• 8:37
  risk and governance teams will go and kind of validate and approve an
• 8:41
  LLM version in particular to say, "Yep, we've looked through this one.
• 8:45
  We've reviewed it.
• 8:45
  We've done all the, the tasks.
• 8:47
  We've done all the testing.
• 8:48
  We've done all the due diligence."
• 8:49
  And so they'll list this as approved for production usage.
• 8:52
  Okay, great.
• 8:53
  So if you are now developing a, a chatbot or an agent that's using that pre-approved
• 8:57
  LLM, that naturally says, all right that's one item we can check to say- Mm-hmm
• 9:00
  … "Yep, we're into that pre-approved bucket." Another one, obviously very
• 9:03
  common, is this internal versus external.
• 9:06
  Internal, that gets, obviously, has a lot less risk potentially around it, so
• 9:10
  that's another one that you can look at.
• 9:12
  And then the, another one is also if you are connecting to different
• 9:16
  data sources, especially the corporate data sources, around it.
• 9:19
  If it's something that you're really not connecting in and pulling in other
• 9:23
  corporate data, if you will- Again, that's something that you can generally
• 9:27
  say, "All right, well, we're limiting our exposure because we're not gonna
• 9:30
  feed it any potentially sensitive data."
• 9:33
  Whether or not that data source has sensitive data, it still
• 9:35
  limits your exposure around it.
• 9:37
  So those are kind of just a, a couple of examples.
• 9:39
  and, you know, each company is actually refining this as they go- right?
• 9:43
  So they'll get a new, you know, a new gen AI, agent that comes through the pipe-
• 9:47
  and they say, "Oh, wait a second. Yeah, this looks the same as this, and this,
• 9:51
  but these two new things came in the door." But I think we can see that, one
• 9:55
  of those two items, we'll see a lot again.
• 9:57
  So then they'll go and approve that and have that as part of that kind
• 10:00
  of pre-approved pattern, if you will, put that into the cycle, et cetera.
• 10:04
  Kevin: So it's a- Mm-hmm … it's a learning piece as well from the, the
• 10:07
  overall kind of execution, protocol from the different governance teams.
• 10:11
  All right, so at, at the bottom there's things that get pre-approved.
• 10:14
  Presumably there are some things that are absolutely prohibited at the top.
• 10:17
  What do the middle tiers look like, and, and what's the, the functionality
• 10:22
  that you help, provide to organizations to go through the necessary processes?
• 10:27
  Dave: The middle tiers is where you get into just making sure that, well,
• 10:30
  first and foremost, you're following all the steps in the corporate-
• 10:32
  Mm-hmm … policy, So this is where part of what we do is we're mainly focused
• 10:36
  on workflow automation and automation- Mm-hmm … of those processes, right?
• 10:39
  So this is where the integrations come into play a lot.
• 10:41
  So integrating with your security scanning capabilities- Mm-hmm … integrating with,
• 10:45
  say, your data profiling, data integrity, type capabilities to make sure that,
• 10:49
  yep, security cans, security scan came back clean- Mm-hmm … data profile came
• 10:53
  back and indeed it does not have any PII.
• 10:55
  And then pulling in, excuse me, other corporate policies like looking at if it's
• 10:59
  a third-party AI, pulling in the Ts and Cs that might come from a procurement system.
• 11:03
  Hey, do we have the Ts and Cs?
• 11:05
  Has legal signed off on them?
• 11:06
  Terms and conditions- Has procurement signed off?
• 11:07
  Right.
• 11:08
  So those are the types of things, and then you move on to testing, right?
• 11:11
  So e- for the, the kind of that middle tranche, you can have a set of, of tests.
• 11:16
  It's not gonna cover every- Use case specific nuance from a testing
• 11:20
  perspective, but there are some baseline tests that you can take a look at
• 11:24
  that, again, if you said, "Hey, there's no PII, let's go and check," right?
• 11:27
  Let's go and test it and make sure there isn't any PII coming in or out of it.
• 11:31
  You can look at things such as, hey, we said that this is going to be, you know,
• 11:35
  a, a chatbot and that this chatbot, we wouldn't expect it to be overly negative.
• 11:38
  Let's go do, run a sentiment analysis- Mm-hmm … to determine, hey, is
• 11:41
  this coming back overly negative?
• 11:42
  We wouldn't not want that for even if it was an internal chatbot, if you will.
• 11:46
  So there is a set of tests that then you can, run and help to automate as
• 11:50
  much of those as possible, looking at deviations from what was requested, as
• 11:55
  well as kind of the accepted norms and risk tolerances that you have around
• 11:59
  some of those tests, if you will.
• 12:00
  So that's kind of what happens in that middle tranche of just- Mm-hmm … let's
• 12:03
  try to, again, run all the different scans and the tests, help to prepare
• 12:08
  some documentation so that the reviewers can then take a look as like,
• 12:12
  all right, well, here's a summary.
• 12:13
  Yep, it passed security scan.
• 12:14
  Yep, it passed the data integrity.
• 12:16
  Mm-hmm.
• 12:16
  Here's the tests that run.
• 12:17
  Here's the results of those tests.
• 12:18
  Here's where it actually failed a threshold.
• 12:20
  So you gotta just prepare a package, right?
• 12:22
  You prepare a package for the different reviewers, whether it's legal or the
• 12:26
  risk team or the technology team.
• 12:28
  And so in that way you're trying to streamline the process.
• 12:31
  You're pulling in the right teams at the right time to try to,
• 12:34
  again, ease that process as much as possible Who are you typically
• 12:39
  selling to in these organizations?
• 12:42
  Yeah, so we're typically selling to, at the end of the day, the person or persons
• 12:47
  that are accountable for AI, right?
• 12:48
  Mm-hmm.
• 12:49
  That are accountable for what's happening, decisions made-
• 12:51
  Accountable for accountable AI, yes.
• 12:54
  So the person… and I say it because that role is different-
• 12:58
  Yeah … within each environment, within each, sorry, enterprise.
• 13:01
  It could be a chief AI officer if they have one, a chief
• 13:03
  data and analytics officer.
• 13:05
  Could be a CIO in, in certain cases as well, if they are the ones that
• 13:09
  are accountable to the, executive C-suite around the use of AI.
• 13:13
  But ultimately, that's the main person just to ensure that we can actually safely
• 13:19
  scale AI out to all the different teams.
• 13:21
  That's obviously the end goal for these large- Mm-hmm
• 13:23
  enterprises.
• 13:24
  Let's get AI into the hands of everyone to help to improve
• 13:29
  various aspects of the business.
• 13:30
  But they can't do that until they know that they got the foundation in place.
• 13:34
  They know that if John over in accounting de- has an AI idea and
• 13:39
  wants to use Copilot Studio, that it's gonna go through the process.
• 13:42
  It's gonna go through that engine that says, "All right, we check,
• 13:45
  check, check, check, check.
• 13:46
  Came out, everything looks green.
• 13:47
  Now we can use it." That's really hard to do at the enterprise when you've
• 13:51
  got accounting that's, you know, not- Mm-hmm … very technology or, risk,
• 13:54
  knowledgeable, if you will, no offense to them, versus somebody that is potentially
• 13:58
  part of the data science team, super technical and pushing the bounds of it,
• 14:02
  but they aren't, they're not experts, they're not subject matter experts- Mm-hmm
• 14:05
  in risk and policy, right?
• 14:07
  So how do you make sure that all the different teams, technical, non-technical,
• 14:11
  are able to effectively go from idea to usage in a way that's scalable and
• 14:17
  also, of course, responsible back to the organization- Mm-hmm … that we're
• 14:21
  following those corporate policies?
• 14:23
  How do you do that?
• 14:24
  Yeah.
• 14:26
  That's the that's the, the reason we exist, to be quite
• 14:28
  honest and overall, Kevin.
• 14:29
  So we do that with a couple of key pieces.
• 14:33
  first and foremost is that we do have a very, comprehensive system of record, we
• 14:37
  call it an inventory, obviously- Mm-hmm … of just so you know every single AI-
• 14:41
  that's out there.
• 14:42
  That could be AI, could be ML, could be regressions, could be vendor.
• 14:44
  It doesn't matter to us.
• 14:45
  We took a lot of time to design how do we make sure we encompass and be
• 14:49
  able to track every single AI system.
• 14:51
  Of course, with that, we have a discovery capability to help organizations
• 14:55
  go and find what's out there.
• 14:56
  But then the real secret sauce from there is what we call our, our
• 15:00
  Modelop AI Delivery engine, or MADE, and that's what takes, as we started
• 15:04
  the conversation, those policies and turns those into enforceable workflows
• 15:09
  so that you can, it doesn't matter if you've got 40 different flavors
• 15:14
  of how you need to do the different steps in the process, but you have
• 15:18
  the ability to go and say, "All right.
• 15:21
  If an idea comes in, here's the workflow that must be followed." Those workflows
• 15:26
  will pull in the right people and the right systems at the right time to
• 15:30
  make sure every- check is done, every tick in the box every analysis, any
• 15:36
  evidence that needs to be captured, all of the, as I said, the risk tiering,
• 15:39
  the automated testing, the triggering of security scan, all of that is check.
• 15:43
  Mm. And really the powerful part about it, and the most important
• 15:46
  part, is that there's actually not one process to rule them all, right?
• 15:50
  Mm. There's actually a variety of different processes.
• 15:53
  It's based on risk, like we talked about, high risk versus low risk.
• 15:56
  Mm. High risk has a lot more involved versus a low risk.
• 15:58
  There's also different processes for, say, customer-facing versus internal,
• 16:03
  or external, you know, third party purchase SaaS-based AI versus internal.
• 16:08
  Mm. As well as in target environment.
• 16:09
  Is this gonna be running within our network, or is it running within a, a
• 16:12
  partner network, or maybe it's in a cloud?
• 16:14
  So there's a variety of different processes, and that's what, again,
• 16:18
  we took a lot of time to say, for the Fortune 500, we have to accommodate
• 16:23
  all those different scenarios.
• 16:25
  Mm. And so what happens is the, again, back to your original question, we take
• 16:27
  their policies with all those variations, right, the 30, 31 flavors, right?
• 16:32
  And then can turn those into actionable workflows that are really AI and model
• 16:39
  context aware to know, okay, it's high risk- Mm … go down this path, and
• 16:44
  then make sure every step is followed.
• 16:45
  It's low risk, go down this path.
• 16:47
  Mm. Oh, it's low risk and it's cloud-based, okay, well,
• 16:49
  we know what to do here.
• 16:50
  So it's that kind of delivery engine is what we call it, that turns that,
• 16:54
  again, very manual paper-based policy that- Mm-hmm … often spreadsheets,
• 16:59
  emails, super ad hoc- Mm-hmm … into something that is scalable, repeatable,
• 17:03
  consistent, and of course auditable and traceable to make sure- Mm
• 17:06
  Kevin: you know, no stone is unturned, if you will.
• 17:08
  And just to, to drill down a little more in where you started, the,
• 17:12
  the system- Sure … of record.
• 17:13
  Yeah … you know, makes sense, seems obvious that you need to know what AI you
• 17:16
  have if you're going to govern it, but we're talking about large enterprises,
• 17:20
  all kinds of things going on, shadow AI, people bringing things in, third party.
• 17:25
  Dave: What are, what are the big challenges and, and the keys to
• 17:28
  building that foundation effectively?
• 17:30
  Yeah, I think the, the, the first one is just making sure that you have
• 17:35
  a system of record, to be honest.
• 17:38
  But then from there it's, th- then making sure that you can tap into the various
• 17:44
  sources, that you can discover where those AI, shadow AI, and other components are.
• 17:50
  So that means that you can tap into the procurement system.
• 17:53
  So for some reason they purchased some SaaS-based AI that, you know, they didn't
• 17:58
  report back to the governance team.
• 17:59
  God forbid.
• 18:00
  Hopefully they don't.
• 18:00
  But- Mm-hmm … you know, being able to tap into the different accounts.
• 18:03
  So you think about your Azure, your Google, your AWS accounts to make
• 18:07
  sure you can go and scan, hey, out in AgentCore right now, I see 10 agents.
• 18:12
  Five of them are registered, five are not, right?
• 18:14
  You know, being able to tap into your Copilot Studio, your Agent 365
• 18:18
  on the Microsoft side, et cetera.
• 18:20
  But then also being able to tap into, at the network level, which, so
• 18:24
  going way down in the stack, right?
• 18:26
  To determine, okay, well, we look like we actually detected
• 18:30
  some MCP traffic, if you will.
• 18:32
  The common security thing to do is just block it.
• 18:34
  If you don't know it- Mm … you block it, right?
• 18:36
  which is what they should do, right?
• 18:37
  But then from there you want to say, "All right, we detected this traffic.
• 18:40
  Maybe we do want to put this in the queue to get approved," right?
• 18:43
  So that, again, you're kind of taking at multiple levels.
• 18:46
  At the network level, the kind of account level for cloud and other type of- Mm
• 18:51
  … platforms, if you will, as well as maybe even tapping into the procurement system.
• 18:55
  So now I'm, and I make it sound easy, but as you can appreciate
• 18:58
  in a large enterprise, there's- Yeah … some red tape involved in that.
• 19:01
  but that's the way that you get into kind of full scope and understanding of here's
• 19:05
  all the different places AI might be used.
• 19:09
  Kevin: What do you see about the organizations that are more effective
• 19:12
  in implementing AI governance versus the ones that struggle?
• 19:16
  Yeah.
• 19:17
  So I think the first is just top-down support, and I know that sounds
• 19:21
  so fundamental and so, "Hey, Dave, that's, like, what a first-grade answer
• 19:25
  would say." But it is so, it is so true because there's so much, focus.
• 19:29
  Dave: There's a lot of, different leaders or different technologists
• 19:33
  that are kind of chasing the latest and greatest technology.
• 19:36
  And so there's a lot of emphasis and a lot of focus on, "Let's just
• 19:39
  go and find the best LLM," right?
• 19:42
  "Let's go and build some bunch of agents." And so that's kind of the--
• 19:46
  unfortunately gets too much attention from leadership, and then you forget, "Okay,
• 19:50
  well, that's not gonna be effective.
• 19:52
  If we can't get any of them over the line, it's not gonna be effective."
• 19:55
  I.e., you need to make sure that you have the right governance, and as I
• 19:57
  like to say, governance can actually be an accelerator, doesn't have to be
• 20:00
  the brakes. So it's, it's first is the top-down to say, "We're going to do this.
• 20:04
  We're gonna build these different AI solutions, but we're gonna do
• 20:07
  so in a way that has governance embedded in about how we do delivery."
• 20:12
  You can't have one without the other.
• 20:14
  And so that, for me, that's the first one.
• 20:15
  The ones that do it the best have the top-down support, and then obviously
• 20:20
  the different program teams underneath it to have that tight interlock-
• 20:23
  Mm-hmm … between AI development, delivery, and governance overall.
• 20:27
  So that's number one.
• 20:28
  the second is I'll just say, really as a kind of a lesson learned from enterprises
• 20:34
  out there is that you can't just use your traditional approaches to governance.
• 20:39
  Kind of you think your traditional GRC type approaches, which were designed for
• 20:43
  a whole bunch of corporate reasons, right?
• 20:44
  Mm-hmm.
• 20:44
  So you could…
• 20:45
  You know, there's physical, there's access to the location itself, right?
• 20:49
  There's a whole bunch of policies, but AI is its own beast, right?
• 20:52
  It's constantly changing.
• 20:54
  Mm-hmm.
• 20:54
  It's related to data.
• 20:55
  It's a, a, a unique combination of software, data, risk that's inherent
• 20:59
  because it's not deterministic, right?
• 21:01
  So trying to take what has been done in the past and apply it to AI just leads to
• 21:07
  a lot of- … slowdowns, a lot of failures on the delivery side, if you will.
• 21:11
  So a- again, the best ones think about, okay, this is fundamentally
• 21:15
  distinct from what we've done in the past from a governance perspective.
• 21:19
  Mm-hmm.
• 21:19
  And second, it's fundamentally distinct from the development and
• 21:23
  execution side of the house- Mm … a- as well as distinct from security.
• 21:27
  And I'm sure you'll have questions, and we can get onto that, right?
• 21:29
  But these are different disciplines, right?
• 21:32
  You develop and you, a student cannot grade their own papers, right?
• 21:35
  So developers cannot grade their own papers, right?
• 21:37
  So governance and developments are distinct.
• 21:40
  Security is focused on a different area, if you will.
• 21:42
  So again, back to your original question, it's, it's a recognition that these
• 21:46
  are distinct disciplines that are interrelated and need to work together,
• 21:49
  but they need to be there, you know, again, in a distinct basis overall.
• 21:54
  Kevin: Mm-hmm.
• 21:55
  You made the argument that governance is actually an accelerator, and I've, I've
• 21:58
  made this argument as well, and I've had conversations like this with a number of
• 22:01
  guests, and so I, I believe that's true.
• 22:03
  But I'm really curious in practice- Yeah … how do you make that case
• 22:08
  and where do you see, evidence in companies that that can actually happen?
• 22:11
  Dave: So just based on what we've done with our customers- over the many years.
• 22:15
  Before walking into our customers, we did some analysis, and the different
• 22:19
  Governance and risk activities generally were causing anywhere between 40
• 22:23
  and 70% of the delays- in what I've talked about from the beginning,
• 22:26
  the idea to actual production usage.
• 22:29
  That's a pretty big range.
• 22:30
  So if you think about, hey, you've got this pent-up business demand for this
• 22:33
  AI solution that's gonna deliver X dollars in benefit- and the governance
• 22:38
  tasks are the things that are slowing it down to the point where, in some of
• 22:42
  our customers before us, it would take them between 9 and 12 months to get
• 22:46
  a- an AI or ML solution out the door.
• 22:48
  Right.
• 22:48
  By the time you get it out the door, it is deprecated, right?
• 22:51
  You know, it's, it's- Mm-hmm … it is fundamentally, not going
• 22:54
  to provide the business value that you would expect overall.
• 22:57
  So that's where we start, is just saying, "Okay, let's look at where
• 23:00
  it is today." And Kevin, to your original question, a lot of it is the-
• 23:03
  manual processes, the ad hoc, you know, cycles that are involved.
• 23:07
  You know, you're a developer and you develop something wonderful, and
• 23:11
  then you go and you say, "All right.
• 23:12
  Well, I'm ready to push a button.
• 23:13
  Let's get it technically deployed." And they're like, "Whoa, whoa, whoa, whoa.
• 23:15
  Did you talk to Johnny?" "Oh, wait.
• 23:17
  No, Johnny says you're using, you know, uh, uh, uh, PII data."
• 23:20
  Then you gotta talk to Jane.
• 23:21
  Jane says, "Oh, nope, you gotta talk to legal." So then you get this
• 23:23
  runaround, and you're just kind of a ping-pong nature of going out to
• 23:27
  the different teams, and nobody has that fundamentally understanding.
• 23:30
  They don't have the blueprint at the end of the day of how do
• 23:33
  you go from idea to production.
• 23:34
  It's a constant, you gotta go and figure it out yourself, read
• 23:37
  the policy document, et cetera.
• 23:39
  So that's the way that we help to become an accelerator.
• 23:42
  I'll go back to that case of the- Mm-hmm … before we came in, it was
• 23:45
  a large financial, it was between 9 and 12 months to get an AI and
• 23:49
  even an ML solution out the door.
• 23:52
  And that was because, as I said, there was just not a cohesive
• 23:56
  collaboration across the teams.
• 23:57
  There wasn't a documented process across here's the different steps
• 24:01
  involved, here's the different teams involved, the different
• 24:02
  approvals, the evidence, et cetera.
• 24:05
  And so we, with our, product overall, but first and foremost,
• 24:08
  lay down here's the blueprint.
• 24:09
  Here's the playbook, right?
• 24:10
  Here's the playbook across all the different teams, the different
• 24:13
  flavors, the different plays.
• 24:14
  They use a football analogy.
• 24:15
  You got your playbook, and you got a bunch of different plays.
• 24:17
  Those plays, as I said, high-risk play versus low risk, right?
• 24:20
  You know, cloud versus internal.
• 24:22
  So you develop that playbook, and then you obviously Modelop is great about taking
• 24:25
  that playbook and enforcing it, using it with automation, putting those processes
• 24:29
  as live orchestrations, et cetera.
• 24:31
  So in that way, we took that 9 to 12 and reduced that to under three months.
• 24:36
  And that was purely because they had the repeatable playbook.
• 24:39
  They had the, streamlined understanding of who needed to be, doing different
• 24:43
  approvals, reviews, testing, et cetera, and orchestrating as much of that as
• 24:47
  possible via automations and integrations.
• 24:50
  Mm-hmm.
• 24:50
  So that, that's like, again, why both you and I agree that it can be an accelerator
• 24:55
  as, as it's proven through some of our case studies with our customers thus far.
• 25:00
  Kevin: speed to market, is one way companies can think about how
• 25:04
  effective they are with AI governance, whi- what you've talked about.
• 25:07
  What are the other ways that, that managers in these enterprises, are
• 25:11
  able to evaluate how they're doing?
• 25:14
  Yeah.
• 25:14
  so I think the, there's a couple ways.
• 25:17
  Speed to market, obviously, from a business perspective.
• 25:19
  Dave: The, the second is that the demand for AI, especially with gen
• 25:24
  AI and now agentic, it's off the charts, and I'm sure, you know-
• 25:27
  Yeah … you've talked about this with some of your guests before.
• 25:29
  Sure.
• 25:30
  So if you think about it from the con- I'll just say control partner perspective,
• 25:33
  the legal risk, compliance, governance, et cetera, they're staring down a gauntlet.
• 25:38
  Oh my gosh, we're gonna have to go and review, you know, 1,000, right?
• 25:42
  Or, we have one customer that's, you know, in the many, many
• 25:45
  thousands of different AI solutions.
• 25:46
  So they're staring down this gauntlet.
• 25:48
  What do I do?
• 25:49
  Like, there's no way I can keep pace with this.
• 25:50
  So the, the second reason is actually they're faced with two choices: I
• 25:55
  go and scale up and have a massive team to handle this demand, or I turn
• 25:59
  to, you know, software that can help us to keep pace with this demand.
• 26:03
  So yes, time to market, but second is, can we have that pace of innovation
• 26:08
  without massively increasing the cost, the human cost, the human headcount from
• 26:16
  that, again, control partner perspective?
• 26:18
  So that's a second reason to do, you know, have a, a effective, a
• 26:21
  responsible, accountable AI program.
• 26:23
  And then the third, of course, is just why it exists in the first place,
• 26:26
  protect the organization, right?
• 26:28
  Yeah.
• 26:28
  So protecting the organization from, you know, undue risk, from potential brand
• 26:33
  exposure, hopefully that never happens.
• 26:34
  Obviously, any audit or regulatory concerns, but at the end of the
• 26:37
  day, you just wanna protect the organization from something going wrong.
• 26:42
  It will go wrong, but how do you protect it from catastrophic wrong, if you will?
• 26:48
  Kevin: Yeah, but, you know, how do executives assess that?
• 26:50
  Because if, if the catastrophic wrong hasn't happened- Yeah … then
• 26:53
  you're proving a negative.
• 26:54
  And, and clearly legal and compliance matters a lot, but- Mm-hmm … there
• 26:59
  are all these gray areas.
• 27:00
  and, you know, we've got the EU AI Act and we've got the, you know, GDPR and
• 27:05
  different data protection requirements.
• 27:06
  Dave: And, and, and those at least it's, it's concrete what the obligations are.
• 27:10
  But there's still a lot of these areas where, for most of these
• 27:13
  organizations, there's not a definitive legal mandate or regulatory-
• 27:17
  Yeah … mandate they have to follow.
• 27:19
  How should they think about how far to go?
• 27:22
  Yeah.
• 27:22
  So, again, first off, we always hope they do the right thing
• 27:25
  and protect the organization and many leaders will do that.
• 27:28
  second, that their AI will go wrong, and there's plenty- Mm-hmm … of examples
• 27:32
  out there that you've seen where, I think one of the first ones was that a
• 27:36
  chatbot, you know, responded to an airline passenger and said, "Oh yeah, sure.
• 27:40
  That's fine.
• 27:41
  You can get the, uh, the grievance rate for, you know, the funeral
• 27:44
  you have to go to." So there's actual financial implications
• 27:47
  should AI go wrong, around it.
• 27:49
  But then third, I would say of why, a- again, a really a catalyst to
• 27:56
  take this seriously is actually the spend, believe it or not.
• 27:59
  Yeah.
• 27:59
  So the spend that is starting to come about because of just the token maxing
• 28:05
  and just in general the variable nature of how the different frontier lab
• 28:10
  providers are doing this and changing kind of their variable rates, the spend
• 28:14
  is growing quite significant, right?
• 28:16
  And so it's not just a regulatory bit, it's also a financial and
• 28:21
  operational governance, if you will.
• 28:24
  So you can, in, in a way, if you have the proper, governance in place,
• 28:28
  you're not just looking at did I, as I said, do all the reviews, approvals,
• 28:31
  regulatory compliance, et cetera, but you're looking at what's it costing us?
• 28:37
  Yeah.
• 28:37
  What's the value we're getting out of it?
• 28:38
  Does this make sense, right?
• 28:40
  Mm-hmm.
• 28:40
  And so having that level of financial due diligence, just to have that, again,
• 28:46
  fiduciary duty back to your company, does it even make sense to continue
• 28:50
  on with this AI solution because the cost is X- … value TBD, right?
• 28:55
  So i- in that way, it, it is something that I'm seeing.
• 28:59
  I'm sure you've heard, Kevin, with some of your conversations and what you're seeing.
• 29:03
  That is being a ca- that is a catalyst.
• 29:04
  Mm-hmm.
• 29:05
  It's a, it's a big catalyst to- Yep … say, "All right. It's time to rein it
• 29:08
  in. Let's get a little bit of control around this Wild West of AI," just
• 29:11
  purely from a, a fiduciary perspective.
• 29:14
  Kevin: And then how much does the rise of agentic AI change what you do?
• 29:20
  it's been exciting, Kevin, from our perspective, it's been super exciting.
• 29:24
  It doesn't change our technology because we designed our technology from the
• 29:27
  very beginning to be able to support all types of AI, both now and in the future.
• 29:31
  Dave: So when LLMs came out, we added it overnight 'cause we
• 29:34
  just designed our technology.
• 29:36
  So it hasn't changed our technology.
• 29:37
  What is it changing again, why it's exciting, is because agentic AI
• 29:41
  really has been the catalyst that's pushed over the edge these enterprises
• 29:46
  that said, "Nah, you know what?
• 29:48
  We can just get by with what we have with our responsible AI program.
• 29:52
  Yeah, we can just continue on with SharePoint and spreadsheets."
• 29:56
  That doesn't work with agentic AI.
• 29:57
  There's too many.
• 29:58
  It's too much.
• 29:59
  There's too much that could go wrong because now the agents can make decisions
• 30:04
  on behalf instead of just informing employees of what decisions could be
• 30:08
  made, but they're making decisions on it, which could have real actions.
• 30:12
  And even if they say, "Well, I don't want it to make decisions," there's still
• 30:15
  the opport- there's still the chance that the agent does it anyway, right?
• 30:19
  Which we've seen all those stories of, out in the, the wild about different agents
• 30:23
  that have gone rogue and even despite the directions not to go rogue, right?
• 30:26
  So for us, it's changed in that, as I said, agents are so, in such high
• 30:32
  demand from the different business teams that it has, as I said, pushed
• 30:36
  over the edge to say, "You know what?
• 30:37
  We need to fundamentally do something different here.
• 30:40
  We need to, have this as a, a comprehensive program and
• 30:44
  not just an afterthought Yeah.
• 30:46
  Kevin: And we're almost out of time, but that's why… so h- how can
• 30:49
  agents though be effectively governed?
• 30:50
  I understand we're talking in the enterprise, we're not talking
• 30:53
  about, you know, unreleased frontier models in testing at the labs.
• 30:57
  But there is still this problem that, you know, we have this alignment issue
• 31:00
  where you can tell the agent something, but it may still do something else.
• 31:03
  Dave: Yeah.
• 31:04
  So there's a couple levels as you probably, well know
• 31:06
  from a control perspective.
• 31:08
  And the first thing on the controls is just limiting the usage and
• 31:11
  making- … well, first defining the usage, and then defining what are
• 31:15
  the limits of that usage, so that you have an understanding, okay, hey,
• 31:18
  this agent is designed to do this.
• 31:20
  Here's the autonomy that we wanna give it or not give it.
• 31:23
  Here's the limits of that autonomy.
• 31:25
  Here's where we need human in the loop, et cetera.
• 31:27
  And then making sure that, again, those limits and usages are, part of
• 31:33
  how the agentic system is designed.
• 31:35
  Then second, you'll go down a level to more the system level controls,
• 31:39
  and you get into place in making sure that especially the tool and data and
• 31:44
  system access is well understood, and that there are appropriate low level
• 31:49
  controls in place to limit it to only those tools, data, and systems that
• 31:54
  you have defined that are possible.
• 31:56
  Mm. And that's where you, need to get down into the execution, and potentially
• 32:00
  even the network layer, because you're defining, hey, we don't want it to reach
• 32:03
  out to the internet, we don't want it to reach out to MCP servers, et cetera.
• 32:06
  So it multiple levels, as you can appreciate, Kevin, from
• 32:09
  again, the limits and the here's the usages, down to the platform level of
• 32:13
  here's type of access and integrations that are allowed, and then all the way
• 32:16
  down to the network level of, okay, now let's enforce that, especially for don't
• 32:21
  let it outside this network, right?
• 32:23
  if the network level is saying nope, can't get out there, then it can't do anything.
• 32:27
  Kevin: So all of those kind of- Mm … examples in the wild that we
• 32:29
  see- Wow … wouldn't be possible if the appropriate network level,
• 32:33
  uh, controls are in place around it.
• 32:35
  So again, multi-layer.
• 32:36
  Got it.
• 32:37
  All right.
• 32:37
  Well, as you say, it is an incredibly exciting time.
• 32:40
  and, I am glad to have the opportunity to talk with you about
• 32:43
  what you're doing at Model N.
• 32:44
  Dave: Thanks a lot, Dave.
• 32:45
  Yeah.
• 32:45
  Thank you so much, Kevin.
• 32:48
  This has been The Road to Accountable AI.
• 32:50
  If you like what you're hearing, please give us a good review and
• 32:53
  check out my Substack for more insights on AI accountability.
• 32:57
  Thank you for listening.
• 33:01
  Kevin: If you want to go deeper on AI governance, trust, and responsibility
• 33:05
  with me and other distinguished faculty of the world's top business
• 33:08
  school, sign up for the next cohort of Wharton's Strategies for Accountable
• 33:12
  AI online executive education program, featuring live interaction
• 33:16
  with faculty, expert interviews, and custom designed asynchronous content.
• 33:21
  Join fellow business leaders to learn valuable skills you can
• 33:24
  put to work in your organization.
• 33:27
  Visit execed.wharton.upenn.edu/acai for full details.