Latest / The Road to Accountable AI / Richa Kaul, Complyance: Asking the Right Questions

Transcript

Richa Kaul, Complyance: Asking the Right Questions · The Road to Accountable AI

• 0:09
  Kevin: Hi, I'm Kevin Warbeck, professor of Legal Studies and
• 0:11
  Business Ethics at the Wharton School of the University of Pennsylvania.
• 0:15
  For decades, I studied emerging technologies from broadband to blockchain.
• 0:19
  Today, AI is promising to transform our world, but AI needs accountability
• 0:26
  mechanisms to ensure it's developed and deployed in responsible,
• 0:29
  safe, and trustworthy ways.
• 0:32
  On this podcast, I speak
• 0:33
  Kaul: with the experts leading the charge for accountable ai.
• 0:39
  Kevin: Richard Call is CEO and founder of compliance with a y,
• 0:43
  an AI first governance risk and compliance platform for enterprises.
• 0:47
  She previously was chief strategy officer of the legal tech company, contract
• 0:50
  pod ai, and a consultant McKinsey.
• 0:53
  In our conversation, we discuss how AI fits into larger frameworks of
• 0:58
  governance, risk, and compliance.
• 1:00
  The key AI risks enterprises are facing the challenge of data leakage
• 1:03
  with generative AI questions.
• 1:05
  Companies should be asking their AI vendors.
• 1:07
  How to address the risks of AI agents and why AI governance is
• 1:11
  still a priority at firms, despite the current pullback on regulation.
• 1:18
  Richa, thank you so much for joining me on the Road to Accountable ai.
• 1:22
  Kaul: Thanks for having me, Kevin.
• 1:25
  Kevin: Your firm compliance is in the the GRC governance risk and compliance area.
• 1:30
  First, before we get to ai, what are the major issues that enterprises are
• 1:35
  dealing with broadly in that space?
• 1:38
  Kaul: Great question.
• 1:39
  I think that it's a little, uh, focused today around risk visibility.
• 1:43
  I think that there's a gap where CISOs and CCOs feel like when they're
• 1:48
  approaching the board or the executive team and talking about risk, they
• 1:52
  don't feel true control and visibility over the risk, uh, in their business.
• 1:57
  And so when they're being asked things like, where is
• 1:59
  the biggest risk coming from?
• 2:01
  They are sharing, you know, what they believe to be true, but they would like
• 2:04
  to have more control and, uh, confidence in some of the answers that they give.
• 2:08
  And I think that that's a huge part of what, you know, makes GRC
• 2:11
  such an important function today.
• 2:14
  Kevin: And where does, uh, AI fit into all of that?
• 2:17
  How much is AI changing the landscape as firms start to adopt it?
• 2:21
  Kaul: Great question.
• 2:24
  So I feel like these two little letters, AI have really changed
• 2:27
  the conversation, maybe more so than they necessarily should.
• 2:31
  So I think that when it comes to ai, I like to think about it from an internal
• 2:35
  perspective and an external perspective.
• 2:37
  And it's helpful to break it down in that way because from a GRC side of things,
• 2:41
  you're talking about governance, risk and compliance, and that means AI that you're
• 2:44
  using internally, both potentially in your product as well as for internal functions.
• 2:49
  Then you have the piece kind of external ai, and that means like AI and the
• 2:53
  vendors that you're considering, and that's a whole other section, AI and
• 2:57
  governance and risk that comes into play.
• 2:59
  So maybe it's useful if you agree to talk about both sides.
• 3:01
  Sure.
• 3:03
  So I think on the internal side, what we are seeing is that AI is
• 3:08
  changing the way that people work.
• 3:10
  And we all know this, but it's especially coming up when you know, we're talking
• 3:15
  about risk observability because right now the risk landscape is increasing so
• 3:19
  quickly and so fast that it's impossible for humans to keep up, no matter how much
• 3:24
  head count you can possibly add to a team.
• 3:26
  And AI creates essentially increased observability and therefore better
• 3:31
  risk visibility across the team.
• 3:33
  And whether that's internally built agents or through softwares like that is one
• 3:37
  of the biggest changes in how people are working and how we're thinking about it.
• 3:40
  People are of course, adding AI into their own products and then that creates
• 3:44
  kind of the question around AI security and AI privacy in the products that
• 3:48
  the company themselves are putting out.
• 3:50
  That creates, you know, things like ISO 40 2001 or NIST RMF and different standards
• 3:55
  that you now have to comply with on top of what you were already doing before.
• 4:00
  In terms of the external piece, I think this is what's really interesting.
• 4:03
  The third party risk management or TPRM processes are kind of.
• 4:08
  Rapidly changing right now based on AI in your external vendors.
• 4:12
  And this is where I feel the fluency around AI is lacking just a little
• 4:16
  bit and needs to be improved quickly to keep pace with AI innovation.
• 4:20
  Because I've found that governance committees, you know, and a lot
• 4:23
  of these AI governance committees are kind of popping up now.
• 4:26
  They're not necessarily asking exactly the right questions.
• 4:29
  I think fluency needs a little bit of time to catch up, uh,
• 4:32
  with the innovation of ai.
• 4:33
  So long answer, but happy to dig into any of those pieces.
• 4:37
  Kevin: Great.
• 4:37
  No, there's a lot there.
• 4:38
  So why don't we split it as you say, and we'll start on the
• 4:40
  internal side and, and go to the, those external third party issues.
• 4:43
  Um, there were two elements in, in which you talked about internally there's,
• 4:47
  you know, AI as improving observability.
• 4:50
  Kaul: Mm-hmm.
• 4:50
  Kevin: But AI also increases the, the risk landscape, uh, as these
• 4:54
  new AI tools are being deployed.
• 4:55
  There are all these.
• 4:56
  These risks that, that come about for ai.
• 4:58
  So how do firms balance that?
• 5:01
  How do they take advantage of AI to address risks while also
• 5:05
  addressing a new set of risks?
• 5:08
  Kaul: Well, of course, there's so many applications of AI that we could talk
• 5:11
  about when it comes to increasing risk, observability, everything from, you
• 5:15
  know, risk, uh, sorry, ai, AI-based like risk monitoring, AI-based cybersecurity
• 5:19
  monitoring, AI-based GRC tooling.
• 5:21
  I think that one of the important.
• 5:24
  Areas that companies to companies need to invest in on the internal
• 5:28
  AI is actually training their teams on what is the security and
• 5:33
  privacy best practices of using ai.
• 5:36
  So I say this because there's all these memes going around of
• 5:40
  people putting in their companies for proprietary information into
• 5:43
  chat, GPT or Gemini, or you know.
• 5:47
  Or Claude, and it's funny because I think that's actually happening.
• 5:53
  You have, you have this happening every day, and it is.
• 5:55
  It is actually a security issue and people are not seeming to be training
• 6:00
  on this yet, and they need to do that because people are using these tools
• 6:05
  to, you know, make their own jobs more efficient or get thoughts on an email
• 6:08
  they're writing or so on, and they're uploading a lot of stuff in there.
• 6:12
  And I think that that needs to be addressed pretty, pretty soon.
• 6:15
  Yeah.
• 6:16
  Whereas on the engineering side, I'm seeing much more fluency.
• 6:19
  Kevin: Right.
• 6:19
  Because that the, the issue of people putting proprietary
• 6:22
  information into a query.
• 6:24
  It's potentially a big problem, but as you say, it's a training problem.
• 6:27
  It's a controls problem.
• 6:29
  If you've got an enterprise license, there are restrictions there.
• 6:32
  It's, it's of course, same as as any other tool.
• 6:34
  It's not about some new technical risk that the AI system is going to
• 6:38
  fail in some, uh, unanticipated way.
• 6:41
  Kaul: Yeah, I think so.
• 6:42
  But I think that a lot of these mid-market organizations and smaller
• 6:45
  enterprises, they don't yet have the enterprise licenses for LLMs.
• 6:49
  And so we're seeing people using free tooling, uh, because it's available.
• 6:53
  They're using a personal, you know, chat GPT subscription or something.
• 6:57
  And so I think that for the largest enterprises, actually this problem
• 7:00
  almost reduces because they've already taken the steps to have, you know,
• 7:04
  this is just based on what I'm seeing.
• 7:05
  Sure.
• 7:06
  But that's really what I'm seeing.
• 7:07
  And so mid-market companies and smaller enterprises aren't necessarily there yet.
• 7:11
  And so you have these folks just using what tools they have available
• 7:15
  to them online and then sharing a heck of a lot more information than I
• 7:18
  think they should really be sharing.
• 7:21
  Kevin: Is the answer to that, just better training or is there
• 7:24
  more that can be done to prevent employees from taking those actions?
• 7:29
  Kaul: Definitely training.
• 7:30
  That is a baseline need.
• 7:31
  I just, for example, internally with our team, you know, we're very
• 7:34
  particular about saying, Hey, you know, you've gotta be careful about
• 7:38
  what you're putting out into LLMs.
• 7:40
  We also obviously.
• 7:41
  Have bought, uh, bought subscriptions for our team.
• 7:43
  So I think that's kind of the next step is arming your team with the right tooling
• 7:47
  that they need and doing so quickly.
• 7:49
  I think there's some organizations right now that are figuring out and
• 7:52
  taking the time to figure out, um, their, you know, AI governance, but.
• 7:57
  Things are moving quick and people are reacting to the tools
• 8:00
  that are available to them.
• 8:01
  So you've gotta be quicker, you gotta be responsive to what's happening
• 8:04
  and set up some basic protections and some basic training and offer basic
• 8:08
  tooling so that you're not getting some of these, you know, human risk
• 8:12
  elements that we're seeing materialize.
• 8:16
  Kevin: What about the, the technical risks, things like hallucinations and, and
• 8:19
  bias and these other problems that we know are there with some of these AI tools?
• 8:23
  Kaul: Yeah, so I think of hallucinations being more of a risk when you're
• 8:26
  actually depending on AI for critical, uh, functions and critical work.
• 8:32
  I will say hallucinations is all have obviously dropped.
• 8:34
  I think that LLMs are doing a good job managing a, uh, reduction
• 8:38
  in hallucination rate overall.
• 8:39
  Of course, there are still issues with it.
• 8:42
  I think that it's about the.
• 8:45
  If, if you're using AI for critical functions in your business and you're
• 8:49
  buying external tooling with AI involved, I think it's about answering
• 8:53
  or asking the right questions of the vendor to make sure that they are
• 8:57
  doing checks to reduce hallucinations.
• 8:59
  So for example, for.
• 9:01
  I think that's a really good question to ask at an AI governance committee meeting.
• 9:05
  And for example, on our side, the way we answer that is around using domain
• 9:09
  training and domain expertise built into the agents, which actually significantly
• 9:13
  reduces hallucination, especially when paired with tighter context
• 9:17
  windows and tighter outputs as well.
• 9:20
  So it's not so open-ended.
• 9:22
  Uh, and, and.
• 9:23
  Does that make sense, Kevin?
• 9:24
  How, how that helps to reduce the hallucination?
• 9:26
  Kevin: Yeah, absolutely.
• 9:26
  Why don't we take, take a step back and unpack, 'cause I've been just
• 9:28
  asking you in general terms, but, but now you're getting into the, the
• 9:31
  agents that you use for, uh, for GRC.
• 9:34
  How does that work and, and how are you able to do something like putting
• 9:37
  that domain training into the agency?
• 9:39
  Kaul: Yeah, so we have a number of ways that we actually do that.
• 9:42
  One is through, you know, actually putting a layer of our own proprietary
• 9:47
  knowledge base on top of the LLM, which basically allows this, uh, sea
• 9:52
  of content that our agents benefit from specific to our platform for our clients.
• 9:58
  Now, we do not use client data to create that proprietary knowledge base.
• 10:02
  We actually use a kind of number of sources on our side, including working
• 10:06
  with auditors to use their knowledge base.
• 10:08
  And thinking about, uh, a very, you know, targeted domain, trained set
• 10:13
  of information that we're getting from experts, which is awesome.
• 10:17
  I think that when it comes to, again, thinking about.
• 10:21
  Um, agents and how do you make them secure?
• 10:23
  How do you make them private?
• 10:24
  There's so much happening every day right now, and it's about the commitment
• 10:28
  of the company to stay on top of some of those changes and make sure that
• 10:32
  they're actually meeting the moment.
• 10:34
  And a lot of companies are, um.
• 10:37
  Unfortunately, I think have taken some shortcuts on that.
• 10:39
  Um, and I think that it's, it's hard to know what a shortcut is, but that's where
• 10:45
  we've gotta see kind of ex improvements in these AI governance committees.
• 10:50
  Fluency in what they ask their vendors about.
• 10:54
  Kevin: Are the questions well defined enough?
• 10:57
  As you mentioned, there are things like the ISO 40 2001 standard NIST Risk I
• 11:02
  Risk Management framework and so forth.
• 11:04
  There are these toolkits out there.
• 11:05
  Is it just a matter of taking those best practices and applying them,
• 11:09
  or do we still have more work to go in terms of, uh, enterprises
• 11:12
  knowing what they should be asking?
• 11:15
  Kaul: I think we still have some work to go.
• 11:16
  I mean, I, I know you can appreciate it.
• 11:18
  I think all consumers right now can appreciate it.
• 11:21
  The speed of AI innovation is happening faster than anything that we have seen.
• 11:26
  And even as a techie, it's hard to keep on top of it.
• 11:30
  So it's really hard if it's not your day job to be keeping on top of it and you're
• 11:35
  trying to do it, you know, outside of audit prep and risk management and all
• 11:39
  these other things that you have to do.
• 11:41
  And I do think that.
• 11:44
  Right now there is a gap in the, uh, fluency of AI governance
• 11:49
  committees in general, and you know.
• 11:52
  As, and I said 40 2001, and what vendors are thinking about.
• 11:56
  But here's a couple things I think we can say right off the bat.
• 11:59
  I think the first thing is that these AI governance committees really need
• 12:02
  to focus on where the data, uh, that they are feeding to the LLM is going.
• 12:08
  I think that's the first question.
• 12:09
  I think people are asking about it, but I think that first, where
• 12:11
  Kevin: it's coming from and then where it's going
• 12:13
  Kaul: first, where it's coming from, then where it's going.
• 12:14
  Absolutely.
• 12:15
  I mean, assuming that it's using.
• 12:17
  Their own data, which it should be to create a more context specific response.
• 12:21
  Mm-hmm.
• 12:22
  But where it's going to is a really important question.
• 12:24
  And I think that folks kind of, they, they ask the question and then they.
• 12:30
  Don't necess and then it's kind of have happy ears on for the answer
• 12:34
  and 'cause they wanna use the tool.
• 12:36
  You don't wanna be a blocker, right?
• 12:37
  And so you want, you wanna kind of believe that what the vendor is saying is right.
• 12:41
  But I do think it's really specific, like really important to ask.
• 12:44
  Like, what is the LLM that you're using?
• 12:46
  Where's my data going?
• 12:47
  Is it being used to train for other clients?
• 12:49
  Is it, is it anonymized?
• 12:51
  So like these are the next level questions that we are not being asked.
• 12:55
  And so I can only speak from the experience mm-hmm.
• 12:57
  Of us presenting at these committees.
• 12:59
  But if we're not being asked, I'm not sure that they're asking others.
• 13:02
  And I think that people should be.
• 13:03
  So that's kind of one, I think the other piece is around how do you
• 13:07
  ensure low ha low to no hallucination.
• 13:10
  Mm-hmm.
• 13:10
  And what you should be looking for in that answer is about, you know, um, like
• 13:15
  one we've actually done number of tests.
• 13:18
  Two, like how, what are, what are your customers saying?
• 13:20
  Three, what have you done technically to reduce the amount of hallucination?
• 13:25
  Mm-hmm.
• 13:25
  And for that, you should be looking for, you know, tighter inputs, tighter outputs.
• 13:29
  Those are the types of things like more control, uh, in the.
• 13:33
  Context as well as what the AI is supposed to produce.
• 13:36
  And then again, that creates like less hallucination because there's
• 13:39
  just less scope for movement and straying from the expected path.
• 13:44
  Uh, so just, just a couple examples of questions that I feel like we're
• 13:47
  not hearing the full extent of what we should be getting asked.
• 13:51
  Um, speaking honestly as a better.
• 13:53
  Kevin: Yeah.
• 13:53
  It's interesting because you, you're talking about organizations that have
• 13:57
  some sort of AI governance or AI risk committee already, so they've taken
• 14:01
  that step, but it sounds like the expertise isn't necessarily there.
• 14:04
  Kaul: Not yet.
• 14:05
  I, I will like folks are trying and I think that's really important.
• 14:08
  I think we're getting there.
• 14:09
  I just think it's gonna be a little bit of wild to till everyone catches up.
• 14:13
  And again, we were talking before, like I think this is
• 14:15
  what we saw also when everything moved from on-prem to the cloud.
• 14:19
  Mm-hmm.
• 14:19
  There was just a little bit of a gap where people were catching up.
• 14:23
  I think this is a unique moment because AI is moving even faster and so you're
• 14:27
  getting this, you know, huge kind of.
• 14:31
  Um, explosion of potential risk without the catch up being fully bridged yet.
• 14:37
  Mm-hmm.
• 14:37
  And that's what we're trying to do, right?
• 14:39
  So like that's like, actually I see that and I want people to know
• 14:42
  so that they know what to ask.
• 14:44
  I also think that that's what.
• 14:46
  Tooling can help you do like GRC platforms like us, and you can ask those questions.
• 14:50
  You know, we have template questionnaires about what to ask your AI vendors and so
• 14:54
  you can use the template questionnaire, then you can use AI to help review
• 14:57
  their answers and decide, you know, parse through and say, have they given
• 15:01
  a satisfactory answer, or is there risk being posed based on what they're saying?
• 15:05
  So I think you can almost automate some of these questions rather than putting it in
• 15:08
  a risk committee or something like that.
• 15:10
  But that's what we're trying to get to and share information about.
• 15:14
  Kevin: Yeah.
• 15:14
  And it gets back to, to where I started.
• 15:15
  We, you know, we, we had the move to the cloud, which everyone mm-hmm.
• 15:18
  Understood really should, should have understood in enterprises,
• 15:20
  raised a whole new set of risks and a whole new set of issues.
• 15:23
  Kaul: Mm-hmm.
• 15:23
  Kevin: Um, we also had, you know, GDPR and the rise of data protection.
• 15:28
  And so most of these organizations have structures around data
• 15:32
  management, uh, in some cases because they, they have to for regulation.
• 15:35
  Mm-hmm.
• 15:36
  With, um, ai there, there's the whole field of ML ops and there's a
• 15:40
  lot of tooling that's going to be in there with existing AI structures.
• 15:44
  Mm-hmm.
• 15:45
  Um, is it just a matter of adding on a new set of questions?
• 15:49
  Um, you talked about how AI is moving fast, but, but are there
• 15:52
  structurally different kinds of risks that are coming up now?
• 15:56
  Kaul: Certainly, I mean, anthropic was just talking a couple months ago about how
• 16:00
  it was their first breach using AI agents.
• 16:03
  Mm-hmm.
• 16:03
  But I think that it is easy to sensationalize the risks around AI when we
• 16:09
  think about them in this broad, open-ended way of saying there's so much risk.
• 16:15
  I think instead we need to break it down by category and which I was trying to do,
• 16:19
  and I think we can make it even neater.
• 16:21
  So I think it's like, okay, there's risk of having AI in your offering.
• 16:25
  There's some risk associated with that.
• 16:26
  The solutions there are doing things like, you know, aligning to something
• 16:30
  like ISO 40 2001, Misty, IRMF, doing risk assessment specific to the
• 16:34
  AI tooling, you know, being really on top of privacy and security.
• 16:37
  Best practice with your ai.
• 16:38
  That's one kind of section.
• 16:40
  There's risks about, from the internal working side, which is things like.
• 16:44
  Your team, uh, sorry, your employees, you know, putting too
• 16:47
  much information into ai mm-hmm.
• 16:49
  Or relying on AI without checking the outputs for critical
• 16:52
  functions and so on and so on.
• 16:54
  And I think there's fixes for that, which are things like training, which are things
• 16:57
  like, don't let AI be used for critical functions without human in the loop.
• 17:00
  Mm-hmm.
• 17:00
  Like all of these workflow process automation and kind of like, um,
• 17:06
  process steps that you can build in.
• 17:07
  So I think there's that kind of bucket of risks.
• 17:10
  There's the bucket of risks of the AI in your vendors' products that you use
• 17:14
  every day, and how do you make sure that your data's not going, places
• 17:17
  you don't know, and all of that.
• 17:19
  Then there's the fourth bucket, which is AI being used for kind of,
• 17:24
  you know, bad act by bad actors.
• 17:26
  Misuse,
• 17:26
  Kevin: right, yeah.
• 17:27
  Kaul: Misused by bad actors to attack you and having, you know, the power
• 17:31
  of the ai, um, kind of, you know, trying to get into, into your, mm-hmm.
• 17:35
  System, and that is obviously, I think the scariest one out of that group, but
• 17:40
  it's also the one that we have to go back and rely on our security practices.
• 17:45
  And it's not, it's not shockingly different from what you need to do to
• 17:50
  be SOC two compliant or to live SOC two compliance, actually, I should say.
• 17:53
  And it's not radically different from some of the measures that we're already taking.
• 17:58
  We just have to be vigilant.
• 17:59
  We have to be prepared like no one does.
• 18:03
  So they do, but no one loves to do your business.
• 18:07
  Disaster recovery tests every year.
• 18:09
  You don't it.
• 18:09
  Mm-hmm.
• 18:10
  You know what I, I say in this world, make it quarterly.
• 18:12
  Like those are the types of things you need to be able to respond quickly.
• 18:15
  Mm-hmm.
• 18:15
  It can't be your first, it can't be your first rodeo when this happens.
• 18:19
  You know?
• 18:19
  You gotta know how to respond on how to react.
• 18:21
  So I guess.
• 18:23
  I don't wanna sensationalize the restaurant ai.
• 18:25
  I wanna be practical.
• 18:26
  Mm-hmm.
• 18:26
  And say these are the buckets of risk that AI, you know, creates for us.
• 18:30
  And what can we tactically do to address each bucket?
• 18:33
  And that's how I think that we move forward here and we lean into what
• 18:36
  is a technological innovation that's gonna move forward with or without us.
• 18:42
  Kevin: So most organizations started with, with principles and
• 18:46
  general frameworks and mm-hmm.
• 18:47
  Presumably at some point, uh, this becomes really about, uh, processes
• 18:53
  and audits and, and things like that.
• 18:56
  As, as we have in these other areas.
• 18:57
  How far are we on that journey?
• 19:02
  Kaul: Good question.
• 19:03
  Kevin: And obviously I'm asking you in, in the abstract in general,
• 19:05
  which, which there's no way to answer, but I, but I guess I'm, I'm curious
• 19:08
  what, what you're finding with, with organizations where they're getting
• 19:11
  stuck, uh, along that trajectory.
• 19:14
  Kaul: I think right now we're very early in the adoption curve, and so
• 19:17
  therefore, we're earlier right now in the auditing curve related to ai.
• 19:22
  I think that auditors are, um.
• 19:26
  They are already doing so much work and so much testing.
• 19:32
  Mm-hmm.
• 19:32
  That I think adding AI to the mix of everything they have to test for opens
• 19:36
  an entire other dimension that most folks haven't necessarily gotten to yet, unless
• 19:40
  they're of course ISO 40 2001 auditor.
• 19:42
  I think that we are.
• 19:45
  Just starting to have conversations with our enterprise clients, saying,
• 19:49
  okay, how can I make sure that the AI is complete and accurate in the
• 19:54
  evidence review that, you know, the, the compliance software is doing for us?
• 19:58
  How can I make sure that these things are, um, you know, looking
• 20:01
  at everything correctly that they're actually following my criteria?
• 20:04
  So that's great.
• 20:05
  Like we're having that conversation internally with these GRC teams who are.
• 20:09
  Barely coming to us.
• 20:10
  I'd say less than 5% of clients so far coming to us and saying, how
• 20:13
  can we now put checks on the ai?
• 20:15
  So I think it's gonna take six to 12 months before that
• 20:18
  5% becomes closer to 50%.
• 20:21
  And in my experience, it's around then with that critical mass
• 20:24
  that auditors start to pick up on it too and bring it forward.
• 20:27
  So I think we have a little bit of time.
• 20:29
  We're honest with ourselves.
• 20:30
  Cabin,
• 20:30
  Kevin: right?
• 20:31
  Well, six to 12 months of the AI world is an eternity, but
• 20:34
  Kaul: it
• 20:34
  Kevin: is.
• 20:34
  It's not that long in the real world.
• 20:37
  Kaul: It is, you know what I think will be interesting?
• 20:39
  I think that AI employees are not that far away.
• 20:42
  And I think, you know, folks feel many different ways about
• 20:45
  it, and I, I'm not commenting on it from a personal perspective.
• 20:49
  I just think that that is where we are headed in 12 to 18 months.
• 20:52
  And then I think we have a whole nother version of AI risk because
• 20:57
  now you're talking about AI making decisions for your business.
• 21:00
  Yeah.
• 21:01
  And that is, yeah.
• 21:03
  We really gotta be auditing the AI that we're using to do those types of things.
• 21:08
  Kevin: Yeah.
• 21:08
  So how, how do firms get ahead of that age agentic AI risk as you, you're
• 21:12
  right, there's a lot of talk about AI employees, but in terms of, you
• 21:16
  know, starting to deploy some of these age agentic systems where they can
• 21:21
  actuate and, and, and have effect in the world, that seems to be happening.
• 21:25
  Uh, and as you say, it does raise the risk threshold.
• 21:28
  So what do, what do we need to do now?
• 21:31
  Kaul: I'm gonna go back and say, I really like to make things tangible and
• 21:35
  solvable, and so here's what I would say.
• 21:39
  This is happening.
• 21:40
  We need to figure out how to deal with it.
• 21:43
  When I think about the most.
• 21:48
  Initial way, like the kind of like the first step we need to take.
• 21:52
  Anytime an agent is performing functions that a human used to do, let's just
• 21:56
  say, or that a human did not use to do anytime they're doing something,
• 22:00
  uh, that is an end-to-end workflow.
• 22:02
  We have to go through the simple process of mapping out literally
• 22:06
  what the agent can possibly do.
• 22:08
  All of the things the agent can possibly do from start to finish and make clear
• 22:13
  points and say, this should be a decision point where a human should be in the loop.
• 22:17
  This should be a decision point where a human should be in the
• 22:19
  loop and build in those guardrails.
• 22:21
  Literally build them in if it's your own tech.
• 22:24
  Build it in, it's a vendor's tech, demand it and say, no, the AI cannot
• 22:28
  make this decision without me.
• 22:30
  I need a human to weigh in.
• 22:32
  And by just doing that simple process, you've now said, okay, these are
• 22:36
  where, you know these, this is where the AI could then go astray outside.
• 22:40
  And before you know those points.
• 22:42
  And if the vendor says, sorry, can't do it, or if your team can't build
• 22:46
  that human in the loop, at least you know where your highest risk is.
• 22:49
  Mm-hmm.
• 22:49
  And you can put eyes on it.
• 22:51
  I mean, I know this sounds really, really basic, but I actually
• 22:55
  think it's gonna help to reduce a significant amount of risk.
• 22:57
  'cause people are not necessarily thinking about it like that today.
• 23:01
  Kevin: Yeah.
• 23:01
  Well, and the phrase human in the loop gets thrown around a lot.
• 23:04
  But I, I find that, you know, when I, when I talk to students, even when
• 23:07
  I talk to, uh, enterprises and, and executive education clients, it's
• 23:13
  often this kind of magical thing.
• 23:14
  Either you have a human in the loop or you don't, but as, as you're alluding
• 23:17
  to what role the human plays, where it is in the humans are in the loop, um,
• 23:22
  it, it's actually a whole spectrum.
• 23:25
  Kaul: You're exactly, exactly right.
• 23:26
  Human in the loop sometimes can mean that there's a human.
• 23:30
  Doing the task before or after, like picking up from the AI
• 23:34
  or, or, um, you know, handing off to the ai and that's fine.
• 23:38
  But human in the loop should actually be about the key decision points that
• 23:42
  the AI agent is making and making sure there's human in the loop to sign off on
• 23:46
  those decisions before the AI actions.
• 23:48
  The next step, that to me is actually.
• 23:51
  AI agents working alongside humans and augmenting their work rather than
• 23:55
  handing over entire workflows to agents that make decisions behind a black box.
• 24:01
  Kevin: What about the, the pressure that so many organizations are feeling
• 24:04
  now that, you know, we need to be, uh, not behind the curve on ai.
• 24:09
  The, the board is pushing us, management is pushing us.
• 24:13
  Kaul: Mm-hmm.
• 24:13
  Kevin: Um, obviously regulated industries, some more risk averse companies, it's,
• 24:18
  it's more in their DNA, but, but I hear this a lot, um, that, that even though
• 24:22
  they're aware there are risks, uh, it's an environment where it's hard to be
• 24:26
  the one saying, no, we need to go slow.
• 24:28
  Kaul: Ugh.
• 24:29
  Yes, I totally agree.
• 24:30
  And that's where I think two things are happening.
• 24:33
  One third party risk management is becoming more important than ever
• 24:36
  because most of the times these companies who wanna adopt, you know,
• 24:39
  AI forward initiatives, which obviously I think is great in many ways, I,
• 24:43
  I think that they're relying on external vendors to, uh, basically.
• 24:49
  Give them the AI tooling and then that makes TPRM so
• 24:53
  important, so, so, so important.
• 24:55
  And that is what I obviously was talking about the beginning, where
• 24:59
  I said, you know, then you've gotta ask the right questions.
• 25:01
  Mm-hmm.
• 25:01
  And I think that the advice that I would give folks who are more risk averse
• 25:06
  is really get to know your vendors.
• 25:08
  Ask the right question.
• 25:09
  Figure out which vendors you can trust, figure out which vendors actually have ai.
• 25:14
  I wish I didn't have to say that, but right now I feel like people are just
• 25:17
  saying everything is AI and it's not.
• 25:20
  And so I think it's all about getting to know your vendors and I think
• 25:24
  that there's something to be said for doing that in a more automated way.
• 25:27
  Again, that's one of the things we do, not only template questionnaires, but
• 25:31
  using AI to review the vendor's answers based on domain expertise to parse out.
• 25:36
  Mm-hmm.
• 25:36
  What is, what is actually, um.
• 25:39
  Meeting the bar of a really strong low risk AI vendor and what is not.
• 25:43
  But if you're not doing it with tech, fine, but you gotta do it.
• 25:46
  That is what I think is the crux of the next 12 months.
• 25:51
  Kevin: And how do you avoid the same risks that we're talking about with the AI tools
• 25:56
  that you're using for these GRC functions?
• 25:59
  Kaul: The AI tools that we built, we are, we are.
• 26:03
  Very particular, we need to eat our own dog food or compliance tech company.
• 26:08
  We've gotta make sure that we are on the front lines.
• 26:10
  And so we are constantly, constantly researching and thinking about
• 26:14
  all of the risks that are posed by different, you know, AI technologies
• 26:17
  and how do we build in corrections.
• 26:19
  And we have, uh, two entire, um, I feel like.
• 26:24
  I don't even know, like, don't wanna use the word dictionary,
• 26:26
  but it feels like books pretty much on this internally mm-hmm.
• 26:29
  That we use.
• 26:30
  Uh, and you know, the engineering team has to live by and our
• 26:33
  customer team has to live by.
• 26:34
  And so we've really brought it onto a science.
• 26:37
  But I think when it comes to us assessing our vendors, we go back and use like our
• 26:42
  own criteria, the ones that are available in our platform and think about, you know.
• 26:46
  What are the questions that we should be asking them and asking those
• 26:49
  questions, seeing how they respond, understanding their risk profile,
• 26:52
  and only then making a decision about bringing on them as a vendor.
• 26:57
  Kevin: We're, we're in a period now where, uh, there's been a somewhat of
• 27:00
  a pullback in regulation, especially in the us but even in, in Europe,
• 27:05
  you hear, you hear discussions about delaying the, the high risk, uh,
• 27:08
  elements of the AI Act and so forth.
• 27:10
  Mm-hmm.
• 27:11
  Um, you, how are the, the companies you're dealing with.
• 27:14
  Thinking about that, uh, because you know, of course the risks are still there.
• 27:18
  Uh, and I always say, you know, you shouldn't be doing this just because
• 27:21
  regulators are telling you to, you should be doing this is the right thing to do.
• 27:24
  But, but I'm curious, since you're actually out there talking to
• 27:27
  so many companies, how are they perceiving this environment?
• 27:31
  Kaul: One of my favorite things about working with security leaders is that they
• 27:34
  don't necessarily wait for the regulation.
• 27:36
  Hmm.
• 27:36
  I think that if they did, we wouldn't have ISO 27,001 and SOC two being so prevalent.
• 27:42
  Obviously Fed ran CMMC are federally mandated, but at the end of the day,
• 27:46
  a lot of the standards that security teams live by and look for are things
• 27:51
  that are actually market expectations set, not even federally regulated.
• 27:54
  And I think that actually makes for a really beautiful ecosystem
• 27:58
  in many ways, because it's.
• 27:59
  Based on what people think of for their risk.
• 28:02
  And so I'm having conversations with security leaders every day talking
• 28:07
  about AI security and AI governance, regardless of what is happening
• 28:11
  from a regulation perspective.
• 28:13
  Because they care about two things, protecting their business and
• 28:16
  protecting their business's customers.
• 28:19
  And that doesn't wait for regulation.
• 28:21
  That is just the here and now.
• 28:25
  Kevin: How did you get into this space and, and how does that
• 28:28
  influence the way you and your company think about this work?
• 28:33
  Kaul: Very honestly, I am a huge data privacy nut.
• 28:36
  Mm-hmm.
• 28:36
  In my personal life.
• 28:37
  I really don't like when governments or businesses have more of my
• 28:41
  personal data than they need.
• 28:43
  And at some point I realized that there's only so many privacy
• 28:47
  configurations I can toggle on and off.
• 28:49
  And the best way to protect consumer data.
• 28:53
  Is to help secure the companies who hold the world's data,
• 28:56
  and that guides us a lot.
• 28:59
  That is the why behind compliance with a why in our name.
• 29:03
  I think it helps keep us focused on the mission.
• 29:05
  This is not a, you know, we're not just trying to sell software to enterprises.
• 29:09
  We were trying to help support enterprises and their GRC team's missions to
• 29:14
  keep consumer data safe and we align to that as much as we possibly can.
• 29:19
  Kevin: Yeah, it's interesting because I, I, I talk to, again, I teach a lot
• 29:22
  of students and they, you know, there's great concerns about digital platforms
• 29:26
  and their use of data and, and so forth.
• 29:28
  Um, and, and then we get into these issues and, you know, and, and you say to a
• 29:32
  company, well, you might think naively, we just want to scrape all the data we can.
• 29:36
  But then you're creating this massive attack surface in this environment, um,
• 29:41
  where, you know, everything is, is at risk if you're not being careful about it.
• 29:44
  So it it, it's good to hear if you're finding that
• 29:46
  resonating with organizations.
• 29:49
  Kaul: I think that we are also, I think it's inherent in who, in what we project,
• 29:55
  and therefore the people with that type of mindset actually find their way to us.
• 29:59
  And that is something that's really nice.
• 30:01
  So compliance with the Y, it's, it appeals to people who wanna
• 30:05
  do, uh, GRC in a more custom way.
• 30:07
  We're super configurable, very customizable, and we really
• 30:11
  focus on making, for example.
• 30:14
  Controls checks very specific to your organization.
• 30:16
  Mm-hmm.
• 30:17
  We focus on making integration specific to your organization, customizing
• 30:20
  the AI agent with your specific organization's risk thresholds.
• 30:24
  Mm-hmm.
• 30:24
  And so I think we end up a little bit of confirmation bias, finding
• 30:29
  the people who also value that.
• 30:33
  Kevin: So last thing, going forward, as you say, everything's
• 30:35
  moving very fast in this space.
• 30:37
  There are many vendors out there, both in GRC more broadly, AI governance,
• 30:43
  different pieces of it and so forth.
• 30:45
  How do you see this space evolving and and what do you think is going
• 30:48
  to differentiate the companies that are most successful?
• 30:51
  Kaul: Yeah, I think that the space is evolving towards, uh, agentic AI being
• 30:57
  embedded in the platform, and I think that's harder than it sounds to actually
• 31:01
  implement because your infrastructure has to be AI native and be able to.
• 31:07
  Actually embed AI work.
• 31:09
  I think a lot of older legacy tooling unfortunately, does have more AI
• 31:14
  wrappers or kind of add-on features.
• 31:17
  And the difficulty with that is that the impact the AI can make
• 31:20
  is actually limited when it is not well connected into the system.
• 31:24
  Uh, also, frankly, there's a lot less control that the enterprise
• 31:28
  can have over the AI when it's kind of a slap on chat bot.
• 31:32
  Um, it's a little counterintuitive, but it's just the difference between
• 31:35
  having AI embedded in the workflow where there's more control and
• 31:38
  customization versus kind of add-on.
• 31:41
  And so I think the winners right now will be folks who actually, um, mm-hmm.
• 31:45
  Use AI to create real value, and I think that one of the enablers of that is the
• 31:50
  AI native abilities of the tech itself.
• 31:56
  Kevin: Great.
• 31:56
  Richard, thank you so much for this conversation.
• 31:58
  Kaul: Thanks so much, Kevin.
• 32:01
  Kevin: This has been the Road to accountable ai.
• 32:04
  If you like what you're hearing, please give us a good review and
• 32:06
  check out my substack for more insights on AI accountability.
• 32:10
  Thank you for listening.
• 32:14
  If you want to go deeper on AI governance, trust and responsibility with me and
• 32:19
  other distinguished faculty of the world's top business school, sign up for the
• 32:23
  next cohort of Wharton's Strategies for Accountable AI Online executive education
• 32:28
  program, featuring live interaction with faculty expert interviews and
• 32:32
  custom designed asynchronous content.
• 32:34
  Join fellow business leaders to learn valuable skills you can
• 32:37
  put to work in your organization.
• 32:40
  Visit exec ed.warden.upen.edu/acai for full details.
• 32:47
  I hope to see you there.