Latest / The Road to Accountable AI / Var Shankar: AI Governance for Smaller Organizations

Transcript

Var Shankar: AI Governance for Smaller Organizations · The Road to Accountable AI

• 0:09
  Werbach: Hi, I'm Kevin Werbach, Professor of Legal Studies and
• 0:11
  Business Ethics at the Wharton School of the University of Pennsylvania.
• 0:15
  For decades, I studied emerging technologies from broadband to blockchain.
• 0:19
  Today, AI is promising to transform our world, but AI needs accountability
• 0:26
  mechanisms to ensure it's developed and deployed in responsible,
• 0:29
  safe, and trustworthy ways.
• 0:32
  On this podcast, I speak
• 0:33
  with the experts leading the charge for accountable ai.
• 0:40
  Most AI governance guidance is designed for large global enterprises,
• 0:45
  but that leaves a significant gap.
• 0:47
  Var Shankar, executive director of the Council on AI Governance
• 0:50
  is working to fill that gap.
• 0:52
  He previously led the Responsible AI Institute and worked at
• 0:55
  the Reg Tech startup enzyme.
• 0:57
  In addition to practicing law at Grava, Swain and Moore and working
• 1:01
  at the Clinton Global Initiative, we talk about why smaller organizations.
• 1:06
  Actually need AI governance and what they need, why workforce AI literacy is
• 1:11
  so important, and AI governance lessons from the healthcare and financial
• 1:15
  services sectors, as well as other important points in this conversation.
• 1:20
  Our pleasure to talk with you.
• 1:22
  Welcome to the Road to Accountable ai.
• 1:25
  Shankar (2): Hey, Kevin, pleasure to be with you.
• 1:26
  Thanks for having me.
• 1:28
  Werbach: Tell us a little bit about your path to working in AI governance.
• 1:33
  Shankar (2): Yeah, so I'm a lawyer by background, and my
• 1:35
  interest has really been in using AI to improve service delivery.
• 1:40
  So that is both in terms of the substance of the service being
• 1:43
  delivered, for example, better healthcare, but also in terms of form.
• 1:47
  So providing services in the right language or in a more accessible way.
• 1:52
  And so that's kind of how I got started in Responsible Tech.
• 1:55
  I worked on these issues for, uh, the government of British Columbia.
• 1:59
  Both digital government and COVID response and you know, AI
• 2:03
  started coming up more and more.
• 2:04
  I started looking for clear guidance on, on responsible AI use, and found that
• 2:10
  there was very little joined and kind of went, eventually went on to lead the
• 2:13
  responsible AI Institute with the support of other large member organizations
• 2:19
  in healthcare, pharmaceuticals, financial services, energy, and others.
• 2:24
  Was putting out really clear guidance on what good looks like for AI governance.
• 2:28
  And working on certification programs.
• 2:30
  Then went on to work on it from the startup side, joined Enzy, which is
• 2:34
  a RegTech AI compliance company, and we were kind of tackling exactly how
• 2:39
  organizations can work across functions to both drive responsible AI outcomes,
• 2:44
  but also to put together the documentation that demonstrates responsible ai.
• 2:49
  And then more recently joined the council on AI governance.
• 2:53
  Where we've been more focused on providing open resources for AI
• 2:57
  practitioners within organizations.
• 3:00
  Yeah.
• 3:01
  And so I think that, so the council is the brainchild of my colleague and,
• 3:05
  and kind of close collaborator, Alexis Cook, who was previously an AI engineer
• 3:10
  and developer advocate at Google.
• 3:12
  And her view, which I share, is that there is.
• 3:15
  A gap in clear guidance for most organizations because the kinds of
• 3:20
  AI standards we're seeing are best suited for large, sophisticated,
• 3:24
  kind of multifunctional, you know, global organizations.
• 3:28
  And about half of American workers are employed by organizations that
• 3:33
  have fewer than 500 employees.
• 3:35
  And so our effort is really to provide those organizations
• 3:38
  and practitioners with.
• 3:40
  Open current and, and pragmatic guidance that's aligned, you know, doesn't take
• 3:45
  any single perspective, just kind of brings together the best practices.
• 3:48
  Mm-hmm.
• 3:48
  For example, you know, this week we're releasing an intellectual
• 3:52
  property primer for people using LLMs.
• 3:54
  Just kind of, you know, a few pages on, on what good do's and don'ts
• 3:58
  would be at a smaller organization.
• 4:00
  And then we also have our AI governance playbook, which provides
• 4:04
  kind of a mental model, uh, so that that's the gap that we fill.
• 4:08
  Yeah.
• 4:08
  And other than being shorter and simpler, uh, is there anything different
• 4:12
  about how small to medium enterprises should be looking at AI governance
• 4:17
  compared to the major big firms?
• 4:20
  Yeah, a couple of things.
• 4:21
  They need to be much better about asking vendors the right questions because
• 4:25
  typically, you know, they're not building.
• 4:27
  Mm-hmm.
• 4:27
  Uh, they're buying almost all of their ai.
• 4:29
  And then also, you know, at smaller organizations, a few people do a lot.
• 4:34
  So the roles and responsibilities, sometimes you just don't have, uh,
• 4:39
  either you don't have the right expertise or you're just not clear on where
• 4:42
  those lines are between, you know, expectations of who's gonna be doing what.
• 4:46
  And so those are two of the main areas that come up frequently
• 4:49
  at smaller organizations.
• 4:51
  What are you hearing from the, the organization that you're talking to,
• 4:55
  either in terms of just generally how interested they are in AI governance
• 4:58
  or what are the, the big problems that they're highlighting for you?
• 5:02
  Yeah, so I think there is a lot of interest in this now.
• 5:06
  I, I think, you know, people at all kinds of organizations are asking
• 5:09
  about how to run a program and, you know, it's, it's, there's a lot of
• 5:14
  kind of typical questions around how do we inventory what we already have?
• 5:19
  So for example.
• 5:20
  If we are using some sort of platform that uses agents within the platform, is that
• 5:25
  part of a master inventory or is it do, do we just log that we're using this system?
• 5:30
  You know, those sorts of really concrete questions around a question
• 5:33
  around issues like inventories.
• 5:35
  Then already mentioned, you know, how, who owns governance internally?
• 5:40
  Mm-hmm.
• 5:40
  How do we ensure that nothing is slipping through the cracks?
• 5:43
  It's really important, especially with regulatory reporting as well as kind of.
• 5:48
  Customer expectations coming into place.
• 5:51
  And then procurement of course is super hot mapping, existing privacy risk
• 5:57
  model, risk management requirements.
• 6:00
  So I, I'd say generally, you know, there, there's.
• 6:03
  Compared to a few years ago when I used to speak with organizations, we
• 6:07
  used to have to make the case for why you should care about AI governance.
• 6:11
  I think that case has been made, you know, for a variety
• 6:14
  of reasons, and now it's more,
• 6:16
  yeah.
• 6:17
  What is it that, that you find is most pushing those organizations
• 6:20
  to feel the need for AI governance?
• 6:22
  Is it regulation or perception about coming, regulation, customer
• 6:26
  trust, what is it that that's primarily on their mind now?
• 6:30
  I think it's a few things.
• 6:32
  So I think that there's str, there's pressure, board pressure
• 6:36
  and CEO pressure frequently.
• 6:38
  On making sure that you're using emerging technology to drive value or to cut costs.
• 6:44
  So I think that that's a big one.
• 6:46
  Definitely risk and compliance, you know, the regulatory stick.
• 6:50
  A fear of missing out.
• 6:51
  I think you see a lot of other companies adopting AI are talking about how they're
• 6:55
  adopting ai, and so I think in every industry and in companies of all sizes.
• 7:02
  There's a little bit of, you know, we, we should be using this regard.
• 7:05
  We should at least be experimenting with this to see where it goes.
• 7:09
  And I find that we frequently have to kind of say, okay, just because it's out there
• 7:13
  doesn't mean you need to be using it.
• 7:14
  You need to kind of go, go slow, go methodically.
• 7:17
  It has, of course, significant potential, but I do think that there
• 7:21
  is a little bit of anxiety around, uh, competitive pressure driving it as well.
• 7:26
  Those seem to be though reasons why companies are investing in ai, but
• 7:30
  that would seem to suggest that, that they might be disregarding
• 7:33
  the need for governance and, and responsibility and and risk management.
• 7:38
  So where does that pressure come from?
• 7:39
  So I think it's.
• 7:41
  A need for quality and predictability because especially with generative ai,
• 7:47
  it's, I think people correctly don't trust, you know, you don't want to give
• 7:52
  an AI agent, for example, browser access or tool access or, or a lot of memory.
• 7:58
  And without kind of understanding how you're kind of, without
• 8:03
  having visibility into it, without having, being able to control it.
• 8:06
  And so I think organizations see that.
• 8:09
  Another thing that's driving it is that just shadow AI use.
• 8:12
  So the concern that, okay, we could just say, you know, all this things
• 8:16
  I just mentioned, you could just say we don't allow AI use in our company.
• 8:20
  Your employees are probably going to be using it anyway, and they're
• 8:24
  probably gonna be using it in ways that you don't want them to be using it.
• 8:27
  And so I think that that's also, in addition to the regulatory push, really
• 8:31
  driving a need for responsible ai.
• 8:35
  Let me dive in a little bit more concretely to some of
• 8:37
  the, the work that you've done.
• 8:39
  So the, the Council's AI governance playbook talks about four focus areas,
• 8:44
  and correct me if I, if I get it wrong, but strategy, risk, and compliance.
• 8:49
  Workforce and literacy and then procurement and
• 8:52
  management across those areas.
• 8:55
  Where do you find there's the biggest gap?
• 8:58
  That organizations are not investing enough?
• 9:03
  Yeah, so, so my view is that that would be.
• 9:07
  Literacy and workforce.
• 9:08
  We already talked about how strategy is, is getting attention from CEOs.
• 9:13
  Risk is getting attention because of the regulatory stick.
• 9:16
  And then, you know, procurement gets attention usually when something's
• 9:19
  not working the way it should.
• 9:20
  Literacy is often kind of a longer term play.
• 9:23
  It's more like more vitamins than painkillers, so it tends to fall
• 9:27
  through the cracks a little bit.
• 9:28
  It's, you know, organizations say, okay, we'll add this to our training
• 9:31
  module and we'll go from there.
• 9:34
  And.
• 9:36
  You can't really scale an AI governance program without literacy.
• 9:40
  It's, it's really kind of a, mm-hmm.
• 9:41
  A, a key piece.
• 9:42
  You have pretty good external resources now, whether, you know, Udemy, Coursera,
• 9:47
  universities, and then AI labs themselves kind of put out pretty good content.
• 9:52
  But what you really need is to get everyone in your organization thinking
• 9:57
  critically around what the opportunities are, and then also what the risks are.
• 10:01
  Mm-hmm.
• 10:02
  Uh, and you can do that.
• 10:03
  If you can learn by doing so, you need to involve them in pilots.
• 10:07
  You need to involve them in documenting what they've learned.
• 10:11
  And people are already very busy, very stretched.
• 10:15
  Mm-hmm.
• 10:15
  Uh, it's kind of an uncertain economy, so it's really difficult to resource
• 10:18
  this kind of big AI literacy, change management push at most organizations.
• 10:23
  So we often see that as, as kind of the, the, the missing piece.
• 10:28
  So let's talk a little more about the educational side of it.
• 10:31
  You ha have actually been teaching AI governance at Purdue, and so I'm curious
• 10:37
  what your experience has been with the students in terms of, you know, what
• 10:40
  they struggle with, what they get wrong or what they potentially get right.
• 10:44
  Yeah, and I'd be curious to how this maps to yours.
• 10:47
  So I've, as you mentioned, you know, I've been helping with the Grail lab at
• 10:51
  Purdue, building out their AI auditing coursework at the master's level.
• 10:55
  And then I've also worked with students in, in various capacities
• 10:57
  at, at other masters in law programs.
• 11:00
  Uh, and generally I feel that they have.
• 11:03
  Positive visions and are optimistic about what AI can achieve, which isn't
• 11:08
  always the case, you know, at at large organizations with people that are
• 11:12
  much further along in their careers.
• 11:14
  Students also, I feel, tend to understand the companies and models really well.
• 11:18
  Like they, they tinker far more than I expected that they would, and
• 11:23
  they also have a good grasp of the pitfalls at for a given use case.
• 11:29
  You know, if you're brainstorming, for example, in a given use case
• 11:32
  fit, is this fit for purpose?
• 11:34
  How do we understand what it's doing?
• 11:35
  Could it be biased?
• 11:37
  Those sorts of questions seem to come naturally.
• 11:39
  On the flip side, I think.
• 11:42
  The areas where they tend to learn a lot and grow a lot is organizational
• 11:46
  governance, uh, and then also networks and networks of people.
• 11:51
  I think governance is organizational.
• 11:53
  You know, we're asking questions of who approves what, what gets
• 11:56
  inventoried, how do you assess a vendor?
• 11:59
  So even if you have good instincts at the use case, you still need to learn.
• 12:02
  All of the kind of governance tools that you would use at a big organization.
• 12:07
  And then also there is a lot of that organizational governance
• 12:11
  is interpersonal skills.
• 12:12
  You know, how are you talking to people in different functions?
• 12:14
  How are you kind of bringing them on board with your ideas?
• 12:17
  And then when I see networks, I mean that students particularly.
• 12:23
  Early in their careers tend to underestimate, you know, the extent
• 12:27
  to which developing long-term trusted networks and relationships is important.
• 12:32
  Mm-hmm.
• 12:32
  Not just generally, but also for your practice of AI governance.
• 12:36
  So, you know, who are you gonna call to bounce an idea or an issue off of?
• 12:40
  Or are you a regular at a chapter meeting of, of some industry organization
• 12:45
  where you can quickly address, you know, a new issue or even just keep.
• 12:50
  Keep current with AI governance approaches.
• 12:53
  Mm-hmm.
• 12:53
  Uh, so those are the areas they, they tend to learn a lot in.
• 12:57
  Yeah.
• 12:57
  Your distinction between the, the organizational structures and, and the
• 13:01
  use case and the tools is, is interesting because some of the structures,
• 13:05
  like for example, the ISO 40 2001 standard are about management systems.
• 13:10
  So what, what exactly is the gap?
• 13:12
  If you understand those frameworks and understand the organizational
• 13:16
  piece, what, what exactly is it?
• 13:17
  This is not just for students, but for.
• 13:19
  People in companies as well.
• 13:21
  What are the more operational tools that they need to understand better?
• 13:27
  Yeah, so I think that ev, every organization has a
• 13:30
  different way of doing things.
• 13:32
  So you can't just overlay something like ISO 40 2001 onto an organization.
• 13:37
  You're going to have existing functions, existing personalities,
• 13:41
  as well as existing governance from cloud and, and kind of.
• 13:45
  Pre other IT frameworks, data governance frameworks, cybersecurity frameworks.
• 13:49
  And so there's this constant dance of making the case for why
• 13:54
  this is needed and what's new.
• 13:56
  And then also kind of making sure that the processes that you
• 13:59
  put in place are sustainable.
• 14:01
  And a lot of that, you know, as I mentioned earlier, comes down
• 14:04
  to having good relationships within the organization.
• 14:08
  Being able to communicate things to different audiences and being
• 14:12
  able to understand, you know, not everybody needs to be a computer
• 14:15
  science PhD, but being able mm-hmm.
• 14:17
  Able to communicate why this is important requires just a little
• 14:21
  bit of technical depth, so.
• 14:22
  Mm-hmm.
• 14:23
  So that, that's kind of where the organizational governance theory,
• 14:27
  where the rubber hits the road.
• 14:28
  Mm-hmm.
• 14:29
  You've done some work on looking at application of AI governance in
• 14:32
  different sectors, and in particular, I wanna ask you about healthcare.
• 14:35
  I know you, you've done some analysis and some writing.
• 14:38
  W what's different about healthcare in particular, and, and is it a,
• 14:42
  an effective or, or an appropriate or good area to do this work?
• 14:47
  Yeah, so I think healthcare has already done a great job of, because of necessity.
• 14:53
  You know, it's, it's, we trust the healthcare system, or we tend
• 14:56
  tend to at least trust components of the healthcare system.
• 14:59
  And that trust is a really valuable kind of infrastructure that you might
• 15:03
  not have in other industries, and it would make sense to kind of build on.
• 15:06
  So I think there's, there's a really specific, at least in the US, a really
• 15:10
  specific regulatory architecture.
• 15:13
  There's hipaa, there's kind of FDA, the FDA's really taken the
• 15:16
  lead on some of these AI issues as they pertain to medical devices.
• 15:20
  And then there's also state level requirements.
• 15:22
  Mm-hmm.
• 15:23
  So you already know what you're working with in terms of regulations, which isn't
• 15:26
  always the case in, in other industries.
• 15:29
  You're also, you know, these are, these are healthcare
• 15:32
  organizations are very complex.
• 15:34
  They're used to handling a wide variety of use cases with
• 15:38
  a lot of kind of variability.
• 15:40
  So, you know, you, you might have to.
• 15:42
  For example, assess third party risk in a diagnostic model, a scheduling tool, uh,
• 15:50
  or some sort of ambient scribe, you, you are, are already accustomed to looking
• 15:55
  at a number of different use cases.
• 15:57
  Mm-hmm.
• 15:57
  It's not one size fits all.
• 15:59
  Then I think that they are, they tend to be already aware of.
• 16:06
  It's not just the system, but it's where the system is used by a physician
• 16:11
  or other medical professional that really helps determine outcomes.
• 16:15
  And so again, that's already embedded in their knowledge.
• 16:19
  And then finally, they, they already have a lot of emphasis on, on
• 16:24
  post-deployment monitoring of the systems.
• 16:26
  And so there already exist a, a, a number of.
• 16:30
  Good pieces upon which you could develop.
• 16:33
  Mm-hmm.
• 16:33
  You know, and so some sort of assurance model, that's not to say of course, you
• 16:37
  know, there, there's, there are a lot of different kinds of, or healthcare
• 16:41
  organizations, they kind of govern things in different ways, but generally
• 16:46
  there are these kind of existing pieces that, that lend themselves to.
• 16:50
  Mm-hmm.
• 16:50
  And ai, AI assurance model.
• 16:53
  And also there's a desire, you know, you, you have this trust that's
• 16:56
  built up over kind of centuries to be worthy of that trust, right?
• 17:01
  Mm-hmm.
• 17:01
  So there's also a real desire to do it within these organizations.
• 17:04
  Yeah.
• 17:04
  You, you mentioned AI assurance model.
• 17:07
  What, what exactly is an AI assurance model?
• 17:09
  And, and maybe talk a little bit about what you're seeing in
• 17:12
  terms of the development of that assurance ecosystem more broadly.
• 17:17
  Yeah, so I think assurance is the connective tissue between an
• 17:22
  organization and other organizations.
• 17:25
  So it's how you would demonstrate to regulators, to customers, to the other
• 17:31
  organizations you do business with, and then to the public more generally,
• 17:36
  that your internal governance is real and that's, that it's effective.
• 17:41
  So in some cases that can be self-reported.
• 17:44
  You know, you, it might be kind of a low risk use case or you might be
• 17:49
  the type of organization that doesn't require a lot of additional governance.
• 17:52
  You're kind of relying on the vendor to do a lot of that.
• 17:55
  And then in other cases it's, you know, in higher risk use cases, it can
• 17:59
  be through an audit to have someone external come and look at your processes.
• 18:03
  So.
• 18:05
  I think to your point about the assurance ecosystem for ai, it's
• 18:11
  important to be realistic on, on what you're able to achieve.
• 18:15
  So I think at this stage it would be good to just understand.
• 18:20
  We want people to share information with each other, uh, so that
• 18:24
  they can gauge and price risk.
• 18:26
  So we're not going to be able to eliminate risk the, the way that the speed at which
• 18:29
  technology is moving, we're probably gonna have an increasing gap in risk, but.
• 18:34
  Can we at least be able to have clear information that everybody kind of
• 18:39
  agrees with and then be able to, to either accept it or at least price
• 18:44
  it and potentially transfer it.
• 18:45
  Mm-hmm.
• 18:45
  So that's kind of where AI assurance comes in.
• 18:47
  Now I wanna be really clear about, you know, it's not an alternative to
• 18:51
  other AI governance methods, so it's not, you know, regulation is really
• 18:56
  important and has kind of democratic legitimacy and regulators of course
• 19:00
  can incorporate assurance tools.
• 19:02
  But really it's kind of its own beast.
• 19:04
  And then internal governance is really where all the work is
• 19:06
  happening in your organization.
• 19:08
  So assurance is separate and you can have an assurance ecosystem
• 19:11
  regardless of what kind of legal regime or regulatory regime you have.
• 19:16
  But the reason I really like it is the tech is moving quickly.
• 19:21
  You need really specialized expertise to, to govern it.
• 19:24
  And so it seems like a lot for internal compliance teams and
• 19:28
  regulators to tackle alone.
• 19:31
  If you kind of bring researchers, Professors, standard setters, auditors,
• 19:36
  everybody on board, then you have a better chance of kind of closing that information
• 19:40
  symmetry and governing AI better.
• 19:43
  There are now a number of firms out there that will do AI audits, and
• 19:46
  you've mentioned you're teaching and master's programs that are training
• 19:49
  AI auditors and all the major audit and assurance firms, you know, have
• 19:53
  practices in this area for a company, especially the the small to medium
• 19:57
  companies that, that you're focused on.
• 20:00
  Can they easily now find someone that they can trust to
• 20:03
  do those kinds of assessments?
• 20:05
  For them,
• 20:06
  I'd say it, it really varies by industry and by what kind of
• 20:11
  assessment they're looking for.
• 20:12
  I think in regulations really do drive markets in many of these cases, so the
• 20:17
  New York City local law around automated employment decision making has a really
• 20:22
  standardized report format, and so.
• 20:24
  That's really kinda stimulated the development of a market around it.
• 20:28
  In other cases, smaller organizations don't necessarily know what to ask for.
• 20:34
  Mm-hmm.
• 20:34
  And they also to, to at risk of kind of getting in the weeds
• 20:38
  aren't clear about what exactly is being assessed or certified.
• 20:42
  Because once you take, especially if you take a foundation model or,
• 20:45
  or a kind of system from a big lab.
• 20:48
  If you make enough changes to it, it's really quite different from
• 20:51
  the model that you purchased.
• 20:53
  And so is that a separate kind of object of analysis or is it, are there elements
• 20:59
  of it that you can kind of break off?
• 21:01
  These are questions that are really difficult for smaller
• 21:03
  organizations in particular.
• 21:05
  And so I think that that it kind of re, it's more helpful for really
• 21:11
  common industry use cases where.
• 21:13
  You have the same patterns over and over again.
• 21:15
  And so a smaller organization can kind of piggyback on that and say, okay,
• 21:19
  I see, you know, I'm a small bank or small financial services organization.
• 21:24
  I've seen the kinds of audits that a big bank does, and so
• 21:28
  I'm gonna piggyback off of that.
• 21:29
  Mm-hmm.
• 21:30
  So, yeah, to answer your question, it kind of really varies.
• 21:32
  Um, but, but, but you're right.
• 21:34
  Increasingly, the ecosystem has, has, is, is developing well and has come
• 21:38
  a long way over the past few years.
• 21:40
  You made an interesting point about foundation models because a lot of
• 21:43
  our initial ideas about assurance are, you know, you build some machine
• 21:49
  learning solution for an application and we're gonna test it for bias
• 21:53
  or for accuracy or whatever else.
• 21:55
  But now increasingly, we're in a world where companies are building
• 22:00
  on top of these foundation models that they have no control over.
• 22:04
  But as you point out, they may be modifying them.
• 22:06
  So how can the assurance ecosystem develop?
• 22:10
  To help companies that are increasingly using these generative AI structures
• 22:15
  that, that are gonna look fairly different than the kind of machine
• 22:18
  learning approaches that original kinds of, uh, audit world came about around.
• 22:24
  So I think that, you know, at, at its surface, theoretically, it
• 22:27
  seems like a really good model for assurance design because, you know,
• 22:31
  the AI labs have understand a model's capabilities better than anybody.
• 22:37
  The organizations that are deploying them.
• 22:40
  Are best positioned to understand fitness for purpose, and then end
• 22:45
  users, whether they're individuals or companies, they can gauge whether
• 22:49
  they're actually getting value.
• 22:50
  And so there is an incentive and kind of need for all of these different
• 22:54
  players to share information.
• 22:56
  And you know, as you're kind of alluding to, there's also
• 22:59
  a couple of complications.
• 23:00
  So.
• 23:01
  There's only a handful of cutting edge AI labs.
• 23:03
  The deploying organizations tend to be a bit limited in what they can
• 23:07
  demand from those cutting edge labs.
• 23:09
  And then of course, the rate of change is quite significant.
• 23:13
  And so you can say things like, okay, everybody needs to share
• 23:16
  information in these specified formats.
• 23:18
  Labs need to do model testing, deployers need to be responsible
• 23:21
  for the context of deployment.
• 23:24
  And you have regimes like the EU AI Act, trying to make that very explicit, but.
• 23:30
  You know these questions of, of.
• 23:32
  What is the thing that we are trying to provide assurance for in, in, in
• 23:36
  the case of a deploying organization or even what level of third party access
• 23:41
  is, should be allowed for a cutting edge model, you know, at what point
• 23:45
  does that become a security risk?
• 23:47
  And we've tried a number of different approaches to this.
• 23:50
  We tried with AI Safety Institutes kind of peering in and doing
• 23:53
  analyses of cutting edge models.
• 23:55
  We've tried providing really structured access to a handful of researchers.
• 24:00
  Then a lot of the time we also just rely on the mm-hmm.
• 24:03
  Labs themselves to, to, to evaluate their models.
• 24:05
  And so I'm not sure that those open questions are going, we're going to
• 24:09
  see a resolution for those anytime soon, but I am optimistic that.
• 24:15
  If we can start with the most common use cases in the largest industries
• 24:19
  that give us some common patterns where you have, you know, the sector
• 24:22
  specific nuance, then you can have the economies of scale to make mm-hmm.
• 24:26
  Assurance exercise worthwhile, and then smaller players can
• 24:29
  kind of piggyback on that.
• 24:32
  Yeah.
• 24:33
  Along those lines, I asked you before about healthcare, but you also were a
• 24:36
  co-author of a paper in 2025 that proposed a certification framework in financial
• 24:42
  services and, and actually did a pilot.
• 24:44
  I'm curious what you found in that work.
• 24:50
  Yeah, so this was kind of pilot that we did with the Standards Council of Canada.
• 24:55
  To test some AI system level certification criteria with a bank
• 25:00
  that had a customer facing use case?
• 25:02
  I'd say the, probably the main takeaways were, you know, we had, we had.
• 25:07
  Really reduced the certification criteria and simplified them as much as we could.
• 25:11
  Mm-hmm.
• 25:12
  But still, you know, just, just by, by the nature of AI and the kind of
• 25:17
  differences between certifying something like a dynamic system, we found
• 25:22
  that our certification criteria were interpreted in different ways by different
• 25:26
  actors, and that we needed to kind of simplify and standardize the scheme.
• 25:30
  Even further and leave some discretion to the auditor.
• 25:35
  And this again, starts, starts to get back at why it's an ecosystem and why
• 25:40
  auditor education is so important.
• 25:42
  The other kind of surprising finding for me was that our public sector partner,
• 25:47
  the Standards Council of Canada is.
• 25:50
  One of the most forward thinking organizations on this topic, kind of
• 25:53
  anywhere in the world, but even they, at the time of the pilot didn't have the
• 25:57
  expertise available to play the standard setter role for something that dynamic.
• 26:03
  Usually, you know, the things that are being certified are, are quite static,
• 26:07
  so it really highlighted the need for more capacity in the public sector.
• 26:12
  More training, more technical expertise to be able to oversee, even
• 26:17
  oversee a, a certification program.
• 26:20
  First is, I think it's really important to, you know, in, in this AI and
• 26:26
  responsible AI and risk world, we tend to focus a lot on risk, and that's
• 26:30
  understandable, but I'd really start with.
• 26:33
  That positive vision?
• 26:34
  Mm-hmm.
• 26:35
  I think it's, it's, it helps you change your mindset to problem
• 26:39
  solving and kind of creativity.
• 26:41
  And so really trying to understand, okay, what are, what
• 26:43
  are we trying to do with ai?
• 26:44
  Is there a way that we can do this in a way that, and that's
• 26:47
  significantly less risky?
• 26:49
  So getting a sense of the strategy, getting the sense of how it's gonna
• 26:51
  help you provide value, I think is really important for any risk,
• 26:54
  risk or compliance professional.
• 26:57
  Beyond that, I'd say getting.
• 26:59
  Observing how your IT governance is working right now.
• 27:04
  So you'll already have data governance, information security,
• 27:08
  analytics, you know, risk.
• 27:10
  All of these functions you'll already have grappled with some of the major issues.
• 27:15
  We discussed it with the council's playbook, whether it's
• 27:17
  third party risk or visibility.
• 27:20
  For who's using these systems across the organization.
• 27:23
  And so just by observing how is this actually working and how
• 27:26
  can we kind of overlay some of the AI concerns on top of it.
• 27:30
  You'll also get a sense of which ecosystems you're already plugged into.
• 27:34
  You know, do you, some organizations might really closely follow FDA
• 27:38
  guidance, others, you know, if you're a government contractor, maybe you're
• 27:42
  following the Pentagon's guidance.
• 27:44
  Are you kind of.
• 27:45
  Really close to NIST or really close to iso.
• 27:47
  I think those sorts of questions can help determine like the path
• 27:51
  of re least resistance for your organization in getting really
• 27:55
  good AI governance guidance.
• 27:56
  So that's what I'd recommend.
• 27:58
  Great Var, thank you for all of your insights.
• 28:01
  Really appreciate it.
• 28:02
  Thanks for having me, Kevin.
• 28:05
  This has been the Road to Accountable ai.
• 28:07
  If you like what you're hearing, please give us a good review and
• 28:10
  check out my substack for more insights on AI accountability.
• 28:14
  Thank you for listening.
• 28:18
  If you want to go deeper on AI governance, trust and responsibility with me and
• 28:23
  other distinguished faculty of the world's top business school, sign up for the
• 28:27
  next cohort of Wharton's Strategies for Accountable AI online executive education
• 28:31
  program, featuring live interaction with faculty expert interviews and
• 28:36
  custom designed asynchronous content.
• 28:38
  Join fellow business leaders to learn valuable skills you can
• 28:41
  put to work in your organization.
• 28:44
  Visit exec ed.warden.upenn.edu/acai
• 28:50
  for full details.
• 28:51
  I hope to see you there.