Latest / The Road to Accountable AI / Emre Kazim (Holistic AI): Why AI Governance is Like Cybersecurity

Transcript

Emre Kazim (Holistic AI): Why AI Governance is Like Cybersecurity · The Road to Accountable AI

• 0:09
  Kevin: Hi, I'm Kevin Werbach, Professor of Legal Studies and
• 0:11
  Business Ethics at the Wharton School of the University of Pennsylvania.
• 0:15
  For decades, I've studied emerging technologies, from
• 0:17
  broadband to blockchain.
• 0:20
  Today, AI is promising to transform our world, but AI needs accountability,
• 0:26
  mechanisms to ensure it's developed and deployed in responsible,
• 0:29
  safe, and trustworthy ways.
• 0:32
  On this podcast, I speak with the experts leading the charge for accountable AI.
• 0:40
  Holistic AI was one of the earliest companies built specifically to
• 0:44
  govern, audit, and red team AI systems.
• 0:47
  In this episode, co-founder and CEO Emre Kazim talks about how his original
• 0:50
  thesis that AI governance would look like data governance anchored in
• 0:53
  compliance regimes has given way to something he now thinks looks more like
• 0:57
  cybersecurity, driven by incidents, technical assessments, and an arms race
• 1:02
  between AI capability and AI oversight.
• 1:05
  We also get into why human in the loop is no longer viable as the primary means
• 1:09
  of addressing AI risks, the vanguard model of enterprise AI adoption, why
• 1:14
  best practices will become the de facto standards for responsible AI, and why
• 1:18
  he believes that meaningful research capacity will be the price of entry for
• 1:22
  anyone working in the AI governance space.
• 1:26
  All right, Emre, welcome.
• 1:27
  Thank you for joining me.
• 1:29
  Fantastic.
• 1:29
  Thank you for having me, Kevin.
• 1:31
  Uh, so you co-founded Holistic AI in twenty twenty, if I'm right.
• 1:35
  That was before there was ChatGPT, before the EU AI Act.
• 1:39
  Uh, there was, you know, a lot of activity around, uh, responsible AI,
• 1:43
  but, but things have changed a lot.
• 1:45
  So what is it that, that originally got you started that, that made you
• 1:49
  see the need for this kind of service?
• 1:52
  Emre: Kevin, yeah, this-- that's actually a great question because it's been, uh,
• 1:56
  uh, the-- it's been so long now since we, uh, got going, um, that there's
• 2:01
  a... Sometimes when you look back in retrospect, you start to realize how many
• 2:05
  different kind of events precipitated certain kind of responses and other
• 2:09
  events which we kind of foresaw and act- and predicted and so on and so forth.
• 2:13
  So let me give you a bit of a primer in terms of how Holistic AI, um, spun out
• 2:18
  and what our kind of origin story is.
• 2:19
  So I'm co-founder of the company, and I co-founded the company
• 2:23
  along with Adriano Koshiyama.
• 2:25
  So Adriano and I met in the computer science department
• 2:28
  of University College London.
• 2:30
  So, uh, I'm sure your listeners are familiar with UCL.
• 2:34
  Um, it's the home of DeepMind, uh, Google's DeepMind.
• 2:38
  Um, we-- I tr- I claim Hinton as one of our Nobel Prize winners, but
• 2:42
  certainly Demis, uh, everyone was here at, was at UCL, and there was
• 2:46
  a lot of interest in foundational, uh, AI work and the generational
• 2:51
  synthetic data and so on and so forth.
• 2:53
  And in that context, there was an emerging and, and, and a very
• 2:57
  important conversation about trust Which was if we're going to use these
• 3:02
  systems or, and meaningfully harness their potential, then we have to
• 3:06
  have confidence in those systems.
• 3:08
  So, um, Adriano was doing his PhD at the time.
• 3:11
  I think Adriano started, um, in 2018 I met him, but he started before then.
• 3:15
  I started my postdoc in 2018.
• 3:17
  That's when I ended up at UCL.
• 3:19
  And the general conversation that we were having at that time was, how do we
• 3:23
  engender trust in algorithmic systems?
• 3:26
  And the use case that we were focused on was the example of a trading algorithm.
• 3:32
  Um, it was, the company was called Knight Capital.
• 3:35
  Kevin, I'm not sure if you're familiar with this use case.
• 3:38
  So it was a trading algorithm that glitched, and I think it wiped out
• 3:42
  $440 million from the business.
• 3:44
  Hmm.
• 3:45
  So it was this- Oh, yeah ... huge, yeah- Now, now I recall.
• 3:47
  Yes, yeah, yeah.
• 3:48
  It was this huge kind of use case in financial services, and there was
• 3:51
  this awareness that, okay, look at how mission-critical these AI systems are.
• 3:56
  You know, we have to have confidence that these systems actually work, and they're
• 3:59
  reliable and safe and so on and so forth.
• 4:02
  What was interesting is that at the same time, that was very much about
• 4:05
  operations, but at the same time, a social awareness was emerging around
• 4:09
  issues of bias and explainability.
• 4:12
  So we weren't sure, okay, how are we gonna go about doing this?
• 4:15
  And we thought our original thesis was that if we create a certification standard
• 4:20
  and a, an algorithm meets that standard, then it is, you know, ergo trustworthy.
• 4:27
  Now, the issue was, you know, what kind of certification standard?
• 4:30
  What, what was the standard that we were going to, uh, generate?
• 4:33
  And also, why were we in a position to forward that standard?
• 4:38
  So what we did, as academics, uh, we had the power to convene, and we
• 4:41
  brought the great and the good in the UK together, so then we had regulators,
• 4:45
  we had the civil service bodies, um, and then we had industry, and
• 4:50
  then we had ourselves as academics.
• 4:52
  And we thought, okay, what we'll do is create a certification standard that all
• 4:55
  of this cohort or group agree to, and then we'll kind of publish it as a public
• 5:00
  statement and then we can get going.
• 5:03
  And what happened pretty quickly is we realized that certification is an
• 5:06
  endpoint rather than a starting point, and the whole problem got flipped on
• 5:10
  its head, and two streams emerged.
• 5:14
  One was conversations about setting up a kind of national synthetic, uh, data
• 5:19
  center, and we were gonna basically set up something where you could test your
• 5:24
  algorithms against these data sets, and then you would know if they were biased
• 5:28
  or whatever it was be-, um, because we knew the data wasn't, so you'd be able to
• 5:33
  test the actual algorithms against that.
• 5:35
  And the second work stream was what we referred to at the time
• 5:40
  as algorithm auditing, which was saying, okay, let's just take a
• 5:43
  single system and let's say, how does one go about auditing an algorithm?
• 5:48
  And what does it even mean to audit an algorithm?
• 5:51
  So, uh, Adriano led the paper, I was second name author on this And he
• 5:55
  wrote, which I, what I consider to be the foundational text in the field,
• 5:59
  which is Towards Algorithm Auditing.
• 6:01
  And the paper, Towards Algorithm Auditing, became a really powerful framework
• 6:05
  to say, "Okay, this is how we would theoretically go about, um, assessing
• 6:10
  an algorithm." And the core thesis was there were technical variables such as
• 6:15
  explainability, privacy, uh, bias, and robustness, and robustness was a kind of,
• 6:20
  uh, umbrella term for re- reliability, reproducibility, security, and so on.
• 6:25
  And that those technical verticals, we were calling them verticals or variables,
• 6:29
  were controlled for by just like good traditional governance, documentation,
• 6:35
  compliance, taxonomies of responsibility, accountability, and so on and so forth.
• 6:40
  Um, and then that was the kind of core thesis.
• 6:42
  Mm-hmm.
• 6:43
  Now, the problem is you can't study medicine in the library, you know?
• 6:46
  Of course.
• 6:46
  You study medicine in the hospital.
• 6:48
  And, uh, Ajorn and I were like, "Okay, if we're actually gonna have impact, we've
• 6:52
  gotta do this in the real world." So we spun the company out, and we got going.
• 6:57
  And what we started to do was say, "Okay, how does one go about, in quotes,
• 7:00
  auditing an algorithm?" Um, and you know, we used this language of auditing
• 7:05
  because we wanted to evoke a kind of seriousness within the community because
• 7:11
  AI ethics and responsible AI, um, quickly became quite fluffy as terms.
• 7:17
  Um, so we were trying to make an analogy to say, "Hey, this is as
• 7:20
  serious as, as financial auditing," even though the analogy is not that great.
• 7:25
  Mm-hmm.
• 7:26
  So the kind of concept and the term changed and morphed, and we can
• 7:30
  talk about that perhaps, into things like, you know, is this AI GRC?
• 7:36
  You know, is this AI compliance?
• 7:38
  Is this AI governance?
• 7:39
  And so on and so forth.
• 7:41
  But actually, the core of it was- Mm-hmm ... you know, this is a, the
• 7:44
  foundational problem of this space, which is we can have all this technology, but
• 7:48
  if we can't use it, what's the point?
• 7:50
  And how can we create the kind of confidence and trust in that?
• 7:53
  And that moved from an academic exercise- Mm-hmm
• 7:56
  to, you know, maximizing our impact through, through a business.
• 8:01
  Kevin: Fast-forward to today then, what has remained of that
• 8:04
  initial vision, and what's changed?
• 8:06
  Emre: So I think at the... So one thing that's changed is I think, um, I think at
• 8:11
  the start, I believed that what we had... So, and I still believe this thesis, which
• 8:17
  is that 2010s was defined by data, and the 20, uh, 20s have been defined by AI.
• 8:24
  You know, a little primer here, I think the 2030s are probably
• 8:27
  gonna be defined by robotics.
• 8:29
  Mm-hmm.
• 8:29
  And, um, which is this kind of greater and greater embodiment or-
• 8:33
  Mm-hmm ... you know, autonomy of systems.
• 8:35
  And we thought, I thought that the way that data, w- a primary way in which data
• 8:41
  was governed was through legislation, and for example, we had the GDPR, and
• 8:46
  the GDPR, it m- created a huge market.
• 8:50
  I think that, um, that was ultimately wrong.
• 8:54
  I think it has not turned out to be the case.
• 8:56
  I think the EU AI Act is not a direct, you know, comparison to GDPR.
• 9:02
  I think AI does something else, and I think that the AI governance space and
• 9:07
  market, if you will, is gonna be defined more by more in analogy to cybersecurity
• 9:14
  than to governance in the way...
• 9:15
  Uh, sorry, to, than to compliance in the way that I thought it was going to be.
• 9:19
  Now- That, that's really interesting.
• 9:20
  Could you say more what that analogy means?
• 9:22
  So let's say, so I always thought, okay, what is AI... Uh,
• 9:26
  let's call it now AI governance.
• 9:27
  Let's, that's the category.
• 9:29
  And when you're doing category creation, uh, you know, no- it doesn't exist.
• 9:34
  You have to think, "Okay, what are the closest analogies here?" And I thought
• 9:38
  about, you know, a bunch of ways.
• 9:40
  One way I thought about was, okay, we've had cloud transformation,
• 9:45
  and then we got cloud governance.
• 9:47
  You know, does this look like that, you know?
• 9:50
  We had data transformation, and then we got data governance.
• 9:54
  And when you look at data governance, yes, there's things like providence, there's
• 9:57
  things like, uh, privacy, there's, uh, not pri- uh, compliance and privacy is not the
• 10:02
  sum total of data governance, of course.
• 10:03
  There's just a whole regime there.
• 10:06
  And then there's cybersecurity, right, which is a different kind
• 10:09
  of problem, uh, and a different form of governance, if you will.
• 10:13
  And my thesis was of this cohort, this would look more like data governance.
• 10:19
  And a big part of data governance, notwithstanding not the, not, not in its
• 10:23
  totality is compliance, was the GDPR.
• 10:27
  And actually, I thought AI is a companion or sister to data, that it
• 10:33
  has such an intricate relationship with the two that it's more likely to
• 10:37
  be the, the AI governance, whatever this cashes out as, is more likely
• 10:41
  to be closest to data governance.
• 10:44
  In fact, we had to make a- And presumably that was
• 10:45
  Kevin: the, that was the thesis of the European Commission as well- Absolutely
• 10:49
  ... in putting together the AI Act.
• 10:50
  Absolutely.
• 10:50
  Emre: Absolutely.
• 10:51
  So, so a couple, you know, we wrote a lot of papers on this.
• 10:54
  Like w- e- right at the start in 2020, I believe the EU published a, their two
• 10:59
  principles upon which they were going to, uh, govern or, uh, um, uh, legislate
• 11:05
  for AI, and it was a, a ecosystem of, uh, competitiveness and an ecosystem of trust.
• 11:11
  And I thought, okay, yeah, these are the legal frameworks and so on.
• 11:14
  Now, the cybersecurity side, it didn't f- it didn't, you know, it didn't seem
• 11:19
  to have the kind of implications of privacy... Or sorry, of like things
• 11:23
  like bias, explainability- Mm-hmm.
• 11:25
  Right ... a lot of the concerns, the social concerns that people were.
• 11:27
  Mm-hmm.
• 11:28
  It was more like, okay, the company needs to be protected from certain
• 11:31
  kinds of incidents, if you will.
• 11:33
  But as we've moved and, and, you know, Kevin, I'd be keen
• 11:36
  to hear your views on this.
• 11:37
  As we've moved, it feels like it's moving more and more into incident management,
• 11:43
  performance management, you know, um, rather than, "Oh my God, we've got this
• 11:47
  compliance thing that we need to tick off And, you know, we n- we need a bunch of
• 11:52
  people with legal and policy backgrounds.
• 11:55
  Uh, so it feels to me like that's where I probab- where
• 11:58
  it is becoming more technical.
• 12:00
  Kevin: Interesting.
• 12:01
  Uh,
• 12:01
  Emre: incidentally for us, Kevin, it's, it's actually a good thing because we
• 12:04
  come- Yeah ... from foundational research.
• 12:06
  Yeah.
• 12:06
  Um, and it turns out to be non-trivial to red team an algorithm, you know?
• 12:12
  Or to, to really think about what are the latest benchmarks, or how
• 12:15
  can we innovate in this space.
• 12:17
  But, um, but yeah, I think it's moving more into, in, in the, into the deeper
• 12:20
  tech space than I think people had anticipated, or even I had anticipated.
• 12:25
  Kevin: Yeah.
• 12:26
  It, uh, fascinating.
• 12:27
  So that's what I was going to ask you, is what would be different for organizations
• 12:32
  taking this cybersecurity mindset or for a, a company like yours that's providing
• 12:36
  these services compared to y- where you started and many people started
• 12:40
  was this is like auditing, we've got a whole literature and an experience and
• 12:44
  infrastructure around financial auditing.
• 12:45
  So what does it look like in
• 12:46
  Emre: practice now?
• 12:47
  I think a couple things.
• 12:48
  One thing is, um, it's not an extension of data governance.
• 12:53
  Kevin: Mm-hmm.
• 12:53
  Emre: Ergo, it's not just adding a few questions, um, to, you
• 12:58
  know, your data governance regime.
• 13:00
  Mm-hmm.
• 13:00
  So it has a huge implication in terms of who owns this within the business.
• 13:04
  Mm-hmm.
• 13:04
  I think secondly, the skill set of the kind of people who are involved in that
• 13:08
  kind of, uh, concern is very different.
• 13:10
  So it's not so much legal and policy, I think it's now gonna become more and
• 13:14
  more technical, or people with technical kind of literacy and so on and so forth.
• 13:18
  I think the third thing is that it dep- it, at the core of
• 13:22
  this, of, of the, let's say, the problem of governing AI is it's...
• 13:28
  Data governance is quite conservative, right?
• 13:30
  It's about conserving, protecting, you know, ensuring, um, a certain kind of,
• 13:36
  um, regime of, of protection, if you will.
• 13:40
  Whereas AI fundamentally is about the exploitation and opportunity
• 13:45
  that it presents, right?
• 13:47
  So companies are racing to, uh, adopt AI, and almost like the winners of
• 13:52
  that race are gonna win this mar- are gonna win the market generally.
• 13:56
  So you have this immense pressure to say, "We need to use these systems." So
• 14:02
  the governance aspect is actually about saying, "How can we ensure as quickly
• 14:06
  as possible we can get the rockets off the ground?" You know, it's not just
• 14:10
  about protecting from the outside, "Okay, let's make sure no one can
• 14:13
  access this," and so on and so forth.
• 14:15
  So I think at the, uh, it, it's much more technical.
• 14:18
  So, like, I thought the market would have landed more on documentation
• 14:22
  and reporting, so we expect... And I think a lot of vendors just
• 14:26
  emerged and did that, basically.
• 14:28
  You know, "We provide documentation, we provide..." Yeah, sure.
• 14:31
  You know, so does everybody else, you know?
• 14:34
  But actually at the core it's saying, "Can you interrogate the systems
• 14:37
  themselves?" Mm. So I've been looking at, um, two concrete examples.
• 14:41
  One is, uh, our thesis on human oversight- So, you know, Kevin, you
• 14:46
  remember, I'm sure, you know, it used to be the kind of panacea for everything.
• 14:51
  You know, we just have make sure you have human oversight
• 14:53
  or there's more human oversight.
• 14:54
  Kevin: Yeah.
• 14:54
  Yeah, yeah.
• 14:55
  And- And everyone always says, "Human in the loop, that's the
• 14:56
  answer." That's it, you know?
• 14:57
  I still hear that all the time.
• 14:58
  Emre: And then it's like, you know, we've got this elephant in the room,
• 15:01
  which is in what meaningful way can a human, um, you know, let's say with
• 15:07
  these agentic systems, meaningfully oversee such a system, you know?
• 15:12
  And then you've got this kind of deeper problem of the systems are so
• 15:15
  sophisticated and dynamic that you... What about real-time monitoring?
• 15:20
  Like, what does it mean to actually have oversight?
• 15:23
  So that's an aspect that has been significantly eroded, I believe.
• 15:28
  You know?
• 15:29
  The second part is the actual what does it mean to assess, and I, I strongly
• 15:33
  believe that it requires a meaningful technical assessment of the system.
• 15:40
  So not just ask-- So if you remember the, for example, the data protection
• 15:43
  impact assessments would just be a form that someone filled in, you know?
• 15:48
  It's just simply, um, it's just inadequate for these systems.
• 15:53
  Kevin: What exactly are those technical capabilities?
• 15:55
  You, you mentioned red teaming before, but, but give us a little bit more
• 15:59
  detail about what that means in practice.
• 16:00
  So, so
• 16:01
  Emre: we'll take a few examples.
• 16:02
  Um, let's say for, let's say, let's say we were inter- Let's say you looked
• 16:06
  at... There was a piece of work that we did, uh, I'm happy to say, was on
• 16:10
  explainability of agentic systems.
• 16:12
  Mm-hmm.
• 16:13
  Really cool piece of work actually, um, Kevin.
• 16:15
  It-- We won a prize.
• 16:17
  We were awarded this prize by one of the co-founders of OpenAI.
• 16:20
  Um, it was a hackathon, actually, 600 teams, uh, competed, and
• 16:25
  it was just about this really cool, we called it Agent Graph.
• 16:29
  Uh, anyone can find it on our website, and it's part of our product.
• 16:33
  And it was basically an explainability function or tool for agentic systems.
• 16:39
  And, you know, that's the kind of stuff that really comes out of the academy.
• 16:43
  You know, it comes from foundational research.
• 16:45
  It's bleeding edge.
• 16:47
  Uh, it's the kind of stuff which is, it's, it's not meaningful to just ask a
• 16:50
  few heuristic questions- Mm-hmm ... um, and say, you know, "Who, who made this
• 16:55
  decision?" Or, you know, "What was this output?" It's about actually can you
• 16:59
  interrogate this system technically?
• 17:00
  So that's one example.
• 17:02
  And then there's like a whole suite of... So you've got a social definition of bias,
• 17:07
  let's say, uh, but then you've got a whole suite of technical tests to assess bias.
• 17:13
  Of course, nothing can, can be done in, in a vacuum.
• 17:16
  Um, each one is contextualizing the other, but you do need
• 17:19
  to assess these technically.
• 17:20
  So we have a, a large open source, um, bias or debiasing library.
• 17:26
  So these are the kinds of things that I'm, I'm referring to.
• 17:28
  We still now industriously publish in, um, leading, uh, computer science
• 17:33
  or AI conferences and journals, uh, because we think fundamentally that's
• 17:37
  probably where the, the direction of travel is, and we believe in it as well.
• 17:42
  Mm-hmm.
• 17:43
  Kevin: How much of it, the discontinuity is there with those technical
• 17:46
  responses with generative AI?
• 17:48
  Because it, for example, there's, there's a lot of bias testing algorithms
• 17:51
  which have been around now for a decade, but when you've got- Yeah ... a
• 17:54
  non-deterministic system- Totally ... then it's become a different environment.
• 17:58
  Emre: Yeah, totally.
• 17:58
  So that's an interesting question, and if I may, Kevin, I'd love to
• 18:02
  indulge a little bit about this kind of drama with the standards-
• 18:07
  Mm-hmm ... and regulatory bodies in our ecosystem- Mm-hmm ... writ large.
• 18:12
  So I think we were desperate as a community to say there's the,
• 18:16
  you know, somebody come out with the framework that we can just...
• 18:20
  That will be our target to assure against, right?
• 18:23
  And I think what happened was, rightly so, um, things were moving so quickly
• 18:28
  that these bodies themselves were having to react in a way which was, you know,
• 18:33
  f- uh, tr- uh, tracking the bleeding edge of these- Mm-hmm ... uh, systems and
• 18:38
  then saying, "Okay, how do we assure it?"
• 18:40
  And I think, and I give absolute credence to this, I think that it,
• 18:44
  it's been almost an impossible task.
• 18:47
  You know, we've got o- There was a period in particular between '23 and '24 where
• 18:52
  it felt like almost every month or every other week, some huge seismic event would
• 18:58
  take place in our ecosystem, new capacity.
• 19:01
  I mean, the generative models weren't around in anything like the way that we're
• 19:05
  talking about them when the EU did its first draft of the, of its legislation,
• 19:09
  and then all of a sudden they had to kind of respond to general purpose systems.
• 19:13
  So this is where I think the analogy with cybersecurity comes in.
• 19:17
  I think the way that cybersecurity works is that you have, like, incidents
• 19:20
  or events, and then this has this huge implication on the industry,
• 19:25
  and then there is a solution, and then that cascades quickly across-
• 19:29
  Yeah ... um, uh, enterprises or, or, or companies writ large.
• 19:33
  And I think basically that kind of best practice becoming normalized is
• 19:39
  probably the way in which it's going to hap- it's, it's going to work.
• 19:43
  So we're gonna see practical responses to, to things that just emerge.
• 19:47
  It's gonna come- Mm-hmm ... from the academy, it's gonna come from
• 19:49
  industry just responding, and then they're gonna become the standards.
• 19:53
  I think it's gonna actually flip on its head.
• 19:55
  I think best practice will become the de facto standard because I'm not sure we
• 20:00
  can really do this in abstract, you know?
• 20:03
  Kevin: Can I push back on you a little on that- Please,
• 20:05
  please ... on the cyber analogy?
• 20:06
  Absolutely.
• 20:07
  Um, certainly, you know, there, there's a lot of great technical
• 20:10
  capability, uh, in cybersecurity.
• 20:13
  Um, but you know, if you look, especially you look across enterprises, so- Yeah
• 20:17
  ... so not just, you know, military, you know, super high security kinds of contexts,
• 20:22
  um, you know, there's incident reporting.
• 20:25
  There are very good firms when you have a data breach that
• 20:27
  will go and clean things up.
• 20:29
  But a- at least what I see is, is we don't really see a race to the top.
• 20:33
  There's still many companies that are saying, "Okay, this is
• 20:36
  a cost of doing business." Yeah.
• 20:38
  Uh, and, and they're not really adopting the best practices.
• 20:41
  Doesn't seem like the incentive structure is really there broadly enough.
• 20:44
  So i- i- if you... You're nodding your head, so if you agree with that, how does-
• 20:48
  Yeah ... AI governance avoid that fate?
• 20:49
  Look,
• 20:50
  Emre: so I agree.
• 20:50
  I, I, I'm not... Uh, because I think the ana- that the analogy with
• 20:53
  cybersecurity breaks down as well.
• 20:55
  Mm. Um, it's just that I was just trying to make it in
• 20:58
  comparison to data governance.
• 20:59
  Kevin: Well, I think it's a good analogy- Uh, yeah ... it's just cybersecurity
• 21:01
  is not necessarily a positive model.
• 21:03
  Emre: Uh, yeah, sure.
• 21:04
  Then, you know, so, so, so absolutely, I agree.
• 21:07
  So I can think, uh, in my, in my mind, I can think of different categories
• 21:11
  where you would have sufficient... You'd have market motivations
• 21:14
  and other categories where you would not have market motivations.
• 21:18
  Uh, and I think probably as we move forward, institutes like NIST and so
• 21:22
  on will probably have to work most aggressively on promo- o- on focusing
• 21:27
  on the areas where there isn't a market motivation to, for a company
• 21:30
  to, to behave in a particular or, or to govern in a particular way.
• 21:34
  But I must talk... I'm talking about really basic stuff, like
• 21:36
  for example, the hallucinations.
• 21:38
  I can just... A m- an immense market motivation to get that right.
• 21:42
  Yeah.
• 21:43
  You know?
• 21:43
  Uh, bias is not... People, I think, fixate on biases in demographic terms,
• 21:48
  but we're talking about, like, a algorithmic bias, like in terms of
• 21:51
  its... So we've got huge, um, market motivation to get that right, you know?
• 21:57
  So I think there is enough.
• 21:58
  It... That's not the same in, in a lot of cybersecurity, of course it isn't.
• 22:02
  Um, you know, whereas I think there is a, there are more powerful forces in-
• 22:07
  Mm ... in our industry, uh, for sure.
• 22:12
  Kevin: What are you seeing right now with the companies that, that you talk
• 22:15
  to or, or that work with Holistic?
• 22:17
  What's, what's motivating them?
• 22:18
  What are the biggest problems that they're bringing to you
• 22:20
  Emre: today?
• 22:21
  I think overwhelmingly... So my, my experience, and I'd love to know if it,
• 22:26
  if it's, uh, analogous to yours, is... Or, or in, or coherent with your experience.
• 22:30
  My, my experience was that in '23 and '24, there was this explosion of pilots.
• 22:36
  Mm. Like a huge amount of kind of speculative work.
• 22:39
  And I felt like at the start of '25, there was a kind of reckoning, um, which
• 22:45
  was, you know, what's going on here?
• 22:47
  You know, where is the return on value on these systems?
• 22:50
  You know, what are we actually seeing here?
• 22:52
  And we're starting to see more and more of this sense of reflecting a bit
• 22:57
  more and saying, "Okay, can we actually derive value from these systems?"
• 23:02
  So there has been... I think we've moved past that pure experimentation phase.
• 23:07
  Uh, I don't think we saw the kind of productivity gains that the industry
• 23:12
  was hoping that it would have gained, but I don't think it was a wasted labor.
• 23:16
  I don't think it was wasted labor.
• 23:17
  I think that capital that was invested, uh, both in terms of human labor and
• 23:22
  actual capital, I think it's gone into knowledge rather than production.
• 23:26
  Uh, and that, and that means, you know, it's almost like a J curve.
• 23:29
  So ultimately, I think now we're gonna start to see the, the upside of it.
• 23:33
  So the kind of... What we're seeing generally is a far higher
• 23:36
  level of maturity in terms of awareness of this problem.
• 23:40
  Um, they've had some periods of experimentation, um, and now
• 23:44
  they're really keen to actually motivate and push this forward.
• 23:48
  Um, typ- so the distribution is usually... Actually, a lot of
• 23:51
  companies are still using traditional, uh, machine learning systems.
• 23:55
  Then they've got a c- a cohort of generative systems that have,
• 23:59
  have been quite reasonable and worked, and now they're really
• 24:01
  experimenting with agentic systems.
• 24:04
  Um, not, not generally not in production, but, uh, you know, really
• 24:08
  keen to move in that direction.
• 24:09
  So it's-- That, that's the kind of distribution.
• 24:12
  Kevin: Yeah.
• 24:13
  How does that impact on their interest and commitment to AI governance?
• 24:17
  Oh,
• 24:17
  Emre: I think it, it's the far... There's, it... So what I'm finding is
• 24:23
  there are different constituents that are concerned- Mm-hmm ... with AI governance.
• 24:27
  Mm-hmm.
• 24:28
  So one of the constituents is whoever owns AI transformation in the business.
• 24:33
  All right, so fundamentally, they are responsible for undergoing... It's for
• 24:37
  motivating this transformation, and governance is just central to that.
• 24:40
  For them to have a single pane, um, view on the use of AI across the business,
• 24:45
  to be able to manage this, to be able to have, you know, a view on who's
• 24:48
  using this and who isn't using this.
• 24:49
  So fundam-- Let's say that's a kind of AI governance as, uh, something that
• 24:55
  falls out of a technical track, you know, which is just about transfor- tran-
• 24:59
  AI transformation across the business.
• 25:01
  And of course, the sec- second cohort is those very much motivated by compliance
• 25:06
  or policy, you know, and saying, "Look, we need to be, we need to be able to...
• 25:11
  We really need to be on our brief on here.
• 25:13
  You know, how are we gonna be accountable?
• 25:15
  How can we document what we're doing?
• 25:16
  How can we ensure that we have sufficient audit trails and so
• 25:19
  on and so forth?" Anticipating the AI Act and other frameworks.
• 25:23
  So they're the two kind of general, uh, camps.
• 25:27
  I don't think, um, the persona that owns this has solidified.
• 25:33
  Um, you know, Kevin, it's funny, I thought at the start of the decade in 2020, I
• 25:37
  thought by the middle of this decade, so last year, we would have had, we would
• 25:41
  have settled on a lot of the core issues like what does it mean to... What...
• 25:46
  Like for example, the documentation I thought would've been solved, you know?
• 25:49
  Um, and then I thought the second half of the decade would be
• 25:52
  codification of technical standards.
• 25:54
  Kevin: Mm-hmm.
• 25:55
  Emre: But actually, the, the market or the general space is still very indeterminate.
• 25:59
  Uh- Yeah ... it's, there is-- We're starting to see the kind of
• 26:02
  repeatability in personas and so on, but actually, it's still quite,
• 26:06
  um, it's still very, very dynamic.
• 26:10
  Mm-hmm.
• 26:11
  Kevin: Well, so what do you see from this vantage point of 2026?
• 26:15
  What, what do you think now we can expect in 2030 in this area?
• 26:19
  Emre: Yeah, I do.
• 26:19
  I think we're gonna base... I think we would have s- I think
• 26:22
  in quotes, we would've solved the problem of productivity Actually
• 26:25
  coming from these systems.
• 26:27
  So like, um, I don't think it's gonna be...
• 26:29
  I think there was, there was a thesis that Copilot was the approach,
• 26:34
  which was every single person is supercharged with their own AI.
• 26:37
  I think it's more gonna be there's gonna be some very, very powerful
• 26:40
  use cases- Mm-hmm ... and we're gonna see immense value from them.
• 26:44
  Mm-hmm.
• 26:44
  And that companies would have solved that, like we would've been
• 26:47
  right in our, fully in our stride, um, in, in that particular path.
• 26:52
  So I think it's probably not so much the in quote- I'm talking
• 26:55
  about in the enterprises.
• 26:57
  Um, it's not so much I think about the kind of transformation of every
• 27:02
  single person having, uh, a system, not the democratic model, as I call it.
• 27:06
  I think it's more the vanguard model of, you know, companies are gonna
• 27:10
  invest heavily in these, you know, mission-critical, probably agentic
• 27:14
  systems that are gonna motivate huge change within the business.
• 27:17
  Mm-hmm.
• 27:18
  And the productivity will come from them.
• 27:20
  And the governance in that context is very different to the governance
• 27:25
  in the more generalized sense.
• 27:27
  So the, the more generalized sense, I think we'll continue to have the problem
• 27:31
  of shadow AI, who's using it, what are they doing, and so on, and that's it.
• 27:35
  And then on the kind of more vanguard models, let's call them, or, or, or
• 27:39
  use cases, it's gonna be very much red teaming and saying, "Okay. How,
• 27:42
  however deep we can go, let's go." You
• 27:44
  Kevin: know?
• 27:46
  Mm-hmm.
• 27:46
  And how much can red teaming be automated and scaled, which presumably is essential,
• 27:51
  especially with go-to a-agentic models?
• 27:54
  Emre: Yeah.
• 27:54
  So I think what we'll have is probably we're in a, it's like an arms race, uh,
• 28:00
  from... So it's funny, is it-- So I guess one of the themes of this conversation
• 28:04
  is it, what, is that I thought it was gonna be the law versus tech, right?
• 28:09
  And as it's progressed, it seems to be tech versus tech.
• 28:12
  So I think what we're seeing is basically it's gonna be the rate at
• 28:16
  which we're gonna see innovation in AI is gonna have to be mirrored and
• 28:19
  will be mirrored in the rate at which we're seeing innovation in governance.
• 28:23
  Mm-hmm.
• 28:23
  I think for the long term, you know, for the next ha-half a decade, decade,
• 28:29
  it's gonna, it, it's impo- it's gonna be impossible to play in this game without
• 28:33
  meaningful, uh, research capacity.
• 28:37
  So I just think it's gonna be... That's, that's the m- that's the direction
• 28:41
  in which we're gonna be seeing this.
• 28:42
  It's, uh, it's... We, we have to anticipate more, uh, and
• 28:47
  significant innovation in this space.
• 28:49
  You know, I mean, we would be... It, it's not static.
• 28:52
  You know, it's not that we just had the LLMs and then now we're,
• 28:56
  you know, responding to it.
• 28:57
  It's like, you know, every year something dramatic is taking place.
• 29:00
  So I think it's fundamentally gonna be whoever's on the core face of the
• 29:05
  research and the innovation in governance is gonna be the only ones that can
• 29:09
  meaningfully govern, uh, these systems.
• 29:13
  Kevin: You mentioned early on that, that human oversight, uh, is, is
• 29:17
  no longer going to be, be viable or just as a, as a simple concept.
• 29:20
  So then what- Yeah ... what replaces that, that serves the needs that people
• 29:23
  were talking about human oversight for?
• 29:25
  Emre: So, so, so I do believe there will be, um, human oversight, but
• 29:28
  I think it'll be at a different level of, of abstraction.
• 29:31
  You know?
• 29:31
  So at the moment it's, I guess, I'm trying to think about what the
• 29:34
  appropriate, um, mental model would be.
• 29:37
  It's a bit like saying, you know, we don't, we don't scan
• 29:41
  for planes using our eyes, right?
• 29:43
  Mm-hmm.
• 29:44
  Um, we have radars, we have so on, and then we have somebody
• 29:46
  viewing the radar, right?
• 29:48
  So e- I see it a bit like, I see it like that.
• 29:50
  I think it's at a high level of, of abstraction.
• 29:53
  We'll see the kind of human oversight.
• 29:55
  It won't be on the every single system or somehow... I think we're gonna
• 29:59
  basically move towards AI governing AI.
• 30:02
  I think effectively that's what we're going to see.
• 30:04
  We're gonna see an increase in automation and an increase in the use of, um,
• 30:11
  things like guardian agents, sentinels.
• 30:14
  Mm-hmm.
• 30:15
  Um, you know, really quiet areas, which I thought, you know, were
• 30:19
  a lot further away are actually closer than, than one imagines.
• 30:23
  So I think that's basically the paradigm we're moving towards, AI governing AI.
• 30:27
  Kevin: Hmm.
• 30:28
  Which presumably makes your point about the need for research capability,
• 30:32
  because obviously you've got to build AI governance systems that don't have
• 30:35
  the same problems as the AI itself.
• 30:37
  Emre: You've got the problem of infinite regress, right?
• 30:39
  Yeah.
• 30:39
  So like you need to, how does that system govern that system?
• 30:42
  How do we govern the system governing the system?
• 30:45
  How do you have confidence in that system and so on.
• 30:48
  And there's regimes in which you can do this, but, but absolutely, I have no doubt
• 30:52
  in my mind, and it's quite extraordinary because, you know, as an entrepreneur,
• 30:56
  when you leave the academy, uh, it's a d- it's a completely different, um, domain
• 31:02
  of expertise and incentive structure.
• 31:05
  But when you're-- But we, we seem to be in the most unique, uh- Mm-hmm ... setting,
• 31:10
  Kevin, where like the f- that research is driving, foundation research is
• 31:15
  still driving the, the market or the industry, which is an extraordinary thing.
• 31:19
  You know?
• 31:20
  Um, two company, uh, from our home institution, UCL, um, two companies just
• 31:25
  raised a significant amount of money.
• 31:27
  Uh, one raised one point one billion, the other raised five hundred million.
• 31:31
  Uh, I think, oh, you probably saw the news recently.
• 31:33
  And for la- effectively labs, you know?
• 31:37
  Mm-hmm.
• 31:38
  And it's, that was quite striking for me, but I, but it, but it kind of makes sense,
• 31:43
  you know, that, um, that still there's so much foundational work, uh, and the
• 31:48
  kind of foundational work that is taking place is coming from, from research.
• 31:52
  Hmm.
• 31:53
  Kevin: Although, you know, we're going to need to wrap up after this.
• 31:55
  The, the, the challenge is w- when you have these, uh, applied companies
• 32:01
  especially, that have valuations where they're raising in the hundreds
• 32:04
  of millions or billions- Yeah ... of dollars, uh, then people who otherwise
• 32:07
  would be in academia and doing research increasingly are doing their work-
• 32:11
  Behind closed doors in organizations that are not necessarily publishing
• 32:16
  Yeah How do we get out of that dynamic?
• 32:17
  Emre: Yeah, I think that's a, that's a great que- great question.
• 32:19
  I don't know.
• 32:20
  Yeah Because they are producing fantastic, there are companies producing
• 32:24
  fantastic work, and at the same time, exactly as you said, so like I
• 32:28
  noticed LeCun left, uh, Meta, right?
• 32:32
  And he's the classic example of somebody I would've expected to go back in, properly
• 32:36
  back into the academy, but instead he basically raised another billion dollars,
• 32:40
  I think it was a billion euros or whatever it was, to set up another lab.
• 32:44
  So, um, there is a whole conversation to be had about, you
• 32:48
  know, what is taking place there.
• 32:49
  Can you, can you do meaningful, purely speculative interest-driven
• 32:55
  research in the context of a commercial, um, uh, entity?
• 33:00
  Um, I mean, um, that's a, a greater question, if you will, but I
• 33:04
  do think, I mean, the US is the best model for this, you know?
• 33:08
  Uh, we've got so many examples in the US of where, um, fundamental
• 33:13
  and foundational innovation is taking place in industry, you know?
• 33:17
  So I'm not as, I'm not, I'm not, I'm not ... It's, this seems to be
• 33:21
  a unique dynamic, um, but I think that's an important question to ask.
• 33:26
  Mm-hmm.
• 33:27
  Absolutely.
• 33:27
  Kevin: Well, it's, as you say, a fascinating time in many ways.
• 33:30
  We could keep going for a long time, but we need to wrap up.
• 33:33
  Emre, thanks again.
• 33:34
  It's really great to talk to
• 33:35
  Emre: you.
• 33:35
  Thank you so much.
• 33:36
  Thanks, Kevin.
• 33:37
  Kevin: This has been The Road to Accountable AI.
• 33:40
  If you like what you're hearing, please give us a good review and
• 33:43
  check out my Substack for more insights on AI accountability.
• 33:47
  Thank you for listening.
• 33:51
  If you wanna go deeper on AI governance, trust, and responsibility with me and
• 33:56
  other distinguished faculty of the world's top business school, sign up for the
• 34:00
  next cohort of Wharton's Strategies for Accountable AI online executive education
• 34:04
  program, featuring live interaction with faculty, expert interviews, and
• 34:08
  custom designed asynchronous content.
• 34:11
  Join fellow business leaders to learn valuable skills you can
• 34:14
  put to work in your organization.
• 34:17
  Visit execed.wharton.upenn.edu/acai for full details.
• 34:24
  I hope to see you
• 34:24
  there.