Latest / The Road to Accountable AI / Brett Levenson (Moonbounce): The Best AI Security System is the One People Will Use

Transcript

Brett Levenson (Moonbounce): The Best AI Security System is the One People Will Use · The Road to Accountable AI

• 0:09
  Hi, I'm Kevin Werbeck, Professor of Legal Studies and Business Ethics at the Wardens School of the University of Pennsylvania.
• 0:15
  For decades, I studied emerging technologies from broadband to blockchain.
• 0:19
  Today, AI is promising to transform our world.
• 0:24
  But AI needs accountability, mechanisms to ensure it's developed and deployed in responsible, safe, and trustworthy ways.
• 0:32
  On this podcast, I speak with the experts leading the charge for accountable AI.
• 0:36
  Yeah, yeah.
• 0:39
  Brett Levinson is co-founder and CEO of Moonbounce, which is building what it calls a semantic firewall for AI.
• 0:46
  And we get into what that means.
• 0:48
  He worked at Apple and then joined Facebook in 2019 to lead integrity work in the aftermath of the Cambridge Analytica scandal.
• 0:56
  We talk about his learnings from the social media content moderation world and the tricky challenges.
• 1:03
  of building mechanisms to ensure that companies' AI systems actually comply with their own policies.
• 1:13
  Brett, welcome.
• 1:13
  Thank you for joining me on the Road to Accountable AI.
• 1:16
  No, thank you.
• 1:17
  I'm I'm ~ excited to be here.
• 1:20
  What were some of the the key issues that you saw in your prior life doing content moderation and business integrity ~ at Facebook that you ~ have brought into your current
• 1:31
  work and movement?
• 1:33
  ~ my god, so many.
• 1:36
  I think the the the core issue that I saw, and I'm happy to kind of get into the backstory of like how all this was figured out.
• 1:45
  ~ but I think that like the main cent center of it ~ was that policies as they were being written were essentially ~ unenforceable consistently.
• 1:57
  The policies that we were asking human reviewers
• 2:02
  To use as their sort of guiding principles.
• 2:04
  And ~ ultimately, you know, the labels and the decisions human reviewers were making were the same data that we were trying to use to train models.
• 2:13
  So garbage in, garbage out, right?
• 2:15
  ~ But ~ the the the the policies that we were giving these human reviewers and the requirements that we were putting on them, ~ essentially what we found was, you know, the
• 2:29
  accuracy wasn't very good as a result of them.
• 2:32
  ~ and more importantly, perhaps in my view, ~ the decisions that were being made were inconsistent, reviewer-to-reviewer.
• 2:40
  And I think that consistency issue ~ kind of became like the center of what I what I got a little obsessed with is how do we define policies in a way that they ~ can be consistently
• 2:55
  enforced upon, ~ that their meaning is clear, that there's as little ambiguity.
• 3:01
  in intent as we can sort of get to.
• 3:07
  and ~ ultimately ~ we ended up coming up with something that I coined info first ~ enforcement that I guess was like the the culmination of a lot of that ~ a lot of that
• 3:21
  research and ~ it created you know the consistency auditability like explainability and
• 3:31
  I as I said, I think in my time at Meta the the main thing that I just discovered was like the the way we're going about this, in my view, is setting us up for failure.
• 3:39
  Like that that that's kind of the that was the my biggest takeaway.
• 3:43
  Yeah.
• 3:43
  Okay.
• 3:43
  Well so you me interested in that.
• 3:44
  So what what is info first?
• 3:46
  So fine.
• 3:47
  Yeah.
• 3:47
  ~ to explain info first, it helps to explain the alternative, which we call decision first.
• 3:53
  ~ so essentially the way that that human review was done ~ when I joined Meta was that there's a policy.
• 4:00
  And the policy was sort of written by the policy team, maintained by the policy team.
• 4:05
  ~ picture, you know, 20 plus page document.
• 4:08
  there are po you know, there were many, many policies, you know.
• 4:11
  ~ I think we had 40 plus categories, everything from ~ hate speech.
• 4:16
  To the sale of restricted goods and services, right?
• 4:19
  ~ so different policies for different things.
• 4:22
  ~ and so these policies are written, they're very complex, they're sort of written like legal documents in many ways.
• 4:29
  ~ and then they get sort of thrown over the wall.
• 4:33
  And the first stop is operations.
• 4:35
  An operations job is to take the policy and turn it into something that we called a protocol.
• 4:40
  And that was essentially like a more detailed
• 4:43
  version of the policy that didn't just explain the definition of the problems, but also explained, okay, well, what's the implementation?
• 4:50
  What's the action that should be taken based on these definitions?
• 4:54
  ~ they would then also train human reviewers on the policies.
• 4:59
  ~ and finally a human reviewer would be like ready to go to work, basically.
• 5:04
  And the the work of the human reviewer, I mean I know Tara Lawson has written about like many people have written about what this is like.
• 5:11
  ~
• 5:12
  But just operationally, it's like they have a job queue to work on, they pick a job, they see content, and they are given 30 seconds to make a decision.
• 5:21
  And that decision, at least the way it was set up when I started, was what should we do?
• 5:27
  Should we block this?
• 5:28
  Should we ~ allow it?
• 5:31
  Should we escalate it?
• 5:32
  ~ should we send it back to the advertiser and say, Hey, you need to make the following changes?
• 5:37
  Like, what should we do here, basically?
• 5:40
  And only then
• 5:41
  Once a an actioning decision was made, why?
• 5:45
  Like which policies policy or policies do you believe, you know, were violated here, essentially?
• 5:51
  That's decision first, right?
• 5:54
  And to me, when I first learned that this is how we did it, suddenly the fact that our accuracy and I don't want to just pick on meta, like accuracy across the board on human
• 6:04
  review for basically any big company that does it is somewhere around 60 to 70 percent for most policies, basically.
• 6:10
  So it's not just meta.
• 6:12
  and I think when you hear what's being asked of these reviewers, you know, 30 seconds to make a decision, you have to keep many, many 20-page policies in your head.
• 6:20
  ~ apply their judgment, not yours.
• 6:24
  I actually sort of came away with this feeling like, wow, our ops group is really good at what they do.
• 6:29
  ~ so we can simultaneously believe ops is really good at this, but also that it's insufficient, right?
• 6:35
  ~ InfoFirst, on the other hand.
• 6:39
  Essentially, we ran this experiment.
• 6:41
  So we we took a policy area ~ and we we essentially said, okay, can we extract a set of yes or no questions from this policy?
• 6:51
  ~ that if answered as yes, would tell us that the policy is violated, basically.
• 6:58
  And what we did was we we figured out how to sort of extract all these questions, hundreds of questions pulled out of a policy, basically, ~ and we organized them into a decision
• 7:07
  tree.
• 7:08
  ~
• 7:10
  And we said, okay, we're gonna do this.
• 7:12
  This was a very quick, we were trying sort of prove this idea out at this point.
• 7:15
  So not complex toy stuff, but essentially we said, okay, we're gonna build a thing so that instead of the reviewer being asked, hey, what decision should we make?
• 7:24
  We're gonna ask them these yes or no questions.
• 7:26
  And these were things like, for example, in hate speech, it could be, is there a reference to a protected group of individuals ~ based on religion, race, ethnicity, sexual
• 7:37
  orientation, et cetera, et cetera, right?
• 7:41
  if it was a pornography policy, it could be you know, is there ~ nudity in this image, basically, something like that.
• 7:49
  And obviously many more questions.
• 7:51
  ~ so we would collect the answers to these questions from the reviewers and then apply the decision tree.
• 7:56
  And the d and ultimately the difference here is the reviewers are not making the decision or deciding what action to take.
• 8:03
  They're just kind of acting like feature extractors.
• 8:06
  They're they're trying to figure out what is this content about.
• 8:08
  Basically.
• 8:09
  ~ and then once we have the answers, now it's just a systematized approach to say, okay, what what action should we take here?
• 8:17
  Basically.
• 8:18
  And our hypothesis, of course, was that this would improve decision making consistency and agreement across reviewers.
• 8:26
  And of course it did.
• 8:27
  Like I I almost feel like this is a no-duh kind of result.
• 8:31
  but it improved it consistently, like 98% consistency from you know, 65, 70%, essentially overnight, you know.
• 8:38
  First first try.
• 8:39
  ~ and I think this became the starting point for a lot of the the additional work I did.
• 8:49
  ~ and ultimately what ~ kind of is at the core of what we try to do at Moon Bounce.
• 8:55
  ~ we we are very focused on trying to disambiguate, ~ trying to figure out
• 9:05
  For this very complex policy, what are the simple atomic questions that we can ask that will help us understand what the right action here is?
• 9:14
  Sometimes those questions are about very simple things, objects, concepts, whatever.
• 9:19
  And sometimes they're more complex.
• 9:20
  Sometimes they're about, well, what context is this thing occurring in, basically.
• 9:24
  ~ but ultimately that atom atomizing things in this way, ~
• 9:33
  does create more consistency of decision making.
• 9:35
  And I believe it it it also helps create explainability.
• 9:41
  Because when we make decisions, it's not just, well, I s I did it because the policy said so.
• 9:46
  It's we made this decision because the answers to this question and this question and this question were yes.
• 9:52
  And the policy says if this is true and that is true and that is true, then the decision must be X, Y, Z, basically.
• 10:00
  It's not quite that simple, but that's that's the I'm simplifying a little bit, but that's the basic idea.
• 10:05
  Yeah.
• 10:06
  No, great.
• 10:06
  So ~ how do you come up with all those questions?
• 10:09
  How much does that have to be a manual process?
• 10:13
  when we first built the system, it was very manual.
• 10:15
  ~ I'm my background is reinforcement learning.
• 10:18
  I've been an engineer 25 years and ~ I don't know, I've been obsessed with figuring out how meaning works for most of that time and and and with being able to predict things.
• 10:28
  ~ so when we first built the system, ~ most of it was a manual process.
• 10:32
  We ~ we basically said to the human operating system, hey, you need to think about this problem differently.
• 10:40
  What we're gonna give you is what I like to call a learning gradient.
• 10:44
  It's a sort of a machine learning idea.
• 10:45
  I'm happy to I'm happy to explain it further if that would help.
• 10:49
  But basically
• 10:50
  for for for the audience that would be useful.
• 10:52
  Yeah.
• 10:53
  Yeah.
• 10:53
  I mean, basically the idea is if you write a prompt, and hopefully most people at this point are familiar with prompts, right?
• 10:59
  And it doesn't do what you want, but it's four or five paragraphs long.
• 11:03
  Well, now you're kind of in this spot where you're like, what do I change?
• 11:07
  What do I change?
• 11:08
  How do I fix it?
• 11:09
  And you might try something, ~ and then try, you know, run this run run the AI again.
• 11:16
  And I don't know, maybe you get the right answer.
• 11:18
  Maybe it makes things worse in some ways and better in others.
• 11:21
  Like you you have
• 11:22
  Really, you have no guide to tell you what the what the right set of changes is.
• 11:26
  So a learning gradient, and it's actually how most of ML works, ~ is about providing information that tells us what's the right direction to move in on this gradient.
• 11:37
  Like, how should we change the instructions ~ so that we get a better outcome?
• 11:43
  And so when we first built the system, it was very manual.
• 11:45
  ~ but what we were doing was we were saying, Hey, we're gonna give you complete insight.
• 11:51
  Into all of the questions that you're asking, how each one is being evaluated across potentially thousands of examples, ~ so that you can understand which questions are doing
• 12:03
  work and which questions are maybe making things worse, basically.
• 12:07
  That was how it started.
• 12:09
  Now at this point, ~ we've built an AI that does the decomposition work for the user, basically.
• 12:15
  ~ and so most of our customers just some of them ~ you know.
• 12:20
  mature enterprises, they come to us and they say, here's our, I just had this with a financial institution the other day.
• 12:25
  They handed us a 200 page content standards document that RAI ingested and and sort of did its thing with.
• 12:34
  ~ But less it less sort of sophisticated customers in some cases it's really okay for them to just say, hey, I don't want my customer service agent to issue refunds.
• 12:49
  that aren't aligned with my refund policy.
• 12:50
  And our agent will then say, Okay, can you share the refund policy with me?
• 12:53
  Like it it you don't have to provide much.
• 12:56
  We're gonna do the best we can with what you give us, basically.
• 13:00
  Let's just take a step back ~ and for context and just describe you what what is the the problem area that you're addressing with mood bounds.
• 13:06
  How is it similar or different than what we were talking about in in content moderation?
• 13:11
  Yeah, so I think ~ the parallels are there, but the the main problem that we solve ~ these days ~ is really AI safety and security.
• 13:20
  ~ to me, ~ the second we developed AI, and I don't think enough people have like appreciated this yet, ~ the appul the application layer in all software that's using AI
• 13:34
  became non deterministic.
• 13:37
  And
• 13:38
  Every security protocol we have developed for 20, 25 years assumes that the application is deterministic.
• 13:46
  permissions, I am, even privacy policies, like they all assume that you can look at the code and know what the range of possible behaviors will be, basically.
• 13:57
  And just overnight that became not true, basically.
• 14:01
  The example I always give people is because I think most people at this point.
• 14:05
  Or many people at least have used, you know, Claude or Open or ChatGPT or something like that.
• 14:10
  ~ so the example I give is if you've ever seen that allow button when Claude wants to do a thing, you probably clicked allow.
• 14:18
  You may have even clicked always allow.
• 14:19
  And I actually think always allow is a very rational behavior, right?
• 14:23
  Because otherwise, what's the point?
• 14:25
  Like you may as well not use it.
• 14:27
  ~ but if you allow your agent to send an email on your behalf, it can do
• 14:33
  But you might say, well, it's just sending an email.
• 14:35
  What's the big deal?
• 14:36
  Well, okay, sure.
• 14:37
  It could send a confirmation that you're attending lunch with your friend next week.
• 14:41
  It could also send your entire customer database to a hacker whose email it found in a website that it searched.
• 14:47
  Like both of those outcomes are possible from the same allow thing.
• 14:53
  ~ And the problem that we solve is really to me what what's at the core of that permissioning model.
• 15:00
  The problem
• 15:01
  That we that we in security and safety need to be concerned about with AI is no longer is it okay to send an email?
• 15:07
  Maybe it is, maybe it isn't.
• 15:10
  The question is, is it okay to send this email in this context with this list of recipients and with this body and these attachments?
• 15:20
  That's what we need to be able to evaluate.
• 15:21
  And so what Moon Mounts does is we are ~ a security layer, essentially.
• 15:27
  ~ we operate a hidden underneath the AI.
• 15:31
  We evaluate every input, every output, every action the AI wants to take, every document it ingests.
• 15:39
  And we decide in real time, as the AI is doing its thing, hey, should this output, should the AI be allowed to say this?
• 15:47
  Should it be allowed to do this?
• 15:49
  Should the request that the user made of the AI should that be allowed, basically?
• 15:54
  And we decide that based on the business's policies, essentially.
• 15:58
  ~
• 16:00
  That's our business.
• 16:01
  You know, we work with everyone from I mentioned customer service agents ~ to companies that are using AI to generate images and video.
• 16:10
  ~ so essentially we're a f we are a semantic firewall around AI applications, we like to say.
• 16:17
  Wha what are the the hardest parts of building this?
• 16:19
  I know your your co-founder was a senior engineering executive at Apple, so you w what w what was tough to to do this effectively?
• 16:27
  ~ I think many sorry, many, many things, but I would say actually the the the the most difficult challenge we ran into was doing it quickly and cost efficiently.
• 16:40
  ~ ultimately what we do, there's no way around it.
• 16:44
  It's a tax on ~ on inference, on what companies are already spending to make the AI work in the first place, basically.
• 16:52
  ~ and so we are a cost on top of that.
• 16:56
  And I
• 16:57
  think it's it's important to be respectful of the fact that like AI is already really expensive.
• 17:03
  So it's important that we be able to do this cost efficiently enough ~ that the company is sort of getting something for their money, basically.
• 17:12
  ~ and as I mentioned, being able to do it, we we sort of have this SLA where we say we will always make decisions in 300 milliseconds or less.
• 17:21
  That's around the amount of time it takes to blink, basically.
• 17:25
  And I believe that's really important.
• 17:26
  Because I say this a lot.
• 17:28
  The best security system is a security system that someone will actually use.
• 17:35
  and I think if we disrupt the user experience, no one's gonna use us.
• 17:39
  Right.
• 17:40
  And so our goal has always been to fit inside that user experience window, to be able to be fast enough ~ that we can prevent the harm from occurring before the user sees it.
• 17:54
  And
• 17:55
  we operate sort of with like what you might think of as a broadcast delay in television, right?
• 17:59
  Like, you know, ~ we operate 500 milliseconds ahead, basically.
• 18:03
  ~ but any more than that, and things like voice agents stop working, right?
• 18:08
  ~ I was just talking to a customer the other day who builds AI receptionists that work over the phone.
• 18:13
  And, you know, to him, his biggest questions were about how fast can you can you do this?
• 18:19
  Because to a human on who's on a phone call with an AI, if the AI is like,
• 18:23
  Working, working, working.
• 18:25
  Eventually it's just a crappy experience and people hate AI for it, you know?
• 18:29
  ~ so ~ definitely like getting the latency right was hard.
• 18:33
  I mean, I could go into a ton of details about how making judgment work is is difficult if you'd like.
• 18:42
  ~ but it's a bit of a rabbit hole.
• 18:44
  So I'm happy to go there if you want to talk about it.
• 18:45
  But yeah
• 18:46
  I am curious about it.
• 18:47
  ~ you know, because that that always seems to be the the the challenge.
• 18:51
  E everyone would agree that the AI system should do what they want it to do, but what you want it to do is is not a a simple thing.
• 18:59
  I would agree.
• 19:00
  ~ so I'm I'm gonna feel very silly talking about this because I feel like you're gonna be the expert more than me.
• 19:06
  ~ but ~ one of the things I learned at Meta is that policy cannot be static.
• 19:12
  ~ the world changes, bad actors adapt.
• 19:16
  Like there's you you can't just write a policy once and say, Okay, we're good.
• 19:20
  We got our policy now.
• 19:21
  ~ the analogy that I I use a lot is the legal system and and case law.
• 19:27
  ~ and
• 19:29
  You know, the legal system has so many mechanisms for how case law work.
• 19:33
  And that's why I'm saying, like, I'm please correct me if I if I'm if I'm off on any of this stuff.
• 19:37
  But you know, there's a trial, there's a judge, there are jurors, there's an appeal system to make sure that we made the right decision in the first place.
• 19:45
  ~ but the main reason that case law and precedent matters, I I mean, I would love to debate this with you, but to me, when I sort of boil it down through my engineer brain,
• 19:56
  it's like we write a law.
• 19:58
  But that law we sort of acknowledge it cannot anticipate the messiness of the real world.
• 20:03
  It's just not possible.
• 20:04
  Yeah, yeah, no, and there's always this rule of standard tension.
• 20:06
  Yeah.
• 20:07
  Yeah.
• 20:08
  And so the case law system and and the legal system is there to evaluate those cases when they come up and to decide, hey, how should should this law be applied to this situation?
• 20:19
  And how should it be applied if so?
• 20:21
  And what should the penalties be?
• 20:23
  ~ and we realized, you know, I I I will stand by the fact that our info-first approach just on its own makes decision making better and more consistent.
• 20:35
  But we realized pretty quickly ~ that you cannot, even with that, you need the policy to be adaptive.
• 20:43
  You need it to be able to bend.
• 20:45
  You need to be able to like understand ~ what the what our we need to be able understand what our customers' judgment is as they see new cases, basically.
• 20:57
  This reminds me when I was at Meta, we did this exercise at one point where we looked at.
• 21:05
  How well was our decision making aligned with the policies that we had written down, basically?
• 21:13
  And it was really interesting.
• 21:14
  I mean, ~ let's just say it was less than half aligned in most cases.
• 21:20
  and I think the the knee-jerk reaction is to go, well, you just explained human review, like decision-making is bad.
• 21:26
  And that's true.
• 21:27
  There's some portion of that that was the case.
• 21:29
  ~ but if you looked at
• 21:33
  Alignment over time.
• 21:35
  What you saw was that alignment got worse and worse as time went on, basically.
• 21:39
  ~ and to me, this spoke to like the idea that policy must not be static, that it must be able to adapt.
• 21:46
  ~ and so one of the hardest things to your question, I think for us was figuring out okay, how do we incorporate the customer's judgment ~ over time in a way that remains audible?
• 22:03
  That doesn't have silent drift in it, basically.
• 22:05
  ~ and we eventually develop this system that we just call it decision memory, basically.
• 22:11
  ~ but essentially what we say to customers is okay, get started with your policy.
• 22:16
  Our system is gonna help you decompose it.
• 22:18
  It's gonna break it down into all these little parts to try to make decision making more consistent and understandable.
• 22:22
  Great.
• 22:25
  I never recommend to a customer that they go live on day one.
• 22:27
  I'm always like, hey, put us in observe mode.
• 22:29
  Let's take a look at the decisions we make.
• 22:31
  And we want you we want you to issue corrections to our decision making.
• 22:35
  ~ over
• 22:38
  Actually a reinforcement learning process?
• 22:40
  I as I said, I'm a reinforcement learner cat.
• 22:42
  That's what I figured.
• 22:43
  I can't help it.
• 22:43
  ~ so yeah, so you know, they issue corrections.
• 22:49
  We actually have a sampling mechanism we use to try to pick out the highest leverage examples for them to take a look at.
• 22:55
  ~ but aside even if we didn't, like the point is is issue corrections when you see problems.
• 23:00
  ~ they issue corrections and the main thing that I think we ask for that I've never seen anyone else ask for is we ask, hey, what what was the right decision?
• 23:10
  ~ and why?
• 23:12
  Importantly, why?
• 23:13
  What is your judgment about this?
• 23:15
  Like what leads you to that belief?
• 23:17
  ~ and ultimately that all goes into our AI.
• 23:21
  ~ and the AI uses that r reasoning to update the policy that is written.
• 23:28
  That policy then gets what we call back tested.
• 23:31
  So essentially we say, what let's pretend what if this new version was active for the last week or the last month?
• 23:40
  ~ and then we can show the controlling user, hey, ~ if you had had this new policy active, here's what would have changed.
• 23:49
  Right?
• 23:50
  And that gives them information.
• 23:52
  Now they can make a coherent decision about, hey, ~ is is this better?
• 23:56
  Does it in fact incorporate my judgment?
• 23:57
  Or maybe it fixed some things but broke others, and we need to make another correction.
• 24:03
  ~ the point, I guess, is that we're storing
• 24:07
  the examples as part of the decision memory, but it is my contention that we never want hidden state.
• 24:14
  That for AI security to actually work into the future, it needs to be auditable and explainable.
• 24:22
  And so it can't be that it's like hidden weights in a model.
• 24:25
  And which is why we always try to translate the reasoning and the examples back into something that's written down in the policy where someone can see it.
• 24:35
  or at least they can see that, hey, this example is attached and this was ~ this was the decision that should be made in these cases to reference the legal system again.
• 24:43
  Like here's the scope to which this example applies.
• 24:47
  It doesn't apply to all cases, it only applies to cases that have this context.
• 24:51
  ~ you know, here's how severe this case is, all all that stuff.
• 24:55
  ~ I I I think that was one of the hardest things for us to figure out how to do, and I will admit we had some really good design partners.
• 25:04
  That we got to work with, ~ who I think not only helped us with the data that we needed to train that system to do what it needed to do, but also taught us about like what are
• 25:16
  customers willing to contribute to this process?
• 25:19
  Again, to like go back to my, you know, the best security system is one that people will use.
• 25:24
  ~ so I think ~ certainly I think that was ~ one of the most challenging things to figure out.
• 25:30
  And to be honest, we're still figuring it out.
• 25:31
  We're still figuring it out every day.
• 25:32
  We're still
• 25:33
  trying to improve and ~ you know we're now handling ~ video as well because AIs generate video so we need to be able to process video and ~ you know getting this getting the same
• 25:44
  process right for an element for a a ~ an artifact in video that now has a time access to it was a whole nother process.
• 25:54
  So ~ definitely pretty complicated ~ stuff.
• 25:58
  What are the the biggest ~ areas that companies come to you with or that are that are, you know, they get most excited about you being able to address?
• 26:07
  what what what kind ~ I mean, I I'm sure it varies industry to industry and company to company, but do you see a pattern right now about you know what is the biggest driver?
• 26:16
  So I do think a lot of our customers, we learned this pretty early.
• 26:19
  You know, we're a vendor, and I think many of them want us to be the experts.
• 26:24
  And I sort of understand that impulse.
• 26:26
  It's like I actually don't think every single engineer, every single product manager who is trying to deploy AI should have to be an expert in AI security and safety.
• 26:36
  And we ended up developing this thing that we we just call it label library.
• 26:40
  and it it basically has 60 plus policies that
• 26:45
  we've identified as kind of the the lowest common denominator of security and safety problems that generally I'm not gonna say every single one applies to every single
• 26:55
  customer exactly the same.
• 26:56
  I you know, I think a lot of my argument is that business rules differ.
• 27:00
  ~ but ~ you know one of the things that we offer that I definitely do see customers kind of get excited about is plug us in on day one.
• 27:08
  We're fixing kind of a lot of the the baseline stuff.
• 27:14
  ~ so I I definitely think that's that's one thing that we've seen customers ~ just ~ appreciate.
• 27:21
  ~ the the other area that we have seen a lot of, frankly, is ~ customers coming to us kind of hair on fire ~ because their payment providers ~ are threatening essentially to cut
• 27:35
  them off to say, look, we just can't underwrite your business anymore.
• 27:38
  ~ we've seen that a lot in the AI image and video gen space.
• 27:45
  and I think it's starting to change now with the with the EU ~ with the ~ EU governance that finally got released and stuff.
• 27:52
  But for quite a while, ~ I really think the payment providers were kind of operating as this de facto standards and regulatory body, perhaps.
• 28:01
  ~ and I think understandably, like they obviously do not want to be underwriting the generation of illegal content, basically.
• 28:09
  ~ and I think our ability
• 28:12
  in many cases to draw extremely fine lines, especially as customers engage in the feedback process, ~ has been I I think a real strength of ours, at least in in that case.
• 28:25
  ~ with a lot of our ~ sort of real more real-time applications, ~ not that that's not real time, but you know, in cases where there is a human who is interacting with an AI and the
• 28:36
  expectation is that the AI does not take forever, basically.
• 28:41
  I think a lot of
• 28:42
  ~ customers have appreciated and and been pretty excited about how quickly we can do our work and and how accurately we generally ~ can do it.
• 28:52
  There is one thing that we're rolling out right now that I am very excited about and we have a couple of customers testing for us.
• 28:59
  And it it's basically ~ I've called it active guardrails in the past.
• 29:05
  ~ but essentially the idea I'm not a big believer that that the
• 29:12
  That allow and block or allow and deny should be the only two actions.
• 29:15
  I I think actually it's like a false binary there.
• 29:20
  ~ and I think what it ends up doing, and and we've actually heard some of the, I don't want to call anyone out by name, but we've heard some of the cases with AI friends and
• 29:30
  girlfriends where this where, you know, we've heard a CEO say, look, you know, we didn't implement as many safety protocols as we could have ~ because we were worried about the
• 29:40
  user experience.
• 29:41
  ~ and I think I I'm not gonna say it's right, but I think it's it's understandable, perhaps.
• 29:48
  ~ if the only actions available to you are allow and block, and the cost of a block is that the user experience breaks, basically.
• 29:57
  ~ so we have essentially built something where now there's ~ there there's a there's four actions basically.
• 30:06
  There there is allow, there is deny, of course.
• 30:08
  ~
• 30:10
  There's escalate.
• 30:10
  We can escalate to a human because I do think there are cases where we should say, hey, we don't know the right answer here, and this one needs to get escalated.
• 30:17
  ~ I think that's reasonable.
• 30:19
  ~ but we also now have what we call rewrite or transform.
• 30:22
  And the idea is, well, we know what your policy is supposed to be.
• 30:26
  So if an AI is about to do something ~ or say something dangerous, ~ or you know, maybe just not so nice, ~
• 30:38
  Why can't we rewrite that output before it gets to the user to let the experience continue, but make the experience compliant at the same time?
• 30:47
  ~ and I think at least recently, last couple of months, since we only built built the infrastructure for this recently, like that has been a a common area of excitement that
• 30:57
  we've seen.
• 30:58
  I mean it makes a lot of sense why that would be valuable, but I would assume that's also somewhat scary that that a company is turning over to you ~ that ability to you know
• 31:08
  essentially silently rewrite what's happening.
• 31:10
  yeah.
• 31:11
  Yeah.
• 31:11
  And I think this is why, you know, it's still in beta.
• 31:13
  It it hasn't been, you know, widely rolled out yet.
• 31:16
  ~ there's definitely going to need to be ~ additional controls.
• 31:24
  Yeah.
• 31:25
  ~ obviously we now take on responsibility for making sure that whatever we do output is also compliant with ~ the policy.
• 31:33
  ~ so it definitely is ~
• 31:38
  For us an active area of like development and and research, and I I wouldn't I wouldn't claim it's ready yet.
• 31:43
  Yeah.
• 31:44
  ~ but I think my argument is just that if we can make this work and make it work well, ~ that it will encourage more more companies that are deploying AI to put the appropriate
• 31:59
  controls in place, Facebook.
• 32:02
  Yeah, it it sort of raises for me as as a lawyer the the this tension.
• 32:06
  And it it's also true what you were describing before in terms of the the auditability and explainability of of how you're managing the policies, that, you know, on the one hand,
• 32:16
  you know, organizations want to be accurate and they want to be able to, you know, have a set of rules and a you know and make a set of decisions.
• 32:24
  the the fear is always the more you have this audit trail.
• 32:28
  then the more that can be the basis that ~ a plaintiff or a regulator says, look, you knew exactly what you were doing and you're never going to get it right a hundred percent of
• 32:36
  the time.
• 32:37
  ~ so, you know, obviously, you know, these are open legal questions, these AI responsibility questions and the regulatory system is is still evolving.
• 32:45
  But how does how does that factor into what what you're doing with companies, that concern?
• 32:51
  I mean, I I I I haven't had anyone complain about the auditability and observability yet.
• 32:56
  ~ I certainly understand the the concern.
• 33:02
  ~ but I think generally like our because there is no maybe it will change, you know, as I'm a big believer actually that that the industry needs regulation.
• 33:13
  ~ and importantly I think standards that are measurable and falsifiable.
• 33:20
  in some way.
• 33:21
  ~ it's up to like the standard to to define ~ you know what we're looking for.
• 33:28
  ~ it's up to the deployer to then decide what to do, I guess, when our standards are violated.
• 33:33
  ~ but
• 33:38
  Right now I think most of our auditability tools are actually being used by CISOs and CTOs to sort of understand the the the way the AI is behaving, what failure modes exist, how it
• 33:49
  needs to be improved, how many ~ control events are are actually occurring.
• 33:56
  ~ I don't know, you've kind of got me on this one.
• 33:59
  Like I I I I admit I haven't I haven't thought enough about, you know, the potential like fear in companies around, well now I have this giant audit trail.
• 34:08
  My what I will say is like my hope is that we do get regulations that dictate like what what are the standards and what disclosure requirements are there, basically.
• 34:21
  Right.
• 34:22
  ~ because certainly once that exists, we will implement it.
• 34:25
  And then it's not a matter of like, well, I'm afraid of of having auditability because I could end up in a in a problem.
• 34:32
  ~ now it's just okay, these are the requirements.
• 34:35
  So this is ~
• 34:37
  what I want to do.
• 34:38
  And I also believe it's important, like one thing we have thought a little bit about is, you know, how could our tools be misused ~ on in s in s on some level, right?
• 34:47
  Like, you know, we don't define the policy.
• 34:50
  The customer defines the policy.
• 34:51
  ~ and you could, I guess, I I don't want to give anybody I any ideas, but like you you could use our tools to make your AI really bad, just as you could use your tools to make
• 35:05
  it really good.
• 35:07
  And it's part of the reason why I do think disclosure is important because ~ the only control for like a company that's a bad actor is sunlight.
• 35:18
  It's visibility, right?
• 35:21
  and so like to me that that is why I I I really would like it if you know the government would come along and say, look, here are the stand-ins.
• 35:29
  This is what you have to do.
• 35:31
  ~ because otherwise it's a free-roll.
• 35:34
  I was and I know in one of your recent episodes, I think.
• 35:38
  w what was it like around ten percent of companies actually have like a governance framework or something like that versus around fifty percent who have you know who have a
• 35:46
  strategy for AI.
• 35:47
  Like that's a that's frightening to me that like
• 35:50
  There's a yeah.
• 35:51
  Yeah.
• 35:51
  No, and and I I'm glad to hear you say that because I yeah, there there's always this knee-jerk reaction that regulation is bad and certainly there's you know, there's lots of
• 35:58
  problematic regulation.
• 36:00
  But when I talk to companies, especially on on the deployment side of AI, you know, they don't want to be in this situation where you just don't know what the right answer is and
• 36:10
  whatever they do, someone can say you got it wrong.
• 36:12
  So so I think that's an important insight that you have about the the value that that can create for.
• 36:17
  Yeah.
• 36:17
  I mean, I'm sure as a lawyer, like ~ I guess this is more your area than mine, but I I I you know, now that you brought it up, it's like I'm sure it it is problematic to feel like
• 36:27
  these are non deterministic systems.
• 36:28
  You're never gonna get it right a hundred percent of the time.
• 36:31
  It's it's a it's an you know, we can strive for perfection, but everybody needs to accept like we're not gonna get there.
• 36:38
  and I can certainly see where in that environment,
• 36:43
  It's like, well, the more information that I have, like the more risk there suddenly is that you know, I get caught doing something that I didn't mean to do, but that happened
• 36:53
  anyway.
• 36:54
  I mean, I think the advice I would give companies is is write standards that are actually measurable, be clear about what your tangents are, make them measurable.
• 37:02
  ~ we have a little test we do basically where we say, like, I'm actually about to run this exercise at the end of the month at at Harvard.
• 37:10
  ~ but
• 37:12
  if you write down a a rule and you show that rule to N people and you get different answers, it's not consistent enough yet.
• 37:21
  Like it's it's that simple.
• 37:22
  ~ so write standards that are actually measurable, that can be enforced on ~ consistently.
• 37:28
  ~ decide what your actions are when standards are violated, and then enforce those standards consistently.
• 37:35
  And and I know that sounds like kind of simple, but but I think like where where I think
• 37:42
  meta got into trouble years ago was not being consistent yeah about their enforcement, about their actioning.
• 37:48
  Right.
• 37:48
  That's when pe I think everyone can deal with tran with with the fact that some controls are necessary ~ and that people make mistakes if they're being transparent about it.
• 38:00
  Yeah.
• 38:00
  And they are at least, you know, doing their level best to enforce their own standards consistently.
• 38:07
  that's just my that's like my take and sort of the advice I would I would give anybody.
• 38:10
  Yep.
• 38:11
  Terrific.
• 38:12
  Well, we have to wrap up, but ~ it's been a fascinating conversation and Brett, I really appreciate your joining me.
• 38:17
  No, thank you.
• 38:18
  It's ~ it's been wonderful to talk.
• 38:21
  This has been the Road to Accountable AI.
• 38:23
  If you like what you're hearing, please give us a good review and check out my Substack for more insights on AI accountability.
• 38:30
  Thank you for listening.
• 38:33
  ~ If you want to go deeper on AI governance, trust, and responsibility with me and other distinguished faculty of the world's top business school, sign up for the next cohort of
• 38:43
  Wharton's Strategies for Accountable AI Online Executive Education Program.
• 38:48
  Featuring live interaction with faculty.
• 38:51
  Expert interviews and custom designed asynchronous content.
• 38:54
  Join fellow business leaders to learn valuable skills you can put to work in your organization.
• 39:00
  Visit execed.warden.upen.edu slash acai for full details.
• 39:07
  I hope to see you there.
• 39:08
  ~