Latest / The Road to Accountable AI / Trey Causey: Is Responsible AI Failing?

Transcript

Trey Causey: Is Responsible AI Failing? · The Road to Accountable AI

• 0:01
  Trey. thanks so much for joining me on the road to Accountable AI.
• 0:07
  What would you say is the biggest thing that people on the outside don't understand about the practice of responsible AI or AI governance?
• 0:18
  Yeah. I think there's really two things that kind of intersect actually. and I guess for one.
• 0:22
  The first is this is just. you know. brand new technology and exposing kind of problem and you can say like impact services or problem areas in ways that we've never seen before.
• 0:32
  So. you know. before when you were building traditional software. if you wanted someone not to do something in your software. you just wouldn't enable that functionality.
• 0:40
  And now you have these general purpose models like LLMs that are powering things that by design enable kind of. you know. whatever you would like to do.
• 0:48
  The other thing I think is there's so much attention paid to AI right now and the dialogue is so heated and charged and there's a lot of polarization and there's a lot of
• 0:58
  expectation that we are further along in how to regulate and govern and use this technology than we are.
• 1:05
  And so those two things together is kind of a potent mix. right?
• 1:07
  And I think a lot of people outside don't understand that we're working really hard to make sure we're developing and deploying safely here. but it's really brand new territory.
• 1:18
  What can you do to address those fundamental problems that it's such a general purpose technology and some of these systems are non-deterministic?
• 1:27
  Yeah. it's a great question.
• 1:28
  mean. I think uh some of it is. is unknown still. right?
• 1:31
  We have this. you know. we have a lot of guardrails we can build into.
• 1:35
  can build technical solutions. observability and uh moderation. detection of bad behaviors. things like that.
• 1:43
  And of course there's the political solution of regulation and so on. we actually don't know how to perfectly deploy these systems yet.
• 1:51
  uh And so in a lot of ways. like we have to think about what
• 1:55
  how we design the software. who's involved in designing the software. who's involved in building it. how we kind of take into account humans and how they use software throughout.
• 2:03
  It's a software development lifecycle problem kind of end to end.
• 2:06
  But even at the end of that. you can be the most careful developer. the most well-governed and kind of accountable organization. and you're still fundamentally dealing with models
• 2:14
  that are quite unpredictable sometimes.
• 2:17
  You wrote an essay recently. which is the reason I wanted to talk to you now. that was called Responsible AI is Dying. Long Live Responsible AI.
• 2:25
  And I found a lot of what you said there really compelling.
• 2:27
  Let me first just ask you what motivated you to put out that piece.
• 2:32
  Yeah. it's actually something I've been stewing on for a while.
• 2:35
  I recently left a role as the head of Responsible AI for Indeed.
• 2:39
  And I didn't want it to seem like it was a commentary on Indeed because it wasn't.
• 2:44
  was a commentary kind of industry-wide.
• 2:46
  But through talking to my peers in Responsible AI and AI governance and through watching how these things unfold. uh I'm just getting the sense. the growing sense that we haven't
• 2:58
  yet figured out how to...
• 3:00
  really build robust. responsible AI organizations. or if that even is a thing we should be building to answer your previous question around how do we do this responsibly and
• 3:09
  effectively?
• 3:10
  I think what I saw was really large variance and influence of organizations across industry.
• 3:18
  We saw really differing approaches to governance. sometimes very compliance and regulatory focused. sometimes very much more on the kind of humanistic and ethical and kind of social
• 3:28
  justice side of things.
• 3:30
  And regardless of those approaches. what I often saw was a conflict between the business and the intent to kind of govern the space.
• 3:39
  And I mean. that's the story is all this time. obviously.
• 3:41
  But I think because this space is so new and so much is being asked of these small organizations. these responsibly I organizations. I think it was. I wanted to kind of. you
• 3:53
  pull back the curtain a little bit to show like. hey. there's a lot of problems here for how these things are moving.
• 3:57
  And I'm not sure that we've settled on the model yet for how to be effective.
• 4:01
  I still fundamentally believe in the missions of Responsible.
• 4:04
  I still think it's something we absolutely have to do. but I'm watching as companies divest from their OpenAI. from their Responsible teams and as they kind of downsize and.
• 4:14
  you know. I kind of jokingly said it in the piece. but I meant it like.
• 4:18
  It's very rare that people that have AI in their job title right now are getting laid off. but that's not true for responsible AI organizations.
• 4:24
  so. repealed preferences and so on.
• 4:27
  mean. it seems like either the value is not being seen or it's not being seen as important.
• 4:31
  So. why don't you maybe start a conversation around that.
• 4:34
  And here we are.
• 4:35
  think you did.
• 4:36
  Absolutely.
• 4:36
  It's really important one.
• 4:37
  I want to get into the specific points you make. but let me first just ask about something you said that there is this tremendous variance in terms of how organizations are going
• 4:45
  about this activity. how they talk about it. how they describe it.
• 4:49
  Is that a feature or a bug. especially given the newness of it?
• 4:53
  Shouldn't we have a lot of experimentation as opposed to saying we know the one way for every company to do responsible AI?
• 5:00
  Absolutely.
• 5:01
  I think that's really. that really is a feature.
• 5:03
  And the reason I even highlight it is when I see that they're kind of equally being hit in terms of downsizing and kind of lack of influence. which kind of suggests to me that it's
• 5:12
  not necessarily the organizational model or approach that is the. that's kind of what's driving the disconnect that I'm seeing.
• 5:18
  But. but I do think you're right.
• 5:19
  Like we haven't landed on it is brand new.
• 5:21
  uh There's a lot of competition and the kind of marketplace of ideas in this space.
• 5:25
  So you have the AI safety folks.
• 5:27
  who tend to be a little more focused on existential risk and super intelligence.
• 5:31
  have the kind of very. you know. AI ethics folks. which tends to be more academic and very much focused on systemic social issues.
• 5:37
  I think responsible AI floats in the middle somewhere with. you know. uh security and privacy touching those pieces as well.
• 5:44
  And so I do think a variety of approaches will be needed.
• 5:48
  uh I also think when budgets are up and reviewed and hiring and laying off is done. one of those viewpoints is going to win ultimately.
• 5:57
  I'm not sure what it is.
• 5:58
  Yeah.
• 6:00
  The first point that you make is about technical competence. or at least the lack thereof in a lot of responsibility I teamed.
• 6:06
  Why is that so important?
• 6:08
  Yeah. you know. this is actually something that I was reluctant to believe for a while.
• 6:12
  Uh. it's something that I like personally had a bias for because I come from a data science background. I worked in machine learning engineering and kind of. and coming from
• 6:20
  that.
• 6:20
  So in cognitive in the fact that I might just be saying like. Hey. you should do what I do because that's right way to do it.
• 6:25
  Uh. but. but I think what I ended up seeing in my own. uh. kind of work as well as talking to colleagues is being able to really work closely with the engineers and product teams
• 6:36
  that are developing. deploying those models.
• 6:38
  is it like being able to help build the solutions to build those things responsibly is really important.
• 6:44
  Being able to kind of pitch in at their level. speak their language. et cetera.
• 6:48
  Like not only is there like a credibility component to that. which is real. don't get me wrong.
• 6:52
  Like there's also just a. hey. I know you can't prioritize this because you're trying to build the features that are going out right now.
• 6:57
  So let me help you by building the scaffolding that you can ship these things with.
• 7:01
  Let me help you by building the testing and evaluation frameworks that you can ship these things with.
• 7:05
  And then that's going to help you ship faster and more confidently and so on.
• 7:08
  And that was actually what really landed.
• 7:10
  And when I've talked to some of my peers in the industry and the responsibility industry. they would often say like. wow. that's. like a really different approach to go that way
• 7:19
  rather than kind of other ways.
• 7:21
  And really impressed to see how you've made it happen.
• 7:23
  Cause I think there was some skepticism that we could do that.
• 7:26
  And I'm proud of that.
• 7:29
  What was your pathway to get into doing this kind of work?
• 7:33
  Yeah. you know. it's interesting.
• 7:34
  I kind of stumbled into it.
• 7:36
  So I was a sociologist originally and I did computational sociology. which is something not a lot of people know about. but you know. I use natural language processing to answer
• 7:45
  social questions.
• 7:46
  And that was 15-ish years ago now.
• 7:49
  I was in my PhD program during 2008 as an academic.
• 7:52
  You probably understand the implications of that for academic jobs.
• 7:56
  And luckily at the time. this thing called data science was popping up and I said. hey. like I'll give it a shot.
• 8:03
  And so actually didn't do much with responsible AI for a while.
• 8:09
  And then a gentleman that I managed. Dan Tabor is his name. he's now at Spotify. brought it up as an idea to Indeed.
• 8:17
  we said. yeah. this is great.
• 8:18
  And this was six years ago. five. six years ago.
• 8:20
  And he said. great.
• 8:21
  And he ended up leaving.
• 8:22
  And it was one of those situations where I was like. you know. I think. I think I have a lot to offer here.
• 8:26
  And if nothing else. I know how to build teams and how to start things up new.
• 8:29
  And then it just kind of became the more I did it.
• 8:32
  the more I realized this was something that I could offer and I could drive at the company.
• 8:37
  I understood the company well enough to have the influence to have like the seat at the table and make things happen.
• 8:44
  So a little bit of a full circle. know. like I spent a lot of my life thinking about things like inequality and social change and things like that and sociology.
• 8:53
  Didn't think about it for a while and then kind of came back full circle.
• 8:57
  And talk a little bit more. you alluded to some of the things that you were able to do at Indeed in terms of actually bringing that technology expertise in.
• 9:06
  Yeah. sure.
• 9:06
  So. uh. you know. we had this kind of model of working. was we had data scientists and engineers and researchers on our team.
• 9:13
  They would kind of approach things as research in some ways. like how would we identify bias in a model or how would we. how would we moderate the output of an LLM or something
• 9:21
  like that?
• 9:21
  And kind of would build in a researchy way. the code to do that.
• 9:25
  And then we kind of proved that it worked and it was like. okay. well we have hundreds of models.
• 9:29
  Uh. we probably need to do this in an automated way.
• 9:32
  So let's automate these analysis that we did.
• 9:34
  And then once that kind of worked. we say like. you know. this could probably be something that we could ship to the rest of the company and like through an API or something. you
• 9:40
  know. a library they can. they can embed.
• 9:42
  And then it would go to that.
• 9:44
  So we kind of just continually did that research automate scale.
• 9:49
  And so we had. you know. a variety of things bias analysis for sure.
• 9:54
  Uh. LLM guard rails and moderation for sure.
• 9:56
  Um. all of those things kind of working at really large scales with the kind of data that you see at a large internet company.
• 10:03
  And that was generally our approach for almost everything we did.
• 10:06
  Now we did a lot of like one-off analysis and. you know. advising and things like that as well.
• 10:10
  But anytime we could. we thought of how can we turn this into a tool that if you were an engineer. you could just reach for and use.
• 10:18
  Isn't the endpoint though that those become tools that someone just builds and sells into companies as opposed to every responsible A group having to build it themselves?
• 10:28
  Yeah. absolutely.
• 10:29
  I mean. I think there is a lot of that.
• 10:31
  And don't get me wrong.
• 10:31
  I looked at plenty of vendors and we engaged with vendors and bought their tools sometimes too.
• 10:36
  know. every tech. it's like an anti-coronavirus problem.
• 10:39
  Every tech company has their own infrastructural issues that require uh specific integrations.
• 10:44
  And so sometimes it's easy to build things yourself.
• 10:47
  But it was also true that when we were doing it. there weren't a lot of vendors selling these things.
• 10:52
  And so we were kind of first movers.
• 10:54
  And it was also the case that the things that were being sold were very much
• 10:57
  not at the level of rigor that we wanted necessary for work.
• 11:01
  And because indeed is a. know. jobs. you know. matching employment marketplace. like that's a highly regulated space.
• 11:07
  And so a lot of times the level of rigor we wanted to achieve just to even kind of operate in that space was not something we could ask a vendor to do necessarily.
• 11:18
  Is there anything that's fundamentally different though about AI versus other kinds of technology risk and governance that companies deal with cloud computing and software as a
• 11:29
  service and data governance and so forth?
• 11:32
  Is this a. you think an AI issue or should all of those be more technically driven?
• 11:38
  Well. if I look at the kind of really successful places where those things are run. you kind of have the two options.
• 11:43
  have the compliance focus. which is often. you know. lawyers and policy experts and so on.
• 11:48
  And then you have the implementation side. which is often engineers.
• 11:50
  So if I think of like. you know. DevOps became SRE in production engineering at places like Meta and Google and so on.
• 11:59
  It's actually pretty similar trajectory.
• 12:01
  would say is that we actually were more successful when we found we had engineers who specialized in this area.
• 12:07
  and we have privacy engineering now and you know. all these things too.
• 12:10
  I don't think one obviates the other.
• 12:14
  You absolutely still need the legal and compliance side of things.
• 12:18
  The other thing I would say though too is we sometimes use legal and regulatory and governance and accountable all as kind of synonyms.
• 12:27
  They all mean different things.
• 12:29
  And I don't have to tell you as someone who went to law school. but lawyers are there to protect the company and make sure they're not breaking the law.
• 12:35
  They're not there to...
• 12:37
  advocate for user interests or make sure we're following best practices or anything like that.
• 12:40
  um You know. they have enough on their plate already.
• 12:42
  And so having that domain expertise is also really important.
• 12:46
  um But I think TBD. how much AI is really different. right?
• 12:52
  Like I do think the non-deterministic part of it is pretty large.
• 12:56
  And I think there's a level of certainty that maybe you can get in other domains that you can't get an AI.
• 13:02
  uh
• 13:03
  And we're expecting it. like I said. we're expecting it faster than we did in other domains. but you know. that could all change.
• 13:08
  It's a very fast moving field.
• 13:10
  yeah. absolutely.
• 13:11
  So second point you make in the essay is that the failures of responsible AI people see and the successes they don't see.
• 13:18
  You don't get credit that you prevented something bad from happening in a company.
• 13:22
  So I guess I had the same question.
• 13:24
  Why is that a particularly AI problem?
• 13:27
  don't think it's actually particularly a problem.
• 13:29
  think that that actually is a problem or. you know. potentially uh a uh characteristic of any kind of governance organization.
• 13:37
  I think. and maybe this gets two points later in the essay. responsible AI is a field kind of positioned itself a little differently. I think. than potentially security. privacy.
• 13:46
  other functions that they position themselves as more proactive as like. we're going to make the software better for people. we're going to instill the right. you know. values
• 13:55
  and principles and so on.
• 13:56
  so
• 13:56
  That kind of proactive stance in a way is their own undoing or our own undoing in a lot of ways. because I think you're like. well. what are you doing to make it better then?
• 14:04
  Like. it sounds like. you know. this bad thing happened.
• 14:05
  You didn't stop that.
• 14:06
  And that's of course an unfair comparison because you can't see the good things that were prevented.
• 14:11
  But I do think positioning yourself as. the. we're going to make things better group can be a challenge.
• 14:17
  Does that make sense?
• 14:18
  Yeah. absolutely.
• 14:20
  Are there metrics or KPIs or something that a responsibility group can use to service more of the benefits they're creating?
• 14:29
  I think the closer you can get to the core business metrics. the better off you're gonna be.
• 14:33
  Like I don't think. you know. bad things avoided is a metric you can really track and it's. you you can't measure a counterfactual sometimes.
• 14:42
  I kind of jokingly one time asked if maybe we could track the number of. you know. the amount of money we would lose and find if we were found in violation of a law.
• 14:50
  And I was told I couldn't do that because that might be seen as we were thinking about not complying with the law and we didn't want that to happen.
• 14:55
  And we never were thinking about not complying.
• 14:57
  yes.
• 14:58
  Yes. exactly.
• 14:59
  uh But that up and to the right metric is like. is really hard to find for a lot of things.
• 15:04
  And so I do think that's in the kind of OKR. KPI driven world that we're in right now.
• 15:11
  That's a real challenge.
• 15:12
  And if you look at other kind of hard to measure functions like design. design has struggled with this for a long time. right?
• 15:19
  What's our up and to the right?
• 15:20
  Like. yes. we can measure revenue and so on. but we don't know how much of that was because of the design and so on.
• 15:24
  uh And they've kind of faced similar uh
• 15:28
  struggles. although we all have a kind of intuitive sense that design matters for product development and so on and so forth.
• 15:33
  But it took a long time to kind of make that case.
• 15:36
  Is the answer then for a responsibility eye group to just reframe to not have the perception be that that's what they have to do?
• 15:44
  It's a good question.
• 15:45
  think that might be part of it. right?
• 15:46
  I think in some sense there is a line in the sand that your company's leadership draws to say. we believe in doing what's best for our users.
• 15:55
  We believe in not being biased.
• 15:57
  We believe as much as possible.
• 15:58
  We believe in trying to make all the right choices that we can.
• 16:03
  uh And if there's not a regulatory uh impetus for them to do that. then it really does become what's the company's values and how are you going to embody those values?
• 16:13
  Sometimes that is just what it is.
• 16:15
  I'm not going to say that's going to be everywhere.
• 16:17
  And maybe it is also the case that. you know. we will find those metrics.
• 16:20
  My. my old organization. we would celebrate our ships of tools and. and APIs and everything. just like another engineering team would say. like. you're the things we
• 16:27
  delivered.
• 16:28
  ah And of course there are steps removed away from revenue and so on. but that was a little easier. I think. for leadership to grok then we. we made sure that we're. we're
• 16:37
  acting. you know. ethically. and that's. can be hard to swallow.
• 16:39
  I think from a financial standpoint. sometimes.
• 16:42
  Yeah. absolutely.
• 16:44
  The third point that you make. and I love the title for this section. was epistemic overreach and the lack of product market fit.
• 16:50
  And especially for me as a business school professor. uh I know people who use terms like epistemic and I know people who use terms like product market fit. but they usually have
• 16:59
  the same person.
• 17:00
  So I'm really curious how you thought about and put those two things together.
• 17:04
  It's a funny point when you put it that way.
• 17:06
  I hadn't thought about that.
• 17:07
  I think sometimes when you're kind of like deep in the AI community. it is one of like the more academic crossover communities that you find in tech.
• 17:13
  And so like there are a lot of PhDs and former non-PhDs and grad school dropouts.
• 17:18
  And so I think you hear these kinds of words more than you might hear in normal kind of engineering circles.
• 17:25
  Also. I think because there's just been a lot of ink spilled about what the epistemic foundations of responsible AI and AI safety and AI in general are.
• 17:34
  I have random people who I think five years ago wouldn't know what AGI is. asking me what I think about specific arguments about AGI sometimes.
• 17:43
  And so I think these words are leaking into uh the zeitgeist as a result.
• 17:48
  But I think they're actually very related for me. right?
• 17:51
  Like getting back to that. here's what responsible AI said it was gonna be. and here's what we said we were about and the problems we were gonna solve.
• 17:58
  And it touches every part of the business and its design and its engineering. its privacy. its safety. and so on.
• 18:04
  ah That's kind of the epistemic overreach thinking that one group can kind of set the tone across many other disciplines.
• 18:11
  I am speaking very broadly here.
• 18:13
  Like I don't think there's one person sitting out there saying like. I think we're going to do all this. you know. painting with a broad brush.
• 18:18
  ah And then at the same time. there wasn't a demand for that necessarily.
• 18:21
  ah And so like a lot of companies. like they don't need an overall person to tell them if things are going in the right direction or not across the board.
• 18:29
  They need them to say like. here's what we need to fix.
• 18:31
  Here's how we're to fix it.
• 18:32
  here's how we're gonna develop it.
• 18:34
  And so I think when we were in the zero interest rate phenomenon era of a years ago. it was a very lofty thing to have those kind of broad thinking. very ambitious. epistemically
• 18:47
  ambitious groups.
• 18:49
  And that became harder to swallow. I think. when money was no longer free.
• 18:55
  Yeah. does that mean then. again. is the answer you create a responsibly i group that just very narrowly defines its function?
• 19:04
  I think the answer is maybe the opposite of that. which is you try and embed responsibly eyes of many functions.
• 19:09
  and I think. you know. that's. you can't do that just. you from. you can't start from zero there.
• 19:17
  Like you probably do have a responsibly I group that sets the tone and helps drive things.
• 19:21
  But I think asking that one small group of people to kind of own that entire space as companies scale. as they're building more. as AI enables them to build even more.
• 19:29
  Uh. I think that's unrealistic.
• 19:31
  And so I think it comes down to how do we.
• 19:33
  going back to the values part. like how do we make sure we have a set of values?
• 19:36
  These are the things we believe about what we should and shouldn't do with AI.
• 19:39
  And here's how we bake that into how we develop software.
• 19:43
  Now. other disciplines have done this successfully.
• 19:46
  mean. human centered design is a good example of this.
• 19:48
  Like I think even mobile first is a good example of this.
• 19:51
  I I was at Facebook not long after the mobile code read when. you know. it was decided like. my God. mobile is actually a thing. more than half our traffic's here.
• 20:00
  And like now everybody gets an Android feature phone to try and figure out like.
• 20:03
  how to use Facebook on it.
• 20:04
  ah And so I do think it's possible.
• 20:07
  It's just we're like with everything with AI. we probably shouldn't be having this conversation yet because like the industry so new and it feels so new and yet it feels
• 20:16
  like it's overdue at the same time.
• 20:18
  And so I think that's kind of where we're at.
• 20:20
  And I think to the epistemic overreach part. if you make the responsibility. not to overlay the term. of how AI goes at a company and the kind of purview of a small group of
• 20:30
  people. that also kind of like.
• 20:32
  divest the agency of the rest of the company to also have a hand in that too. you know. and they feel like I'm going go to them. make sure they sign off and then I'm going get
• 20:40
  back to doing my thing.
• 20:40
  And so I think there's actually kind of a counterintuitive effect there of. of they're watching it.
• 20:45
  I don't need to worry about it.
• 20:46
  Let's let me just do my thing.
• 20:49
  I hear this a lot from people in this area. which makes a lot of sense that ultimately responsibility needs to be driven throughout the business.
• 20:58
  It can't just be up somewhere in one group.
• 21:00
  But there's not a lot of people out there like you who have data science expertise. sociology expertise. and thinking about these big normative questions.
• 21:09
  So are there things companies can do to really promote that activity throughout the company?
• 21:15
  Yeah. this is tough enough to crack.
• 21:17
  feel like on the one hand. I feel like there's lots of resources out there. right?
• 21:20
  Like so in the hiring space. Indeed got together with LinkedIn and ADP and Workday and the Future Privacy Forum and actually produced a set of best practices for AI and HR.
• 21:29
  Said. here's. you don't have to hire a responsible team to come up with these. like. here you go.
• 21:33
  And to get Indeed and LinkedIn and ADP and Workday to all agree on something is like no mean feat. right?
• 21:39
  So I think those kinds of best practices exist.
• 21:43
  I do think...
• 21:44
  it's harder when you start evaluating vendors who are saying they're going to solve your problems for you. or you don't incentivize acting responsibly within your performance
• 21:54
  management system or something like that.
• 21:56
  so there really is a. there's like a cultural component. there's like an incentive component. and there's also an education component.
• 22:02
  And all when you hear those three words together. you don't hear like quick change.
• 22:07
  You're like. oh. this is going to take a long time to implement and so on.
• 22:10
  That doesn't mean it's not the right choice though. I think.
• 22:15
  As you mentioned. employment is a very regulated area and it's an area where there's been a whole lot of attention for a long time to questions about. with regard to AI and
• 22:25
  analytics. bias. explainability. and so forth.
• 22:28
  Can you talk about any of the more distinctive things that you were involved with and indeed in terms of doing AI governance in that sector?
• 22:37
  Yeah. absolutely.
• 22:37
  I mean. we had a very rigorous evaluation procedure for our models.
• 22:41
  uh Before there was any regulatory burden to treat AI any differently. we had bias evaluations in place.
• 22:49
  We asked job seekers to self-identify with their demographic information so that we could use real data. evaluate how these models were behaving with respect to various groups. uh
• 23:00
  and really did a really rigorous practice.
• 23:03
  So like to give you a sense for how rigorous that was for every bias analysis we did.
• 23:07
  we would write up a design document.
• 23:09
  That design document would have what's our definition of bias. because as it turns out. it means a lot of things.
• 23:15
  How are gonna measure it?
• 23:16
  What are the decision criteria?
• 23:18
  Like. how will we know if we've detected bias or not?
• 23:21
  How are we controlling for false positives?
• 23:23
  How are we setting kind of power and things like that appropriately?
• 23:26
  And then everybody involved. like legal. us. the product teams. would all sign off on that before we did any analysis. any work. any data collection as well.
• 23:34
  So I was kind of said like.
• 23:36
  You can't. you know. bet on roulette. like after the wheels stopped turning.
• 23:38
  And so you also can't define bias after you've started looking at the data.
• 23:42
  And so we always took like a really rigorous approach to like. we're all agreeing this is what we're going to do if we find X. Y. or Z.
• 23:49
  And I felt that that was really like. it's not innovative. it's just science. but like. you don't find a lot of like pure science sometimes in industry.
• 23:57
  And so I was really proud that that was the approach we took.
• 23:59
  And then we turned that into automatic APIs and so on and so forth so that we could increase our coverage of our models.
• 24:04
  And so then when regulation came along. was often not difficult to comply with it.
• 24:11
  The first point that you made though about defining bias. uh as you note. is not a simple matter and they're different tests and in some cases they're formally inconsistent and
• 24:21
  they're trade-offs.
• 24:22
  uh By definition. yeah. so how would you work through those questions where there were multiple choices. all of which could be a reasonable way to define fairness?
• 24:34
  It's a really good question.
• 24:35
  um One of the things that we always tried to do was take the perspective of the user. the job seeker. to say like. what's the worst kind of error the model could make. right?
• 24:46
  And so. you we all like. if you've taken an insurance stats class. you've heard of the like. would you rather your smoke alarm chirp a little too often. or would you rather like
• 24:55
  the automatic pilot. you know. disengage too often?
• 24:57
  And so like. those are different kinds of errors and false positives and false negatives matter differently.
• 25:01
  And so we would always kind of take the perspective like.
• 25:04
  what would deny an opportunity unfairly or what would potentially send the signal that a job seeker is not as qualified as they were incorrectly and then kind of work backwards
• 25:15
  from there to say. this. because of this. this is the kind of metric we're gonna use.
• 25:18
  And then some of it was determined like empirically like. it's this kind of number. it's a score. it's a category.
• 25:23
  So you had to pick like specific statistical tests to work with that.
• 25:27
  uh And then we would use best practices for determining effect size.
• 25:31
  And those often came from things like the four fifths rule.
• 25:33
  and other regulations say like. you know. this would stand up in most legal arguments as kind of prima facie. you know. bias.
• 25:40
  And so like. let's make sure we meet that as well.
• 25:43
  ah There was. that was actually one of the most contentious parts of the design process though. was defining that and making sure this was the right number.
• 25:50
  And what happens if the numbers. know. cause we would often have multiple measures and what do do if they disagree and who makes that decision and so on.
• 25:58
  So ah that was.
• 26:00
  That's still an open question. I think. for a lot of people.
• 26:02
  And I went to several events with EEOC attorneys and so on.
• 26:07
  And they would even say. as we all know. the Four-Fifths Rule is not a law.
• 26:11
  It's a heuristic.
• 26:12
  And they would say it's often not even the most important heuristic.
• 26:15
  And yet. it's the best we have to go
• 26:18
  Yeah. and people don't know that's the. as you said. the heuristic in US law that if the difference is basically more than 20 % between the two groups. that's initial indicator
• 26:28
  that there's bias.
• 26:29
  yeah. but as you say. people just sort of take that as a given and yet it's just a guideline.
• 26:37
  What do you think about the very. yeah.
• 26:42
  But it's clear. right?
• 26:43
  mean. 80 % is something that everyone can see.
• 26:47
  em Some of the efforts to legislate in this area. like New York City has this local law 144. they basically said do an adverse impact analysis and so you can see. for example. em
• 26:57
  what's your take on that and some of the other efforts to legislate and regulate in this area?
• 27:04
  I think. I think local L144 was a well-intentioned law that. that really went through the wringer in a lot of ways.
• 27:12
  And I think what came out. a lot of people. including industry and the regulators are not super happy with the actual final product.
• 27:22
  And I think both sides will say it's the other side's fault why it ended up looking like that.
• 27:26
  I thought it was a good first pass.
• 27:28
  The thing that I found potentially frustrating about it was
• 27:32
  how prescriptive the analyses were in some sense.
• 27:34
  And when I say that. it's because I actually think we did more rigorous analyses sometimes. and that can create friction with your legal department.
• 27:43
  So like. it doesn't say to do that. it says to do this.
• 27:46
  And so then you have to make your decisions.
• 27:49
  And I think specifically with the. this is gonna get very wonky. but like the continuously valued outcome variables. so like rather than a categorization. but like a score. the
• 27:59
  analysis proposed there doesn't.
• 28:01
  make total sense to me statistically.
• 28:03
  And I think others have raised this question as well. but now that it's kind of encoded into law. like we're doing that analysis and I'm not sure it's actually making the systems
• 28:12
  any fairer.
• 28:13
  And so that for me is kind of deadweight loss. right?
• 28:15
  It's like we've introduced complexity and bureaucracy without improving uh outcomes necessarily.
• 28:20
  That being said. they were the first and they were pioneers and it's. I think. know. been the template for lot of other people and I think lessons learned.
• 28:27
  I'm glad we went through it for sure.
• 28:29
  Yeah. and I know you're not a lawyer. if you were going to advise a government uh on how to address this issue of employment bias or AI and algorithmic bias more general. what
• 28:42
  would you suggest in terms of the right of
• 28:45
  You know. uh I thought about this a number of times and I always kind of come back to like. it's a very regulated space already and it's not always clear to me that we need
• 28:54
  additional regulation for AI.
• 28:56
  In fact. like. you know. there are some regulations out there that basically say it's illegal to discriminate with AI.
• 29:03
  By the way. it's already illegal to discriminate.
• 29:05
  We're just letting you know it's actually illegal to discriminate with AI as well.
• 29:08
  And which to me is like. I'm not sure what the purpose is there necessarily.
• 29:13
  And I think when you. so as a sociologist. when you introduce new institutions. new regulations. you're going to create constituencies and you're going to create feedback
• 29:22
  effects and so on.
• 29:23
  And so I worry about having regulation for regulations sake in those senses where in a place that's already regulated.
• 29:29
  I do think um it's a delicate balance as always between industry and the government.
• 29:35
  But I think this is an area where there's got to be better partnership and there's got to be.
• 29:39
  kind of good faith dialogue.
• 29:40
  When I say like. this analysis is actually impossible to do.
• 29:43
  We have to understand I'm saying that because it's impossible. not because I'm saying like. I don't want to do it. which is also. you know. something people do.
• 29:51
  Yeah.
• 29:51
  So one more thing is we need to wrap up just to get back to this question about the responsible AI field.
• 29:57
  As you point out. the pendulum has been swinging in the negative direction.
• 30:02
  Do you see a scenario. scenarios where that shifts back and what does it look like?
• 30:09
  I think it has to in some sense.
• 30:10
  I think the. you know. all gas. no breaks approach we're seeing right now to AI and industry only goes so far.
• 30:18
  And I think we're in the interim before regulation catches up.
• 30:22
  And then I think there will be a real. you know. whiplash effect after that.
• 30:26
  And I think the further we go potentially in kind of a quasi regulated state. the bigger that whiplash effect could be in a lot of ways. because. know. we'll push the boundaries.
• 30:36
  I do think we...
• 30:37
  I mean. this is obviously. I'm talking my book here. but I think we could see more embeddings into teams and kind of more like distributed responsible AI versus a
• 30:47
  centralized responsible AI. capital R. capital A. capital I team um that maybe kind of sheds a little bit of the political or ideological kind of associations that might be held
• 30:59
  there as well.
• 31:00
  I think that's. we haven't really touched on. but that's kind of obviously looming as well that.
• 31:05
  there's a macro political economic environment that this is all happening in.
• 31:10
  but I do think as we. since we haven't solved any of these problems yet. it's hard for me to imagine that we don't have some kind of responsible eye going forward. but maybe I'm
• 31:18
  just. I don't have a big enough imagination.
• 31:22
  No. I'm with you on that one.
• 31:23
  Absolutely.
• 31:24
  Trey. thanks so much.
• 31:24
  It's really been a pleasure speaking with you.
• 31:26
  It's been great.
• 31:27
  Thank you.