Latest / The Road to Accountable AI / Alice Xiang: Connecting Research and Practice for Responsible AI

Transcript

Alice Xiang: Connecting Research and Practice for Responsible AI · The Road to Accountable AI

• 0:01
  Thank you so much for joining me on the podcast.
• 0:04
  Thanks so much for having me.
• 0:06
  You have somewhat of an interesting background with degrees in both law and statistics.
• 0:13
  How does that combination contribute to the work that you've done on AIF?
• 0:19
  So I think AI ethics is a particularly great space for folks that have diverse backgrounds and that it is such a socio-technical space.
• 0:29
  On the one hand, there's a strong technical component of course, in terms of how AI technologies work, how various mitigation strategies work in terms of how you can address
• 0:39
  issues at the data layer, at the model layer.
• 0:42
  But at the same time, especially nowadays, there is a significant legal and regulatory
• 0:48
  to AI ethics.
• 0:49
  It's not just a matter of what does any individual developer think would be better from an ethical perspective, but we increasingly do have different rules to comply with as well.
• 1:01
  And so figuring out how to bridge those two elements is a lot of the challenge in the AI ethics space, but one that really attracted me to it since it's one of the rare.
• 1:14
  subfields where you can actually leverage both of these areas fully in your day job.
• 1:20
  So that's something I really appreciate about it.
• 1:24
  Yeah, and similarly, you spent most of your career as a research scientist, but then moved over to Sony in this role that you're in now.
• 1:31
  What made you decide to make that switch?
• 1:35
  Yeah, so a few things.
• 1:36
  So for one, it was just a very unique role, both in terms of leading a research team, but also leading the AI governance function in terms of more of the legal and compliance side
• 1:49
  as well.
• 1:51
  And I thought that was really exciting because one without the other can often feel a bit siloed.
• 1:57
  So if you're just doing research, but you don't have any connection to the governance side.
• 2:02
  of the organization, then your research goes out into the world, but there's not a clear funnel in terms of how does it make impact within the company?
• 2:09
  How do you actually implement the research you're doing?
• 2:12
  And also, how do you actually learn about what the real world challenges are that folks are experiencing?
• 2:18
  And then on the flip side, just legal and compliance without any connection to the research side can often be sort of miles behind in terms of what the latest and greatest
• 2:29
  research is around what
• 2:30
  can you actually do from an AI ethics perspective.
• 2:33
  The field is moving so fast that it can, you know, every single month we have new publications coming out on different techniques.
• 2:42
  And so I think it's really great to have both of these hats so we can move more at the cutting edge when it comes to AI ethics.
• 2:53
  How much do you think that that's a cultural issue or a company issue?
• 2:56
  There's some companies that have a very technical culture.
• 2:59
  There are others where it's more management driven.
• 3:03
  Do you think that this kind of approach is the right one for everyone or is it something particular about Sony and companies like it?
• 3:11
  it's, that's a good question.
• 3:12
  I think it varies a bit.
• 3:14
  Like not every company can afford to have a research team that's doing more like kind of basic research in a way, in terms of trying to push for the frontiers of what is possible
• 3:24
  in AI ethics.
• 3:25
  It is much more common to have folks that are more focused on just implementation when it comes to kind of the compliance side of things.
• 3:35
  That said, I think there's been an interesting shift in recent years in terms of how AI ethics in general has been considered.
• 3:41
  used to only be the major tech companies that would have anybody whose full-time job was on AI ethics, but especially with recent laws coming into place now, increasingly,
• 3:54
  especially in terms of AI lawyers, you're seeing more positions like that at companies because...
• 4:00
  It is quite difficult now if you are an AI developer to ensure that you're in compliance with all of these laws and having someone with that specialized expertise is quite
• 4:10
  important.
• 4:12
  So from that perspective, I have seen a lot of growth in that area while the research is maybe still concentrated in a few major companies plus academia.
• 4:23
  And you've been in this role at Sony for, think, close to four years now.
• 4:27
  How has the AI governance structure and activity developed in that time?
• 4:33
  Yeah, so we've made tremendous strides during that time.
• 4:38
  in terms of how our structure looks overall, so Sony is a huge company with very diverse businesses, spanning electronics, entertainment, financial services.
• 4:48
  And so given that diversity, we have in each business unit, a CEO appointed AI ethics executive to work with my team, which is the AI ethics office within headquarters.
• 5:00
  And so we work with each of them to develop AI ethics structures within their business units that can assess AI cases, flow to high risk cases, and also work with them on
• 5:16
  compliance with relevant laws plus internal policies on how AI is developed or used.
• 5:24
  And so, you know, with some of our business units, are, we have a more hands-on
• 5:29
  where we actually are very involved with many of the day-to-day assessments.
• 5:34
  for example, a lot of our early work was with our electronics business units where there was already a lot of kind of structure around how they evaluated their technologies
• 5:44
  because in the hardware space, of course, there is a lot of consideration and quality assurance that's necessary before you ship out any product.
• 5:52
  And so we were able to layer on AI ethics as part of that.
• 5:57
  and be very involved with them in developing that structure.
• 6:02
  And now we're working more in the entertainment space in terms of looking at how that might, how we can structure ethics assessments there.
• 6:11
  But I do think we've done an impressive job in terms of just like the scope and depth of the type of AI governance that we have in the organization, especially just given how
• 6:23
  diverse the different AI applications are.
• 6:27
  It's an interesting challenge, because presumably you want to address the needs of the unique businesses, but you want to have some consistency in the approach.
• 6:36
  So how do you make that balance?
• 6:39
  Yeah, it's always a constant kind of back and forth.
• 6:42
  So a key part of my team's role is harmonization.
• 6:46
  So for example, when it comes to internal policy, sometimes we'll put out a first draft or at least whenever different businesses come out with their versions, we'll ensure that it
• 6:57
  is harmonized with our across businesses as well.
• 7:01
  So there's the allowance, of course, for businesses to have specific
• 7:07
  approaches that are necessary for their own business idiosyncrasies.
• 7:11
  But at the same time, we want to make sure there's a sufficiently unified Sony approach such that it's not very confusing for employees.
• 7:21
  It doesn't create unmitigated categories of risk, all of that.
• 7:25
  yeah, it definitely keeps my team busy in terms of trying to be that central node.
• 7:33
  And you have something called the AI ethics flagship project.
• 7:36
  What does that mean?
• 7:38
  Yeah, so that's on the research side.
• 7:40
  So Sony research is structured around these major research initiatives, which we call flagships.
• 7:47
  And on the ethics side, a lot of our focus today has been on topics of ethical data collection and algorithmic bias, since these are some major areas that
• 8:01
  Even though there's been a lot of awareness and discussion of these issues in recent years, there's still a lot of challenges in terms of how do you folks actually do this in
• 8:10
  practice?
• 8:10
  So ethical data collection is a great example of that.
• 8:13
  When we first started working in this space, it was a bit more of a niche issue.
• 8:17
  Nowadays, it's one of the biggest topics because especially in the age of Gen.ai, folks are very concerned with the huge amounts of data that are being
• 8:27
  ingested by models, typically uncurated data.
• 8:31
  And so you have all sorts of challenges in terms of issues of consent, intellectual property, biases within the data, transparency around the collection process, compensation
• 8:43
  of individuals involved in the data creation process.
• 8:46
  So there's so many of these issues and it's easy to kind of list out the issues, but it's very hard to say how would you actually try to tackle all of those.
• 8:54
  and address them and even at a small scale, let alone at a big scale.
• 8:59
  So there's a lot of research questions within that that we've been working on as a result.
• 9:06
  It is interesting that you were still engaged in publishing research and you were the chair of the FACT conference, it one of the major conferences in the field.
• 9:15
  It's hard enough for people who are full-time researchers to keep on top of what's going on, even in some subset now of AI ethics and governance.
• 9:24
  So, how are you able to pull in insights from the research process in ways that are helpful for wearing your AI governance hat?
• 9:34
  Yeah, so it's kind of a mixture of things.
• 9:37
  I will say there's a little bit of code switching.
• 9:39
  the one hand, we're trying to push the cutting edge in terms of research.
• 9:44
  And then on the other hand, there is the more practical implementation side.
• 9:48
  Now, there is a little bit of overlap in this Venn diagram in that a lot of what's tricky from an implementation side is that last mile, which is where research tends to start
• 10:00
  getting a bit more nebulous in terms of
• 10:02
  what to do, but if you kind of dig a few layers deeper, that can also be interesting fodder for research questions.
• 10:11
  Because again, it's sort of like very easy to kind of list out the Desiderata, but going to like, how would you actually do this in practice?
• 10:20
  There are like a bunch of challenges within that that can also provide, can be interesting research questions in and of themselves.
• 10:30
  But that's one thing that we try to balance, like on the one hand with the research team giving them enough room to work on projects that are a bit more high level in terms of
• 10:42
  what should ideally happen, but also trying to tie it to some practical elements too, such that we can learn from that in advising the businesses.
• 10:53
  For example, going to the fairness evaluation side of things,
• 10:58
  There's a lot of talk about the need to do to assess systems for bias, but it's still quite difficult in practice.
• 11:07
  Like if you talk to a business unit, there's like a lot of very basic questions they have that aren't really quite rigorously answered yet.
• 11:14
  So, you know, it's like, okay, what makes for a good fairness evaluation?
• 11:18
  What is sufficient?
• 11:19
  Like how, like what actually counts as problematic bias?
• 11:23
  And like if there's a confounding factors, it's still
• 11:26
  problematic, what kind of magnitude and direction of bias is problematic.
• 11:34
  And there's all of these sorts of questions that are actually really challenging to resolve and can't really be answered once and for all.
• 11:43
  And so that can also be kind of interesting to drill down as well from a research perspective.
• 11:49
  How do you answer those questions?
• 11:53
  So in the absence of very clear best practices or regulatory standards on this, we tend to take a case by case approach with many of these things.
• 12:05
  So, for example, when it comes to these questions of like how bad is bad, like, you we try to tie it to, what are the actual harms tied with this sort of use case?
• 12:16
  So, you know, it's a very different question if...
• 12:20
  If you have, for example, we don't have these applications at Sony, but if you hypothetically had facial recognition being used in a law enforcement context, then to
• 12:32
  identify potential suspects, then any sort of false positives are very harmful.
• 12:38
  And so you would really care, especially about trying to minimize those and kind of the...
• 12:48
  the rigor with which you would want to evaluate bias issues is extremely high.
• 12:53
  Whereas if it's a context where the AI system is maybe just like roughly counting people in a crowd to say like, there's too many people in an area, then you care a little bit
• 13:04
  less.
• 13:05
  I mean, you still care about fairness issues, but it's a little bit less sensitive in terms of if the accuracy is a bit lower for one group versus another.
• 13:17
  What would you say are some of the most important unanswered questions in the research world in the fairness and biospace?
• 13:26
  Great question.
• 13:26
  think there's a lot of things.
• 13:28
  So I guess first off in terms of fairness and bias, I think there's a significant lack of fairness benchmarks in a lot of areas.
• 13:37
  you have individual papers that have assessed particular models for bias concerns, but there's not a clear set of benchmarks that everyone can use and then compare their models
• 13:50
  reliably.
• 13:51
  And I think that's
• 13:53
  is very important to actually make this part of an everyday process.
• 13:57
  If every single model developer kind of needs to ad hoc create their own test, essentially, it's both not the best incentives and also makes it really difficult for
• 14:09
  third parties to assess how biased is this model versus that model.
• 14:15
  So I think that's one major area that's missing.
• 14:18
  In addition, there tends to be a lack of
• 14:22
  demographic data that would enable folks to do these sorts of checks.
• 14:26
  And so typically, you in the literature, people assume, yeah, you know, who is male or female, or, you know, who is of different racial background and such.
• 14:38
  And that's not really how things work in practice.
• 14:42
  then layered on top of that, there's a lot of questions around appropriate taxonomies.
• 14:47
  So like, for example, in fairness, it's very common for people to just have male and female.
• 14:51
  categories and then of course not everyone identifies in those categories and there's a lot of unanswered questions in terms of how to address the limitations of taxonomies in a
• 15:08
  practical and scalable way across a wide variety of applications, especially given that that sort of data is very sensitive.
• 15:15
  Like really any data that's used for fairness assessments tends to be very sensitive.
• 15:20
  So these are some areas that like researchers on my team have worked on and that I also encourage other folks to explore as well because I think what's really interesting about
• 15:31
  AI ethics is we're never going to fully solve any of these issues.
• 15:35
  The best we can do is put out solutions that are, that make incremental improvement in certain dimensions, but there's never, there's no such thing as like a perfectly ethical
• 15:45
  solution and everything has trade-offs in the ethics space.
• 15:50
  Yeah, absolutely.
• 15:50
  Well, so let me ask you a related question that's more on the practical side.
• 15:53
  Let's say someone's listening to this from an organization and they recognize, let's say, take fairness.
• 15:59
  It's an important issue.
• 16:00
  But as you know, it's an issue that crops up everywhere from the training data to the training to the outputs and everywhere in between.
• 16:07
  So where should an organization start?
• 16:13
  It's yeah, I think in terms of I think the most important thing is to start somewhere actually that that one step of just starting is where is where things usually don't happen
• 16:23
  because you know in practice You know I've seen on the ground is typically people get stuck at that first layer They're like well, you know I don't really know like where I
• 16:33
  would get this data to do this fairness evaluation Or I don't really know what would be the best way to test my model and
• 16:42
  I think what's important is to at least start doing something, getting development teams used to thinking about these issues and whether they're applying the latest and greatest
• 16:52
  methodologies or not is less important than having some sort of baseline of actually checking for these issues.
• 17:00
  Overall in the industry, I think the bigger problem is that folks just don't even check for these things.
• 17:06
  they, it's very common just to kind of.
• 17:10
  for folks to throw their hands up and say, well, you I gave you all the information about like general performance and such.
• 17:15
  And that's what, you know, downstream users or clients care about.
• 17:19
  But if we don't even start thinking about how to do this in practice, it's hard to make any improvements.
• 17:27
  And that's something we really, you know, stress on our end, like even if you can't do the best fairness evaluation, just try to at least check.
• 17:35
  for something, because it can be very eye-opening as well from the developer's perspective, because inevitably there's pretty much always some degree of bias.
• 17:45
  And it then leads to those questions of what is problematic and such.
• 17:50
  But those are very useful questions for folks to start interrogating as a first step.
• 17:57
  then how do you get the conversation to go from there, given as you say, it's not that, we found the bias and we just take it out because nothing is perfect.
• 18:06
  How does an organization use that information constructively?
• 18:12
  So this goes into more of the kind diagnosis process.
• 18:17
  So as I was mentioning, how big of a bias and what kind of bias matters depends on the particular application, because it's not always just a technical fix.
• 18:28
  Sometimes the fix can be more about the human-computer interaction, for instance.
• 18:34
  So if you have an AI model that you are worried about certain shortcomings of bias or otherwise,
• 18:41
  one mitigation strategy is like, okay, if a human is checking the output, for example, or a human that is like well-trained, or alternatively, the output of AI model is only one
• 18:54
  component, or you provide more transparency about the shortcomings of the model.
• 18:59
  So there's these additional levers that can be pulled in addition to the technical ones.
• 19:04
  And of course, like with the technical ones, if you know about the...
• 19:09
  bias from a quantitative perspective, that's an important first step there.
• 19:13
  So I think the first step tends to be the hardest and then it kind of opens up the door to these other possibilities in terms of how you can address these issues.
• 19:23
  But this whole conversation stops if folks just throw up their hands and say, can't even check for this property.
• 19:31
  How do you overcome that resistance or skepticism, whatever it is, whether it's from developers or business people who say, of course, bias is bad.
• 19:39
  These are all problems.
• 19:40
  But my job is just to make the system work according to certain performance.
• 19:46
  It's a very challenging question.
• 19:48
  it's definitely, I would say, an area where buy-in is extremely important and executive buy-in and internal policy making is quite important here.
• 20:00
  Because basically you want this sort of thing to become habit.
• 20:04
  And once it becomes habit, it's not so onerous for people because it's always hard when you first start out and you come in in the middle of someone's development cycle and you
• 20:13
  say, hey,
• 20:15
  to do like a fairness assessment or I want you to fill out this model card or I want you to tell me how you interact with your stakeholders and what you told them about the
• 20:25
  functionality of this model.
• 20:27
  that always is tricky at the beginning because there's no way around it in terms of there is work involved in checking for these things.
• 20:35
  There's no way to just, you know.
• 20:38
  figure out all this information without somebody actually putting pen to paper and thinking about it and often running some analyses.
• 20:47
  But once it becomes more of a habit, people factor it into their overall timelines and it becomes just another part of the process.
• 20:55
  you know, in terms of my role, I've seen it kind of both ways.
• 21:00
  You when we first came in, it was a lot more of like the...
• 21:03
  the pushback at every single stage and like, do we really need to do this?
• 21:06
  Why do we need to do this?
• 21:08
  Like, it's going to take me this much time and that sort of negotiation.
• 21:12
  But for the, you know, for the business businesses where this has become much more of a habit now, like we're getting much less of that and much more like kind of constructive,
• 21:24
  like how do we do this better?
• 21:25
  We want to do this at like, we want this to be useful.
• 21:29
  You know, we want
• 21:30
  this to be at a good level and we're not just trying to tick the boxes and move on.
• 21:37
  And just in general, how important is this AI to Sony?
• 21:39
  And either Kinetza in terms of machine learning generally, or in particular, I'm curious about generative AI.
• 21:48
  so it's, let's see, how should I?
• 21:54
  Yeah, go ahead if you need a minute to think about
• 21:58
  Yeah, sure.
• 22:00
  So AI does show up basically in all of our businesses in various ways.
• 22:05
  So we have what's really interesting about working at Sony in this area is we have so many different elements of the AI ecosystem represented.
• 22:15
  So on the one hand, we have technology developers in both the hardware and software context, especially in our electronics business units, where really, AI can show up in all
• 22:26
  sorts of
• 22:27
  different places, whether it be like face verification on a phone or like facial detection in terms of camera technologies and things like that.
• 22:38
  But we also have a lot of interest in terms of broader conversations around the role of AI in the ecosystem, especially vis-a-vis creators, because we are a company that, you know,
• 22:51
  the one thing that unifies a lot of our businesses, both entertainment
• 22:55
  electronics is the focus on creators and that's a huge part of our corporate purpose.
• 23:02
  And as anyone who's opened the news nowadays knows, it's a very active public conversation in terms of how AI is used in this space and whether it can be used to augment human
• 23:16
  creativity versus replace it.
• 23:19
  and that's a very important question for us from an AI ethics perspective too, that informs a lot of our work.
• 23:25
  Like, I think it, it's a nice connection between the goals of AI ethics and the goals of the company that like a lot of what we're doing from an AI ethics perspective is we want
• 23:36
  to ensure that, AI is in this more enhancing role in terms of, making people's experiences better, enabling them to do more create, creatively rather than.
• 23:49
  know, disenfranchising them or replacing them in terms of like, you know, leveraging their works as fodder.
• 23:57
  So, that is one nice thing that motivates our work as well.
• 24:03
  Yeah, absolutely.
• 24:03
  And this is a really huge question that's coming up in lots of contexts in the creative economy, whether it's copyright or job displacement or just changing the nature of
• 24:16
  creative work.
• 24:17
  How are we going to get to that result that the AI is additive as opposed to purely just replacing all the human?
• 24:29
  It's a challenging question and it's something, on the plus side, think we have a lot of work.
• 24:36
  So there's not gonna be any easy resolution.
• 24:41
  But in terms of my team's work, one thing we do focus a lot on kind of going back to the ethical data side of things is thinking very deeply about these questions, especially of
• 24:52
  consent, compensation, the various rights involved in data.
• 24:58
  And there's obviously a legal, increasing legal component to that, but from an ethics side, we are just kind of taking more of the approach of if you are trying to, as much as
• 25:09
  possible, protect people's rights and make sure everyone is fairly treated in the process of data collection, what would that look like?
• 25:16
  And kind of raising awareness of these issues and making sure that folks are checking for these things and there's education about these issues as well.
• 25:26
  I think that's a...
• 25:28
  very important starting point.
• 25:30
  In general, I think there needs to be a lot more focus in the research field on how to bridge this sort of gap, because on the one hand, you have the need for tremendously large
• 25:42
  data sets, where it's very hard to envision how do you actually respect all of the different rights involved with every single element of that data set.
• 25:54
  And then on the other hand, you have a long list of
• 25:57
  Desiderata of what you would ideally do.
• 26:01
  And I think we kind of need a reimagining of how that would actually work in a scalable operational way.
• 26:08
  And I know some of that doesn't sound sexy because I said the word operation.
• 26:13
  especially like from a research standpoint, there can be this attitude of like, okay, that's not really our job.
• 26:19
  But I think if someone can solve that, that would be tremendous because right now,
• 26:24
  Developers are in a very challenging situation where you can imagine how you could do this properly for small datasets, but it's very hard to see how you would scale up to trillions
• 26:35
  of data points, for example.
• 26:38
  Yeah, absolutely.
• 26:39
  So one more question about Sony.
• 26:40
  It's a very global company, a company whose roots are not in Silicon Valley, but in Japan.
• 26:46
  how do those cultural factors play into the work that you do on AI ethics?
• 26:52
  So one aspect that I really appreciate in terms of working at Sony is there is a huge emphasis on diversity in our culture.
• 27:02
  So that is in part due to our extremely global nature.
• 27:07
  And that's made it a lot easier actually for us as AI ethics team, especially one that focuses on issues of fairness to make the case for why.
• 27:20
  folks in the company should care about this because it's something that is already in kind of all of our corporate cultural materials, training materials, that emphasis on
• 27:29
  respecting people of different cultures and trying to leverage diverse perspectives as a value add in our organization.
• 27:36
  So that for me has been really nice.
• 27:39
  And it's also practically had the impact of, so my teams are extremely diverse and global.
• 27:44
  So on the research side,
• 27:46
  where actually at both the research team and the legal compliance team were split around the world.
• 27:52
  have folks in the U S, in the EU, in India and Japan, the UK.
• 28:01
  So, you know, we're really kind of all over the place and, that's been very beneficial in informing our work, especially because,
• 28:10
  The AI ethics field in general tends to be very US and Europe centric.
• 28:15
  And that of course imposes a certain value system and perspective that is quite limited.
• 28:22
  you know, for example, even just kind of going back to the taxonomy question earlier, if we talk about issues of like racial bias, for example, what are relevant categories of
• 28:33
  race?
• 28:33
  people.
• 28:34
  very commonly in the literature use like the US census categories, for instance, and that's not really resonant to folks outside of the US.
• 28:42
  And so a lot of these insights have been raised in my team in part because we just have such a diverse team where folks are pointing out these ways in which the perspective can
• 28:54
  be limited if we're just going off of what is commonly used.
• 29:00
  And one big change that has happened in recent years in this area is the coming of many legal requirements and regulations around AI such as the
• 29:12
  things all around the world.
• 29:13
  How does that affect what you do on the ground from a governance standpoint?
• 29:19
  Yeah, so it's made my job, I guess, easier and harder in different ways, easier in the sense that I would say, you know, when I first started my job, oftentimes a lot of what I
• 29:33
  had to prove to execs is why do you why should they even think about AI ethics?
• 29:39
  Why is this relevant if like they are not like the technology person in their org or the AI person there?
• 29:45
  Or why is this a broader issue that like requires
• 29:48
  kind of cross-functional collaboration in order to address.
• 29:54
  Now that's, that pitch is way, way easier.
• 29:56
  In fact, it's more a lot of the other side of like, now I tend to be more inundated with folks asking questions like, my God, how do we comply with this?
• 30:05
  How do we address these issues?
• 30:07
  Like AI is now very top of mind for many folks throughout the organization.
• 30:13
  So.
• 30:14
  But of course that has also created its own form of difficulties in terms of now, like we have to move extremely fast in terms of figuring out how to actually wrap our heads around
• 30:24
  all of this.
• 30:27
  it's not just like, yeah, we're building AI governance systems because like we're ahead of the game and we're trying to like do things that are extra and like.
• 30:37
  Instead, it's like, yeah, we actually really need to do this.
• 30:40
  So there is more external pressure for that as well.
• 30:43
  But overall, I think it's a direction that is helpful in that.
• 30:58
  Yeah, you can just pick that up.
• 31:02
  But overall, the helpful aspect has definitely been kind of moving AI ethics from an area where it's really just like a nice to have and has a very strong subjective component to
• 31:13
  it to something that is, there is much more of that demand both internally and externally for us to be thinking very critically about these issues.
• 31:26
  And finally, what do you see as the biggest challenge that you're facing in your work in the coming years?
• 31:33
  The biggest challenge I would say is that AI ethics is just such a new space that's moving so fast.
• 31:40
  so, you know, typically there doesn't need to, for example, be such a close connection between like research and implementation because you can have like a 10 year lag and
• 31:52
  that's completely normal.
• 31:54
  But in the AI ethics space, the entire field of AI ethics is an extremely new field or subfield, depending on how you consider it.
• 32:03
  And we're going from, okay, we have a lot of high level research to suddenly now there's these extremely complex regulations that are complex, but not necessarily very specific
• 32:15
  from a technical perspective of like, do you actually do?
• 32:18
  And there's also a dearth of industry best practices to lean on because again, it takes a long time for there to be any coalescing around what everyone is doing in the space and
• 32:29
  what folks think is.
• 32:31
  sufficient.
• 32:32
  And so, you know, for my teams in this space, we're having to kind of fill that gap of like, okay, there's, it's not so easy as we can, it's not as easy as just having a
• 32:43
  checklist where we can say, okay, did you go through this checklist and do this?
• 32:47
  Instead, there's a lot of challenges to define what the right approach is.
• 32:53
  And, you know, on the plus side, though, it keeps the job interesting and makes it, you know, again, like creates
• 33:00
  the reason for why you would have the research side as well, tied into all of this.
• 33:07
  But it is a significant challenge at the same time in terms of to fill that ambiguity.
• 33:17
  Alice, it's been a pleasure speaking with you.
• 33:18
  Thank you so much for your time.
• 33:20
  Thank you