Latest / The Road to Accountable AI / Diya Wynn: People-Centric Technology

Transcript

Diya Wynn: People-Centric Technology · The Road to Accountable AI

0:01
Dear welcome.
0:02
You were, as I understand it, the first AI ethicist or responsible AI person at AWS.
0:09
So how did you find your way into that kind of a role?
0:13
Well, I'd say first, like, formal...
0:17
in that context, but there were certainly other people before me doing work in other
0:22
capacities in the organization focused on sort of ethical and responsible AI.
0:27
But specifically that title and function was developed in our professional services
0:36
organization.
0:38
And interestingly enough, probably, I guess about five years ago, a little bit
0:43
over five years ago, I started this internal camp
0:48
I thought that we had an opportunity to be able to assess the tens of thousands, now
0:54
hundreds of thousands of customers that we're building on top of our AI ML
0:57
services.
0:59
It's just them in a way that we do with like our cloud.
1:03
We have something called Well -Architected and it's a set of practices and sort of a
1:09
structure around pillars that give people the foundational information that they
1:14
would need to build.
1:16
and take full advantage of the capacity and all the capabilities of the cloud.
1:21
And my thought was that we have the benefit of our experience using AI and
1:28
implementing that for Amazon in various capacities for a number of years.
1:34
All of the expertise of the...
1:37
probably hundreds, thousands of data scientists that are helping to build our
1:41
services.
1:42
And those that are working in research areas and like Amazon Science and our
1:47
scholars that we could bring all of that together in a way to be able to assist our
1:52
customers with understanding the risk and having the best sort of insight to be able
1:59
to address that.
2:01
So we have this culture at Amazon, which is we call thinking big and writing a
2:06
narrative, basically a proposal.
2:08
And that's exactly what I did.
2:09
So I wrote a narrative with the things that I believed were going to provide the
2:16
best support for us to lean in and respond to our customers in this way.
2:21
I will tell you at the time we didn't have as much customer demand and a lot of our
2:26
innovation comes from that, you know, from our customers.
2:30
this was a little bit more forward leaning or forward thinking in that we were seeing
2:36
some increased conversations around the need for ethical AI.
2:41
That was the term at the time.
2:44
And it felt like we have an opportunity to be able to lean in and provide support and
2:49
assistance for our customers given their expertise and all of the knowledge.
2:52
So it took a little bit to find the right organization internally that would sponsor
2:58
the work.
2:58
But...
2:59
I started our first customer facing practice on responsible AI and that's sort
3:04
of where that all began.
3:07
So how do you work with customers?
3:10
As I assume they've got their own initiatives and their own business needs,
3:14
and AWS is the cloud provider that they're using.
3:19
What are the kinds of issues that you get into with them?
3:22
So it varies.
3:25
One is we'd have customers that would be looking for some sort of guidance.
3:31
They had some familiarity or awareness of this need for responsible AI or ethical
3:36
AI.
3:37
They wanted to understand what we were doing internally in the company.
3:41
And so that would be an inquiry that they made through their sales team or other
3:45
folks on their account.
3:46
team.
3:47
Sometimes it was directly with our professional services resources where they
3:52
saw an opportunity that they believed would be best suited for us to be engaged.
3:58
I would venture to say at the beginning that you know there was some discretion.
4:03
is this a use case that we should have some other oversight in or some other
4:09
support with?
4:10
And then there would be some outreach from our perspective in the initial stages
4:16
where we were looking at customers in certain industries that we felt would be
4:20
much more amenable to these conversations around responsible AI.
4:27
I've made a bit of a pivot in terms of...
4:30
like that initial structure and certainly what we've seen over the course of the
4:35
last year or so helped to make that pivot and shift as well because one, my firm
4:42
belief is that there shouldn't be any AI without responsible AI, without the
4:48
accountability and the intentional focus on understanding risk and seeking to
4:54
mitigate those and address those.
4:56
But what happened with...
4:58
sort of the generative AI now being in the hands of so many individuals and some of
5:04
its public challenges or the challenges that were made public, that's helped
5:10
others to now seek out much more our assistance and understanding of how we can
5:18
be much more diligent in our work and what things they can do to protect their data
5:22
and.
5:24
And so our work would be educational just in terms of general awareness.
5:28
One person may know or have some context and wants to make sure that they can
5:33
expand that to their entire team.
5:36
The other would be strategic advisory and then implementation support.
5:41
So those were sort of the three sort of broad areas that we would end up engaging
5:46
with customers as well as helping them understand what we.
5:49
we're doing as an organization and our commitment to responsible AIs that they
5:53
have some context for.
5:55
What are we seeing?
5:56
What are we doing?
5:57
And then how do we step in and do this as well?
6:01
Yeah, could you say a little bit more about that?
6:02
How does the internal work of AI accountability or responsibility in AWS
6:09
and Amazon relate to what you do engage with customers?
6:12
Yeah, so very much related.
6:14
We have established a four -part strategy in order to drive our commitment to
6:22
responsible AI and institutionalize that in the organization.
6:27
And that four -part strategy looks like one helping, well, I often start off with
6:34
one that is later in the context of our slides.
6:37
but it's having a people -centered focus.
6:40
And that people -centric of focus or approach helps us extend beyond just
6:45
looking at technology, but also incorporating people, process, and culture
6:50
to help make the shift so that responsible AI becomes an integral part of the
6:55
culture.
6:56
It also means that we are committing to doing the work that's necessary given our
7:00
understanding of the risks and the ways in which that it can affect marginalized and
7:04
underserved populations.
7:06
or underrepresented populations that we increase the value in building diverse
7:11
teams, incorporating their voice and perspective into how we're establishing
7:18
and developing products to ensure that they're fair and inclusive.
7:22
So that looks like us investing in scholarship programs, our machine learning
7:29
course.
7:31
a partnership with HBCUs and Hispanic serving institutions in order to provide
7:37
education.
7:39
And then we have something like our AI ready program that or initiative that
7:43
where we committed to training 29 million individuals in AI by the year 2025.
7:53
Then the other part of the strategy is,
7:56
taking a holistic approach to building our AI services responsibly.
8:01
And so that's incorporating responsible AI into every aspect of the machine learning
8:05
lifecycle, from how we make decisions about the technology at the leadership
8:10
level into design, development, and deploying, as well as ongoing use.
8:15
And that kind of goes in line with the idea that we don't.
8:21
want responsible AI to be a separate workstream, but integrated into the way
8:25
that we build our services and ultimately how we're doing our business.
8:32
And then it is helping our customers transform responsible AI from theory to
8:41
practice.
8:41
And that's essentially what we do or part of what we do, my teams or the team that
8:47
we're customer facing as well as.
8:50
other teams in the organization, it is our ability to be able to give guidance as
8:56
well as resources and tooling to be able to incorporate responsible AI into their
9:02
practice, essentially operationalizing responsible AI.
9:05
So that may be the team of responsible AI practitioners and consultants or experts
9:11
that will come to partner with our customers.
9:15
to provide education as well as awareness, as well as support as far as how they're
9:21
implementing specific services, like helping them understand how to detect bias
9:26
with something like SageMaker Clarify or looking at privacy preserving techniques.
9:31
And then the last bit of this is our ongoing investment in responsible AI
9:40
research, advancing the science of responsible AI.
9:44
And that's important because there are still challenges or areas where we don't
9:50
have all of the answers to.
9:52
We're constantly innovating and technology is also advancing at a rapid pace.
9:58
It requires a sort of constant evolution and expanding and advancing the science to
10:04
stay up with that, but also to find new novel approaches to some of the challenges
10:11
and be able to take that.
10:13
ongoing research and innovation into how we're building and ultimately the kind of
10:20
support and assistance that we're giving to our customers.
10:23
So that looks like some of our integration with National Science Foundation and we
10:29
have a Fairness in AI grants program where we've invested $20 million, strategic
10:35
partnerships with like California Institute of Technology as well as.
10:41
engaging in standards bodies, et cetera.
10:43
So that is the kind of insight that helps to inform the, again, the work that we're
10:49
doing in terms of building our services, but also what we're doing in terms of
10:53
providing support and assistance to our customers.
10:55
And so that's the overall strategy that helps drive all of the internal work in
11:03
the organization.
11:05
You mentioned the evolution in the increase you're getting from companies.
11:10
I'm curious how much at this point, when you're working with customers, is, OK,
11:16
here's the template.
11:17
If you want to be responsible AI, you've got to do the following eight things and
11:21
just check them off versus really having to be more of a bespoke conversation and
11:28
distinctive kinds of issues they're facing.
11:30
Well, so we tried to develop a structure in terms of helping us guide conversations
11:37
and engagement with customers.
11:39
Because just like you've seen pillars of responsible AI or principles that
11:46
different organizations have established, all of those provide a framework for being
11:51
able to unpack where there are areas of risk and to help us be intentional in
11:56
considering.
11:57
you know, who's incorporated and how we're building the system and addressing the
12:01
proposed risks.
12:03
So that structure for us is really what helps us guide the conversation, but it
12:07
does end up being somewhat bespoke in the sense that not every customer is at the
12:13
same degree of maturity and their unique, their needs might be different.
12:20
So, you know, we've had an opportunity to sit with a customer who was at the very
12:24
early stages.
12:26
that conversation or engagement with them was one, just focus on training and
12:31
education.
12:32
What do we do to help our teams and developers understand our commitment to
12:37
responsible AI, our commitment to AI overall?
12:41
And that was in part like building out a learning path and strategy for how we
12:51
incorporate general AI training.
12:56
into the organization so that they understand commitment somewhat like some
13:00
of the compliance, security compliance and information security kind of training, but
13:05
then also equipping people at different stages of their product development cycle
13:10
with knowledge and understanding of what they should be thinking about and how.
13:13
And then it was also like helping them develop their own set of tenants or...
13:21
or guardrails for what responsible AI looked like for them and their
13:25
organization.
13:26
And again, that sort of baseline work happens with some organizations.
13:31
And then others, we had a customer that came to us that had already created their
13:36
own principles and had a certain structure.
13:38
They really wanted to now get into tools and wanted to understand what things that
13:42
we have available that could help support them in that.
13:45
And fortunately, in our early stages, we had, I'll say, I'm probably.
13:51
A handful is not accurate.
13:53
There are a number of different tools because there are things that we're doing
13:56
in MLOps that support different areas of responsible AI as well in terms of its
14:04
automation or being able to look at performance and measure, right?
14:07
But then also adding on additional capability like the things that have
14:12
evolved with our being able to establish model cards in SageMaker or...
14:19
you know, detect for bias and have explainability.
14:21
Those are additional tools that now are services that now round out, you know, a
14:26
good, I think robust set of services that help support customers as they're
14:30
implementing.
14:32
Yeah, let me drill down on that issue of bias and fairness.
14:35
As you say, there are a variety of tools out there where you can do an analysis,
14:40
look at your training data, look at your model performance and so forth.
14:43
But how far along do you think we are in terms of organizations really seriously
14:49
and effectively addressing that challenge?
14:52
So this is, I think it's twofold.
14:57
I think you're absolutely right.
14:58
We have tools that are helping organizations understand bias in their
15:06
algorithms and or their data of themselves.
15:10
There's an element of this.
15:11
And I remember, I think it was a 2019 AI Now study that said, the problem with or
15:18
the challenge with AI systems is,
15:22
is a diversity problem, right?
15:24
Essentially, some of that challenge is because of the lack of diversity, the lack
15:31
of those that are underrepresented to see and help evaluate the data differently or
15:37
to help provide a different perspective as we're evaluating some of our challenges.
15:42
So I think that's one element of it.
15:44
But the part that I wanted to get to in particular was around the notion that we
15:49
as human beings have bias, right?
15:51
And we're the ones building the systems.
15:53
But that actually wasn't the AI Now study.
15:55
That was something that came out of Gartner that said 85 % of the projects up
15:59
till I think it was 2023 would have erroneous outcomes due to bias in the
16:05
people themselves, bias in the data, as well as in the algorithms.
16:11
And so we've got to be able to address or look at ways to be able to tackle all of
16:16
those.
16:17
How do we...
16:19
address challenges with the bias that we bring to a problem and the way that we
16:27
design a system.
16:29
The diversity helps to support that, but then also understanding that.
16:33
And then putting in systems and structure and process so that we could hopefully
16:38
unpack and eliminate our own individual bias and have a much more objective view
16:42
as we are evaluating things.
16:44
And so I would say that I think we're taking steps in the right direction.
16:49
The tools are essential to being able to support us in that process.
16:54
The process is necessary as well as us coming in alignment and seeing the value
16:59
and importance of doing that kind of work and making sure that we're considering the
17:04
voice and perspective of those so that we can eliminate areas of bias.
17:09
And then there's this bit where we likely will have bias at some level in systems
17:15
all the time.
17:16
And what do we do in terms of making different decisions given the context and
17:21
awareness of bias in our data perhaps, right?
17:26
Or systemic and institutional bias that might be infiltrated.
17:31
How do we then evaluate how we're releasing a product?
17:34
What does that look like in deployment?
17:36
And those are conversations that we're starting to have a lot more because of
17:42
this sort of move and direction.
17:44
I think those are all.
17:46
good things that are getting us further along.
17:50
What do those conversations look like?
17:52
Or, you know, you probably can't give a specific example with a particular
17:56
company, but you posed a bunch of really good questions there about things to think
18:01
about, but how do you help a customer actually work through what they need to do
18:06
for bias and fairness?
18:08
Yeah, so I'll kind of talk through the framework a little bit.
18:13
And we often start off with a value alignment in terms of like, how well is
18:19
this specific use case that we're considering aligned to our values with the
18:24
output of such, infringe on or not be in alignment with our values.
18:30
I've used this example a lot, but I think it's one of the best ones to articulate
18:36
this.
18:36
And this was with.
18:37
you know, someone who was responsible for serving a certain population and, you
18:45
know, gaining information or intelligence, you know, from calls could have, could
18:52
mean that they violated their, their, you know, core tenets in terms of their, their
19:01
ideals or their mission, right?
19:02
And so,
19:03
For us, it was more of a question of how would this use case align with your car
19:07
values?
19:07
Does this support your mission, vision, and values, any of the things that are
19:13
regulatory, et cetera, that you have to comply with?
19:16
Then it's about inclusion.
19:18
And one of the things that we do when we talk about inclusion is not just do we
19:24
have diversity at the table, because that's often been a challenge in terms of
19:29
bringing in resources,
19:32
broadly in organizations, we still are struggling with that.
19:37
But how do we bring in that voice and perspective?
19:39
And so a couple of the things that we would use is certainly like looking at
19:42
personas and identifying those.
19:44
Are we considering all stakeholders, not just 80 % of them?
19:49
And then also looking at, have we thought about anti -personas?
19:54
Who do we intend not to serve?
19:55
Because that's often where unintended impact occurs.
19:58
And so that would be another opportunity for us to think about and consider.
20:02
We actually mean this for this particular population, and we don't want to harm
20:08
others, even though we're not considering them one of our primary customers.
20:13
Then it's looking at things around training and education.
20:18
How are we equipping those that are responsible for building the systems or
20:22
any part of the product lifecycle with the knowledge and awareness that they need to
20:25
know, as well as what do we do?
20:27
in an organization where we might be introducing systems that have to be
20:32
integrated into work and augment the workforce?
20:36
Or what responsibility does that organization have to consider those
20:39
things?
20:40
And then we'll also then step through the other areas of the process.
20:43
And I know I won't go into all of those, but then transparency, accountability is
20:48
one.
20:48
Like, are we thinking about roles and responsibilities?
20:51
Who will help us understand?
20:55
What are the next steps to take?
20:56
How do we look at the specific use case, understand its risk and impact, and then
21:03
have the appropriate human oversight as necessary?
21:09
Transparency and explainability.
21:12
Then we look at privacy and security, privacy, security, and safety, and then,
21:17
and of course, fairness, right?
21:19
And the fairness is the technical sort of...
21:21
approach to ensuring that we have the appropriate behaviors that we want across
21:26
the system consistently irrespective of any subgroup or demographic.
21:31
And that element is important for us to be able to define at the beginning because
21:36
making this sort of commitment to building responsibility responsibly requires that
21:42
we think about this in design, right?
21:45
And...
21:46
and not just as an afterthought.
21:49
Like after we've designed the system, then we try to run some tests and look at F1
21:53
scores and hope that we've got the right result.
21:56
It's ultimately like defining what is fair for this particular use case and having
22:00
that integrate throughout the entire system is the goal.
22:05
So what exactly does that mean, designing it in from the beginning?
22:14
I'm sorry, I lost you for a moment.
22:16
I didn't...
22:16
that's right.
22:16
And we can just edit out if we something like this.
22:18
And yeah, I know that the one or whatever came up as we were talking there, but
22:22
that's fine.
22:24
What exactly does it look like in concrete terms to design in fairness from the
22:29
beginning of the process?
22:30
What are the things that an organization would need to do in order to achieve that?
22:34
it's actually thinking about, so it's in part like thinking about the customers
22:38
that we want to serve, but it's also thinking about what would be an
22:42
appropriate outcome that's going to meet the needs of both or all demographics.
22:49
So if I am building something that we want to make sure is providing benefit or
22:54
service to an aging population, right?
23:00
Making sure that we're respected if it's supposed to support.
23:04
people of a certain age, right?
23:06
So we're gonna say seniors, for instance.
23:08
I've been dealing with that quite a bit with my family.
23:10
So with my aging father, but you know, looking at like a specific service is
23:19
going to benefit them.
23:21
Then we would want to also say, well, well, aging has some cross sections,
23:26
right?
23:26
The aging population has some cross sections.
23:28
Do we consider?
23:30
what that performance might look like for people who are aging across different
23:33
demographics.
23:34
How do we think about people who are, who have different, who might be a different
23:40
ability, right, or might be disabled?
23:42
They're still part of the aging population.
23:44
And so I have to look at those cross -sections as well, and then define what
23:47
would be the appropriate system outcome, right, across those, so that we're
23:53
defining that and then integrating that into how we build the design.
23:56
So at the end,
23:57
If our system is not actually producing the right outcome that we've defined here,
24:01
then we get to reevaluate that or make determinations about, well, again, did we
24:07
meet the measure and fulfill the requirements ultimately desired for the
24:11
system?
24:13
So let me pull back a bit.
24:15
How do you define success when you're working with a customer on one of these
24:19
engagements?
24:22
Well, it'll vary some in terms of...
24:34
to do it lose again.
24:36
Yeah, I heard the lawn mower again.
24:39
So never.
24:41
Now that's right.
24:42
Yeah, you can go and I can ask you a question again, or you can just go ahead
24:45
and start the answer again.
24:47
No foul.
24:48
in the response, so I paused for a moment.
24:50
And I'm sorry, you'll have to cut this through a little bit.
24:53
fine.
24:53
Now I've got an editor, he'll do that.
24:55
Okay, so it will depend on what we're actually doing, right?
25:00
So in the context of something that is educational, it's looking at over time,
25:05
the percentage of individuals that we've reached in terms of educational content
25:09
and how that's informed the work that they do, right?
25:12
So that's fairly standard.
25:13
And when I say informed, right, if we are helping, you know, the developers now use
25:20
tools that are helping with detecting bias, as we were just talking about that.
25:25
how many of them are actually using that now, using those tools and making them an
25:30
active part of their job and responsibility versus just occasionally
25:35
using it or not using it at all.
25:36
What percentage of those are now adopting the tools that are going to help benefit
25:40
them?
25:41
So that could be one, right, from an education perspective.
25:43
But if it's something strategic, it's actually looking at how well we've defined
25:50
the...
25:53
principles or tenants that they might be leveraging, how those get integrated into
25:57
the organization, looking at a bit of a sort of organizational change management
26:02
process and how that gets implemented and creating measures across the board.
26:09
With implementation and with it's going to be something much more technical, if we're
26:14
again talking about bias, actually measuring before and after in terms of a
26:19
process, if we are now
26:23
re -evaluating a system that might have been already functioning and then what
26:27
that looks like in the end because now we're introducing bias detection tools and
26:33
mitigation techniques.
26:36
Ultimately, it's ensuring that when we look to address issues of risk, that those
26:42
risks have been mitigated and we have a way to be able to track those.
26:47
And you've been doing this for, as you said, a number of years now.
26:49
Are there mistakes along the way that you've learned from or ways that you've
26:54
evolved the way that you think about doing this kind of?
26:58
Absolutely, I think.
27:00
So one of the first things that come to mind is when we first started this ethical
27:05
AI was the conversation at DeJour and while there was a lot in marketplace in
27:13
terms of ethical AI, when we got into conversations or when we were trying to
27:18
approach customers, that actually was a bit of a turnoff in some ways, right?
27:24
There was sometimes a visceral reaction to the idea that
27:28
ethical, like we were passing judgment or making some assertion about their moral
27:34
values, et cetera, because of a particular use case.
27:37
And that's not actually the role that we wanted to take.
27:41
So we pivoted early from conversations around ethical AI to responsible AI.
27:46
I think our outcomes are still ultimately the same.
27:50
And we've used a number of these terms interchangeably.
27:54
But it meant a
27:56
It meant for us being able to meet our customers at a different place and remove
28:01
some of the initial resistance that might have occurred or did occur in some cases
28:05
so that we could have a conversation.
28:08
This notion of responsible AI is very much our responsibility is also very much in
28:12
alignment with discussions that we have from a cloud perspective.
28:17
We have a responsibility model and people understand that there's a role that we
28:22
play as a provider of technology, but there's also a role that they have.
28:26
in being able to, you know, protect.
28:33
There's a role that we have as far as providers of technology, but there's a
28:36
role that customer has as well as the builder and deployer of that technology.
28:41
And so this conversation around Responsible AI was certainly one that I
28:44
think was much more amenable to our customers, but also gave us an opportunity
28:48
to meet them much more where they are.
28:51
I think the other is to think that we could have this grand set of tenants
29:01
and...
29:02
this framework and perhaps that everyone would take all of those and start to
29:07
integrate them fully.
29:09
And understanding where people were in terms of their maturity was important.
29:13
So we had some customers initially that may have only done something around
29:18
education awareness and didn't necessarily step into using every tool or service and
29:26
addressing all of these pieces, but that understanding that this is a journey.
29:30
There are none of us that...
29:32
have, I think, arrived fully and have addressed everything that we can, but that
29:37
we're making steps in the right direction.
29:38
And so depending on where that organization is, as far as their maturity
29:44
and use of AI and deploying those things into production, it will dictate in some
29:51
ways what they take on as far as the principles.
29:55
And we had to certainly adjust our perspective about that because.
30:00
I would love that everyone would do all of the things, right?
30:03
But that's not a reality.
30:05
And I think the other is just recognizing that those trade -offs that we make with
30:14
how much we're investing in responsible AI and where that slows us down and what we
30:23
can prioritize given timeframes is very real.
30:26
And we've had to certainly...
30:29
you know, help figure out where it makes sense to invest, given a certain use case,
30:35
and others, you know, perhaps not so because you had these competing business
30:42
priorities.
30:44
One thing that's also changed is now there's this flood of either regulation or
30:49
frameworks or proposed obligations and laws that are relevant to AI and
30:55
responsible AI.
30:56
Does that make your job easier or harder?
30:59
I think it makes it harder.
31:01
Well, actually, I think it makes it easier.
31:05
I'm going to say that in the sense that where our initial start with responsible
31:11
AI was in some ways relying on people's will and desire to do good, to incorporate
31:22
these principles and to develop these practices internally.
31:27
And while there are customers
31:29
and companies for sure that are committed to doing good and not doing harm, again,
31:36
that business prioritization and the lack of necessity to necessarily invest could
31:46
help or could allow them to de -prioritize some of those areas of focus.
31:53
And what...
31:55
What legislation and some of the standards now has done is elevate the importance of
32:02
these, you know, beyond just one organization or a few advocates making
32:07
this demand, but seeing it across the board in, you know, governments, you know,
32:13
White House, executive order, et cetera.
32:16
But it's also adding the reinforcement, right, or we'll have, you know, some of
32:21
that.
32:22
reinforcement where it's a now and we can't ignore it.
32:26
It really is a business imperative and these areas are drawing people much more
32:34
to the conversation.
32:36
So last thing is, what do you, what do you focus on going forward or what do you
32:40
think are the greatest challenges in the next few years for organizations trying to
32:47
think about responsibility.
32:50
Well, that is, who knows what's going to happen in the next few years, but I'll
32:56
tell you what I think about a lot.
33:00
And it's an area of excitement, in one sense, the great opportunity that we have
33:06
with AI.
33:07
In some ways, we are looking at technology and opportunity for us to hopefully create
33:14
more equitable experiences and access, right?
33:17
That is a hope.
33:20
It requires a lot of intentional focus from organizations to build in these
33:27
practices, to make that integral in the way that they're not just designing, but
33:31
also deploying these systems.
33:33
I think the other thing is just to be conscious and aware of while technology
33:39
has the ability to do a lot of things, some things we probably should keep off
33:44
limits.
33:45
Right?
33:45
And how do we balance that?
33:47
Because there's such excitement and consideration for all of the things that
33:53
it can do.
33:54
But I think that there are some things that it probably shouldn't do.
33:58
And us still balancing this great opportunity, all that the advancement of
34:06
technology can bring while preserving humanity.
34:09
All right, well, that's an encouraging, challenging, but overall positive way to
34:16
wrap up.
34:17
Dia, thank you so much for joining me on the road to accountable AI.
34:20
Thank you as well for having me.