Latest / CXOTalk / AI Inequality EXPOSED: When Algorithms Fail | CXOTalk #882
Transcript
- 0:00Today on CXO Talk episode 882, we're examining a critical issue
- 0:07that affects millions, yet is largely invisible to most
- 0:11business leaders. How AI is failing our most
- 0:14vulnerable citizens on an unprecedented scale.
- 0:18I'm Michael Krigsman, and I'm delighted to welcome our guest,
- 0:22Kevin Delebin. As founder of Tectonic Justice,
- 0:26Kevin has witnessed first hand how AI systems determine who
- 0:30gets healthcare, who finds housing, who gets hired, and who
- 0:34receives government benefits. His ground breaking report,
- 0:38Inescapable AI, reveals that 92 million Americans now have
- 0:43fundamental aspects of their lives decided by algorithms,
- 0:48often with devastating consequences.
- 0:51He'll share how these systems fail, why traditional
- 0:55accountability mechanisms don't work, and what this means for
- 0:59your organization. We're discussing real systems
- 1:03built by major vendors causing real harm right now.
- 1:08Tectonic Justice is a new nonprofit organization launched
- 1:11last November to protect low income people from the harms
- 1:17that AI causes them. And I come to this work after 12
- 1:21years as a legal aid attorney representing low income folks in
- 1:25all sorts of civil legal matters.
- 1:26And it was there that I first saw the ways that these
- 1:29technologies were hurting my clients lives.
- 1:32And I was involved in several battles and won one several of
- 1:36them as well, and started understanding that this was a
- 1:39bigger problem that needed more attention and more focus.
- 1:41Kevin, you were an early pioneer actually winning cases relating
- 1:46to AI harms. It was really in 2016 when I had
- 1:51clients who were disabled or elderly on a Medicaid program
- 1:55that pays for an in home caregiver to help them with
- 1:59their daily life activities so that they can stay out of a
- 2:02nursing facility. And this is better for their
- 2:05dignity and independence and generally cheaper for the state
- 2:07as well. And what happened is the state
- 2:09of Arkansas replaced the nurse's discretion to decide how many
- 2:15hours a day of care a particular person needed with an
- 2:19algorithmic decision making tool, and that ended up
- 2:21devastating people's lives. People's care hours were cut in
- 2:25half in some cases, and it left people lying in their own waste,
- 2:29left people getting bed sores from not being turned, being
- 2:32totally shut in just intolerable human suffering.
- 2:34And we ended up fighting against that in the courts and also with
- 2:37a really big public education and kind of community activation
- 2:40campaign. And we won.
- 2:41And that's one of the relatively few examples still to this day
- 2:46of kind of successful advocacy against sort of automated
- 2:49decision making. Algorithms, AI are neutral
- 2:54mechanisms, neutral devices. They're just math without
- 2:59feelings, without interests, without malice.
- 3:03So given that, what is the problem here?
- 3:08I would challenge some of the the the assumptions even in that
- 3:12question of them being neutral, right?
- 3:13I mean, they're programmed by humans.
- 3:15The statistical science that underlies a lot of this stuff is
- 3:19determined by humans using various choices that they have,
- 3:22using historical data that they have.
- 3:25And that isn't a wholly objective exercise.
- 3:28And so I think what you really have to look at is the purpose
- 3:31for which the technology is being built to understand it and
- 3:35understand a lot of like even the technical aspects that
- 3:38underpin it. And in my world, when we're
- 3:40talking about low income people and sort of automated decision
- 3:43making for them, these are not neutral technologies at all.
- 3:46These are designed oftentimes to restrict access to benefits or
- 3:50to empower whoever's on the other side of the equation,
- 3:54whether it's a landlord, a boss, a school principal, a government
- 3:57official to do something that they want done.
- 4:00That might not be what the person who I'm representing is
- 4:05interested in. So I would challenge that
- 4:07premise first. So you're saying that the design
- 4:10of the system is intended to cause harm.
- 4:14Is that correct what I'm hearing you say?
- 4:16In some cases, it's intended outright to cause harm.
- 4:19In some cases, it's just intended to, you know, sort of
- 4:25facilitate a decision by the decision maker, right?
- 4:28Make a landlord's life easier, make a boss's life easier, make
- 4:32a government official's life easier.
- 4:34The problem is inherent in making their life easier ends up
- 4:38being making somebody else's life harder.
- 4:41And so I think that's where the push and pull is of this is
- 4:45there is the intent issue. There are very clearly stuff
- 4:48that's built to be harmful. But then there's also this Gray
- 4:51area where nobody is, you know, scheming in a dark room about
- 4:55plotting to take over the world and destroy people's lives.
- 4:58But the nature of their power positions and the decisions that
- 5:01they're making and what makes their life easier ends up
- 5:04translating into that for low income people.
- 5:07Can you give us an example of where the the goals or the
- 5:12incentives are misaligned between the developers of this
- 5:17technology, of these technologies or algorithms and
- 5:22the can we say the intended recipients?
- 5:25Is that even a a correct way to phrase it?
- 5:27You have the hiring process for example with most big companies
- 5:31now is riddled with AI. Everything from resume, resume
- 5:35review and screening to video interviewing and to oversight.
- 5:42Once somebody gets the job, right, there's nothing inherent
- 5:46in that process that really benefits the person who's
- 5:48seeking work or is an employee, right?
- 5:51That's all intended to facilitate the life and the work
- 5:55of the employer. The bosses, same thing a lot of
- 5:58times with, you know, with public benefits, you've got
- 6:04relatively, you've got really dedicated public servants, but
- 6:07oftentimes they're unsophisticated in technology
- 6:09issues. They're thinking, OK, well, this
- 6:12new piece of technology is going to suddenly help expand our
- 6:15limited capacity, so let's implement it.
- 6:18And then they don't have what they need to do that in a non
- 6:23destructive way. And so the people who end up
- 6:25bearing the risk of, you know, sort of their own lack of
- 6:29knowledge or incompetence are the low income people that are
- 6:31subject to the decision making. These systems are complex.
- 6:35They are developed with algorithms and data as well.
- 6:44Can you isolate where a a primary source of the problem
- 6:51lies? I realized underneath it all,
- 6:53you have a human intention, trying to solve a problem,
- 6:57trying to achieve a goal. But if you can, you drill into
- 7:00that a little bit, kind of dissect this for us.
- 7:03There were a couple aspects to the way the algorithm worked.
- 7:07One is the mechanics of it right?
- 7:09What inputs turn into what outputs?
- 7:13And that's hard enough to discern, but then there's the
- 7:16reason that those inputs are chosen to lead to those outputs,
- 7:20right? Like why do you look at this
- 7:22factor and not this factor? Why is this factor shaped to
- 7:26look back three days instead of five days?
- 7:29All of those things, those are all human decisions Now.
- 7:32They're informed by, in the best cases, statistical science.
- 7:38In a lot of cases, there is no science is a bad descriptor for
- 7:43that. A lot of times it's junk, right,
- 7:44that somebody just invented and comes up with.
- 7:47But in the best cases in statistical science, that still
- 7:49is riddled with various assumptions.
- 7:52And so in our example, for example, in Arkansas, for
- 7:55example, whether or not somebody could use the bath on their own
- 8:07might not have been a factor that the algorithm considered.
- 8:10And that's weird, right? I mean, we're talking about home
- 8:12care for elderly or disabled people.
- 8:15Being able to bathe on your own should be 1 factor that decides
- 8:17how many hours of care you need. It wasn't or your ability to
- 8:21prepare meals wasn't a factor. And so you see this disconnect
- 8:25of like, we know instinctively or, you know, through medical
- 8:29discretion and judgment, how to answer this question of how much
- 8:33care somebody needs. It might be imprecise, but we
- 8:36know we know what we should be looking at.
- 8:38But the algorithm didn't do that.
- 8:40They looked at a lot of factors that weren't kind of intuitive,
- 8:43and then they ignored a lot of factors that were intuitive.
- 8:48How does this come about? Is it simply lack of
- 8:51understanding of the the subjects of the the target?
- 8:57How did? What happens here?
- 8:59Some of it is real ignorance about, you know, the lives of
- 9:03poor people and the ways that decisions are made and the
- 9:06impact of the decisions. Some is ignorance about certain
- 9:10program standards or laws or anything else.
- 9:13I've seen that a lot in the technology.
- 9:15Some of it is the lack of having to get it right.
- 9:19You know, for a lot of the developers of these algorithms
- 9:22in particular, they're shielded from any sort of consequences
- 9:26for their actions. And so they do what they think
- 9:30is best or what they can sell to a client and that's that.
- 9:34And then the clients that are using it, the government
- 9:36agencies or the employers or whatever, they might not be
- 9:40vetting it or, you know, they are also insulated from the
- 9:44accountability because if it hurts poor people, what's going
- 9:48to happen to them? Like what's going to happen to
- 9:50the person who decided to use it?
- 9:52I mean, poor people often times are not a particularly empowered
- 9:55political bloc. They're usually aren't scandals
- 9:57that end up resulting in lost jobs or lost elections for
- 10:02officials who are in charge of this stuff.
- 10:04And so it's easy to get away with really harmful actions just
- 10:09because you're doing it to people who don't have a lot of
- 10:13power that's ready at hand. You know, low income communities
- 10:16have always been super involved in advocating for themselves and
- 10:20organizing and everything else. But that's a huge effort, right?
- 10:24And takes like a concentrated movement.
- 10:26And it's not like you can just call your elected official and
- 10:29you have that kind of access and say, hey, this is a problem.
- 10:31Can you take care of this for me?
- 10:32Or organize a lobbying effort to get rid of something?
- 10:35Now, if you're doing something with poor people and it hurts
- 10:38them, you're not going to face immediate consequences for the
- 10:40most part. Folks who are listening, I want
- 10:44you to ask questions. We have some questions that are
- 10:48starting to come in on LinkedIn and Twitter and we're going to,
- 10:51we're going to get to them in a in a couple of minutes.
- 10:55If you're watching on Twitter, just insert your questions into
- 11:00Twitter using the hashtag CXO Talk.
- 11:03If you're watching on LinkedIn, just pop your questions into the
- 11:06chat. For those of you who are
- 11:08developing these kinds of systems and we hear a lot of
- 11:13discussion of ethical AI and responsible technology, here's
- 11:18an opportunity to ask somebody who's dealing with the actual
- 11:23fallout of this. So ask your questions, Kevin,
- 11:28what about the scale of the problem?
- 11:30How? How big an issue is this
- 11:32actually? All 92 million low income people
- 11:35in the United States have some key aspect of their life decided
- 11:38by AI, whether that is sort of housing, healthcare, public
- 11:43benefits, work, their kids, school, family stability, all of
- 11:47these issues. Not everyone might have all of
- 11:50those issues decided by AI, but everyone has at least one of
- 11:53those issues decided by AI. And then it extends beyond low
- 11:58income people as well into higher income things.
- 12:01So there have been a lot of stories, for example, about
- 12:03employer use of AI, sort of the screening aspect and then sort
- 12:07of the bossware management aspect of it being used against
- 12:12finance executives or against hospital chaplains, against
- 12:16therapists. Recently there was a story about
- 12:19Amazon programmers who are now subjected to AI based oversight
- 12:24and measurement and it's affecting their lives.
- 12:26So even though a lot of this stuff is most prevalent and
- 12:29probably most severe in the lives of low income people, it's
- 12:32happening to all of us. Healthcare is another great
- 12:35example, right? If our doctor recommends a
- 12:37treatment for us, many of the more expensive treatments are
- 12:41subject to health insurance company review prior to being
- 12:43offered, and those health insurance companies are using AI
- 12:46generally to deny those requests.
- 12:50We all know about United Healthcare and their use of
- 12:53algorithms that they say is neutral.
- 12:57And we don't do that. But it's, you know, you hear
- 12:59doctors complaining about how algorithms are interfering with
- 13:05their ability to render the kind of care that they want.
- 13:08And so it becomes pretty evident that what what was once targeted
- 13:15at lower income people now through the acceleration of AI
- 13:21is broadening and touches all of us at this point, I would
- 13:25imagine. In one of the examples of the
- 13:27health insurance companies, they ostensibly had a human reviewer
- 13:32reviewing the the AI's outputs, but when the investigation.
- 13:39Dug into what that human review looked like, it showed that the
- 13:43doctor was approving something like 60 prior authorization
- 13:46requests a minute like they had one or two seconds per one.
- 13:49There's no human reviewing that right.
- 13:51And it's bad faith to assert otherwise.
- 13:53And that's I think one of the key data points and there.
- 13:56Are a lot of others that help us show that this isn't just all
- 13:59purely accidental, this can't be just attributed to mistakes or
- 14:03errors, that there's a lot of thought and intention that goes
- 14:06behind, you know, developing and implementing these systems that
- 14:11are denying people really fundamental needs.
- 14:16Subscribe to our newsletter, go to cxotalk.com, check out our
- 14:20newsletter, and check us out for our next shows.
- 14:24We have great shows coming up. Let's jump to some questions.
- 14:27Let's begin with Arsalan Khan on Twitter.
- 14:32Arsalan's a regular listener. Thanks so much, Arsalan, for
- 14:35your regular listenership. And Arsalan says this.
- 14:39He says, who whoever sets the AI guardrails has the power, but
- 14:46who checks if those guardrails are equitable?
- 14:50And he asks, why don't we have a Hippocratic Oath for us as IT
- 14:57professionals? He's, he's an enterprise
- 14:59architect. So.
- 15:00So this notion of whoever sets the AI guardrails has the power,
- 15:04but who checks that the guardrails are right are
- 15:07equitable? The Hippocratic Oath idea is not
- 15:10a meaningful source of systemic change to insulate society from
- 15:18these harms because, you know, doctors have Hippocratic oaths.
- 15:24And while that might be useful, it doesn't a lot of times
- 15:27prevent some of the abuses in in medicine either or lawyers have
- 15:30obligations and it doesn't present prevent us from going
- 15:33and doing all sorts of random harmful things.
- 15:36So I think what you need is actual regulation to reinforce
- 15:40kind of the guardrail notion, right?
- 15:42Safeguard people from having any exposure to the harms in the 1st
- 15:47place, or if there are, because those kind of institutional and
- 15:51ethical safeguards fail, then there are real consequences for
- 15:57that. They go beyond somebody just
- 15:58violating their oath and feeling bad that way.
- 16:02So I don't know if that's getting at the full essence of
- 16:05the question, but that's where some of my thoughts go.
- 16:10And also, not everybody's as ethical as the person asking the
- 16:13question either, right? And some people are perfectly
- 16:15happy to just do whatever the client wants or program the
- 16:18system in whatever way is going to make it most profitable and
- 16:22attractive. And as long as they don't have
- 16:24anything holding them back formally, officially real
- 16:27consequences and accountable, we're not going to get any major
- 16:30change. So self policing is not
- 16:33sufficient in your view. Definitely not.
- 16:36And I know those questions are asked in good faith and are
- 16:39posited in good faith. But the people who are pushing
- 16:43that at the policy level are definitely not pushing it in
- 16:46good faith. They don't want any
- 16:48accountability. They don't want anything that
- 16:49would restrict how they use it, and they're perfectly happy to
- 16:54shunt off all the risks and all the dangers of their systems
- 16:58being bad or going wrong or doing something destructive to
- 17:02the people who are subject to those decisions.
- 17:05Are you talking about government policy or in corporate policy,
- 17:09people designing products? Government policy, the tech
- 17:13industry has been vociferous in their opposition to any sort of
- 17:18meaningful regulation of AI, automated decision making
- 17:23technologies and so forth. And that's the reason why we
- 17:28don't have any real societal protections against this stuff
- 17:31outside of existing laws. And even now they're targeting
- 17:36some of the European Union's restrictions, which are modest,
- 17:40but big tech doesn't like those. So that's where I'm talking
- 17:44about is sort of how kind of corporate interests end up
- 17:47shaping their policy positions in ways that are detrimental to
- 17:51really all of us that are not in that in that world, but
- 17:55particularly low income people. You also have many of the major
- 18:00tech companies pushing forth their own ethical AI initiatives
- 18:08and lots of discussions around the data and creating building
- 18:13bodies of data that try to weed out bias.
- 18:17I mean that you see this everywhere is happening.
- 18:20That's true, and there are a lot of.
- 18:21Good people who share my values in these companies and are
- 18:26trying to make the companies do as right as possible.
- 18:30But I think when the rubber hits the road, we've seen repeatedly
- 18:34that the folks speaking out for ethical uses are sidelined.
- 18:38You know, a few years ago in Google, for example, the whole
- 18:41ethical AI team, I think, was fired because they wanted to
- 18:44publish a paper that Google didn't want published.
- 18:48Or more recently, when, you know, Twitter was taken over by
- 18:53its current owner, the whole ethical AI team was disbanded
- 18:57instantly. You have Google's retrenchment
- 19:02of its ethical AI things and now it's technology is being
- 19:06deployed in unemployment hearings, right, for people who
- 19:08are desperate for benefits, even though we know that a lot of the
- 19:13AI technology involved can be faulty.
- 19:15So again, you do have these ethical components within
- 19:18institutions that are pushing, I believe in good faith a lot of
- 19:21times for changes, but the people who are pushing for that
- 19:26don't have the same interests as the institutions who are
- 19:28allowing it. A lot of times the institutions
- 19:30are allowing ethical AI because it allows them to go out and
- 19:32talk about their their concept of social responsibility.
- 19:36But we we see repeatedly when the rubber hits the road, ethics
- 19:39will go by the wayside and the company's profit incentives and
- 19:42motives are going to be what dictates what happens next.
- 19:45So basically, money talks, nobody walks.
- 19:47Yeah, I mean it's. Complicated, right?
- 19:49Because, again, there's a lot of good people in there that are
- 19:51pushing really hard for these major institutions that have
- 19:54lots of power to do right. And the fact that the
- 19:58institutions allow that to happen is noteworthy.
- 20:02I think it just comes, yeah, at the end, it it, it, it ends up
- 20:05being the money. Talks.
- 20:06I will say that you are up against the marketing budgets of
- 20:12some really, really large companies here.
- 20:16I am. This is going to change
- 20:17everything though, Michael. See CXO talk.
- 20:19This is going to be, this is going to be the entryway.
- 20:21This is going to this is better than all the all the marketing
- 20:23budgets of of the big tech companies right now.
- 20:26Let's jump to some other questions.
- 20:30And I'm seeing some themes developing in the questions
- 20:34here. And this next one is from Preeti
- 20:38Narayanan. And she says, given your work
- 20:42exposing large scale harm caused by AI and public services, what
- 20:47practical guardrails would you recommend to technology leaders
- 20:51like her, Like many of our listeners who are building
- 20:55enterprise AI systems so we don't unknowingly replicate
- 21:01those same failures at scale? Basically, it's the same
- 21:06sentiment as Arsalan Khan just brought up.
- 21:11What can we, as the people creating these systems, do?
- 21:15OK, one thing is push for regulation, right?
- 21:19And push for meaningful regulation of what it is that
- 21:21you do, because that way it bakes in consequences for
- 21:25getting it wrong. And as long as you have good
- 21:27faith and are doing things the right way, those consequences
- 21:31shouldn't be terribly severe. You shouldn't be exposed to them
- 21:33that you know, in in a way that's wholly destructive.
- 21:37So I think pushing for regulation is actually in your
- 21:39own interest, but kind of in the context of developing a
- 21:44particular product. You can ask, is this a
- 21:46legitimate use for AI? For example, should we be using
- 21:49AI to deny people, disabled people benefits and home care?
- 21:54That might not be a legitimate use of AI.
- 21:56And if it isn't a legitimate use, maybe we shouldn't do it
- 21:58and we should just say that's off, off limits.
- 22:00We're not going to do that no matter how much somebody's going
- 22:02to pay us because we just don't believe that's fair.
- 22:04Now, if it is a legitimate use, and I acknowledge there's a lot
- 22:07of kind of Gray areas in this, then you've got to have a really
- 22:10intensive development and vetting process.
- 22:14What are you doing? What are you, what data are you
- 22:16using? Are you projecting out the
- 22:18harms? Are you consulting in a
- 22:21meaningful way with actual oversight?
- 22:23The people who are going to be subjected to these decisions, do
- 22:26they have some sort of say in how it's developed in a way that
- 22:29would actually stop you from moving forward or force a
- 22:31different development of it? Are you willing to disclose
- 22:35things that might traditionally be considered a trade secrets or
- 22:37intellectual property in the interests of having more public
- 22:40accountability? Are you willing to ensure
- 22:45ongoing oversight so that if your product is developed or is
- 22:50is deployed, it's deployed first of all in narrow, short phased
- 22:56ways so that we can test the harm before it's applied to
- 22:58everybody? And then two, are we willing to
- 23:00look over time in a three month span and see, hey, does our
- 23:04projected impact which we have documented and have disclosed to
- 23:07the public differ from what the actual impact is?
- 23:10And if so, is there an automatic off switch?
- 23:12Is there some way to to course correct that?
- 23:15And all of those things, when combined with meaningful, you
- 23:20know, legislation that means that their people have
- 23:23enforceable rights if they're hurt by it, would lead to reduce
- 23:27chances of harms on systemic society wide scales.
- 23:31If I were a corporate leader, you made the assertion that we
- 23:37should question whether AI is the appropriate decision making
- 23:42tool to use in some of these situations that could Causeway
- 23:47real downstream harms. But I would push back an I would
- 23:51say, Sir, you don't know what you're talking about because AI
- 23:56is a decision tool. It is not autonomous.
- 24:00It's overseen by humans. The data that we collect is
- 24:04carefully vetted to be biased. And it's unfortunate that these
- 24:09downstream harms are happening, but it's not a result of our
- 24:13decision making. There are systemic underlying
- 24:16societal issues and frankly, the AI is making the right decision.
- 24:22I would challenge almost everything that you said there,
- 24:24Michael, from, you know, the the sophistication of the vetting
- 24:29process. The people who are developing
- 24:32enterprise software might be doing a better job when the
- 24:36people are going to buy their software are wealthier or richer
- 24:39than when it works for low income people.
- 24:40So first of all, I think like who the audience is, who's going
- 24:43to be subjected to this dictates a lot of how careful the kind of
- 24:48development process is. And if it's going to be deployed
- 24:50against poor people, the development process doesn't need
- 24:52to be as intensive probably as it would be for corporate
- 24:57clients, right? So I think there's there's that.
- 25:01So a lot of the so-called science in AI is really junk
- 25:05when it applies to poor people. 1 great example of that is
- 25:11identity verification, for example, during the pandemic.
- 25:15And hopefully some of your listeners listeners will have
- 25:17some some frame of reference. But during the height of the
- 25:20pandemic, right, masses of people were unemployed.
- 25:23Congress expanded unemployment benefits to help people float
- 25:27during these, you know, desperate times.
- 25:31At some point, states, encouraged by the federal
- 25:35government, implemented ID verification measures and what
- 25:38they would algorithmic ones. And So what they would do is
- 25:40they would run every active claim and every application that
- 25:44was outstanding through these ID verification algorithms.
- 25:48And the algorithm would flag claims that it noted as
- 25:51suspicious. And then what would happen is
- 25:53the person would have to who is flagged would have to present
- 25:56physical proof that they are who they say they are.
- 25:59That happened. And then still the state didn't
- 26:02have capacity to process that verification.
- 26:06And so you ended up with millions and millions and
- 26:08millions of people who are in desperate circumstances, can't
- 26:12keep their lights on, can't pay their rent, can't get.
- 26:15School supplies for the kids who had their benefits stopped or
- 26:19delayed by months and months and months because of this identity
- 26:22verification algorithm. Now what would happen?
- 26:25How did it work? One of the factors is, are you
- 26:29applying from the same address as somebody else, including with
- 26:33apartment buildings? So if I live in unit one O 1 and
- 26:37somebody else is applying for unemployment benefits that lives
- 26:39in unit 3O3, both of us are flagged.
- 26:42That's ridiculous. That's somebody in their
- 26:43basement coming up with some junk that make that they think
- 26:46would be associated with fraud. There's nothing statistical
- 26:50about that. There's nothing scientific about
- 26:51that. That's somebody just inventing
- 26:53stuff, right? But it invents stuff and it
- 26:55causes millions and millions of people desperation that you
- 26:58couldn't imagine. I had clients who were calling
- 27:01with active mental health crises talking about self harm because
- 27:04they couldn't get unemployment benefits even though they were
- 27:07who they said they were. And they showed that to the
- 27:09state. So that's an example of, you
- 27:12know, maybe the maybe, you know, maybe some companies care more
- 27:18than others. But here when rubber hit the
- 27:20road, it didn't matter. And ultimately studies came that
- 27:23came out after that we're assessing sort of the validity
- 27:28of these tools showed that the for the, for the most part, they
- 27:32caught eligible people, right, They weren't targeted narrowly
- 27:36to ensure that we're only getting the few.
- 27:38That are actively suspicious No, they end up catching essentially
- 27:41everybody and then just leaving, leaving folks to to to try to
- 27:46wade through the mess on their own.
- 27:48And that's just not, you know, that's not acceptable.
- 27:50There's no justification for that kind of stuff.
- 27:52Michelle Clark on LinkedIn says, can the problem of bias data be
- 27:58solved? And let me just reframe that.
- 28:00How do you manage the fact that that people are struggling to
- 28:04have data that is lacking bias? I've spoken with many of these
- 28:08folks on CXO talk, but but it's a really tough challenge from a
- 28:12technical standpoint. So.
- 28:14So what do we do about that bias data?
- 28:16Biased data is only one part of the problem, right?
- 28:18And that there are other parts of the problem.
- 28:20You can have unbiased algorithms that still cause massive harms
- 28:23and that I think would still be illegitimate in a lot of ways.
- 28:27So we want to make sure we talk about the risk in more ways than
- 28:31bias. But bias is a big one.
- 28:34And when we talk about it, there have been various ideas about
- 28:38debiasing data. And to be fair, I don't have the
- 28:43full technical background to understand the statistical
- 28:45science between all of the different ways and which is the
- 28:48best at doing what. So I don't want to claim
- 28:51otherwise. But what I do understand is that
- 28:53there's, you know, sophisticated like trying to get more data
- 28:58sets that are validated, trying to account for historical
- 29:02exclusion, again using the data in real world examples, but that
- 29:13don't have real world consequences and so forth, so
- 29:18that you're hopefully getting better data.
- 29:21So I think all that is is very much possible.
- 29:26But you know, again, I think the best, the best test against
- 29:31biased data is going to be, you know, once it's out in the
- 29:36world, are you going to face consequences for what you put
- 29:40out there, right. And if you are going to face
- 29:42consequences, then you're going to make sure or you're going to
- 29:45do your very, very best efforts to ensure that your data is not
- 29:49biased in a way that's leading to to unfair outcomes for folks.
- 29:53Self regulation is not sufficient regulation in this
- 29:56case. Yeah, exactly.
- 29:57We see the bias example all the time, right?
- 29:59There's the obvious healthcare examples about who gets
- 30:01transplant plants or, you know, black folks pain being treated
- 30:06as less real than people who are white.
- 30:09And various other examples in the healthcare context of AI
- 30:13that's deployed with, you know, bias baked in the ad targeting
- 30:17stuff from social media, like all of these things.
- 30:21And then there's another deeper question, which is if you can't
- 30:23figure it out, if you can't debias your data, maybe you
- 30:27shouldn't be using it. Maybe what you're trying to do
- 30:30is not so important that you're going to go out and reproduce
- 30:35long standing societal inequities with your technology.
- 30:38Maybe the money is not worth it. That's a value judgement I guess
- 30:42for every person and every company to make.
- 30:46But of course everybody is going to say, well, we are careful.
- 30:51That's the point. I mean, I think this comes back
- 30:54to one of my points that, you know, ultimately meaningful,
- 30:57robust, enforceable regulations are part of your.
- 31:01Gain your interest. If you are a company that is
- 31:03committed to doing things right, subjecting yourself to
- 31:08accountability is going to be a competitive advantage, right?
- 31:13Because if you have other people who are not doing things right
- 31:16and they can be subjected to lawsuits that are consequential,
- 31:20they can be subjected to regulatory oversight that's
- 31:22meaningful, that's going to be a competitive advantage for you.
- 31:25You can say, look, we are not caught up in any of that stuff.
- 31:28They are. And so we're a safer bet.
- 31:30We're a better bet. You can tap the societal values
- 31:33that that, that you provide all of those things.
- 31:36So I think ultimately regulation is in your interest because it
- 31:40creates a new competitive space for you, a competitive surface.
- 31:43I guess I'd rather say. I just want to mention for folks
- 31:46that are interested in the technology, technical
- 31:50underpinnings of data and bias, just search on the CXO talk site
- 31:56because we have done interviews with some of the leading
- 32:00technologists in the world who are focused on this problem.
- 32:05So just search for data bias and so forth on cxotalk.com.
- 32:09And oh, by the way, while you're there, you should subscribe to
- 32:13our newsletter so we can keep you up to date on shows like
- 32:17this because we have incredible shows coming up.
- 32:19Our next show, not next week, the week after is with the Chief
- 32:23Technology Officer of AMD. So subscribe to the newsletter
- 32:29we have. Our next question is from Greg
- 32:32Walters, who's another regular listener.
- 32:34And Greg, thank you. And Greg says AI is not like old
- 32:40school digital transformation broadly.
- 32:43Can AI help raise us up out of low income?
- 32:49No, not with current incentive structures in the current system
- 32:52that we exist in. People always ask me like, what
- 32:56about AI for good, right? Like what can we do that would
- 32:59advance justice? There's one example I always
- 33:03like to offer, which is with public benefits, say Medicaid or
- 33:08SNAP, which is Nutrition Assistance.
- 33:11The government knows most of the time what income and assets
- 33:16people have, right? That is, that information is
- 33:18accessible to them in some form. They know they could make
- 33:23eligibility decisions oftentimes without any or with minimal kind
- 33:28of involvement from the person who would qualify for the
- 33:30benefits. And so if you could build a
- 33:32system that would accurately, fairly consistently make those
- 33:36eligibility decisions, minimize paperwork and other burden on,
- 33:40on folks, that would be a net wonderful good that would do
- 33:43more good than 100 legal aid or lawyers in our lifetimes ever
- 33:46could. The problem is, is big companies
- 33:49have tried it, big government vendors have tried it, and it
- 33:51repeatedly fails in the same way.
- 33:54Why does it fail? Because of failed accountability
- 33:56mechanisms, right? You don't have political
- 33:58accountability, as we talked about, because hurting poor
- 34:00people generally isn't a scandal that's going to get anybody
- 34:02booted out of office. You don't have market
- 34:05accountability oftentimes in the government vendor contract
- 34:07context is because there's very few government vendors of the
- 34:10size needed to be able to to compete with one another.
- 34:16But even beyond that, you have market failures in terms of
- 34:18transparency of how your product works and what kind of public
- 34:21oversight it's it's subject to. And then you have no legal
- 34:25accountability because the existing laws that we have,
- 34:28while they have been used effectively by advocates like
- 34:31myself, are limited in scope and can only get a certain amount
- 34:35certain kind of relief. A lot of times I can't get money
- 34:37damages for the suffering that people's 'cause you can just get
- 34:41a judge that tells the state or that tells the vendor to change
- 34:44what they're doing. And so you have all these broken
- 34:47accountability mechanisms, which means that even with this good
- 34:50use, right, helping people get the healthcare that they are
- 34:53eligible for, you don't see that brought about in real life.
- 34:56And so if you can't do something like that, you're not going to
- 34:59do anything else in terms of alleviating poverty at scale.
- 35:04You can have some cool projects like in the legal world, there's
- 35:07like know your rights projects, right?
- 35:09Everybody's had a ban. Hopefully everybody's had a bad
- 35:11landlord at some point in their life, right?
- 35:13Where you needed to request repairs or ask for your security
- 35:17deposit back after you left and they were trying to hold on to
- 35:19it. There have been some cool AI
- 35:21based tools that help people do that.
- 35:24And that's cool stuff. It's great, but you know it's a
- 35:27grain of sand on the beach that borders the Pacific Ocean,
- 35:33right? Like it's cool, but it's not
- 35:36scale. This leads us to an important
- 35:41question from Trenton Butler on LinkedIn, who is a community
- 35:46advocate, organizer and project manager.
- 35:50And Trenton says this for those of us committed to ensuring
- 35:53these tools are used ethically, how can we get involved,
- 35:57especially if one does not come from a law or technology
- 36:01background? This is an important aspect is
- 36:04there's a lot of power building and community organizing that
- 36:07can be done. You know, some of the AI stuff
- 36:09happens at a very local level, right?
- 36:11Some school districts, actually about half of school districts
- 36:14use AI to predict what kids might in the future commit
- 36:19crime, right? And then targets them for law
- 36:23enforcement harassment or terror in in some cases, right?
- 36:27That's something where you could find out as a citizen.
- 36:29You don't need to be a lawyer. You can do open records
- 36:32requests. You can go to school board
- 36:33meetings. You can ask people, hey, is AI
- 36:35being used here and how does it, how does it work?
- 36:37And if it is and it looks bad, and most of the time it is bad,
- 36:42you can help organize people to get involved.
- 36:45Another local fight that there is is data centers.
- 36:48These are a big deal right there.
- 36:49The way that all the data that AI depends on is processed.
- 36:53They're subject to local land use laws, local regulation
- 36:57around utility prices and other things.
- 37:00So there are a couple ways really at home that you can get
- 37:03involved in this and building knowledge of yourself, building
- 37:07knowledge of journalists and the public, holding meetings,
- 37:10getting your neighbors involved and all that stuff.
- 37:13And it can be daunting. So and there's a huge gap in
- 37:17helping people do that right now.
- 37:18And that's one of the reason tectonic justice exists.
- 37:21So in the in the self interested plug, please follow us, please
- 37:24stay in contact. And as we're building out, it's
- 37:26just me and it's been my first two employees joined last month.
- 37:30So we're still very much in the building phase.
- 37:32But as we get more established, we want to be doing working in
- 37:36partnership with folks who want to be engaged around these
- 37:38issues. So, so please stay up with us.
- 37:40We have another question now from Twitter and this is from
- 37:48Chris Peterson, who says what agency or is there is or is
- 37:54there just one? Would you suggest as the AI
- 37:57Ombudsman in the US? He says also that for folks in
- 38:02charge of big AI, 99 plus percent of us are lower income
- 38:08in quotes. There is no one ombudsperson
- 38:12kind of yet around AI. And I mean, that's an
- 38:16interesting idea in terms of meaningful accountability
- 38:19because there are ombuds people in healthcare and in nursing
- 38:22homes and other similar entities.
- 38:26It's a huge gap. That's part of why we exist is
- 38:28right to be focused people on the ground, right?
- 38:31Like I was a lawyer working with hundreds and hundreds of low
- 38:34income people to try to fight this stuff.
- 38:38So I think in the ecosystem, sort of the nonprofit ecosystem,
- 38:44there are few organizations that are trying to build up the
- 38:46capacity to do some of this stuff to watchdog the use of AI.
- 38:51And then there are a lot of established organizations that
- 38:53are more focused on kind of the policy level.
- 38:55So there is no one ombudsperson. In terms of the other aspect of
- 39:02the question, I guess I would need more context about what it
- 39:06means that 99% of us are subject.
- 39:10Maybe it's that we're subject to the big tech.
- 39:12What he was saying is that it's it's the billionaire question.
- 39:18Now is the time to get involved before these technologies become
- 39:23entrenched as legitimate ways to make decisions about these core
- 39:26aspects of life. Because even though AIA lot of
- 39:30times purports to be, or at least it's it's hype.
- 39:33It's hype, it's hype men purported to be kind of this
- 39:38objective way to make decisions. Whose objectivity is it, right?
- 39:43If it's always limiting access to benefits, if it's always
- 39:45making housing or jobs or education harder to get, then
- 39:50it's not really objective. It's it's, you know, the people
- 39:53who are developing or using the AI, It's achieving their ends.
- 39:57So now is very much the moment for it, I think, because this
- 40:00field is. Relatively new is sort of a
- 40:03social phenomenon and a social movement.
- 40:05There isn't a lot of the infrastructure that needs to be
- 40:08there to help people get organized and engaged around it.
- 40:11So a lot of my answers are probably unsatisfying.
- 40:14It's like, we'll talk to your community, organize around it,
- 40:16stay up with tectonic justice, these kinds of things.
- 40:19Because that's what we're trying to build, is build the
- 40:21infrastructure for people to be able to channel their front,
- 40:24their concerns, their frustrations, their energy
- 40:27towards ensuring something that looks more like justice.
- 40:30But aren't you in a way trying to turn back the clock to a a a
- 40:38simpler and easier time Before we had AI and AI is not going
- 40:44away and it's growth is going to continue to make incursions into
- 40:49every aspect of decision making. That's the overwhelming you
- 40:54talked about earlier, the PR budgets, right, of big tech.
- 40:58And that's the overwhelming sense is that it's inevitable.
- 41:01But is it really right? Why can't a nurse make a
- 41:05decision about how much home care a disabled person needs?
- 41:08Why is that not viable anymore? Why shouldn't that be the case?
- 41:12Why can't we use technology in a way that supports human based
- 41:18decision making rather than essentially making the decision
- 41:22for us with like cursory human oversight at all?
- 41:25And I think that has to be the questions, what is the
- 41:28legitimate use of AI? And then even where the use is
- 41:33sort of legitimate, let's go through all the vetting we
- 41:35talked about earlier. But let's also talk about the
- 41:37bigger picture. Questions in terms of what it
- 41:39means for the Earth, right? We know that AI has
- 41:42environmental consequences. There's debate about how many
- 41:44liters of water each ChatGPT you know, prompt uses or whatever,
- 41:48but like we know that it's draining water in certain places
- 41:52where water is scarce. We know that it's responsible or
- 41:55at least correlated with energy price increases.
- 41:58We know it's correlated with the use of non renewable energies.
- 42:02So you have to factor all these things into the equation in
- 42:06terms of its societal value and its societal costs.
- 42:09And it may be that if we actually do a concerted,
- 42:13reflected effort that accounts for all these externalities, we
- 42:18realize, you know what, this isn't that harm.
- 42:20Maybe we shouldn't do it or we should only do it in these
- 42:22limited circumstances. And I think that's what we have
- 42:25to be engaging. In and that's why I always
- 42:27reject the frame of inevitability.
- 42:29I'm a practical person. I generally need to solve
- 42:31problems for my low income clients and that doesn't always
- 42:34allow me to like, be pie in the sky principled.
- 42:37But we can be pie in the sky principled.
- 42:39While also being practical and, and start thinking like, is this
- 42:43really worth it? Is the productivity gain really
- 42:47worth all the cost? And so far even that in the
- 42:50corporate sphere hasn't been clear that there are really net
- 42:54productivity gains, particularly when you factor in the required
- 42:58human oversight for its continued use.
- 43:01And so I don't think it's inevitable.
- 43:03I think it will be inevitable if we don't in the next, you know.
- 43:06Decade or two really reckon with the implications of it.
- 43:10My friend, you have an uphill fight.
- 43:12I have to say on this point, you and I have to agree to disagree
- 43:18because as I look out over the developments of AI and automated
- 43:24decision making, I cannot see, I cannot fathom, and maybe I
- 43:32reflect kind of a typical technology viewpoint, but I
- 43:36cannot fathom that AI is not going to grow much as the steam
- 43:42engine influenced every facet of our lives.
- 43:45And you can say the steam engine also caused a lot of problems.
- 43:48Potentially. I mean, I think in society, it's
- 43:50not like we just accept technology inevitably without,
- 43:54you know, restricting its use. I mean, certainly nuclear energy
- 43:59has had significant use restrictions around it and its
- 44:05development and where it can be used and everything else.
- 44:08Cars have had a lot of restrictions around how they can
- 44:12be used. Everybody, I'm sure thought
- 44:13Ralph Nader in the 70s was ridiculous for, you know,
- 44:17advocating for seatbelts, right? And now that's just an accepted
- 44:20facet of the cars. Now that doesn't take care of
- 44:22all the harms that cars are potentially causing, right?
- 44:25And I'm not saying that it does, but it's one example of of
- 44:30movement that way. And all of these things are
- 44:34have, you know, essentially corporate power and lots of
- 44:38money going against, you know, people who seem like they're in
- 44:42the way of inevitability. But we have to be a little bit,
- 44:47you know, what's the word? We have to believe that
- 44:50something more is possible. Otherwise we just resign
- 44:53ourselves to accepting the worst version of whatever it is that
- 44:57we're fighting against. And that's not a, that's not a
- 44:59concession I'm willing to make. Like I'll fight like hell, maybe
- 45:02I'll lose, but I bet you that we're better off because of the
- 45:06fight than if nobody fought. Let's jump to another question,
- 45:10and this is from Ravi Karkara on LinkedIn.
- 45:14He says, oh and I should mention he is Co founder and author of
- 45:19AT AI for food global initiative.
- 45:22And he says, how should global stakeholders navigate the
- 45:27ethical challenges and data governance differences posed by
- 45:31China's AI strategy, particularly its state centric
- 45:36data policies, while promoting international norms for
- 45:40responsible and transparent AI development?
- 45:44Not sure how much expertise you have in in China, but thoughts
- 45:48on kind of a global perspective? In the global context, the AI
- 45:52discussion becomes even more interesting because there's a
- 45:57lot of people who are pushing AI as a solution to kind of global
- 46:00poverty, right, and inaccessible healthcare, right.
- 46:04You get the the story of like people in remote villages, you
- 46:11know, in the majority world who are now suddenly able to access,
- 46:14you know, medical care or at least knowledge about medical
- 46:17care that they couldn't because they couldn't travel to cities
- 46:19and so forth. And I think there's, you know,
- 46:22who am I to say sitting here and you know, in the US, in Los
- 46:26Angeles, CA, to say that that's a bad use of AII.
- 46:31Think where I care about. There are a few things.
- 46:33One is the data extraction that comes from, you know, expanded
- 46:39use of AI. Is it fair to be extracting all
- 46:41the data about people's behaviors, who they are,
- 46:43etcetera, etcetera, when you're going to monetize that and when
- 46:46they really don't have meaningful consent, right?
- 46:48Opting into opting into the terms of service on a contract
- 46:52for social media, for example, that's not a real thing of
- 46:55consent for most people. So what's the data extraction
- 46:59relationship? What's the labor relationship,
- 47:01right? Because just as there's a person
- 47:03who needs to seek healthcare in a village, right?
- 47:06And this is an archetype, I'm not trying to use a specific
- 47:09example, there's. Somebody not too far away.
- 47:13Who's being paid pennies on the dollar to view really horrific,
- 47:17traumatic data and label it right?
- 47:20There are people being exploited for the supply chain and
- 47:23everything else. So I think as we transition to
- 47:26the global discussion, you're going to have a lot of these use
- 47:29cases of AI for good that are going to be uplifted to justify
- 47:33the continuance of the AI regime.
- 47:35And if we're being reflective people that are serious about
- 47:39kind of the policy implications of this, we need to factor in
- 47:42all the costs. What is the what are the costs
- 47:44of the data extraction of the labor exploitation?
- 47:47What are the downstream costs of having other people's lives
- 47:49decided by the not so good and not so innocent uses of AI?
- 47:53This is from Elizabeth Shaw, who says how does today's AI differ
- 48:00from previous algorithms from the view of social harms, and
- 48:05can AI be part of the solution? So really the question is what's
- 48:09unique about AI and can AI help solve these problems?
- 48:14A lot of the technologies that are used in government services
- 48:17right now are not the latest generation AI, you know, LLMS
- 48:22and other things like this. A lot of them are more older
- 48:26algorithms that are based, you know, that were supervised
- 48:29learning that were based on statistical regression and these
- 48:33sorts of technologies and those are really harmful.
- 48:36I don't think AI has any like the latest generation AI has
- 48:39anything to offer in a lot of these, in a lot of these
- 48:44contexts. Again, so long as it's existing
- 48:46for purposes that are to, you know, essentially limit life
- 48:50opportunities. And in this, you know, vacuum of
- 48:53accountability, I don't think the technological sophistication
- 48:56is going to make much of a difference because they're going
- 48:59to be making the same decisions with the same incentives, right?
- 49:02And one way that we are seeing this now is, excuse me, in the
- 49:08recent developments federally, right, where the administration
- 49:12has implemented AI and for example, Social Security offices
- 49:17and it's made Social Security harder to access.
- 49:19It's made people have to wait longer.
- 49:21It's made people not get their benefits.
- 49:23And that is technically the latest generation of AI.
- 49:26So I think that's an example of, I always challenge the promise
- 49:32that the premise that AI is going to somehow fix existing
- 49:35problems just because of the technology is going to get more
- 49:37sophisticated. No, what's going to happen is
- 49:39it's going to make those problems even harder to fight as
- 49:41the technology becomes even more inscrutable, more insulated from
- 49:45public accountability and transparency and all of those
- 49:48things. You are not of the school of
- 49:51thought that AI is going to be the great savior.
- 49:54Oh, God, no. Oh God, no, no.
- 49:57If anything, it's the opposite way, right?
- 49:59It's immiserating people. And I think, you know, the
- 50:03recent use of AI in the last few months by the administration
- 50:09helps show us this is devastating.
- 50:11AI is being used to destroy government, destroy government
- 50:16capacity, destroy lives, and violate the law left and right
- 50:22and everything else. It is a weapon that is uniquely
- 50:25suited, uniquely suited to authoritarianism, right?
- 50:30Even by its nature, it has inscrutable.
- 50:32It's sort of just as like an the Oracle that tells you what the
- 50:35decision is, but doesn't tell you why it's making the decision
- 50:37doesn't allow you to disagree with it.
- 50:39All of that, That's like an authoritarian approach to
- 50:42thinking. And to decision making.
- 50:45So no, if if anything, AI is a greater threat.
- 50:48So I think our continued existence as a, you know, as a
- 50:52democratic society, it's antithetical to a lot of
- 50:55egalitarian notions, if anything is going to make things worse.
- 50:59Can you offer advice to folks who are working in the corporate
- 51:05realm, who really have a conscience and who don't want to
- 51:08see the perpetuation of these kinds of harms that you've been
- 51:12describing? The current uses of AI are
- 51:14destroying its reputation. I think that's brand risk for
- 51:17your companies. I think that's brand risk for AI
- 51:20as it's as a as a sort of venture.
- 51:23And I think opposing authoritarianism, particularly
- 51:28authoritarianism that's being fueled by AI is a really
- 51:32critical thing for your long term survival for various
- 51:36reasons. Then you know on a less sort of
- 51:39global and do me scale is all of the things that we're talking
- 51:44about push for meaningful regulation.
- 51:47What are you scared of? That's my question.
- 51:49Like, if you've got this great product that's backed by the
- 51:51most sophisticated science we have, what are you scared of?
- 51:53You should be proud of that. You should be putting that out
- 51:55there and saying, you know what? Subject us to accountability
- 51:58because our stuff is so strong, so scientifically sound and
- 52:02produces such clear value for the public that we're willing to
- 52:07embrace the being under a microscope.
- 52:11And I don't see that yet. And that's why I even challenge,
- 52:13you know, the the notion of an inevitability in terms of pure
- 52:17efficiency. There haven't been clear one
- 52:22sided efficiency gains that have made adoption of AI, even for
- 52:27non decision making purposes, universally sensible.
- 52:31Help make AI an electoral issue. Let's start talking about the
- 52:34injustices. I mean, I think there's going to
- 52:36be in some incentive problems there because, you know, there's
- 52:39big tech money that funds both parties.
- 52:41And I think there are a lot of people who don't want to be
- 52:43accused of being a Luddite. And, you know, there's other
- 52:45incentives there. But I think policy makers have a
- 52:49responsibility to educate the public much more intensely than
- 52:52they currently do about the harms of AI.
- 52:55Engage the public, hopefully create a base of people so that
- 52:59there's a balance, a counterbalance to the to the
- 53:01weight of big tech in these discussions so that you can push
- 53:05for meaningful legislation and regulation and ongoing and
- 53:08enforcement and oversight. And I think that's going to be
- 53:10vital to, you know, again, sustaining a democratic society,
- 53:16pushing for less inequality and ultimately having an environment
- 53:21where people have a real chance to thrive.
- 53:23What advice do you have to individuals who are victims of
- 53:30on unjust AI decisions? This is really hard.
- 53:36A lot of times you don't even know that AI is the reason for
- 53:40that you're suffering. So what I would say is contact
- 53:44your local legal aid program if you're hurt by this stuff.
- 53:46Legal Aid provides free legal services to folks throughout the
- 53:49country on civil legal matters. Talk to your neighbors, talk to
- 53:53other people in the same situations.
- 53:55Try to see if things are going, what's going on, and gather
- 53:58information and start kind of engaging in the things that are
- 54:02needed to push back. And if you're in a position to
- 54:04sue, if you're in a position to offer your story to a
- 54:08journalist, take those opportunities to speak for
- 54:12yourself. Because there are relatively few
- 54:16stories out there. And the discourse doesn't have
- 54:20the people who are hurt the most, doesn't have the people
- 54:22who are having to live with the consequences of what powerful
- 54:25people do. And any chance that we have for
- 54:27long term success is going to depend on you being able to
- 54:31become a leader and to Share your story and share your
- 54:34passion and share your injustice so that we can make it better
- 54:37for everybody. Kevin Delebin, founder of
- 54:40Tectonic Justice, thank you so much for taking your time to be
- 54:45here. And I'm very grateful for you to
- 54:47for you sharing a point of view that honestly is quite different
- 54:52from that that we usually hear on CXO Talk.
- 54:56So thank you for taking your time and being here with us.
- 54:58This is really fun and thank you also to the audience for all the
- 55:01great questions and you for having me, Michael.
- 55:03Audience, thank you guys. You guys are awesome.
- 55:06I mean truly that your questions are so thoughtful.
- 55:08You guys are so smart. Before you go, subscribe to our
- 55:13newsletter, go to cxotalk.com, check out our newsletter and
- 55:19check us out for our next shows. We have great shows coming up.
- 55:22And if you're interested again in topics like data bias, all of
- 55:27these issues that we've been discussing, search on the CXO
- 55:31Talk site because we have had lots of perspectives on this
- 55:36from business leaders, from politicians, you name it.
- 55:39So dig into the interviews on cxotalk.com.
- 55:42It's it's truly a great resource.
- 55:45Thanks so much, everybody. We'll see you again next time,
- 55:47and I hope you have a great day.