Latest / CLEs You Actually Want to Hear / Nevertheless She Persisted: Gender Bias in the Age of Gen-AI
Transcript
- 0:03Welcome everybody to Nevertheless She Persisted,
- 0:07Gender Bias in the Age of Gen .AI. I'm Mark Hoag,
- 0:13chair of the Beverly Hills Bar Association's
- 0:16freshly minted AI in the law section, along with
- 0:19my co -chair Amanda Harris. And I'm the founder
- 0:21of hoagelaw .ai, where I serve as fractional
- 0:24general counsel to AI first SaaS companies and
- 0:27tech startups generally while spending my nights
- 0:29and weekends. vibe coding, all sorts of apps
- 0:32and websites, because of course I do. I am delighted
- 0:36to introduce our speaker today, Professor Chris
- 0:39Chambers Goodman. Professor Goodman is a professor
- 0:42at Pepperdine University School of Law, where
- 0:45she teaches constitutional law, evidence, and
- 0:48comparative anti -discrimination law. She writes
- 0:51and speaks frequently on equal protection topics,
- 0:54including implicit bias, algorithmic bias, affirmative
- 0:57action, and diversity. She's the author of California
- 1:01Evidence in Aspen's Examples and Explanations
- 1:05series and co -author of the Mendez -Goodman
- 1:08and Mineiro Evidence textbook. So with that,
- 1:12let's get started. Professor Goodman, please
- 1:14take it away. Thank you so much Matt. I am so
- 1:18excited to be here with you all today and looking
- 1:20forward to talking to you a little bit about
- 1:23some of the research that I've done on gender
- 1:26bias and adding in some of my some new research
- 1:30on gender bias. under the age of artificial intelligence.
- 1:35And so what I want to do is I have an agenda
- 1:39on the next slide. And that's what I'm going
- 1:42to talk through. So I'm going to give you a quick
- 1:44introduction. And then we'll talk a little bit
- 1:47about some of the historical background and the
- 1:50statistical context that impacts women in the
- 1:55legal profession. After that, I'm going to give
- 1:59you a bit of a primer on sources of of bias data
- 2:03in machine learning processes. So you can understand
- 2:06a little bit about how these processes may exacerbate
- 2:11and enhance existing biases that people have.
- 2:16We'll talk a little bit about some specific manifestations
- 2:19of gender bias in the law and in legal practice.
- 2:23And then in the final segment, we'll talk about
- 2:26some strategies for reducing bias in real life
- 2:30as well. as well as online. And then finally,
- 2:34I'll try to leave about five or 10 minutes to
- 2:37get to the Q &A feature questions that you may
- 2:42have. And so in terms of just an introduction,
- 2:49We've been talking about women in the legal profession
- 2:51for decades, right? And we've seen steady advances,
- 2:55but yet we still see a lot of attrition. And
- 2:59there are many reasons for that attrition, but
- 3:03one reason that comes to mind and is part of
- 3:06the focus of today's conversation is the persistence
- 3:10of gender bias in the legal profession. And so
- 3:14I want to take a look back and let's take a look
- 3:16at some of the historical background. and some
- 3:19of the statistics that many groups, including
- 3:23the American Bar Association, have gathered about
- 3:26the progress of women in the legal profession.
- 3:29So if we go on to the next slide, you will see
- 3:33that we have numbers over time. We're gonna go
- 3:39back to, on some of the slides, 1980, 1991, 2013,
- 3:46and take a look. at the way that gender representation
- 3:52in the legal profession has progressed. So here
- 3:55you see JD degrees by gender, and you see a decline
- 4:00in the blue line, which is male degrees, and
- 4:05a substantial increase in the orange line on
- 4:08this slide of JD degrees being offered and being
- 4:14earned by female lawyers. And this is a 10 -year
- 4:16period. from 2013 to 2024. Now this of course
- 4:20tracks what we see in college admissions as well
- 4:24with an increasing number of women getting college
- 4:27degrees and a decreasing percentage of men getting
- 4:31their college degrees. So when you look at the
- 4:35JD degrees by gender, we see that that we have
- 4:39more women graduating with JD degrees than we
- 4:43do men, which should bode well for perhaps achieving
- 4:47parity in the legal profession along gender lines.
- 4:51When you go to the next stage, female law firm
- 4:56associates. And I recognize not every female
- 4:59goes to a law firm. We have other statistics
- 5:01on government representation and those who go
- 5:06in -house and those sorts of things. But this
- 5:08is one of the easier ones to gather. In terms
- 5:12of female percentages of law firm associates,
- 5:16back in 1991, way back in the days when some
- 5:20of us might have graduated from law school, we
- 5:23were below 40%. 38 percent. Obviously before
- 5:27then the numbers were even lower. In the 1980s
- 5:32it was below 30%. And so now, as of 2023, that
- 5:38number of female law firm associates has increased
- 5:41to over 50%. And that number continues to climb
- 5:46in the few years since these statistics have
- 5:50been gathered. Now, what we see though, this
- 5:54more than 50 % of law students and law graduates
- 5:57are women, and a slightly more than 50 % of law
- 6:02firm associates are women, but when we get to
- 6:05the next slide, we see that once we get to the
- 6:08partnership track, partners who identify as female
- 6:13are below 30%. And again, of course, the limits
- 6:18of statistics, we have to recognize, but based
- 6:22on the firms that we're serving, and again, these
- 6:24are statistics gathered by the American Bar Association,
- 6:27so they're nationwide statistics. They may differ
- 6:30in some in some states, but we're still at under
- 6:3430%, 27 .8 % law firm partners. We could further
- 6:40break this down into equity versus of counsel,
- 6:44equity and non -equity partners. And again, we
- 6:47see at the lower ranks, women are still well
- 6:52represented. And as we go through the higher
- 6:56ranks, women are less represented. We see a similar
- 7:03increase in government lawyers on the next slide.
- 7:07We have our orange line, which you see, it's
- 7:11just surpassed right around the time of COVID,
- 7:15it looks like, 2020. The number of female government
- 7:22lawyers surpass the number of male government
- 7:25lawyers, and it's by about 1 ,500, 1 ,300 as
- 7:30of 2023 when our last statistics were available.
- 7:34So we are continuing to see these types of increases,
- 7:41which is good news, right? If we continue on
- 7:44to the next slide, we're also looking at general
- 7:46counsel. Those who are general counsel, this
- 7:50chart is limited to Fortune 100 companies. They
- 7:53were at 16 % female in 2004. almost less than
- 8:0020 years later at 34%. So those, again, the numbers
- 8:05are trending in good directions. Similarly, with
- 8:11judges, we've seen a sharp increase in female
- 8:15federal judges, going from virtually none until,
- 8:20you know, 1980s, right, when larger numbers of
- 8:24women started to graduate from law school and
- 8:27have their time. served to be appointed. And
- 8:31now we're up quite high, 484 female federal judges
- 8:38as of 2024. And again, more have been appointed
- 8:41in the last year and a half as well. So that
- 8:44number continues to climb. In the state court
- 8:51system, we have some statistics on the state
- 8:54high court judges. We can also get them on the
- 8:59appellate courts, but didn't want to give you
- 9:01sort of too many slides in the very beginning
- 9:03of of just statistics. But we see women are at
- 9:06forty three percent of high court judges. So
- 9:10the state Supreme Court or whatever the state's
- 9:13highest court is is called. Some states, of course,
- 9:16have a larger percentage, some a smaller percentage.
- 9:19But we're again getting getting closer to parity.
- 9:25in that situation. So this is some of the good
- 9:29news, right? Steady increases, but we still have
- 9:33issues of, as we said, retention and advancement
- 9:39within the ranks. So now I want to put this in
- 9:44the context of Gen AI and artificial intelligence
- 9:49and machine learning processes, all of these
- 9:52things have become a greater part of our lives
- 9:56over the last five years -ish, or maybe for some
- 10:00of us just the last year. And we are beginning
- 10:04to hear information about biases that occur in
- 10:09machine learning processes. Now, we have to step
- 10:12back, of course, because many people think about
- 10:15machines as as objective, neutral. They don't
- 10:19have biases, right? algorithms can't favor one
- 10:25thing or the other. They do what they are programmed
- 10:28to do. And so just to push back a little bit
- 10:31on this, we have to recognize that these algorithms
- 10:35are created by humans. These machines are trained
- 10:39by humans. These processes are corrected by humans.
- 10:44And many of the biases that humans already have
- 10:49will also become a part of these processes and
- 10:54have become a part of these processes. All right.
- 10:58So I'm going to talk to you now a little bit
- 11:00about some of the sources of bias in the machine
- 11:03learning processes. And I'm going to go over
- 11:06six different sources. There may be more, but
- 11:09we've narrowed it to six for purposes of this
- 11:12presentation. So the first sort of bias in machine
- 11:16learning processes is simply history. History
- 11:18is what history is. The number of people who
- 11:23have been president of the United States is a
- 11:25fixed number. And the number of people who have
- 11:28been president of the United States who have
- 11:30been female is absolutely zero, right? And so
- 11:34if the machine learning system is trained on
- 11:37historical data who have been all the past presidents
- 11:40of the United States and give me a prediction
- 11:44about whether or not this candidate or this candidate
- 11:46would be the stronger for a future presidential
- 11:49run, well, of course, being male will be a very
- 11:54large part of that equation because of history.
- 11:58And so, while large language models pride themselves
- 12:02on the vast volumes of data that they're able
- 12:06to use and gather and work through and learn
- 12:11from, a lot of that data actually is going to
- 12:16perpetuate existing biases. Now, in part, women
- 12:21were not presidents for a lot of reasons, right?
- 12:25None have been elected. But for much of our nation's
- 12:28history, women weren't allowed to vote. And so
- 12:31women weren't allowed to vote. They weren't allowed
- 12:33to run for office either. But From a data perspective,
- 12:38just looking at the data and seeing no females
- 12:41does not give the machine the context, right?
- 12:46That women weren't allowed to vote, women weren't
- 12:48allowed to hold offices in many places. And so
- 12:51that would be an explanation for why the history
- 12:54may not be as useful of a tool in predicting
- 13:00future presidential candidates. So in terms of
- 13:06history, there are lots of historical gaps, historical
- 13:09gaps of women's recognition in the law, women's
- 13:14recognition as legal actors, as those who are
- 13:17able to testify, to own property, to enter into
- 13:22contracts, all those types of things. have a
- 13:25very different historical background for women
- 13:29than they do for men. And so to the extent that
- 13:33these systems are being trained on historical
- 13:35data that kind of treats men and women equally
- 13:39at a time when they were not treated equally,
- 13:44that absence will be noticeable. Now, similarly,
- 13:49unrelated to historical bias is representation
- 13:52bias. And there, again, we're looking at who
- 13:55is represented in the data sets. And to the extent,
- 14:00again, that large language models are gathering
- 14:03lots of data, they may say, oh, we have data
- 14:06from millions of data points, right? But if the
- 14:10same groups are overrepresented in those data
- 14:13points and other groups are underrepresented
- 14:15in those data points, they is it really representative?
- 14:19And any prediction based on that volume of data
- 14:25may be less accurate. And again, we'll have less
- 14:31predictive power. And so when we think about
- 14:37Representation, for instance, one aspect, one
- 14:41thing that you may be familiar with is facial
- 14:44recognition software, right? And facial recognition
- 14:47software, we've learned, can be highly accurate,
- 14:50but it's highly accurate for particular groups.
- 14:53For white men and for Chinese men, it is highly
- 14:57accurate because of the large volume of faces
- 15:00that have been used to train that data. For Black
- 15:04women, it's very for people with darker skin,
- 15:09for people with non -Caucasian or Asian features,
- 15:13it's really inaccurate. But if you lump in and
- 15:16say, you know, we tested this on, you know, people,
- 15:21populations throughout various cities in the
- 15:23United States, again, yes, it will be accurate
- 15:26for a segment of that population, but for another
- 15:29segment of that population, it will be entirely
- 15:33inaccurate. Now, in terms of measurement bias,
- 15:37again, we measure what we think counts, and therefore,
- 15:41because we've measured it, it ends up counting.
- 15:47characteristics, for instance, that are not considered
- 15:50by these large language models, by those who
- 15:53are designing the algorithms, and if they don't
- 15:56consider these characteristics, which may not
- 15:59be sort of on their face apparent in terms of,
- 16:04for instance, doing an employment application,
- 16:08right? Well, we all know that some aspect of
- 16:12having a good employee is not merely how they
- 16:16present orally, how they present in writing,
- 16:20but also if they have some sort of initiative,
- 16:24right? They will take the bull by the horn, so
- 16:26to speak. They will jump in and look for solutions
- 16:31instead of sitting back and waiting for solutions
- 16:35to drop into their laps. If that's not measured
- 16:40in your online employment application, someone's
- 16:44sort of stick -to -itiveness, someone's initiative,
- 16:47well, then those who have initiative will not
- 16:51make it to the next round of the interview process
- 16:55because that wasn't asked in the questions, all
- 16:58right? That wasn't part of the application process,
- 17:00and it wasn't tested. And so then what we get
- 17:04is, again, a gathering data but we're gathering
- 17:10data about particular data points, but to the
- 17:13extent we're ignoring other data points, then
- 17:16the results that we see when we plug in candidates
- 17:22or applicants into our algorithms and see what
- 17:25pops out, the results that we see are not going
- 17:29to be entirely bias -free. So those are three
- 17:35of the sources of bias data in machine learning
- 17:39processes. And now I'm going to talk to you about
- 17:42three additional sources of bias in machine learning
- 17:46processes. So aggregation is can be helpful,
- 17:56but it also can be a big problem, again, when
- 17:59you're dealing with large data sets. So we all
- 18:02know from statistical analysis that if you have
- 18:05too small of a sample, then any results you have
- 18:08will not be... Statistically significant, right?
- 18:11Say that three times fast. Statistically significant.
- 18:14And so what we sometimes do is we will lump groups
- 18:17together so that we have enough data from which
- 18:20to make a prediction, for instance, or to make
- 18:25a recommendation. But the problem is when we
- 18:28aggregate disparate groups, then again, we may
- 18:32be over -representing certain groups in our data
- 18:36set and under -representing others. We may be
- 18:40misrepresenting certain groups. For instance,
- 18:42we can say, the average woman is, I'm just going
- 18:46to say a number, is five foot four and a half,
- 18:50because that's how tall I am. So we say the average
- 18:52woman is five foot four and a half. That may
- 18:54be true, but the average woman from particular
- 18:59ethnic groups or racial groups will have very
- 19:02different heights and very different height averages
- 19:05depending on their race, depending on their location,
- 19:09country they're from, nourishment levels, socioeconomic
- 19:14status, all those types of facts. can make a
- 19:17difference in the height that someone grows to.
- 19:24And so when we aggregate the data and put all
- 19:27of the women together, while we have an accurate...
- 19:30we have an accurate number on the average overall,
- 19:34it really doesn't mean much in terms of the individuals.
- 19:38And so this aggregation of data can become dangerous
- 19:45and even pernicious when the data is basically
- 19:50devouring minority and underrepresented groups.
- 19:56And so the greatest part of the data will be
- 20:00the majority groups. And so then again, we're
- 20:02not making accurate predictions for people within
- 20:06those underrepresented groups because of this
- 20:09aggregation. All right. The way that data is
- 20:15evaluated and again how we determine what is
- 20:20valuable and what is less valuable is another
- 20:23place where biases can creep in. So taking again
- 20:27an example of algorithms used for hiring, so
- 20:33they'll be used for both recruiting applicants,
- 20:38for encouraging applicants to apply. to screen
- 20:42resumes and even to conduct digital interviews,
- 20:46screening interviews, before humans get involved
- 20:50at all. But one thing that we find with the evaluation
- 20:54that's being done by some of these algorithms,
- 20:59in the employment context is that they've been
- 21:05programmed, each has been programmed in a way
- 21:08to look for certain things, but some of the things
- 21:11they look for can actually result in discrimination
- 21:16against people based on a lot of different factors
- 21:21that aren't being taken into account. For instance,
- 21:24many of these programs give high scores for eye
- 21:28contact, making eye contact with the camera,
- 21:31with the screen during the whole time of the
- 21:34interview. But in some cultures, it's disrespectful
- 21:37to make eye contact with someone who's senior
- 21:40to you, who's at a higher social status than
- 21:43you. potential employer. That would be disrespectful.
- 21:47But in terms of evaluating that interview, what
- 21:51we see is, oh, they got a low score on eye contact,
- 21:55and therefore that's going to play a role. But
- 21:58without being able to examine the cultural factors
- 22:01that may explain the lack of eye contact or the
- 22:08lower amount of eye contact, we're going to be
- 22:11missing out on probably very qualified candidates.
- 22:17In addition, for many people on the autism spectrum,
- 22:22eye contact is also very, very difficult. And
- 22:26so to judge... them based on the lack of eye
- 22:30contact could violate anti -discrimination provisions
- 22:35under the Americans with Disabilities Act, the
- 22:38California Fair Employment and Housing Act, and
- 22:41UNRU Act, and other things like that. So that's
- 22:44another place where we want to think about how
- 22:48people are being evaluated, what are the metrics
- 22:51that are being used, and how were those metrics
- 22:54programmed into the algorithm. tools that we're
- 23:00using in order to make these decisions. And then
- 23:05a final source of biased data in machine learning
- 23:09processes is deployment bias. Excuse me. With
- 23:18deployment bias, there... We have algorithms
- 23:22that are created, machine learning processes
- 23:24that were created and designed to do a particular
- 23:28job. And when we deploy them out into the real
- 23:32world, out into our legal work, our offices,
- 23:36whatever, if we use them for something else,
- 23:40something that they were not designed to do,
- 23:43well, then we may end up with... worse results
- 23:46than if we hadn't used artificial intelligence
- 23:49at all. And one very, very prescient example
- 23:54of this is algorithms that were designed to provide
- 24:00sentencing recommendations and bail recommendations
- 24:04for judges. So judges get a lot of sentencing.
- 24:09They have a lot of bail issues that they need
- 24:12to consider. And some were concerned that judges
- 24:15may be biased in their sentencing decisions,
- 24:18biased based on race or ethnicity or many other
- 24:23factors, poverty, and the same with bail decisions.
- 24:27And so it was thought we can design some programs
- 24:29that can take all the information about the convicted
- 24:34defendant to make a sentencing recommendation
- 24:38or take all the information about the person
- 24:42who's accused to determine their future dangerousness
- 24:46or flight risk and provide a score that can help
- 24:50the judge to determine whether or not someone
- 24:54should be released on money bail in the states
- 24:58where there still is money bail. or to be held
- 25:02over as a danger or, you know, whatever. And
- 25:06so what happened was the judges would input the,
- 25:10you know, they'd have their clerks input the
- 25:11information, they'd get the recommendation, and
- 25:14then they'd think about the recommendation, and
- 25:16then they'd kind of tweak it. They'd do something
- 25:19with it, and then that would be what their sentencing
- 25:22recommendation was, or that would be what their
- 25:24bail recommendation was. And some studies were
- 25:27done after the fact to show that The algorithm
- 25:30acting alone was better than the algorithm plus
- 25:35the judge. The judge acting alone was better
- 25:38in terms of predicting future dangerousness,
- 25:41in terms of predicting who was a flight risk,
- 25:43who would come back for their trial or their
- 25:46hearing, than the two together. And that's because
- 25:50the algorithm wasn't designed to also factor
- 25:53in the human judge. Now, this was a few years
- 25:56ago. And obviously, AI is increasing, expanding,
- 26:01and getting better and better at so many things.
- 26:04So we'll have to take a look at this again in
- 26:07the future. But recognizing that, oftentimes,
- 26:11systems are created for one purpose and used
- 26:15for a different purpose. I'll give you one other
- 26:17example of this deployment bias. believe it was
- 26:22in Japan. There were algorithms that were used
- 26:28to help companies figure out which applicants
- 26:33had the greatest likelihood of accepting their
- 26:37offer of employment. And so these applications
- 26:41went out, I mean, these applicants only happened
- 26:44once a year. And so it was really, really important
- 26:48to the company to give offers to people that
- 26:51were more likely to accept, because if they made
- 26:54a bunch of offers to people who did not accept,
- 26:56there was no opportunity to make offers again
- 26:58for one year. And these were offers for recent
- 27:02university graduates. And so there was a program
- 27:06designed to determine which people are more likely
- 27:11to accept offers. And so employers were using
- 27:15that program to make their offers, to figure
- 27:18out which people were more likely to accept.
- 27:21But what happened is the company unbeknownst
- 27:24to the candidates, applicants, or employers at
- 27:27the time, retained the data from previous years
- 27:30about individual applicants. And then, so that
- 27:35when the next year or the next time an applicant
- 27:38applied, that data was also taken into account
- 27:41as to whether or not that individual had accepted
- 27:44offers or rejected offers in the past. And so
- 27:47that ended up being a deployment bias problem.
- 27:51It was not what the algorithm was designed to
- 27:55be used for, and it ended up being a source of
- 28:01shame in the company. had to issue many apologies.
- 28:05And so just keep in mind that when you are getting
- 28:09artificial intelligence technology systems from
- 28:13vendors, find out what is it exactly designed
- 28:16to do. Make sure that you are using it in the
- 28:19ways that it's designed to be used. Or if you
- 28:23need some other product or some other focus,
- 28:27then keep that in mind as well. And so those
- 28:30are some of the of biased data in machine learning
- 28:34processes that a lot of people seem to just not
- 28:38even think about. They think, oh, I plugged it
- 28:40in. It's searching the whole internet. I've got
- 28:44the whole world at my fingertips. And what it
- 28:47comes up with is a neutral, unbiased response.
- 28:50And that's just not the case. And so now I want
- 28:58to just step back and talk just a little bit
- 29:02about implicit bias and explicit bias and a little.
- 29:08tiny bit about the science of those two. And
- 29:12so I know you've all had implicit bias CLEs and
- 29:15we have this continuing legal education obligation.
- 29:21And so just as a reminder, implicit bias is that
- 29:25intuitive, reactive, immediate, that quick. automatic
- 29:29response that you have. It's not the one that
- 29:32you sit and think about, not the one that you
- 29:35deliberate about, not the one that you, you know,
- 29:38hem and haw over. It's just what is your instantaneous
- 29:42reaction? Many people will deny that they exhibit
- 29:52implicit biases, but that's part of the thing
- 29:54is it's not part of your consciousness, so you
- 29:57don't really know that you have an implicit bias,
- 30:01all right? So when we're talking about implicit
- 30:04bias in the gender context, we see that we have
- 30:09lots of stereotypes, generalizations, pictures
- 30:14that come into our mind when we're told, you
- 30:17know, we may be told someone is a female litigator
- 30:20or you have a female judge or something like
- 30:24that. For many people, there will just be, you
- 30:26know, thoughts that come into their minds whenever
- 30:30they hear a particular person's gender. There
- 30:34are certain associations that are made. So when
- 30:37we're talking about implicit bias, I better stay
- 30:40in the camera. When we're talking about implicit
- 30:42bias, I usually show a picture of a brain, the
- 30:46brain with all the nooks and crannies. And so
- 30:49I like to think about those nooks and crannies
- 30:53as the well -worn paths that are easy for our
- 30:58brains to take. So when we see one stimulus,
- 31:01we see one person who looks a particular way,
- 31:04there's a path in here that takes me to a spot.
- 31:08So you may see a police officer and you may have
- 31:12a path that goes to, oh good, everything's safe,
- 31:15there's a police officer present. Or you may
- 31:18have a path that goes to, oh, there's a police
- 31:21officer present. I better mine my P's and Q's
- 31:24and make sure I don't do anything wrong. And
- 31:26maybe I want to get out of here, right? There's
- 31:28these paths that are in our brains. We don't
- 31:31have to think about them. It's just our instant
- 31:33reaction, our uneasy feeling. Some call it that
- 31:38gut feeling, something like that. And so these
- 31:41well -worn paths are our implicit biases. It's
- 31:45very easy for me to go from seeing the police
- 31:49officer to feeling a little bit of fear, all
- 31:52right? That's not a step that's difficult to
- 31:55make. Now, if I wanted to see a police officer
- 31:58and greet him warmly and be genuinely excited
- 32:02to see him, that's going to require some deliberation.
- 32:05That's going to require something in my thought
- 32:07process that is conscious because my instant
- 32:11reaction is not that. Okay. And so when we are
- 32:17thinking about what are our instantaneous reactions,
- 32:24we know that those are based on generalizations,
- 32:28they're based on stereotypes, and they're based
- 32:30on our past experiences. Our past experiences
- 32:34are very present. Even if we don't remember all
- 32:37of the experiences and they're not at the top
- 32:40of mind, past experiences are imprinted on our
- 32:44brains and have an influence on how we behave
- 32:48in particular situations. And so we have scientists
- 32:56have figured out a way to kind of test implicit
- 33:00bias, to measure the levels of implicit bias
- 33:04that people have without sort of, you know, opening
- 33:07their brain up. And so the science of implicit
- 33:11bias, which was conducted by Project Implicit
- 33:15at Harvard many, many years ago, many of you
- 33:17may have taken these implicit attitude tests,
- 33:21they're called. But what they've done Sorry.
- 33:25What they've done is defined and identified lots
- 33:35of different mech and lots of different axes
- 33:38upon which people can be biased. So we have a
- 33:41couple here on this slide, associating women
- 33:44with home and career, or associating judges with
- 33:48males and females, right? And so what we find
- 33:52is when you're making a quick association, one
- 33:55that is one of those well -worn paths in your
- 33:59brain, you can do that very easily. When you
- 34:02have to make one of those associations that is
- 34:04not a well -worn path in your brain, it takes
- 34:07you longer. You have to stop and think about
- 34:11it, okay? And so the difference between how quickly
- 34:14you can associate women with home versus how
- 34:18quickly or slower you associate women with career,
- 34:23that is the implicit bias score that Project
- 34:27Implicit has come up with. And so the way that
- 34:32these tests work, for those of you who have not
- 34:34taken any, and I put the link there in the slide,
- 34:40is they'll flash pictures. you have a keyboard
- 34:44in front of you, and you click the left key for
- 34:48certain, you know, to make a good association,
- 34:52let's say, and the right key to make a negative
- 34:55association, all right? And then it will ask
- 34:57you to make associations between women and career.
- 35:02And so it'll show you a picture of a woman, it'll
- 35:05show you a picture of a career that we might
- 35:07expect a woman to have, or it'll show you a picture
- 35:11of a career that you didn't... expect women to
- 35:13have and how quickly you can make those identifications
- 35:15and sort these photos that you see, that's how
- 35:20they get a baseline for sort of where you're
- 35:24starting. And then they flip it and ask you to
- 35:26do the reverse association and associate what
- 35:29you used to associate with good, you now need
- 35:32to associate with bad. And so how quickly can
- 35:35your brain make that adjustment? And that difference
- 35:38is, as we said, is the score. And so one thing
- 35:42that we find from these tests and the meta studies
- 35:45that were done analyzing these tests is that
- 35:49implicit bias manifests more under conditions
- 35:52of stress, under conditions of duress, under
- 35:55time constraints. The quicker you have to do
- 35:58things, the more likely you're going to revert
- 36:00to those well -worn calves in your brain. And
- 36:03so you're going to make the decision that you
- 36:05already have a pathway for. are implicit biases
- 36:10when people take these tests, a lot of times
- 36:12they're very surprised by their results. They're
- 36:14surprised that a working woman, a professional
- 36:17woman who's worked all of her life actually does
- 36:20have a slight preference for associating women
- 36:22with home and men with career than vice versa.
- 36:27So that is something that, like I said, is surprising
- 36:32because these biases are implicit. In addition,
- 36:36what they found, and this is going to be useful
- 36:38for many lawyers and the judges in the room,
- 36:41is that people who profess to be objective, people
- 36:45who think that they are really fair, actually
- 36:49have higher incidence, higher levels of implicit
- 36:52bias when they take these tests. So thinking
- 36:56that you're fair, trying to be fair, expecting
- 36:59that you're fair is not going to be enough to
- 37:03guarantee that fairness. And that's why we all
- 37:07want to stop and take a breath and think about
- 37:11where we may be behaving in a biased manner,
- 37:15even though we're not intending to, even though
- 37:17we don't think we are, even though we're not
- 37:23expecting that. So, some of the critiques about
- 37:30implicit bias research are, first of all... A
- 37:34lot of us didn't grow up as digital natives.
- 37:37So anything that requires me to react quickly
- 37:39to a keyboard, I'm not going to be as fast at.
- 37:43And a response to that critique is that you're
- 37:46only tested against yourself. How quickly do
- 37:49you make the association that is stereotypical
- 37:52versus how quickly can you make the association
- 37:55that is counter stereotypical? All right. And
- 37:58so that is one critique is, you know, it's a
- 38:02physical processing thing more so than a brain
- 38:04processing thing. Another critique about the
- 38:08implicit bias research, of course, is the sample.
- 38:13It's a voluntary thing to take these implicit
- 38:17attitude tests. They are sometimes required in
- 38:21college and university classes. But that's a
- 38:24whole different demographic, right? College and
- 38:27university students do not represent the United
- 38:30States or the world in large measure. And so
- 38:34there's a sampling issue with, you know, it's
- 38:38people who actually are curious about whether
- 38:40or not they have implicit bias. And so maybe
- 38:42that is over -representing those who are predisposed
- 38:47towards implicit bias. So that is another critique.
- 38:51Another critique is that causation, as we know,
- 38:56is not the same as correlation. So the fact that
- 38:59I do score higher on an implicit bias test regarding
- 39:05people who are overweight, for instance, versus
- 39:09underweight or something like that, doesn't necessarily
- 39:12really mean that I'm going to treat people differently
- 39:14when I see them in real life if they are overweight,
- 39:18all right? And there's not a real good way for
- 39:20scientists to test that anyways. We ask people
- 39:23about their motivations. why they chose a particular
- 39:27candidate over another, those sorts of things.
- 39:29But if they don't tell us the reasons, we can't
- 39:33really get in their heads to know for sure what
- 39:36the basis is for people's conduct. Some additional
- 39:40studies were done and found about a 4 % causation.
- 39:44So that's still a small part of a decision, 4%,
- 39:48but it is something that was measurable. So those
- 39:52are some of the critiques. the physical processing
- 39:56issue, the sampling size and representation within
- 40:00the samples, and the idea of correlation and
- 40:04causation not being the same. And then a final
- 40:08critique that I often hear, but we have lots
- 40:11of responses to that one, is, well, if everyone
- 40:14has implicit bias, then doesn't it even out?
- 40:17Because I have implicit bias against people like
- 40:19you, you have implicit bias against people like
- 40:21her, and she has implicit bias against people
- 40:23like So in the long run, everyone's affected
- 40:28by implicit bias a little bit, and therefore
- 40:31we don't need to worry about it. But of course,
- 40:33the issue that we see from years and years of
- 40:37people taking these tests is that people who
- 40:40are in minority groups still have implicit bias
- 40:44against minority groups. In part, it's not just
- 40:48based on our experiences ourselves as a person
- 40:52of color or a person of a majority group, but
- 40:56it's also based on our exposure, our exposure
- 40:58to others, our exposure to media, our exposure
- 41:01now to social media. media. And so those exposures
- 41:06have an impact on people so that when taking
- 41:09a race IAT trying to associate darker skinned
- 41:14people or African American men's with with good
- 41:17and positive words and lighter skinned or white
- 41:21people with negative words that's difficult for
- 41:26people of all races, okay? And so to the extent
- 41:29that implicit bias is present in people of all
- 41:34races, ethnicities, genders, and all those types
- 41:36of things, still the benefit is going to inure
- 41:42to those who are represented in the majority
- 41:45group. So that's a response to that final critique.
- 41:53And now I just want to give you a couple of highlights
- 41:56on algorithmic bias in hiring. So I've done some
- 42:01research on the use of algorithms in the hiring
- 42:06context. There's here four points I have on this
- 42:10slide here that you want to be thinking about
- 42:12because I know some larger firms and organizations
- 42:16are using these AI tools for these employment
- 42:22AI tools and some of your clients may be using
- 42:25them as well. And so the first step is this discrete
- 42:32elimination, right? There's no record. When I
- 42:35apply for a job online and an algorithm decides
- 42:38I don't even get to move to the next stage, that's
- 42:42it, right? Maybe I get a rejection letter, but
- 42:44that's it. So there's no... audit trail. There's
- 42:48no information about sort of, what did I do wrong?
- 42:51What information do I, should I have included?
- 42:54That sort of thing. And it happens behind the
- 42:57scenes. So you all probably have heard the story
- 43:01of one of the big tech giants who decided to
- 43:04use AI to try to higher and wanted to increase
- 43:10their representation of women in their organization.
- 43:15But what ended up happening is the resume tool
- 43:20that they developed was rejecting any resume
- 43:23that said something about women, about playing
- 43:25on a women's sports team, about women's issues.
- 43:32The algorithm was not designed to do that. They
- 43:35didn't say and reject women, but what they trained
- 43:38the algorithm on was the resumes of all of the
- 43:43highest ranking men in the company. They were
- 43:45trying to emulate the people who were in charge
- 43:49and they happened to mostly be men. And so as
- 43:53a result, women's resumes were just booted out,
- 43:56booted out, booted out. That's still happening
- 44:00despite years of people being aware of this sort
- 44:04of problem with resume screening. And a company
- 44:09was recently sued. for screening out women over
- 44:1460 and men over 65. Again, without asking questions
- 44:20about age, without being programmed to violate
- 44:24the Age Discrimination in Employment Act or anything
- 44:27like that, it had decided based on the pool of
- 44:32people in the job that people over this age weren't
- 44:36good. That elimination can happen. With so many
- 44:44companies being mailed heavy, at least at the
- 44:46leadership ranks, or for so many jobs being dominated
- 44:50by men, this discrete elimination can occur without...
- 44:55without any programming, right? And so it really
- 44:59requires an effort to debug the program, to retrain
- 45:04the program, to make sure that these kinds of
- 45:09eliminations aren't made. And so even if the
- 45:14AI technologies are instructed, do not consider
- 45:17race or gender or whatever. There are proxies
- 45:21for these characteristics that can be observed.
- 45:24from the data. And it may be things that humans
- 45:27wouldn't necessarily pay attention to or that
- 45:30humans may not even recognize, but the AI technologies
- 45:36can. And so they end up basing decisions on protected
- 45:41categories inappropriately, but again, without
- 45:46anyone really knowing that that's the basis for
- 45:50the decision. Now, we've also got intellectual
- 45:53property and trade secret issues. So that could
- 45:56be a whole other talk. But one thing that I found
- 46:00in the research on employment algorithms is that
- 46:06some people end up getting permanently excluded
- 46:09from job searches because they've been rejected.
- 46:13by the system, and then they apply again a year
- 46:17later to a different job. But if they've used
- 46:20the same system or have access to the same data,
- 46:23then that person now becomes the person who was
- 46:26rejected by this other company two years ago.
- 46:29And then they get rejected again, and now they're
- 46:32the person that has been rejected four times.
- 46:34That's sort of how they're seen in the data representation.
- 46:39And so it ends up that some people are never
- 46:41going to get jobs based if the decision -making,
- 46:44or at least the initial decision -making, is
- 46:47determined by AI technologies. So that risk of
- 46:53permanent algorithmic exclusion is really a great
- 46:57one, especially now when so many people are using
- 47:00AI to not only create their resumes, but also
- 47:03to send it out to all these places. And so that
- 47:06saturation of applications can actually come
- 47:10back to bite the person. And so in my article
- 47:14about algorithmic bias and accountability for
- 47:18job applicants, one thing that I recommend is
- 47:22having clear systems. for deleting data after
- 47:26a certain point. And data retention policies
- 47:30should be as narrow as possible in order to avoid
- 47:35this kind of permanent algorithmic exclusion.
- 47:38So that is just like a quick highlight on some
- 47:44of the issues that specifically arise in the
- 47:47hiring context. And now in our last few minutes,
- 47:55we'll talk a little bit about some of the specific
- 47:59manifestations of gender bias. And I want you
- 48:02to recognize that when we think about gender
- 48:04bias in the law, we're thinking about it from
- 48:07many, many, many different sources. Usually,
- 48:11back when I started, we kind of looked at it
- 48:14from the senior partners or managing partners,
- 48:19that sort of thing, judges, opposing counsel.
- 48:23But it also happens from peers, supervisors.
- 48:28So when you're thinking about manifestations,
- 48:30think about all the different... places that
- 48:35bias can sort of bombard people. And so one very
- 48:41common, and I'm sure there are lots of people
- 48:43in this room that have experienced this, is the
- 48:46misidentification bias. And so I used to get
- 48:50this all the time, and I still do. So I walk
- 48:54into the courthouse, and they would say, I'm
- 48:58sorry, that's council table. And I say, yes,
- 49:02I'm counsel. But why did they say that's counsel
- 49:07table? Because they thought I wasn't, because
- 49:09I was young, because I was black, because I was
- 49:10female, who knows, right? But for some reason,
- 49:13even though I was wearing a suit like everyone
- 49:15else, had a briefcase, business cards, you know,
- 49:18so that misidentification and the percentage
- 49:20of women, women lawyers who have been misidentified
- 49:23as secretaries, paralegals, other staff who have
- 49:28not been identified as lawyers or judges or,
- 49:31you know, partners is really astounding. It's
- 49:35something that continues to happen. And people
- 49:37say, well, oh, you know, I didn't mean that.
- 49:39I just assumed, right, I just assumed is part
- 49:43of the problem. Why do you assume that the woman
- 49:46is not an attorney and you assume that the man
- 49:49is? even when he's not. Alright, so think about
- 49:52these times when you misidentify people. It happens
- 49:55in the office, it happens in the hallway, it
- 49:58happens in the courtroom, it happens in in the
- 50:01classroom, right? Another Example is misimpressions.
- 50:09We form impressions of people very, very quickly
- 50:12in a lot of situations, and those impressions
- 50:16are hard to shake. They become a basis for things
- 50:19like confirmation bias and attribution bias.
- 50:24And so when we are putting people into categories
- 50:29and forming impressions about them based upon
- 50:32other people we know in that category, then we're
- 50:36creating misimpressions or potentially creating
- 50:40misimpressions. So much of this is basically
- 50:43when we are making assumptions about people before
- 50:47we actually give them a chance to prove themselves.
- 50:52That is something, again, to keep in the back
- 50:54of your mind when someone walks in late and you,
- 50:57oh, I guess you had trouble with your kids or
- 51:00I guess, you know, why would you say that about
- 51:03a female, but not about a male attorney? And,
- 51:06you know, hopefully you wouldn't say it out loud
- 51:07in any event. Another thing that a lot of women,
- 51:12lawyers and business people are concerned about
- 51:17is missing invitations. Colleagues will invite
- 51:21other colleagues out for drinks, other male colleagues
- 51:24out for lunch, out for golf, out for outings,
- 51:27events. And what I hear from some males who feel
- 51:32uncomfortable inviting females out because they're
- 51:34worried about sexual harassment. They're worried
- 51:37about discrimination issues. And so they figure
- 51:40it's easier if I just go out with the guys. But
- 51:43unfortunately, so much of is relationships, so
- 51:47much of business development is having those
- 51:49relationships and the confidence of your partners
- 51:52and your peers and those around you. So those
- 51:55missing invitations end up being a manifestation
- 51:58of gender bias as well. And then this one I call
- 52:03Ms .ing, and that's... this idea that so many
- 52:08people have that, you know, if we're going to
- 52:11be having a meeting, let's get Chris to take
- 52:13notes in the meeting. She's a very thorough note
- 52:16taker. All right. Well, if Chris is female and
- 52:19Chris is always the one taking notes, well, then
- 52:22that's a Ms .ing of Chris. You're giving me the
- 52:24housekeeping tasks. And, you know, Chris can
- 52:27order the food. Chris can get the coffee. Oh,
- 52:30Chris makes good biscotti. Let's have Chris do
- 52:32the snacks for the meeting. Right. So when women
- 52:37are often given housekeeping tasks like scheduling,
- 52:41being in charge of the minutes, of taking notes,
- 52:45those sorts of things, again, we just want to
- 52:47be conscious of that. Yes, women can do those
- 52:50tasks, but they shouldn't be the only people
- 52:52doing those tasks. And yet these are things we
- 52:54see in the office, in chambers, in business areas
- 53:02as well. And so on the next slide, I'm going
- 53:07to give you just a couple of examples of gender
- 53:12bias. And so we've already talked about confirmation
- 53:16bias, expectation bias. Some people think of
- 53:19women as either Mary Eve or Superwoman, right?
- 53:22So they're either the Virgin Mary, can do no
- 53:24wrong, chaste, or they're Eve, the temptress,
- 53:27or they're Superwoman and they can do everything.
- 53:30They can juggle their work in their home and
- 53:33work on a brief overnight while they are flying
- 53:36to their child's graduation, all those sorts
- 53:39of things. And so again, some people tend to
- 53:43characterize women in one of these three categories,
- 53:47which again will lead to the misimpressions that
- 53:50I was talking about. When you use the word...
- 53:53aggressive versus assertive and we often see
- 53:57this men are deemed to be assertive when they
- 54:00are firm right and raise their voice and women
- 54:03are deemed to be aggressive if they raise their
- 54:05voice or try to be firm right. It's a little
- 54:10bit different in terms of the language, but it
- 54:12makes a big difference in terms of people's impressions
- 54:15and their abilities to rise up, to move forward.
- 54:20Similarly, when emotions are at play, if a man
- 54:24is emotional when delivering a closing argument
- 54:27or in a motion hearing or something like that,
- 54:31many people observe that as righteous indignation.
- 54:35If he's emotional, he must feel really wronged,
- 54:39or this must be a very horrible situation. It's
- 54:43not about him not being able to control his emotions.
- 54:46Whereas with a woman, she's considered overly
- 54:49emotional if she demonstrates the same level
- 54:52of what we otherwise would call righteous indignation.
- 54:56She may be deemed hysterical, but we don't use
- 55:00those terms with men. So be careful in these
- 55:03descriptions when emotions are involved. In terms
- 55:07of assertions versus tag questions, we see from
- 55:11lots of observations and studies that there are
- 55:15a number of women who raise their voice at the
- 55:21end of a sentence. suggests a lack of certainty,
- 55:27like they're asking a question, but they're not
- 55:29really asking a question. And so that also is
- 55:32a place where when people hear that, they kind
- 55:35of react as lack of confidence, not sure, you
- 55:39know, not ready for leadership type things. But
- 55:43What you want to also consider is there are different
- 55:47speaking styles, there are different cultural
- 55:49speaking styles, and many younger people, men
- 55:55or women, also speak in that way. All right.
- 55:59Now, again, do looks matter? Yes. And we found
- 56:01that looks matter more in terms of a professional
- 56:04context and how women are treated than how men
- 56:07are treated. And so, again, there's lots more
- 56:10to be said about that. And then in terms of voice
- 56:13modulation, voice tone versus voice pace, women
- 56:18with very high -pitched voices aren't taken seriously
- 56:21by a lot of people. It doesn't mean they know
- 56:25less than people who speak in a lower tone, but
- 56:30sometimes that's just an instant place where
- 56:33people are dismissive. And so I'm going to give
- 56:37you just a couple of strategies for addressing
- 56:41bias in the legal profession. And it's pretty
- 56:45simple. It's A, B, C as easy. No, it's not as
- 56:48easy as one, two, three. But the first one is
- 56:50awareness. You want to be aware of the possibility
- 56:53that you may be acting in a biased manner, that
- 56:57you may be responding, deciding, making an impression,
- 57:05developing an impression about someone that may
- 57:08be based on some biases that you have. So awareness
- 57:12is the first step. The second step is to think
- 57:15about some bias interrupters. So I'm going to
- 57:18give you those on the slide. And then I have
- 57:21just a few communication strategies or things
- 57:24to think about. So these will be sort of checklists
- 57:27for you to look at and try to incorporate as
- 57:33you can through your daily lives. So on the next
- 57:36slide, we have A for awareness. Awareness makes
- 57:39a difference. So if you just be conscious of
- 57:41the fact that sometimes you act on assumptions,
- 57:44sometimes you jump to conclusions about people,
- 57:46and then you'll start to see, which people do
- 57:49I jump to conclusions about? And you can see
- 57:51if there's some gender or age or race or ethnic
- 57:56or class or some other issue that really seems
- 58:00to be the trigger for for this. When I say gender
- 58:04perks, one of the things that people sometimes
- 58:08comment about is some women using their looks
- 58:13or their charm to get benefits, right? And then
- 58:17that makes it harder for other women. And so
- 58:20that is something to be conscious of as well.
- 58:24But again, if you take the time to stop and think,
- 58:27to be aware of the possibility of bias, then
- 58:30that will help you avoid defaulting to bias.
- 58:34On the next slide, bias interrupters, you want
- 58:38to try to stop it when you see bias being exhibited.
- 58:46You want to try to stop it. And so if you're
- 58:49in a position to say, hey, wait a minute. I don't
- 58:52think we should have Krispy taking notes in the
- 58:53meeting. I mean, why can't we have John take
- 58:56notes in the meeting or something like that?
- 58:57Just small things like this can make a difference.
- 59:03So on the next slide on bias -interruptors, another
- 59:06thing to do is to enhance your exposure to people
- 59:09that do not comport with any stereotype that
- 59:14may exist. All right. So with this, we're thinking
- 59:19about, you know, if If one of the stereotypes
- 59:22is about women who leave early because of their
- 59:24kids, then what about women who have kids who
- 59:29work very hard, right? Activating these types
- 59:32of stereotypes or counter stereotypes in your
- 59:35mind will also make a difference. Notice your
- 59:39messaging. How do you talk to people? And what
- 59:44words do you use? We talked about aggressive
- 59:46versus assertive. And so in another bias in...
- 59:50is simply to stop and ask a question. Why do
- 59:53you say that? Or what do you mean by that? No
- 59:57challenge necessary, but you can sometimes make
- 1:00:02a difference there. And now on the next slide,
- 1:00:06we have communication strategies. So you want
- 1:00:09to think about when you're going to be communicating,
- 1:00:12whether this does involve a potential bias or
- 1:00:15other equity issue, and then think about when.
- 1:00:18Is this something that needs to be addressed
- 1:00:20in the moment? Or is it something where we need
- 1:00:23to take break, we need to cool down, and I'm
- 1:00:25going to have a conversation at a calmer level
- 1:00:28after the fact. So those are some of the decisions
- 1:00:31that you want to think about. You also want to
- 1:00:35think about where to have this conversation.
- 1:00:36Is this a private conversation? I tend to recommend
- 1:00:40private conversations to pull someone aside so
- 1:00:42they don't feel defensive. But sometimes if you're
- 1:00:45in a group or particularly if you're in a leadership
- 1:00:48position and there's a public statement that
- 1:00:51can be very harmful, to a member of your group,
- 1:00:55then you may need to make a public statement.
- 1:00:59And then when you are making statements, when
- 1:01:01you are having these conversations, just make
- 1:01:04sure you use I and we statements rather than
- 1:01:08the accusatory or finger pointing you. And so
- 1:01:12those are just a few quick strategies. Awareness,
- 1:01:16making use of bias interrupters, and effective
- 1:01:20communication. in order to address some of the
- 1:01:25gender bias issues that we see in the legal profession.
- 1:01:29And now we are right at time, so I will stay
- 1:01:33on to answer any questions that we have in the
- 1:01:38Q &A. And thank you so much for your attention
- 1:01:40today, and I wish you a good evening. All righty,
- 1:01:45so it looks like we have one question from a
- 1:01:47landwin. We are all human and make mistakes.
- 1:01:49What is a good way to recover from making an
- 1:01:52incorrect assumption or identification about
- 1:01:55someone? Yeah, that's a really great question.
- 1:01:58So we all do make mistakes. I mean, we may call
- 1:02:01someone the wrong name. We may attribute something
- 1:02:04to them that wasn't appropriate. I think the
- 1:02:09best way to recover from those mistakes is to
- 1:02:13admit them, to admit them both to yourself and
- 1:02:16forgive yourself, but also to the person to whom
- 1:02:20you made that mistake. So when I call a student
- 1:02:24by, you know, by the wrong name, and I am putting
- 1:02:27that student in a category with other students
- 1:02:30who maybe look similar, but obviously is not
- 1:02:33the same person, I'm going to go later. You know,
- 1:02:36I can apologize in a moment. Oh, I'm sorry, I
- 1:02:38meant Steve instead of Stefan or whatever. But
- 1:02:40then later I'm going to pull them aside and say,
- 1:02:43you know what, I really apologize. for that misspeaking,
- 1:02:47I've been on the receiving end of that type of
- 1:02:51confusion, people thinking I'm someone else,
- 1:02:53and I know how hurtful it can be, so I just wanted
- 1:02:55to let you know that I apologize. Now, if you
- 1:02:58do that in front of the whole group, that could
- 1:03:00be even more embarrassing for them. So that's
- 1:03:02why I try to do a quick, oh, sorry, and then
- 1:03:05later say something if need be. And then the
- 1:03:10other thing is when you do apologize, when you
- 1:03:14do forgive, yourself, you also want to think
- 1:03:16about, how am I going to make sure that I don't
- 1:03:18do this again? Because that's really part of
- 1:03:21it. If you just keep calling the person the wrong
- 1:03:24name or whatever it is, you keep misattributing
- 1:03:29something to them, well, then your apology is
- 1:03:31really hollow and meaningless. So what steps
- 1:03:34can I take to make sure that I remember this
- 1:03:37is how this name is pronounced or this is where
- 1:03:40you're from, not there. And so that's what I
- 1:03:44would suggest. So own up, apologize, pull them
- 1:03:49aside and forgive yourself and try to do better
- 1:03:51next time. All right, Professor Chambers Goodman,
- 1:03:58it looks like that's it for questions today.
- 1:04:00Thank you so much for leading this program and
- 1:04:02thank you so much to our audience. You're very
- 1:04:06welcome. I really appreciate this opportunity
- 1:04:08and I hope everyone has a great day. Thank you.
- 1:04:12Bye bye. Bye.