Latest / Hope + Possibilities: Trust, Leadership + Strategic Foresight / Hilke Schellmann - The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted & Fired & Why We Need to Fight Back Now

Transcript

Hilke Schellmann - The Algorithm: How AI Decides Who Gets Hired, Monitored, Promoted & Fired & Why We Need to Fight Back Now · Hope + Possibilities: Trust, Leadership + Strategic Foresight

0:00
I'll see you next time.
0:24
Nola Simon: so much for joining me.
0:25
I'm Nola Simon.
0:26
I'm your host of the Hybrid Remote Center of Excellence and joining
0:29
me today is Hilke Schellman.
0:31
Did I pronounce your last name right?
0:32
Yes.
0:33
So it's
0:33
Hilke Schellmann: Hilke Schellman.
0:34
Hilke.
0:35
Yeah, it's a, German first name.
0:37
Very hard.
0:38
Sorry, my dad is actually German,
0:39
Nola Simon: but I don't actually speak
0:41
Hilke Schellmann: German.
0:41
The funny thing is Germans don't know the name either.
0:44
So they're often like Heike, Zilke, what?
0:48
So it is, it is an enigma for anyone I encounter.
0:51
Nola Simon: Okay that's fine.
0:52
I'm glad I asked.
0:53
So thank you.
0:54
She's the author of the algorithm, how AI decides who gets hired,
0:59
monitored, promoted, fired, and why we need to fight back now.
1:03
And so it was actually interesting how I came across your book, and I
1:06
want to tell the audience a little bit about how I became aware of your work.
1:10
And so it started, honestly, I started on my own noticing like AI
1:14
being a trend back when I started my podcast back in like December of 2021.
1:18
And I interviewed a guy from Eightfold AI and he told me all about the way
1:24
they were using it in hiring and firing.
1:27
But also he was telling me about how they can identify transferable skills.
1:31
And I found this very interesting.
1:33
They have contracts with government to really identify transferable skills that
1:37
existed in long term unemployed people.
1:40
With the goal of actually getting them back to work.
1:42
And I'm like, wow, that's fascinating.
1:44
I never considered that was a possibility.
1:47
And he came up with this example that, if you have somebody who has
1:52
went to university, they have a math degree, but they have always
1:54
worked in customer service, you could literally get them into data analysis.
2:00
because the skill sets are really the same, but data analysis is
2:04
really more future focused and also pays a lot more, right?
2:09
So people are sitting on skill sets that are extremely valuable.
2:14
They don't even realize what the value is of those skill sets.
2:17
And that really appealed to me because I have a degree in math and I spent
2:21
20 years working in customer service.
2:23
Now that's how it started.
2:26
So then I also was aware of a company called Plum who worked with Scotia
2:31
Bank to actually replace their resumes.
2:33
And so basically nobody submits a resume for Scotia Bank in Canada.
2:38
They use this.
2:39
AI profile called Plum.
2:41
Now, I ran a personal profile for myself, and it came up with all kinds
2:46
of interesting things, except one of the lines was, basically, you're better
2:49
doing work that doesn't really take any initiative, that it's repetitive,
2:54
and you need structure, and you need to be told what to do, basically.
2:58
And I'm like, Yeah, that's not me.
3:00
Thank you.
3:01
I'm working for myself.
3:02
And my podcast, I've got, I'm nearing a hundred episodes.
3:05
So it's yeah, nobody's been telling me to do my podcast.
3:09
Thank you very much.
3:10
So I wanted, but what I wanted to know and why I had approached the CEO of Plum
3:15
was what is, What does AI actually know about me that is pulling that result?
3:21
Because if I'm running my own profile and I notice an inaccuracy
3:26
in it, so what's causing it?
3:28
Where is it pulling from?
3:29
How can I influence that result so that, I'm going to draw something
3:33
that's more accurate, but also what are the repercussions of that?
3:38
inaccuracy, right?
3:39
And how do you get it removed?
3:41
How do you get it fixed?
3:42
And how do you do the due diligence on doing like on any of that information?
3:49
Because if I applied for a job and that came up and the job is about
3:53
innovation and, taking initiative and, every job that I'd be interested
3:58
in, and that's what's coming out of that profile, that's gonna shoot you
4:02
dead in the water right there, right?
4:04
Yeah.
4:05
And then I could not get an answer from the company.
4:06
She didn't initially respond to me because I'm like, I want to interview you.
4:10
And then she goes to me and she even actually walked by me on a stage.
4:14
I had told her that I would be there and she walked by me and wouldn't talk to me.
4:17
So it's okay.
4:18
So I, I've given up on, on that, but I was fascinated with reading your
4:22
book because you focused on both of those companies and I was like, Oh,
4:28
somebody else noticed this and put it together and put it into the book.
4:33
And that's where I knew I really wanted to interview but we started,
4:37
there's one more piece of the story.
4:39
I didn't, I'd introduced you to somebody else that I had as a guest on
4:42
my podcast, and that's Swetha Redney.
4:44
She's a career coach up in Sudbury.
4:48
And I had noticed Swetha trying to get.
4:51
Information and trying to get media to actually interview her about concerns
4:57
that she had about AI and immigrants.
5:00
If you're if English is not your 1st language, how does
5:02
that influence an AI interview?
5:05
These 1 way video interviews, do you have accommodations, right?
5:09
If you have a stutter, if you have a lisp, all of these things
5:12
factor into how you perform.
5:14
And personally, again, my own personal story going back, there's
5:18
something about my voice that automated systems do not like.
5:22
So Siri and Alexa hate me.
5:25
My kids used to razz me because they could get Alexa to talk and
5:29
Alexa wouldn't respond to me.
5:30
So they would harass me using Alexa.
5:33
I've turned Siri off because it's so reliably does not understand my voice
5:37
that it's more hassle than it's worth.
5:40
And one time I had a bad accident they actually sorry, I've
5:43
got music playing somewhere.
5:45
Can you just pause for a moment?
5:47
Yeah.
5:48
Hang on a second.
6:14
Oh my God.
6:14
It was an electric.
6:16
It started playing in my ear.
6:19
Now it heard you.
6:20
Finally.
6:21
Now it hears me.
6:22
This is why I hate it so much.
6:23
Because, literally, it was played like the theme song from the Mickey Mouse Club.
6:27
I'm like, did I say anything at all about the Mickey Mouse Club?
6:33
Hilke Schellmann: Wait till it starts again.
6:34
Nola Simon: Oh, God.
6:36
Apparently that's dangerous to talk about.
6:38
Yeah, so I used to have to, I was checking the status because
6:41
I was waiting for my license to be reinserted after the accident.
6:44
And I had to wait my eight year old up to say the letter S.
6:48
Because the system would not recognize me saying the letter S.
6:51
I literally have concerns about if I ever had to do one of those video interviews,
6:55
how would that actually affect my voice?
6:58
English is my first language.
7:00
I don't have a concern otherwise.
7:02
But there's something about how automated systems, Recognize my voice that I don't
7:08
trust that they're going to get it right.
7:10
So I have a lot of concerns.
7:11
Hilke Schellmann: So we could totally test that in different ways.
7:14
Yeah, exactly.
7:16
That's right.
7:17
We usually have people who have maybe a dialect or an accent, right?
7:21
Or have a speech disability that we mostly concerned about.
7:24
I've never had somebody who is a native English speaker who
7:28
has automatic systems that.
7:30
Don't want to listen to you.
7:32
I know
7:32
Nola Simon: it's so aggravating and so annoying.
7:34
And I actually I was working for a bank before my job was restructured and they
7:38
had an automated system as well, too.
7:40
And I had run tests for them demonstrating like how badly you
7:45
couldn't recognize me and they didn't know how to, Handle that right?
7:49
And again, I volunteered to be a subject for them so they could test it out.
7:53
But I got restructured before that project could go anywhere.
7:57
But yeah, no, I find that fascinating to me because again, if this can
8:00
happen to me, then how does it happen to, how does it affect people who
8:06
have bigger barriers than I face?
8:09
Yeah.
8:09
And that's where it's the only thing that I know how to do anything about is
8:13
to ask questions and to talk about it.
8:16
And so that's why I'm very grateful to have you on the podcast so we can
8:18
amplify this conversation because I do think that it's extremely important.
8:24
Because it's only the tip of the iceberg where we are right now.
8:27
Yeah.
8:27
So before we really get into it you're a professor at New York
8:30
University and you're In the journalism
8:33
Hilke Schellmann: department, teaching journalism.
8:35
I'm in, I'm a professor in the journalism department, teaching
8:38
journalism, what I love.
8:40
So I have the dream job.
8:41
I get to teach what I love to do.
8:43
I get to do journalism.
8:45
So I'm I feel very blessed with that job.
8:47
Nola Simon: Yeah.
8:48
And the AI really fell in your lap too, when you were actually
8:51
in a cab and this guy told you he had an interview with a robot.
8:56
And that started, that question really just started opening up the
9:00
whole Pandora's box for you, right?
9:02
Hilke Schellmann: Yeah, totally.
9:02
We always wonder like where do journalists get their ideas from and
9:06
this was literally like me in the back of a Lyft ride talking to the driver
9:10
and he was like, I've had a weird day that doesn't ever happen to me.
9:12
It's usually they say I'm fine.
9:14
How are you?
9:15
And he's yeah, I've had a weird day.
9:16
And I was like, Oh, yeah, what happened?
9:18
He's I was interviewed by a robot for a job.
9:20
And I was like, robot?
9:22
He's yeah, I got a call from a robot.
9:24
So probably like a pre recorded voice.
9:26
And he had applied for a baggage handler position at at a local airport.
9:31
And he was just really just almost speechless about the process, because
9:34
he was like, that has never happened so weird and I took note of that and
9:38
was like, Oh, I've never heard of that.
9:40
And then, I went to an AI conference a few weeks later, and somebody talked
9:43
about AI and algorithms being used to check people's calendars and absences.
9:47
And she also did mention, oh, they use it for hiring and.
9:50
She told me about, could the company hire of you, which is one
9:53
of the largest providers in this space and it just started this.
9:56
And when I started looking into it, I was like I didn't know how ubiquitous it is.
9:59
And I think talking to your point of feeling like, whoa,
10:02
like if, It's inaccurate for me.
10:04
What is it for people who have way less agency?
10:06
And I think that in introduction of algorithms and AI and hiring does shift
10:11
the power balance, it's always been more power with the employers, right?
10:15
Because make the final decision, of course, but now it feels like
10:18
that, you could like Think about what am I put in my application?
10:22
Who do I list as a reference?
10:23
So you were a little bit of the curator of your passport, maybe.
10:26
But now companies assess you with AI or without even without knowing, right?
10:32
Like you may be thinking you're doing a video interview, little did you
10:35
know that they use AI on you or you send in your application material.
10:38
And if you send it through like LinkedIn or any of the large job platforms,
10:42
they all use AI, no recruiter wants to have just a folder with 2000 applicants
10:47
Transcribed Their resumes, right?
10:48
There's all kinds of ranking in the way this ranking happens.
10:51
It's like really curious, right?
10:53
Like, how do we know that the person ranked at number one is better than
10:56
the person ranked at 100 and those are like feel interesting questions.
11:00
And I started looking into that.
11:02
And at the time, like a lot of the I was first used on people.
11:07
What the industry calls like high turnover high volume job.
11:11
So we hire a lot of people and you have a lot of turnover often for like
11:14
retail positions, fast food service.
11:18
Yeah, customer service call centers have a really high turnover too.
11:21
So we see, in one of the in one of the tests that we did the company
11:25
is actually for a call center.
11:26
So the video is like somebody, you have this irate person on the phone.
11:29
How are you going to calm them down with one of the tests, right?
11:32
So we see that.
11:33
And I would say that folks who are looking for those kinds of job have probably one
11:37
of the least power in the workplace and the least time and get don't make often
11:41
a living wage or not near living wage.
11:43
And they're being subjected to this first, whereas CEO is another like sort
11:48
of leadership positions very rarely get subjected to these kinds of tools.
11:52
We've seen it like climbed a little bit, I've seen it used for flight attendants,
11:56
for teachers now I've seen it being used for a lot of recent graduates because a
12:01
lot of people feel hiring managers feel like, Oh, there's so many people, they
12:05
all have great bachelor's degree, but they don't have a lot of work history.
12:09
So they all look alike.
12:10
So let's do a skills capability assessment because we don't know a lot about them,
12:14
so we see it used mostly for that.
12:16
And we see it, I think what happened with the dawn of job platforms and
12:20
now with generative AI companies are flooded by applications.
12:25
And I think that is, there's a real need there.
12:27
IBM said they get about 5 million applications.
12:30
Google get about 3 million applications per year.
12:33
Goldman Sachs said for their summer internship alone, they
12:35
got over 220, 000 applications.
12:38
There's not enough humans to go through all of them.
12:40
And we know that humans are biased too.
12:42
So a lot of companies turn to technology because they, they're
12:46
drowning under applications.
12:47
We now more than ever, they're drowning because, a lot of job applicants use
12:51
generative AI to generate a cover letter, to generate Resume and now we have
12:56
a I that can actually apply for you.
12:58
So you don't even have to do anything.
13:00
Nola Simon: The battle of who has the best a I
13:03
Hilke Schellmann: it is a I versus a I out there and, at one point, you might
13:06
have to ask okay, what's still real here?
13:08
What are we talking about?
13:09
What are the capabilities?
13:10
How do we actually hire the best people?
13:13
And I don't think we necessarily have an answer there
13:16
Nola Simon: well, and how many people are actually avoiding that whole
13:19
process and just working the network and, That then becomes problematic too,
13:24
because then it's like, who, and in terms of equity and, equality, that's,
13:31
Automatically problematic because, people tend to have closed networks, right?
13:38
Like they, they like people who are like them, right?
13:40
So if you're building a process, that's automatically going to be biased,
13:49
then is it's example of systems working as systems are designed
13:54
to work and that's really what the outcome is that everybody wants.
13:58
Yeah, we don't Diverse workforce.
14:01
Hilke Schellmann: Yeah.
14:01
That's a
14:02
Nola Simon: question that you have to ask, right?
14:03
Hilke Schellmann: Yeah.
14:04
And especially like when you build tools based on the
14:06
current workforce that you have.
14:08
Yeah.
14:08
Most companies are not diverse, right?
14:10
So the the problem is that you might replicate, right?
14:13
The historical inequities that you've built into your workforce for hiring
14:17
more men or more white people, right?
14:19
Over time, right?
14:20
And then the tools pick up.
14:22
Their facial expressions, the words that they use, their manners, their
14:26
way of behaving and game playing might just hire more white men
14:31
which I don't think is anyone's intention, but is a likely outcome.
14:36
Nola Simon: Because they the systems have access to your hierarchy
14:40
through performance reviews or anything that's written, right?
14:42
You have an example in the book about, this AI had learned that the name Thomas.
14:47
It was, yeah, it was getting more points.
14:48
My hypothesis is that this was built.
14:49
This is a resume screener.
14:49
And if the resume happened to include the name, Thomas, it was rated more, it was,
14:55
Hilke Schellmann: it was, yeah, it was getting more points.
14:57
That my hypothesis is that this was built.
15:00
This was a resume screener.
15:01
that it was probably built with a bunch of resumes of people who are currently
15:05
successful in the role, which often means the people who are doing the job now.
15:09
Nola Simon: Yeah.
15:09
Hilke Schellmann: So you give the tool like thousand resumes of the
15:12
people you have employed right now, or you have been recently employed
15:15
the last six months or so made it to the last round of interviews.
15:18
And you say those people are the successful people.
15:20
And then the tool does a statistical analysis.
15:22
And apparently in this Statistical analysis, the word Thomas came up and
15:27
became statistically significant, so then the tool gave people who had the
15:31
word Thomas on their resume more points.
15:33
So obviously, any human knows that Thomas, the word Thomas,
15:37
doesn't qualify you for anything.
15:39
The machine obviously doesn't have a conscience and doesn't have any
15:42
ethical ideas and I should also say, Humans are also problematic, right?
15:47
We know from like social science that like we send resumes with more
15:51
Caucasian sounding names versus African American sounding names, right?
15:54
There's a lot of human bias too, that we know that people get fewer
15:58
callbacks who have African American first sounding names, but like a machine
16:02
was supposed to be objective, right?
16:04
That's what.
16:05
These AI vendors sell that it like democratizes hiring.
16:08
It has no bias.
16:09
It is absolutely fair.
16:11
And that has not been the case.
16:12
And I was really surprised when I talked to industrial organizational
16:16
psychologists who said like all of the tools that you looked at had problems.
16:20
And other employment lawyer told me like every fourth tool he looked at
16:23
resume parser had problematic keywords.
16:26
He found in one of them, the word Africa and African
16:29
American were used as keywords.
16:31
Another tool.
16:32
The word baseball.
16:34
If you had the word baseball on your resume, you got more points.
16:36
If you had the word softball on your resume, you got fewer
16:38
points, which points, which is
16:40
Nola Simon: which is excellent reason that you don't want to put your
16:43
activities your after curriculars.
16:46
And I don't
16:47
Hilke Schellmann: know what happens for the people who
16:48
don't have any hobbies, right?
16:49
We don't know if they get penalized or not, right?
16:52
But that's the problem.
16:53
A tool will look at everything that's on a resume, doesn't know that maybe
16:58
Python, a programming language, is a more important skill for a software developer
17:02
job than hobbies, which really should be, not be part of the decision making
17:06
at all, because hobbies doesn't say more about like your socioeconomic background.
17:10
If you put snowboarding or skiing, probably means in some societies
17:15
that you have more money than others that you come from a privileged
17:17
background because you can afford it.
17:19
That means more that tells you more about your background than your actual
17:23
skills and capabilities to do the job.
17:25
Unless it's a skiing instructor, they probably need skiing skills.
17:28
But other than that it doesn't have any bearing on the job.
17:31
And we see this again and again, and in a lot of these tools and that does worry me.
17:37
And we have so little oversight where in this case is the vendors themselves
17:41
didn't find the problem, right?
17:42
It was like when a vendor talked and talked to an employer
17:46
and, about using the system.
17:48
And then some of the employers do their due diligence.
17:51
They bring in outside counsel, they start the system, and
17:54
then they find the problem.
17:55
And a lot of times companies use deep neural networks, which you don't
18:00
need to know what that means, but it means that you have training data.
18:03
to build the model, and you can look at the results, but you don't necessarily
18:06
know what happens inside the machine.
18:08
We can ask the machine what exactly happened, but most
18:11
companies don't do that.
18:12
So a lot of AI vendors don't know what is the machine inferring upon?
18:17
And what we see a lot of machine learning, there's been like a famous example
18:21
where I think somebody built a tool that was supposed to understand what's the
18:24
difference between huskies and wolves.
18:26
and fed a lot of pictures in the model.
18:28
And, the model miraculously learned what's a husky and a wolf because if
18:31
you send in a new photo it was like, oh, husky, yes, until it didn't work.
18:36
And then the the folks asked the tool what did you infer upon?
18:40
Was it the nose?
18:41
Was it like the fur of the husky or the wolf?
18:44
How did you know the difference?
18:45
And the tool highlighted the snow in the background of the huskies.
18:50
Nola Simon: Because it turns out
18:50
Hilke Schellmann: the Huskies had like snow in the
18:52
background, the Wolves did not.
18:53
So the tool that actually didn't know what's the difference
18:56
between Husky and Wolves, what?
18:57
The Wolves didn't have snow?
19:00
Or the Huskies didn't have snow?
19:02
I don't remember, actually.
19:03
I would have expected Wolves to have more snow.
19:05
One of, one of them didn't have snow.
19:09
Yeah.
19:09
And that was and that's what the tool learned.
19:11
And so that's what we call, and this is actually.
19:14
More prevalent than you think in these systems, like we think it
19:17
does X, but when we tested, we learned that it doesn't do anything.
19:21
So there's another example with The COVID cough, like during the pandemic, there
19:25
were a lot of companies that tried to build AI systems where you like cough into
19:29
the phone to your doctor, the tool will tell you, do you have COVID or not, right?
19:33
And that never worked out.
19:34
And what we saw in even like academic literature where people said, Oh, we
19:37
found the COVID cough when you tested it.
19:40
What it actually found was that it had the people who were, had the
19:44
COVID cough and most often in the ICU.
19:47
So you heard the beeping of the machines and the people who didn't
19:49
have COVID were in, whatever scenario where they were coughing.
19:54
So the tool had learned that COVID means machines beeping in the background.
19:58
So obviously had to learn anything about COVID.
20:00
So we see this again and again, it's a real problem.
20:03
And I'm.
20:05
Not too hopeful that obviously that the hiring space isn't spared
20:09
from this problem because I found this problem now many times.
20:12
And that's just, as you said, the tip of the iceberg, right?
20:14
I haven't looked at all the tools.
20:15
I don't have access to all the tools.
20:17
So we need a lot more scrutiny.
20:19
We need a lot more testing.
20:21
So I hope that people in talent acquisition or in hiring will hear this.
20:25
Oh, hear me speak like please test these tools and do these like
20:28
super cheaper tasks that I do.
20:30
If I can speak German to a tool and still get a six out of nine English proficiency
20:35
score that the tool probably doesn't work.
20:37
Do some kind of testing and to really understand what does this tool do?
20:42
And if it doesn't do what it's supposed to do, you really might want
20:45
to rethink using this for hiring.
20:48
It matters who gets a job.
20:49
It matters to job seekers.
20:51
I'm relying on making money to put food on the table to have an apartment.
20:54
I'm nervous before a job interview because it matters if I get the job.
20:58
I know for employers, often it feels ah, so many candidates.
21:01
We reject any, them anyways, but It doesn't matter.
21:04
And I'm sure it mattered to the hiring manager and the talent
21:06
acquisition manager at one point that they did get the job, right?
21:10
Nola Simon: And there's legal ramifications as well too.
21:13
So if you're using tools that have built in discrimination, like you
21:18
said, like they're learning things that aren't necessarily relevant.
21:21
An example would be like just university related to like
21:24
social economic class, right?
21:26
If you're filtering out your AI tools, filtering out,
21:28
Harvard or whatever university.
21:31
There's long time studies and proof that show that those aren't necessarily tied
21:36
to skills and ability, but those are definitely tied to socio economic class,
21:41
Hilke Schellmann: right?
21:41
Yeah.
21:42
Yeah, and I feel especially with the resume parser there was 1 example that it
21:45
had highlighted, had learned that Syria and Canada are, Predictions of success.
21:50
So everyone else was fascinated with that one since I'm
21:52
Nola Simon: from Canada.
21:55
Hilke Schellmann: Exactly.
21:55
I was like, look, the Canadians, like they're, moving forward at a faster clip.
21:59
But you know what this means for people from other places that
22:01
they would be potentially get discriminated against, which is
22:05
illegal in the United States, right?
22:06
Like they're illegal place.
22:08
Yeah.
22:08
I think what's a little bit problematic is like that the tools themselves obfuscate
22:13
some of the decision making, right?
22:14
Because we don't actually know that Canada and Syria were indicators of success,
22:19
which we shouldn't be using, because unless somebody digs deep into the system,
22:23
we just see the results of the ranking.
22:25
And they look very convincing.
22:28
Even when I did tests, I know that these tests are bogus that I was like,
22:32
okay, there's no science behind them.
22:35
But when you still see your score, you're like 78%, blah, blah, blah.
22:39
It's really hard to not see that score and feel like, oh, that means I'm this.
22:43
You still put meaning towards it, even though, it's actually meaningless.
22:48
And I think that's.
22:49
What's really problematic, right?
22:50
As an hiring manager, you get this list of 1000 people, and
22:54
you're going to call the first 20.
22:55
I'm not going to call 998 that person, even though I have no idea how this
23:01
banking was really put together, right?
23:02
I assume the machine, somebody did their due diligence, and these
23:06
machines work and pick the best people.
23:09
But I think it was interesting from one survey that Joe Fuller did at Harvard.
23:13
We know that he surveyed like 2200 or so people in leadership
23:18
when their company use an AI tool.
23:21
Almost 90 percent said, Oh, we know that they reject qualified candidates.
23:25
So we know that they don't actually fulfill that promise that they
23:28
find the most qualified candidate.
23:30
But we still use it because I think the tools make hiring so much more
23:33
efficient, so much easier, cut down the number of days that you're hiring.
23:38
And that is something that companies care about, which I think is
23:41
probably the wrong incentive.
23:42
Who cares if you found somebody in two days?
23:45
Are they the most qualified?
23:46
Can they actually do the job the best?
23:48
If it takes three weeks maybe that's okay.
23:50
That isn't necessarily the incentive that is being used,
23:54
Nola Simon: right?
23:55
Yeah, exactly.
23:56
That's right.
23:56
And it comes down to trust and accountability.
23:59
Really?
24:00
Are you walking the talk?
24:01
And are you being responsible for what it is?
24:03
You're doing and that seems to be lacking in companies
24:07
that are actually using these.
24:08
These AI systems.
24:10
And again, I don't know that's their intention, but it is.
24:13
I don't think it's
24:13
Hilke Schellmann: at all their intention.
24:14
Yeah, this is like a tale of like good intentions, maybe gone bad or
24:19
having unintended consequences, right?
24:21
I don't think actually anyone building these systems or using
24:25
these systems has any ill intent.
24:27
I think everyone.
24:28
Really does probably want to diversify the workforce, want to
24:31
give more people a chance but it's not really fully understanding how
24:34
are we actually using this system?
24:36
How do these systems make decisions?
24:38
Yeah, because they are literally black boxes, like that, what we always say,
24:42
and I think that's, really problematic.
24:45
And I think, a lot of companies come into this.
24:47
They want to have they want to save money.
24:49
They want to have more efficient hiring technology.
24:51
So they don't want to necessarily now hire someone to oversee these
24:55
systems and test them continuously, like they're easier and faster.
25:00
So they're not going to maybe apply like a lot of scrutiny to this.
25:04
They're just like, okay, it works.
25:05
It cuts down hiring.
25:06
We need to employ fewer people.
25:08
That's all I want.
25:09
And I
25:09
Nola Simon: mean, it's basically software as a service, right?
25:13
So like the vendors could be updating the software as they go, right?
25:17
So they really should keep on top of that.
25:20
Just from a compliance point of view, you would think that you'd want
25:22
to make sure that you're reviewing the terms and conditions every time
25:25
there's an update to the software.
25:28
I don't understand what the risk people are like, where's
25:30
the cybersecurity and the risk assessment that goes along with this.
25:35
Hilke Schellmann: Yeah.
25:35
And I'm sometimes wondering about this too, because, like we reported
25:39
very early on in 2018, after I met the Lyft driver and the next year I was
25:42
working for the Wall Street Journal.
25:43
And we did this like a 10 minute, as a video investigation, because we
25:46
felt like, oh, this is, it's about video interviews mostly and other
25:50
AI and hiring, but it felt like, oh, we want to show this in video.
25:53
And we looked at HireVue and HireVue at the time still use like facial expression
25:59
analysis Ation of our voice and analysis.
26:02
And it also transcribed the words that you used.
26:05
And, when I started digging deeper, at first I was like so surprised.
26:09
I was like, oh, I didn't know that facial expression and job interviews
26:11
are predictive of success in a job.
26:13
What Interesting way to measure things.
26:16
And then when I dug deeper and I talked to expert, they're
26:18
like, there's no science here.
26:20
I was like.
26:21
Wait, what?
26:21
There's no science here.
26:23
They're like, yeah, we don't have any science.
26:24
What facial expressions you have to have in a job interview that predicts how
26:28
good you should be at any given job.
26:29
And I was like, Oh but they're using it.
26:33
And and then I talked to folks, there is the effective computing companies
26:36
that, that, that put out the tools.
26:38
They also say we have the same six emotions.
26:40
If you're smiling, you're happy.
26:42
If you're frowning, you're angry.
26:43
And when I talked to psychologists, they said Oh yeah, that's 60 years ago.
26:47
We thought that, but we know now that like different cultural
26:51
backgrounds, different societies, we don't have the same emotions.
26:55
Like we are likely to have similar emotions, but not always.
26:59
And so this is not predictive at all either.
27:00
And I was like, Oh, I was like, this is really problematic, but large
27:05
fortune 500 companies were using the technology without questioning it.
27:10
From what I know.
27:11
So I was really surprised that I was like, I do wonder, like, where are those people?
27:16
Where are the compliance people?
27:17
I don't know if each into that.
27:20
hiring space or they feel like because there is like almost no way for
27:25
people to get access to these systems.
27:27
If the person who builds the system doesn't know what the tool infers
27:30
upon, like the people using it, the companies don't know it, job
27:33
seekers are never going to find out.
27:35
So maybe the risk is very low.
27:37
Nola Simon: Yeah.
27:37
There's no transparency.
27:38
Like even as a job seeker, it's only because of press releases and whatnot.
27:42
Like I knew that Scotiabank was using Plum, for example, and
27:45
that's why I was researching Plum.
27:47
But most of the time they don't identify who's providing the documentation.
27:53
the services, right?
27:54
Yeah,
27:55
Hilke Schellmann: I think after, like some sort of scandals, right?
27:57
Amazon tried to build an algorithmic tool that scans resumes, and they built
28:02
a tool that was downweighing women.
28:04
If you had the word woman or women on your resume, it would downweigh you.
28:09
So I think, They got a lot of backlash for that as I think since that companies
28:13
have also been very careful to publicize anything bad about these tools, right?
28:18
Like I have talked to, like after the book came out, I got lucky.
28:22
Before too, but like really after the book came out, I had talked to a lot
28:24
of CHROs, like chief human resource officers, chief people officers,
28:29
and Oh yeah, we use that tool.
28:31
We have the same questions and problems that you uncovered.
28:33
And then we like stopped using it.
28:35
And I was like that's great.
28:36
I'm glad to hear it.
28:37
Did you tell anyone?
28:39
Because Company X is still using it.
28:41
Could you publicly say that you found out these tools didn't work?
28:44
And they're like, No, we can't publicly acknowledge that because they're
28:47
afraid of class action lawsuits.
28:49
They're afraid of press that they use the tool that's flawed.
28:53
You can understand it.
28:54
But the problem is we don't really have much progress then, right?
28:57
No one calls out the bad actors, or not the bad actors, but like when tools don't
29:03
work, they don't call out the companies.
29:04
And, I think the vendors are unlikely to dig deep and really check out
29:09
their systems and change them.
29:11
They do want to market and sell, right?
29:13
And there's still companies that merely believe them that this works.
29:17
So yeah.
29:19
Nola Simon: So what's the solution?
29:21
Like you've been on record saying that, regulation is needed.
29:25
Government involvement is needed.
29:27
Hilke Schellmann: Yeah.
29:27
We just did a recent election in this country, so I'm not sure.
29:31
under a Trump administration, there will be more regulation, right?
29:34
In fact, we've seen a lot of deregulation under the last Trump administration.
29:38
So I'm not very hopeful.
29:39
We haven't seen some forward movement under the Biden administration.
29:43
There has been like AI not lost, but like hopeful, like we
29:48
should really look into this.
29:49
And there has been like progress but we really haven't seen an agency stepping in.
29:55
starting to approve these tools or test them or anything.
29:59
I think that is, there isn't a whole lot of political will, I think, to
30:02
do that, and maybe also not a lot of.
30:04
knowledge because it is tricky to, and I wouldn't recommend that government agency
30:09
do this like little testing that I do.
30:11
I sometimes do this thing where bring a question to like computer scientists and
30:15
sociologists and we do larger Studies.
30:19
Yeah.
30:19
That's how we found the problem and their hallucinations and whisper.
30:22
That's also how we found out that some AI tools who take your personality from your
30:26
LinkedIn or your Twitter feed don't work and that you have multiple personalities
30:32
and even the same AI tool has different things finding out about your personality.
30:36
, if it looks to you at your LinkedIn or Twitter, which is, against the
30:39
theory, like if you use personality, you would assume that it is a stable.
30:44
Thing, right?
30:44
Otherwise, if it changes every 5 minutes, then you really
30:47
shouldn't use it for hiring.
30:48
So we had also
30:49
Nola Simon: Social media is a filter to begin with, right?
30:52
Who you are on LinkedIn may be very different from who you present yourself
30:55
to be on Twitter or threads or whatever.
30:57
I absolutely agree.
30:58
You adopt personas, or at least some people do.
31:02
Hilke Schellmann: Yeah but I think the companies would say you
31:04
didn't write this to get a job.
31:06
Especially your tweets, so there's something about you
31:08
that you're not hiding, right?
31:09
This idea to look under the hood of a job seeker.
31:13
It is pretty prevalent.
31:14
I think the same idea with checking your facial expressions like you
31:17
get something up about yourself that you weren't even knowing about.
31:21
And we know from history of companies analyzing like handwriting
31:26
and other things and hiring that, there is no science there.
31:28
There is no science.
31:29
But we've seen this again and again, because it does feel.
31:34
So intuitive, right?
31:35
It does feel intuitive that something in your writing says something about you.
31:39
So I could analyze that and find the real you that you can't hide.
31:42
But so we fall into that again.
31:44
Those are those are assumptions that, that, that worry me in this space.
31:47
Let me see.
31:49
Yeah.
31:50
Nola Simon: But Europe isn't having the same sort of problem
31:52
because of the GDPR, right?
31:54
And the privacy?
31:57
Hilke Schellmann: Yes and no.
31:58
I do think that Europe is like a more regulated place on earth, right?
32:02
Like we see much more regulation coming out of the European Union.
32:06
And we see, they have, we call those like omnibus legislation
32:10
because there are large laws.
32:12
So one is GDPR, the general data protection privacy law even the
32:16
UK adopted, although it's no longer in the European union.
32:19
And I was like lucky to find who I call like patient zero Martin Birch who got
32:23
his who had to take a plum assessment, like a gamified personality test to,
32:29
to really basically describe it you.
32:31
Get some, I don't know, you do some Tetris thing games.
32:34
And you have to answer a couple of questions.
32:36
What would you in the workplace, if you encounter this or that and
32:39
they come up with the personality and capability analysis of you.
32:43
So he was he had applied for a job in data.
32:47
And he was like, I was data scraping all day at my current job.
32:50
So he was like, I was really surprised that, I wasn't, asked to do an assessment
32:54
on my capability of data analysis and data scraping, which I do in my job,
32:58
which I applied for at Bloomberg.
33:00
But I was asked to answer these questions about my personality and
33:03
puzzle Tetris and other things.
33:05
And he was also surprised that he pretty much got an instant
33:09
within a day, a rejection.
33:10
He was like, Huh.
33:12
For a job that I'm already doing that I applied for at this other company.
33:15
That's interesting.
33:16
So he was in a journalism adjacent job, right?
33:19
He was for a news organization.
33:21
He wasn't like a reporter, but he knew what reporters do.
33:25
So he thought about the GDPR.
33:26
He asked for his data.
33:28
And then he got the data and he scored really low and then he found
33:31
out that Bloomberg had automatically rejected him, which is under, some
33:38
other laws, is not legal, you have to disclose that if you use automatic
33:45
And Bloomberg didn't disclose that.
33:46
So he started, he started like a inquiry, a legal inquiry.
33:50
And he settled with the company, but, it's one of the first people that I ever
33:54
heard from that like actually had the data to look at and ask about it exactly.
33:59
And I think that happens so rarely.
34:01
And it doesn't happen very often in Europe, right?
34:03
Because a lot of Europeans don't even know that they have this
34:06
right or they know how to get it.
34:07
And then when you get the data, it's also really hard to really understand it.
34:11
Okay.
34:11
Capability or not, I'm, I don't know if it's right or not.
34:14
There's no authority to go to and say I don't think it's right.
34:17
Who would you call?
34:18
We see a little bit more and now we see the EU AI Act, which actually outlaws
34:22
outright the emotion recognition.
34:24
So that kind of tools would not, will no longer exist in
34:28
the European Union legally.
34:30
If you use it, it would be illegal.
34:32
And there's a high bar for hiring, which I'm really happy about because
34:35
I've been saying for a while now that I think, hiring is high risk decision
34:39
making, like it makes It is important to understand who gets hired and we're not.
34:43
So it should be up there with like other high risk things about
34:46
who gets a mortgage, who gets a loan, who goes to jail, like those
34:49
are or how long you go to jail.
34:51
Those are also high risk decision making and hiring should be one of them.
34:54
So there is a little bit more scrutiny that companies have to face.
34:59
We not exactly know how it's going to interpret it in every country and
35:02
every like sort of societal system.
35:05
There's like general guidance.
35:07
How is this actually going to be implemented?
35:08
It's going to be interesting.
35:10
So we're not 100 percent sure, but we see, a little bit of that.
35:13
We've seen promising things in California, but they've recently been turned down
35:17
by the government, governor there.
35:19
With the Trump administration, I'm not seeing that we'll
35:21
have a lot of regulation.
35:22
Maybe on the state level, but not on the federal level.
35:25
Probably not.
35:26
So we'll see what happens.
35:27
I do think there needs to be more regulation, but I think actually, I don't
35:30
know if every regulation will catch.
35:33
Problematic use case.
35:35
So I think that like companies need to speak out and push the vendors to
35:40
do better, or we need to have more non for profit initiatives to build
35:44
tool in the public interest, right?
35:46
I think as the barriers of building some of these tools come down and
35:49
make it easier for a lot of people.
35:52
To do the question is could we, could civil society build tools in the public
35:56
interest that are better than some of the commercial companies put out
36:01
there and we can actually make public how the tools were built, what the
36:05
assumptions are built in and all of those things that that we criticize that we
36:09
don't know from commercial suppliers.
36:11
That's all I do.
36:12
Nola Simon: Are you familiar with Amy Webb?
36:13
She's a futurist.
36:14
Hilke Schellmann: Yes.
36:15
Yes.
36:15
I've interviewed her actually for an article.
36:17
Oh, have
36:17
Nola Simon: you?
36:18
Okay.
36:18
What do you think about her point?
36:20
Her point is that she thinks that regulation is going to be
36:24
so challenging, especially to get it consistent across the globe.
36:28
And really it has to be a financial incentive for companies to really build it
36:35
ethically and Responsibly and if companies then hire the are used the AI produced
36:41
by that company, then they get a reward for choosing the ethical AI company.
36:48
Do you think that's viable?
36:49
Hilke Schellmann: Yeah.
36:50
I don't know if it's viable because I think.
36:52
I do agree that regulation can't catch them all, right?
36:56
Like it's not really going to work out.
36:57
It might work for like the European union, but that doesn't necessarily
37:00
transfer to all these other jurisdictions.
37:02
So it is really problematic and it can't catch them all.
37:05
So if we rely on industry itself, like maybe a system where we reward, or
37:09
we have some sort of certification we require we publicly shame companies
37:14
who don't use certification.
37:15
But the question is like.
37:17
does the certification work?
37:18
And I would argue that we've seen in some industries that we have
37:22
like thick leaf as certifications.
37:24
We've seen this about diamonds where diamonds come from.
37:27
We see this, we call it greenwashing, right?
37:30
That companies like buy a certification that they look like a green, organic,
37:34
super environmentally friendly company, but then they dump their
37:37
trash in the landfill and they actually don't recycle, whatever
37:40
their promise it's actually not true.
37:42
They're just buying.
37:43
The certification.
37:44
So I am worried about that too, that we then have another level of that looks
37:49
like this is certified and this is a good algorithm, then no one looks at it
37:55
and knows is this actually good or not?
37:57
And the, I wish we had like clear standards on how to judge algorithms.
38:01
They're being developed in the European Union, but there's still a lot of
38:04
we don't know a lot of things here.
38:06
So I hope we will have standards and maybe if we had a little
38:10
bit more transparency and demand accountability and explainability.
38:15
A lot of times when I do some of these tests, like talking German to
38:18
the tool and then, I get a, I got a six out of nine English proficiency
38:22
and I talk to the developers.
38:23
Oh, before we publish, we always go back.
38:25
Nola Simon: Yeah.
38:26
Hilke Schellmann: Ask for comment and also maybe we made a mistake, right?
38:29
Like we want to know.
38:31
So we go back to the developers.
38:32
And I was just, I was listening and listening.
38:35
I was like, wow.
38:36
He was like, it's maybe because German and English is in this 5D space.
38:40
The languages overlap.
38:41
And I was like, 5D, I was like, I'm really not understanding.
38:44
I was like, I can't follow.
38:46
I was like, can you just explain to me if you were in front of a judge,
38:49
how, why I was scored six out of nine?
38:51
Can you just explain how you would explain in front of a judge?
38:54
And he couldn't.
38:56
And I was like, I, 5D doesn't mean anything to me.
38:59
It seems very complex.
39:01
Can you just explain it?
39:02
Like how the score came together.
39:04
And I think if people can't do that tells us something like if you
39:07
can't explain why this person was scored this way, we have a problem.
39:13
So I think we need much more explainability, transparency.
39:16
If he could get that through government regulators, that would be great if he
39:20
could have like benchmarks standards, we've now seen in New York, there is
39:24
a law that companies who use AI and hiring or algorithms and hiring they
39:29
do have to get an audit every year.
39:31
It's up to the companies to actually decide how much
39:34
AI and algorithms they use.
39:35
So that's flawed.
39:36
And then the audits.
39:38
It's also unclear necessarily it's like usually follows a lot of people follow the
39:42
four fifth rule which is actually already.
39:45
a guideline by the Equal Employment Opportunity Commission in, in, in the US.
39:49
So it's like just auditing what's already being audited.
39:52
And we know that those kinds of audits are really flawed.
39:55
They just look at different genders, like male and females and
39:58
different sexes against each other.
40:00
They don't look at the intersection of Black, Black women versus
40:04
Caucasian men, for example.
40:06
We know at those intersections where marginalized identities crossover That's
40:11
where I see most of the problems in the system because marginalized groups
40:14
are underrepresented in the tool and in the training data, often not the tools.
40:19
Then they become underrepresented in the tools and the models.
40:22
And then we have the the problem multiplies.
40:25
And so we don't look at that.
40:26
There's no mandate to look at that.
40:28
And we've actually, when somebody looked at that, we found problems.
40:32
I'm hopeful ish.
40:34
That this will help.
40:36
A little bit.
40:37
Nola Simon: Let's talk about the whisper network.
40:38
So you just published an article in the Associated Press.
40:42
I read it this morning.
40:43
And it's basically about how you found problems in terms of a
40:48
transcription having a hallucination.
40:50
And honestly, this is 1 of the use cases that I use AI for often
40:55
when I do podcast, I'll take the transcript and I'll just put it into.
40:58
perplexity.
40:59
I prefer that.
41:00
And it spits out my show notes and I read it briefly.
41:03
And usually it's I don't really have a concern about it.
41:08
Also it's, that doesn't have a lot of effect.
41:10
Couldn't tell you anybody who's ever called back to me saying you
41:13
wrote really great show notes.
41:14
So I don't spend a lot of time on it.
41:16
But the whisper network is actually being used in hospitals and this is
41:20
where it can be very interesting in terms of transcripts for medical use that.
41:26
Are then actually they're deleting the audio.
41:29
And so the only record that remains is this transcript.
41:33
And you found in your test that there are hallucinations and the transcripts are
41:38
including text that actually was never
41:41
Hilke Schellmann: said.
41:42
Just chat GPT, but this is a transcription tool, right?
41:45
Like a benign.
41:47
Tool that companies use every day people use every day.
41:50
It's very ubiquitous.
41:52
So let me take one step back and tell you how we got to this because it
41:55
actually relates to AI and hiring and it relates to you know that I work with
41:59
computer scientists and sociologists.
42:01
So I've looked into AI.
42:03
one way video interviews, right?
42:05
And even regulators have asked me, and I've wondered about this too what happens
42:08
with people who have a speech disability, or have a dialect, or have who have an
42:13
accent, exactly what we talked about.
42:15
If they use these systems, or are being asked to use these systems
42:19
the tool then takes the audio and transcribes it into text.
42:22
Is it fair?
42:24
To people with different dialects or accent and speech disabilities.
42:28
So I was asked by regulators and others and I'm like, why are you asking me?
42:31
I'm just a lone journalist.
42:34
But I was like I guess no one is looking into this.
42:36
So I talked to a computer scientist and a sociologist and I was like,
42:39
I think we should really test this.
42:41
So I asked the companies which tools do you use, right?
42:44
Because most companies in the space don't build their own tools.
42:47
You buy one on the market because there's large companies that use those.
42:50
So we took those tools and then, as we were doing, we found a a database
42:55
with audio recordings of people who have a speech disability and people
42:59
who don't have a speech disability.
43:00
And we had about, 13, 000 recordings as a whole.
43:04
And we ran it through these transcription software.
43:07
And as we were starting to do the experiment, we also heard about open
43:10
AI snooze, new whisper tool, which everyone was like raving about.
43:14
And millions of people were already using it's open source.
43:16
It's very cheap.
43:17
It has a very high accuracy rate that everyone was raving about.
43:22
It can also use it can translate between different languages and be like, okay,
43:25
this becomes, seems to be like one of the market leaders, we should also.
43:28
And when we looked at the scatter plots, we did see that for Whisper, like a
43:33
lot of, there were very low error rates and some, but there were some plots
43:37
that had like ginormous error rates.
43:40
And we were like, how do you get a thousand percent error rate?
43:42
Like everything must be wrong in these transcripts compared to the ground truth.
43:47
So we did a human.
43:50
Intelligence investigation and looked at these two because you
43:52
know we were like did we upload a wrong clip like what happened here.
43:56
And when we looked into it we saw that it like in some instances
44:00
just hallucinated whole stories like made up whole paragraphs that
44:04
weren't there in the first place.
44:06
And when we did a close analysis like it also had facial commentary,
44:10
it, it had hallucinated a new medication, like all these new
44:14
things that like, were never there.
44:17
A medication that didn't actually exist.
44:19
Yes, it was, wasn't it psychoactivated?
44:23
Yeah, it was a hyper, hyperactivated.
44:25
Antibiotics.
44:26
It also like hallucinated violence and all kinds of things, because, it
44:30
learned from everything on the internet.
44:31
So like the stuff that we put on the internet and in books
44:34
and stuff, it learns from that.
44:36
But we hadn't seen this in a transcription software.
44:38
And when I started looking at GitHub, where a lot of like open source developers
44:41
congregate, there were just like so many entries of what is happening?
44:46
What is this tool doing?
44:47
I was like, Okay, we're not the only one.
44:48
So it wasn't like a problem that only we encountered.
44:51
This is like much, much bigger.
44:53
And then I found this company called Nabla that does we see this
44:56
a lot now, medical transcription.
44:58
So where doctor patient conversations are being recorded.
45:01
And then a system conscribes it and summarizes and you.
45:05
Does these doctor notes, right?
45:06
So your doctor can spend more time with you instead of documenting
45:09
everything and taking notes.
45:11
It sounds like a great idea, but some companies, I use whisper and not the
45:16
only one that uses whisper, but it is the, I think the, the only one that
45:20
I talked to that, so they work with about at least 30, 000 or so medical
45:24
providers in the U S and large.
45:27
systems and hospital systems.
45:29
And, they built their own tool on top of the whisper interface.
45:33
And it summarizes the recordings and what the tool also does, they,
45:39
it, because of privacy concerns, it throws away the underlying recording.
45:42
So if I am a doctor, I look at my notes and I'm like, Was that really sad?
45:48
I can go back to the transcript, but the transcript might not be accurate
45:52
because there might be hallucinations in there, and I can't go back to
45:55
the actual recording to check.
45:56
Was that actually said did anyone talk about antibiotics or, whatever
46:00
the hallucination might be.
46:03
And, to a lot of people who look at AI, this was really problematic that
46:08
a company was using that, not, and I also asked them, I was like, we
46:11
know that whisper hallucinates and they're like, oh yeah, we know too.
46:14
We try to mitigate it.
46:15
And I was like, oh.
46:16
Were you able to get rid of it 100 percent because I've never heard of
46:19
anybody being able to mitigate it 100 percent and they were like, no,
46:22
but I think we have it under control.
46:24
And I was like, okay So what are the, what are
46:26
Nola Simon: the impacts of having a medical transcript that is inaccurate
46:29
and includes a hallucination?
46:30
So that could be, it could go to court.
46:34
It can be used as evidence to actually deny coverage for insurance claims.
46:42
Hilke Schellmann: Yeah, it also is in your presumably in your
46:45
electronic medical records forever.
46:47
So you might have so it could lead to misdiagnosis.
46:50
Yeah it could have a lifelong implications.
46:53
You might also get a prescription for the wrong medication
46:57
all kinds of consequences.
46:58
Nola Simon: It can even, honestly, with the changes in law, if it picks up somehow
47:02
that you had an illegal abortion, does that then lead to legal consequences?
47:09
Hilke Schellmann: Yeah,
47:09
Nola Simon: if that's not something that haven't happened.
47:14
Hilke Schellmann: Yeah, and I think that's, it could be that it hallucinates
47:16
something, but it could also be like what happens if the transcript is actually
47:21
accurate and the transcript is used for more, it's not protected by HIPAA, right?
47:25
Which is the privacy loss in the United States because that.
47:30
mandates that medical information stays private between a patient
47:33
and a doctor, but it's also now shared with other companies.
47:37
And what we've seen is like now patients are asked to sign a
47:40
release to actually sign away.
47:42
Their HIPAA rights that, says I acknowledge that the transcripts or the
47:46
end of recording sometimes is being shared with some large tech companies, these
47:51
providers and, that does worry me too, that when we go to the doctor, we get
47:55
this tablet where you just sign things and, just sign here, you don't even read
47:58
it, and people aren't even aware of the potential consequences that these very
48:03
personal I think a lot of things that we tell our doctor, we might not want
48:07
to tell others very private information gets shared with large tech companies.
48:11
Then they use it against for for training data and other things.
48:14
And who knows who uses it, who is being sold to the next time what happens if
48:19
a company goes bankrupt, like that is it's usually very valuable material
48:23
that then the next company buys.
48:25
And, yeah, or even just
48:26
Nola Simon: like you run the test and go, tell me what you know about
48:29
this person and all of a sudden your private medical records show up in
48:34
like a general inquiry that anybody in the world could actually ask.
48:39
Hilke Schellmann: Yeah, and, I find that, they're like anonymized,
48:42
but we also know from like anonymized data that they can be.
48:46
Pretty easily right there.
48:48
There's a lot of data on us out there.
48:51
So it does worry me.
48:52
And I know of companies use this kind of medical data to then predict
48:57
this is what happens to people.
48:58
They need a therapist.
49:00
They need back surgery.
49:01
We already know it's not.
49:03
100 percent predicted protected.
49:04
I'm not saying it's the specific transcript, but there is like data
49:07
lakes of medical data on Americans and other people out there that is already
49:12
used for kind of predictions and and things that I think people would find
49:16
very creepy if they know why they're being suggested that everyone has a bad
49:21
day and they should see a therapist.
49:23
I might, When my company sends me a nudge like that, I might be like, Oh, yeah,
49:27
maybe I'm having a bad day if I would know that the company has derived this
49:32
information because I through my spouse of my medical benefits, and that might be a
49:37
signal of divorce and I need a therapist.
49:40
I might feel very different that my company shares that kind of
49:42
information with a third party provider.
49:45
Nola Simon: Yeah, it's so concerning.
49:47
Hilke Schellmann: Yeah, that's probably going to be the subject of
49:50
my next book is AI and health care.
49:52
Yeah, that's fascinating because we see a lot of AI also moving
49:57
into the workplace, right?
49:58
That can find out like, are we depressed?
50:00
Are we anxious?
50:00
And I think there is something that like, if you share this with
50:03
your doctor, I hope gets protected.
50:06
We just talked about that the protection isn't ubiquitous, but but also What
50:10
happens in the workplace with like, when we have these kinds of tools are they
50:14
used only for the benefit of employees?
50:16
Are they also being used against them?
50:18
That was
50:18
Nola Simon: one of the questions in your book that I found the most fascinating,
50:21
because it goes back initially to my interest in the eightfold, which was,
50:25
if you identify transferable skills that are valuable, but that Employee
50:30
doesn't necessarily want to use that skill because they find it draining or
50:34
they have trauma associated with, that work in the past or whatever reason
50:39
they don't want to do it anymore.
50:41
What are the repercussions for turning down what the companies
50:44
perceives to be an opportunity?
50:46
Yeah.
50:47
To benefit from your skill.
50:49
Hilke Schellmann: Yeah.
50:50
And I think also what I thought is interesting about eightfold is like,
50:53
it has these like career laddering that people in your position did
50:57
this, they became vice president five years because they did X, Y and Z.
51:01
Here's some, LinkedIn learning or whatever learning platform that
51:04
you can use to get those skills.
51:06
And that's interesting.
51:07
So but it could also be used for managers to find like high performers
51:12
and it could also be used to find.
51:15
Maybe what we might see from the platform is low performance right
51:18
people who didn't advance in five years to vice president who took longer.
51:22
But the platform doesn't tell you why that is are they slow learners or
51:25
did they have difficult pregnancy or did they have to take care of their
51:29
parents and you know didn't want.
51:31
Yeah, there's multiple reasons why you might.
51:33
And like.
51:34
When I asked them that question, I was like it could also
51:36
be used to penalize people.
51:38
They're like everyone knows about it.
51:40
The employer knows about it, about our assessment of them
51:43
and the employee knows as well.
51:45
And the employer you could question it, but I don't think it uses it.
51:49
Nola Simon: If you're in a state that has at will employment and
51:52
you don't like the answer to that particular question, then you could
51:55
also use it as a reason to terminate.
51:58
Hilke Schellmann: Yeah, totally.
51:59
Exactly.
52:00
Exactly.
52:00
And I think there, there isn't a whole lot of protection.
52:02
It could also be that maybe you used another learning platform, right?
52:06
To have new skills and that get never registered in the company's
52:11
system, which uses maybe another event or you read a book.
52:15
Yes, and that never is part of the system either, right?
52:18
Like we can think of so many use cases that are actually not part of
52:21
the what a company collects on us.
52:24
And there's no way to like manually entry that and be like, no, but I
52:28
did get these skills somewhere else.
52:30
And we don't necessarily know how company executives or managers
52:34
actually use these systems.
52:36
And I think that opens it up for some doubtful things.
52:39
And we've seen this, unfortunately, again and again, that companies
52:42
like collect data for one thing, and then they want to analyze it.
52:46
And it's actually not then their layoffs, they want to look at the data
52:49
that they collected for one thing.
52:51
It's actually not used for layoffs, but it's not supposed to be used
52:54
for layoffs, but they still use it because they have the data, right?
52:57
Once you have it, I know it.
53:00
Exactly.
53:01
And I think that's another problem.
53:02
Once you have the data, once you do the analysis, like if I think that
53:06
you are at flight risk, meaning you're going to leave the company,
53:09
the, there are these indicators that say this person is 80 percent likely
53:12
to leave the company in a year.
53:14
Am I in a manager and I want to keep you maybe I'll give you a promotion.
53:19
Did you deserve it?
53:20
I don't know.
53:20
Probably somebody else is not gonna get a promotion or a raise because
53:23
there's only a limit, a limited pod, and I only do that based on this.
53:28
flight risk indicator.
53:29
How can I unsee that?
53:30
Am I going to put you forward for leadership training if you
53:33
think you're already one way out?
53:35
I don't know.
53:36
But the question is you don't even know that the tool predicted that
53:39
you're with one foot out the door and you may or may not be, right?
53:42
Like it's right.
53:43
Exactly.
53:43
Nola Simon: There might be something keeping you there like
53:45
a personal relationship that isn't going to show up on any tool.
53:48
Hilke Schellmann: Yeah.
53:49
Nola Simon: You follow the advice to have friends at work.
53:52
I
53:53
Hilke Schellmann: mean, there are so many things connected to this.
53:56
It's just, it's wild.
53:57
So
53:57
Nola Simon: I wanted to make sure that we were going back to the book.
54:00
You've fabulous book.
54:02
I highly recommend it really got a great job.
54:04
Thank you.
54:05
What are the successes that you've had with the book that
54:08
you really want to highlight?
54:09
Because it's done so well.
54:13
Hilke Schellmann: Yeah, I got really lucky that I think, it was
54:14
like a, it was like a a topic that is really in the zeitgeist, right?
54:18
And I think a lot of job seekers also feel like I've heard of
54:21
algorithms, how do they really work?
54:23
And do you think that people in talent acquisitions Are also like
54:26
thinking do these tools work?
54:28
And there may have like similar questions.
54:30
So I think it was also really interesting for them.
54:32
So I've been lucky to talk in actually like HR, talent acquisition conferences.
54:37
And I'm very grateful for that.
54:38
Cause I think those are the people that I actually have maybe more
54:41
than job seekers, actually maybe the power to actually investigate these
54:44
tools, maybe make changes, I've also spoken at data science and technology
54:49
companies for the people who build these tools to be like much more.
54:55
thoughtful and think through and talk to different people and not just assume
54:59
that they're domain experts about hiring.
55:02
And really, they have no clue how to really hire and what are their best case
55:06
scenarios and how to not discriminate.
55:08
So I also tell them like, here's how, what you should avoid.
55:11
And here's exactly how you avoid it.
55:12
Don't do this, don't do that.
55:14
So I'm really grateful that I feel like people who built the technology and use
55:18
the technology are actually listening.
55:21
And then I love just connecting with like readers from all over the
55:24
world, it's amazing when you get a message from like Brazil and the next
55:28
day from Sweden, somebody read your book was really meaningful to them.
55:31
Like I was just like, very honored.
55:33
And then recently I gave a TED talk about the subject.
55:36
Oh, I listened to the podcast.
55:38
Yeah.
55:38
And getting new finding new audiences.
55:41
So hopefully, it's also reached some lawmakers.
55:43
So hopefully there will be change.
55:44
And I'm just like, so grateful for people who pick up the book,
55:47
spend so much time with it.
55:48
And I hope to illuminate a new world that maybe they didn't know about.
55:52
And it's endlessly fascinating to me how we quantify human beings.
55:57
So I, I hope that like rubs off on people too, that it is interesting, like
56:01
how we try to make sense of the world,
56:03
In in, in different ways.
56:05
Nola Simon: Yeah, no, it's fascinating.
56:06
And it validated me because I'm like, I don't have a journalistic background
56:09
at all, but I went back and I'm like, Oh, look, I was hearing really a lot of
56:13
the similar work and I'm like, exactly.
56:15
And I think that's what I want to do.
56:17
journalistic instincts to say, Hey, I'm noticing these things.
56:22
This is good.
56:22
Yeah.
56:22
Hilke Schellmann: Welcome to our world.
56:24
Like the next book about AI and hiring and in the world of work.
56:28
But I actually think I always want to encourage people.
56:30
I tell them like steal my methods.
56:32
Like some of them are super basic.
56:34
Some of them are more elaborate, but yeah, these methods to scrutinize
56:38
these AI tools, because as we can see, like technology companies just put
56:42
out these tools and, for example, with Open AI knew about the hallucinations.
56:47
In fact, they put it in the paper.
56:48
They did say in one of their disclosures, it shouldn't be
56:50
used for high risk use cases.
56:53
Lots of people don't need the model card.
56:55
So they ignored the advice from the company.
56:57
They took it from it's not like they didn't know, but you wonder like,
57:00
why did they release a flop system?
57:02
There's all kinds of questions that we can ask about this.
57:05
So I feel like.
57:06
We need to be much more skeptical users, testers of
57:10
these technologies and push back.
57:12
So I want everyone to steal my methods and do, invent your own ones, test
57:17
these tools push back against the companies so we build better tools.
57:22
Because you think that AI can be transformative technology.
57:25
The jury is still out if we use it to predict the future of job seekers.
57:29
It's really a question if we actually, if AI can help with that.
57:32
But I think it's really helpful in in, in other use cases.
57:35
But in some maybe we shouldn't use it and we should push back and
57:38
like highlight how it doesn't work.
57:40
So I think everyone can do the work.
57:41
We don't have enough journalists to do all this work.
57:43
Nola Simon: Yeah.
57:44
This is what my fourth podcast episode about it.
57:46
So
57:49
Hilke Schellmann: I'm doing my part.
57:51
Thank you.
57:52
Yeah.
57:52
And if anyone wants to be in touch with me I'm on LinkedIn.
57:54
You find my email on my pet faculty page.
57:56
There's only one Helga Shellman.
57:58
Please be in touch.
57:58
Like I'm happy to I always love to hear from job seekers or people
58:02
who work in talent acquisition.
58:04
I love talking to folks.
58:05
It might take a few days to like respond.
58:09
And, I have a bunch of obligations.
58:11
I also have a four year old kid I like to spend time with.
58:14
And, she demands a scavenger hunts for sweets with her mom.
58:18
But, I love to be in touch with people and I, I'm so grateful for
58:21
everyone who's listening and has wrote me and, has engaged with the book.
58:25
It's just wonderful.
58:26
Nola Simon: Okay.
58:27
That's good.
58:27
I think make sure that everything is linked in the show notes.
58:30
And I really appreciate you making the time.
58:32
So thank you so much.
58:34
I'll see you next time.