Latest / The Science of Leading / Algorithmic Monoculture: Why Hiring Tools Reproduce Bias

Transcript

Algorithmic Monoculture: Why Hiring Tools Reproduce Bias · The Science of Leading

• 0:00
  Welcome to the show everybody!
• 0:01
  I'm Claire Monroe, here with Edwin Carrington.
• 0:04
  Edwin, I want to start with a phrase I can't stop thinking about: algorithmic
• 0:08
  monoculture.
• 0:10
  It comes from a 2022 paper out of Stanford by Rishi Bommasani,
• 0:14
  Percy Liang and their colleagues, titled Picking on the Same Person.
• 0:18
  Their argument is that when lots of organizations rely on the same model,
• 0:22
  or models trained on the same data, the same individuals end up rejected by all of
• 0:26
  them.
• 0:27
  They call it outcome homogenization.
• 0:29
  It's an important idea, Claire, and it changes how we should think about bias in
• 0:33
  hiring.
• 0:35
  For most of history, a biased hiring manager was a local problem.
• 0:39
  He was wrong, and he cost a good candidate one job.
• 0:42
  That candidate walked across the street, met a different manager with different
• 0:46
  blind spots, and got hired.
• 0:48
  The errors were scattered.
• 0:50
  Scattered errors are survivable.
• 0:52
  Because every door had a different lock.
• 0:54
  Exactly.
• 0:56
  Now imagine every door has the same lock.
• 0:59
  If a thousand companies screen through the same tool, and that tool has learned to
• 1:03
  undervalue a certain kind of résumé, that person isn't rejected once.
• 1:08
  They're rejected a thousand times, by the same logic, in the same week.
• 1:13
  And no single employer ever sees the pattern, because each of them only sees their
• 1:17
  own funnel.
• 1:18
  And the Stanford team found something even more uncomfortable.
• 1:21
  When models share training data, the effect is strongest for specific individuals,
• 1:26
  even more than for whole demographic groups.
• 1:28
  So it's not only that a group is disadvantaged on average.
• 1:31
  It's that particular people get locked out of the entire market.
• 1:35
  That's the part leaders miss.
• 1:37
  They ask, is this tool accurate?
• 1:40
  That's the wrong question on its own.
• 1:42
  The better question is, whose mistakes am I inheriting, and how many other companies
• 1:47
  are inheriting the same ones?
• 1:48
  There's a second paper that backs this up.
• 1:50
  Jon Kleinberg and Manish Raghavan at Cornell published a study in the Proceedings of
• 1:55
  the National Academy of Sciences in 2021.
• 1:58
  Their finding was that even when an algorithm is more accurate than any individual
• 2:02
  human evaluator, having everyone use the same one can make overall hiring decisions
• 2:07
  worse.
• 2:07
  More accurate for each company, worse for the whole system.
• 2:11
  That is the monoculture in one sentence.
• 2:14
  Independence of judgment is a hidden asset, and most organizations are trading it
• 2:18
  away without noticing.
• 2:19
  So let's talk about how these tools learn to pick on the same person in the first
• 2:23
  place.
• 2:24
  The study that stopped me in my tracks came from the University of Washington in
• 2:28
  October 2024.
• 2:30
  Kyra Wilson and Aylin Caliskan tested three large language models on more than five
• 2:36
  hundred real résumés against more than five hundred real job descriptions.
• 2:40
  Over three million comparisons in total.
• 2:43
  The models favored white-associated names eighty-five percent of the time.
• 2:47
  Female-associated names were favored only eleven percent of the time.
• 2:51
  And the models never favored Black male-associated names over white male-associated
• 2:56
  names.
• 2:57
  Never, Edwin.
• 2:58
  Not once.
• 2:59
  Eighty-five percent.
• 3:00
  And notice what changed between those résumés.
• 3:03
  Not the experience.
• 3:05
  Not the skills.
• 3:06
  The name.
• 3:08
  That tells you the model isn't evaluating the work.
• 3:11
  It's evaluating a proxy.
• 3:12
  And this isn't new.
• 3:14
  Back in 2018, Reuters reported that Amazon scrapped an experimental recruiting tool
• 3:19
  because it had taught itself to downgrade résumés that included the word women's,
• 3:23
  as in women's chess club captain.
• 3:25
  It learned from ten years of résumés that came mostly from men.
• 3:29
  That's the mechanism, Claire, and it's worth saying plainly.
• 3:32
  A screening model trained on your past hires learns who you hired.
• 3:35
  It does not learn who performed.
• 3:37
  Those are two very different datasets.
• 3:40
  If your history carries a bias, the model doesn't correct it.
• 3:43
  It automates it, at a speed and volume no human recruiter could ever match.
• 3:48
  So the bias gets in through the training data, and then it gets amplified through
• 3:51
  consistency.
• 3:52
  And then it spreads through concentration.
• 3:55
  When the market converges on a handful of vendors, one vendor's blind spot becomes
• 4:00
  an industry's blind spot.
• 4:03
  The résumé filter that quietly penalizes a gap year or a particular zip code isn't
• 4:08
  one company's problem anymore.
• 4:10
  It's a gate across an entire profession.
• 4:13
  Which is scary, because most companies buy these tools to reduce bias.
• 4:17
  That's the pitch.
• 4:18
  The pitch assumes the tool is neutral.
• 4:20
  No tool is neutral.
• 4:22
  Every filter encodes a definition of what good looks like.
• 4:25
  The only question is whose definition, and whether anyone has checked it against
• 4:29
  actual performance.
• 4:30
  Okay, so what do leaders actually do?
• 4:32
  Because I don't think the answer is to throw out structure and go back to gut feel.
• 4:36
  No, gut feel is its own monoculture, the monoculture of one person's instincts.
• 4:42
  The answer is to change what you measure and where you measure it.
• 4:45
  Two moves.
• 4:47
  First, measure against your own standard, not the market's.
• 4:50
  Second, measure early, before the résumé screen has already narrowed the pool on
• 4:55
  proxy signals.
• 4:56
  Say more about your own standard.
• 4:58
  Most screening tools compare candidates to a generic picture of a good hire,
• 5:02
  built from someone else's data.
• 5:04
  OAD works the other way.
• 5:06
  It measures behavioral drive, meaning how someone is wired to operate under
• 5:11
  role-specific pressure, and compares candidates against a Top Performer Benchmark
• 5:16
  built from your own best performer.
• 5:18
  Every candidate is scored on the same five dimensions: Drive,
• 5:22
  Influence, Resilience, Structure and Leadership.
• 5:27
  So your filter reflects what success looks like inside your organization,
• 5:31
  in that role.
• 5:32
  That's the opposite of a monoculture.
• 5:34
  Because my top performer isn't my competitor's top performer.
• 5:37
  Precisely.
• 5:38
  And the benchmark measures how people operate, not what their name is or which
• 5:43
  school is on the résumé.
• 5:45
  OAD has been EEOC compliant since 2001, with no adverse impact on any protected
• 5:52
  class, and zero successful legal challenges in thirty-eight years.
• 5:57
  But I want to be careful here, Claire.
• 5:59
  It is built as one structured input in the hiring decision.
• 6:02
  Never the sole basis for a hire.
• 6:05
  A human stays accountable for the decision.
• 6:07
  I like that, because it gives leaders a practical checklist for any vendor,
• 6:11
  not just OAD.
• 6:13
  What data was this trained on?
• 6:15
  Who else in my industry uses it?
• 6:17
  Can I see pass-through rates by demographic group at every stage of my funnel?
• 6:21
  And can you explain why a candidate was rejected?
• 6:23
  Those four questions will tell you more than any sales demo.
• 6:27
  And if a vendor can't answer them, you have your answer.
• 6:31
  You would never let one supplier set the price of every input in your business.
• 6:36
  Don't let one algorithm set the definition of talent for your whole industry.
• 6:40
  So if you want to see what your own standard looks like, start with the person you
• 6:44
  already trust most on your team.
• 6:47
  OAD's free Top Performer Profile takes seven minutes, you get results in twenty-four
• 6:51
  hours, and there's no credit card.
• 6:53
  It's a clear look at what a validated, structured input looks like next to whatever
• 6:57
  you're running today.
• 6:59
  That's O-A-D dot A-I.
• 7:01
  The best companies have never won by thinking like everyone else,
• 7:04
  Claire.
• 7:04
  Hiring is no different.
• 7:06
  Build a filter that is yours.
• 7:07
  Perfect place to leave it.
• 7:08
  Thanks for the wisdom, Edwin.
• 7:10
  Always a pleasure, Claire.