Latest / Hiring & Firing with Fexingo: Recruitment, Performance, and Workforce Decisions / The Hiring Algorithm That Learned Bias From Human Data
Transcript
- Lucas: So there's this story from a few years back — Amazon built an AI resume screener, and it basically taught itself to be sexist. Luna: I remember that. The algorithm started penalizing resumes that included the word 'women's,' right? Lucas: Exactly. And it wasn't some rogue engineer slipping in a gender filter. The team trained the model on ten years of Amazon's own hiring data. The problem was, that data reflected a tech industry that had historically hired far more men than women. Luna: So the algorithm saw patterns like 'women's chess club captain' or 'women's soccer team' and inferred those candidates were less likely to succeed — because past successful hires didn't have those phrases. Lucas: Right. And it went deeper. It also downgraded candidates who attended all-women's colleges. It favored verbs like 'executed' over 'participated' — verbs that appeared more frequently on men's resumes. Amazon's engineers tried to fix it, but the model kept finding proxies for gender. Eventually, they disbanded the project. Luna: But here's the thing — Amazon wasn't trying to be biased. They were trying to scale hiring. How common is this kind of blind spot in AI hiring tools today? Lucas: It's everywhere. A lot of companies are using off-the-shelf AI screening tools that claim to be 'bias-free' because they don't explicitly use gender or race as inputs. But the data they're trained on — historical hiring decisions — already contains human bias. The model just learns to replicate it, sometimes in subtler ways. Luna: So it's not the algorithm's fault, it's the data's fault. But the algorithm amplifies it. Lucas: Exactly. There was a famous study from MIT Media Lab where they found that commercial facial analysis systems had error rates of less than one percent for light-skinned men, but up to thirty-five percent for dark-skinned women. Same problem: training data was mostly white men. Luna: And with hiring, the stakes are even higher because it directly affects people's livelihoods. Lucas: Right. And the scary part is, most companies using these tools don't audit them. They assume that because the software is 'AI,' it's objective. But the AI is only as good as the historical decisions it's trained on — and those decisions were made by humans with all their biases. Luna: So what should a company do if they want to use AI in hiring without baking in old biases? Lucas: First, don't train your model on raw historical data. You need to clean it — remove or reweight data from periods where you know bias was worse. Second, define success metrics that aren't just 'who got hired before,' but include measures of actual on-the-job performance that control for bias. Luna: So you're saying use objective performance data, not just past hiring decisions. Lucas: Yes. And third, you need to actively test for fairness. There are statistical tests — like measuring whether the model's false positive or false negative rates differ across demographic groups. If they do, you have a problem. Luna: But a lot of small and mid-size companies don't have data scientists on staff. They're buying a tool from a vendor who says 'our AI is fair.' Lucas: That's a huge risk. The vendor may not know. Amazon's own team didn't realize the depth of the bias until they dug in. So the buyer needs to ask: what data was this trained on? How did you validate it? Can you show me the results of a fairness audit? Luna: And if the vendor can't answer those questions, that's a red flag. Lucas: Big red flag. There's also a growing movement toward 'algorithmic impact assessments' — basically a required audit before deploying AI in hiring. Some states and cities are already passing laws around this. Luna: So regulation is catching up, but slowly. Lucas: Yeah, New York City passed a law last year requiring bias audits for automated hiring tools. More will follow. But the tech moves faster than the law. Luna: I want to go back to the Amazon story for a second. They reportedly tried to fix the model multiple times but couldn't stop it from finding gender proxies. What does that tell us? Lucas: It tells us that bias can be deeply embedded in the training signal. For example, even if you remove gender-related words, the model might learn that candidates who use more 'action verbs' are better — but if men historically used those verbs more, you've just moved the goalpost. Luna: So it's not enough to remove obvious markers. You have to rethink the entire modeling approach. Lucas: Right. Some researchers argue that you should instead train a model to predict something like 'job performance,' using data that's been corrected for bias — like using blind performance reviews, or multi-rater assessments. Luna: But even that data could have bias if the performance reviews themselves are biased. Lucas: Exactly. There's no perfect solution. But awareness is the first step. Companies that assume their AI is neutral are the ones most likely to get burned. Luna: So if a listener is in HR or talent acquisition right now, what's one actionable takeaway from this episode? Lucas: Ask your vendor or your data team: what is the model actually learning? Run a simple test: take resumes that differ only in demographic proxies — like swapping 'women's soccer' for 'soccer' — and see if the score changes. If it does, you have bias. Luna: That's a concrete test. And it doesn't require a PhD. Lucas: Exactly. And if these kinds of conversations help you think differently about your hiring process, that's the whole point of the show. We keep it ad-free and listener-supported — if you find value in it, you can toss a few bucks our way at buy me a coffee dot com slash fexingo. No pressure, just helps keep the lights on. Luna: Yeah, it's a small way to say 'this matters.' And we appreciate every bit. Lucas: Alright, back to the core point. The Amazon case shows that even the most sophisticated tech companies can build biased systems without intending to. The lesson for everyone else is: you have to look under the hood. Luna: And maybe the safest path is to keep humans in the loop — but informed humans who know what to look for. Lucas: Absolutely. The algorithm isn't the enemy. The unexamined algorithm is. Luna: Well said. Thanks, Lucas. Lucas: Thanks, Luna. Catch you next time.