Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / The Biased Recruiter Inside Your Job Application
Transcript
- Lucas: So, back in 2018, Reuters published an investigation that should have been a five-alarm fire for anyone building AI for human resources. Amazon had developed an internal tool to screen job applicants — a machine learning model that would read resumes and rank candidates from one to five stars. Luna: I remember this. The system was supposedly designed to find the best technical talent without human bias. But it ended up doing the exact opposite. Lucas: Exactly. The model had been trained on ten years of resumes submitted to Amazon — mostly from men. And because the tech industry is male-dominated, the algorithm learned to penalize anything it associated with women. It literally downgraded resumes that included the word 'women's' — like 'women's chess club captain' or 'women's soccer league president.' Luna: That's the part that still makes me angry. The algorithm didn't just miss qualified women — it actively punished them for signaling their gender. Lucas: Right. And here's the thing — Amazon's engineers knew this was happening. Internal emails, later obtained by Reuters, showed that teams flagged the bias during development. But the tool was deployed anyway, and it ran in production for about a year before Amazon finally killed it. Luna: Wait — they knew about the bias before launch and shipped it anyway? How does that happen in a company that prides itself on engineering rigor? Lucas: It's a classic case of AI ethics being treated as an afterthought. The team was likely under pressure to deliver a recruiting efficiency tool. Amazon receives hundreds of thousands of resumes a year — if you can automate the initial screen, you save millions of dollars. So the bias was seen as a 'known issue' that could be fixed later. Luna: But it never got fixed. They just scrapped the whole project after the Reuters story broke. Lucas: And that's the pattern we see over and over. A company builds an AI tool, the bias is discovered internally, there's a promise to 'de-bias' it, but the fix never happens because the metrics that matter to leadership are speed and cost savings — not fairness. Luna: So what was the actual technical mechanism? How does a model learn to associate 'women's' with a lower rank? Lucas: It's a classic case of historical bias baked into the training data. Amazon's training set came from resumes submitted over the prior decade. Since the tech workforce is roughly 75 percent male, most of the 'good' resumes — the ones that led to hires — came from men. The model learned that patterns like 'women's' were predictive of a resume being less likely to lead to a hire, because in the training data, they were. Luna: So the model was just pattern-matching the past — a past that was already unfair. Lucas: Exactly. And to be clear, the model wasn't explicitly told to penalize women. It was told to predict which resumes would result in a successful hire. It just so happened that the easiest way to make that prediction was to use gender as a proxy. Luna: But couldn't they have just removed the gender signal? Like, not include the word 'women's' or the candidate's name? Lucas: They tried that. Amazon actually had a version of the tool that stripped out explicit gender markers — names, pronouns, gendered terms. But the model still found proxies. It learned that graduates from all-women's colleges were less likely to be top-ranked, because in the training data, those graduates were less likely to have been hired. Luna: So the bias was so deeply embedded in the data that you couldn't just 'turn it off' with a filter. That's terrifying. Lucas: That's the core lesson from the Amazon case. Bias in AI is not a simple bug you can patch. It's a feature of the data. And if your data reflects systemic inequality, your model will reproduce that inequality, often in surprising and hard to detect ways. Luna: What I find frustrating is that this wasn't some small startup cutting corners. This was Amazon — a company with arguably the best AI research team in the world. Lucas: And yet, the incentives worked against an ethical outcome. The recruiting team wanted faster screening. The engineering team wanted to ship. The legal team probably didn't even know about the bias until the story hit. And there was no internal ethics review board with the power to say 'no.' Luna: So what happened after the story broke? Did Amazon change its approach to AI ethics? Lucas: Publicly, Amazon said it would continue to invest in 'fair and unbiased' AI. But internally, the recruiting tool was shelved. The team was reportedly disbanded or reassigned. There was no major restructuring of how AI projects are approved. Luna: So the lesson was learned — but only by the people who read the Reuters article. Lucas: In some ways, yes. But the story did have an impact. It became a case study in every AI ethics syllabus. It's one of the main reasons why the field of 'algorithmic auditing' emerged — independent researchers and startups who test AI systems for bias before they go live. Luna: And now, in 2026, are things any better? Are companies actually auditing their hiring algorithms? Lucas: Some are. New York City passed a law in 2023 requiring audits of AI hiring tools. A handful of other cities and states have followed. But enforcement is spotty, and many audits are done by the companies themselves, not independent third parties. Luna: So it's still the Wild West, just with a few more cautionary tales. Lucas: That's a fair summary. But there's also been progress on the technical side. Researchers have developed techniques like 'adversarial debiasing' — you train two models, one that predicts outcomes and one that tries to detect bias, and you force them to compete until the bias is minimized. Luna: But those techniques only work if companies actually use them. And if the incentive to ship fast outweighs the incentive to be fair... Lucas: Then you get another Amazon. Or worse, you get a system that is biased but never uncovered because no journalist happens to dig into it. Luna: So the Amazon case really is the cautionary tale of our time. It showed that even the best-funded, smartest team can build a weapon of bias if they don't ask the right questions from day one. Lucas: And that's the note I want to end on. AI ethics is not a separate step you add at the end of development. It has to be part of the data collection process, the model design, the testing, and the deployment monitoring. The moment you treat it as an afterthought, you've already lost. Luna: So for anyone building an AI tool today — or using one — the takeaway is: ask what's in your training data. And if you can't answer that question clearly, you have a problem. Lucas: Next episode, we're going to look at a different kind of bias — one that affects medical diagnosis algorithms, and how a skin cancer detection model got it terribly wrong. Until then, keep questioning the algorithm.