Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Recruiter Questions Your Name
Transcript
- Lucas: So Luna, I want to run a little experiment with you. Imagine you're applying for a job — same exact resume, same exact qualifications, same exact work history. But on one version, your name is Emily Watson. On the other, it's Fatima Al-Rashid. Luna: I know where this is going. And I'm guessing the AI recruiter isn't treating them equally. Lucas: Correct. And this isn't just speculation — there's a very specific 2025 study from researchers at Princeton and the University of Amsterdam that put this to the test. They submitted over 5,000 matched resumes to AI screening tools used by major employers. Luna: Matched meaning literally identical except for the name? Lucas: Identical. Same skills section, same education, same formatting. The only variable was the name at the top. And the result was a 14 percentage point difference in callback recommendations — Emily Watson got recommended for an interview 14 percent more often than Fatima Al-Rashid. Luna: Fourteen percent — that's not a rounding error. That's systemic. And these are tools that companies are using to filter hundreds of applicants a day. Lucas: Exactly. And the really troubling part is that these systems weren't explicitly told to discriminate. The AI learned it from historical hiring data — because past human recruiters favored certain names, the model internalized that pattern as a 'signal' of a good candidate. Luna: So it's basically encoding old human bias into an automated system that runs at scale. You don't fix it by just saying 'don't look at the name.' Lucas: That's the key insight. The researchers tested what happens if you strip the name field entirely before the AI processes the resume. And it barely helped. The gap only shrunk by about 3 percentage points. Luna: How is that possible? If the AI can't see the name, how does it still know? Lucas: That's where the technical term 'residual name embeddings' comes in. The AI reads the entire resume as a block of text, and it picks up subtle cues — the name of a university in Egypt, a previous employer in the UAE, a community organization that's predominantly one ethnicity. These are correlated with name origin, so the model essentially reconstructs the demographic signal from other parts of the resume. Luna: So it's like the AI is playing a game of Where's Waldo with ethnicity. Even when you hide the obvious clue, it finds the pattern in the noise. Lucas: Exactly. And that makes it much harder to fix. A lot of companies have responded by saying, 'We already removed names from our screening process.' But this study shows that if the rest of the resume still contains correlated information, the bias persists. Luna: So what actually works? Are there technical solutions that go beyond just removing the name field? Lucas: There are a few approaches, but none of them are silver bullets. One is something called 'adversarial debiasing' — you train a second model that tries to predict the protected attribute — like ethnicity — from the AI's internal representations, and you penalize the first model if the second model succeeds. Luna: So you're basically teaching the primary AI to hide the demographic signal from a detective. Lucas: Right. Another method is to use 'counterfactual data augmentation' — you generate synthetic resumes where you swap the ethnicity-linked cues, like changing the name of the university to a comparable domestic one, and then you train the model to treat those as equivalent. Luna: Both sound technically complex and expensive. Are companies actually doing this, or is it still mostly academic? Lucas: It's moving from academic to early adoption. A few major tech firms and some AI auditing startups have started offering these tools. But here's the thing — the European Union's AI Act, which came fully into force in stages last year, now requires 'high-risk' AI systems — and recruitment tools are classified as high-risk — to undergo bias audits and implement mitigation measures. Luna: So regulation is forcing companies to stop relying on the 'we removed names' excuse and actually measure whether the bias is still there. Lucas: Exactly. And that's a huge shift. Under the AI Act, companies have to demonstrate that their hiring AI doesn't produce disparate impact across protected groups — and they have to do it with statistical evidence, not just good intentions. Luna: What about in the US? We don't have an equivalent federal law yet. Lucas: We don't. But the Equal Employment Opportunity Commission has been paying close attention. In 2024, they issued updated guidance saying that employers are liable for discriminatory outcomes caused by AI tools, even if they didn't intend it. And there have been a couple of high-profile settlements — one large retailer settled a class action last year over an AI screening tool that disproportionately filtered out older workers. Luna: So the liability risk is real. But I wonder — are the technical fixes reliable enough to hold up in court? If a company uses adversarial debiasing, and the bias drops from 14 percent to 2 percent, is that good enough? Lucas: That's the billion-dollar question. The Princeton study found that after debiasing, the gap fell to about 2 percent — which is statistically significant but much smaller. But here's the catch: debiasing can sometimes hurt the model's accuracy for everyone. You might reduce bias at the cost of making worse hiring decisions overall. Luna: So there's a trade-off between fairness and predictive power. That's a deeply uncomfortable conversation for a lot of companies. Lucas: It really is. And it gets even more uncomfortable when you consider that 'accuracy' itself is defined by past hiring data — which is already biased. So you're essentially asking: do we want an AI that replicates the status quo perfectly, or one that tries to correct for historical injustice but might be less 'accurate' by that same flawed metric? Luna: That's a philosophical question as much as a technical one. And I imagine the answer changes depending on whether you're the candidate being filtered out or the HR director trying to hit hiring targets. Lucas: Right. And that's why transparency matters. The AI Act requires companies to document their bias testing methodology and results. Some startups are now offering 'model cards' — standardized reports that explain how an AI was trained, what data it used, and what bias tests were run. Luna: That sounds a lot like nutrition labels for AI. I could see that becoming an industry standard. Lucas: It's already starting. A few large employers have voluntarily published model cards for their screening tools. But the vast majority haven't. And the study we're discussing found that only about 12 percent of companies using AI in hiring had conducted any kind of bias audit in the past year. Luna: Twelve percent. So we're still in the early adopter phase when it comes to actually checking if these tools are fair. Lucas: Exactly. And that's a problem, because the adoption rate of AI in hiring is already over 70 percent among Fortune 500 companies. So we have a situation where the technology is widespread, but the oversight is minimal. Luna: One angle I want to push on — the study you mentioned used names like Emily Watson and Fatima Al-Rashid. But there are other dimensions of bias too. Gender, age, even names that sound 'too creative' or 'too ethnic' in different ways. Lucas: Absolutely. The same researchers ran a secondary test comparing names like 'Emily Watson' and 'Ebony Washington' — which is a name strongly associated with Black Americans — and found a 10 percent callback gap. So it's not just about non-Western names; it's about any name that signals a marginalized group. Luna: And I imagine the AI is learning these associations from the broader internet text it was trained on, not just the hiring data. So you have this compounding effect — societal bias in language, plus historical bias in hiring, plus statistical noise. Lucas: That's exactly right. And that's why merely 'cleaning the training data' is insufficient. You have to actively intervene in the model's learning process. The Princeton team actually showed that fine-tuning a pre-trained language model on a balanced dataset of resumes reduced the bias by about half — but the remaining half still requires the debiasing techniques we talked about. Luna: So it's not hopeless, but it's not a one-and-done fix either. This is going to require ongoing monitoring and adjustment. Lucas: Exactly. And that's where we need to be honest about costs. These audits and debiasing procedures aren't free. They require specialized expertise, and many companies — especially smaller ones — don't have that in-house. Luna: Which brings us back to the regulatory question. Should we mandate bias audits, or let the market sort it out? The evidence so far suggests the market isn't sorting it out very quickly. Lucas: The European approach says mandate it. The US approach has been more cautious, but with the EEOC guidance and the settlements, the legal pressure is building. I think in the next two to three years, we'll see some form of federal legislation — at least for hiring AI. Luna: That feels like a long time for the thousands of people who are currently being filtered out by these systems. But I also understand that regulation takes time. Lucas: It does. And in the meantime, there are things job seekers can do. Some researchers have shown that using a 'white-sounding' name on your resume increases callback rates — which is a terrible workaround, but it's a real one. There are also tools like 'Resume Worded' that let you test how your resume performs against AI screeners, though they don't fully address the bias issue. Luna: It's depressing that we're advising people to game the system rather than fixing the system itself. But I get it — individuals need to survive. Lucas: Yeah. And that's why I think the most important takeaway from this study is that awareness matters. The more we talk about how these systems actually work — and don't work — the harder it is for companies to hide behind the black box. Luna: And speaking of keeping conversations like this going — if today's tech conversation gave you something usable, something to think about at work or in your own job search, a couple of dollars a month is genuinely what keeps these episodes ad-free and accessible. Buy me a coffee dot com slash fexingo, if you've gotten something out of them. Lucas: Yeah, it makes a real difference. And it lets us keep digging into studies like this one without any sponsor constraints. So, back to the bias — there's one more finding I want to highlight. Luna: Please — because I think listeners might wonder: if I'm a candidate who's affected, what's the single most effective thing I can do? Lucas: That's a great question. The study actually tested one surprisingly effective intervention: adding a 'skills summary' section at the top of your resume that lists concrete, quantifiable achievements. That section — which the AI processes as high-signal text — partially overrides the name bias because it's so packed with relevant keywords. Luna: Interesting. So you're essentially giving the AI more good data to latch onto, drowning out the noise of the name. Lucas: Exactly. The gap for candidates with strong skills summaries dropped from 14 percent to about 5 percent. Still there, but much smaller. So there's a practical takeaway: if you're a job seeker, lead with measurable impact. But for the broader system, the solution has to be regulatory and technical — not putting the burden on the individual. Luna: Right. And that's the distinction that often gets lost in the conversation. It's not about blaming the AI or the candidate — it's about designing systems that are fair by default. Lucas: Well said. And that's where we'll leave it for today. Thanks, Luna. Luna: Thanks, Lucas. On to the next.