Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Recruiter Has a Gender Bias Problem
Transcript
- Lucas: There's a number that's been sticking with me since I read the EU Fundamental Rights Agency's 2024 report on AI hiring tools. They tested a popular recruiting algorithm against two sets of resumes — identical qualifications, but one set included activity from a women's college and used more female-coded language, like 'collaborative' or 'nurtured,' while the other used male-coded terms like 'competitive' and 'led.' Luna: What was the difference in callback rates? Lucas: Twelve percent. The female-coded resumes got 12 percent fewer callbacks. That's not a bug — that's a pattern the model learned from historical hiring data where men were more likely to get hired for that role. The algorithm essentially memorized the gender imbalance and automated it. Luna: And this is the kind of system companies are buying off the shelf. It's not some sketchy startup — this was a widely used platform. Lucas: Exactly. And the scary part is, many companies think they've solved this by simply removing gender from the data — no name, no pronouns, no gender markers. That's called 'fairness through unawareness,' and it doesn't work. Luna: Because language itself is gendered. You can strip out explicit gender fields, but the words people use — or the schools they attended — are proxies. The model still figures it out. Lucas: Right. The EU study found that even after removing gender indicators, the algorithm still penalized women at a statistically significant rate. It picked up on correlations like 'member of the National Organization for Women' or 'attended Smith College' and treated those as negative signals. Luna: So what's the fix? Don't just say 'remove gender' — that clearly fails. Lucas: One approach is something called 'equal opportunity auditing.' You don't just test the model on a held-out set — you deliberately construct counterfactual pairs: two candidates who are identical except for a protected characteristic, and you see if the model treats them differently. Then you retrain or adjust the decision threshold. Luna: But that requires companies to actually care enough to do the audit. And to know what they're looking for. Lucas: Which is why regulation is starting to catch up. New York City's Local Law 144, which took effect in 2023, requires any company using AI for hiring in New York to conduct an annual bias audit and publish the results. It's not perfect — the audits can be gamed — but it's a start. Luna: And there's a precedent for this failing at massive scale. Amazon scrapped its own AI recruiting tool in 2018 after discovering it penalized resumes that included the word 'women's' — as in 'women's chess club captain.' The model learned to downgrade candidates from women's colleges. Lucas: That's the canonical example. Amazon tried to build a system that would rank candidates based on ten years of hiring data. The problem was that ten years of data reflected a male-dominated engineering culture. So the model taught itself that male-coded patterns were better. They couldn't fix it, so they killed the project. Luna: And yet, here we are in 2026, and similar biases are still showing up in vendor products. So what's the disconnect? Lucas: Part of it is that bias is not a one-time fix. It's not like you clean the data, run an audit, and you're done. The world changes — job descriptions change, applicant pools change. A model that was fair two years ago might drift. You need continuous monitoring. Luna: But continuous monitoring costs money and expertise. Most HR departments don't have a machine learning engineer on staff. They're buying a SaaS product that promises 'ai powered recruiting' and assuming it's neutral. Lucas: That's the core problem: the assumption of neutrality. Algorithms are not objective — they're mirrors of the data they're trained on. If your historical hires were mostly white men from certain schools, the model will encode that as the ideal. Luna: And this isn't just about gender. It's race, age, disability — any pattern that correlates with success in the training data can be amplified. Lucas: Right. The EU report also found that AI hiring tools in some sectors penalized candidates with non-European-sounding names, even when qualifications were identical. The bias is intersectional. Luna: So what's the practical takeaway for a company that wants to use AI in hiring but do it responsibly? Where should they start? Lucas: First, demand transparency from your vendor. Ask: what data was the model trained on? What bias audits have been run? Can you provide a third-party audit report? If they can't answer those questions, that's a red flag. Luna: Second, I'd say run your own internal audit. Even if the vendor says they've tested it, the model might behave differently on your applicant pool than on the vendor's test data. Lucas: And third, consider using the AI as a screening tool, not a decision-maker. Let it flag candidates for human review, but don't let it reject people automatically, especially for junior roles where the signal in the data is weaker. Luna: That's a good principle: keep a human in the loop. But the human has to be aware of their own biases too. If the recruiter is biased, they might just rubber-stamp the AI's decision. Lucas: Absolutely. The human-in-the-loop only works if the human is trained to challenge the model, not defer to it. There's research showing that people tend to over-rely on algorithmic recommendations, especially when the algorithm is presented as 'AI' or 'machine learning.' Luna: Automation bias. We assume the machine is smarter. Lucas: Exactly. So the solution isn't just technical — it's organizational. You need a culture that encourages critical thinking about the tools you use. Luna: Speaking of tools that help us think critically, if today's episode gave you something useful — maybe a new way to think about hiring or a concrete action to take — that's the kind of thing listener support makes possible. We keep this show ad-free because of people who find value here and want to share it. Lucas: Yeah, it's a small gesture that goes a long way. If today was worth a coffee to you, that link is buy me a coffee dot com slash fexingo. No pressure, just a way to keep the conversation going. Luna: And we really do appreciate it. It lets us spend time digging into reports like that EU study instead of chasing ad revenue. Lucas: Alright, back to the practical side. Let's say you're a mid-size company and you want to adopt AI recruiting tools responsibly. One model program I've seen is from a tech firm called Buffer — they published their entire hiring algorithm's logic and audit results online. That's rare, but it's a benchmark. Luna: Transparency as a differentiator. If you're proud of your process, you should be willing to show it. Lucas: And there's a business case for it too. Companies with more diverse workforces tend to outperform their peers on innovation and profitability. If your AI is filtering out diverse candidates, you're hurting your own bottom line. Luna: So bias isn't just an ethics problem — it's a talent problem. You're missing out on great people because your algorithm learned from a flawed past. Lucas: That's the argument that gets CFOs to pay attention. You can talk about fairness all day, but when you frame it as 'your model is systematically excluding 50 percent of the talent pool,' that lands differently. Luna: And it's not just about the hiring stage. Bias can creep in at every step — from job ad targeting to resume parsing to interview scheduling to assessment scoring. You have to audit the whole pipeline. Lucas: Right. One thing the EU study highlighted is that many companies only audit the final ranking model, not the earlier stages. But if your job ad is shown primarily to men because the platform's algorithm optimizes for engagement, you've already narrowed the pool before the model even sees a resume. Luna: So the solution requires a holistic view of the hiring system, not just the AI component. Lucas: Exactly. And that's where regulation might push things. The EU's AI Act, which is supposed to come into full effect in the next couple of years, classifies hiring AI as 'high risk.' That means stricter requirements for data quality, transparency, and human oversight. Luna: But regulation is slow, and technology moves fast. By the time the rules are in place, the models have changed. Lucas: That's the eternal tension. But I think the best defense is a combination of smart regulation, third-party audits, and internal accountability. Companies shouldn't wait to be told to do the right thing. Luna: And for job seekers who suspect they're being filtered out by a biased algorithm, what can they do? Lucas: It's hard because the algorithm is a black box. But some advocates suggest tailoring your resume to include more 'power words' — action verbs and quantifiable achievements — that tend to score well. It's gameable, but it's a short-term workaround. Luna: Long term, we need more transparency and accountability. And that starts with people asking the right questions. Lucas: Exactly. Every time a company says 'our AI eliminates bias,' you should ask: whose data was it trained on? What was the historical hiring rate for women and minorities? What audits have been done? If they can't answer, the AI might be making things worse, not better. Luna: And that's the note to end on: skepticism is healthy. Don't trust the AI — test it. Lucas: Well said. That's all for this episode of AI Ethics with Fexingo. We'll be back next week with another real-world look at responsible AI.