Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Recruiter Rejects Based on Your Commute
Transcript
- Lucas: So here's something that came across my desk this morning: a 2025 Stanford study found that some AI recruitment tools are penalizing candidates based on their estimated commute distance to the office. Luna: Wait — commute distance? As in, how far they live from the job site? Lucas: Exactly. The AI was trained on historical hiring data, and it learned that people who lived farther away tended to quit sooner or be less reliable. So it started weighting commute distance as a negative signal — and it disproportionately flagged applicants from lower-income neighborhoods, where long commutes are often a necessity, not a choice. Luna: That's a textbook example of a proxy for socioeconomic status. The system probably didn't have direct access to income data, but it found a correlated variable. Lucas: Right. And this is the insidious part: the company deploying the AI might not even know it's happening. The model is a black box — it outputs a score, but the reasons are opaque unless you specifically audit for proxy discrimination. Luna: Did the study name names? Which companies or platforms? Lucas: It kept vendors anonymous — they tested four commercial recruitment AIs used by Fortune 500 companies. Three of the four showed statistically significant commute bias. One vendor's model was so sensitive to it that a candidate living 30 minutes away was ranked 15 percentiles lower than an identical candidate living 5 minutes away. Luna: Fifteen percentiles. That's enormous. And the candidate with the longer commute could be more qualified across the board — but the AI never even surfaces their resume to a human. Lucas: Exactly. The researchers then showed that simply removing the commute variable from the model didn't fix it — because the AI learned proxy features. Zip code, public transit access, even the time zone of the applicant's IP address. It reconstructed commute distance from other data. Luna: So the bias is baked into the correlations in the training data. You can't just delete a column and call it fair. Lucas: Right. And this connects to a bigger pattern we've seen in lending and insurance AI. In 2023, a major auto insurer was fined for using credit score as a proxy for race — even though they claimed to be race-blind. Same dynamic: a legitimate business metric that disproportionately harms protected groups. Luna: So what's the fix? The EEOC put out guidance in 2024 on algorithmic fairness — does that cover this? Lucas: It does. The EEOC's guidance says employers are responsible for disparate impact even if the AI vendor built the tool. They recommend independent audits that test for adverse impact across the entire pipeline — not just the final score, but the intermediate features. But here's the problem: very few companies do that kind of deep audit. It's expensive, it requires access to the model internals, and most vendors don't want to share. Luna: And if the vendor won't open the black box, the employer is still on the hook legally. Lucas: Exactly. That's the tension. The Stanford researchers proposed a solution called 'counterfactual auditing' — you generate synthetic applicants that differ only on the protected proxy, like commute distance, and you check if the scores change systematically. But that requires the auditor to know which proxies to test. Luna: Which means you need domain expertise. You can't just hand a dataset to a generic data science team and expect them to catch all the subtle ways bias creeps in. Lucas: Right. And this is why I think the most promising approach isn't just technical — it's regulatory. The EU's AI Act, which goes into full effect next year, classifies hiring AI as 'high risk' and requires transparency documentation and human oversight. The US doesn't have anything equivalent yet, but there are bills in Congress. Luna: Speaking of human oversight — what about the human recruiters? Don't they catch these biases before making final decisions? Lucas: That's the hopeful scenario. But in practice, the Stanford study showed that when human reviewers saw the AI scores, they were heavily influenced by them — anchoring bias. If the AI ranked a candidate low, the human rarely overrode it. And if they did, they usually only adjusted by a few points. Luna: So the human becomes a rubber stamp. That defeats the purpose of oversight. Lucas: Exactly. One of the study's recommendations was to blind the human reviewers to the AI score until after they make their own assessment. That way the human's judgment isn't contaminated. Luna: That's elegant. But it also assumes the human is trained to recognize their own biases. Lucas: Which is a whole other can of worms. But I think the commute example is a really useful lens — it's concrete, it's measurable, and it shows how a seemingly neutral variable can become a discrimination vector. And it's not just hiring. Same logic applies to loan approvals, insurance pricing, even university admissions. Luna: You know, this reminds me of something I read about a health insurance algorithm that used 'medication adherence' as a proxy for health outcomes, but it penalized people with chronic conditions who couldn't afford their meds. Lucas: That's the same pattern — a variable that looks objective but is actually correlated with socioeconomic factors. The difference is that in healthcare, there's more regulatory scrutiny because of HIPAA and the Affordable Care Act. But in hiring, the regulatory framework is still catching up. Luna: Let's talk about what companies can actually do right now, without waiting for legislation. What's a practical first step for a mid-size company using an AI recruiting tool? Lucas: First, demand transparency from your vendor. Ask them for a fairness audit report that specifically tests for proxy discrimination — not just overall accuracy. Second, run your own disaggregated analysis: break down the AI's pass rates by zip code, by commute time bucket, by any variable that could be a proxy. If you see a steep gradient, investigate. Luna: And third, have a human-in-the-loop who's empowered to override the AI, with clear criteria for when they should do so. Lucas: Exactly. And document everything. Because if you get sued, the EEOC will ask for that documentation. We've seen cases where companies couldn't produce any audit trail, and they settled for millions. Luna: It's wild that we're still in a phase where the burden is on employers to prove their AI isn't biased, rather than on vendors to prove it is fair before selling it. Lucas: Yeah, that's the current state of play. And it's why episodes like this matter — the more people understand these dynamics, the more pressure there is for change. Luna: Speaking of pressure for change — we deliberately keep this podcast ad-free. If that kind of independent, deep-dive conversation is valuable to you, and you want to support the work, there's a link at buy me a coffee dot com slash fexingo. Lucas: Yeah. No ads, no sponsors, no pressure to sell anything. Just us trying to make sense of a fast-moving area. Every contribution helps keep the lights on. Luna: Back to the commute bias — one thing I want to come back to: the idea of counterfactual audits. Are there tools or open-source libraries that companies can use to run those themselves? Lucas: Yes, there are. IBM's AI Fairness 360 toolkit has a module for generating counterfactuals. Microsoft's Fairlearn also supports some of this. But they require significant technical expertise to set up correctly. The Stanford team actually open-sourced their audit methodology, so that's a good starting point. Luna: So there's no excuse for not auditing, at least for large companies with data science resources. Lucas: Exactly. And for smaller companies, third-party auditors are starting to emerge — firms that specialize in algorithmic fairness. The cost is coming down as the market matures. Luna: Good. Because the alternative — waiting for a lawsuit or a regulatory fine — is both expensive and damaging to trust. Lucas: And trust is the one thing AI vendors can't afford to lose. If every hiring tool is seen as biased, companies will go back to manual processes, and we'll lose the efficiency gains that AI can bring when it's done right. Luna: So the goal isn't to avoid AI — it's to build AI that's genuinely fair, not just fair on paper. Lucas: Exactly. And that starts with understanding where bias hides — sometimes in something as mundane as how far you live from work.