Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Hiring Tool Rejects You Before a Human Sees Your Resume
Transcript
- Lucas: So here's a number that stopped me this week: 72 percent of large employers in the US now use some form of AI to screen resumes before a human ever reads them. Luna: Seventy-two percent? That feels like one of those statistics where you think it's maybe a third, and then you realize it's the norm. Lucas: Right. And the source is a Harvard Business School study from late 2025, so it's fresh. They surveyed hiring managers at companies with over 5,000 employees. And the finding that really got me was less about the prevalence and more about the pattern: the systems are consistently downgrading candidates who have career gaps. Luna: Career gaps — meaning someone who took a year off to raise kids, or care for an aging parent, or recover from illness. Lucas: Exactly. The study focused on a specific Fortune 500 company — they didn't name it, but they got access to their internal hiring data. And the AI model, which was trained on the company's own past hiring decisions, learned that candidates with continuous employment were more likely to be hired. So it started penalizing any resume with a six-month or longer gap. The result: women aged 30 to 45, who are statistically more likely to take career breaks for caregiving, saw their resumes pushed to the bottom of the queue at a rate 38 percent higher than men in the same age bracket. Luna: So the AI is basically baking in the 'motherhood penalty' automatically, without anyone at the company having to consciously decide to do that. Lucas: Exactly. And the scary part is that the company's human recruiters weren't even seeing most of these rejected applications. The system was set to only surface the top 20 percent of candidates based on its score. Everyone else got a form rejection — or just silence. Luna: So the person never knows an AI made the call. They just think their resume didn't land. Lucas: That's the transparency problem. And it's not just career gaps. Another example from the study: the algorithm also penalized candidates who listed non-profit or volunteer experience instead of paid corporate roles. The model saw those as less relevant, even when the skills were directly transferable. Luna: So someone who ran a fundraising campaign for a charity — that's project management, budgeting, stakeholder communication — but the AI doesn't know how to value that if the title doesn't say 'manager'. Lucas: Right. And here's where the ethics get really knotty. The company in the study was using a third-party vendor's AI screening tool. They didn't build it themselves. They bought a system marketed as 'bias-free' and 'merit-based.' But the vendor trained their model on a dataset of resumes from a different set of companies — mostly tech firms in coastal cities. So the model was optimized for a certain kind of candidate profile: continuous employment, for-profit experience, no gaps. Then the vendor sold it to a manufacturing company in the Midwest with a very different workforce. Luna: So the training data itself carries a bias that the buyer doesn't even see. It's like buying a pre-screened filter that already has someone else's assumptions baked in. Lucas: And the vendors don't always disclose what's in their training data. There's no regulatory requirement to do an audit. The Equal Employment Opportunity Commission — the EEOC — issued guidance in 2023 saying that AI hiring tools can violate anti-discrimination laws, but it's non-binding. No enforcement mechanism. So the burden is on employers to figure out if their vendor's model is biased. Luna: Which most of them don't have the expertise or the incentive to do, because the whole point of buying the tool is to save time and money. Lucas: Right. And the Harvard study found that only 12 percent of companies using AI screening had conducted any kind of bias audit in the previous year. So we have a system where the most vulnerable applicants — people with non-linear careers, caregivers, career-changers — are being silently filtered out, and no one is checking. Luna: It reminds me of the earlier episodes we did on credit scoring and loan decisions. The same pattern: an opaque algorithm makes a decision with real consequences, and the person affected has no recourse because they don't even know why. Lucas: Which brings me to a point I wanted to make about the whole 'human-in-the-loop' idea. A lot of companies say, 'We have a human review the AI's recommendations before making a final decision.' But the Harvard study showed that when humans are presented with an AI score, they tend to defer to it — especially if they're busy or the volume is high. They looked at one company where the human recruiters were supposed to override the AI if they saw something the system missed. In practice, they overrode it less than 4 percent of the time. Luna: So the human loop is basically a rubber stamp. The AI decides, the human clicks approve. Lucas: And the recruiters in that company told researchers that they trusted the AI because it was 'data-driven' — the same kind of automation bias we see in medical diagnosis and criminal sentencing. If a machine gives you a number, people assume it's objective. Luna: But it's only objective if the data and the model are fair, which in this case they weren't. Lucas: And that's the core of it. Now, I want to take a quick step back here — because conversations like this are exactly why we do this show without ads. Luna: Yeah, it's a deliberate choice. We don't have sponsors or commercial breaks, and that means we can follow the story wherever it goes without worrying about upsetting an advertiser. Lucas: Exactly. And if you find that valuable — if this kind of deep dive into how algorithms actually affect people's lives is something you want to see more of — the way to keep it happening is listener support. There's a page at buy me a coffee dot com slash fexingo where people can chip in. No tiers, no perks, just a way to say 'keep going.' Luna: It's a small thing, but it adds up. And it means we never have to sell your attention to anyone. Lucas: Right. So, back to the hiring screen. One of the most interesting proposals I've seen comes from a group called the Algorithmic Justice League. They're advocating for what they call a 'resume receipt' — basically, if an employer uses AI to screen candidates, they have to tell you. And if you're rejected, you have the right to request an explanation of what factors the AI considered. Luna: That's a transparency mandate similar to what the EU's AI Act is trying to do for high-risk systems. But in the US, there's no federal law like that. Lucas: Not yet. But there are state-level efforts. Illinois passed a law in 2024 requiring companies that use AI in hiring to notify applicants and get their consent. New York City has a law that requires bias audits for automated hiring tools. But enforcement is weak — the NYC law only applies to companies that already have a certain number of employees in the city, and it doesn't require public disclosure of the audit results. Luna: So it's a patchwork. And if you're a job seeker in Texas or Florida, you probably have no idea if an AI is rejecting you. Lucas: And that's the practical takeaway for our listeners: if you've been applying to jobs and hearing nothing back — especially if you have a gap on your resume or you've worked in non-traditional roles — know that an algorithm may be the reason. And there are things you can do. Luna: Like what? Because telling someone to 'optimize for the AI' feels like playing a rigged game. Lucas: It is a rigged game. But there are strategies. One is to include keywords from the job description directly in your resume — the AI is often doing keyword matching, so if the posting says 'project management' and you say 'managed projects,' that helps. Another is to avoid PDFs that the parser can't read — use plain text or a standard Word format. Some experts also recommend explicitly addressing a career gap in a cover letter, because the AI might not parse the cover letter, but if a human ever does see it, the context is there. Luna: It's depressing that the burden falls on the job seeker to reverse-engineer a biased system. Lucas: Absolutely. And that's why the structural fix has to come from regulation and corporate accountability. But until that happens, the least we can do is help people understand how the game works. Luna: So to wrap up — the key number from today is 72 percent. And the key question: if you're applying for a job and you never hear back, should you have the right to know whether a machine made that decision? Lucas: I think the answer is yes. And I think the more people ask that question, the harder it becomes for companies to hide behind the algorithm.