Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / AI Systems That Automatically Reject Your Job Application
Transcript
- Lucas: So you spend two hours tailoring your résumé for a job you really want. You hit submit. And within thirty seconds, a piece of software has already decided you're not getting an interview. No human ever reads it. Luna: That's not a hypothetical — that's the reality for the vast majority of job applications today. The technology is called an applicant tracking system, or ATS, and it's the gatekeeper for something like 75 percent of all large-company job postings. Lucas: Right. And these systems are not neutral. They're trained on historical hiring data, which means they absorb whatever biases existed in the company's past decisions. One of the most famous examples is Amazon's 2015 automated hiring tool. The company trained a machine learning model on ten years of résumés submitted to the company — which were overwhelmingly from men, because tech was male-dominated. So the model learned to penalize résumés that contained the word 'women's,' as in 'women's chess club captain' or 'women's leadership program.' Luna: Amazon eventually scrapped the tool, but the underlying problem hasn't gone away. ATS software is still making similar kinds of calls every day. And the bias isn't always about gender — it can be about age, about gaps in employment, about the font you used. Lucas: Let's talk about the age bias specifically. Older workers often have longer résumés, and many ATS systems are configured to penalize anything past a certain length — say, two pages. So a senior professional with twenty years of relevant experience gets truncated or scored lower. There's also a subtler signal: the ATS can date-stamp your graduation year, and if it's more than fifteen years ago, the system might downgrade your application. That's age discrimination, but it's hidden behind a 'process optimization' label. Luna: And the irony is that the companies buying these ATS products often think they're reducing bias by removing human subjectivity. They believe an algorithm is more objective. But the algorithm is only as fair as the data it's trained on. Lucas: Exactly. The ATS doesn't know what a 'good' résumé looks like in any absolute sense — it just learns patterns from past hires. If your company historically hired from Stanford and MIT, the ATS will prioritize those schools. If your past hires all had 'communication' as a keyword in their résumés, the ATS will weight that keyword heavily. It's not evaluating merit — it's mimicking past preferences. Luna: There's a well-known case from Unilever, the consumer goods giant. In 2016, they started using AI to screen entry-level candidates. The system analyzed video interviews for facial expressions and tone of voice, and it also scored résumés. Unilever said it saved them 50,000 hours of recruiter time and made hiring more diverse. But critics pointed out that the system could easily pick up on race and class markers — a candidate's accent, their background, even the lighting in their video setup. Lucas: Right, and Unilever eventually backed away from the video analysis component after public pressure. But the résumé screening part is still in widespread use. And it's not just Unilever — companies like Hilton, Goldman Sachs, and Siemens have all used some form of automated screening. The market for ATS software is worth roughly one point five billion dollars, and it's growing fast. Luna: So what can a job seeker actually do? There's a whole industry of advice on 'beating the ATS' — formatting your résumé in a single column, using standard section headers, including keywords from the job description verbatim. Lucas: And that advice works — up to a point. But it also creates an arms race. Candidates stuff their résumés with keywords, the ATS gets updated to penalize keyword stuffing, so candidates get more creative. But the deeper issue is that these systems are not transparent. Most employers don't tell you they're using an ATS, and they certainly don't tell you the criteria the system is using to rank you. New York City's Local Law 144, which took effect in 2023, tried to change that by requiring employers to conduct bias audits of their automated hiring tools. But compliance has been spotty. Luna: I want to step back and ask a more fundamental question: should résumés be screened by AI at all? There's an argument that the sheer volume of applications — some companies get thousands per opening — makes human screening impractical. Lucas: I think the honest answer is that it depends on the stakes. For a seasonal retail job where the main qualification is availability, an automated screen is probably fine. But for a role where you're evaluating judgment, creativity, or leadership potential? No algorithm can assess those things from a piece of paper. And the risk is that we optimize for what's measurable — keyword matches, years of experience, specific degrees — and miss the candidate who would have been the best hire. Luna: There's also a class dimension. Candidates from privileged backgrounds are more likely to know how to tailor a résumé for an ATS. They have access to career coaches, resume-review services, and networks that tell them the unwritten rules. So the ATS can amplify existing inequality. Lucas: Let me give you a concrete number. A study by the National Bureau of Economic Research found that when identical résumés were submitted with white-sounding names versus Black-sounding names, the white-sounding names got 50 percent more callbacks. That's with human screeners. An ATS trained on those same human decisions would learn that same bias — and then apply it at scale, across thousands of applications, every single day. Luna: So the mitigation isn't just about better data — it's about designing the system differently from the start. Some companies are experimenting with 'blind' ATS systems that strip out name, age, gender, and school. But even those can leak demographic information through other signals, like hobbies or the format of the résumé. Lucas: One approach that shows promise is to use the ATS not as a filter but as a matching tool — presenting recruiters with a ranked list of candidates alongside the reasons for each ranking, rather than automatically rejecting anyone below a threshold. That way, a human can override the algorithm. But that only works if recruiters are actually trained to look for their own biases. Luna: And that brings us back to regulation. The EU's AI Act classifies employment-related AI as 'high-risk,' which means companies using these systems have to conduct conformity assessments, ensure human oversight, and provide transparency to candidates. But the AI Act won't be fully enforceable until 2027. In the meantime, the patchwork of local laws — like New York's and a similar one in Illinois — is all we have. Lucas: I think one thing that would help is if more companies adopted a 'rejection explanation' requirement. If an ATS rejects your application, you should be able to request a summary of the factors that led to that decision. Not the full algorithm, obviously — that's proprietary — but something like 'your résumé scored low on years of experience in project management.' That would at least give job seekers actionable feedback and create accountability. Luna: That's a practical idea. And it's similar to what the GDPR's right to explanation was supposed to provide, though enforcement has been weak. But even without regulation, companies that voluntarily offer that transparency could differentiate themselves and attract applicants who are wary of automated screening. Lucas: You know, something that struck me while researching this episode is that the people building these ATS systems often have no background in ethics or fairness. They're software engineers who are told to optimize for 'efficiency' — reduce time to hire, increase number of screened applicants, minimize recruiter workload. And without explicit instruction to optimize for fairness, that variable simply doesn't get weighted. Luna: So it's a system-design problem as much as a data problem. And that's something we, as listeners, can keep in mind. If you're in tech, or if you're ever involved in procuring HR software, you can ask the vendor: 'How did you test for bias? What's your false-positive rate for rejecting qualified candidates? Is there human review?' Lucas: Yeah, exactly. And if today's conversation gave you something usable — a question to ask, a perspective to share — I'll just mention that the show stays ad-free and independent because of listeners who choose to support it. If that's something you'd like to do, you can find us at buy me a coffee dot com slash fexingo. No pressure, genuinely. We're just glad you're here. Luna: And on that note, I think the key takeaway is that résumé screening AI isn't inherently bad — but it's being deployed without enough safeguards. The same technology that can efficiently route a strong candidate to the top of the pile can also systematically exclude entire groups of qualified people. The difference is in how we design, audit, and oversee it. Lucas: So the next time you hit 'submit' on a job application and get a rejection email two minutes later, remember: it might not have been a person who made that call. And that's a problem we all have a stake in solving.