Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / How AI Models Are Learning Bias From Your Job Descriptions
Transcript
- Lucas: So there's a 2025 study from the AI Now Institute that looked at how AI hiring models actually learn bias — not from some rogue data set, but from something as mundane as job descriptions. Luna: Job descriptions? That's — I mean, we've talked about biased data before, but I thought the problem was usually in historical hiring decisions or flawed test scores. Lucas: Right, that's the conventional story. But what this study did was scrape tens of thousands of real job postings from Fortune 500 companies — positions in tech, finance, healthcare — and then fed them into a standard large language model that companies use to screen resumes. Lucas: And what they found was that the model, after training on those descriptions, started penalizing candidates who didn't have a traditional four-year degree from a name-brand university — even when the job didn't explicitly require one. It learned that preference from the way the descriptions were worded. Luna: So the bias isn't just in the data about past hires — it's baked into the language that companies themselves use to advertise the role. That's a whole new layer. Lucas: Exactly. The researchers used a technique called causal mediation analysis to trace which parts of the description drove the bias. And it wasn't just degree mentions. It was subtle stuff — phrases like 'fast-paced environment' or 'self-starter' that were far more common in descriptions at firms that historically hired fewer women and minorities. Luna: So the model picks up on those linguistic patterns and then — what, weights candidates who use similar language in their resumes more highly? Lucas: Precisely. And it goes further. The model also learned to associate 'gap in resume' with lower scores, not because gaps are explicitly mentioned in the description, but because the descriptions implicitly assumed continuous employment. The model basically reverse-engineered a stereotype about what a 'good' candidate looks like. Luna: That's insidious. And it's happening in real time, every day, at companies that probably think they're being objective by using an AI screener. Lucas: Yeah. And the scale is massive. According to the study, about seventy-five percent of large US employers now use some form of AI in hiring — whether it's resume screening, video interview analysis, or skills assessments. So the impact is enormous. Luna: Have any companies actually tried to fix this? Or is it just a problem that everyone's aware of but nobody's solved? Lucas: A few are trying. IBM, for instance, has a tool called AI Fairness 360 that lets companies audit their models for bias on multiple dimensions — race, gender, age. Unilever has been more transparent about their own hiring algorithm, publishing audit results annually. But the study found that most companies aren't doing any auditing at all. Luna: And the ones that do audit — they're usually checking the final output, not the training data itself. Which means the bias in the job descriptions never gets caught. Lucas: Right. And that's the key insight from the AI Now study — they showed that even if you have a perfectly unbiased historical hiring record, if your job descriptions carry subtle linguistic biases, your model will learn them. It's a data quality problem at the input stage. Luna: So what's the fix? Do companies need to rewrite every job description from scratch? That sounds like a massive undertaking. Lucas: You know, speaking of massive undertakings — and this is a quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many of these. It's listener-supported, no ads, and we try to dig into things like this because people actually care about the details. Luna: Yeah, it's a small group, but they make it possible to keep the show ad-free and independent. So thank you to anyone who does that. Lucas: Anyway, back to the fix. The study actually proposes something fairly pragmatic: they suggest companies run their job descriptions through a bias-detection model before posting them. There are startups — like Textio and Applied — that offer this service. They flag language that correlates with biased outcomes. Luna: So it's like a spellchecker, but for bias. That seems doable. Lucas: It is. And some companies are already doing it. But the researchers found that only about twelve percent of the firms they studied used any form of pre-screening on their job descriptions. So there's a huge gap between what's possible and what's actually happening. Luna: Is there any regulatory pressure to close that gap? I know the EEOC has been looking into AI hiring tools, but I'm not sure how far they've gotten. Lucas: The Equal Employment Opportunity Commission issued guidance back in 2023 saying that employers are responsible for the outcomes of their AI tools, even if a vendor built them. But enforcement has been sparse. There was a case last year where a large retailer was found to have used an AI screener that disproportionately screened out older workers — the EEOC settled for about sixty million dollars. Luna: So there are consequences, but it's reactive — after the damage is done. Lucas: Exactly. The AI Now study argues for proactive regulation — requiring companies to audit their training data and their outputs before deployment. Some states are moving in that direction. New York City already has a law that requires bias audits for automated hiring tools. But it's only for certain industries, and enforcement is still ramping up. Luna: And in the meantime, millions of job applications are being processed by models that are learning bias from job descriptions written by humans who may not even realize they're doing it. Lucas: Right. And that's the scary part — it's not malicious. It's structural. A hiring manager writes 'must be able to hit the ground running' because that's what they've always written. They don't realize that phrase, statistically, correlates with a preference for candidates who haven't taken career breaks — which disproportionately affects women and caregivers. Luna: So the solution isn't just better AI — it's also better writing. And better awareness. Lucas: Yes. And the study offers a concrete number: if companies ran their job descriptions through a simple bias-detection tool before posting, they could reduce unwanted demographic skew in their applicant pool by about thirty percent — without changing who they ultimately hire. That's a huge potential impact for relatively low effort. Luna: Thirty percent — that's significant. I wonder if the companies that already use those tools see a better quality of hire, too. Lucas: Some early evidence suggests yes. A 2024 paper from the National Bureau of Economic Research looked at a tech firm that switched to an audited hiring system and found that the diversity of shortlisted candidates increased, and the performance ratings of hired candidates didn't drop — in fact, they went up slightly. So it's not a trade-off between fairness and quality. Luna: That's the kind of data point that might actually convince skeptical executives: this isn't just about being fair — it's about being better at hiring. Lucas: Exactly. And that's the argument the researchers are pushing. They're not saying 'be nice.' They're saying: your training data — your job descriptions — contain bias that makes your model less accurate at identifying the best candidates. Fix the data, and you improve both fairness and performance. Luna: It reframes the conversation from ethics to economics. Which might be what it takes to get companies to act. Lucas: Yeah. Look, we've talked before about AI in hiring — about landlords, about insurance. But this episode is about the hidden layer: the text we write that the model learns from. And the encouraging thing is, we have control over that text. We can change it. Luna: But only if we know what to look for. That's the hard part — most managers don't think 'I need to audit my job posting language' when they're trying to fill a role. Lucas: Right. And that's where the regulation or internal policy comes in. Make it a standard step: before a job description goes live, it goes through a bias check. Just like you'd check for spelling or legal compliance. Luna: So we have a clear action item for any company listening: audit your job descriptions. Or if you're a job seeker, maybe be aware that the language you use in your resume might be working against you if it doesn't match the hidden patterns in the description. Lucas: And that's a good point — it cuts both ways. The model is trying to match patterns. So if you know the company uses an AI screener, tailoring your resume to mirror the language in their job posting could help. But that's a workaround, not a solution. Luna: A temporary one, at best. Long term, we need systems that don't penalize people for having a different path. Lucas: Yeah. And I think that's where we'll leave it. The AI Now study is a good reminder that bias in AI isn't always about 'bad data' — sometimes it's about the everyday language we use, without thinking, that gets amplified by the machine. Luna: And amplified at scale. Which is exactly why it matters.