Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / How AI Models Are Learning Gender Bias From Performance Reviews
Transcript
- Lucas: So there's this dataset that I think about a lot — 1.5 million performance reviews from Fortune 500 companies, collected by researchers at Stanford and Anthropic last year. Luna: 1.5 million — that's enormous. What were they looking for? Lucas: They were running natural language processing on the language used to describe employees. And what they found — depressingly, but not surprisingly — is that certain adjectives are applied very differently to men and women. Luna: We've heard this before in hiring, but in performance reviews it feels more insidious. This is feedback that shapes someone's career trajectory. Lucas: Exactly. The study found that words like 'aggressive,' 'confident,' 'decisive' were used about twice as often for male employees. Words like 'helpful,' 'collaborative,' 'supportive' — skewed heavily female. And here's the thing: this isn't just a problem for the people being reviewed. That language gets scraped and fed into AI models. Luna: Right, because companies are now feeding decades of performance reviews into large language models to train internal tools — like automated resume screeners, or AI that helps write new reviews. Lucas: Precisely. So you get a feedback loop. The model learns that men are 'strategic' and women are 'detail-oriented,' and then when it's asked to recommend someone for a promotion, it reproduces that pattern. One researcher I spoke to called it 'bias laundering' — the model just inherits and amplifies whatever patterns are in the data, regardless of whether they're fair. Luna: And it's worse because the model doesn't flag it. A human reading a review might catch a pattern over time, but the model just sees predictive features. Lucas: Right. The study also looked at word embeddings — the mathematical relationships between words that models learn. They found that the vector for 'male' was closer to 'leadership' and 'vision,' while 'female' was closer to 'administrative' and 'support.' These aren't just stereotypes — they're encoded into the model's geometry. Luna: So when a model generates a performance review for a woman, it might default to 'she is a great team player' rather than 'she drove revenue growth.' Lucas: Exactly. And there's a real-world consequence. Another study — this one from 2024 by researchers at MIT and the University of Chicago — looked at promotion rates at a large tech company after they introduced an AI tool to screen candidates for internal roles. They found that women were 14 percent less likely to be recommended for promotion than their performance scores would suggest. Luna: Fourteen percent — that's not a rounding error. That's a systemic drag on women's careers. Lucas: And the company had no idea. They thought the AI was being objective. But because it was trained on historical data where women had been reviewed less favorably, it just replicated the bias. Luna: That reminds me of something I read about Amazon's now-defunct AI recruiting tool from a few years back. It penalized resumes that included the word 'women's' — like 'women's chess club captain' — because historically, the company had hired more men. Lucas: Same mechanism, different context. The performance review bias is actually harder to fix, because the data is more nuanced. With resumes, you can filter out gendered language. But performance reviews are full of subjective language that's hard to disentangle from actual performance. Luna: So what do companies do? Just stop using AI for HR? Lucas: Not necessarily. But the researchers from the Stanford study recommend a few things. First, audit your training data. If you're going to train a model on performance reviews, you need to measure the distribution of descriptive terms across gender, race, and other protected attributes. If you find imbalances, you can rebalance the dataset — oversample reviews for women that use 'strategic' and 'visionary' language, for instance. Luna: But doesn't that risk creating a new kind of bias? Forcing language that wasn't naturally there? Lucas: It's a trade-off. But the argument is that the natural data is already biased, so doing nothing is also a choice. The other approach is to use debiasing techniques on the model itself — like removing gender-specific word embeddings after training. Luna: And does that actually work? I've seen papers where debiasing reduces accuracy. Lucas: It can. But the researchers I talked to said the accuracy drop is often small — one to two percent — and worth it if it means fairer outcomes. And some companies are starting to take this seriously. Microsoft, for example, has published guidelines for using AI in HR that explicitly call for bias testing. Google has a toolkit called the What-If Tool that lets you test model performance across different subgroups. Luna: Still, I wonder how many companies actually do this. It's expensive, it requires expertise, and most companies just want a tool that works out of the box. Lucas: That's the scary part. The market for ai powered HR tools is growing rapidly — it was estimated at over two billion dollars last year. And many of those products are built on general-purpose language models that have never been audited for performance review bias. They're just fine-tuned on whatever data the vendor has. Luna: If today's conversation gave you something to think about — maybe a reason to look at how your own company uses AI for reviews — that's the kind of thing we hope for. If it was worth a coffee to you, you can find us at buy me a coffee dot com slash fexingo. It's a small way to keep this ad-free. Lucas: Yeah, listener support is what lets us dig into these studies instead of just reading headlines. Appreciate anyone who does. Luna: Alright, back to it. So Lucas, what about the companies that develop the AI — the Anthropics, the OpenAIs — shouldn't they be filtering this out before releasing a model? Lucas: They should, and some do. Anthropic, of course, was co-author of the performance review study, so they're very aware. But the issue is that a general-purpose model doesn't know what it's going to be used for. A model trained on the entire internet will have absorbed billions of performance reviews, job descriptions, and biased text from all over. Luna: So the bias is baked in at the foundation level. Lucas: Exactly. And once it's in the foundation model, even if you fine-tune it for HR, the baseline associations persist. The researchers showed that even after fine-tuning a model on a balanced dataset, the original gender associations in the word embeddings remained detectable. It's like trying to erase a stain that's soaked through the fabric. Luna: That's discouraging. But I'm guessing there is some progress — are there any success stories? Lucas: A few. One example: a large financial services firm — I can't name them, but they're in the top ten — worked with a research team to retrain their internal promotion model after an audit revealed bias. They went back and relabeled performance reviews with gender-neutral criteria, then retrained from scratch. The result was a model that recommended women for promotion at rates much closer to their actual performance. Luna: So it's possible, but it takes intentional effort and resources that most companies don't allocate. Lucas: Right. And that's the core ethical challenge. The technology isn't inherently biased — but the data is. And as long as we treat AI as a neutral tool, we're going to keep encoding our own biases into systems that then amplify them at scale. Luna: I think that's the key takeaway. It's not about whether AI should be used in HR — it's about whether we're willing to do the work to make it fair. Lucas: And that work starts with asking the right questions. Like, what data are you training on? Who's reviewing the outputs? And are you measuring outcomes across different groups? Luna: One last thing — I've seen a few startups offering 'bias audits' for HR AI. Are those legit, or is it mostly marketing? Lucas: Mixed bag. Some are run by actual researchers who use rigorous methods — they'll look at your model's predictions across demographic groups, test for statistical parity, and so on. Others are basically consultants who run a few word counts and call it an audit. The MIT study I mentioned has a good framework for what a proper audit looks like: you need to test on held-out data, stratify by protected attributes, and measure both false positive and false negative rates. Luna: So buyer beware. But the fact that there's a market for audits is a good sign — it means companies are starting to feel pressure. Lucas: It is. And regulators are starting to pay attention too. The EEOC in the US has issued guidance on AI and hiring, and the EU's AI Act will classify HR tools as high-risk, which means they'll require conformity assessments. That's a big deal. Luna: It feels like we're at a turning point. The technology is powerful, but we're finally having the right conversations about how to deploy it responsibly. Lucas: Yeah. And performance reviews are just one piece of a much larger puzzle. But if we can fix that feedback loop — if we can make AI stop encoding our worst biases — then maybe we can start to build systems that actually help people grow, not just sort them into boxes. Luna: That's a good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. Talk next time.