Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Interviewer Holds a Grudge
Transcript
- Lucas: So there's a new wrinkle in the AI hiring story that I don't think we've talked about yet. It's not just about bias in training data or biased prompts — it's about something more subtle: the AI holding a grudge. Luna: A grudge? Like, the AI remembers a bad experience with one candidate and then penalizes similar ones later? Lucas: Exactly. And it's not science fiction — a team at the MIT Media Lab published a study last year, 2025, where they simulated a hiring AI that learns from its own past decisions. The AI was given a batch of candidate profiles to screen, and after each hire, it got feedback on whether that hire worked out. If a hire turned out to be a poor performer, the AI would start marking candidates with similar features — same university, similar experience, even similar phrasing in their cover letters — as higher risk. Luna: So it's essentially learning a stereotype based on one bad data point. That's dangerous because the sample size is tiny. Lucas: And the effect is persistent. Once the AI learns that, say, graduates from a certain university tend to underperform — based on one or two cases — it can take hundreds of positive hires from that same school to correct the bias. Meanwhile, perfectly qualified candidates are being filtered out for years. Luna: Right. And this is different from the usual bias we talk about — it's not a static bias baked into training data, it's a dynamic bias that evolves based on the company's own hiring outcomes. That feels harder to audit. Lucas: That's the key insight. It's a feedback loop. The AI's decisions affect who gets hired, which affects the performance data, which then reinforces the AI's preferences. If the initial data is skewed — say your first hire from a particular school didn't work out — the AI might never give another candidate from that school a fair chance. Luna: And by the way, this kind of conversation — the kind that makes you stop and think about how these systems actually work — is exactly why listener support matters so much for this show. We keep the podcast ad-free because we want to dig into these nuances without any sponsor pressure. If you find value in episodes like this, you can support us at buy me a coffee dot com slash fexingo. It's a small way to keep these deep dives going. Lucas: Yeah, and we really appreciate everyone who's already chipped in. It makes a difference. So back to the grudge-holding AI — one of the researchers I spoke with told me about a real-world example from a large retailer that uses an AI to screen customer service applicants. The AI noticed that candidates who used certain words — like 'enthusiastic' or 'people person' — were more likely to quit within three months. So it started downgrading anyone who used those words. The problem is, those words also correlate with conscientiousness and good performance in some roles, so the AI was basically filtering out potentially great hires. Luna: So the AI was making a spurious correlation based on a few bad apples. That's the classic overfitting problem, but applied to people's careers. Lucas: Exactly. And the worst part is, the company didn't know this was happening until an external auditor ran a simulation. The AI had been in use for two years, and the bias had become deeply embedded. The researchers call this 'historical feedback bias' — it's not about the training data, it's about the ongoing interaction between the AI and the environment it's shaping. Luna: What are the proposed fixes? Can you just reset the AI's memory periodically? Lucas: That's one idea. Some researchers suggest a 'memory budget' — the AI can only retain a certain number of past outcomes before it has to forget the oldest ones. That way, the impact of any single bad hire diminishes over time. Another approach is to keep a human in the loop for any decision that creates a negative bias signal — if the AI flags a candidate as high risk because of a pattern, a human recruiter has to confirm that pattern actually exists. Luna: But humans are biased too. Aren't we just replacing one problem with another? Lucas: Sure, but the point is to break the feedback loop. If the AI learns a bias and then acts on it autonomously for months, that's much worse than a human who might have a bad day. Also, humans can be trained to recognize their own biases. The AI can't — unless you explicitly program it to check for fairness constraints. Luna: So what should companies be doing right now to prevent this? I'm thinking of the HR departments that might be listening. Lucas: The first step is awareness. Many companies don't realize their AI is learning from its own decisions because they only audit the training data, not the ongoing behavior. The MIT team recommends running a 'counterfactual audit' — take a candidate who was rejected, slightly tweak their profile to make them look like a previously successful hire, and see if the AI changes its mind. If it doesn't, you might have a grudge problem. Luna: That's smart. So it's like a bias stress test. But who's actually doing these audits? Is it standard practice yet? Lucas: Not even close. The study surveyed fifty HR tech vendors and found that only about a third even monitor for this kind of feedback loop bias. Most are still focused on static fairness metrics — like whether the AI's acceptance rates match across demographic groups. But dynamic bias is a whole different beast. Luna: And regulatory bodies haven't caught up either. The EU's AI Act talks about bias monitoring, but it's pretty vague on the feedback loop issue specifically. Lucas: Right. The law is still playing catch-up. In the meantime, I think it's on companies to be proactive. One thing that helps is keeping the AI's training data and its real-time decisions separate — don't let the AI update its own model based on outcomes without human review. That's basic, but a lot of systems are designed to be fully automated for efficiency. Luna: Yeah, efficiency is the enemy here. The whole reason companies use AI is to speed up hiring, but if you slow it down with human checks, you lose that advantage. Lucas: Exactly. It's a trade-off. But I'd argue that a slightly slower hiring process that's fair is better than a fast one that systematically excludes qualified people. And in the long run, fair hiring probably leads to better outcomes anyway. Luna: Speaking of long-term outcomes, what about the candidates themselves? If someone is rejected by an AI that holds a grudge, they'll never know why. They just get a generic rejection email. Lucas: That's the transparency problem. Even if the AI is biased, the candidate has no way to challenge the decision. Some advocates are pushing for a 'right to explanation' in hiring AI — the candidate should be able to get a meaningful reason for their rejection, not just a form letter. But right now, most vendors treat their algorithms as trade secrets. Luna: There's a company called HireVue that used to do that — they'd say 'our model is proprietary, we can't tell you why you were rejected.' They got a lot of pushback and eventually started offering more transparency. But they're an exception. Lucas: Yeah, and even with transparency, the feedback loop problem is hard to explain to a candidate. 'Sorry, we rejected you because our AI had a bad experience with someone from your college five years ago.' That's not exactly satisfying. Luna: It's also potentially illegal if it leads to discriminatory impact. The EEOC has been looking at AI hiring tools, but they haven't issued clear guidance on this specific issue yet. Lucas: Right. I think we'll see more regulation in the next couple of years. But for now, it's a bit of a Wild West. The good news is that researchers are developing tools to detect and measure this kind of bias. The MIT team released an open-source audit toolkit that any company can use to test their own AI. We'll link to it in the show notes. Luna: Good. So what's the one takeaway for our listeners? If you're in HR, what should you do on Monday morning? Lucas: Check if your hiring AI has a memory. Specifically, ask your vendor: 'Does your model update based on the performance of the people we hire?' If the answer is yes, ask how they prevent feedback loop bias. If they don't have a good answer, that's a red flag. Luna: And if you're a job seeker? Is there anything you can do? Lucas: Unfortunately, not much. But you can try to avoid being too similar to a hypothetical 'bad hire' — which is impossible to game. I'd say just be yourself. The system might be unfair, but you can't change that by tweaking your resume. It's on the companies to fix their AI. Luna: And on regulators to catch up. It feels like every month there's a new AI bias story. I wonder when the public outcry will be loud enough to force change. Lucas: Maybe it already is. The MIT study got a lot of press when it came out, and I know several large employers have started taking feedback loop bias seriously. It's slow, but it's moving. Luna: Let's hope so. Because the last thing we need is an AI that holds a grudge.