Latest / MBA Training AI / AI bias testing before deployment: the EU AI Act playbook

Transcript

AI bias testing before deployment: the EU AI Act playbook · MBA Training AI

• 0:00
  Host: Welcome to the Leaders Insights podcast.
• 0:07
  Host: Today's episode, Testing AI Systems for Bias and Fairness Before Deployment, a practical
• 0:12
  Host: playbook.
• 0:13
  Host: Today, a playbook landed with a line I can't shake.
• 0:16
  Host: Deploying an AI system without bias testing is like shipping software without QA, quality
• 0:21
  Host: assurance, the testing step, except the bugs are people.
• 0:25
  Host: Is that just a spicy metaphor?
• 0:27
  Expert: It's the least spicy way to say it, honestly.
• 0:30
  Expert: The classic case is still the 2018 Amazon hiring tool that downgraded resumes containing
• 0:36
  Expert: the word women's, as in women's chess club.
• 0:40
  Expert: They scrapped it.
• 0:41
  Expert: But teams keep rediscovering that same failure in 2026, just with fancier models.
• 0:46
  Host: Fine, but that's eight years old.
• 0:49
  Host: Why does the playbook matter right now?
• 0:51
  Expert: Because the EU AI Act, the European law classifying AI by risk level, started biting
• 0:57
  Expert: this year.
• 0:58
  Expert: High risk systems, meaning hiring, credit, anything that gates someone's life, now legally
• 1:04
  Expert: require documented testing before you ship.
• 1:07
  Expert: So bias testing went from nice to have to something a regulator can find you over.
• 1:16
  Expert: You define the protected groups and the outcome you care about.
• 1:20
  Expert: Sounds obvious?
• 1:21
  Expert: Everyone skips it.
• 1:22
  Expert: You pick, say, gender and age, and you decide, are we measuring whether approval rates are
• 1:27
  Expert: equal or whether error rates are equal?
• 1:31
  Expert: Those are different fairness definitions, and you cannot satisfy both at once.
• 1:35
  Host: Wait, you're telling me fairness is mathematically impossible to fully achieve?
• 1:40
  Expert: Correct, and anyone selling you unbiased AI is lying.
• 1:44
  Expert: There's a proven result.
• 1:46
  Expert: If base rates differ between groups, you can equalize approval rates or equalize error
• 1:51
  Expert: rates, not both.
• 1:53
  Expert: So step one is choosing which unfairness you're willing to tolerate, out loud, in writing.
• 1:58
  Host: That's uncomfortable.
• 2:00
  Host: Give me the tools people use to actually measure this.
• 2:03
  Expert: Open source ones.
• 2:04
  Expert: IBM's AIF 360, the AI Fairness 360 toolkit, gives you about 70 metrics.
• 2:13
  Expert: Google's What If tool lets you eyeball outcomes across groups.
• 2:17
  Expert: And Fairlearn, a Microsoft-backed library, is the one I'd start with because it's readable.
• 2:23
  Expert: All free, which kills the, we couldn't afford it, excuse.
• 2:27
  Host: IBM and Microsoft giving away fairness tools.
• 2:30
  Host: Is that generosity or reputation insurance?
• 2:33
  Expert: Both, and I'd flag it.
• 2:35
  Expert: These are vendor tools, so they nudge you toward their cloud.
• 2:38
  Expert: But the metrics are standard math, not proprietary magic.
• 2:42
  Expert: Use them, just don't assume the defaults match your context.
• 2:45
  Host: So what's the mistake that actually gets teams into trouble?
• 2:49
  Expert: Testing on the training data.
• 2:50
  Expert: The same examples the model learned from.
• 2:53
  Expert: Your model looks perfectly fair because it memorized those people.
• 2:56
  Expert: Then, production hits, the real population is different, and the gap explodes.
• 3:01
  Expert: You test on a held-out set that mirrors who you'll actually serve.
• 3:05
  Host: What about the dodge where a company says, we don't collect race or gender, so we can't
• 3:10
  Host: be biased?
• 3:11
  Expert: That's the most dangerous sentence in the room.
• 3:13
  Expert: If you don't collect the attribute, you can't measure the bias.
• 3:17
  Expert: But the model finds it anyway through proxies.
• 3:20
  Expert: Zip code stands in for race, first name stands in for gender, blindness isn't fairness, it's
• 3:25
  Expert: just not looking.
• 3:27
  Host: Give me a number that shows the stakes.
• 3:29
  Expert: A 2019 study in science found a healthcare algorithm used on roughly 200 million Americans
• 3:34
  Expert: a year systematically underestimated how sick black patients were because it used past medical
• 3:40
  Expert: spending as a proxy for need and less money had historically been spent on them.
• 3:45
  Expert: Same proxy trap, enormous scale.
• 3:48
  Host: So even a well-meaning team building a health tool walked straight into it.
• 3:51
  Expert: Every time, nobody wrote discriminate in the spec.
• 3:55
  Expert: They picked a convenient target variable and the historical inequity wrote in on it.
• 4:00
  Expert: That's why you test outcomes, not intentions.
• 4:03
  Host: We've got a minute.
• 4:04
  Host: What's the thing a team should distrust most in their own process?
• 4:07
  Expert: A single fairness number that looks good.
• 4:10
  Expert: One clean metric means you probably tested one definition on friendly data.
• 4:15
  Expert: Real testing is ugly and produces trade-offs you have to argue about.
• 4:19
  Host: Give me the one thing a listener can do tomorrow.
• 4:22
  Expert: Take whatever model you've got in production right now, pull a fresh sample of real users,
• 4:27
  Expert: split it by one protected group, and compare error rates across the two.
• 4:31
  Expert: If you can't do that by Friday, you don't know if your system is fair.
• 4:35
  Expert: Yee, you're just hoping.
• 4:37
  Host: That's a wrap.
• 4:38
  Host: Fresh AI briefings drop daily at MBA-training.com.