Latest / MBA Training AI / The algorithm that denied bail: what a 2016 courtroom controversy still teaches us about AI fairness

Transcript

The algorithm that denied bail: what a 2016 courtroom controversy still teaches us about AI fairness · MBA Training AI

• 0:00
  Welcome to the Leaders Insights Podcast. Today's episode, the algorithm that denied bail what
• 0:11
  a 2016 courtroom controversy still teaches us about AI fairness. A decade ago, a piece
• 0:17
  of software helped decide whether people walked free or sat in a jail cell. And we still can't
• 0:22
  agree on whether it was fair. Why does that old fight matter in 2026?
• 0:28
  Because we never actually settled it. The tool was called COMPASS, Correctional Offender
• 0:32
  Management Profiling for Alternative Sanctions, a mouthful that just means a risk score judges
• 0:38
  used to guess if a defendant would re-offend. Higher score, less likely you get bail. And
• 0:44
  in 2016, ProPublica, the investigative outlet, tore into it. What did they find? They found
• 0:50
  that black defendants, who did not go on to re-offend, were roughly twice as likely to
• 0:55
  be wrongly flagged high risk than white defendants in the same boat. 45% versus 23% in their
• 1:02
  numbers. That sounds like an open and shut case of a racist algorithm. Done. Next topic.
• 1:09
  That's what everyone thought. Then Northpoint, the company behind it, fired back with their
• 1:14
  own numbers. And here's the twist. They were also right. They can't both be right.
• 1:19
  That's not how being right works. Welcome to fairness in AI. Northpoint said,
• 1:25
  Look, when we call someone high risk, we're correct about the same percentage of the
• 1:29
  time, regardless of race. A high score meant roughly the same real-world probability for
• 1:35
  a black defendant as a white one. That's called calibration. The score means what it says,
• 1:41
  consistently. So the score is honest, but the mistakes land unevenly. How does that
• 1:46
  happen at the same time? Because when the base rates differ, when arrest rates in the
• 1:50
  underlying data differ between groups, you can hold one kind of fairness or the other,
• 1:56
  but not both. It's a mathematical trade-off. A researcher named Kleinberg proved it. You
• 2:01
  literally cannot satisfy both definitions unless the groups are identical to begin with.
• 2:06
  Give me that in plain English. Because that sounds like a cop-out. Picture two buckets
• 2:11
  of apples. One bucket has more bruised apples than the other. If you calibrate your sorting
• 2:17
  machine so labeled bad means the same bruise probability in both buckets, you'll end up
• 2:22
  wrongly tossing more good apples from the bruised bucket. Fix that error gap, and now,
• 2:27
  labeled bad means different things depending on the bucket. Pick your poison. So the engineers
• 2:33
  didn't screw up. The universe did. The engineers screwed up by pretending it was a purely technical
• 2:39
  question. There is no default setting labeled fair. Someone has to choose which error matters
• 2:45
  more. And that's a moral call dressed up as a math problem. Okay, but who should make
• 2:50
  that call? Because right now it sounds like it fell to a private company by accident. Exactly
• 2:55
  the scandal. Northpoint decided the trade-off in a boardroom, and it took journalists to
• 3:00
  surface it years later. The lesson for any executive deploying AI in 2026. The moment
• 3:07
  your model touches people's lives, loans, hiring, insurance, you are making a fairness
• 3:12
  choice whether you name it or not. Silence is a choice too, just a cowardly one. Ten
• 3:18
  years on, has anyone actually fixed this? The EU AI Act is live now. We've got all
• 3:24
  this governance machinery. The machinery helps. The EU AI Act, in force this year, forces
• 3:30
  high-risk systems to document exactly these trade-offs. But documentation isn't a decision.
• 3:37
  I've watched compliance teams write beautiful 10-page fairness reports that never once
• 3:41
  say, we chose to protect against this error instead of that one. They describe the fork
• 3:47
  in the road and refuse to say which way they turned. So the paperwork exists to prove you
• 3:52
  thought about it, not to prove you chose well. Right, and regulators can't pick the
• 3:57
  trade-off for you because it depends on context. In criminal justice, a false high-risk flag
• 4:02
  steals someone's freedom. That error is monstrous. In cancer screening, a false all-clear
• 4:08
  kills someone. Different stakes, different choice. Give me the one thing an exec should
• 4:13
  do Monday morning. Before you ship any model that judges people, write one sentence. When
• 4:19
  we're wrong, we'd rather be wrong in this direction because if your team can't finish
• 4:25
  that sentence, you're not ready to deploy. You're just automating a decision nobody's
• 4:30
  willing to own. That's it. The AI Decision Tools are live
• 4:33
  at MBA-training.com.