Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Judge Recommends a Sentence Based on a Typo
Transcript
- Lucas: So there's a case out of Rhode Island, 2023 — a guy named Patrick Dahm. He's up for sentencing on a drug possession charge, pretty standard. The court runs his name through a recidivism risk assessment tool, and it comes back: high risk. The algorithm flags him as a repeat violent offender. Luna: That sounds like it would lead to a longer sentence, for sure. But what did the tool find on him? Lucas: That's the thing — it found records of a different Patrick Dahm. His name is spelled d a h m. The other guy spelled it d a m. But a probation officer typed it in with a typo, and the system matched the misspelling to a guy with a violent felony history. The algorithm then recommended a three-to-five year sentence based on that other person's record. Luna: Wait — so a typo almost sent the wrong guy to prison for years? How did anyone catch it? Lucas: His public defender noticed the name mismatch during review. But here's the scary part: the tool's developer, a company called Equivant — they used to be Northpointe, same company behind the COMPAS algorithm that got famous for racial bias in Broward County — they argued the tool was only as good as the data entered. Which is technically true. But it also means the system has zero error-correction built in. Luna: So it's not a bug — it's a feature of a brittle system. And this is exactly the kind of thing we talk about on this show. If today's conversation made you think about how these systems affect real lives, and you want to support ad-free, independent coverage of AI ethics, listeners can help at buy me a coffee dot com slash fexingo. Even a small contribution keeps us going. Lucas: Yeah, exactly. And we're grateful for anyone who chips in. So back to Dahm — the typo case is a perfect example of what researchers call 'garbage in, gospel out.' The algorithm is treated as objective, but it's completely dependent on the quality of the input. And in many courtrooms, there's no routine audit of that input. Luna: How common is this? I mean, name mismatches can't be that rare in a system that processes millions of records. Lucas: It's more common than you'd think. A 2022 study by the Vera Institute found that in five states, about 12 percent of risk assessment scores contained data entry errors that changed the risk level. That could mean a harsher sentence or a denied parole. And it's not just names — it's arrest dates, charge codes, prior convictions. One wrong digit and the score jumps. Luna: And the defendant doesn't even know the algorithm is being used, let alone that it might be wrong. There's no right to confront your accuser when your accuser is a database glitch. Lucas: Right. And that's the due process problem. In most states, the risk assessment output is shared with the judge, but the underlying data isn't. So the defense can't verify the inputs. There's no cross-examination of the algorithm. You're just expected to accept the score. Luna: Has any state tried to fix this? I know there have been bills around algorithmic transparency. Lucas: A few. California passed a law in 2023 requiring that any risk assessment tool used in sentencing must have its inputs disclosed to the defense. But enforcement is weak. And the bigger issue is that even when you see the inputs, you might not know how they're weighted. The algorithm is a black box, even to the judges. Luna: So transparency alone isn't enough — you need explainability. And you need independent audits. Lucas: Exactly. There's a push for what's called 'algorithmic impact assessments' before a tool can be deployed. The Pretrial Integrity Act, introduced in Congress in 2025, would require federal courts to validate any risk assessment tool for accuracy and fairness before using it. It hasn't passed yet, but it's gained bipartisan sponsors. Luna: Let's talk about the broader pattern. This isn't just about sentencing. Similar tools are used in child welfare, housing, policing. Every time, it's the same story: a system trained on historical data that embeds existing biases, and then we act surprised when it reproduces them. Lucas: Take predictive policing. In 2024, a study of the Los Angeles Police Department's PredPol system found that it sent officers to neighborhoods that were already over-policed, creating a feedback loop. More arrests in those areas meant more data for the algorithm, which then predicted more crime there. It's a self-fulfilling prophecy. Luna: And the people in those neighborhoods don't get to opt out. They don't even know they're being 'predicted.' Lucas: There's no consent. No notice. No appeal. The closest we've come to a legal remedy is the case of Willie Allen Lynch, a man in Ohio who was falsely flagged by a gang database algorithm. He sued in 2022, arguing that the secret algorithm violated his due process rights. The case settled, but it didn't set a precedent. Luna: So what would meaningful reform look like? If you had a magic wand for the criminal justice system specifically? Lucas: First, a moratorium on new tools until they're independently validated. Second, mandatory human review of any algorithmic recommendation that could affect liberty. Third, a public registry of all government-used algorithms, with their performance metrics and error rates. Some of that is in the Algorithmic Accountability Act, which has been introduced a few times but never passed. Luna: And what about the tools already in use? There are dozens, probably hundreds, operating right now without oversight. Lucas: That's the hardest part. Existing tools are embedded in court workflows. Judges rely on them. Withdrawing them could cause chaos. But there's a intermediate step: auditing. Some non-profits, like the Electronic Privacy Information Center, have started auditing risk assessment tools on their own, publishing error rates. They found that COMPAS has a false positive rate for recidivism that's almost twice as high for Black defendants as for white defendants. Luna: And yet it's still used in at least a dozen states. Lucas: Because it's cheap. And it offers the illusion of objectivity. A judge might think, 'I'm not being biased — the algorithm says so.' But the algorithm learned bias from historical sentencing data, which was itself biased. So you're just laundering that bias through math. Luna: It feels like we're in a phase where the problems are well-documented, but the political will to fix them is lagging. What would change that? A high-profile wrongful conviction that gets national attention? Lucas: Maybe. Or a Supreme Court case. There's a petition for certiorari right now in a case called State v. Loomis, which challenges the use of COMPAS in Wisconsin. The argument is that it violates the defendant's right to a fair trial because the algorithm is proprietary and can't be examined. If the Court takes it, that could be a landmark. Luna: We should follow that. But for now, what's the takeaway for someone listening who might encounter these systems — maybe as a juror, or a defendant, or just a voter? Lucas: Ask questions. If you're on a jury and the judge mentions a risk score, ask what it's based on. If you're a voter, ask your local DA what tools they use and whether they've been audited. The worst thing we can do is treat algorithms as neutral arbiters. They're not. They're tools made by people, with all the flaws that entails. Luna: And a typo away from sending the wrong person to prison. Lucas: Exactly. A typo. That's the margin of error in a system that's supposed to be more just.