Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When AI Recommends Your Sentence
Transcript
- Lucas: So there's this tool used in courtrooms across at least four states that recommends how long a sentence a person should serve. It's called the Public Safety Assessment, and it's been around for nearly a decade. But a study just published last fall from the University of Michigan looked at over 200,000 sentencing recommendations from it and found something pretty troubling. Luna: Let me guess — the recommendations were longer for Black defendants. Lucas: Yeah, on average about 12 percent longer than for white defendants with the same charges and similar criminal histories. The tool itself doesn't see race — it's not supposed to — but it's trained on historical sentencing data. And that data already reflects decades of systemic bias. Luna: So it's essentially learning from the outcomes of a biased system and then perpetuating those biases. That's the classic 'garbage in, garbage out' problem we've seen in hiring algorithms and healthcare algorithms. Lucas: Exactly. And what's interesting is that the Public Safety Assessment was designed specifically to reduce bias. It was created by the Arnold Foundation — now called Arnold Ventures — as an improvement over older, more opaque tools. The idea was to use only a handful of objective factors: age, current charge, prior convictions, things like that. Luna: But if those factors correlate with race because of policing and sentencing disparities, then the tool still produces a racially disparate impact. Even if it never looks at race directly. Lucas: Right. And the Michigan study showed that the tool's recommendations were followed by judges about 60 percent of the time. So it's not just a theoretical exercise. It's directly shaping real sentences. Luna: That's a huge rate of adherence. I'd assumed judges would override it more often. Lucas: A lot of judges say they find it useful as a baseline. But here's the thing — the tool doesn't explain its reasoning. It gives a score from 1 to 6, and the judge gets a recommended sentence range. But there's no transparency about why a particular defendant got a 4 instead of a 3. Luna: So it's a black box. And that's especially problematic in sentencing, where the stakes are someone's freedom. Lucas: Exactly. The study's lead author, a criminologist named Megan Kurlychek, told me that even she couldn't get the full details of how the algorithm weights its factors. The company that commercialized the tool — it's called the Evaluation and Compliance Institute — considers it proprietary. Luna: Proprietary algorithms used in criminal sentencing. That feels like a fundamental due process problem. You can't challenge a sentence if you don't know how the recommendation was reached. Lucas: And that's exactly what defense attorneys have been arguing in court challenges across several states. There have been cases in New Jersey, Kentucky, and Arizona where defendants have tried to force disclosure of the algorithm's inner workings. So far, the courts have mostly sided with the tool's creators, saying it's a trade secret. Luna: Trade secret protection trumping a defendant's right to a fair trial. That seems like a constitutional crisis waiting to happen. Lucas: It might be. There's a case working its way up in Wisconsin that could eventually reach the Supreme Court. The argument is that using a proprietary risk assessment in sentencing violates the Sixth Amendment right to confront witnesses — because you can't cross-examine an algorithm. Luna: I want to zoom out for a second. How widespread are these tools? I know COMPAS got a lot of attention a few years ago, but is this the same thing? Lucas: Related but different. COMPAS is a recidivism risk assessment — it predicts the likelihood that someone will reoffend. The Public Safety Assessment is specifically for pretrial detention and sentencing recommendations. But they share the same core problem: they're trained on biased data and they're opaque. Luna: And we now have evidence that they produce racially disparate outcomes across multiple states. So what's the path forward? Should we just stop using them? Lucas: Some advocates say yes. The Electronic Frontier Foundation and the ACLU have both called for a moratorium on using predictive algorithms in criminal justice until there's independent validation and transparency. But the counterargument is that these tools can reduce human bias if designed carefully. A judge might have implicit biases that a well-calibrated algorithm doesn't. Luna: But we haven't seen that in practice. The empirical evidence so far shows the opposite effect. Lucas: True. And there's a newer approach that some jurisdictions are experimenting with — it's called 'algorithmic impact assessment.' Before a tool is deployed, an independent body evaluates it for bias and transparency. The EU AI Act actually mandates something similar for high-risk systems. Luna: So that's like a pre-market approval process, similar to how the FDA evaluates drugs. Lucas: Exactly. And a few US states — California and Vermont have proposed bills — are looking at that model. But it's early. Most courts still use whatever tool the local prosecutor's office bought. Luna: And that purchase decision itself is often made without any public input or oversight, right? Lucas: Right. It's a procurement process that happens behind closed doors. The company pitches the tool to a judge or a district attorney, and within a few months it's being used on thousands of cases. There's rarely any pilot testing or community consultation. Luna: That's a governance failure. And it's not unique to this one tool — we've seen the same pattern with predictive policing software and child welfare algorithms. Lucas: Yeah, it's a broader pattern in how government agencies adopt AI. And it's often justified by efficiency — these tools save time, they standardize decisions, they reduce backlogs. But at what cost? Luna: The cost is borne by the people who get longer sentences because of an algorithm they can't challenge. And disproportionately, those people are Black and brown. Lucas: Speaking of cost — and this is a bit of a tangent, but it connects — one of the reasons these tools proliferate is that they're often marketed as cost-saving. A county can process more cases with fewer judges or probation officers. And that budget pressure is real. Luna: It reminds me of something you said earlier about how the show is possible because of listener support. If people find these conversations valuable, they can help keep it ad-free and independent. Lucas: Yeah, exactly. If today's episode gave you something to think about — maybe it's worth the price of a coffee. There's a link on our website, buy me a coffee dot com slash fexingo. That kind of support really does make a difference in letting us dig into topics like this without any outside pressure. Luna: And it keeps the show free for everyone. So appreciate anyone who does that. Lucas: Back to the sentencing tools — there is one jurisdiction that's trying a different approach. In Kentucky, they've started publishing annual bias audits of their risk assessment tools. The audits are done by an independent research group at the University of Louisville. Luna: And what do those audits show? Lucas: They show that the tool's predictive accuracy is roughly the same across racial groups — about 65 percent — but the false positive rate is higher for Black defendants. That means Black defendants are more likely to be classified as high-risk when they actually aren't. And that classification leads to longer sentences. Luna: So even when the overall accuracy is similar, the real-world impact is still unequal because of how the tool's errors are distributed. Lucas: Precisely. And that's a subtle but crucial distinction. It's not enough to say 'the algorithm is equally accurate for everyone.' You have to look at who bears the cost of the mistakes. Luna: I think that's the kind of nuance that gets lost in the public debate. People hear 'AI in sentencing' and they either think it's a magic bullet for bias or a dystopian nightmare. The reality is more complicated. Lucas: It is. And the Michigan study shows that even well-intentioned tools can entrench existing inequalities if we don't build in safeguards from the start. Transparency, independent auditing, and the right to challenge a recommendation — those need to be non-negotiable. Luna: Given how quickly these tools are spreading, do you think we'll see federal legislation requiring those safeguards? Lucas: There's a bipartisan bill in the House called the Algorithmic Accountability Act, but it's been stalled for years. I think it'll take a Supreme Court case — or a high-profile wrongful conviction linked to an algorithm — to really move the needle. Luna: That's a sobering thought. We have the evidence now, but we're waiting for a crisis to act. Lucas: That's often how it works. But in the meantime, individual judges are starting to push back. Some have publicly said they won't use risk assessment scores at all. Others require that the scores be accompanied by a written explanation of the algorithm's limitations. Luna: So there's hope in judicial discretion — but it's uneven. And it depends on judges being educated about the technology. Lucas: Which they often aren't. There's a new training program at the National Judicial College that teaches judges about algorithmic bias. It's voluntary, but a few hundred judges have taken it so far. Luna: That's a start. But it feels like we're running a pilot program while the system is already in full production. Lucas: Yeah. And that's the tension at the heart of AI ethics across every domain we've covered — the technology moves faster than our ability to govern it. The question is whether we'll catch up before too much damage is done.