Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Judge Predicts Recidivism Based on Zip Code
Transcript
- Lucas: If today's tech conversation gave you something usable, something to think about, that's the whole point of this show. Luna: Yeah, it really is. Lucas: And the reason we can keep doing it without ads or sponsors is that a small group of listeners chips in monthly through buy me a coffee dot com slash fexingo. Luna: It's not a big ask. It's just what keeps the lights on and the episodes ad-free. Lucas: Exactly. So, back to something that's been on my mind: predictive algorithms in the criminal justice system. Luna: You're thinking about COMPAS, right? The recidivism risk assessment tool? Lucas: Right. COMPAS stands for Correctional Offender Management Profiling for Alternative Sanctions. It's a proprietary algorithm developed by a company called Northpointe, now Equivant. It's been used in courtrooms across the U.S. to help judges decide bail amounts, sentence lengths, and parole eligibility. Luna: And the controversy blew up in 2016 when ProPublica published an investigation showing that COMPAS was biased against Black defendants. Lucas: That investigation is still the most cited example of algorithmic bias in criminal justice. ProPublica looked at more than seven thousand people arrested in Broward County, Florida, between 2013 and 2014. They found that the algorithm incorrectly flagged Black defendants as future criminals at nearly twice the rate it flagged white defendants. Meanwhile, white defendants were more likely to be labelled low risk and then go on to commit another crime. Luna: So it was wrong in opposite directions depending on race. That's not just a statistical quirk. Lucas: No, it's structural. And the really troubling part is that the algorithm's inner workings are proprietary. Northpointe never released the full model, so independent researchers could only test the inputs and outputs. They found that variables like zip code and employment history were heavily weighted. Zip code, in particular, is a proxy for race and economic status in many American cities. Luna: So the algorithm was encoding historical patterns of over-policing in certain neighborhoods. If more police patrol a zip code, more arrests happen there, and the algorithm learns that zip code equals higher risk. It's a feedback loop. Lucas: Exactly. And the consequences are real. A higher risk score can mean a higher bail amount, which means someone might sit in jail for weeks while awaiting trial, even if they haven't been convicted of anything. There's research showing that pretrial detention itself increases the likelihood of a future conviction, because people lose jobs, housing, and the ability to mount a defense. Luna: So the algorithm isn't just predicting recidivism, it's partially causing it. Lucas: That's the paradox. Northpointe defended COMPAS by arguing that the tool was equally accurate across racial groups when measured by something called 'predictive parity' — meaning the percentage of people predicted high risk who actually reoffend was similar for Black and white defendants. But that metric doesn't address the false positive disparity. Luna: Right, because the base rates are different. If Black defendants are arrested and charged at higher rates overall, the algorithm will naturally assign them higher risk scores. But the question is whether that's fair. Lucas: And that's the ethical knot. There are multiple competing definitions of fairness in machine learning. One is 'equal false positive rates,' which COMPAS failed. Another is 'predictive parity,' which COMPAS passed. The problem is that you can't satisfy both at the same time unless the base rates are equal, which they aren't. Luna: So we're stuck with a technical trade-off that has human consequences. Has anything changed since 2016? Lucas: Some states have passed laws. New York's AI Bias in Criminal Justice Act, signed in 2024, requires any automated decision tool used in court to undergo an independent audit for racial and socioeconomic bias. The audit results must be made public. That's a big step. Luna: But that only applies to New York. And even with audits, the algorithms themselves remain proprietary. Lucas: Right. There's a movement toward 'open source' risk assessment tools. One example is the Public Safety Assessment, developed by the Arnold Foundation. It's transparent, uses only nine factors like age and prior convictions, and explicitly excludes zip code and employment history. But it's not as widely adopted as COMPAS. Luna: Because courts buy the proprietary tools, and there's inertia. Plus, the companies selling them argue that transparency would allow people to game the system. Lucas: That's the classic security through obscurity argument, and it's largely rejected by the academic community. Most researchers say that transparency actually improves fairness because it allows for external validation. And if someone could 'game' a risk assessment by changing their zip code, that's a sign the model is flawed. Luna: What about the judges themselves? Do they even understand what the algorithm is doing? Lucas: That's another layer. Studies have shown that judges often override algorithmic recommendations, but when they do, they tend to make more biased decisions. In one study from Wisconsin, judges who received COMPAS scores gave harsher sentences to Black defendants even when the algorithm recommended leniency. So the algorithm can amplify bias, or judges can add their own. Luna: So it's not just the algorithm, it's the human-algorithm interaction. Lucas: Exactly. And that's why explainability matters so much. If a judge sees a risk score but doesn't understand why it was assigned, they might either blindly trust it or blindly distrust it. Explainable AI — models that can show which factors drove a prediction — can help judges make more informed decisions. Luna: But explainability can also reveal uncomfortable truths. If a judge sees that a high risk score was driven primarily by zip code, they might decide to ignore the score entirely. Lucas: And maybe that's the right call. Some jurisdictions, like the state of California, have actually banned the use of risk assessment tools for certain decisions, like setting bail. The California Supreme Court ruled in 2021 that using algorithmic risk scores in bail decisions violated due process. Luna: So we're seeing a patchwork. Some places embrace it, some ban it, some try to regulate it. Lucas: And the conversation is still evolving. As of mid-2026, the federal government hasn't passed comprehensive AI ethics legislation, but the Department of Justice has issued voluntary guidelines for the use of risk assessment tools in federal courts. They recommend transparency, ongoing validation, and training for judges and attorneys. Luna: Which is better than nothing, but voluntary guidelines don't have teeth. Lucas: No, they don't. And the financial incentives for companies like Equivant are strong. Risk assessment is a multi-million dollar industry. There's a tension between profit and fairness. Luna: So what's the bottom line? Can we ever build a fair recidivism algorithm? Lucas: Some ethicists argue that you can't have a fair algorithm in an unfair system. If the data reflects historical discrimination, any model trained on that data will inherit that bias. You can try to de-bias the data, but that's technically difficult and philosophically fraught. Others say that transparent, open-source models, combined with human oversight, can be less biased than human judges alone. Luna: So we're left with a choice between imperfect options. But the first step is admitting that the algorithm isn't neutral. Lucas: Exactly. COMPAS was treated as an objective tool for years, but it was encoding value judgments from the start. The question is whether we're willing to have that conversation publicly. Luna: And to fund the kind of independent auditing that's needed. Which brings us back to how a small group of listeners makes this show possible. Lucas: It does. buy me a coffee dot com slash fexingo. That's where you can join them.