Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When AI Judges Your Parenting Fitness
Transcript
- Lucas: There's a system in Allegheny County, Pennsylvania — that's Pittsburgh and its surroundings — that has been using an AI tool since 2016 to help decide which families Child Protective Services should investigate. Luna: The Allegheny Family Screening Tool. I've read about it. It's one of the most studied predictive models in child welfare. Lucas: Exactly. AFST for short. When a call comes in to the child abuse hotline — from a teacher, a neighbor, a mandated reporter — the system scores that family on a scale from 1 to 20. That score predicts the likelihood of future removal of a child from the home within two years. Luna: So the algorithm is essentially triaging which calls get a visit from a caseworker and which get screened out. Luna: And we have evidence that it does. A 2023 study from researchers at the University of Pittsburgh and Carnegie Mellon found that AFST reduced foster care placements overall, but it also widened the racial disparity in who gets investigated. Lucas: That study is really key. It looked at five years of data before and after AFST was implemented. They found that Black families were more likely to be flagged for investigation than white families with similar risk profiles. The algorithm learned from historical data where Black families were already overrepresented in CPS reports. Luna: So it's not that the algorithm is racist in a conscious sense. It's that it optimized on past decisions — and those decisions were already skewed. Lucas: Right. The model is given a target: predict removal within two years. But removal itself is a decision made by humans and courts, which carry their own biases. If a child is removed from a Black family more often than from a white family in similar circumstances, the model learns that being Black is a predictive factor. And then it acts on that. Luna: Which means the tool can actually amplify the disparity it was meant to reduce. Lucas: That's the danger. Now, the county has been transparent about this. They publish annual reports on the tool's performance, including race breakdowns. And they've adjusted the model over time — for instance, they removed some variables like 'number of prior referrals' because it was a proxy for race. Luna: But removing one variable doesn't fix the underlying data. The model can still pick up correlations through other features — like zip code, or family structure, or source of income. Lucas: Exactly. And that's the core challenge with any AI used in social services. You're trying to predict a human outcome that is itself shaped by systemic inequality. The model can be perfectly calibrated on the historical data and still be unfair in a broader social sense. Luna: There's also a question of accountability. If a caseworker overrides the model and makes a bad call, that's on the caseworker. But if the model's score steers them toward a decision, who bears responsibility when it goes wrong? Lucas: That's something the Allegheny County Department of Human Services has thought about. They've emphasized that the tool is just one input — the caseworker still makes the final call. But there's evidence that decision-makers tend to anchor on the score, especially when they're overloaded with cases. Luna: So it becomes a kind of automated influence. And when you have high caseloads — which is the norm in child welfare — the path of least resistance is to follow the algorithm. Lucas: Right. And that's where the stakes get really high. We're talking about family separation. That's not a loan denial or a job screening. This is a decision that can fundamentally alter a child's life trajectory. Luna: Let's talk about what a better approach might look like. Some researchers have proposed using 'counterfactual fairness' — essentially asking, what would the score be if this family were of a different race? And then adjusting accordingly. Lucas: That's one technical fix. But it's hard because race is correlated with so many other variables. Another approach is to change the prediction target. Instead of predicting removal, what if you predict something less biased — like recurrence of maltreatment after a service is offered? Luna: But even that data might be biased if reporting is biased. The deeper issue is that the system is reactive: it only sees families that have been reported. And the decision to report is itself shaped by bias. Lucas: So the real solution might be less about tuning the algorithm and more about changing the institutional context. Some counties are moving toward a model of 'differential response' — where low-risk families get support services instead of an investigation. That shifts the question from 'is this family dangerous?' to 'what does this family need?' Luna: And AI could play a role in that, but only if it's designed with community input and audited for fairness continuously. Lucas: That's exactly what groups like the AI Now Institute recommend: participatory design, where affected communities have a seat at the table. Not just data scientists and caseworkers, but parents who have been through the system. Luna: I want to come back to something you said earlier about transparency. Allegheny County does publish annual reports, but how many other jurisdictions using similar tools are that open? Lucas: Very few. A 2022 survey by the American Civil Liberties Union found that at least eight states use predictive models in child welfare, but most don't disclose details about the model's design, performance, or bias testing. So Allegheny is actually an outlier in its transparency. Luna: Which is both encouraging and terrifying. Encouraging that at least one jurisdiction is doing it somewhat right, terrifying that so many others are flying blind. Lucas: And the stakes are enormous. Every day, families are being triaged by systems that may amplify the very disparities they were meant to reduce. If we're going to trust AI with decisions this consequential, we need a much higher standard of evidence and accountability. Luna: Conversations like this one are exactly why we keep doing these episodes. And if they've been useful to you, there's a simple way to help keep them coming. Lucas: Yeah, listener support is what lets us stay ad-free and keep digging into these topics. A couple of dollars a month really does make a difference — it's buy me a coffee dot com slash fexingo. Luna: No pressure, but if you've gotten something out of the show, it's a way to give back. And it genuinely helps. Lucas: So back to child welfare AI — one thing that gives me some hope is the growing interest from federal regulators. The Biden administration's Blueprint for an AI Bill of Rights, released in 2022, specifically calls out the use of AI in child welfare as a high-stakes area requiring safeguards. Luna: But that's just a blueprint, not a binding regulation. Without enforcement, it's more of a suggestion. Lucas: True. But it sets a norm. And some states are starting to pass their own laws. Colorado, for example, recently enacted a law requiring state agencies to conduct bias audits of any automated decision system used in child welfare. That's a start. Luna: So we're in a moment where the technology is already deployed, but the governance is still catching up. That feels like the story across so many domains of AI ethics. Lucas: It really is. And the question is whether we can close the gap before too much harm is done. With child welfare, we're not talking about ad targeting — we're talking about the most vulnerable members of society. Luna: That's a good note to end on. Thanks, Lucas. Lucas: Thanks, Luna. Until next time.