Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When AI Redlines Your Neighborhood Without a Map
Transcript
- Lucas: Luna, I want to talk about a city map that decides who gets stopped by police before a single officer has left the station. Luna: You mean predictive policing — the software that generates hot spots based on past crime data. Lucas: Exactly. And the specific system I have in mind is Palantir's Gotham platform, which several mid-sized U.S. police departments adopted starting around 2022. There's a new study from the AI Now Institute — released just last month — that looked at three cities using Gotham for patrol allocation. Luna: And what did they find? I'm guessing it's not great. Lucas: It's a textbook case of label bias. The system is trained on historical arrest data, which we already know reflects decades of disproportionate policing in certain neighborhoods. So Gotham learns that those neighborhoods are high-risk, sends more officers there, those officers make more arrests, and the data confirms the original prediction. Luna: A feedback loop that compounds the original bias. The map doesn't show crime — it shows where police already go. Lucas: Right. The study's lead author, Rashida Ahmed, calls it 'digital redlining.' One of the cities they examined — I won't name it because the contract is sealed — saw a 34 percent higher false-positive rate in low-income census tracts compared to affluent ones. The system was flagging hot spots that didn't actually produce more crime when patrols were random. Luna: So the prediction itself is self-fulfilling. But what about the technical side — how do you even audit a system like that when the city's contract with Palantir is confidential? Lucas: That's the core problem. The researchers had to use publicly available arrest data and Freedom of Information requests to reconstruct the system's outputs. They couldn't see the model weights or the training dataset. So they reverse-engineered the bias by comparing predicted hot spots to actual 911 calls for service, which are less influenced by proactive policing. Luna: And the gap between the hot spots and the 911 calls — that's your bias metric. Lucas: Precisely. In the second city, they found that after Gotham was deployed, misdemeanor arrests went up 22 percent, while serious crime reports stayed flat. That suggests the system was driving officers to look for low-level offenses in flagged areas, not responding to serious incidents. Luna: Which raises a bigger question: who is accountable? The police chief? The city council that signed the contract? Palantir? Lucas: Right now, nobody. The study notes that none of the three cities had a civilian oversight board with access to the system's performance data. One city's contract included a clause that explicitly prohibited sharing 'trade secret' information with outside auditors. So the algorithm operates as a black box. Luna: I remember a similar fight with bail algorithms a few years ago — same pattern: proprietary software, no transparency, biased outcomes. Lucas: The difference here is the scale. Gotham isn't just a recommendation — it's a resource allocation system. It decides where patrol cars sit, where beat officers walk, which neighborhoods get 'proactive enforcement.' And those decisions shape the data that trains the next version of the model. Luna: So the loop gets tighter over time. Do any cities push back? Lucas: A few. In 2024, the city of Oakland passed an ordinance requiring any AI tool used by police to undergo an independent bias audit every two years, with results made public. But most jurisdictions don't have that. And Palantir has pushed back hard, arguing that revealing model details would compromise security. Luna: Security from what? The public? Lucas: Essentially. They claim that knowing the algorithm's triggers could let people game the system — avoid hot spots, commit crimes in blind spots. But the AI Now researchers counter that the real risk is already playing out: communities being over-policed based on biased predictions. Luna: So what's the alternative? If we want to use data to allocate police resources, can we do it fairly? Lucas: Rashida Ahmed's team proposes a framework called 'community-centered predictive policing.' Instead of training on arrest data, you train on victimization surveys and calls for service that are less influenced by enforcement patterns. And you require that the system's outputs be benchmarked against random patrols in a control area. Luna: So you actually test whether the hot spots are real, not just assumed. Lucas: Exactly. One of the cities in the study — the one with the most transparent contract — ran a six-month pilot where half the patrol zones used Gotham and half used a simple grid rotation. The grid rotation produced comparable crime reduction with fewer arrests and fewer racial disparities. But the city still expanded Gotham citywide after the pilot because, in their words, 'it felt more scientific.' Luna: That's frustrating. The data didn't support the decision, but the perception of precision won out. Lucas: And that's the ethical trap. AI systems carry an aura of objectivity that makes their recommendations harder to question, even when the underlying data is deeply flawed. Luna: If today's conversation gave you something useful — a new way to think about algorithmic bias, or a concrete example to raise at a city council meeting — you can help keep these conversations ad-free. It's buy me a coffee dot com slash fexingo. Lucas: Yeah, listener support is what keeps us independent, and it means we can dig into studies like this one without worrying about sponsors. So if that matters to you, it's buy me a coffee dot com slash fexingo. Luna: And we're back to the data. Lucas, you mentioned that one city expanded Gotham despite evidence it didn't outperform random patrols. What was the public reaction? Lucas: A coalition of civil rights groups sued in March of this year, arguing that the expansion violated the city's own racial equity ordinance. The case is ongoing, but it's already forced the city to release some internal emails showing that the police union lobbied for Gotham because it justified overtime. Luna: So the algorithm is being used as cover for a staffing decision. Lucas: Exactly. And that's the thing about predictive policing: it's never just about prediction. It's about power, resources, and who gets to define what 'safety' means. Luna: I think that's the real takeaway. We need to ask not just 'does the algorithm work?' but 'who does it work for?' Lucas: And until those questions are answered publicly, every hot spot on the map should come with a grain of doubt.