Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Landlord Evicts You
Transcript
- Lucas: So there's this lawsuit filed last October in federal court in California. Thirty-seven tenants versus one of the largest property management firms in the country. And at the center of it is a piece of software that decides who gets to keep their apartment. Luna: We've talked about AI in credit scores, in hiring, in insurance. But this one feels closer to home for a lot of people. Housing is kind of the bedrock. Lucas: Exactly. The system in question is called SafeRent — and yes, it's a real product. It's used by property managers across the US to screen applicants and flag tenants who might be 'high risk.' But the lawsuit alleges that it's not just screening new tenants — it's also used to recommend evictions for existing ones. Luna: So the same algorithm that says 'yes' or 'no' at the door can later say 'time to kick them out.' That's a lot of power in one black box. Lucas: Right. And the plaintiffs point to a pretty damning number. According to the complaint, tenants who were flagged by SafeRent as high risk had a 34 percent higher rate of eviction filings compared to tenants whose files were reviewed manually — even when those tenants had similar payment histories and lease violations. Luna: Thirty-four percent. That's not a rounding error. That's a systemic pattern. Lucas: It gets worse when you break it down by race. The lawsuit alleges that the algorithm disproportionately flagged Black and Hispanic tenants — even after controlling for income and credit score. So you have a situation where a machine is essentially automating housing discrimination, but because it's a machine, the property manager can say 'we're just following the software's recommendation.' Luna: This feels like the same problem we saw with automated bail recommendations and sentencing guidelines. The algorithm inherits bias from historical data — and then it gets to wear a cloak of objectivity. Lucas: And what's striking here is how little we know about SafeRent's actual model. The company hasn't disclosed what variables it uses. But the plaintiffs' lawyers dug into some discovery materials and found that the system incorporates — get this — social media sentiment scores. Luna: So they're scraping your tweets and Facebook posts to decide if you're a good tenant? That's insane. Lucas: According to the complaint, yes. The algorithm pulls in publicly available social media data and assigns a sentiment score. Negative sentiment — complaints, venting, frustration — that apparently contributes to a higher risk score. Never mind that everyone vents online sometimes. Luna: And I'm guessing there's no way to appeal or even know what went into the score. Lucas: That's exactly what the plaintiffs argue. The Fair Credit Reporting Act requires that consumers be able to see and dispute the data used in certain decisions. But SafeRent is arguing that its model is a proprietary trade secret — so they don't have to disclose how it works. Luna: Trade secret versus basic fairness. That's a tension we keep seeing across all these ai in decision making cases. Lucas: And it's not just one company. There's a whole ecosystem now. SafeRent, AppFolio, Yardi — these are major players in property management software. They're processing millions of tenant records. The question is whether we need regulation that says: if your algorithm makes decisions that affect someone's housing, it has to be auditable and explainable. Luna: Is there any precedent for that kind of requirement? Lucas: New York City passed a law in 2021 — Local Law 144 — that requires employers using AI hiring tools to conduct annual bias audits and publish the results. It's not perfect, but it's a model. Some lawmakers have proposed similar rules for housing algorithms. Nothing's passed yet. Luna: So right now, it's basically the Wild West. And tenants are the ones getting evicted based on a score they can't see or challenge. Lucas: And that's the core ethical problem. Housing is a fundamental human need. If we let opaque algorithms decide who gets to keep a roof over their head, we are essentially ceding a basic right to a profit-driven black box. Luna: And the thing is, this is a technology show. If today's conversation gave you something to think about — maybe a concrete example to bring up next time you're talking about AI and fairness — then it's worth reflecting on how these systems are already embedded in daily life. Lucas: Yeah. And speaking of that — quick honest thing. This show doesn't have ads. It's funded entirely by listeners who chip in through buy me a coffee dot com slash fexingo. It's a small group, but it's what makes it possible for us to do deep dives like this, episode after episode. Luna: And we really appreciate it. It keeps the conversation independent and focused on what matters. Lucas: So — back to SafeRent. There's another dimension here that I think is worth exploring. Luna: What's that? Lucas: The data sources. Beyond social media, the plaintiffs' discovery also revealed that SafeRent uses what they call 'lifestyle scores' — things like whether you've bought lottery tickets, or how frequently you change addresses, or even the kind of car you drive. Luna: Buying lottery tickets? That's not a credit factor. That's a judgment call about someone's character. Lucas: Exactly. And those judgments are built into a model that's never been tested for accuracy the way a credit score is. FICO scores have decades of data showing they predict default risk. SafeRent's model — we have no independent validation. Luna: So property managers are paying for a tool that might not even be better than a simple random assignment. But it feels scientific because it's an algorithm. Lucas: And that's the dangerous part. The illusion of objectivity. When a human property manager denies an application, they can be questioned. When a computer says no, it's 'just the system.' Luna: So what would a fair system look like? If you were designing it from scratch? Lucas: First, full transparency on inputs. Tenants should know what data is being used — and have the right to correct errors, which is already the law for credit reports. Second, mandatory bias testing before deployment. Third, a human review option — any adverse action should be appealable to a person who can override the algorithm. Luna: And maybe a ban on certain types of data — like social media sentiment or lifestyle proxies that have no proven link to rental risk. Lucas: Right. Some states are already moving in that direction. California's proposed AB 2930 would restrict the use of algorithms in housing decisions and require impact assessments. It didn't pass last session, but it's been reintroduced. Luna: So there's momentum. But in the meantime, thousands of tenants are being scored every day by systems they don't understand. Lucas: And that's why lawsuits like this one matter. Even if they don't win on every claim, they force the company to open the black box a little. They generate discovery. They create a public record of how these systems actually work. Luna: And they give tenants legal standing to fight back. Which is more than the algorithm ever gives them.