Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Landlord Evicts You
Transcript
- Lucas: So there's this company called SafeRent. It's a tenant screening service that uses a machine learning model to assign a 'rental risk score' to every applicant. The score is supposed to predict how likely you are to pay rent on time or get evicted. Luna: Right, and I've seen that used by property managers across the country. It's basically a credit score for renting, but without the transparency of a traditional credit report. Lucas: Exactly. And here's the thing — in 2023, the Massachusetts Attorney General sued SafeRent, alleging that their algorithm discriminates against Black and Hispanic tenants. The state claims the model relies on factors like housing voucher status, which disproportionately screens out minorities. Luna: Wait, so using a housing voucher — which is often a proxy for race — is baked into an algorithm that denies people housing? That seems like a textbook case of disparate impact. Lucas: That's exactly the legal theory. The Fair Housing Act prohibits practices that have a discriminatory effect, even if unintentional. So Massachusetts is arguing that SafeRent's model violates that law, regardless of whether the company intended to discriminate. Luna: And SafeRent isn't the only one. There are a handful of startups — like CoreLogic's Rental Property Solutions, or First Advantage — that sell similar scores. It's a whole ecosystem of algorithmic gatekeeping for housing. Lucas: Right. And what makes these systems particularly opaque is that tenants never see their own score, and they can't challenge the data that goes into it. If a traditional credit report has an error, you can dispute it. With SafeRent, there's no equivalent process. Luna: So the algorithm becomes a black box that landlords trust blindly, and tenants have no recourse. That feels like a huge power imbalance. Lucas: It is. And the Massachusetts case is just one example. There have been class action suits in other states too. In 2022, a group of tenants in California sued a property management company that used an AI screening tool, claiming the algorithm was more likely to flag Black applicants as high risk. Luna: I remember that case. The company settled for about a million dollars. But that's a drop in the bucket compared to the harm caused — people being denied housing, forced into less safe neighborhoods, maybe even into homelessness. Lucas: And the problem isn't just screening for new tenants. There's also a growing trend of using AI to predict which existing tenants are likely to fall behind on rent — and then using that prediction to start eviction proceedings. Companies like RealPage offer 'eviction risk scores' to landlords. Luna: That feels like a self-fulfilling prophecy. If the algorithm flags you, the landlord might start eviction before you even miss a payment. Then you actually miss a payment because you're dealing with the stress and legal fees. Lucas: Exactly. And the data used to train these models often comes from historical eviction records, which themselves reflect systemic racism. So the algorithm learns to associate certain zip codes or demographic patterns with higher risk, perpetuating the cycle. Luna: So what's the solution? Should we ban these algorithms outright? Or can they be made fair? Lucas: Some advocates say a ban is the only way, because even well-intentioned algorithms can encode bias. But others argue that if you design the model carefully — using only income, rental history, and credit reports — and you audit it regularly for disparate impact, it could be less biased than a human landlord who may act on gut feeling. Luna: But can you really audit an algorithm when it's proprietary? SafeRent has refused to disclose the exact factors in its model, citing trade secrets. Lucas: That's the core tension. Without transparency, there's no accountability. Some states have started to pass laws requiring that any algorithm used in housing decisions be explainable and routinely tested for bias. But federal legislation still lags. Luna: It's interesting — we've seen similar debates around AI in hiring, in lending, in healthcare. But housing feels particularly urgent because shelter is so fundamental. Lucas: Absolutely. And a couple of dollars a month is genuinely what keeps these conversations going — buy me a coffee dot com slash fexingo, if you've gotten something out of them. Luna: Yeah, it makes a real difference. We don't run ads, so listener support is what keeps us ad-free and focused on these deep dives. Lucas: Right. So back to the SafeRent case — a judge actually ruled against some of the state's claims in 2024, but the litigation is ongoing. The core question remains: can an algorithm comply with fair housing law if its inner workings are hidden? Luna: And even if the algorithm is transparent, can it ever be truly race-neutral when the data it's trained on is soaked in historical discrimination? Lucas: That's the deeper problem. Some researchers argue that you can never completely de-bias a model because the real world is biased. The best you can do is mitigate the most harmful outcomes and continuously monitor. Luna: Which is a far cry from the tech industry's typical promise of 'just trust the algorithm.' Lucas: Exactly. And until we have regulatory frameworks that require independent audits and meaningful transparency, these systems will continue to operate as black boxes that can deny someone a home with no explanation. Luna: It's a sobering reminder that AI ethics isn't a theoretical exercise — it's about whether people get a roof over their heads. Lucas: And that's why episodes like this matter. Thanks for listening.