Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When AI Fails the Gig Economy Workforce
Transcript
- Lucas: There's a number that's been stuck in my head all week. According to a 2025 study from the UC Berkeley Labor Center, one major ride-hailing platform deactivates roughly 8 percent of its drivers every year based on an automated algorithm. That's tens of thousands of people who lose their primary income source, and the decision is made by lines of code. Luna: And those deactivations, they're not evenly distributed, are they? I remember the study showed that drivers from lower-income neighborhoods were hit way harder. Lucas: Exactly. The researchers found that drivers whose home ZIP codes fell in the bottom quartile of median income were almost three times more likely to be deactivated compared to those in the top quartile. And the algorithm itself — it's not looking at income or ZIP code directly. It's using proxy metrics like acceptance rate, cancellation rate, and customer ratings. Luna: But those metrics are shaped by where you drive, right? If you work in a dense urban area with lots of short trips and traffic, your acceptance rate might be lower because you're constantly deciding whether a trip is worth it. Lucas: That's exactly the structural bias the study identified. Drivers in wealthier neighborhoods tend to get longer, higher-paying trips, so they accept more rides and get better ratings. Drivers in lower-income areas may face more surge pricing volatility, more short trips, and more passenger complaints about unrelated things. The algorithm treats all of them as equally capable, but the environment isn't equal. Luna: So the AI is basically punishing drivers for circumstances beyond their control. That's the kind of fairness problem that's hard to spot if you're just looking at the aggregate numbers. Lucas: Right. And the thing is, the company's response has been basically, 'We're just enforcing our quality standards.' But when your quality metrics are contaminated by socioeconomic bias, you're not measuring quality — you're measuring privilege. Luna: That's a really sharp way to put it. Lucas: Look, I know we talk about bias in hiring algorithms and healthcare AI a lot on this show. This gig economy angle feels different because it's not a one-time decision — it's a continuous performance evaluation that determines whether you can work tomorrow. And the worker has almost no recourse. Luna: There's a lawsuit in California right now — a group of drivers sued their platform specifically over algorithmic termination without a meaningful explanation. They're citing California's new AI transparency law that took effect this year. Lucas: And that case is fascinating because it's testing the limits of the 'right to explanation.' The drivers want to know exactly which data points triggered the deactivation and how much weight each one had. The company argues that's proprietary trade secret. Luna: So we have a clash between algorithmic accountability and intellectual property. And the workers are caught in the middle. Lucas: Yeah, and this is where the EU AI Act actually goes further than anything in the US. Under that framework, high-risk AI systems — which would include worker management tools — have to provide a clear explanation of decisions. No trade secret exemption for the core logic. Luna: If today's conversation gave you something to think about — and I hope it did — the way we keep the podcast ad-free is through listener support. If you found this useful, you can buy us a coffee at buy me a coffee dot com slash fexingo. It genuinely helps us keep digging into stories like this. Lucas: And we're really grateful for every person who chips in. It's what allows us to spend the time on research and not worry about sponsorship constraints. Luna: So back to the lawsuit — what's the current status? Lucas: It's still in the discovery phase. But what caught my attention was the expert witness report from a computer scientist at MIT. She audited the algorithm using a technique called counterfactual analysis — basically asking, 'If this driver had the same behavior but lived in a different ZIP code, would the outcome change?' And the answer was yes, in a statistically significant number of cases. Luna: That's powerful evidence. Did the platform challenge the methodology? Lucas: They tried, but the judge allowed it. And the MIT report also found that the algorithm's internal weight for customer ratings was actually lower than the weight for acceptance rate — which the company had previously denied. They said acceptance rate was just one of many factors, but the audit showed it was the dominant one. Luna: So the company was being opaque even about the basic structure of the algorithm. Lucas: Right. And this gets to a broader issue: when you rely on automated decisions, you're essentially transferring judgment from humans who can be questioned to a system that can't explain itself. The burden of proof shifts to the worker to prove they were unfairly treated, but they don't have access to the evidence. Luna: That's the exact opposite of due process. Lucas: Exactly. And the Berkeley study also surveyed deactivated drivers — over 60 percent said they received no explanation beyond a generic email saying their account didn't meet quality standards. No specifics, no appeal, no human review. Luna: So what's the fix? Do we need regulation mandating human review before deactivation? Lucas: Some advocates are pushing for that — a 'human-in-the-loop' requirement for any automated termination. But there's a cost argument. Platforms say they process thousands of deactivations a month, and human review would slow things down and increase costs. Luna: But if the alternative is systematic bias that destroys livelihoods, maybe the cost is worth it. Lucas: I think so. And there's also a technical fix: you can build fairness constraints into the algorithm itself. For instance, you could require that the deactivation rate across different ZIP codes doesn't differ by more than a certain threshold. That's called demographic parity, and it's a standard fairness metric in machine learning. Luna: But the companies resist that, don't they? They say it would force them to keep on underperforming drivers. Lucas: They do. But the counterargument is that 'underperformance' is defined by a biased metric. If you define performance in a way that penalizes drivers in low-income areas, then enforcing that definition is just entrenching inequality. The platform could redesign the metric to be more robust — for example, comparing drivers only within similar geographic areas, or controlling for trip distance and time of day. Luna: That seems like a reasonable middle ground. Not eliminating performance standards, but making them context-aware. Lucas: Exactly. And some platforms are starting to experiment with that. But it's slow going because the incentive to change isn't there yet. The cost of litigation is lower than the cost of overhauling the system. Luna: So what would shift the calculus? Public pressure? Regulation? Lucas: Probably a combination. The EU AI Act is a big deal because it applies to any company operating in Europe, including us based gig platforms. So if a platform wants to operate in the EU, it'll have to comply with transparency and fairness requirements. And once they build that system for Europe, it's easier to roll it out globally than to maintain two separate systems. Luna: That's the Brussels effect — EU regulation becomes de facto global standard. Lucas: Right. And we're already seeing hints of that. One of the major ride-hailing companies recently announced a pilot program in a few European cities where deactivated drivers can request a human review within 48 hours. That's directly tied to the AI Act requirements that kick in next year. Luna: So there's some movement. But it's still a far cry from the kind of algorithmic auditing that the MIT researcher did. Should there be a legal requirement for independent audits of worker management AI? Lucas: That's exactly what a coalition of labor groups is advocating for. They want a mandatory third-party audit before any algorithmic management tool can be deployed at scale. And I think that's a smart idea — similar to how financial audits are required for public companies. If an algorithm can affect thousands of people's income, it should be subject to independent scrutiny. Luna: But who pays for the audit? And who sets the standards? Lucas: Those are tough questions. One model is the New York City law on hiring algorithms — employers have to conduct a bias audit and publish the results. But enforcement has been weak. Another model is the proposed federal Algorithmic Accountability Act, which would require impact assessments for high-risk systems. That bill has been introduced a few times but hasn't passed. Luna: So we're at a moment where the technology is outpacing the law. And workers are the ones bearing the risk. Lucas: Yeah, and I think that's the core of this episode's story. The gig economy promised flexibility and independence, but what we're seeing is a new kind of algorithmic management that can be more arbitrary and less accountable than any human boss. And that's not just a labor issue — it's an ethics issue about how we design systems that have power over people's lives. Luna: And that's why we're talking about it on AI Ethics. Thanks for listening to this episode.