Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / AI Systems That Predict Your Job Performance Without Your Knowledge
Transcript
- Lucas: So there's a case that landed on my radar a few weeks ago — a Fortune 500 retailer, name's in the lawsuit filings, but I'll hold it — that used a vendor called Workforce AI to score every single employee on something called 'retention risk' and 'performance trajectory.' Luna: And the employees didn't know they were being scored? Lucas: That's the kicker. No disclosure, no consent. The system ingested email metadata, Slack message frequency, calendar meeting patterns — not the content, just the metadata — and ran it through a model trained on historical performance reviews. Output was a score from one to a hundred. Below forty? You get flagged for 'low potential.' Luna: I'm already uncomfortable. And I bet the model had a field day with anyone who works async or has caregiving responsibilities. Lucas: Exactly. The lawsuit — filed in California, February of this year — alleges the system systematically penalized employees who logged fewer Slack messages on weekends or had shorter average meeting durations. The plaintiffs argue that's not just unfair, it's illegal under California's labor privacy laws, which require explicit consent for automated profiling. Luna: Right, because California has that 2018 California Consumer Privacy Act, and then the 2020 Privacy Rights Act. But those were written for consumer data, not employee data. There's a gap. Lucas: A huge gap. Employee monitoring is explicitly exempted from CCPA in most interpretations. So companies like this retailer are essentially operating in a legal gray zone. The case is testing whether that exemption holds when the monitoring is used to make consequential employment decisions — not just security or productivity tracking, but who gets promoted and who gets managed out. Luna: And the vendor, Workforce AI — what's their defense? I imagine they say the model is 'validated' or 'fair.' Lucas: Their public statement said the tool is 'designed to surface objective patterns that reduce human bias.' But here's the problem — the training data was the company's own historical performance reviews, which themselves contain all the biases of the managers who wrote them. So you're encoding past bias into a model that then appears neutral because it's an algorithm. It's bias laundering. Luna: Bias laundering — I'm keeping that phrase. So what kind of data points are we talking about? You mentioned email metadata. Give me a concrete example of something that sounds benign but isn't. Lucas: Sure. One of the features in the model was 'response latency' — how quickly an employee replied to internal messages. The model assumed faster replies correlated with higher engagement. But an employee who blocks deep-focus hours might have a high response latency simply because they're, you know, working. Or an employee with a disability that affects typing speed. The model doesn't know the difference. Luna: That's exactly what I was thinking. And the employee never gets to see their score or contest it. They just suddenly get a bad performance review and no one says 'by the way, our AI decided you're low potential because you take two hours to answer a non-urgent Slack.' Lucas: Right. There's no appeal mechanism. And that's the core ethical problem — not just the surveillance, but the decisional opacity. The EU's proposed Workplace AI Directive, which is still in draft as of June 2026, would require companies to notify employees when an AI system is used to evaluate them, and to provide a meaningful explanation of the output. But it's not law yet, and it only covers the EU. Luna: So outside the EU, it's basically the Wild West. And even within the EU, enforcement is uneven. Let me ask you this — is there any version of people analytics that is ethical? Or is the power imbalance just too fundamental? Lucas: I think there's a version that could be ethical, but it requires conditions that almost no employer today meets. First, informed consent — not buried in an employee handbook, but a clear opt-in with no penalty for refusal. Second, transparency — employees get to see their own scores and the factors that drive them. Third, a human appeal process that actually has teeth. And fourth, periodic independent audits for bias. Luna: That's a high bar. And I think the reason companies don't do it is because if you give people that much transparency, the system loses its power. The whole point of a black-box score is that it's hard to game — or to challenge. Lucas: Exactly. And that's why I think the regulatory route is the only real lever. Because individually, employees have almost no bargaining power. They can't ask their prospective employer 'hey, what AI tools are you using to evaluate me?' without seeming difficult. But a law — like the California one being tested in that lawsuit — can set a floor. Luna: Speaking of the conversation we're having right now — if this topic gave you something to think about, that's the kind of thing we try to do every episode. And listener support is what keeps this show ad-free and independent. If you've gotten value from our deep dives, you can support us at buy me a coffee dot com slash fexingo. It's a small gesture that makes a real difference. Lucas: And we genuinely appreciate every single one. Alright — back to the case. I want to talk about one more dimension: the vendor's responsibility. Workforce AI didn't create the bias; they just encoded it. But is that a defense? I think not. Luna: I agree. If you build a system that you know will be used to make high-stakes decisions about people, and you don't build in safeguards — or even transparency — you share the liability. Especially when you market it as 'objective.' Lucas: There's a parallel here to the credit scoring industry. When FICO scores first became widespread, they were also opaque. But over time, regulation forced disclosure — you can now see your FICO score and the key factors. And that didn't destroy the model; it made it more legitimate. I think the same could happen with workplace AI scores. Luna: But the difference is, with credit scores, the consumer is the customer. With workplace AI, the employer is the customer and the employee is the product. The incentives are misaligned. Lucas: That's a really good point. And it's why the EU directive is so interesting — it treats employees as data subjects with rights, not just inputs to a system. The question is whether the US will follow. Given that California lawsuit, we may see state-level action before federal. Luna: What's the timeline on that case? Any chance it sets a precedent? Lucas: It's in early discovery right now. The plaintiffs are asking for the model's feature weights and training data. If the court orders disclosure, that alone could reshape the industry. Because then we'd know exactly what signals are being used — and whether they correlate with race, gender, or other protected classes. Luna: And that's the thing — we don't even know yet. It could be worse than we think. Or it could be relatively benign. But the fact that it's a secret is itself the problem. Lucas: Secrets in employment decisions are antithetical to due process. And I think that's the ethical bottom line: no one should be judged by a system they cannot see, understand, or appeal.