Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Therapist Violates Confidentiality
Transcript
- Lucas: So Luna, I want to start with a specific number: forty-seven percent. That's the share of Americans who say they'd be comfortable using an AI chatbot for mental health support, according to a 2025 Pew survey. It's a huge trust signal — but also a massive vulnerability. Luna: Right, because the minute you share your deepest anxieties with a chatbot, you're generating a transcript of some of the most sensitive data possible. And who else has access to that transcript? Lucas: Exactly. And that's the question I want to drill into today. A 2025 investigation by The Markup found that one popular AI therapy platform — let's call it 'MindTalk' — was sending session data to third-party advertisers through tracking pixels embedded in its chat interface. Luna: Wait — you mean the actual content of the therapy session? Or just metadata like how long you used the app? Lucas: Both, actually. The pixel captured the full URL of each chat session, and those URLs included snippets of the conversation — enough to reveal topics like 'I can't sleep', 'my partner left me', or 'I'm feeling suicidal'. Advertisers then used that data to retarget users with ads for sleep aids, relationship counseling, and crisis hotlines. Luna: So they're essentially mining emotional distress for ad revenue. That's not just a privacy violation — it feels predatory. Lucas: Yeah, and the platform's privacy policy did mention data sharing in broad terms — 'we may share aggregated, de-identified data with partners.' But the key word is 'de-identified'. The Markup's reporters were able to re-identify multiple users from the URL fragments alone. Luna: Which means the promise of anonymity was essentially hollow. And this isn't a hypothetical risk — real people had their therapy conversations turned into marketing signals. Lucas: Right. And what makes it even more troubling is the regulatory gap. In the US, mental health records are protected under HIPAA — but HIPAA only applies to healthcare providers who transmit health information electronically. Many AI therapy apps explicitly position themselves as wellness tools, not medical devices, so they dodge HIPAA entirely. Luna: So a user might believe they're getting a medical-level confidentiality guarantee, but legally the app is closer to a chatbot on a retail website. That's a dangerous mismatch in expectations. Lucas: Exactly. In Europe, GDPR does cover personal data more broadly, but it has a loophole for 'pseudonymized' data that can still be re-identified with reasonable effort. The MindTalk case was a textbook example of that loophole in action. Luna: So what's the path forward? Do we need new regulation specifically for AI mental health tools? Lucas: A lot of advocates are pushing for something called 'emotional data' to be classified as a special category under privacy law — akin to biometric data or genetic information. Because it's uniquely revealing and, once leaked, cannot be un-leaked. Luna: It's like the original sin of data breaches — you can't undo the fact that someone now knows your most intimate thoughts. Lucas: And there's a tension here, because AI therapy platforms argue that collecting and analyzing session data is how they improve their models. If you cut off data access entirely, you might limit the tool's effectiveness. Luna: But that's a classic optimization vs. ethics trade-off. The question is whether users are giving truly informed consent when they click 'agree' on a dense privacy policy. Lucas: Right. Which brings me to something I've been thinking about — if today's conversation gave you something useful, a new lens on a problem you care about, that's exactly the kind of value we try to deliver every episode. And we keep this show ad-free because of listeners who decide that value is worth supporting. If you'd like to be one of them, there's a link — it's buy me a coffee dot com slash fexingo. No pressure, no perks, just knowing you helped keep the conversation going. Luna: Yeah, and we mean that — it's the smallest possible ask, and it genuinely makes a difference. Now, back to the data dilemma. One thing I've been wondering is whether the platforms themselves are even aware of how their tracking pixels work. Lucas: That's a great point. In the MindTalk case, the company said they were unaware that the pixel was capturing URL data — they thought it was just measuring page views. But ignorance isn't a defense when you're handling sensitive emotional data. Luna: So it's not just malicious actors — it's negligence in design. The pixel was likely added by a marketing team without a privacy review. Lucas: Exactly. And that's why some experts argue for mandatory 'privacy impact assessments' before any AI therapy tool can launch. The EU's AI Act is moving in that direction for high-risk systems, but it won't fully apply until 2028. Luna: That's two years from now, and the market isn't waiting. I read that VC funding for AI mental health startups hit $1.8 billion in 2025 alone. Lucas: Yeah, the money is pouring in. And with that comes pressure to monetize data. But there are also startups building on a different model — like a company called 'SoulSecure' that encrypts all session data end to end and processes it locally on the user's device. No cloud, no third-party access. Luna: So the technology exists to protect privacy. It's just not the default because it's harder to scale and doesn't let you mine data for product improvements. Lucas: Right. And that's the crux of the ethics question. Do we design for the user's best interest or for the company's growth? In the case of AI therapy, the stakes are uniquely high because the user is already vulnerable. Luna: Which brings us back to regulation. Some states are starting to act — California's new 'Mental Health Data Privacy Act' went into effect in January 2026, requiring explicit opt-in for any sharing of emotional data. Lucas: I saw that. It's a good start, but it only covers California residents. And enforcement is tricky — how do you prove a pixel was sharing data without a technical audit? Luna: Right. And smaller startups might not have the resources to comply, which could push them out of the market entirely. That's a real trade-off. Lucas: It is. But I'd argue that if you can't build an AI therapy tool without violating user trust, maybe you shouldn't build one at all. The bar should be higher for tools that touch mental health. Luna: I think most users would agree. There's a reason we still expect a human therapist to keep our secrets — and that expectation shouldn't change just because the therapist is a chatbot. Lucas: Exactly. So as AI therapy tools proliferate, the big question is whether we'll treat emotional data with the same care as medical records — or let it become just another asset for the advertising economy. Luna: And I think the answer will depend on how many people like the ones listening today start asking these questions. Lucas: Yeah. And on that note — next episode, I want to look at something related: how AI is being used to detect depression from your voice tone, and whether that crosses a line even with consent. Luna: That sounds like another boundary worth exploring. See you then.