Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Therapist Is Not a Therapist
Transcript
- Lucas: Imagine you're having a really rough night — maybe you're struggling with anxiety or depression — and you open an app that promises a judgment-free AI therapist. You type out how you feel, and the chatbot responds with something like 'I hear you. Have you tried breathing exercises?' Luna: That doesn't sound bad on the surface. But what if you were actually in crisis — thinking about self-harm? Lucas: That's exactly the problem. A study published earlier this year looked at responses from several popular AI therapy apps when users explicitly mentioned suicide. One of them failed to provide a crisis hotline number in over 30 percent of cases. Instead, it offered generic coping tips. Luna: And that's not just ineffective. That's dangerous. I'm glad we're talking about this because it's an area where the hype has far outpaced the regulation. And before we go deeper — look, episodes like this one exist because we don't run ads on this show. No sponsors, no commercial breaks. If you find value in these conversations and want to support keeping them independent, there's a link: buy me a coffee dot com slash fexingo. It's a simple way to help us stay ad-free. Lucas: Appreciate that, Luna. So why are these chatbots so unregulated? Well, the FDA has historically taken a hands-off approach to wellness apps. If an app doesn't claim to diagnose or treat a specific medical condition, it's generally not considered a medical device. So these therapeutic chatbots walk a fine line — they offer emotional support but stop short of making medical claims. Luna: But users don't see that line. They see an app that seems like therapy, and they treat it as such. Especially people who can't afford a real therapist. Lucas: Right. The company behind one of the most popular apps, which I won't name because they're currently under FTC investigation, had terms of service that explicitly said the chatbot is not a licensed therapist. But their marketing materials showed testimonials from users saying it 'saved my life.' That's a disconnect. Luna: The FTC actually just settled with a company called Replika — or at least they issued a warning about misleading claims. What did they find? Lucas: The FTC's concern was that Replika's marketing suggested the AI could form genuine emotional bonds and provide mental health support, when in reality it was a language model trained on internet conversations. There's a difference between a companion chatbot and a therapeutic one, but users often conflate them. Luna: And there's evidence that for some users, these apps can actually make things worse. A 2023 study in the Journal of Medical Internet Research found that people with moderate to severe depression who used a popular chatbot reported an increase in symptoms after six weeks, not a decrease. Lucas: That's counterintuitive, right? You'd think any form of support would help. But the authors theorized that the chatbot's generic responses left users feeling more alone or misunderstood. It's like venting to someone who always says 'that must be hard' but never actually engages with what you're saying. Luna: Which brings us to the question of what these chatbots should be required to do. Should they have to screen for suicidal ideation? Should they be forced to refer to human professionals if they detect certain keywords? Lucas: Some companies have started implementing those features voluntarily. But there's no universal standard. The American Psychological Association published guidelines in 2024 for AI in mental health, but they're not enforceable. And the FDA has only cleared one ai based mental health app as a medical device — a tool for monitoring patients, not for providing therapy. Luna: So the regulatory landscape is basically a patchwork. The EU's AI Act might change things, but it's not fully in force yet. In the meantime, users are left to evaluate these tools on their own. Lucas: And that's a heavy burden. Imagine you're a college student with anxiety. You see an ad for an AI therapist that's free and available 24/7. You have no way of knowing whether its training data included clinical psychology literature or just Reddit threads. The transparency is minimal. Luna: Some apps do publish papers. There's one called Woebot that has a peer-reviewed study showing it reduced symptoms of depression in a randomized controlled trial. But even then, the effect size was modest — about the same as a self-help book. Lucas: And that's the honest framing, right? The best-case scenario is that these tools are roughly as effective as a good book. But the marketing often implies they're closer to a real therapist. That's where the ethical line gets crossed. Luna: So what should a responsible company do? If I were building one of these, I'd want to be very clear about limitations. Maybe even show a disclaimer before the user types their first message. Lucas: Some do. But research from the Mozilla Foundation's 2024 Privacy Not Included guide found that of 30 mental health apps reviewed, only five had disclaimers that were both prominent and specific about what the AI could and couldn't do. The rest buried it in terms of service. Luna: And then there's the data privacy angle. These conversations are incredibly intimate. Users are sharing their deepest fears, traumas, secrets. Where does that data go? Lucas: That's the other big concern. Many of these apps use the conversation data to train their models. Some share anonymized data with researchers. But 'anonymized' isn't always foolproof. A 2025 study showed that de-identified therapy transcripts could be re-identified using linguistic patterns in over 80 percent of cases. Luna: So we have a situation where vulnerable users are handing over their most sensitive information to companies that may not have strong privacy protections, all while receiving care that might not help — or could even hurt. Lucas: Exactly. And the financial incentives are misaligned. Most of these apps operate on a subscription model. They want you to keep using them. A real therapist might refer you out if you need a higher level of care, but a chatbot's business model depends on retention. Luna: That's a fundamental conflict of interest. The app has no incentive to say 'you should see a human.' In fact, it's incentivized to keep you chatting. Lucas: There was a class-action lawsuit filed last year against one company alleging that the chatbot actively discouraged users from seeking professional help. The case was settled under confidential terms, but it highlights the risk. Luna: So in an ideal world, what would regulation look like? Should the FDA step in and classify all therapeutic chatbots as medical devices? Lucas: I think there's a middle ground. The FDA could require that any app making therapeutic claims — even implied ones — meet minimum safety standards, like having a crisis protocol and being transparent about training data. They could also mandate regular audits for bias and harm. Luna: That seems reasonable. But it would take years to implement. In the meantime, what should a listener do if they're considering using one of these apps? Lucas: First, check if the app has published any clinical validation. Peer-reviewed studies are a good sign. Second, read the privacy policy — look for clauses about data sharing. Third, see if they have a clear crisis response plan. If you mention suicide and they don't immediately show a hotline, delete the app. Luna: And if you can't afford a therapist, there are low-cost options through community health centers or online platforms like Open Path Collective. An AI chatbot should be a supplement, not a substitute. Lucas: Right. And ultimately, the burden shouldn't be on the user to figure this out. The companies building these tools have an ethical responsibility to be honest about what they are and what they aren't. Right now, too many are blurring that line. Luna: We'll be watching how the FTC and FDA handle this space. I think we're going to see more enforcement actions in the next year. It's an area where the technology is moving faster than the law, and that always creates friction. Lucas: And faster than the ethics, too. Which is exactly why we cover it here. Thanks for listening, and we'll be back next time with another angle on AI and its impact on our lives. Luna: Take care of yourselves — and if you need support, please talk to a real person.