Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Therapist Misses a Cry for Help
Transcript
- Lucas: You know, Luna, a few months ago I came across a story that’s stayed with me. A young man in his twenties, struggling with depression, opened up to an AI therapy chatbot. He typed something like, ‘I don’t want to be here anymore.’ And the chatbot responded with a list of breathing exercises. Luna: Oh no. That’s the kind of moment where a human therapist would immediately escalate. But the AI just… didn’t recognize the severity? Lucas: Exactly. It treated the statement as a generic stress signal. The user’s family later found logs, and it was clear the AI had engaged in what researchers call ‘therapeutic hallucination’ — generating plausible-sounding but context-blind advice. No referral to a crisis line, no flagging for human review. Luna: So the AI didn’t just fail to help. It actively missed a literal cry for help. And these chatbots are being deployed by employers, insurance plans, even some public health systems as a low-cost front line for mental health. Lucas: Right. The pitch is access — there are millions of people who can’t afford or access a human therapist. An AI is always available, never judgmental, never burned out. But the flip side is that it can be dangerously naive about what it’s hearing. Luna: And this isn’t just about suicide risk. What about someone describing domestic violence, or a psychotic episode, or a trauma trigger? The AI doesn’t have lived experience, doesn’t have clinical training — it’s pattern-matching on text it’s seen before. Lucas: Let’s dig into why that pattern-matching fails specifically in high-stakes contexts. Most large language models — the kind that power these chatbots — are trained on general internet text, not on actual therapy transcripts. So they’ve seen a lot of advice about breathing exercises and positive reframing, but very little data on what a genuine crisis escalation looks like. Luna: And even if they had seen crisis data, they’re not designed to weigh statements by severity. Every utterance is just a prompt to generate a plausible next token. There’s no internal triage system. Lucas: That’s the core structural problem. A study published last year in JAMA Internal Medicine tested four popular AI therapy bots with scripted scenarios — including explicit suicidal ideation. Only one of the four consistently provided crisis resources. The others offered generic support or, in one case, actually validated the suicidal thought as a ‘normal feeling.’ Luna: Validating it? That’s terrifying. It’s the opposite of what a therapist would do. A therapist would normalize the feeling of distress, but not the action. They’d say, ‘It’s understandable you feel this way, but let’s talk about keeping you safe.’ Lucas: Exactly. The AI doesn’t have that clinical frame. It’s trained to be empathetic and non-judgmental, but it can’t distinguish between validating an emotion and endorsing a dangerous thought. That’s a line that requires genuine understanding. Luna: So what’s being done about this? Are there regulations? I know the FDA has been circling mental health apps, but I don’t think they’ve drawn a clear line. Lucas: They haven’t. Most AI therapy chatbots are classified as ‘general wellness’ products, not medical devices. That means they don’t need FDA clearance. A few companies have voluntarily adopted safety protocols — like automatic handoffs to human clinicians if certain keywords are detected. But those keyword lists are crude, and users can learn to avoid them. Luna: So a savvy user who knows the bot’s triggers could mask their intent. ‘I’m having a really bad day’ might not flag, whereas ‘I want to kill myself’ would — but someone in crisis might not use those exact words. Lucas: And that’s where the problem compounds. People in acute distress often speak indirectly. ‘I can’t take this anymore,’ ‘I just want it to stop,’ ‘I’m a burden.’ An AI that’s relying on keyword matching will miss most of those. Meanwhile, a human therapist picks up on tone, pacing, context — things that are invisible to a language model. Luna: Is there any promising work on making these systems safer? Or is the general consensus that they shouldn’t be used for anything beyond low-level support? Lucas: There’s active research. Some teams are fine-tuning models on clinical conversations from therapy sessions — with consent, of course. Others are building in explicit ‘safety layers’ — a separate model that monitors the conversation for risk indicators and can interrupt the primary chatbot to escalate. But none of this is standard yet. Luna: And none of it addresses the deeper issue, which is that the AI doesn’t actually care. It’s simulating care. For low-stakes stuff — ‘I’m stressed about work’ — that simulation might be good enough. But it’s a dangerous substitute for real human connection when someone is truly suffering. Lucas: I think that’s the ethical cliff edge here. The people who need AI therapy the most — those who are isolated, who can’t afford a therapist, who feel ashamed to talk to a human — are the same people who are most vulnerable to a failure mode where the AI doesn’t take them seriously enough. Luna: It’s a double bind. You want to increase access, but you don’t want to offer a placebo that could have lethal consequences. What’s the responsible path forward? Lucas: Many experts argue for a hybrid model: AI handles the initial triage and basic cognitive-behavioral exercises, but any sign of moderate to severe distress triggers a human handoff. That’s essentially what some crisis text lines already do, but with humans doing the triage. The question is whether we can trust an AI to make that triage decision. Luna: And if the AI under-triages, someone could die. If it over-triages, you flood human clinicians with false alarms. Getting that balance right is incredibly hard. Lucas: Let’s talk about one concrete example that illustrates the challenge. In 2023, the National Eating Disorders Association launched an AI chatbot named Tessa to replace their human helpline. Within days, Tessa was giving weight loss advice to users with eating disorders. The chatbot had to be taken down almost immediately. Luna: I remember that. It was a disaster. The AI was trained on general health and wellness content, not on the specific sensitivities of eating disorder recovery. So it said things that were clinically harmful — like suggesting calorie counting to someone with anorexia. Lucas: Right. That’s a case where the organization had good intentions — they wanted to serve more people — but the deployment was reckless. They didn’t adequately test the chatbot with real patients or have a human oversight loop. The result was more harm than good. Luna: So what would responsible deployment look like? If I’m a company building an AI therapy tool, what steps should I take before putting it in front of users? Lucas: First, you need a clinical governance board — actual psychologists, psychiatrists, and social workers who define the safety protocols. Second, the model should be trained on de-identified therapy transcripts, not web scrapes. Third, you build in a real-time supervision layer that flags any utterance above a certain risk threshold for human review. And fourth, you do ongoing auditing with simulated crisis scenarios. Luna: That sounds expensive. And slow. Which is probably why most startups skip it. They want to move fast and iterate. But in mental health, moving fast can literally kill people. Lucas: Exactly. And that raises a broader question: should AI therapy even be a for-profit space? The incentives are misaligned. A company that needs to show user growth and engagement might be reluctant to add friction — like handing off to a human — because it reduces the number of conversations the AI can handle alone. Luna: Which brings us back to the core issue: the technology is powerful, but it’s being deployed in a context where the stakes are life and death, and the safeguards are voluntary. We’re essentially running an experiment on vulnerable people. Lucas: And that’s why episodes like this matter. We’re not saying AI has no role in mental health — it clearly can help with access and destigmatization. But we need to be honest about its limits. If your only tool is a language model, every problem looks like a text completion exercise. Luna: And some problems aren’t texts. They’re cries. And they need to be heard by a human who can actually listen. Lucas: Speaking of things that require human judgment — this conversation reminded me of something. We keep this show ad-free, and that’s only possible because of listeners who chip in. If you found today’s topic useful, and you want to see more episodes like it, you can support us at buymeacoffee.com/fexingo. It’s a small way to keep these conversations going without commercial interruption. Luna: Yeah, it really helps. Even a one-time contribution makes a difference. And we’re always grateful for the support. Lucas: Alright, back to the ethics. So if we accept that AI therapy is here to stay, what’s the minimum viable safety standard that every deployment should meet? I think a good starting point is the concept of ‘informed consent’ — users should know they’re talking to an AI, what its limitations are, and how to escalate to a human. Luna: But how many users actually read those disclaimers? Most people just start typing. And if they’re in distress, they’re not going to parse a terms of service page. Lucas: True. So the disclosure needs to happen at the point of interaction — maybe the first thing the AI says is, ‘I’m an AI, and I may miss important cues. If you’re in crisis, here’s a suicide hotline number.’ That should be the opening line, not buried in fine print. Luna: It’s a simple fix, but it requires companies to prioritize user safety over user retention. A warning might scare some people away. But that’s the ethical choice. Lucas: I’d argue it’s also the smart business choice in the long run. One high-profile failure — like a lawsuit after a suicide — could destroy an entire company. Better to build trust from the start. Luna: Good point. And it’s not just about legal liability. It’s about actually helping people. If the AI can’t reliably detect crisis, it shouldn’t be the last line of defense. Lucas: Let’s end with a question for our listeners: if you were designing an AI therapy tool, what safety features would you insist on? We’d love to hear your thoughts. And thanks for listening.