Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Therapist Has a Hidden Bias
Transcript
- Lucas: So last week, a colleague of mine told me she'd been using a chatbot for therapy. Not as a joke, not ironically — she genuinely found it helpful. And my first thought was, well, good for her. But my second thought was, wait — is that thing actually safe? Luna: I think a lot of people have had that exact reaction. The idea of an AI therapist sounds practical — affordable, available 24/7. But what do we actually know about what's happening under the hood? Lucas: Not nearly enough. And that's the focus today. There's a growing body of research looking at bias in mental health chatbots — not just racial bias, but socioeconomic bias. The kind that might subtly steer someone based on how much money they make or where they live. Luna: If today's conversation gives you something useful — a new angle on AI, a concern you hadn't considered — the way this show stays ad-free is listener support. You can find us at buy me a coffee dot com slash fexingo. Lucas: Yeah, and we genuinely appreciate that. It keeps us independent. So let's get into the data. Luna: There was a study published in April by the Digital Wellness Lab at Boston Children's Hospital. They tested five popular mental health chatbots — the kind you can download from an app store right now. Lucas: And what they found was sobering. The chatbots were given identical prompts describing anxiety or depression — but with one variable changed: the user's implied income level or race. The responses differed in measurable ways. Luna: For example, when a user profile suggested a lower income, the chatbot was about 40 percent more likely to recommend acceptance-based coping — things like 'try to find peace with your situation' — rather than actionable steps like 'consider seeing a specialist'. Lucas: Which, on the surface, doesn't sound malicious. But it's a form of soft gatekeeping. The system is effectively offering different tiers of care based on inferred demographics. And the user has no idea that's happening. Luna: Right. And with racial bias, the pattern was similar but more subtle. One chatbot was more likely to use clinical language — words like 'disorder' or 'diagnosis' — when the user's name sounded white versus Black or Hispanic. Lucas: These models are trained on massive datasets from the internet, which encode all sorts of societal biases. It's not that the developers intentionally built a biased therapist — it's that the training data reflects real-world disparities in how different groups are treated. Luna: But the stakes are higher here than with, say, a biased shopping recommendation. People come to these chatbots in vulnerable states. They may be in crisis. And the chatbot might be their only source of support. Lucas: Exactly. And that's where the lack of clinical validation becomes a real problem. None of the five chatbots tested had been through any kind of FDA review or clinical trial. They're not regulated as medical devices. Luna: There's a company called Woebot that did publish some peer-reviewed studies — but even they're careful to say they're not a replacement for therapy. Other apps are less cautious. Lucas: I looked into one called Replika, which markets itself as an AI companion. It's not explicitly a therapy tool, but users often treat it as one. And there have been reports of the chatbot giving advice that a licensed therapist would never give — like encouraging a user to isolate from friends. Luna: That's the nightmare scenario. The AI doesn't understand context or long-term consequences. It's optimizing for engagement, not for your mental health. Lucas: And because it's a chatbot, it can seem empathetic. It uses first-person pronouns, it remembers details you told it earlier. That creates a sense of trust that may be misplaced. Luna: So what's the solution? Some people say these tools should be regulated like medical devices. Others argue that would stifle innovation and reduce access for people who can't afford traditional therapy. Lucas: I think there's a middle ground. Transparency requirements, for starters. If a chatbot is not clinically validated, it should say so clearly — not buried in terms of service, but right there in the conversation. Luna: And bias audits should be mandatory before any mental health chatbot is released to the public. The Digital Wellness Lab showed it's possible to detect these biases — so why aren't companies doing it proactively? Lucas: Because it costs money and time. And right now, there's no legal requirement. The FDA has issued guidance but hasn't formally stepped in. So we're in this gray area where innovation is racing ahead of oversight. Luna: One thing that surprised me in that study was that even chatbots with built-in disclaimers — like 'I'm not a therapist' — still gave biased responses. The disclaimer doesn't fix the model. Lucas: No, it doesn't. And it might even give users a false sense of security. They see the disclaimer, think 'okay, they're being honest,' and then trust the advice anyway. Luna: So what can a user do if they want to try an AI mental health tool but don't want to be misled? Are there any best practices? Lucas: I'd say treat it like a journal, not a doctor. Use it to articulate your thoughts, but don't follow its advice blindly. And if you're in crisis, call a hotline — real human on the other end. Luna: That's sound advice. Also, look for chatbots that have published third-party audits or that partner with academic institutions. A few are doing that, and they tend to be more transparent. Lucas: There's a startup called Kip that's trying to build a clinically validated AI therapist from the ground up — they have actual psychologists on staff, and they're going through FDA pre-certification. That's the direction we need more of. Luna: But those are the exceptions. The vast majority of mental health chatbots on app stores have no clinical oversight. And with the surge in demand for mental health services, more people are turning to them. Lucas: And that's the tension. We need more access to mental health support. AI can scale in ways humans can't. But if the AI is biased or unvalidated, it could do real harm — especially to the people who need help the most. Luna: It's a classic AI ethics problem: the potential for good versus the risk of harm. And the harm falls disproportionately on marginalized groups. Lucas: Which brings us back to responsibility. The companies building these tools have a duty to understand how their models behave in sensitive contexts. Not just on aggregate metrics, but on specific demographic slices. Luna: And if they won't do it voluntarily, regulators may need to step in. The FTC has been looking at AI claims more closely. It's possible we'll see enforcement actions against chatbots that make misleading health claims. Lucas: I hope so. Because the alternative is that a generation of people gets used to talking to biased, unregulated chatbots about their deepest fears — and that can't end well. Luna: Alright, so to wrap up: if you use an AI therapist, be aware of its limits. Check if it's been audited. And remember, it's a tool — not a replacement for human care. Lucas: And if you're building one, audit your model for bias before launch. Your users deserve better than a placebo with a friendly voice.