Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Fitness Coach Pushes Unsafe Workouts
Transcript
- Lucas: So there's this fitness app that promised to be your personal AI trainer — you tell it your goals, your health history, and it designs a workout plan. Sounds great, right? Luna: On paper, sure. But I'm guessing there's a catch. Lucas: Big one. A 45-year-old user in Texas with a known heart condition — he disclosed it in the app — followed the AI's recommended high-intensity interval routine and ended up in the ER with cardiac distress. Luna: Wait, he told the app about his heart condition? And it still pushed HIIT? Lucas: Exactly. The app's algorithm apparently flagged his condition but still generated a plan that included sprints and burpees. The company's defense? The algorithm only 'suggests' routines, and users should consult a doctor before starting. Luna: That feels like a cop-out. If you're marketing an AI as a personal trainer, you're implicitly endorsing its advice. Lucas: That's the core tension here. These apps walk a line between helpful tool and unlicensed medical advice. And unlike a human trainer, there's no certification, no liability insurance, no real oversight. Luna: So what's the regulatory landscape? Are there any rules governing AI fitness coaches? Lucas: Almost none. The FDA regulates medical devices, but fitness apps are generally considered 'general wellness' products, so they're exempt. The FTC can step in for deceptive claims, but that's after the fact. Luna: So essentially, the user bears all the risk. That seems backward. Lucas: It is. And it's not just this one app. Major wearables like Fitbit and Apple Watch have faced similar questions — their algorithms detect irregular heart rhythms, but they're careful to frame it as 'not a diagnostic tool.' Still, people act on that data. Luna: There was a study a couple years ago — I think from Stanford — that found many fitness apps gave unsafe recommendations for people with chronic conditions. The algorithms just weren't trained on that population. Lucas: That's the data bias problem. Training data for these models typically comes from healthy, young, active users. So if you're older, have a pre-existing condition, or are a woman — many models underperform because they're trained on male-dominated datasets. Luna: Right, and that's dangerous. But let's talk about the legal angle. If someone gets hurt, who's liable? The app developer? The AI model? The user? Lucas: So far, courts have been hesitant to hold AI liable. The argument is that the AI is just a tool, and the user is responsible for their own health decisions. But that feels inadequate when the AI is presented as a personalized coach. Luna: Especially because users may not have the medical knowledge to override the AI's suggestions. They trust it. Lucas: Exactly. And there's a precedent in other industries. When a self-driving car causes an accident, the manufacturer is often sued. Why should fitness AI be different? Luna: Maybe it shouldn't. But the companies argue that their AI is just software, not a medical device. It's a regulatory loophole. Lucas: A loophole that's getting wider as more health-adjacent AI products launch. There are now apps that claim to detect skin cancer from photos, or predict your risk of diabetes. Same regulatory gray zone. Luna: That's terrifying. But let's bring it back to the fitness coach. What would responsible design look like in that case? Lucas: I think a few things. First, the app should flag high-risk users and either refuse to generate a plan or require a doctor's sign-off. That's basic safety. Luna: But would that hurt adoption? Most apps want to keep users engaged, not turn them away. Lucas: It might, but that's a business tradeoff. Second, the AI should be transparent about its limitations — it should say, 'I wasn't trained on people with your condition, so take this with caution.' Luna: And third? I'm guessing liability insurance? Lucas: That would help, but I think third is independent auditing. Just like financial models get stress-tested, fitness AI should be evaluated by medical professionals before launch. Luna: So you're basically calling for a regulatory framework akin to what we have for medical devices. Lucas: Not necessarily as strict, but some baseline. The FDA's general wellness policy could be updated to include AI that gives personalized health advice. That wouldn't be a huge lift. Luna: It's interesting — we're seeing a parallel with AI in other domains. Like AI therapists or AI legal advisors. The same pattern: unregulated, user assumes risk, companies hide behind disclaimers. Lucas: It's a pattern that needs to break. And the fitness case is a perfect example because it's so relatable. Everyone wants to get in shape, and the promise of a personalized AI coach is appealing. But the danger is real. Luna: What about the user's responsibility? Should they have known better than to follow an app's advice with a heart condition? Lucas: Some might say yes. But I think it's reasonable to assume that an app marketed as a 'smart coach' would be safe. The average person doesn't know how algorithms work or what data they were trained on. Luna: Fair point. And the app's terms of service probably have a liability waiver buried in there somewhere. Lucas: Almost certainly. But that doesn't make it ethical. And it doesn't protect the user from harm. Luna: You know, this episode is exactly the kind of practical insight that people need. If today's conversation gave you something useful, the way these stay ad-free is through listener support. You can buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small way to keep the lights on. We really appreciate it when folks chip in. So back to the Texas case — the user is actually suing the app company for negligence. Luna: Oh interesting. What's the legal theory? Lucas: They're arguing that by collecting health data and generating a personalized plan, the app assumed a duty of care. And by failing to adequately warn or tailor the plan, it breached that duty. Luna: That could set a precedent. If the court agrees, it might force fitness AI companies to rethink their approach. Lucas: Exactly. And it could also push regulators to act. The FTC has been looking into health apps more closely, and this case might accelerate that. Luna: What about the data privacy angle? The app collected the user's health data — is that protected under HIPAA? Lucas: Not necessarily. HIPAA applies to healthcare providers, not fitness apps. So that data can be used for training models or even sold, with few restrictions. Luna: So another layer of risk. The user might not only get hurt but also have their data exploited. Lucas: Right. And that's why we need a comprehensive approach — not just safety, but privacy and transparency. Luna: So what's the takeaway for someone considering using an AI fitness coach? Lucas: I'd say: be skeptical. If you have any health conditions, talk to a real doctor or human trainer first. Don't assume the AI knows best. Luna: And for companies building these products? Lucas: Invest in diverse training data, include medical review in your design process, and be transparent about limitations. It's not just ethical — it's good business in the long run. Luna: Because one lawsuit can ruin a brand's reputation overnight. Lucas: Exactly. And more importantly, it can ruin someone's health. That's a cost no algorithm should impose. Luna: Well said, Lucas. I think we covered a lot of ground today. Thanks for the conversation. Lucas: Always a pleasure, Luna. See you next time.