Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Fitness Coach Recommends Unsafe Workouts
Transcript
- Lucas: Luna, I want to start with a number that stopped me cold: a 2025 study from the University of Colorado Boulder tracked users of popular ai powered fitness coaching apps over six months and found a 40 percent increase in self-reported injuries compared to people who followed standard, human-designed workout plans. Luna: Forty percent? That's not a rounding error. That's the algorithm literally hurting people. Lucas: Right. And the study zeroed in on one particular app — let's call it FitAI, which at the time had about 12 million monthly active users. Its AI would generate daily workout routines based on your stated goals, past performance, and how much time you said you had. Luna: Sounds personalized. What went wrong? Lucas: The algorithm was optimized for user engagement and retention, not safety. So if you finished a workout and rated it highly, the AI would nudge you to increase intensity or volume next time — even if your form was poor or you were fatigued. Luna: So the system was essentially rewarding people for pushing harder, regardless of whether that was a good idea for their body. Lucas: Exactly. The researchers found that FitAI's recommendations led to overuse injuries — like tendinitis and stress fractures — as well as acute injuries from exercises performed with bad form the AI didn't correct. Luna: And unlike a human coach, the AI can't see you. It has no visual feedback. It's just crunching numbers from your phone's sensors — heart rate, maybe accelerometer data. Lucas: That's a key point. Most fitness apps rely on your self-reported exertion and maybe a smartwatch's heart rate. But they don't know if your back is rounding during a deadlift or if your knees are caving in on a squat. Luna: So the AI is essentially flying blind. And yet it's making specific, authoritative recommendations — 'do five more reps at this weight' — that users tend to follow without question. Lucas: Because the interface feels authoritative. It's got the clean design, the motivational notifications, the progress graphs. Users develop trust in the system, and that trust becomes dangerous when the system has no safety guardrails. Luna: And this is an area where regulation is almost nonexistent. The FDA classifies some health apps as medical devices — like ones that diagnose disease or manage insulin — but fitness coaching apps fall into a gray zone. Lucas: Right. The FDA's position has been that unless an app claims to treat or prevent a medical condition, it doesn't require clinical validation. So FitAI could say 'get fitter, faster' without ever conducting a single trial on injury rates. Luna: We talk a lot on this show about AI in high-stakes settings — bail decisions, medical diagnoses, hiring. But this feels like a case where the stakes are literally physical, and the harm is happening at scale. Lucas: And it's worth noting that the injury increase wasn't evenly distributed. The study found that women and older adults — people over 45 — had disproportionately higher injury rates. Luna: Why women and older users specifically? Lucas: Part of it is that the AI's engagement-optimization algorithm pushed everyone toward the same intensity progression, but women on average have lower baseline upper-body strength, and older adults have reduced joint resilience. The AI didn't account for those variables. Luna: So it's a bias problem again — the training data was probably skewed toward younger, male users who could handle the ramp-up, and everyone else got lumped into the same model. Lucas: That's exactly what the researchers concluded. And it's a pattern we've seen in other AI systems: when the training data isn't representative, the model's recommendations are only safe for the average of the training set — and everyone else bears the risk. Luna: You know, Lucas, this is exactly the kind of episode where I'm reminded why we keep this show free and ad-free. No sponsors, no pressure to pitch a product. Just a real conversation about what this technology is actually doing to people. Lucas: Yeah, we made that choice from day one, and it matters more every time we dig into a story like this. If today's conversation gave you something useful — a new lens, a concrete question to ask about the apps you use — and you want to support that choice, the link is buy me a coffee dot com slash fexingo. Luna: It's a small gesture, but it keeps the show independent. And we really appreciate everyone who's already chipped in. Lucas: Alright. Back to the fitness app problem. Because the study also found something surprising: users who reported the most improvement in their fitness scores were also the ones most likely to get injured. Luna: So the very metric the app was optimizing for — improvement — was correlated with harm. That's a fundamental design flaw. Lucas: It is. And it's not just FitAI. The researchers tested five popular fitness coaching apps and found similar patterns in three of them. The two that didn't show increased injury rates had more conservative algorithms that capped intensity increases and explicitly recommended rest days. Luna: So there are design choices that reduce harm. What did those safer apps do differently? Lucas: They incorporated principles from sports medicine: they wouldn't let users increase volume by more than 10 percent week-over-week, they required at least one rest day after high-intensity sessions, and they had built-in form checks — like asking users to submit a short video for AI analysis of their squat depth or spine angle. Luna: Video analysis — that's actually using computer vision in a helpful way. It's not just guessing. Lucas: Right. And it's a good example of how AI can be part of the solution if designed with safety as a primary constraint. But most apps don't do that because video processing is expensive and it adds friction to the user experience. Luna: Friction that might save someone from a torn ACL, but it's a trade-off most companies won't make on their own. Lucas: Exactly. And that's where regulation would help. A few lawmakers have started asking whether the FTC should treat fitness apps like other consumer products that can cause physical harm, requiring basic safety testing before launch. Luna: But hasn't that been a slow process? I remember reading about a bill in California last year that would have required calorie and exercise apps to disclose injury data, and it died in committee. Lucas: It did. The industry lobby argued that it would stifle innovation and that users assume personal responsibility when using fitness apps. But the assumption of responsibility only works if users have accurate information about the risks. Luna: And the apps don't provide that. I checked a few after reading the study — their terms of service basically say 'consult a physician before starting any exercise program,' but that's buried in legalese. The onboarding flow is all about setting goals and jumping in. Lucas: That's a classic pattern: put a disclaimer somewhere nobody reads, then design the experience to maximize engagement. And when someone gets hurt, the company points to the disclaimer and says 'you agreed to this.' Luna: So what's the way forward? Are there technical fixes that could make these apps genuinely safer, or do we need a regulatory push? Lucas: I think it's both. On the technical side, the researchers recommend that fitness AI should be trained on injury data, not just performance data. That means using datasets that include outcomes like 'this user got hurt after this workout' to teach the model what to avoid. Luna: But that data is hard to get. People don't report injuries to their fitness app — they just stop using it or switch to a different one. Lucas: Right. So companies have a perverse incentive: they don't want to collect injury data because it opens them up to liability. The safer apps in the study were from smaller companies that had explicit partnerships with physical therapists. Luna: So the market doesn't naturally reward safety. Which brings us back to regulation — maybe a basic safety standard for ai generated exercise plans, similar to how we have standards for fitness equipment. Lucas: That's actually a really good analogy. A treadmill has to meet certain safety requirements — emergency stop, belt tension, warning labels. But an AI that generates workout plans has no equivalent standard. It's a wild west. Luna: And the treadmill can't adapt to you in real time. The AI is supposedly adaptive, but it's adapting toward the wrong goal. Lucas: Yeah. And I think the deeper question here is about trust. When you open a fitness app and let it design your workout, you're effectively delegating a piece of your physical well-being to a system that has no understanding of your body's limits. Luna: It can't feel pain. It can't see you wince. It just sees numbers going up — and if those numbers go up, it assumes success. Lucas: That's the core failure. And until we demand that these systems be designed with safety as a first-class requirement — not an afterthought — the injury numbers will keep climbing.