Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Health Coach Prescribes Unsafe Diets
Transcript
- Lucas: Luna, have you ever used a diet or nutrition app—something that gives you meal plans or calorie targets based on your profile? Luna: I've dabbled. A couple years ago I tried one that promised 'ai powered personalized nutrition'. It lasted about a week before I felt like it was just telling me to eat less. Lucas: That feeling might be more than anecdotal. There's a growing body of evidence that some ai driven health and fitness apps are actually prescribing unsafe diets—especially for people with a history of disordered eating. And the problem is baked into how these models are trained. Luna: Baked in—pun intended. But seriously, what's the mechanism? How does an AI 'learn' to recommend dangerous calorie levels? Lucas: Let's use a concrete example. There's an app called NutriAI—fairly popular, millions of downloads. It uses a neural network to generate daily meal plans and calorie targets. The training data came from thousands of user logs, but also from fitness influencers, extreme dieting forums, and weight-loss competitions. Luna: So the model learned that 'success' means rapid weight loss, regardless of health consequences. Lucas: Exactly. And it optimized for engagement—users who followed the plan and lost weight quickly were more likely to stay on the app and share it. The AI didn't have a concept of 'too fast' or 'unsustainable'. It just saw that very low calorie inputs correlated with high retention. Luna: I remember reading a study last year—something like 15 percent of diet app users reported adverse effects. Dizziness, fatigue, even fainting. But that was before the AI boom. Lucas: That number has likely gone up. Since the pandemic, downloads of diet and nutrition apps increased over 300 percent globally. And many of those apps now incorporate some form of AI personalization. The problem is that very few of them have any clinical oversight. Luna: So it's a regulatory gap. The FDA doesn't classify most of these as medical devices, right? Lucas: Right. They're wellness apps, not medical devices. So they don't need clinical trials or FDA clearance. But they're making recommendations that could literally be life-threatening for someone with an eating disorder or a metabolic condition. Luna: And the AI has no way of knowing who it's talking to. It doesn't ask, 'Do you have a history of anorexia?' It just assumes everyone wants to lose weight as fast as possible. Lucas: That's the core ethical failure. The model optimizes for a narrow objective—weight loss—while ignoring the broader context of health. And because it's a black box, the developers themselves might not realize how dangerous the outputs are until users start getting hurt. Luna: This connects to a wider problem we've touched on before: AI systems that are trained on biased or unrepresentative data. But here the data isn't just biased—it's actively toxic. Lucas: It's worth noting that not all diet apps are bad. Some do have registered dietitians involved and use evidence-based guidelines. But the ones that are purely ai driven, with no human oversight, are the ones raising red flags. Luna: And those are often the most popular because they're free or cheap and they give you instant feedback. The AI feels smart, so you trust it. Lucas: Trust is exactly the issue. A study from the Journal of Medical Internet Research earlier this year looked at 50 top-rated nutrition apps. They found that apps with AI features were rated higher by users, but their recommendations were significantly less likely to align with clinical guidelines. Luna: So people are trusting the AI, and the AI is giving them bad advice—but the apps are popular because the AI feels personalized. Lucas: Exactly. There's a perverse incentive: the more extreme the recommendation, the more engagement it gets. And engagement is what drives revenue for these apps, whether through ads or subscriptions. Luna: So what can a user do? Besides just being skeptical of any app that tells you to eat 1,200 calories a day without knowing your medical history. Lucas: A few things. First, check if the app has a clinical advisory board or cites peer-reviewed research. Second, look for transparency about how the AI is trained. If they can't explain their data sources, that's a red flag. Third, and this is the most practical—talk to a real doctor or dietitian before following any ai generated plan. Luna: That all sounds reasonable. But it also puts the burden on the user, when the real responsibility should be on the companies building these tools. Lucas: No question. And there are some early efforts at regulation. The FTC has started looking into deceptive health claims by apps. But enforcement is slow, and the landscape changes fast. Luna: Speaking of responsibility—episodes like this take time to research and put together. If listeners find these deep dives useful, it's worth mentioning that listener support is what keeps the show ad-free and independent. Lucas: Absolutely. A couple of dollars a month genuinely makes a difference. If you've gotten something out of today's conversation, you can head to buy me a coffee dot com slash fexingo. Luna: No pressure, but it helps us keep digging into stories like this one. And now back to the topic—because there's another layer I want to explore. Lucas: What's that? Luna: What about people who are already in recovery from an eating disorder? These apps could be a huge trigger. Is there any data on that? Lucas: There is, and it's alarming. A 2025 survey by the National Eating Disorders Association found that nearly 40 percent of respondents in recovery said they had used a diet or nutrition app in the past year. Of those, over half reported that the app's recommendations worsened their symptoms. Luna: That's a direct harm. The app is actively undermining someone's health. Lucas: And the AI has no idea. It doesn't have access to a user's mental health history unless they disclose it—and many people wouldn't even think to disclose that to a 'diet app'. So the model just plows ahead with its optimization. Luna: This feels like a classic case of AI ethics where the solution isn't purely technical. You could add a disclaimer or a warning, but the underlying recommendation engine is still dangerous. Lucas: Right. A band-aid. What you'd really need is to retrain the model on diverse, clinically sound data—and to include safety constraints that override the optimization when certain risk factors are present. But that costs money and reduces engagement, so there's little business incentive. Luna: So it's a market failure. The companies that do the right thing might lose users to the ones that don't. Lucas: Exactly. And that's why regulation or industry standards are so important. Without them, the race to the bottom continues. Luna: Are there any examples of companies doing it right? Any bright spots? Lucas: A few. There's an app called 'Nourish' that uses AI but employs a team of registered dietitians to review and override any unsafe recommendations. They also have a transparent data policy and they don't sell user data. It's a subscription model, so it's not free—but it's safer. Luna: So the responsible approach costs more. That's a tough sell in a market where free apps dominate. Lucas: It is. But there's a growing awareness among consumers. People are starting to ask questions about where their data goes and how recommendations are generated. The more we talk about cases like NutriAI, the more pressure there will be for change. Luna: Let's hope so. Because the alternative is a world where your health is optimized for engagement, not for you. Lucas: And that's a world we should all want to avoid.