Latest / 1 More Paper / AI chatbots for promoting healthy habits: Legal, ethical, and societal considerations — van Kolfschooten et al. (2025)
Transcript
- You’re listening to “AI chatbots for promoting healthy habits: Legal, ethical, and societal considerations,” by H. van Kolfschooten and colleagues. Published in 2025. Abstract. Machine learning-based artificial intelligence (AI) chatbots are increasingly used to promote health and encourage individuals to adopt healthier behaviors. Chatbots driven by generative AI (genAI) simulate human interactions through text or voice to generate personalized content with guidance on topics such as smoking cessation, nutrition, managing stress, and sleep improvement. The use of AI chatbots for health promotion and wellness has been growing since 2023. While empirical evidence suggests their effectiveness in supporting behavioral change and mental health, the legal, ethical, and societal implications remains largely unexplored. This article presents a qualitative case study of S.A.R.A.H. (Smart AI Resource Assistant for Health), a genAI chatbot developed by the World Health Organization (WHO), analyzed against the six ethical principles outlined in the WHO’s 2021 Guidance on Ethics and Governance of AI for Health. We also gathered exploratory insights from adolescent focus groups. These findings are descriptive and not based on formal thematic analysis. Drawing on this analysis, we identify key gaps between high-level ethical principles and practice and offer policy recommendations to guide responsible use of AI chatbots for health promotion. Introduction. Machine learning (ML)-enabled conversational agents, or artificial intelligence (AI) chatbots, are increasingly deployed by researchers, companies, and public health bod-ies to support behavioral changes, ranging from smoking cessation1 to improved diet,2 mental health,3 and sleep.4 Generative AI (genAI) chatbots powered by large language models (LLMs) simulate human conversation and enable more dynamic, context-sensitive interactions than earlier scripted systems.5 Since 2024, their popularity has surged alongside growing public adoption. While evidence of their effectiveness and cost-efficiency remains limited, early research suggests LLM-based chatbots may prevent non-communicable diseases, expand access to health information, especially in underserved populations, and reduce strain on health systems.6 Yet, the rapid deployment of these tools raises pressing questions about their ethical, legal, and societal implications. Are core principles such as autonomy, equity, and transparency upheld? How is user privacy protected? And can such systems truly deliver on their health-promoting promises? To explore these questions, we analyzed S.A.R.A.H. (Smart AI Resource Assistant for Health), an LLM-enabled chatbot launched by the World Health Organization (WHO) in 2024,7 using the WHO’s own 2021 ethical AI principles outlined in the Guidance on Ethics and Governance of AI for Health.8 Focus groups with adolescents provided further insights into usability, trustworthiness, and value. We con-clude with recommendations for ethical governance of genAI chatbots for promoting healthy behavior. 1Amsterdam Law School, University of Amsterdam, Amsterdam, Netherlands 2Center for Life Sciences Law, University of Basel, Basel, Switzerland 3Department of Media & Communication, Erasmus University Rotterdam, Rotterdam, Netherlands 4Faculty of Technology, Policy and Management, Delft University of Technology, Delft, The Netherlands Hannah van Kolfschooten, Amsterdam Law School, University of Amsterdam, Amsterdam, Netherlands; Center for Life Sciences Law, University of Basel, Basel, Switzerland. Emails: the email address; the email address The current state of AI chatbots for health promotion Healthcare chatbots have long provided basic health information, answered common questions, and assisted with administrative tasks. These early systems relied on rule-based logic with scripted, inflexible responses. Recent advances in ML, particularly the rise of LLMs, have trans-formed chatbots into more dynamic, personalized, and interactive tools. Table 1 provides an overview of the evo-lution from early rule-based chatbots to current genAI systems used in health promotion. The growth of genAI has accelerated the adoption of health chatbots. Notable examples include Ada Health and Buoy Health (symptom checkers) and Headspace (integrated CBT techniques). Specialized chatbots are also emerging, such as Daleela for Arabic-speaking women’s health and Troomi, designed for adolescents.9 Early evidence suggests conversational AI can effect-ively support behavior change. A 2023 systematic review reported improvements in physical activity, diet, and sleep quality,10 while other studies noted that LLM-based chatbots (e.g. ChatGPT, Google Bard, LLaMA-2) can encour-age healthy habits, though impact is greater among motivated users.11 Many users also value chatbots’ non-judgmental tone, empathy, and round-the-clock availabil-ity.12 Despite these benefits, AI chatbots pose risks to their users. We highlight these below, using the case study of S.A.R.A.H. as an example. research on their tool. Figure 2 is a screenshot of a sample conversation with S.A.R.A.H., in this case, about healthy eating habits. It shows how the tool uses facial expressions and records audio, but can also be used for written conversation. It lists the different areas of public health that the tool can assist users with. Promises and pitfalls of genAI chatbots for public health Case study of S.A.R.A.H.: the WHO’s digital health assistant Methods. Our analysis combined three approaches. First, we inter-acted with S.A.R.A.H. to assess its usability and alignment with ethical principles. We documented its responses, privacy notices, and interface design. Second, we reviewed available documentation on its design, intended use, and safeguards from WHO, Soul Machines, and OpenAI between January and April 2025. Third, we drew on focus groups with 25 adolescents (aged 16–23) recruited through community centers in Rotterdam, the Netherlands. These focus groups took place in the first quarter of 2025 and were part of a larger, pre-registered qualitative study on adolescents’ experiences with mental health and health-related apps, for which a full thematic analysis is being conducted.14 Participants used S.A.R.A.H. and shared feedback on usability, trust, and accessibility. In March 2024, the WHO launched S.A.R.A.H., a genAI chatbot developed with Soul Machines (New Zealand).13 Designed as a research prototype, S.A.R.A.H. tests the feasibility of deploying genAI for health promotion at scale, offering 24/7 multilingual conversations on topics such as nutrition, stress management, and tobacco cessation. As the WHO’s first experiment with genAI for public health communication, S.A.R.A.H. represents a valuable governance case study: it illustrates both the promise of AI-enabled health outreach and the complex ethical, legal, and societal challenges these technologies raise as they become integrated into public health strategies worldwide. We analyze S.A.R.A.H. against the six ethical principles WHO’s outlined in 2021 Guidance on Ethics and Governance of AI for Health8 and draw on user insights from adolescent focus groups. Figure 1 shows the general description of S.A.R.A.H. as provided by the WHO and lists the eight available use languages. Notably, it reveals how the WHO is explicitly asking for feedback and ethical For this article, we used only the subset of discussions that addressed S.A.R.A.H., which was introduced during the ses-sions as an example of a genAI chatbot for health promotion. Because this material is limited in scope, we present these findings descriptively to illustrate user perspectives and complement our legal and ethical analysis, rather than as a standalone qualitative dataset. 1. Protecting autonomy. S.A.R.A.H. aims to enhance users’ autonomy by offering accessible health information on topics such as diet, exercise, tobacco cessation, and stress management. This supports individuals in recognizing risk factors for major conditions like cancer, heart disease, and diabetes. However, autonomy is limited by the chatbot’s tendency to offer generic advice unless explicitly prompted for tai-lored guidance. Adolescents in our focus groups noted that S.A.R.A.H.’s responses felt overly broad, sometimes resulting in advice that was either too general or poorly sui-ted to their needs. Privacy and consent issues further complicate autonomy. S.A.R.A.H. encourages users to enable camera and micro-phone access for a more “interactive experience,” capturing sensitive biometric data such as voice and facial expres-sions. While the WHO states that all conversation data is anonymized, multiple privacy policies apply (WHO, Soul Machines, OpenAI), creating a complex, fragmented framework. Users must navigate three separate privacy policies, none of which clearly describe how data flows between entities, where it is stored, or whether it is used for model training. This patchwork approach undermines informed consent and accountability, particularly given WHO’s extraterritorial status and the cross-border nature of the plat-form’s infrastructure. Notably, S.A.R.A.H. provides no clear answers to users asking about how their data is stored, shared, or secured. In focus groups, adolescents expressed discomfort with the ava-tar “watching” and “listening,” which discouraged them from fully engaging with the tool. Without stronger privacy protec-tions and clearer communication, AI chatbots like S.A.R.A.H. risk undermining user autonomy. Figure 3 shows the information the user is given before accessing the AI chatbot. 2. Promoting human well-being, human safety and the. public interest The WHO emphasizes that AI systems must avoid harm and ensure user safety, yet key safeguards appear to be missing in S.A.R.A.H. There are no age checks or warn-ings, leaving minors exposed to content not designed for them. The chatbot’s authoritative tone and polished, human-like avatar risk fostering misplaced trust, with users potentially overestimating its expertise.15 Adolescents in our focus groups described the chatbot’s answers as overly long, repetitive, and at times glitchy or incoherent, echoing broader research showing AI chatbots often struggle to maintain sensitive, fluid conversations. A critical gap is the lack of human oversight: when users dis-close suicidality, S.A.R.A.H. advises seeking clinical help but offers no concrete resources. Generic advice without quality control, particularly given that development and maintenance have been outsourced. Without strong oversight and clear pathways for improvement, S.A.R.A.H.’s ability to uphold safety is uncertain. actionable support may heighten risks, especially where access to care is limited.16 While WHO guidance calls for continuous monitoring, it remains unclear how the organization ensures ongoing 3. Ensuring transparency, explainability, and intelligibility. The WHO emphasizes transparency in its ethical AI S.A.R.A.H.’s principles, yet implementation reveals important gaps. While the chatbot notes that training data can be made available on request, users are not informed about the specific language model powering its responses. Although the landing page links to OpenAI’s privacy policy, it is not made explicit whether OpenAI’s models are used, creating potential confusion. S.A.R.A.H. assures users that conversation data are anonymized and points to privacy policies from WHO, Soul Machines, and OpenAI. However, users are not given the option to opt out of data collection for training purposes, nor are they informed in accessible language about how their data flow between these actors. Our focus group participants noted frustration when the chatbot declined to answer questions about its own privacy safeguards, under-mining trust. Despite WHO’s leadership in setting AI ethics standards, these shortcomings illustrate the difficulty of translating high-level principles into concrete, user-facing transparency. 4. Fostering accountability. The WHO endorses S.A.R.A.H. but disclaims responsi-bility for its content, warning users not to rely on its responses as medical advice or even as factually correct. This disclaimer, though legally sound, may undermine trust, especially given the high expectations users have of WHO-affiliated tools. Accountability structures are also minimal. Users can submit feedback via a general survey, but there is no clear process to report harmful content or engage with a responsible team. This lack of direct oversight weakens responsiveness to errors or risks. Moreover, S.A.R.A.H. operates outside formal clinical care frameworks, meaning it is not subject to medical device regulations or legal protections tied to patient care, such as medical secrecy or liability standards.17 This regulatory gray area highlights an urgent need for clearer governance of AI health promotion tools, particularly when they carry the weight of institutional endorsement by the WHO. 5. Ensuring inclusivity and equity. Conversational AI holds promise for advancing health equity by offering 24/7 access, multilingual support, and adaptable formats. S.A.R.A.H. is available in eight languages, can be accessed via text or voice, and requires no app download, features that make it broadly accessible. However, our focus groups highlighted usability barriers for some groups, including elderly users and people with disabilities, who may struggle with its interface. Personalization remains a challenge. S.A.R.A.H.’s initial responses often assume users are neurotypical and from high-income contexts; tailored advice requires explicit prompting. This risks alienating users with different needs or backgrounds. Bias is an inherent risk in AI health tools.18 While S.A.R.A.H. appears designed to offer culturally specific information, such as regional foods, details on training data and mitigation strategies are lacking. Without transparency on bias checks or safety audits, it is difficult to assess how effectively WHO is addressing equity concerns, underscoring the need for clearer safeguards in future implementations. 6. Responsive and sustainable AI. Responsive AI requires ongoing monitoring to ensure quality and safety in real-world use. For S.A.R.A.H., we found no evidence of active monitoring or mechanisms for users to flag issues, raising questions about how WHO oversees performance. Outsourcing development to private firms may further limit WHO’s ability to audit or adapt the tool over time. Sustainability also means aligning AI with broader health system goals and minimizing environmental impact. It remains unclear how S.A.R.A.H. integrates with existing health infrastructures or whether it contributes to sustainable healthcare delivery. To date, no clear standards or strategies have been outlined to ensure AI health tools like S.A.R.A.H. support long-term public health priorities. Taken together, our findings illustrate a clear AI governance gap. Chatbots like S.A.R.A.H. operate outside established regulatory categories and rely on a patchwork of privacy policies and contracts across multiple actors, each under different legal regimes. This fragmented approach blurs accountability, complicates consent, and weakens pro-tection for sensitive health data. As genAI becomes embed-ded in public health strategies, building coherent, enforceable governance frameworks is essential to safe-guard trust and equity. The way forward AI chatbots for public health promotion are advancing rap-idly, but governance frameworks have not kept pace. Our analysis of S.A.R.A.H. reveals critical gaps between ethical principles and real-world implementation, particularly con-cerning privacy, transparency, accountability, equity, and sustainability. Importantly, this case study is not just about one tool; it serves as a bellwether for the broader field of AI-driven health promotion. Why does this matter? First, S.A.R.A.H. is a flagship initiative by the WHO, a globally trusted authority in health. Its launch signals that genAI is moving from experimental to mainstream use in public health. Second, because the WHO was a key architect of global ethical AI standards, its own struggles to align S.A.R.A.H. with those principles expose fundamental governance and technical challenges that will confront many of S.A.R.A.H. shows, these principles remain aspirational without clear accountability, legal standards, and binding obligations. In this light, we recommend the following policy actions, summarized in Table 2 below: Establish clear legal classifications: Governments 1. and international health organizations should define when AI health chatbots qualify as medical devices 5. Promote inclusivity and mitigate bias: Developers must use diverse, representative datasets and ensure design choices (e.g. language options, accessibility needs.21 features) accommodate varied user Regular independent audits should be mandated to detect and correct bias. 6. Integrate AI into broader health systems: AI chatbots should be viewed as supplements to, not. substitutes for, human-led care. Public health strategies should explicitly define the role of AI tools and ensure they are embedded within existing care pathways and community initiatives.22 7. Support continuous oversight and sustainability: AI chatbots should only be deployed after clear safeguards are established, including protocols for data protection, content accuracy, usability testing, and escalation pathways for users in crisis. Independent panels should be established to monitor AI chatbot performance in real time,23 and environmental standards for digital health tools should be developed to minimize carbon impact. Promising initiatives are emerging. A recent example is a chatbot for maternal health developed in southern Africa, trained in Sesotho, Shona, Ndebele, and English, to expand access to reliable maternal health information in underserved, resource-constrained communities.24 In Australia, researchers co-designed a chatbot with people waiting for eating disorder treatment to deliver single-session ther-apy.25 These examples suggest that effective, ethical deployment is possible if governance keeps pace with innovation. Our analysis of S.A.R.A.H. illustrates the stakes. This case shows that even the world’s leading health authority struggles to meet its ethical commitments when deploying cutting-edge AI. For researchers and policy-makers alike, S.A.R.A.H. provides an instructive example of both the opportunities and the pitfalls of using genAI in public health. It underscores the need to move beyond broad principles and put enforceable, well-designed governance structures in place. Limitations. This article is a case study that uses the WHO’s genAI chatbot S.A.R.A.H. as a lens to explore the legal, ethical, and societal implications of AI chatbots for promoting healthy habits. As such, it does not aim to evaluate the chatbot’s clinical effectiveness or provide a comprehensive technical assessment of its design. The focus groups discussed here form part of a larger pre-registered study of adolescents’ experiences with mental health and health-related apps; this paper draws only on the subset of comments related to S.A.R.A.H. and presents them descriptively rather than through a full thematic analysis. This approach reflects our intention to use qualitative user feedback as illustrative context for a normative analysis rather than as standalone qualitative findings. Because this work is centered on legal, ethical, and governance considerations, its conclusions are not intended to be generalizable beyond the case of S.A.R.A.H. or to offer definitive judgments on all health chatbots. Instead, the case study provides insight into broader challenges in the design, regulation, and implementation of AI-driven health promotion tools, particularly those endorsed by trusted institutions like the WHO. Future research should comple-ment this perspective with formal qualitative analyses of user experiences, empirical studies on chatbot safety and equity, and comparative legal analyses of regulatory approaches across jurisdictions. Conclusions. AI chatbots for health promotion offer new opportunities to make health information more accessible and responsive. They provide scalable, around-the-clock advice in multiple languages and formats, which could benefit underserved populations. This case study of S.A.R.A.H. provides exploratory insights into both the promise and the risks of using these tools to promote healthy habits. While our findings are based on a single chatbot and a descriptive subset of focus group data, they highlight important legal, ethical, and societal considerations, including concerns about privacy, transparency, accountability, equity, and sustainability. As a high-profile prototype backed by a trusted global health authority, S.A.R.A.H. provides enduring lessons for how genAI chatbots should be governed, tested, and deployed in health promotion. To ensure AI chatbots contribute meaningfully to public health, stronger governance, clear legal requirements, and robust oversight are essential. Ethical principles must be translated into enforceable rules, and AI tools must be care-fully integrated into public health strategies that center human expertise and community needs. Ongoing monitoring, transparency, and accountability are critical to keeping these tools safe and effective. If developed responsibly, AI chatbots can help close health gaps and promote better out-comes for diverse populations. However, as S.A.R.A.H. makes clear, there is much work to be done to ensure these tools meet their promise rather than deepen health inequi-ties. Now is the time to build solid legal and ethical founda-tions before these technologies become more deeply entrenched. Acknowledgments. Authors thank Janneke van Oirschot and Luna Willems for their assistance in the preliminary research leading to this publication. Ethical considerations The focus groups were approved by the Human Research Ethics Committee TU Delft. Consent to participate Participants provided written consent to participate in the focus groups. Contributorship HvK did conceptualization, data curation, writing the original draft and writing the review and editing. JG did technical analysis, writing: review and editing. NO did data curation, writing: review and editing. CF did conceptualization, data curation, writing: review and editing. 10. Bak M and Chin J. The potential and limitations of large language models in identification of the states of motivations for facilitating health behavior change. J Am Med Inform Assoc 2024; 31: 2047–2053. Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: CF was supported by the “High Tech for a Sustainable Future” capacity building program of the 4TY Federation in the Netherlands [no grant number] and a Convergence Healthy Start Sprint Grant [grant no: HSF245]. The funders had no role in the content of the manuscript. HvK was supported by the Scientific Exchange Grant of the Swiss National Science Foundation (SNF) [grant no: 234814]. Other authors received no funding for this research.