Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / AI That Runs on Your Phone Changes the Privacy Math
Transcript
- Lucas: You know that feeling when you type something into a chatbot and a little voice in your head goes — huh, I wonder who's reading this on the other end. Luna: Yeah, every time. Especially if you're asking something personal, like a health question or a financial concern. Lucas: Right. That tension is the central assumption behind a lot of AI ethics debates — your data leaves your device, goes to a server, and you trust the company not to misuse it. But there's a growing movement that says: what if the model never leaves your phone in the first place? Luna: So the AI runs locally. No cloud call, no data transmission. That's the pitch behind Apple's on-device foundation model and Google's Gemini Nano. Lucas: Exactly. And it changes the privacy math fundamentally. Because if your data never leaves your device, a lot of the classic AI harms — data breaches, surveillance capitalism, unauthorized training on your inputs — they just become impossible. Luna: But does it fix everything? Or does it just move the problem somewhere else? Lucas: That's the question. Let's start with what's actually happening in the hardware. Qualcomm's latest Snapdragon chips, the 8 Gen 4 and the X Elite for laptops, include a dedicated neural processing unit — an NPU — that can run large language models locally. We're talking models with seven billion parameters, running on a phone, without a network connection. Luna: Seven billion parameters — that's about the size of LLaMA 2, the open-source model from Meta. A few years ago that required a data center. Lucas: Exactly. Apple has its own Neural Engine, and the iPhone 17 Pro — which launched last fall — can run a model with about three billion parameters entirely on device. That's enough for summarization, writing assistance, even image captioning. And Apple has been very explicit: they designed it so that no user data ever reaches Apple's servers for these tasks. Luna: That is a huge shift from the standard ai as a service model. But does that mean the model itself is safe? I mean, the model weights are still stored on the phone. If someone gets physical access — Lucas: Right. That's one of the new risks. With a cloud model, the company can update the model centrally, patch biases, fix safety filters. Once a model is on your phone, it's frozen. If there's a bias in the local version — say, it's less accurate for non-native English speakers — that bias persists until the next software update, which might be months away. Luna: And the model itself is still trained on a dataset. That dataset could have biases. Just because the inference happens locally doesn't mean the training data was fair. Lucas: That's a key point. On-device AI is a privacy improvement for inference, but not necessarily for training. The model Apple ships to your phone was trained on a massive dataset — and we don't know exactly what's in it. Apple publishes a transparency report, but it's high-level. They say they use 'publicly available data' and 'licensed data' — but they don't release the full dataset for audit. Luna: So the privacy win is for the user's input data, but the model itself is still a black box. And if the model has a blind spot — say, it's worse at understanding African American Vernacular English — that blind spot is now on every iPhone, and Apple can't fix it until the next iOS update. Lucas: Right. And that's not hypothetical. Researchers at Stanford found that several large language models, including some used in consumer products, perform worse on text written in AAVE. If that model is running locally, the user never gets the better version that a cloud model could update in real time. Luna: Let's talk about a concrete use case where this really matters: mental health chatbots. There's a company called Koko — they had a controversy a few years ago where they used GPT-3 to generate mental health support messages without telling users. But there's a newer generation of apps that run entirely on-device, like the one from a startup called Limbic. Lucas: Limbic's model is interesting. They specifically built a small language model that fits on a phone, designed for cognitive behavioral therapy exercises. The selling point is total privacy — what you type never leaves the device. Not even Limbic's engineers can read it. That's powerful for something like mental health, where the stigma of someone else seeing your thoughts is a real barrier to seeking help. Luna: But here's the ethical question: if the model is completely private, who ensures it's safe? If a user types something indicating self-harm, the model can't alert a human. It's not connected to anything. So either the model has to handle that entirely on its own — which is a huge responsibility for a small language model — or it just doesn't handle it, which is dangerous. Lucas: And that's the trade-off. Limbic's response is that their model is trained to recognize crisis language and, when it does, it can trigger a local intervention — like displaying a suicide hotline number — but it cannot notify a therapist. That's by design, for privacy. But it also means the user is alone with the model in a moment of crisis. Luna: So the privacy win comes at the cost of human oversight. And that's a very different risk profile than a cloud model where a human could step in. Lucas: Precisely. And that's the conversation we need to have about on-device AI. It's not a simple upgrade. It's a different set of trade-offs. Less risk of data breaches, less surveillance, but more reliance on the model's training data being correct and its safety systems being robust. And less ability to intervene after the fact. Luna: It reminds me of the early debates about encryption. You have the 'no back doors' camp — and I'm generally with them — but then you have law enforcement saying, what if a terrorist uses end to end encryption? The same logic applies here: perfect privacy means no one else can help. Lucas: That's a really good parallel. And the difference is that with encryption, the data is unreadable to everyone except the parties communicating. With on-device AI, the model itself is the interpreter. And if the model has a bug, or a bias, or a blind spot, that interpreter is the only one reading the message. Luna: So what do we do? Regulate on-device models differently than cloud models? Lucas: That's the direction some policymakers are thinking. The EU AI Act has a provision that applies different rules to 'general purpose AI models' — they're still figuring out the specifics. But one idea is that any model deployed on a consumer device should have to pass a safety audit before release, because once it's out there, it's hard to patch. Luna: Apple actually does something like that. They review each third-party app that uses their on-device ML frameworks. But the built-in models — like the ones powering Siri or the Photos app — those are Apple's own, and they're not independently audited. Lucas: Right. And that's the gap. Independent auditing of the model's behavior on representative data — including edge cases — should be a requirement before a model gets shipped to millions of devices. Because once it's there, you can't recall it like a bad batch of food. You have to push an update and hope people install it. Luna: And not everyone updates. There are still people on iOS 17 who never updated to 18. So a model with a bias could persist on those devices indefinitely. Lucas: This is one of those topics where the technology is moving faster than the safeguards. And it's worth pausing to appreciate that the hosts of this show — we've been talking about AI ethics for 22 episodes now — we're able to do this because we don't have to sell ad space or chase sponsors. We deliberately keep the show ad-free. If you find these conversations useful, and you want to support that choice, the link is buy me a coffee dot com slash fexingo. It's a small gesture that helps us keep the focus on the substance rather than the commercial break. Luna: Yeah, and it really does make a difference. Every little bit helps us keep digging into these nuanced angles that might not get covered on a bigger network. Lucas: So back to the core question: on-device AI is coming to more and more products. Apple, Google, Qualcomm, Samsung — all of them are pushing this as a privacy feature. And it is, for inference. But we need to be clear-eyed about what it doesn't fix. Luna: And what about the data used to train the model in the first place? If the model was trained on biased data, the on-device version inherits that bias. And there's no way to audit it after deployment because the model is local. Lucas: So one proposal is 'federated learning with differential privacy' — where the model can be improved over time using anonymous updates from users' devices, without ever seeing their raw data. Apple uses something like that for QuickType keyboard suggestions. It works, but it's slow. And it still requires the company to have a server coordinating the updates, which reintroduces some trust. Luna: So there's no perfect solution. But I think the key takeaway is: don't assume 'runs on my phone' means 'no ethical risk.' It means a different kind of risk. Lucas: Exactly. And as consumers, we should ask the same questions we ask about cloud AI: who trained it, on what data, and what happens when it gets something wrong? The answer might be different, but the question is the same. Luna: And for the mental health case, I think the ethical bar should be higher. If a model is going to interact with vulnerable users, and it's designed to be completely private, then it needs to be extraordinarily robust. And I'm not sure any current model meets that bar. Lucas: That's the frontier. And it's why we cover this stuff — because the technology is here, the decisions are being made, and the public debate is still catching up. On-device AI is not a magic bullet. It's a new tool with new trade-offs. And we need to be honest about both.