Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Home Assistant Spies on Your Kids
Transcript
- Lucas: So there's this story from last month that I can't shake. A family in Portland — the Millers — bought a popular smart speaker to help with their toddler's bedtime routine. White noise, lullabies, that kind of thing. Luna: I remember this. It recorded a private conversation and sent it to one of their contacts, right? Lucas: Exactly. The mom was talking to her husband about their kid's doctor's appointment — nothing sensitive, really — and the speaker interpreted her words as a command to send a message. So it recorded the whole minute-long conversation and sent it to an employee at the mom's office. Luna: And that employee just got a random audio file of her boss's family chat. No warning, no context. Lucas: Right. The company said it was a 'false activation' — a one-in-a-thousand error. But here's the thing: for the Millers, it's a one-in-one violation of their privacy. And they have a young child. Luna: Which raises a bigger question. We're putting these devices in our kids' rooms, but we have no idea what they're actually hearing, storing, or sharing. Lucas: And that's the angle I want to drill into today. Not the general smart home privacy debate — we've covered that on other shows — but specifically how these devices interact with children. Because the stakes are different. Luna: How so? Lucas: Children can't consent to data collection. They don't understand that a conversation with their mom might be recorded and stored on a cloud server. And the data is incredibly intimate — voice biometrics, sleep patterns, emotional states. Luna: There's also a legal framework here. The Children's Online Privacy Protection Act — COPPA — applies to kids under 13. But it was written in 1998, long before conversational AI existed. Lucas: Right. COPPA requires companies to get parental consent before collecting personal information from kids. But how do you get consent from a parent when the device is always listening? The parent bought it, set it up, but the data collection happens automatically. Luna: And the companies argue they aren't collecting data 'from children' — they're collecting data from the device, which happens to be near a child. Lucas: That's exactly the loophole. The Federal Trade Commission has fined companies like Amazon and Google for violating COPPA. In 2023, Amazon paid a $25 million fine over Alexa's data practices with kids. But fines are a cost of doing business when the revenue from voice commerce is in the billions. Luna: Let's talk about the technical side. How do these devices even work when they hear a child's voice? Lucas: So most smart speakers use what's called 'keyword spotting' — they're constantly running an on-device model that listens for a wake word. That part is usually local and doesn't record. But once the wake word is detected, the audio is sent to the cloud for processing. Luna: And that cloud processing is where the privacy risk lives. The company gets a recording of everything said after the wake word — and sometimes before, due to buffering. Lucas: Exactly. And those recordings are often reviewed by human annotators to improve the AI. Amazon and Google both admitted to having contractors listen to recordings, including ones that contained private moments. Luna: Which is especially concerning when the voices are children's. Kids don't modulate their speech around a device. They'll ask it questions, argue with it, say things they wouldn't say to a stranger. Lucas: And there's research showing that children as young as three treat these devices like social actors. They say 'please' and 'thank you' to Alexa. They don't understand it's a microphone connected to a corporation. Luna: So what can parents actually do? I mean, beyond not buying one. Lucas: There are some practical steps. You can review and delete voice recordings in the device's settings. You can disable the microphone when you don't need it. But the default is always on, always listening. Luna: And that's the UX problem. Privacy is possible but not convenient. Most people won't dig into settings. Lucas: Right. And the companies design it that way. They want the frictionless experience. So the burden falls on the parent. But there's a broader systemic issue too. Luna: Which is? Lucas: The lack of a clear regulatory framework for voice data specifically. COPPA covers online data, but voice is a different category — it's biometric, it's contextual, it's hard to anonymize. The European Union's GDPR is stronger, but even there, enforcement is spotty. Luna: And in the US, there's no comprehensive federal privacy law. So it's a patchwork of state laws — California, Virginia, Colorado — and sectoral regulations. Lucas: Which means the companies are effectively writing their own rules. And they have a financial incentive to collect as much data as possible, because voice data is incredibly valuable for training AI, for targeted advertising, for building user profiles. Luna: Speaking of which — and maybe this ties in — we've been able to keep this show ad-free and independent because of listener support. It's actually a small group of people who chip in monthly through buy me a coffee dot com slash fexingo. That's what lets us dig into stories like this without worrying about sponsors. Lucas: Yeah, it's a good reminder that independent tech journalism depends on listeners who find value in it. So thanks to anyone who's part of that. Luna: Alright, back to the kids and the speakers. I want to bring up one more case — this one from Germany last year. Lucas: Oh, the one where the speaker recorded a couple's fight and the husband used it in divorce proceedings? Luna: Exactly. That case had a child present too. The device captured not just the adults' argument but also the child's reactions. And the data was subpoenaed. So now you have a child's emotional response to a parental conflict, recorded and entered into evidence. Lucas: That's a legal nightmare. And it shows how voice data can be weaponized. The companies say they protect user privacy, but they comply with lawful requests. And the law hasn't caught up to the fact that a smart speaker is basically a surveillance device. Luna: So what's the solution? Is it better tech — like on-device processing that never sends audio to the cloud? Lucas: That's one path. Apple's Siri does more on-device processing now. And there are startups building privacy-first voice assistants. But the incumbents have a data advantage, so they're reluctant to give up the cloud model. Luna: Another path is regulation. The proposed American Privacy Rights Act would create national data privacy rules, but it's stalled in Congress. Lucas: Right. And even if it passes, enforcement is the question. The FTC is underfunded and outgunned. So we're left with a choice: either trust the companies to do the right thing — which history says is naive — or change our behavior as consumers. Luna: Or push for transparency. Like, wouldn't it be powerful if every smart speaker had a mandatory privacy label — like a nutrition label — that told you exactly what data it collects, how it's used, and how long it's stored? Lucas: That's actually a proposal from the FTC in 2024. A 'privacy label' for IoT devices. But it's voluntary so far. No major manufacturer has adopted it. Luna: So the status quo remains: we invite a microphone into our children's rooms, trust a corporation to behave, and hope for the best. Lucas: And that hope is the problem. Because the incentives are misaligned. The company profits from data, the parent gets convenience, and the child's privacy is the externality. Until that equation changes — through regulation, through market pressure, through informed consumer choice — we're going to keep seeing stories like the Millers'. Luna: And every one of those stories is a reminder that AI ethics isn't abstract. It's happening in living rooms, in bedrooms, in the conversations we thought were private. Lucas: Yeah. And the most vulnerable are the ones who can't say no.