Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Copilot Broadcasts Your Private Chat
Transcript
- Lucas: So back in January of this year, a major telecom company rolled out an AI copilot for their customer support agents. It was supposed to summarize call histories, surface account details, suggest responses — the kind of thing that sounds great on paper. Luna: Right, until it started showing one agent's screen the private chat transcripts of a completely different customer. Lucas: Exactly. A customer called in about a billing issue, and the copilot pulled up not that customer's data, but a previous caller's entire conversation history — including their home address, social security number, and a detailed complaint about a medical debt. Luna: That is the stuff of nightmares. And it wasn't a hack — it was just the AI doing what it was built to do, poorly. Lucas: And that's what makes it interesting from an ethics standpoint. The system wasn't compromised by an attacker. It was a design failure in the retrieval mechanism. Most modern AI copilots use something called retrieval-augmented generation, or RAG. Luna: RAG — where the model queries a vector database of documents to find relevant context before generating a response. Lucas: Right. So the copilot is supposed to pull only the current customer's records. But in this case, the vector similarity search grabbed a different customer's data because the embeddings were too similar — both calls mentioned 'billing issue' and 'medical expense.' The system didn't have a hard access-control layer on top of the retrieval. Luna: So it's not that the AI 'hallucinated' — it retrieved real data, just the wrong data. That feels almost worse, because it's a real privacy leak, not a fabrication. Lucas: Exactly. And the scary thing is how hard it is to catch. The agent who saw the leak had to report it manually. The copilot's own logging didn't flag the retrieval mismatch because, technically, the retrieval succeeded — it just returned the wrong vector. Luna: If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. Lucas: Yeah, it's that simple. A lot of people tell us they use these episodes in team discussions or training sessions, and that's exactly the kind of thing that keeps us doing it. Luna: So back to the telecom case — once it was reported, what did they actually do about it? Lucas: They took the copilot offline for about three weeks. When it came back, they had added a 'customer ID filter' that explicitly restricts the vector search to documents tagged with the current caller's account number. It's a band-aid, but it's a necessary one. Luna: But here's the thing: that filter only works if the tagging is correct. If the data pipeline mislabels a document, you're back to square one. Lucas: And that's exactly what worries me. The company didn't release a post-mortem, but security researchers who analyzed the incident pointed out that the root cause was a 'shadow leak' — a class of vulnerability that standard red-teaming rarely catches because the tests assume the retrieval is scoped correctly. Luna: So the AI passes the 'adversarial' tests because the tests aren't designed to probe retrieval boundaries — they're designed to probe generation boundaries. Lucas: Right. The industry has spent years worrying about prompt injection — someone tricking the model into ignoring its instructions. But this is almost the opposite: the model follows its instructions perfectly, but the data it's instructed to use is wrong. Luna: I want to go back to the vector similarity problem. The system matched on 'billing issue' and 'medical expense.' That means the embeddings are encoding topic similarity, not identity. So if Customer A and Customer B both talk about medical bills, the AI sees them as nearly identical. Lucas: Exactly. And the embedding models are getting better at semantic understanding, which actually makes this worse. They capture nuance so well that two conversations about 'medical billing disputes' end up in a very tight cluster in vector space. Without explicit identity metadata, the system can't distinguish them. Luna: So the fix isn't just a filter — it's redesigning the retrieval architecture to separate 'what is this about' from 'who does this belong to.' Lucas: And that's a fundamentally different engineering challenge. Some companies are experimenting with 'data isolation layers' — essentially encrypting documents by user ID and only decrypting them after retrieval. But that adds latency and complexity. Luna: It also raises a question: if the AI can't read the data until after retrieval, how does it know which documents to retrieve? You'd need a separate index for the encrypted metadata. Lucas: Some startups are building exactly that — a 'searchable encryption' layer for vector databases. But it's early. Most enterprises are still using off-the-shelf RAG with basic access control lists. Luna: And those ACLs are often bolted on after the fact, because the vector database wasn't designed for multi-tenant isolation. Pinecone, Weaviate, Chroma — they all have 'namespaces' or 'tenants' now, but they weren't built with privacy-first architecture. Lucas: That's the deeper issue. The entire RAG stack — embeddings, vector DB, LLM — was designed for accuracy and speed, not for access control. Privacy is an afterthought. Luna: So what does this mean for regulation? The EU AI Act has provisions for high-risk systems, and customer support copilots probably fall into a gray area. But would this incident trigger a mandatory notification? Lucas: Under GDPR, absolutely. The telecom would have to report a personal data breach within 72 hours. But the AI Act's requirements are more about pre-deployment conformity assessment. The question is whether the copilot's architecture was documented as part of that assessment. Luna: I suspect a lot of companies are going to discover that their RAG systems aren't compliant with Article 14 — the transparency obligation — because they can't explain exactly how the retrieval works at a granular level. Lucas: And they can't audit it either. Most auditing tools check the LLM's outputs for bias or toxicity. They don't check whether the retrieved context actually belongs to the user. Luna: So we have a blind spot in the audit pipeline. And that's exactly where the next scandal will come from. Lucas: I think so. We've seen ai generated content go wrong — deepfakes, misinformation. But the next wave of AI ethics failures might be about data leakage, not about what the model says, but about what it sees. Luna: And unlike hallucination, where you can spot a weird answer, a leakage event looks completely normal to everyone except the person whose data was exposed. Lucas: That makes it harder to detect and harder to litigate. The customer whose data leaked might never know, unless the company tells them. And the company has an incentive not to tell. Luna: Right, because notification triggers investigation, fines, reputational damage. So there's a perverse incentive to quietly patch the bug and move on. Lucas: And that's why we need third-party auditing mandates for RAG systems. Not just for the generative layer, but for the retrieval layer. Companies should have to prove that their vector database enforces per-user isolation. Luna: That's a technical standard that doesn't really exist yet. We have SOC 2, ISO 27001, but they're about general security controls, not about rag specific privacy. Lucas: There's an opportunity here for a new certification. Something like 'RAG Privacy Verified' — a standard that requires testing for cross-tenant leakage, prompt injection resilience, and retrieval boundary enforcement. Luna: Who would do that? Maybe a consortium of AI safety researchers, or a regulatory body like the ICO or the CNIL. Lucas: Or a major cloud provider. If AWS or Azure offered a 'privacy-guaranteed RAG' product with a certification badge, enterprises would flock to it. Luna: But that also creates a new kind of lock-in. Once you build your retrieval pipeline on a proprietary privacy layer, you can't easily switch. Lucas: That's a fair concern. Open-source alternatives like LangChain or LlamaIndex could develop their own open standard. But the key is that someone — anyone — needs to start treating retrieval privacy as a first-class problem, not a patch. Luna: I think that's the bottom line. The telecom incident was a warning shot. The companies that ignore it are going to have a much worse leak in the next twelve months. Lucas: And that leak will be worse not because the technology is more malicious, but because the incentives are misaligned. Speed to deployment, cost savings, user engagement — all push toward looser controls. Luna: So what's the one thing a product manager listening to this should do tomorrow? Lucas: Ask your engineering team one question: 'In our RAG pipeline, can the vector search ever return a document that belongs to a different user?' If they can't answer with a hard 'no,' you have a problem. Luna: And if they say 'we're working on it,' that means the answer is really 'yes, and we haven't fixed it yet.'