Latest / Future of Work Tech with Fexingo: Remote Tools, AI Productivity, and Workplace Software / How AI Is Automating Your Internal Knowledge Security
Transcript
- Lucas: There's a scene playing out inside a lot of companies right now that most employees don't even know about. Their internal AI chatbot — the one that answers questions about HR policies, engineering docs, sales playbooks — that thing is a security risk nobody planned for. Luna: You mean because it's pulling from everything, and accidentally surfacing stuff people shouldn't see? Lucas: Exactly. A Fortune 500 company — I'll keep them unnamed — discovered last quarter that their internal AI search tool had been answering questions like 'What's the salary range for a senior software engineer?' with real, live payroll data. Not just to HR. To anyone who typed the question. Luna: Wait — to any employee? Or to anyone who could access the chatbot at all? Lucas: Any employee who had access to the chatbot — which, in this case, was the whole company. The tool was indexing the internal wiki, and salary bands were sitting in a spreadsheet that someone on the compensation team had linked to a page about 'Total Rewards Philosophy'. The AI didn't know that spreadsheet was supposed to be restricted. Luna: So the AI just... served it up. No access check. Lucas: Correct. This is the fundamental tension: we've spent the last two years deploying AI to make knowledge more discoverable, but the access-control infrastructure hasn't caught up. The AI is trained on the full corpus, but the permissions model is still designed for humans clicking folders. Luna: What did the company do after they found the leak? Lucas: They shut down the chatbot for three weeks, restored it with a completely reworked permission layer. But here's the thing — they only discovered the leak because an employee in IT ran a routine audit of the chatbot's query logs and saw the salary question pop up with a response. That's not a scalable discovery mechanism. Luna: So the problem is that AI is creating a new kind of data exposure that's invisible until someone stumbles on it. Lucas: Exactly. And the vendors are starting to respond. Glean, the enterprise AI search company, announced in June a feature they call 'access-aware summarization'. It essentially runs every query through the same permissions model that would govern a manual file search — if you wouldn't see the original document, the AI shouldn't reference it in its answer either. Luna: Is that hard to implement? It sounds like it should be table stakes. Lucas: It's harder than it sounds because AI embeddings don't carry metadata about who owns them. Glean had to build a real-time permission boundary check that sits between the retriever and the large language model. Coveo, another player in this space, is doing something similar with what they call 'contextual access control' — basically tagging every chunk of text with an access label at indexing time. Luna: So the indexing process itself has to become security-aware. Not just the query layer. Lucas: Right. And there's a second-order problem: even if the AI only retrieves documents you're allowed to see, it can still synthesize information across those documents in ways that reveal something you shouldn't know. Say you're in marketing, and you can see the product roadmap doc and the headcount planning doc separately. The AI might combine them to say 'The Q3 feature launch is delayed because the engineering team is short-staffed.' That inference might be accurate — but it's not information you were supposed to have as a cohesive insight. Luna: So the AI is effectively creating new knowledge by association — and that knowledge might violate compartmentalization. Lucas: That's exactly the phrase: associative leakage. A few security researchers at MIT published a paper in April that demonstrated this — they called it 'cross-document inference attacks' on enterprise AI. They showed that even with strict per-document permissions, an AI could answer questions like 'Which departments are planning layoffs?' by stitching together hints from meeting summaries and budget spreadsheets that individually look benign. Luna: So how do you defend against something like that? Do you limit the AI's context window, or do you have to redesign the entire knowledge architecture? Lucas: Both, honestly. Some companies are moving toward a 'knowledge tier' system — you tag documents by sensitivity level, and the AI is only allowed to draw from a single tier per query. So a question about vacation policy only touches Tier 1, general-access docs. A question about strategic planning only touches Tier 3, which requires executive-level authentication. The AI never mixes tiers in one response. Luna: That's a lot of overhead. Is anyone actually doing this at scale? Lucas: A few. I know of one large financial services firm — again, unnamed — that rolled out a tiered AI system across their compliance division. They have four tiers, and the AI literally refuses to answer if the query would require blending information from different tiers. It says 'I can't provide an answer that combines multiple sensitivity levels.' Employees found it frustrating at first, but it's been running for six months with zero data incidents. Luna: Interesting — they accepted a worse user experience in exchange for security. That's a tradeoff a lot of companies aren't willing to make. Lucas: Not yet. But I think they'll have to. The incident I mentioned at the top — the salary leak — that's going to become more common. And regulators are starting to pay attention. The SEC has indicated that it considers ai driven data exposure a subset of insider information risk. If a company's AI discloses material non-public information to the wrong employee, that creates a trading risk. Luna: Right — if someone in marketing learns about a pending acquisition through the AI even though they're not on the deal team, they could trade on that. Lucas: Exactly. And the company is liable. So the CFO and the general counsel are now getting dragged into AI procurement conversations that used to be entirely it driven. Luna: Speaking of which — who's actually winning in this space? Are there startups focused purely on AI security? Lucas: There's a small but growing category called 'AI access governance'. Companies like StrongDM and Apona — though Apona just rebranded — they're building tools that sit between the AI and the data sources and enforce policy in real time. It's essentially an API gateway with a policy engine that understands both semantic queries and access control lists. Luna: So it's like a firewall for AI queries. Lucas: Exactly. And the interesting thing is that these tools are being adopted faster by mid-market companies than by large enterprises. The logic: a company with two thousand employees can't afford a dedicated security team reviewing every AI response, so they automate the policy enforcement. The Fortune 500 firms are still trying to retrofit their existing permissions, which is slower. Luna: Makes sense. The smaller guys have less legacy infrastructure to deal with. Lucas: Right. And they also have fewer documents, so tiering is less painful. One startup CEO I talked to said his company of 300 people implemented a four-tier system in two weeks. The AI went from 'answer anything' to 'answer only what you should know' overnight. Luna: What about open-source tools? Is there a community effort around this? Lucas: There is. There's a project called OPA — Open Policy Agent — that some companies are extending to cover AI retrieval. It's not purpose-built, but it's flexible. A few security engineers have built 'OPA for AI' plugins that let you write rules like 'If the query contains a department name and the user is not in that department, deny.' It's early, but promising. Luna: So the takeaway is: don't deploy an internal AI chatbot without first auditing what it can see. Lucas: Absolutely. And even then, assume it will find connections you didn't intend. The safest approach is to limit the AI's scope from day one — don't index everything, index only what you're comfortable being asked about. Then expand slowly. Luna: Quick honest thing — a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many episodes. We don't run ads, so that small group is what keeps us going. Lucas: Yeah, and we're grateful for it. The show stays ad-free because of that support. Now back to the security angle — I think the next frontier is going to be auditing AI responses after they're delivered. Not just blocking bad queries, but detecting when a response turned out to be more sensitive than it should have been. Luna: Post-hoc analysis. That would catch the associative leakage that the real-time filters miss. Lucas: Exactly. And I know of at least two startups working on that right now. They're watching the AI's output the way a compliance officer reviews trader emails. It's a whole new category of surveillance. Luna: And a whole new set of tradeoffs between transparency and privacy. Lucas: Always. But the baseline is clear: if your knowledge graph is ai accessible, you need to know exactly what's in it and who is asking. That's the work.