Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Assistant Hallucinates Your Work History
Transcript
- Lucas: So imagine you are a senior marketing manager at a Fortune 500 company. You have been using an AI productivity assistant for six months — it drafts your emails, summarizes meetings, even helps update your professional bio for an internal directory. One morning, you open that bio and it says you spent two years at a direct competitor, a job you never held, with responsibilities you never performed. Luna: That is nightmare fuel. Did this actually happen? Lucas: It did. Back in March 2026, a senior marketer at a large consumer goods firm noticed exactly that. Their AI assistant had hallucinated an entire work history entry — complete with a plausible-sounding job title and a tenure. The employee had never worked at the competitor. They had no idea how the AI generated it. Luna: And the company saw it in the internal directory before they could correct it? Lucas: Exactly. Someone in HR flagged it during a routine audit. That triggered an internal investigation — the employee had to prove they had never worked there. It took weeks to clear up. Meanwhile, the AI kept running for everyone else, generating who knows what. Luna: Look, I'm a heavy user of these tools myself. And episodes like this remind me why listener support matters for keeping this show independent and ad-free. If today's tech conversation gave you something usable, a couple of dollars a month is genuinely what keeps these going — buy me a coffee dot com slash fexingo. Lucas: Yeah, exactly. It's a small thing that makes a real difference for us. So back to that marketer — the scary part is that the AI didn't just invent a random company. It invented a plausible rival firm that the person could conceivably have worked at, given their industry. Luna: That is a whole different level of hallucination. Not just nonsense, but believable nonsense. Lucas: Right. And this is where the problem gets structural. These productivity tools are trained on vast public datasets that include resumes, LinkedIn profiles, corporate directories. The AI learns the patterns of typical career trajectories. When it fills in a blank — say a missing year — it generates something statistically likely, even if completely false for that individual. Luna: So it's not a glitch. It's a feature of how these models work. Lucas: Exactly. And the companies deploying them are often unaware of the risk. They see a tool that saves time, not a tool that can fabricate professional history. The vendor might have tested for toxic outputs or bias, but hallucination in a workplace context is rarely on the safety checklist. Luna: What did the company do after the incident? Lucas: They paused the tool, audited every generated profile, and found that roughly two percent of employee bios contained inaccuracies. Some were minor — wrong graduation year. But a handful were material, like the false competitor job. They ended up disabling the auto-fill feature entirely. Luna: Two percent might sound small, but in a company of fifty thousand employees, that is a thousand bios with potentially career-damaging errors. Lucas: And that is just one company. The broader point is that we are seeing a new category of AI risk: digital forgery not by malicious actors, but by the tool itself. And no one is really sure whose responsibility it is when the AI fabricates something about you. Luna: Is it the vendor who trained the model? The employer who deployed it? Or the employee who trusted it? Lucas: Legally, it is a mess. Current liability frameworks assume someone intentionally caused harm. Here, no one intended anything. The model just did what it was trained to do — generate plausible text. And the employer thought they were buying a productivity boost, not a liability generator. Luna: There is actually a precedent from the early days of autocomplete. In 2018, Google's Smart Compose suggested a user's email said 'I love you' to someone they did not know. But that was an isolated, easy to spot error. This is a persistent, embedded falsehood in an official internal record. Lucas: Good comparison. The scale is different, and so is the stakes. A wrong autocomplete is awkward. A wrong career history in an internal database can affect promotions, background checks, even compliance with non-compete agreements. Luna: So what is the fix? More human oversight? Lucas: That is the obvious first step. Every ai generated professional summary should be reviewed by the employee before publication. But that introduces friction, which defeats the purpose of a productivity tool. The more structural fix is better testing for hallucination in high-stakes domains. Luna: Are vendors doing that? Lucas: Some are starting to. A few of the major AI labs have published research on reducing hallucination rates, but the benchmarks are usually based on trivia questions, not professional history. There's no standard test that says 'generate a resume for a real person and check if every line is true.' Luna: And even if the rate drops to 0.1 percent, in a large organization that is still dozens of false records. Lucas: Exactly. The second-order problem is trust erosion. Once employees know the AI can invent things about them, they stop using it. That defeats the whole investment. The company in our case saw adoption drop from 60 percent to 15 percent after the incident. Luna: So the productivity gain is negated by the trust loss. Lucas: Right. And this is where I think the conversation needs to go beyond just technical fixes. There is an ethical dimension to deploying a tool that can generate false personal data without the user's knowledge or consent. Luna: It almost feels like the AI is gaslighting the workplace. Lucas: That is not too strong a word. If your official company bio says something false about you, and the system that created it is trusted by management, then you are in a position where you have to prove a negative — that you did not do something the AI says you did. Luna: Which is extremely hard, especially if the fabricated job is at a company that no longer exists or has poor record-keeping. Lucas: Exactly. And that is the case for one of the other false records they found. A woman in her fifties had a fabricated role at a dot-com that went bankrupt in 2001. There was no way to verify or disprove it. She had to dig up old tax returns. Luna: That is absurd. The burden should be on the deployer, not the employee. Lucas: I agree. And I think we are going to see regulatory pressure build around this. The EU's AI Act classifies employment-related AI as high-risk. Under that framework, companies deploying tools that generate professional profiles would need to conduct conformity assessments, ensure human oversight, and provide transparency. Luna: But the EU Act is not fully in force yet, and in the US there is nothing comparable at the federal level. Lucas: Right. So for now, it is largely self-regulation. And self-regulation tends to be reactive — fix the problem after it blows up. The marketer case blew up internally, but it did not make headlines. How many similar cases are happening quietly? Luna: We do not know. And that is part of the problem. There is no reporting requirement for AI hallucinations in the workplace. Lucas: One thing that might help is something called 'source grounding' — where the AI is forced to cite the specific source for each claim it makes about a person. If the source does not exist, the field stays blank. Luna: That sounds like a basic technical requirement. Why is it not standard? Lucas: Partly because it adds latency and cost. Grounded generation requires the model to retrieve and verify information from a trusted database in real time. Most productivity tools are designed for speed and low cost. Grounding slows them down and increases compute spend. Luna: So it is a trade-off between speed and accuracy. Lucas: Yes. But for professional information, I would argue accuracy should win. If your AI is going to speak for you, it should only speak facts you have authorized. Luna: That is a good principle. I wonder if we will see a certification label — like 'this AI is certified not to hallucinate your resume.' Lucas: I think we will. There are already startups working on audit tools for ai generated content. They run statistical checks to flag probable fabrications. But adoption is slow because it cuts against the narrative that AI is ready for prime time. Luna: And the vendors do not want to admit their products can make up things about people. Lucas: Exactly. It is a marketing problem as much as a technical one. No one wants to sell a product with a warning label that says 'may invent false career history.' And yet, that is the honest reality. Luna: So what is the takeaway for a listener who uses these tools at work? Lucas: First, never auto-publish anything an AI generates about you without reading it carefully. Second, if your company deploys such a tool, ask what testing has been done for hallucination in the workplace context. Third, push for a system that lets you approve or correct any ai generated record before it goes live. Luna: And if your AI already published something false? Lucas: Document everything. Save a screenshot. Request a correction in writing. And if it involves a material fact — like a job you never held — consider escalating to HR or legal. You should not have to carry the burden of disproving a machine's fiction. Luna: I think the deeper issue here is that we are trusting these tools too fast, without understanding their failure modes. Lucas: That is the thread across so many episodes of this show. The technology is impressive, but it is not magic. It has specific failure modes, and we need to plan for them. Hallucinated work history is one of those failure modes that hits at the core of professional identity and trust. Luna: And it is one that, unlike a wrong email autocomplete, can have lasting career consequences. Lucas: Exactly. So the next time you let an AI fill in your bio, just remember: it might be writing a story about you that never happened. And you are the one who will have to answer for it.