Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Writing Assistant Plagiarizes Without You Knowing
Transcript
- Lucas: So there's this case that hit my feed a few weeks ago — late May — that I think gets at something a lot of people using AI writing tools haven't fully reckoned with. Luna: I saw that too. The freelance journalist who realized her AI assistant had copied entire paragraphs from a paywalled Atlantic article? Lucas: Exactly. Her name is Sarah K. — she's a tech writer who covers AI, actually — and she was using one of the major models to help draft a piece about remote work trends. She asked it to 'expand on the rise of asynchronous communication' and it produced three paragraphs that were nearly verbatim from a 2023 Atlantic piece by Derek Thompson. Luna: She caught it because she remembered reading that article. But what if she hadn't? That's terrifying for anyone who produces original work. Lucas: Right. And the model didn't cite it. It didn't say 'this is from an Atlantic article.' It just served it up as if it were original text. That's the core problem — these models are trained on massive corpora that include paywalled and copyrighted material, and sometimes, especially with well-known phrases or specific data points, they essentially regurgitate. Luna: There's a term for that in the research literature — 'memorization.' It's not just paraphrasing poorly; it's actual near-exact recall. Lucas: Right. Researchers at Google and Stanford have shown that large language models can memorize sequences of hundreds of tokens — that's dozens of words — especially when those sequences appear multiple times in the training data. The Atlantic article was widely shared and quoted. So it was probably in the dataset many times over. Luna: And the journalist's prompt wasn't asking for a quote or a source. She was asking for analysis. The model just made a mistake — it treated a copyrighted passage as the right answer to 'expand on this concept.' Lucas: Which is exactly how these models work. They're not thinking 'is this plagiarism?' They're predicting the most likely next word based on patterns. If the pattern is 'asynchronous communication allows teams to' and the most statistically probable continuation is the exact phrasing Derek Thompson used, the model will output it. Luna: But the companies building these tools — OpenAI, Anthropic, Google — they all have usage policies that say users are responsible for ensuring their outputs are original. Is that fair? Lucas: It's the standard legal liability dodge. They say 'our tools are for inspiration, not final copy.' But if you're a freelance writer on deadline, you're not running every output through a plagiarism checker. You trust the tool. Luna: And even plagiarism checkers might not catch it if the source material is behind a paywall and not indexed in their databases. Lucas: Exactly. So there's this accountability gap. The model does the copying, the user unknowingly publishes it, and the original author gets attribution only if someone happens to notice. Sarah K. actually did something smart — she tweeted about it with a screenshot of the output next to the Atlantic piece. It got a lot of attention. Luna: What was the company's response? Did the model provider say anything? Lucas: The company acknowledged it in a support thread. They said it was a 'rare occurrence' and that they were refining their deduplication and citation mechanisms. But rare is cold comfort if you're the one it happens to. Luna: And this isn't just about journalism. Coders using AI assistants have reported getting chunks of gpl licensed code that the model didn't attribute. That's a legal liability if you incorporate that code into a proprietary product. Lucas: There was a class-action lawsuit filed last year against GitHub Copilot over exactly that — reproducing open-source code without the license. That case is still moving through the courts. And it raises the same question: who is responsible when the AI copies? Luna: I think the deeper issue is that we're treating these tools as 'creative partners' when in reality they're just statistical parrots with very large training sets. The hype says 'AI helps you write better.' The reality is 'AI might help you plagiarize without knowing.' Lucas: That's a good way to put it. And it's not just about copyright. It's about trust. If you're a student, a journalist, a marketer — you need to be able to certify that what you produce is your own work. And the AI can't certify that for you. Luna: So what's being done about it? Are there technical solutions? Lucas: There are a few approaches. One is 'differential privacy' during training — adding noise to prevent exact memorization. But that can reduce model quality. Another is post-hoc detection: tools like Originality.ai or GPTZero that scan outputs for signs of memorization. But they're not foolproof. Luna: There's also a startup called Attrib — I think they launched in April — that specifically builds a 'citation layer' into the model so when it produces a phrase close to a known source, it flags it and links back. They claim they can reduce regurgitation by 90 percent. Lucas: I read about them. They're using a retrieval-augmented generation approach but with a twist — they compare outputs against a cached index of copyrighted content before displaying them. It's promising, but it only works if the model provider integrates it. And it adds latency. Luna: Right. So there's a trade-off: speed and convenience versus safety and attribution. Most users will pick speed until something goes wrong. Lucas: And that's the ethical knot. Because the companies have an incentive to keep the product fast and frictionless. Adding a check that slows down every output by half a second is a hard sell to product managers. But from a user perspective — especially a professional one — that half second is worth it if it saves you from a plagiarism accusation. Luna: Or from accidentally violating the terms of your own freelance contracts. A lot of publications now include clauses about 'original content only' specifically because of AI. Lucas: Yeah, I've seen those. They basically say 'you cannot submit ai generated content as original.' But the line between ai assisted and ai generated is fuzzy. If the AI wrote three sentences and you rewrote two of them, is that ai assisted? What if you didn't know it was copying? Luna: That's exactly why this case is so important. Sarah K. was a careful user. She wasn't trying to cut corners. She was using AI the way the marketing materials suggest — as a brainstorming partner. And it still failed her. Lucas: So what's the takeaway for someone listening who uses these tools every day? I think it's this: treat every AI output as a draft from a junior assistant who might have copied from a source you can't see. You have to verify. You have to check. Luna: And that's extra work. But the alternative is risking your professional reputation. Speaking of which — we actually have a small request on that front. If you find value in shows like this that dig into the real-world implications of technology, listener support is what keeps it ad-free and independent. Lucas: Yeah, it's a thing. We don't run sponsor spots, we don't have a network pushing us to soften our takes. That's only possible because some of you chip in at buy me a coffee dot com slash fexingo. It's completely optional, but it means we can spend the time on episodes like this one. Luna: And if today's conversation about AI plagiarism gave you something to think about — that's exactly the kind of thing we want to keep doing. So thanks for considering it. Lucas: Alright, back to the topic. The other thing I want to mention is that there's a regulatory angle here. The EU's AI Act, which is supposed to be fully in force by August, includes requirements for providers to disclose training data and to implement safeguards against memorization. If enforced, that could force a lot of changes. Luna: But enforcement is the question. The EU has limited resources, and the AI Office in Brussels is still ramping up. It might be a year or more before we see actual fines or mandates. Lucas: True. But just having the legal framework creates pressure. Companies like Microsoft and Google are already pre-emptively adding attribution features. Google's 'double-check' in Gemini, for example, lets you verify statements against web sources. It's not perfect, but it's a start. Luna: So we're in this awkward in-between period where the technology is mature enough to cause real harm, but the safeguards are still experimental. What do you think the next year looks like? Lucas: I think we'll see more lawsuits, more journalists and coders getting burned, and eventually a shift in user behavior. People will start treating AI outputs as unverified drafts by default. And the tools that bake in attribution and memorization detection will win the trust of professionals. The others will be relegated to low-stakes tasks like generating fluff. Luna: That sounds about right. And hopefully, the Sarah K. case serves as a wake-up call — not just to users, but to the companies building these tools that 'rare' isn't good enough when the stakes are someone's career. Lucas: Yeah. One concrete thing to watch: the next version of GPT or Claude — if they can show a significant reduction in memorization without losing quality, that'll be the benchmark. Until then, trust but verify.