Latest / AI Tools for Practicing Lawyers / Field Note: 21 Ways AI Can Hallucinate in Your Legal Brief

Transcript

Field Note: 21 Ways AI Can Hallucinate in Your Legal Brief · AI Tools for Practicing Lawyers

• 0:03
  SPEAKER_00: Welcome to AI Tools for Practicing Lawyers.
• 0:07
  SPEAKER_00: Practical, no-nonsense guidance to help attorneys put AI power to work in their practice right now.
• 0:14
  SPEAKER_01: For this field note, I'm going to start with something that should get every lawyer's attention if you haven't heard of it already.
• 0:22
  SPEAKER_01: A few weeks ago, Sullivan and Cromwell, major world firm, filed a letter to the federal bankruptcy court, admitting that the motion that they filed contained AI hallucinations, specifically identified in what they called Schedule A.
• 0:39
  SPEAKER_01: Now, most of the commentary has focused on the obvious fake citations, misquotes, non-existent authority.
• 0:49
  SPEAKER_01: And yeah, that was all there.
• 0:51
  SPEAKER_01: But when you actually read Schedule A, and you should, because to me, Schedule A is much more interesting than the confession contained in the letter to the judge, because Schedule A is a real-world example of the kind of hallucinations and the breadth of hallucinations that AI can deliver to your legal writing if you're not paying close attention.
• 1:22
  SPEAKER_01: So, in addition to the obvious hallucinations, there are some obscure and underappreciated hallucinations right on Schedule A.
• 1:33
  SPEAKER_01: And here are two that don't get talked about, but absolutely should.
• 1:37
  SPEAKER_01: One is called hybrid citation corruption or citation drift.
• 1:43
  SPEAKER_01: You'll see citations like 503, 543, BR571.
• 1:50
  SPEAKER_01: And dates and volumes get mashed together, parallel citations partially merged.
• 1:57
  SPEAKER_01: Now, this isn't a fake case, but it's something much more subtle.
• 2:01
  SPEAKER_01: It's a real citation that's been structurally corrupted.
• 2:06
  SPEAKER_01: And that's dangerous because it looks real, it passes a quick skim, it may even partially validate, but it's wrong.
• 2:15
  SPEAKER_01: Now, a second dangerous kind of hallucination that we saw in Schedule A were language level mutations inside quotes.
• 2:26
  SPEAKER_01: So there's this quote correction in paragraph 33, and the the Sullivan Cromwell letter is posted on the website.
• 2:33
  SPEAKER_01: Assets will be dissipated.
• 2:36
  SPEAKER_01: Now, these are sentences spliced together, judicial language partially rewritten.
• 2:42
  SPEAKER_01: Now, this is not a fake quote, but it's worse.
• 2:46
  SPEAKER_01: It's a mutated quote assembled from fragments, altered just enough to sound judicial, fit the argument, and escape casual verification.
• 2:57
  SPEAKER_01: So looking at this Schedule A, studying the Schedule A, it hit me if this is happening in one high-profile filing from one of the most sophisticated firms in the world.
• 3:09
  SPEAKER_01: I mean, they represent OpenAI, the creator of ChatGPT, in major groundbreaking litigation.
• 3:20
  SPEAKER_01: They know what's going on in the world of AI.
• 3:23
  SPEAKER_01: They're training their lawyers to find hallucinations, and they still got through.
• 3:29
  SPEAKER_01: So because this Schedule A was so mind-bending to me, I asked my AI tools to help me map the landscape.
• 3:38
  SPEAKER_01: And what came back was much more than I expected.
• 3:42
  SPEAKER_01: So let's talk about the three most well-known hallucinations.
• 3:47
  SPEAKER_01: These are the ones everybody thinks about.
• 3:49
  SPEAKER_01: You know, listen, they're real risks.
• 3:51
  SPEAKER_01: First of all, there's the completely fabricated case.
• 3:54
  SPEAKER_01: That's the classic.
• 3:56
  SPEAKER_01: That's it, it looks real, it's in proper citation format, but it doesn't exist.
• 4:02
  SPEAKER_01: And this is what's getting lawyers sanctioned the most.
• 4:06
  SPEAKER_01: When you read articles about they had 27 citations, and 23 of them were hallucinated cases.
• 4:14
  SPEAKER_01: I mean, that's extreme, it's unforgivable, but that's the classic hallucination example.
• 4:22
  SPEAKER_01: But there's more.
• 4:23
  SPEAKER_01: There's the real case, the wrong holding.
• 4:26
  SPEAKER_01: So the case exists, but the outcome is flipped.
• 4:30
  SPEAKER_01: The reasoning might be distorted.
• 4:32
  SPEAKER_01: The support is fictional.
• 4:34
  SPEAKER_01: Now, this one passes the basic citation check.
• 4:37
  SPEAKER_01: So if you were going to verify, oh, just just verify, right?
• 4:41
  SPEAKER_01: That's the mantra that we're hearing in efforts to stem the tide of AI hallucination that's polluting legal briefs all around the country.
• 4:53
  SPEAKER_01: Just verify the sites.
• 4:55
  SPEAKER_01: But this one would verify the site.
• 4:58
  SPEAKER_01: The site is real, but the content is not real.
• 5:03
  SPEAKER_01: Then, all right, it gets worse from there.
• 5:06
  SPEAKER_01: You get invented quotations.
• 5:08
  SPEAKER_01: And this is where AI really shines and then fails.
• 5:12
  SPEAKER_01: It produces perfect sounding judicial language that's tailored exactly to your argument, except no judge ever said it.
• 5:20
  SPEAKER_01: So those are kind of like three really well-known hallucinations that we're seeing.
• 5:26
  SPEAKER_01: Now, here is some lesser-known, but really conceivably more dangerous hallucinations because it's very subtle.
• 5:35
  SPEAKER_01: So these are the ones that should be keeping lawyers up at night.
• 5:39
  SPEAKER_01: And it's pulled directly from the companion field.
• 5:42
  SPEAKER_01: The first one is a subtle semantic drift.
• 5:46
  SPEAKER_01: This is an extremely dangerous category.
• 5:49
  SPEAKER_01: Nothing looks wrong.
• 5:50
  SPEAKER_01: But all of a sudden, the standard of reasonable likelihood becomes reasonable possibility, right?
• 5:58
  SPEAKER_01: Lawyers know that these are very different ideas.
• 6:02
  SPEAKER_01: So another thing that AI might hallucinate is that the burdens shift slightly.
• 6:07
  SPEAKER_01: Standards soften or harden, no red flags, but AI has changed the law quietly.
• 6:14
  SPEAKER_01: Now, another big problem: fake multi-case consensus.
• 6:19
  SPEAKER_01: So AI stitches together a few real cases from different contexts with different holdings and presents it as courts consistently hold, except they don't.
• 6:30
  SPEAKER_01: The cases exist, but the consensus doesn't.
• 6:34
  SPEAKER_01: So that's a scary kind of hallucination because you've got to dig really deeply into this writing and these cited cases in order to find out that you're gonna have to make changes to that brief.
• 6:49
  SPEAKER_01: So another category of hallucinations are logical hallucinations.
• 6:54
  SPEAKER_01: This one isn't about facts, it's about structure.
• 6:57
  SPEAKER_01: So the argument sounds right, it reads smoothly, it feels persuasive, but maybe it'll skip an essential element, or it'll misapply a burden, or it'll collapse legal steps together that need to be kept apart.
• 7:14
  SPEAKER_01: It's not a fake citation, it's just a broken argument that looks complete.
• 7:20
  SPEAKER_01: So AI hallucinations, they're no longer edge cases.
• 7:24
  SPEAKER_01: They're part of the operating environment of modern legal writing.
• 7:29
  SPEAKER_01: The question isn't whether they happen, it's whether your workflow is designed to catch the ones that don't look like mistakes, but they are mistakes.
• 7:39
  SPEAKER_01: So what I did, with the help of Claude, of course, because Claude has the best deliverables, I put together a companion downloadable for this field note.
• 7:49
  SPEAKER_01: It's not designed to be something that you just read once or you skim once and then you file away.
• 7:56
  SPEAKER_01: It's really intended to be a reference tool.
• 8:00
  SPEAKER_01: Use it during what I call your hallucination verification process.
• 8:06
  SPEAKER_01: That's something that you tell the partner, what are you working on now?
• 8:09
  SPEAKER_01: Well, I finished the brief.
• 8:11
  SPEAKER_01: Now I'm going through the hallucination verification process, and the partner will probably nod knowingly and leave you alone.
• 8:20
  SPEAKER_01: Because the old rule, just check the citation, make sure it's real.
• 8:25
  SPEAKER_01: It's not enough.
• 8:26
  SPEAKER_01: The new rule is you got to verify the proposition, verify the quote, verify the context, verify the procedural posture.
• 8:35
  SPEAKER_01: That's the checklist now.
• 8:37
  SPEAKER_01: So I'm gonna leave you with this.
• 8:39
  SPEAKER_01: This feels like it's filled with kind of dire information that you're never gonna catch.
• 8:47
  SPEAKER_01: But I'm gonna do a field note that's a companion to this field note.
• 8:52
  SPEAKER_01: I went through, I'm constantly going through social media to find content for this podcast.
• 8:58
  SPEAKER_01: And there's a there's a subreddit called Big Law.
• 9:02
  SPEAKER_01: And big law associates are actually adapting to the hallucination problem in legal drafting.
• 9:08
  SPEAKER_01: I mean, look, they they've been trained on legal AI.
• 9:12
  SPEAKER_01: They've been told, listen, we're paying$1,000 a month per seat for you to have this tool.
• 9:20
  SPEAKER_01: You need to be using it.
• 9:22
  SPEAKER_01: So big law associates are using the tool.
• 9:25
  SPEAKER_01: And this was a fascinating thread in Reddit that talked all the way, ways that associates in these big law firms have concocted uh through learned experience and earned experience when they are trying to write their legal briefs in a way that they won't be deluged by hallucinations and turn whatever time savings there may have been by using AI into time losers because they have to spend so much time verifying.
• 10:03
  SPEAKER_01: So the name is 12 ways big law associates are quietly optimizing AI in legal drafting.
• 10:12
  SPEAKER_01: This is not a theoretical thing.
• 10:14
  SPEAKER_01: This is what associates in big law firms have learned in how to deal with AI in their legal drafting.
• 10:22
  SPEAKER_01: So this field note wasn't intended to be about fear.
• 10:27
  SPEAKER_01: It's really about clarity.
• 10:29
  SPEAKER_01: AI does not eliminate judgment, it raises the stakes on it.
• 10:33
  SPEAKER_01: And if you understand how it fails, you can actually use it better than most lawyers ever will.
• 10:40
  SPEAKER_01: This is Ron Dresher, and this is AI Tools for Practicing Lawyers.
• 10:45
  SPEAKER_00: That's it for today's episode of AI Tools for Practicing Lawyers.
• 10:50
  SPEAKER_00: Thanks for listening.
• 10:51
  SPEAKER_00: We'll see you next time.