Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Paralegal Misses a Key Precedent
Transcript
- Lucas: So we've talked on this show about AI grading students, AI deciding bail, AI recommending medical treatment. But there's one profession where the stakes are uniquely high when the model gets it wrong, because the entire system is built on precedent and citation and getting the facts exactly right. Luna: Lawyers. You're talking about AI in legal practice. Lucas: Exactly. And I want to look at a specific case that every attorney I know is still talking about. In early 2025, a federal appeals court in the Second Circuit issued a sanctions order against a law firm because their ai powered legal research tool hallucinated a Supreme Court precedent. The model invented a case citation that looked completely real — docket number, court, year — but the case never existed. Luna: Wait — the attorney didn't check the citation before filing the brief? Lucas: That's the part that shocked the legal community. The associate who ran the research assumed the AI was correct. They didn't open Westlaw or LexisNexis to verify. The opposing counsel, who did their own research, couldn't find the case and alerted the court. The judge was furious. The firm was fined, and the attorney was referred to the state bar for potential ethics violations. Luna: It reminds me of the Mata v. Avianca case from a couple years ago, where a lawyer used ChatGPT to draft a brief and it cited six nonexistent cases. That was the first big wake-up call. Lucas: Right. That was 2023, and people thought it was a one-off — a rookie mistake. But what we're seeing now is that even with more sophisticated legal AI tools — platforms like Harvey or Casetext's CoCounsel — the hallucination problem hasn't gone away. It's just become rarer, and therefore more dangerous, because attorneys let their guard down. Luna: So the better the AI gets, the more trust lawyers place in it, and the less they double-check? Lucas: Exactly. And the thing is, legal research is an area where a 99 percent accuracy rate is not good enough. If a model is right 99 times out of a hundred, but the one mistake is a key precedent that changes the outcome of a case, that's a catastrophic failure. The legal system is built on the idea that every citation can be verified. AI undermines that assumption. Luna: Speaking of trust — if this topic resonated with you and you got something useful out of it, honestly, if today's tech conversation gave you something usable, that's the link. It's buy me a coffee dot com slash fexingo. Keeps this show ad-free and independent. No pressure, just a small way to say it was worth your time. Lucas: Yeah, we appreciate that more than we can say. And we'll get right back to the legal ethics question, because it gets even more interesting when you look at what the American Bar Association is now doing about it. Luna: Perfect. So what is the ABA's latest stance on AI in legal practice? Lucas: They formed a task force in late 2025 specifically on generative AI and the practice of law. Their draft recommendations, which I read last week, propose that any AI tool used for legal research or document drafting must be 'certified' by an independent auditor for accuracy in the specific practice area. So a family law AI would need different certification than a patent litigation AI. Luna: That's a serious departure from the current approach, which is basically caveat emptor — let the lawyer figure it out. Lucas: Right. And the task force also recommends that attorneys disclose to clients when AI was used to generate any part of their legal work product. Not just in billing — which is already a gray area — but in the actual substance of the representation. Some states like California and New York are already considering rules that would make nondisclosure an ethics violation. Luna: I can see the argument against that — that it creates unnecessary paperwork, that clients don't care as long as the work is good. But after the hallucination cases, it seems prudent. Lucas: The counterargument I've heard from some big law partners is that AI is just a tool, like a calculator or a word processor. You don't tell a client when you used a calculator. But I think that comparison fails because a calculator doesn't invent numbers. It does exactly what you tell it. A generative AI model is fundamentally probabilistic. It can produce plausible-sounding falsehoods. Luna: And a calculator error is usually obvious — you know if your numbers don't add up. A legal hallucination can look perfectly correct to someone who isn't an expert in that specific area of law. Lucas: That's exactly the risk. And it's not just about citations. I read about a case in a federal district court in Texas where an ai generated brief used a legal standard from a different circuit without realizing it, because the model conflated two similar-sounding doctrines. The judge caught it and asked for re-briefing. That cost the client time and money. Luna: So what's the practical solution for lawyers who want to use AI but don't want to get sanctioned? Lucas: Most ethics experts I've talked to recommend a two-step approach. First, never use a general-purpose chatbot like ChatGPT for legal work. Use a tool specifically trained on legal databases — like Lexis+ AI or Westlaw's Ask Practical Law — that have guardrails and citation verification built in. Second, always, always verify every citation manually. Treat AI as a research assistant, not a partner. Luna: But doesn't that defeat the efficiency gains? If you have to check everything, you're not saving that much time. Lucas: It's a trade-off. The time savings come from drafting and summarization, not from eliminating verification. A lawyer who uses AI to produce a first draft of a brief in two hours instead of ten, then spends one hour verifying citations, still saves seven hours. The problem is when people skip that verification hour. Luna: It's a discipline problem, not a tool problem. Lucas: Exactly. And the legal profession is wrestling with this right now. Some firms have outright banned generative AI for research, while others have embraced it with strict protocols. The ABA task force is hoping to create a uniform standard so that lawyers in different states have the same ethical obligations. Luna: I want to go back to something you mentioned earlier — the idea of 'certified AI' for specific practice areas. How would that work in practice? Would a company like Harvey need to get its model audited by a third party every time it's updated? Lucas: That's one of the open questions. The draft recommendations suggest a certification process similar to how financial audits work — an independent evaluator tests the model on a standardized set of legal queries and measures its accuracy against a verified answer key. But models change frequently. A certification from January might not be valid in June after a fine-tuning update. Luna: So you'd need continuous testing, not just a one-time certification. That's expensive. Lucas: Expensive, but potentially necessary. And it's not just about cost. There's also the question of transparency. Most legal AI tools are proprietary — you don't know what data they were trained on, how they handle updates, or what their error rates are. The task force is calling for more disclosure from vendors. Luna: It sounds like the legal profession is moving toward a regulatory framework that's similar to what we've seen in medicine and finance — where AI systems that affect important outcomes need to be validated before they're used. Lucas: Exactly. And I think that's the right approach. Because the alternative — letting every lawyer figure it out on their own — leads to situations like the one we started with: a sanctioned attorney, a frustrated judge, and a client whose case was harmed by an error that should have been caught. Luna: It's a reminder that AI is only as good as the human who supervises it. Lucas: And the legal system has centuries of precedent for why supervision matters. I'm curious to see whether the ABA's final recommendations will be adopted by state bars, and whether they'll be strict enough to prevent the next hallucination disaster. It's one of the most concrete ethical debates happening in AI right now, and it affects every person who might one day need a lawyer.