Latest / AI Ethics with Fexingo: Bias, Safety, and Responsible Artificial Intelligence / When Your AI Gives You Wrong Legal Advice
Transcript
- Lucas: So there's this moment, right, that I think a lot of people have had recently — you need a simple legal document, nothing crazy, maybe a non-disclosure agreement or a simple will, and you think, 'I'll just ask an AI to draft it, save myself a few hundred bucks.' Luna: I've definitely done that. Or at least thought about it. Lucas: Right. And the tools are getting better — there are now dozens of legal-specific AI assistants, from startups to features inside larger platforms. But here's the problem: they can be confidently wrong. And when it comes to law, confident wrongness can cost you real money or real rights. Luna: What's the specific example you're thinking of? Lucas: There was a case that got some attention back in February of this year. A small business owner in Oregon used a popular legal AI tool to draft a service agreement. The AI generated a perfectly readable contract, looked great, used proper legal language. But the contract included a non-compete clause that is explicitly unenforceable under Oregon law. The AI didn't flag that. It just produced a document that looked right but had a hole in it. Luna: And the person who used it had no way of knowing. That's the scary part — the AI doesn't show uncertainty. Lucas: Exactly. And this isn't an isolated glitch. Researchers at Stanford's RegLab tested several legal AI tools earlier this year. They found that when asked to draft contracts or answer basic legal questions, the tools gave responses that were incomplete or flat-out wrong roughly 30 to 40 percent of the time — but they almost never said 'I don't know.' They just made something up. Luna: That's the classic hallucination problem, but with higher stakes. If an AI tells me the wrong recipe for banana bread, I get a sad breakfast. If it tells me the wrong legal standard, I could lose a case. Lucas: Right. And the particular danger here is that people trust legal AI more than they trust other kinds of AI. A survey from the American Bar Association last year found that over 60 percent of respondents said they would trust legal advice from an AI tool if it was marketed as 'attorney-reviewed' or 'jurisdiction-aware.' Luna: But most of those tools aren't actually jurisdiction-aware. They're trained on a huge mix of case law and statutes, but they don't have a reliable way to know that Oregon's non-compete law is different from, say, California's. Lucas: That's exactly the issue. The training data is a broad soup of legal texts — federal rulings, state laws, law review articles, maybe some Canadian or UK materials thrown in. The model learns patterns, not jurisdiction. So when you say 'draft a non-disclosure agreement,' it might pull language from a New York case, a UK template, and a law review article about Delaware, and blend them into something that doesn't actually reflect any single jurisdiction's requirements. Luna: And the user, understandably, assumes that if the output looks like a real legal document, it must be correct for their situation. Lucas: Right. And this isn't just about contracts. There are AI tools that claim to help with immigration forms, tenant rights, even criminal record expungement. A nonprofit in Texas that helps low-income tenants facing eviction told me that they've seen people show up to court with documents generated by AI that missed key filings or used wrong statutes. In some cases, that meant the case was dismissed before the tenant even got a hearing. Luna: That's devastating. And it hits people who can't afford a lawyer hardest. The irony is that legal AI is often marketed as a tool for access to justice — but if it's wrong, it can actually make justice harder to reach. Lucas: Exactly. A broken promise of access. So what are the solutions? Some legal tech companies are starting to add disclaimers — 'this is not legal advice,' 'consult a licensed attorney.' But studies show that users often ignore those disclaimers, especially when the AI sounds authoritative. Luna: Right, because the medium itself conveys confidence. If a chatbot says 'here is your contract,' it feels like a finished product. Lucas: One approach that some companies are experimenting with is jurisdiction-locked models. So instead of one big model trained on everything, you have smaller models trained specifically on the laws of a single state or a single area of law. That reduces the hallucination rate significantly — but it also makes the tool more expensive to build and maintain. Luna: And it still might not catch everything. Laws change. New court rulings come out. A model trained on Oregon law from 2024 might not know about a ruling from last month. Lucas: That's the maintenance challenge. Legal knowledge isn't static. So some companies are trying a hybrid approach: the AI drafts the document, but a human attorney reviews it before it's finalized. That adds cost, but it's probably the only way to get real accuracy. Luna: There's also a regulatory dimension. The Federal Trade Commission has started looking at AI tools that give legal or medical advice. They've sent inquiry letters to a few companies asking about their accuracy claims and how they handle errors. Lucas: I saw that. And the American Bar Association has a new task force on AI and legal ethics. They're wrestling with questions like: if a lawyer uses an AI tool and the AI makes a mistake, is that malpractice? And if a non-lawyer uses an AI and gives bad advice to a friend, is that unauthorized practice of law? Luna: Those are hard questions. And they're not theoretical — people are already doing these things. So the legal profession is in this awkward spot where they want to embrace innovation but also protect the public. Lucas: One thing that gives me some hope: there's a growing open-source effort to build benchmark datasets for legal AI. Basically, a standardized set of legal questions with known correct answers. If a company claims their AI is accurate, they can be tested against the benchmark. That's the kind of transparency that could help consumers make informed choices. Luna: That's smart. And it's similar to what's happening in medical AI, where there are established benchmarks for diagnosis accuracy. Lucas: Right. The legal world is about a decade behind medicine on this, but they're catching up. The question is whether they can catch up before a lot of people get hurt. Luna: So what's the takeaway for someone listening who might be tempted to use an AI for legal help? Lucas: I'd say: use it as a starting point, not a final product. Get a draft, but then — if you can — run it by a real lawyer, even for a short consultation. Some states now have low-cost legal clinics or online services where you can get a document reviewed for fifty or a hundred dollars. That's a lot cheaper than dealing with an unenforceable contract. Luna: And if you're a developer building legal AI, what's the responsibility? I mean, you can't just slap a disclaimer on it and call it ethical. Lucas: I think the responsible thing is to be transparent about what the model can and can't do. Don't call it an 'AI lawyer.' Don't say 'jurisdiction-aware' if it's not rigorously tested. And build in obvious friction: if someone asks for a legal document, the tool should clearly say 'this may not be valid in your state' and suggest they verify. Luna: I was reading about one startup that actually shows a confidence score for each clause — green, yellow, red. So the user can see that the non-compete clause is flagged as 'low confidence for your jurisdiction.' Lucas: I love that. That's the kind of design thinking that treats the user like an adult. Give them information, not just a polished output. Luna: It's a reminder that AI ethics isn't just about avoiding bias — it's about managing expectations and communicating uncertainty. Lucas: Exactly. And on that note — this is actually a good moment to mention something. If this kind of conversation is useful to you — thinking through the real-world implications of AI, not just the hype — it's exactly why we do this show ad-free. A lot of listeners have asked how they can support that. There's a simple way: buy me a coffee dot com slash fexingo. It's not a subscription, no pressure. Just a way to keep the show independent and focused on topics like this. Luna: Yeah, and it genuinely makes a difference. We don't run ads, we don't have sponsors, so listener support is what keeps the lights on. Lucas: Right. And back to legal AI — I want to mention one more thing that's happening in the regulatory space. The Uniform Law Commission is drafting a model act on ai generated legal documents. It would require clear disclaimers and impose liability on companies that market their tools as legally accurate when they aren't. That's still in early stages, but it's a sign that lawmakers are starting to take this seriously. Luna: That's promising. Because right now, the burden is entirely on the user to know better. And that's not fair when the AI is designed to seem reliable. Lucas: Exactly. So the next time you're tempted to ask an AI to draft a will or a contract, just remember: it might look perfect, but it could have a hidden flaw. And the only way to find it is to have a human who actually knows the law look at it. Luna: Or at least run it through a few different tools and compare. But yeah, no substitute for a real lawyer when it matters. Lucas: Alright, that's our take on legal AI. Next time, we're going to look at a very different kind of AI ethics problem: what happens when your AI wedding planner plans a disaster. That should be fun. Luna: Oh, I've got stories for that one. See you next time. Lucas: Thanks for listening.