Latest / AI Tools for Practicing Lawyers / Episode 027 Is it Safe to Go to the Rave?

Transcript

Episode 027 Is it Safe to Go to the Rave? · AI Tools for Practicing Lawyers

• 0:03
  SPEAKER_00: Welcome to AI Tools for Practicing Lawyers.
• 0:07
  SPEAKER_00: Practical, no-nonsense guidance to help attorneys put AI power to work in their practice right now.
• 0:14
  SPEAKER_01: Suppose your firm gets a massive document production on Friday afternoon.
• 0:20
  SPEAKER_01: Traditionally, that means some unfortunate associates and paralegals are canceling all their weekend plans and spending the next two days reading documents.
• 0:31
  SPEAKER_01: But now AI can do much of that work in hours, maybe minutes.
• 0:38
  SPEAKER_01: So everybody gets to go to the rave.
• 0:40
  SPEAKER_01: Except who checked the AI?
• 0:43
  SPEAKER_01: And if your answer is that a lawyer has to go back and check every document, then what exactly did we accomplish?
• 0:52
  SPEAKER_01: My guest today, Jim Casey, says we're asking the wrong question.
• 0:57
  SPEAKER_01: AI is going to make mistakes.
• 0:59
  SPEAKER_01: So AI is going to make mistakes.
• 1:04
  SPEAKER_01: But so will the exhausted associate who wanted to go to the raid.
• 1:09
  SPEAKER_01: The real question is how much trust can we safely place in either one?
• 1:15
  SPEAKER_01: And how do we build a system that catches the mistakes that actually matter?
• 1:21
  SPEAKER_01: Because lawyers have what Jim calls a non-delegable duty to think.
• 1:27
  SPEAKER_01: AI doesn't change that.
• 1:30
  SPEAKER_01: So is it safe to go to the rave?
• 1:33
  SPEAKER_01: Let's find out.
• 1:34
  SPEAKER_01: Welcome to another episode of AI Tools for Practicing Lawyers.
• 1:39
  SPEAKER_01: I'm Ron Tresher.
• 1:41
  SPEAKER_01: I'm Heather Gardner.
• 1:43
  SPEAKER_01: And our guest today is my friend Jim Casey.
• 1:47
  SPEAKER_01: Jim is vice president and chief privacy officer.
• 1:51
  SPEAKER_01: We're going to have to figure out what that even means at Akamai Technologies, where he leads the company's global data protection team.
• 2:01
  SPEAKER_01: And that means Jim spends his professional life thinking about questions like privacy, technology regulation, AI governance, and cybersecurity.
• 2:12
  SPEAKER_01: Not just as interesting legal questions, but as real-world problems that a global technology company actually has to solve.
• 2:21
  SPEAKER_01: Jim has written two terrific pieces that I think have a lot to say to practicing lawyers.
• 2:27
  SPEAKER_01: One is on knowledge in an agentic world.
• 2:30
  SPEAKER_01: And frankly, I distrust anything that starts with a preposition.
• 2:35
  SPEAKER_01: And agentic workflows, a new frontier for compliance.
• 2:40
  SPEAKER_01: And they really boil down to a question every lawyer using AI is going to have to answer.
• 2:46
  SPEAKER_01: When should we trust these systems?
• 2:48
  SPEAKER_01: And what happens when we give them the power to act?
• 2:51
  SPEAKER_01: Jim, welcome to the show.
• 2:54
  SPEAKER_01: Thanks, Ron.
• 2:54
  SPEAKER_01: Thanks, Heather.
• 2:55
  SPEAKER_01: I'm really happy to be here.
• 2:57
  SPEAKER_01: This is going to be fun.
• 2:58
  SPEAKER_01: So, Jim, you're going to not be surprised that I didn't read either of your articles.
• 3:05
  SPEAKER_01: But I did have AI study them, and I had lengthy chats with ChatGPT about what do they mean and what should we bring to the show?
• 3:17
  SPEAKER_01: So I'm not going to spend a lot of time talking about the articles, which are really have some important ideas, but I want to I want to bring up a hypothetical for you and let's work off that.
• 3:30
  SPEAKER_01: Sounds good.
• 3:32
  SPEAKER_01: So I've got in my in my mind a lawyer who's been practicing bankruptcy for at least 10 years.
• 3:40
  SPEAKER_01: She does some chapter 11s, some 13s, some 7s, she does some creditor work.
• 3:46
  SPEAKER_01: And you know, she sees a demo.
• 3:49
  SPEAKER_01: And the demo has AI reading and summarizing a hundred and eighty-page court order in seconds.
• 3:57
  SPEAKER_01: And she thinks, okay, I need to start bringing AI into my practice.
• 4:04
  SPEAKER_01: She goes back to her office Monday morning.
• 4:07
  SPEAKER_01: How do these articles that you've written help her figure out what she should and shouldn't do next?
• 4:15
  SPEAKER_03: Well, that that's uh thanks, Ron.
• 4:17
  SPEAKER_03: That and that shouldn't is the is the key point.
• 4:22
  SPEAKER_03: What there's a very simple concept here that is simple to say and high and not necessarily simple to to deploy.
• 4:31
  SPEAKER_03: But the uh what is it that a lawyer brings to the table and cannot give up judgment?
• 4:40
  SPEAKER_03: Judgment and accountability.
• 4:42
  SPEAKER_03: These tools don't know anything, they don't have experience, they don't have knowledge, they aren't cognitive, they aren't thinking in any sense, in any human sense.
• 4:54
  SPEAKER_03: So at the end of the day, the answer belongs to the attorney.
• 4:58
  SPEAKER_03: And you have to, the attorney has to think about under what circumstances they can trust.
• 5:05
  SPEAKER_03: And and what I've tried to lay out here is when can we trust?
• 5:10
  SPEAKER_03: And I, of course, went down my normal rabbit hole.
• 5:12
  SPEAKER_03: I always sort of start from first principles and I said, well, what does a what does an AI know?
• 5:17
  SPEAKER_03: And and went all the way back to Plato and really came very quickly to the conclusion that they don't know anything, classically.
• 5:25
  SPEAKER_03: They don't have anything.
• 5:26
  SPEAKER_03: But the answer, the question is, who do we when do we care?
• 5:30
  SPEAKER_03: And so what I've tried to give here is is is uh is a few simple questions to understand when you should care.
• 5:35
  SPEAKER_03: If the things are really good at doing what they're doing, go with it.
• 5:39
  SPEAKER_03: But but verify.
• 5:41
  SPEAKER_03: Trust but verify to you to use an often overused uh phrase.
• 5:44
  SPEAKER_03: And uh and and uh learning when to do that is is the important is the important.
• 5:49
  SPEAKER_03: Well, let's go back to our lawyer.
• 5:51
  SPEAKER_02: I'm gonna call her Wilma Simpson, all right?
• 5:55
  SPEAKER_02: Because like she's gonna read the whole 180 pages if she's Wilma Simpson.
• 6:00
  SPEAKER_01: Well, you know what?
• 6:00
  SPEAKER_01: That's what lawyers had to do.
• 6:02
  SPEAKER_01: Yeah, that's what lawyers had to.
• 6:04
  SPEAKER_01: They either had to read the 180 pages or they had to say, they had to try to kind of guess what parts of it apply to her.
• 6:13
  SPEAKER_02: You're defeating the purpose of the AI here, if you're reading the whole right, that's what I I mean.
• 6:18
  SPEAKER_02: I I feel like Jim's answer didn't really answer.
• 6:21
  SPEAKER_03: No, I haven't answered it yet, Jim.
• 6:22
  SPEAKER_02: I haven't heard an answer in there.
• 6:24
  SPEAKER_02: I heard him talk a lot, but I feel like he's got something else to say.
• 6:27
  SPEAKER_01: I'm ready.
• 6:28
  SPEAKER_01: I'm ready, Jim.
• 6:29
  SPEAKER_01: What how how are you gonna help Wilma figure out how to safely use AI to get through her daily work?
• 6:38
  SPEAKER_01: Because, you know, she's kind of been staying away from it.
• 6:42
  SPEAKER_02: Well, and what is this 180-page order?
• 6:44
  SPEAKER_02: What is reasonable verification look like for Wilma on that order?
• 6:48
  SPEAKER_02: Let's just go there.
• 6:50
  SPEAKER_03: Yeah.
• 6:51
  SPEAKER_03: And and so so the there are certain things, and it requires a little bit of learning ahead of time for Wilma.
• 6:58
  SPEAKER_03: Wilma's not going to be able to always just utilize the tool out of the box.
• 7:02
  SPEAKER_03: She's gotta learn what these these things are good and what they're not good at.
• 7:07
  SPEAKER_03: Summarization systems are generally pretty good.
• 7:11
  SPEAKER_03: They tend to do a uh that's that's one domain where where uh the AIs do a pretty good job when you point them at something very specific, you can get a pretty good uh summary uh and and view of what it has said.
• 7:27
  SPEAKER_03: Where AI is not good is in something truly novel or something requiring human human moral judgment or human answers to questions, that's where they tend to, if they get it wrong, they get it wrong badly.
• 7:42
  SPEAKER_03: So what Wilma needs to think about is what am I doing right now?
• 7:45
  SPEAKER_03: Right now, she wants a summary.
• 7:47
  SPEAKER_03: She wants to understand the the outline of what this order says.
• 7:50
  SPEAKER_03: What are the what are the main findings and all that?
• 7:53
  SPEAKER_03: That falls into a category where one, it's for her, so the harm if it gets it wrong is is is reversible.
• 8:03
  SPEAKER_03: She can sort of scan through and see, oh, wait, that doesn't sound right.
• 8:08
  SPEAKER_03: I've been practicing law for 10 years, and that that doesn't seem correct, and can go in and look.
• 8:13
  SPEAKER_03: And so so she has an ability to go back if it gets it wrong.
• 8:17
  SPEAKER_03: And so that's good.
• 8:19
  SPEAKER_03: So so it's a domain that the LLM is good at, and she can she can get that general summary, get the outline, and can safely work with that as a cognitive tool.
• 8:31
  SPEAKER_01: I'm gonna jump in and disagree with you.
• 8:34
  SPEAKER_01: Okay, and I'll tell you why it's it's not really reversible.
• 8:38
  SPEAKER_01: It's really not reversible because she knows that she's got a client who's paying her to analyze this very long order and is relying upon her judgment as to what to do next relative to this long order.
• 8:57
  SPEAKER_01: So she is going to be getting a summary from AI that looks great, looks detailed and incisive, and you know, there's provenance, right?
• 9:07
  SPEAKER_01: It'll go right back to where in the order the summary pieces, you know, are derived.
• 9:13
  SPEAKER_01: And she's going to advise a client on next steps based upon her assessment of that order.
• 9:21
  SPEAKER_01: And that client is going to act based upon the lawyer's advice.
• 9:26
  SPEAKER_01: So is it truly reversible?
• 9:28
  SPEAKER_01: Or are we not using the word reversible the correct way?
• 9:33
  SPEAKER_03: No, I you well, in your explanation there, you use reversible in a different part of the chain than I was using.
• 9:40
  SPEAKER_03: Okay.
• 9:40
  SPEAKER_03: And that's worth that's worth conversation.
• 9:42
  SPEAKER_03: Step away from your example for a second.
• 9:44
  SPEAKER_03: So one of the things I have to do within my job is get is get an understanding of a regulatory area of the law for a global company that I may not know the end.
• 9:54
  SPEAKER_03: I may not know the area of the law.
• 9:56
  SPEAKER_03: Whether it's a re whether it's a geographic thing or a new law coming out of Congress in the US or whatever.
• 10:02
  SPEAKER_03: I need to go to outside council and get help with something.
• 10:05
  SPEAKER_03: I could go to outside council and have them tell me everything about the law and spend 20 hours of outside legal help, right?
• 10:14
  SPEAKER_03: Or I can go to a system like this and get a get a general summary of the law and an understanding of the basic outlines of that law.
• 10:24
  SPEAKER_01: When you mean system like that, you mean like a chat GPT frontier model.
• 10:29
  SPEAKER_03: Exactly.
• 10:29
  SPEAKER_03: Okay.
• 10:29
  SPEAKER_03: Yeah, now uh Claude or whatever.
• 10:34
  SPEAKER_03: And I can get a memo that summarizes the law in the area.
• 10:38
  SPEAKER_03: I now have something that as a cognitive tool, I've helped myself start to think where I have to zero in.
• 10:44
  SPEAKER_03: And I do that, and then I know the questions I need to go to my outside counsel and ask.
• 10:48
  SPEAKER_03: And I've now taken a five-hour conversation or five hours of work from the law firm and turned it into a half hour or an hour-long conversation.
• 10:55
  SPEAKER_02: And who's verifying that?
• 10:57
  SPEAKER_02: This sounds like that just sounds like the biggest landmine when we're talking about an attorney verifying it.
• 11:02
  SPEAKER_02: You're talking about a layperson who really wouldn't understand taking hours of AI generated work.
• 11:09
  SPEAKER_02: Someone has to verify that.
• 11:12
  SPEAKER_03: Well, no, we're talking about uh I think we're talking about lawyers using using this, these two.
• 11:17
  SPEAKER_01: We're talking about a lawyer, an in an in-house lawyer at the talking into an out-house lawyer.
• 11:22
  SPEAKER_01: Getting up to speed on an area of the law with which the lawyer is unfamiliar, and and and then going to the to the you know big law attorney and turning a$5,000 bill into a$500.
• 11:39
  SPEAKER_01: Exactly.
• 11:40
  SPEAKER_01: Exactly.
• 11:41
  SPEAKER_03: Both lawyers, so I'm not going to go directly to uh that lawyer and say, here, here's my memo of what of what I think the law is, and then ask them to verify it.
• 11:54
  SPEAKER_03: I'm going to distill it myself and and uh and then go and ask my questions.
• 11:59
  SPEAKER_03: So what have I done?
• 12:00
  SPEAKER_03: At that stage, what the what the LLM produced, what the what the frontier model produced, it is reversible because I haven't acted on it myself yet as the lawyer.
• 12:10
  SPEAKER_03: I haven't taken it to the next step.
• 12:13
  SPEAKER_03: If I take it to the next step and then tell the client this is what you do and they act on it, yeah, reversibility goes away because I've caused that action to happen.
• 12:22
  SPEAKER_03: But in this first step, summarize, having the AI summarize that 180-page court order is only the first step.
• 12:31
  SPEAKER_03: There's a time savings.
• 12:32
  SPEAKER_03: It would have taken a long time to summarize that thing.
• 12:34
  SPEAKER_03: Now I'm going to go through it as a lawyer, and I'm still going to spend time going through it and saying, oh, okay, so this is the important point for what I would tell the client.
• 12:44
  SPEAKER_03: What is the let me double check in the in the order over here?
• 12:47
  SPEAKER_03: Okay, that that got it right.
• 12:49
  SPEAKER_03: There's still that work that has to go, that has to go forward.
• 12:53
  SPEAKER_03: Where you hear cases where the where lawyers have been burned and sanctioned in in some cases now, they have taken the output from the LLM, the the frontier model, the Chat GPT or whatever, whichever one they used, and put it into their into their uh order or into their into their briefing or whatever they're they're filing and handed it over to the court or gave it to their client.
• 13:20
  SPEAKER_03: That is an abdication of your duty as a lawyer because you've taken someone else's answer and given it over.
• 13:27
  SPEAKER_03: So what does what does Wilma Simpson get from doing that hundred?
• 13:31
  SPEAKER_03: She doesn't save all the time she might have done analyzing that order.
• 13:36
  SPEAKER_03: She has drastically reduced and is using the LLM as a cognitive tool and expanding her ability and making her more efficient in how she's going to do it.
• 13:45
  SPEAKER_03: But she's still got to be a lawyer.
• 13:46
  SPEAKER_03: And that's that's the bottom line is you the lawyer still owns the job, the work.
• 13:52
  SPEAKER_03: The lawyer is still accountable.
• 13:54
  SPEAKER_03: So sometimes it's going to save you a lot of time, help you write an email.
• 13:58
  SPEAKER_03: Cool, that's easy.
• 14:00
  SPEAKER_03: Detailed, detailed analysis of a court order, a court order that you then have to tell a client how to act.
• 14:07
  SPEAKER_03: That's gonna that's gonna take some of your own time.
• 14:10
  SPEAKER_03: And you're you're gonna save 50% instead of 80% of the time spent.
• 14:15
  SPEAKER_01: But but see, this is the problem I get with the whole verification concept, which is which is critical.
• 14:22
  SPEAKER_01: Nobody on on the nobody listening, nobody speaking is going to disagree that lawyers are ultimately accountable.
• 14:30
  SPEAKER_01: Everybody knows that and it's reality.
• 14:33
  SPEAKER_01: But, you know, what I to me, what I want for this show, and maybe you know somebody like it, I want the lawyer that did a million document production in intense, complex litigation, and used AI to sort the documents, to summarize the documents, to look for duplication in the documents, all those wonderful things that it used to take paralegals and junior associates a full weekend.
• 15:04
  SPEAKER_01: You know, forget about it.
• 15:05
  SPEAKER_01: You're not going to that rave.
• 15:06
  SPEAKER_01: We got a document production we got to go through.
• 15:09
  SPEAKER_01: All right.
• 15:10
  SPEAKER_01: So that lawyer is now going to dump everything into a secure environment where we're not talking about like a consumer level, you know, Chat GPT or something.
• 15:24
  SPEAKER_01: We're talking about an enterprise level.
• 15:26
  SPEAKER_01: They got Harvey, they got Laguerre, whatever they got.
• 15:28
  SPEAKER_01: They got the right stuff, all right?
• 15:31
  SPEAKER_01: But they're gonna put a million documents or 5,000 pages of a document production in there, and and it's and they're gonna get to go to their rave on Saturday night, as it turns out.
• 15:41
  SPEAKER_01: Story has a happy ending.
• 15:43
  SPEAKER_01: But they haven't looked at every one of these 5,000 pages, and no one expects them to.
• 15:50
  SPEAKER_01: Otherwise, you said million.
• 15:53
  SPEAKER_01: I started out with the million.
• 15:55
  SPEAKER_01: I started out with the million, but but a 5,000 is the same thing.
• 15:59
  SPEAKER_01: A million documents, 5,000 documents, whatever.
• 16:02
  SPEAKER_01: The lawyer's not looking at it.
• 16:04
  SPEAKER_01: Exorbitant, of course, yeah.
• 16:05
  SPEAKER_01: So where has the verification process landed in that obviously desirable use for the AI tool?
• 16:15
  SPEAKER_03: Right.
• 16:15
  SPEAKER_03: Well, now you're now you're getting into my compliance world, uh, a piece of the puzzle.
• 16:21
  SPEAKER_03: How do you build, how do you put trust into this process?
• 16:26
  SPEAKER_03: And it flows in two places.
• 16:29
  SPEAKER_03: There's sort of system level trust.
• 16:32
  SPEAKER_03: Have I designed the system and have I designed the data?
• 16:36
  SPEAKER_03: And have I built built this tool in a way that I can rely, that I can trust it in most cases.
• 16:44
  SPEAKER_03: Then there's the in the real time, the use by the individual.
• 16:49
  SPEAKER_03: I can build a perfectly wonderful, cool, approved tool, and a company can approve it and give it to give it to a lawyer to use, and they can that lawyer can use it in really bad ways and do really dumb things with it and cause all kinds of problems.
• 17:02
  SPEAKER_03: And so no matter what you do, you land back on this individual accountability problem.
• 17:09
  SPEAKER_03: But the answer to your where are the there are tools that are allowing these kind of these kinds of massive re analyses.
• 17:20
  SPEAKER_03: And I I I have not worked with one that I have seen, heard somebody say I've had a million, you know, a product document production and I've analyzed it using the tool.
• 17:30
  SPEAKER_03: They exist, I'm sure they're out there.
• 17:32
  SPEAKER_03: I just haven't personally worked on one.
• 17:34
  SPEAKER_03: But I can tell you I have had conversations with a with a group at a large uh e-commerce platform.
• 17:40
  SPEAKER_03: And that e-commerce platform is selling, is, is selling goods from many different producers and many different different sources, and they have to deal with questions of product law, product liability law.
• 17:54
  SPEAKER_03: And what reporting do they have to do about the safety of a product?
• 17:58
  SPEAKER_03: And the the signals for that reporting obligation come from the complaints and the uh the issues that come in from users who have bought a bed and the bed collapsed and they hurt their back, or or they they uh they cut themselves on a product packaging that was was faulty product packaging or something like that.
• 18:17
  SPEAKER_03: And and to go through potentially hundreds of thousands of these kinds of comments and reports coming through, figure out which ones are real.
• 18:26
  SPEAKER_03: The person that hurt their back with the bed, for example, hurt their back because it was delivered at their uh at the front of their driveway and they had to carry it back into the house and they hurt their back carrying the box back into the house.
• 18:38
  SPEAKER_03: Well, that's not a product problem.
• 18:40
  SPEAKER_03: But the person who built it followed the instructions, laid down and it collapsed.
• 18:44
  SPEAKER_03: Yeah, that's a problem.
• 18:46
  SPEAKER_03: So that takes a huge amount of time for people to sit and sift through that and then sort it and figure out what the reporting obligations that go along.
• 18:56
  SPEAKER_03: Well, they worked for a good, I think they said a good six months of pulling all a bunch of that data together with known responses from the past, and they trained the model and they, they, they tested that model.
• 19:08
  SPEAKER_03: They they worked and they trained the model on all that data and all the responses they had done in the past, and trained the model to actually write a report.
• 19:15
  SPEAKER_03: That would be what you would actually file.
• 19:17
  SPEAKER_03: And they they ran it on new data the first time out of the box, and uh, it was horrible.
• 19:22
  SPEAKER_03: It didn't help, it was wrong, it was not, it didn't do the things that it was supposed to do.
• 19:28
  SPEAKER_03: And so they went back and they reworked the data and they re and they and they retested and they trained again.
• 19:35
  SPEAKER_03: And over time, what they what they did is they got to a situation where they were getting a 90 to 95% accuracy rating on going through this 100,000, you know, a month piles of stuff and brought it down to something where they could actually produce a trustable answer.
• 19:54
  SPEAKER_03: So what that takes is the accountability, you've slid the accountability back into the design of the tool.
• 20:03
  SPEAKER_03: And and you are partially.
• 20:06
  SPEAKER_03: Partially, and but there are still going to always be mistakes, and we can we'll talk about that.
• 20:11
  SPEAKER_03: But the but but the answer, the answer is very few of these tools will get you what you want right out of the box.
• 20:21
  SPEAKER_03: You have to put some work into it to put it together in a way that will actually meet the needs and actually give you the level of trust, and you test it and you and it and and it and it's horrible, and you fix it and do it, and you go back and you test it again, and it's a little bit better.
• 20:38
  SPEAKER_03: There's a lot of work that goes into making this as something a lawyer could trust.
• 20:43
  SPEAKER_03: So any tool that can do that kind of full full on analysis uh has had a lot of work put in it into it.
• 20:49
  SPEAKER_03: Now, something like a Harvey, you can use a lot of what Harvey does right out of the box because teams of lawyers have been building that product and tuning it and training it and working it.
• 21:00
  SPEAKER_03: And and that's where you that's where you get the trust.
• 21:03
  SPEAKER_03: It's kind of like hiring your outside law firm.
• 21:06
  SPEAKER_01: I kind of look at using AI like going on a roller coaster at Universal Studios in Orlando versus going on, you know, one of these crazy scrambler rides that you get at the local fireman's carnival.
• 21:24
  SPEAKER_01: Which one do you trust?
• 21:25
  SPEAKER_01: Right.
• 21:26
  SPEAKER_01: Where they put a stick in there to make sure that the gears don't slip.
• 21:30
  SPEAKER_01: Either way, you're if you're you're gonna get onto that ride, and what do you trust and how much prep do that did they put into it, how much work goes into it?
• 21:40
  SPEAKER_01: There are different levels of trust on the tool, but ultimately you do have to trust it.
• 21:46
  SPEAKER_03: Right.
• 21:46
  SPEAKER_03: And and that trust will come with you know, trust is often betrayed.
• 21:51
  SPEAKER_01: Isn't it a little crazy to you?
• 21:53
  SPEAKER_01: Isn't it a little crazy to you that in the legal field, the most risk averse field.
• 22:00
  SPEAKER_01: I could even think of maybe even more than banking.
• 22:03
  SPEAKER_01: The most risk-averse field is now going full speed ahead on these AI tools that require a degree of blind trust.
• 22:15
  SPEAKER_03: Yeah.
• 22:16
  SPEAKER_01: Well, it's true.
• 22:17
  SPEAKER_01: How do we deal with that?
• 22:18
  SPEAKER_03: Yeah, no, that's a that's a that's a a very good observation.
• 22:22
  SPEAKER_03: And it's one I've been thinking about.
• 22:24
  SPEAKER_03: AI has changed the economics around being a lawyer.
• 22:30
  SPEAKER_03: It has changed the economic model that we operate under.
• 22:34
  SPEAKER_03: Right, right or wrong, good or bad, it's a reality.
• 22:38
  SPEAKER_03: And so at some level, we we lawyers that use AI, it's something that I've heard said multiple times in different contexts.
• 22:51
  SPEAKER_03: AI is not going to replace lawyers anytime soon.
• 22:54
  SPEAKER_03: Lawyers that use AI are going to replace lawyers who don't.
• 22:59
  SPEAKER_03: And so as a result, if you believe that to be true, if you believe that the economic model is changing, absolutely you have to have some blind trust.
• 23:08
  SPEAKER_03: You're going to have to do your best compliance in figuring out whether which tools do the best job.
• 23:14
  SPEAKER_03: When we bought Harvey here at Akamai, for example, we tried about six different approaches to AI models and tested them with use cases and judged answers.
• 23:26
  SPEAKER_03: And it was it was a long, complex process before we landed on that tool.
• 23:30
  SPEAKER_03: And there are other tools now that may that are better, probably.
• 23:33
  SPEAKER_03: It has such a fast moving field.
• 23:35
  SPEAKER_03: But in the end, there are going to be times when when the system gets it wrong.
• 23:40
  SPEAKER_03: By design, AI gets it wrong.
• 23:42
  SPEAKER_03: They do not track truth.
• 23:45
  SPEAKER_03: And accuracy does not mean true.
• 23:48
  SPEAKER_03: They track coherence.
• 23:50
  SPEAKER_03: They're designed to develop a coherent response.
• 23:52
  SPEAKER_03: And a coherent, bad, incorrect answer is just as good an answer from an AI point of view as a correct, truthful answer.
• 24:05
  SPEAKER_01: Okay, let's throw another another hypothetical.
• 24:08
  SPEAKER_01: Okay.
• 24:09
  SPEAKER_01: Because you talked about uh interactions with outside counsel.
• 24:13
  SPEAKER_01: Let's assume that as in-house counsel, you have come to the decision that you can save a substantial amount of money by engaging counsel that use AI tools in the engagement.
• 24:30
  SPEAKER_01: Right?
• 24:31
  SPEAKER_01: We talked about going from a$5,000 bill to a$500 bill earlier.
• 24:36
  SPEAKER_01: So you're insisting upon this because you legitimately believe it's going to save you money.
• 24:42
  SPEAKER_01: Law firm turns around and says, Of course, we use the best AI tools imaginable, but we have a clause in our engagement agreement where you waive any malpractice or incorrect analysis that comes from our use of the AI tools that you are insisting that we use to save you money.
• 25:04
  SPEAKER_01: What's your response?
• 25:05
  SPEAKER_03: I'll call you, don't call me.
• 25:07
  SPEAKER_03: Okay, that's right.
• 25:09
  SPEAKER_03: That's your response.
• 25:10
  SPEAKER_01: Your response is I'd rather pay$5,000.
• 25:12
  SPEAKER_03: I'd rather pay more if that's if that's the position that you take.
• 25:16
  SPEAKER_01: Okay.
• 25:17
  SPEAKER_03: Because because I so I have I have gone to my law, my law firms, I use a number of them for different things, and I have said, let's let's get past this silly dance where you tell me that you you don't rely on AI for anything, and and and and I tell you that that you know I don't want you to use my data and train, blah, blah, blah.
• 25:39
  SPEAKER_03: And let's just cut down to the chase.
• 25:41
  SPEAKER_03: You will use AI, your associates will use AI.
• 25:45
  SPEAKER_03: I'm still holding you responsible for your product.
• 25:47
  SPEAKER_03: I will to make the proper decisions about when to use AI and when not to use AI.
• 25:54
  SPEAKER_03: And that's what I tell, that's what I tell the firm.
• 25:56
  SPEAKER_03: And I fully expect you to use AI where it's going to help me by reducing my bill while still giving me quality output.
• 26:04
  SPEAKER_03: And that's a balancing act that you as the law firm have to do.
• 26:07
  SPEAKER_03: And if I see, if if I see savings, or if you see savings, then I get to see those savings too.
• 26:13
  SPEAKER_03: And I expect that that savings will be passed through.
• 26:15
  SPEAKER_03: And I actually have this in my letters with them now.
• 26:17
  SPEAKER_03: So I fully, I have fully embraced, even as a compliance lawyer where I worry about what can go wrong, I have fully embraced AI as part of the legal ecosystem.
• 26:28
  SPEAKER_03: It is now another cognitive tool that we have available to us.
• 26:32
  SPEAKER_03: And what I expect people to do and what lawyers have to do is they have to, they have to make a value judgment as to where the where that balance, where the balance lies, is when to use it and when not to use it.
• 26:44
  SPEAKER_01: So really Wilma Simpson is, without even realizing it, is radically increasing her overall risk by adopting these AI tools, whether she likes it or not, because otherwise she won't be able to compete.
• 27:01
  SPEAKER_03: Right.
• 27:01
  SPEAKER_03: Well, and I mean that's it.
• 27:02
  SPEAKER_03: You you accept the risk and you you make efforts to mitigate that risk wherever you can, and then and then uh and then you move on.
• 27:11
  SPEAKER_03: And the decision-making process that you have to do though, and this is the piece that people don't don't always do.
• 27:17
  SPEAKER_03: What is the harm that is caused by that confidently wrong answer that you get from from the from the the LLM from the AI?
• 27:26
  SPEAKER_03: What harm can it cause in a situation where I'm just summarizing a bunch of stuff from my own understanding and I'm gonna go back and do my own verification or reading and things, and it's gonna be a it's gonna be a joint cognitive effort to get to an answer.
• 27:42
  SPEAKER_03: I may say that's okay.
• 27:44
  SPEAKER_03: The confidently wrong answer will not kill me at this stage.
• 27:49
  SPEAKER_03: But if the answer comes back, the confidently wrong answer will result in somebody getting the death penalty, for example, then we as lawyers have to say that may not be a place where I can use AI.
• 28:10
  SPEAKER_03: I need to just resort to good old-fashioned lawyer work.
• 28:14
  SPEAKER_01: I gotta tell you something.
• 28:16
  SPEAKER_01: I think AI for the most part is so much elevated over normal legal work over and over and over again.
• 28:26
  SPEAKER_01: I mean, Heather, I want to ask you a question uh that that comes up in in our field, in the bankruptcy field.
• 28:33
  SPEAKER_01: So let's say you've got like one of the big tensions in uh consumer bankruptcy firms is that clients have given enough documents.
• 28:43
  SPEAKER_01: I've given you all the documents that I'm gonna give you, and that's it.
• 28:46
  SPEAKER_01: And then the lawyer says, Well, listen, I need 12 months' worth of bank statements so that I can feed them into my LLM and have it report back to me whether or not there are likely transfers to your friends and relatives that are going to cause them to be sued when you file your chapter seven case.
• 29:07
  SPEAKER_01: Right.
• 29:08
  SPEAKER_01: And then the client says, Yeah, I'm really glad you have those AI tools, but I'm not going to give you those bank statements.
• 29:14
  SPEAKER_01: All right.
• 29:15
  SPEAKER_01: Where where do where do lawyers go with that?
• 29:19
  SPEAKER_02: You're asking me.
• 29:21
  SPEAKER_01: Yeah.
• 29:23
  SPEAKER_02: Well, that's up to the attorney.
• 29:24
  SPEAKER_02: I mean, many attorneys I work with fire a client if they won't cooperate.
• 29:29
  SPEAKER_02: And I feel like that's the most responsible thing to do.
• 29:32
  SPEAKER_02: If the client doesn't give you what you need, how can you move forward with their case?
• 29:36
  SPEAKER_01: All right.
• 29:37
  SPEAKER_01: And so let's say they give you the 12 months and you put them through, and and it it turns out to be nothing.
• 29:44
  SPEAKER_01: And the client's irritated that it had to come up with those 12 months.
• 29:48
  SPEAKER_01: I mean, is is that is that really an AI question?
• 29:52
  SPEAKER_01: Or what not really.
• 29:54
  SPEAKER_02: No, I mean not an AI question.
• 29:56
  SPEAKER_02: It's it's a question of whether you have a personal or business bankruptcy, truthfully, whether you need six months or 12 months.
• 30:02
  SPEAKER_03: Well, well, that was that and that scenario also raises raises a uh another another uh simple point that you have to keep in mind as a lawyer is uh that's personal information associated with that client, correct?
• 30:16
  SPEAKER_01: Yes.
• 30:18
  SPEAKER_03: In many countries and in many in many laws, laws in the laws in certain states of the United States, that person has the right to say to you, I'm going to give you this data, but you may not use AI to process it.
• 30:30
  SPEAKER_02: They can say that, of course.
• 30:31
  SPEAKER_02: Yeah, any attorney would honor that.
• 30:33
  SPEAKER_02: Yeah.
• 30:34
  SPEAKER_01: Well, you can say, then that's why that's why attorneys who want to use AI in those contexts put that in their engagement agreement.
• 30:42
  SPEAKER_01: Yeah.
• 30:43
  SPEAKER_01: Right.
• 30:43
  SPEAKER_01: You put it, you, you, you deal with it right up front.
• 30:45
  SPEAKER_01: You get the you disclose parents, you get the consent right up front.
• 30:49
  SPEAKER_01: Yep.
• 30:49
  SPEAKER_01: And that way you know what you're dealing with.
• 30:51
  SPEAKER_01: Now, here's another question.
• 30:52
  SPEAKER_01: Let's say the attorney has that and he's got he's got a client who says, no, you're not putting my stuff in an AI chatbot.
• 31:01
  SPEAKER_01: All right.
• 31:01
  SPEAKER_01: Do you think do you think attorneys are going to be more likely to say, you know what?
• 31:06
  SPEAKER_01: Find another lawyer.
• 31:08
  SPEAKER_01: Do you see that or do you see attorneys adapting?
• 31:10
  SPEAKER_03: I think in the beginning, attorneys will feel they have to adapt.
• 31:14
  SPEAKER_03: Long run, as as the tools become more, more prevalent and more accepted, and bar associations put out their opinions about what's acceptable and what's not and the like, it'll it I think it'll become a it'll become a stage where it'll be exactly what you say, uh Ron, is is the the consent will be in the engagement letter.
• 31:33
  SPEAKER_03: And customers would will will or lawyers will basically say, you consent to my using these tools.
• 31:42
  SPEAKER_03: Or yeah, I'm not taking you as a client.
• 31:45
  SPEAKER_03: So so take it, take AI out of it for a second.
• 31:47
  SPEAKER_03: If I came to you and said, Ron, here I want you to work on this for me, but I don't want you using this internet thing.
• 31:57
  SPEAKER_03: I want you to do my case, but don't use this internet thing.
• 32:00
  SPEAKER_03: Tell me what you need, and I'll drive over and I'll bring you my stuff, and and I've got it in a shoebox back here, so I'll be able to give you whatever you need.
• 32:09
  SPEAKER_03: Do you take that client?
• 32:10
  SPEAKER_03: Do you take me as a client?
• 32:12
  SPEAKER_03: If I'm going to be that hard to work with.
• 32:14
  SPEAKER_01: You know what?
• 32:15
  SPEAKER_01: I always believed from day one that Flintstone's lawyers should you should be with Flintstone's clients and Jetson's lawyers should be with Jetson's clients.
• 32:26
  SPEAKER_01: That's fair.
• 32:27
  SPEAKER_01: Yeah.
• 32:28
  SPEAKER_01: I I I I believe very strongly in that because there are so many ways to have bottlenecks.
• 32:34
  SPEAKER_02: Do you think people are really do you think people are really categorizing themselves correctly, though?
• 32:39
  SPEAKER_02: Let's be real.
• 32:40
  SPEAKER_02: How how could anyone be choosing the right thing?
• 32:44
  SPEAKER_02: Many people probably believe that they're Jetsons that are not.
• 32:48
  SPEAKER_02: Yeah.
• 32:48
  SPEAKER_02: And vice versa.
• 32:50
  SPEAKER_02: Well, we have to do it.
• 32:58
  SPEAKER_01: I think I think the sorting hat is appropriate, Heather.
• 33:01
  SPEAKER_01: I think it's it would make everyone's life easier, sure.
• 33:05
  SPEAKER_01: I think people know when they're Flintstones and when they're Jetsons.
• 33:09
  SPEAKER_01: Yeah, I don't think they know when they're Simpsons, though.
• 33:11
  SPEAKER_01: Sometimes that's a hard one.
• 33:14
  SPEAKER_02: You're probably hovering on either side of Simpsons often.
• 33:18
  SPEAKER_01: Yeah.
• 33:18
  SPEAKER_01: But but all right, Jim, so in your articles, you draw a distinction between humans in the loop and humans on the loop.
• 33:29
  SPEAKER_04: Yeah.
• 33:30
  SPEAKER_01: And I know we're not talking about Chicago traffic.
• 33:34
  SPEAKER_01: All right.
• 33:35
  SPEAKER_01: We're talking about what's the difference?
• 33:38
  SPEAKER_01: Yeah.
• 33:38
  SPEAKER_03: Yeah.
• 33:38
  SPEAKER_03: So what we're actually talking about, funny enough, or maybe not funny, uh, that that language comes from uh decisions on autonomous weaponry.
• 33:48
  SPEAKER_03: Uh so wait, did you say autonomous weaponry?
• 33:50
  SPEAKER_03: Autonomous weaponry.
• 33:52
  SPEAKER_03: So so that language is really important.
• 33:55
  SPEAKER_03: It hits some really high-stake things.
• 33:56
  SPEAKER_03: And so it's it's a good, it's a good, it's a good way to think about it.
• 34:01
  SPEAKER_03: Because again, it's about what are the stakes when you decide to trust.
• 34:06
  SPEAKER_03: So there it's human in the loop, human on the loop, human out of the loop.
• 34:11
  SPEAKER_03: Out of the loop is what it sounds like.
• 34:13
  SPEAKER_03: No human involvement.
• 34:14
  SPEAKER_03: You let the computer do what it's gonna do, and you just hope the outcome is what it's supposed to be.
• 34:20
  SPEAKER_03: Hope is not a plan, but but in some cases, you the accuracy may be good enough that you can rely on the machine just to do it and you you trust.
• 34:30
  SPEAKER_03: Okay, that's fine.
• 34:32
  SPEAKER_03: Those are those are more rare cases in my mind, certainly for a lawyer.
• 34:35
  SPEAKER_03: On the loop, which is what really, when you talk about verification, that's really what the lawyers are, what a lawyer with AI these days is going to be doing.
• 34:46
  SPEAKER_03: Summarize the 180 pages.
• 34:48
  SPEAKER_03: I'm not gonna go back and reread the 180 pages just to make sure that the summary was correct.
• 34:54
  SPEAKER_03: But I am going to use my training and my judgment to go through that, and things are gonna jump out at me as, huh, that doesn't seem right.
• 35:03
  SPEAKER_03: Is that really what that that that court said?
• 35:06
  SPEAKER_03: I better, I better, I better look at that.
• 35:08
  SPEAKER_03: I better look at that section and take and then read that, go back.
• 35:11
  SPEAKER_03: Nah, it didn't quite get it right.
• 35:13
  SPEAKER_03: It missed a nuance.
• 35:14
  SPEAKER_03: I better deal with that.
• 35:15
  SPEAKER_03: And we'll go through that's human on the look.
• 35:19
  SPEAKER_03: The human is watching the process and is part of and is part of the the cognitive flow that's happening, and says, Whoa, wait a minute.
• 35:28
  SPEAKER_03: In an autonomous weaponry situation, that would be the system saying firing the missile in in five, four, and the person going, wait, that's a flock of birds, or that's it, or that's an airline.
• 35:41
  SPEAKER_03: Yeah, or that's an airline, and you know, that's a commercial airline, and they stop it.
• 35:45
  SPEAKER_03: So that's all in the loop.
• 35:47
  SPEAKER_03: In the loop is nothing happens until I've said go.
• 35:52
  SPEAKER_03: And you talk to compliance lawyers and they'll tell you, well, it's gotta be a human in the loop.
• 35:57
  SPEAKER_03: Economically, that makes no sense.
• 35:58
  SPEAKER_03: Why are you gonna spend all this money on these systems, trust them to do things, and then stop them every time and make a human do the work anyway?
• 36:06
  SPEAKER_03: That doesn't make sense economically, and that's not where this whole thing is going.
• 36:10
  SPEAKER_03: So on the loop is probably where we are.
• 36:12
  SPEAKER_03: But that's what that's what that's what those three refer to.
• 36:14
  SPEAKER_03: Trust question again.
• 36:16
  SPEAKER_03: We get very wrapped up or uh we're very wrapped around the axle on this question of can we trust the computer?
• 36:23
  SPEAKER_03: When is computer going to be wrong?
• 36:25
  SPEAKER_03: And as a compliance lawyer, that's a my that's my first question.
• 36:28
  SPEAKER_03: You know, I'll pound the table and say, but they don't know anything.
• 36:31
  SPEAKER_03: You can't trust them to get it right.
• 36:34
  SPEAKER_03: But I understand that we're gonna use it.
• 36:36
  SPEAKER_03: So let's take the AI out of it again.
• 36:40
  SPEAKER_03: Is the AI going to do better or worse than the third-year associate who's had four hours of sleep in the last three days, who's been asked to sit and go through these documents instead of going to the rave, is is is tired, resentful, overworked, and uh is is making decisions that nobody else is going to be able to verify because nobody else is going to sit back down and do all that review.
• 37:10
  SPEAKER_03: Do we trust that?
• 37:11
  SPEAKER_03: Do we trust that lawyer?
• 37:13
  SPEAKER_03: And is that and how different is that than trusting the AI knowing it is going to make the occasional mistake?
• 37:22
  SPEAKER_01: Well, on those terms, you have another phrase that I really liked in your articles, and that is compliance theater.
• 37:31
  SPEAKER_01: Yes.
• 37:32
  SPEAKER_01: And you have you have a phrase like, you know, an audit log nobody reviews isn't actually a safeguard.
• 37:41
  SPEAKER_01: What does that really mean?
• 37:42
  SPEAKER_01: How does that play out in real world terms?
• 37:46
  unknown: Right.
• 37:46
  SPEAKER_03: So in real world, so that that comes from the pure compliance side of things, where from a systems approval point of view, from a workflow approval point of view, you get a given use case using using client data to analyze and come up with a an answer, you know, on a bankruptcy or whatever.
• 38:10
  SPEAKER_03: And you are worried that that uh that system that system is going to occasionally make mistakes.
• 38:18
  SPEAKER_03: So you build a logging system in that's looking for certain indicators that would indicate that something has gone wrong with your model.
• 38:25
  SPEAKER_03: Where you see this a lot is in is in uh systems designed to protect against cyber attacks, where you log for certain things that show up and would indicate that there's a problem.
• 38:36
  SPEAKER_03: And so those, so those, so those are those are data collected in order to give you the ability to see if things are going wrong or to make a judgment as to whether they are going right.
• 38:48
  SPEAKER_03: So you will tell your regulator, I've got these checks.
• 38:51
  SPEAKER_03: I do these logs and it's logged every 15 minutes and and I log all the following things.
• 38:58
  SPEAKER_03: And the uh the regulator goes, that looks really good.
• 39:00
  SPEAKER_03: That's exactly the kind of kind of checking we we need you to be doing to keep this safe for people.
• 39:06
  SPEAKER_03: If I never if I never hire the person whose job it is to look at those logs, or I never set up the AI system to go scan those logs, or whatever.
• 39:15
  SPEAKER_03: If I never then do something with those those checks, that's that's compliance theater.
• 39:21
  SPEAKER_03: I've created the image of that I've got controls, but nobody's at the controls.
• 39:27
  SPEAKER_03: And for in a legal framework, somebody always needs to be at the controls.
• 39:32
  SPEAKER_01: What I'm hearing is that is that uh accountability is not enough.
• 39:37
  SPEAKER_01: And that for certain high-risk matters, you gotta catch things before they go wrong.
• 39:45
  SPEAKER_01: Knowing why it went wrong after the fact because you have that log, that's not enough.
• 39:49
  SPEAKER_03: You've got to catch it at some point.
• 39:51
  SPEAKER_03: Yeah.
• 39:52
  SPEAKER_01: I mean, it it's you won't because I think an audit log is a great thing.
• 39:57
  SPEAKER_01: Because it's gonna show me exactly what happens and when and and and who's responsible or not responsible.
• 40:03
  SPEAKER_01: Yeah.
• 40:04
  SPEAKER_01: But there's value.
• 40:05
  SPEAKER_03: But are you gonna catch everything, 100% of things before they happen?
• 40:09
  SPEAKER_03: Probably not.
• 40:09
  SPEAKER_03: You're not.
• 40:10
  SPEAKER_03: No, you it's a law of averages.
• 40:12
  SPEAKER_03: Are you gonna build a control that allows you to catch most things before they happen?
• 40:16
  SPEAKER_03: And we know some bad things are gonna get through.
• 40:18
  SPEAKER_03: Perfect systems, perfect solutions are really expensive, and nobody does them.
• 40:24
  SPEAKER_03: That's why you have acceptable error rates and you have acceptable because you nobody can afford consumers couldn't buy products if they were if they were deemed 100% safe at all times.
• 40:35
  SPEAKER_03: So, so yeah, it but but but that's right.
• 40:38
  SPEAKER_03: The person responsible for the auditing, the person responsible for the compliance, has to be engaged and has to be part of the process.
• 40:47
  SPEAKER_03: The minute the lawyer steps out of the out of it and says, I'm not involved anymore, it doesn't matter.
• 40:53
  SPEAKER_03: There was there was one uh something I read recently, and I love the way this lawyer at Cornell put it.
• 41:00
  SPEAKER_03: He referred to the non-delegable duty to think.
• 41:05
  SPEAKER_03: And that's what lawyers have.
• 41:07
  SPEAKER_03: We cannot delegate our duty to think about what we are telling our clients.
• 41:14
  SPEAKER_03: In this case, in the in the lawyer at Cornell, the the professor at Cornell was talking about judges and judges using AI in their opinions.
• 41:22
  SPEAKER_03: Uh but there's a non as a lawyer, there's a non-delegable duty to think.
• 41:26
  SPEAKER_03: And so we have to understand how these tools work, and we have to understand where their risks are, and we have to make judgments as to when we're going to use them and when we're not.
• 41:35
  SPEAKER_03: It's it's really not as complex as we make it to be, even though they are very complex tools.
• 41:44
  SPEAKER_03: But we had the same battle when when when Google search became a thing.
• 41:49
  SPEAKER_03: And oh, we can't use Google search because you know you can't trust it.
• 41:52
  SPEAKER_03: And I'm sure way back in early on in the day when uh when we first had computers, questions came about.
• 41:58
  SPEAKER_03: I was once told by a a prof uh a teacher at my in my high school that I couldn't use a computer to take my notes because the the that that teacher thought I was just pressing a button and the computer was printing out all the information I needed from his class.
• 42:13
  SPEAKER_03: Every new technology goes through this process and it goes through this analysis of when can we trust and when can't we trust.
• 42:19
  SPEAKER_01: And that's R and AI.
• 42:21
  SPEAKER_01: What are you gonna do if that wheel breaks?
• 42:23
  SPEAKER_01: How are you gonna how are you gonna get your grain to the market?
• 42:26
  SPEAKER_01: Yeah, yeah, exactly.
• 42:27
  SPEAKER_01: Yeah.
• 42:28
  SPEAKER_03: Yeah.
• 42:28
  SPEAKER_03: I'll just keep dragging it.
• 42:29
  SPEAKER_03: I'll just keep putting it on my back and carrying it, or I've got my I've got my donkey or whatever.
• 42:33
  SPEAKER_03: Yeah.
• 42:34
  SPEAKER_01: That's right.
• 42:34
  SPEAKER_03: That's right.
• 42:35
  SPEAKER_01: Well, well, this brings us to our practice signals segment of the show, which is where we look at at reports from lawyers in the wild and the the struggles they're having.
• 42:47
  SPEAKER_01: And maybe there's an AI solution, maybe there's not an AI solution, but it's it's worth thinking about.
• 42:54
  SPEAKER_01: So this one is they all come from Reddit because you know everything's anonymous.
• 43:01
  SPEAKER_01: And people will say whatever they want.
• 43:03
  unknown: All right.
• 43:04
  SPEAKER_03: A lot of AIs are trained on Reddit.
• 43:06
  SPEAKER_01: That's kind of interesting.
• 43:08
  SPEAKER_01: I think that's true.
• 43:09
  SPEAKER_01: All right.
• 43:10
  SPEAKER_01: So a firm owner has an associate who's kind and hardworking, which you really can't train those two traits, and they're very valuable, but is struggling badly with vocabulary, grammar, reading comprehension, nuance, focus, and even completing forms without skipping things.
• 43:34
  SPEAKER_01: So, you know, I don't know that.
• 43:35
  SPEAKER_01: I don't know how they did on the LSAT if they can't do those, but they became a lawyer.
• 43:40
  SPEAKER_01: The owner is genuinely trying to help this person, but says, I'm worried that if we can't fix the vocabulary and comprehension problems, this lawyer is gonna get the firm sued.
• 43:54
  SPEAKER_01: I mean, I think that's more of a client facing issue than really getting sued, but but that's the real concern.
• 44:00
  SPEAKER_01: All right.
• 44:01
  SPEAKER_01: So my first reaction is well, you know, AI can compensate for a lot of these problems.
• 44:09
  SPEAKER_01: Is that a good thing?
• 44:10
  SPEAKER_01: Or could AI actually make this associate more dangerous?
• 44:15
  SPEAKER_03: It kind of goes to exactly in my mind what we were just talking about, what we've been talking about this entire time.
• 44:21
  SPEAKER_03: If you're saying that that associate's incapable for some reason to be able to do these things, if you have the AI do it, one side says, yeah, it's more dangerous because when the AI gets it wrong, there's no one watching that, there's nobody minding the store.
• 44:37
  SPEAKER_03: There's nobody that's going to be able to say, well, wait, that's not right.
• 44:40
  SPEAKER_03: That's wrong.
• 44:41
  SPEAKER_03: That's not what I meant.
• 44:42
  SPEAKER_03: They're just going to accept it and go.
• 44:44
  SPEAKER_03: And uh, you know, the person that accepts the output of the AI without question is the dangerous voter.
• 44:52
  SPEAKER_03: You know, the voter who never bothers to learn what they're voting on.
• 44:55
  SPEAKER_03: They just vote one way or the other without, without, you know, they just check a box.
• 45:00
  SPEAKER_03: The so that it could be very, it could be very dangerous.
• 45:02
  SPEAKER_03: But used correctly, there are tools like fixing grant, you know, fixing the grammar, fixing the vocabulary, and things like that could actually be very valuable and might get the help get the person to a point that that they have a cognitive process for doing their work that uses that tool to improve their day-to-day.
• 45:22
  SPEAKER_03: And so, yes, there could very well be a very good path forward for that where you're using those tools.
• 45:28
  SPEAKER_03: And and I use I use AI that way.
• 45:30
  SPEAKER_03: I'll one of those articles that uh that you're talking about here, the the uh the editor came back and said, Well, this is you're writing like you're in grad school, and we need we need to quote dumb this down, unquote, a bit to make it to make it more easily accessible and readable.
• 45:45
  SPEAKER_03: And so I went I went to Claude and I said, make that make this more accessible and more conversational uh to for the average reader.
• 45:55
  SPEAKER_03: And by God, it did it.
• 45:56
  SPEAKER_01: And it's not bad.
• 45:58
  SPEAKER_01: I thought it was great.
• 45:59
  SPEAKER_01: And by the way, what was really great to me was in the text message you sent me with that article, you you said that it was going to be written on the grade school level, and then you changed it to grad.
• 46:16
  SPEAKER_01: And I love that that that spectrum from grade school to grad school with just a AI tool silent D.
• 46:23
  SPEAKER_03: That was the AI tool in in my messaging app that decided I meant grade and not grad the first.
• 46:31
  SPEAKER_02: Ron, you could get a design to develop something like bench sim to help disassociate the this uh problem, you know.
• 46:38
  SPEAKER_02: So you you I was gonna ask you, Heather, what you thought about this practice.
• 46:43
  SPEAKER_02: I think Chris Ryan would be the perfect guy to talk to.
• 46:45
  SPEAKER_02: He created Bench Sim to help new associates learn to argue in court better.
• 46:49
  SPEAKER_02: And I think that, or law students, I don't remember, was it law students or new associates?
• 46:55
  SPEAKER_02: But remember, he built an AI program to help train young associates or young lawyers to get past their sort of nerves and learn to argue in court better.
• 47:05
  SPEAKER_02: This is a, I feel like there could be an AI tool developed to help this attorney develop and improve those skills.
• 47:12
  SPEAKER_01: I think so too.
• 47:13
  SPEAKER_01: I think that by the way, the Benjam AI example is is fantastic because it really is a a training opportunity that that is created by AI.
• 47:24
  SPEAKER_01: By the way, for listeners who may remember the Benjam AI episode, he has sub, he has subsequently gotten his stock to compliance and he's now video capable.
• 47:40
  SPEAKER_01: Very exciting.
• 47:41
  SPEAKER_01: So that's really excited for Chris.
• 47:43
  SPEAKER_02: Yeah, and uh and I maybe you send him this uh practice signal and say, what could you do for this poor young associate?
• 47:49
  SPEAKER_01: That's maybe I will.
• 47:51
  SPEAKER_01: Maybe I will.
• 47:52
  SPEAKER_01: All right.
• 47:53
  SPEAKER_01: So yeah, I think AI is is the right answer because you can't train kindness.
• 47:59
  SPEAKER_01: And you know, clients will respond to that.
• 48:04
  SPEAKER_01: And if it's the right client, right?
• 48:06
  SPEAKER_01: If you've got a client that's just, you know, a meat eater, a fire breather, you don't want the kind, hardworking associate on that case.
• 48:16
  SPEAKER_01: But you know what?
• 48:17
  SPEAKER_01: If you're a personal bankruptcy lawyer or a divorce lawyer or a criminal lawyer, a kind, hardworking lawyer might really resonate with that client.
• 48:29
  SPEAKER_01: And if you have a tool that's going to elevate their work product, that really might be the answer.
• 48:37
  SPEAKER_03: We keep we we we keep saying the word tool, and I want to make sure we we we focus on that for a second.
• 48:42
  SPEAKER_03: These are tools.
• 48:43
  SPEAKER_03: The AI is a tool, and we get wrapped around the axle on things like, oh, yeah, do they think, or like I did with do they have knowledge, or our our killer robot's going to take over the world or whatever.
• 48:56
  SPEAKER_03: But the when you br when you bring it back down to prostacks, AI is a tool that's here to stay.
• 49:04
  SPEAKER_03: And we as l as lawyers have to have to carefully and and and with accountability on our own, under our own ethical rules, think about where those tools can help us do a better job for our clients, and then where we shouldn't use them because the client hired us, not the AI tool.
• 49:27
  SPEAKER_03: And and when and in finding that balance, we're still in the middle of trying to figure out where that is.
• 49:32
  SPEAKER_03: Uh, and it's something that's going to continue for a while longer because the the technology is moving at high speed.
• 49:38
  SPEAKER_03: But as long as we maintain that awareness and we accept our ethical role as lawyers, we should be, we should be able to walk this path.
• 49:48
  SPEAKER_01: And it brings me to our season three premiere of the final segment, which is the users and builders segment.
• 49:58
  SPEAKER_01: You know, we went through Flintstone Simpsons Jetsons and tips.
• 50:02
  SPEAKER_01: And candidly, if you've been listening to this podcast or viewing it from the beginning, you're way beyond Flintstones.
• 50:12
  SPEAKER_01: Where and it's not really a valuable, in my opinion, it's not really a valuable exercise to figure out ways to help Simpsons or Flintstone's lawyers become a little bit more AI fluent.
• 50:26
  SPEAKER_01: I I think we're past that.
• 50:28
  SPEAKER_01: And uh I, you know, I think I I've said this before.
• 50:30
  SPEAKER_01: I think 2026 is gonna go down as the turning point, the inflection point in AI development, unless amazing things happen in 2027.
• 50:40
  SPEAKER_01: But 2026 started, and we were first starting to really kind of get into the rhythm of these, you know, LLMs.
• 50:48
  SPEAKER_01: And now everybody is like a builder.
• 50:51
  SPEAKER_01: You should you need to build with these tools, you need to create these skills in Claude.
• 50:57
  SPEAKER_01: And I don't think that every lawyer that is already involved in the AI tool use wants to build.
• 51:06
  SPEAKER_01: And I think they just they want to continue to buy off-the-shelf products and and use them well.
• 51:12
  SPEAKER_01: So that's really the point of the users and builders season, where we're gonna try to meet lawyers where they are, whether they're users or builders.
• 51:23
  SPEAKER_01: Jim, what do you think about that?
• 51:25
  SPEAKER_03: Oh, I think I think that's an interesting way to way to uh to separate it.
• 51:29
  SPEAKER_03: And there will be those like like myself who are a little bit of both.
• 51:33
  SPEAKER_03: Uh yeah, a little bit of country, a little bit of rock and roll.
• 51:35
  SPEAKER_03: But uh the the uh the the AI is very, very quickly getting to the point that maybe goes against what I said earlier was that that that you can't use these out of the box.
• 51:45
  SPEAKER_03: There really is a lot of good out-of-the-box capability there that the builders who understand the law and who understand these tools are building for the users.
• 51:55
  SPEAKER_03: And I I think you're right.
• 51:56
  SPEAKER_03: We're going to get to a point to where there will be things we can just pull down and use.
• 52:00
  SPEAKER_03: And we just have to decide again is the is this the right tool for this use case.
• 52:07
  SPEAKER_01: Heather, what do you think about season three users and builders?
• 52:10
  SPEAKER_02: I love it.
• 52:11
  SPEAKER_02: I love the whole concept.
• 52:13
  SPEAKER_02: I think that if you're building, you obviously are or should be a very sophisticated user, but there are a lot of people just using AI very intelligently that don't have a need to build, and that's okay.
• 52:27
  SPEAKER_01: So, so the first tip that I have, because I've been insanely involved in building uh a product that I hope to launch sooner rather than later, is make failure visible.
• 52:43
  SPEAKER_01: You're gonna fail.
• 52:44
  SPEAKER_01: Your your your work is gonna fail.
• 52:46
  SPEAKER_01: Don't just build the happy path.
• 52:48
  SPEAKER_01: You know, design the system to recognize uncertainty, missing information, and the systems requiring human judgment.
• 52:56
  SPEAKER_01: Don't expect that what you're building is gonna come out of the box and is going to be just fantastic from the word go.
• 53:04
  SPEAKER_01: So build failure into your system.
• 53:07
  SPEAKER_01: Uh, and and that will expose its limits.
• 53:10
  SPEAKER_01: And there's value in that.
• 53:12
  SPEAKER_01: Yeah, absolutely.
• 53:13
  SPEAKER_01: You know, now if you're a user, you're gonna be having intense and in-depth conversation with vendors.
• 53:20
  SPEAKER_01: So ask the vendor to show you what happens when AI doesn't know.
• 53:26
  SPEAKER_01: You know, show me what your product does when the documents conflict.
• 53:30
  SPEAKER_01: Show me what happens when information is missing.
• 53:33
  SPEAKER_01: Show me what happens when it's uncertain.
• 53:36
  SPEAKER_01: So the tip is ask better questions.
• 53:40
  SPEAKER_01: If you're gonna be a user and you're gonna be relying upon the output of a vendor, learn to ask the right questions.
• 53:49
  SPEAKER_03: And and uh core uh along with that, Rod, is you is uh to the extent that you're using a tool with a chat interface, like Chat GPT and things like this, learn to ask and ask again.
• 54:02
  SPEAKER_03: The the the best output from these tools is a conversation where you say, is that really right, or is that really what it said, or didn't they change that law?
• 54:12
  SPEAKER_03: And uh you that interrogation process to using the tool as a user will help to cover up any of those those those failures that haven't been made visible as you as as you just discussed.
• 54:25
  SPEAKER_03: So it's I think I think this is gonna be an interesting, uh, interesting set of uh segments.
• 54:30
  SPEAKER_03: I'm I'm looking forward to it.
• 54:31
  SPEAKER_01: What do you think, Heather?
• 54:32
  SPEAKER_01: Uh pushing back against uh against your uh chat bot.
• 54:36
  SPEAKER_02: I always do.
• 54:38
  SPEAKER_02: But I mean everyone really misses that line at the bottom that says chat GPT can make mistakes, workspace data isn't so it was uh I think that that's something you know we've stressed always is I've always said collaborate, keep talking, interact with your AI.
• 54:54
  SPEAKER_02: But my always tells me, like, it's good that you're pushing back on this.
• 54:58
  SPEAKER_02: It often says that to me.
• 55:00
  SPEAKER_02: Right.
• 55:01
  SPEAKER_02: All right.
• 55:02
  SPEAKER_02: So so Jim, what do you think?
• 55:03
  SPEAKER_01: Any parting thoughts?
• 55:05
  SPEAKER_03: No, I I think what if if if somebody takes anything from like my articles or what I've been saying is as a lawyer, as counsel, we have to be ready to answer the question what do we do with the confidently wrong answer?
• 55:23
  SPEAKER_03: And if we can answer that and know how we're gonna use the tool with that in mind, I think we're gonna find our way through.
• 55:31
  SPEAKER_03: AI is going to be a very valuable tool.
• 55:33
  SPEAKER_03: If we ignore that problem, we're not gonna stay a lawyer for very long in this world.
• 55:38
  SPEAKER_01: Jim, I couldn't have said that better.
• 55:41
  SPEAKER_01: Thank you so much for being here.
• 55:43
  SPEAKER_01: I I think this was a really useful episode.
• 55:46
  SPEAKER_01: Um, we we sometimes get distracted by all the the bright, shiny AI tools, and and we lose the reality that you know we have to be careful.
• 55:57
  SPEAKER_03: I'll always be I'll always come on to be your philosoph your AI philosopher.
• 56:01
  SPEAKER_03: I'll uh we'll talk Platoh, we'll talk Chalmers, we can we can do all kinds of fun stuff.
• 56:08
  SPEAKER_01: Well, listen, this has been another episode of AI Tools for Practicing Lawyers.
• 56:13
  SPEAKER_01: Thank you so much for listening, and we'll see you next time.
• 56:16
  SPEAKER_03: Thank you.