Latest / CLEs You Actually Want to Hear / What AI Hallucinations Can Teach Lawyers About Judgment, Verification, and Professional Responsibility
Transcript
- 0:03Hello, everyone. Welcome to today's program,
- 0:06What AI Hallucinations Can Teach Lawyers About
- 0:09Judgment, Verification, and Professional Responsibility.
- 0:13I'm Alapoli Castro, Director of Education and
- 0:16Professional Development at Beverly Hills Bar
- 0:18Association. And now it's my pleasure to introduce
- 0:20our speaker for today, Professor John Landy.
- 0:23Professor Landy is the Isidore Lowe Professor
- 0:26Emeritus at the University of Missouri School
- 0:29of Law, where he directs the LLM program in dispute
- 0:33resolution. He holds a JD from Hastings College
- 0:36of Law and a PhD in sociology from the University
- 0:39of Wisconsin Madison, and he has practiced law
- 0:41and mediation since 1980. A prolific scholar
- 0:45and author, Professor Landy has been recognized
- 0:47by the American Bar Association's Section of
- 0:50Dispute Resolution with the Award for Outstanding
- 0:52He has authored multiple ABA published books
- 0:57and has developed the Real Practice Systems Negotiation
- 1:00and Mediation Coach AI tool to promote good decision
- 1:04-making by parties, lawyers, and mediators. And
- 1:07this is a public service tool available for free,
- 1:09by the way. He writes the theory meets practice
- 1:12column for alternatives to the high cost of litigation
- 1:15and contributes regularly to the indisputably
- 1:18.org blog. Professor Landy looks forward to answering
- 1:22all your questions today. So please submit them
- 1:24to the Q &A box on your screens. And now I'm
- 1:27going to turn it over to him. Thank you very
- 1:30much, Ala. And thank you for inviting me to.
- 1:34uh give this presentation and thank you all for
- 1:38taking the time to come and spend some time uh
- 1:42on this so i am going to share my screen now
- 1:46and hopefully it's going to work right okay can
- 1:50you all see this yes it looks good okay so um
- 1:59you will be getting a copy of this powerpoint
- 2:02which includes links to some resources at the
- 2:06end. And as Allah said, feel free to use the
- 2:10chat with questions and comments. And I can stick
- 2:15around at the end. We may have enough time during
- 2:17this program to answer any questions you have.
- 2:20But if we get to the end of the half hour and
- 2:23you want to stick around and ask more questions,
- 2:25I'm happy to answer them. So there we go. So
- 2:30who do you trust? Now, the question with hallucinations
- 2:36is, do you trust AI? And of course, the answer
- 2:40with hallucinations is no, you shouldn't. You
- 2:44need to verify. And that's actually not, I mean,
- 2:49it's different in some ways, but you need to
- 2:51verify things from humans. So for example, with
- 2:55your clients. Now, of course, presumably, you
- 2:59would assume that your clients are being honest
- 3:03with you and telling the truth, but you may find
- 3:06that they don't tell the whole truth and nothing
- 3:10but the truth for a variety of reasons that don't
- 3:14necessarily indicate bad intent. And what about
- 3:17experts? Now, of course, they have, by definition,
- 3:20expert knowledge. So presumably you would trust
- 3:25them. theoretically, although a lot depends,
- 3:30their answers may be related somehow to who has
- 3:33retained them. But judges, judges, of course,
- 3:37you're going to trust judges, they're neutral.
- 3:40Except lawyers don't always trust judges. And
- 3:44indeed, on average, you spend half of your time
- 3:47arguing that judges are wrong. And indeed, some
- 3:50of the time you actually believe that. And you
- 3:53need to identify reasons why they're wrong and
- 3:58argue why your view is correct. I didn't put
- 4:04it on the slide here, but your spouse or your
- 4:07partner, I mean, for sure, your spouse or partner
- 4:10is always, no, they're probably, you wouldn't
- 4:13tell them that they're not always accurate, but
- 4:18they may not always be. And what about yourself?
- 4:22Well, you can owe it. No, you can't always trust
- 4:25yourself either because you don't know everything
- 4:28and sometimes you make mistakes. So the point
- 4:30obviously of this slide is that every human and
- 4:36also AI makes mistakes and you need to verify
- 4:41them. And indeed, what lawyers are is we are
- 4:47professional skeptics. We are looking at information
- 4:52given to us by a wide range of humans, and we
- 4:57don't necessarily agree with them all. And so
- 5:02there are things that lawyers do routinely. This
- 5:05is part of our job. You ask people questions.
- 5:09You will look at information. You'll do some
- 5:11research so that you can have more confidence.
- 5:15in the information that you receive and that
- 5:18the information you put out in oral statements
- 5:23and in writing, particularly in materials that
- 5:26you submit to the court. So what about hallucinations?
- 5:31Now, again, you wouldn't be here if you hadn't
- 5:34heard of hallucinations and you didn't think
- 5:36that they were significant in some way. You probably
- 5:40know that, of course, hallucinations are inaccurate
- 5:43outputs that may sound plausible. And indeed,
- 5:47it can be very tricky because they do sound so
- 5:50plausible. And so if you use AI and you find
- 5:53a site that just sounds too good to be true,
- 5:56or that sounds like it really nails the point
- 5:59that you want to make, it would be tempting to
- 6:01just believe it and cite it. But you know that
- 6:05hallucinations are things that AI tools produce
- 6:09that certainly we hear about fabricated sites
- 6:13of cases, but also they make up quotations, rules,
- 6:18facts, and summaries. So there's a lot of stuff
- 6:21that's inaccurate, and they're extremely visible.
- 6:25A lot of the mistakes that humans make are not
- 6:30so visible. In some ways kind of funny, but actually
- 6:35they're embarrassing and problematic. They're
- 6:38funny if you're not the one who made the submitted
- 6:43some materials with hallucinated citations, but
- 6:48they're embarrassing and problematic. So if you
- 6:51do one, you are subject to sanctions by the court.
- 6:56You doesn't instill confidence by your client.
- 7:00If you have. submitted a paper with hallucinated
- 7:05authorities. Your colleagues in your firm probably
- 7:09don't think well of you and it's not good for
- 7:13your reputation as the person who submitted papers
- 7:16with hallucinated citations. And it's also not
- 7:19good for the legal system to have all these news
- 7:22stories popping up about lawyers who are sanctioned
- 7:27and sheepishly have to acknowledge that they
- 7:30have submitted AI output without having carefully
- 7:35checked it. There's a database that keeps track
- 7:39of reported cases where lawyers have been sanctioned,
- 7:43and it's this running total. The last time I
- 7:46checked, there were more than 1 ,250 reported
- 7:49cases in the United States. There are cases all
- 7:52over the world, and courts have imposed sanctions.
- 7:57So it's important, obviously, that you read material
- 8:01carefully. And it's important not only to read,
- 8:04obviously, your own material, but also others.
- 8:07Even if you do not use AI yourself, you may find
- 8:19that your opposing counsel has submitted material
- 8:25that has hallucinations in it. So you need to
- 8:30check carefully. Now, theoretically, if you get
- 8:33material from your counterpart lawyer, your opposing
- 8:36counsel, you should be reading it carefully with
- 8:38an eye towards picking out what you see as errors
- 8:43in their argument. And you should be reading
- 8:46their their sources to make sure that they're
- 8:50valid. So it's really important that you obviously
- 8:54check for hallucinations and indeed There are
- 8:57rules about this and the rules require you to
- 9:01do what you should be doing anyway. So the basic
- 9:05one is that you're responsible for the accuracy
- 9:08of your work and you can be sanctioned if you
- 9:12provide false information to the courts. And
- 9:16there are a whole range of duties under the rules
- 9:22of professional conduct relating to competence,
- 9:25diligence, candor, reasonable inquiry and others
- 9:29that apply to the use of AI. And you should be
- 9:35familiar with them. And again, certainly if you
- 9:39use AI in your work, but even if you don't, it's
- 9:42important to be aware of them. The American Bar
- 9:46Association issued this formal opinion 512 in
- 9:502024 that was specifically about AI. And as you
- 9:54may know, there are some proposals for revisions
- 9:58of the California rules or actually comments
- 10:00to the rules that would make it much more specific
- 10:05about requiring verification and taking care
- 10:09to use AI output carefully. So there are some
- 10:17surprising values of hallucinations, and they
- 10:20asked me to do this presentation because I wrote
- 10:23a an article with that title, and you might wonder,
- 10:29well, what the heck good is a hallucination if
- 10:34it is false? And the answer is that it can make
- 10:40you think, it can give you ideas, and it can
- 10:42require your judgment, which is really the key
- 10:46thing that lawyers have is your judgment in lots
- 10:49of contexts, not just with AI. So I think about
- 10:54when I was practicing law a long time ago, I
- 10:57don't practice now. I've been a law professor
- 11:01and now I'm a retired law professor. But I remember
- 11:05when I was practicing, I had an experience that
- 11:08I suspect that probably everybody on this call
- 11:11has had where I got a citation to something that
- 11:14looked like a promising case or authority and
- 11:17I'd look at it and it didn't really help. the
- 11:22argument that I wanted to make. But in the course
- 11:25of finding this case that wasn't so helpful,
- 11:30I found other things that actually were helpful.
- 11:33And that's a surprising value of hallucinations.
- 11:37It can lead you to find things that you might
- 11:40not otherwise have found that are really helpful.
- 11:43And it can prompt you to think and give you ideas.
- 11:47So it can stimulate your thinking. And you can
- 11:50think about it this way. And the fact that AI
- 11:54routinely now does produce hallucinations forces
- 11:59you to think carefully. Consider a counterfactual.
- 12:05Suppose that AI produced responses that were
- 12:09100 % accurate. You could take it to the bank.
- 12:13It gave you a citation. You knew that the case
- 12:16existed, that the point that it was making was
- 12:20on point, well, what would happen? You would
- 12:24lose an incentive to really use your judgment.
- 12:28You would just become complacent. And part of
- 12:32what having the value of hallucinations is, it
- 12:35forces you to actually think, to look, and to
- 12:39consider what makes sense or not, and consider
- 12:43the opposite. I talked with a lawyer. who told
- 12:48me that she used AI and it made a mistake. And
- 12:53so she doesn't use it anymore. And well, that
- 13:00isn't that, I mean, everybody makes mistakes.
- 13:03You think of your clients, your experts, judges,
- 13:06yourself, even your spouses and partners. Are
- 13:09you going to get rid of them because they make
- 13:11mistakes? No, you you used your judgment and
- 13:15you think about how you would assess various
- 13:17statements and try and figure out what makes
- 13:20sense or not. So here's an example. So consider
- 13:26if you asked AI to suggest sites for writing
- 13:31a brief and it provides a bunch of really good
- 13:34sites, a questionable one, and a hallucination.
- 13:39Is that a good response? For the lawyer I was
- 13:48talking about who said well, there's a hallucination.
- 13:51I'm not going to use AI. It's not a good response,
- 13:54but I would submit it is a good response because
- 13:58you're getting a bunch of really helpful sites
- 14:02and you should be checking to determine whether
- 14:06any of them are accurate and helpful for your
- 14:11purpose. So the fact that there are hallucinations.
- 14:15in and of itself should not be a problem. It's
- 14:21really a question of whether the cost -benefit
- 14:24makes sense. And you obviously have to think
- 14:28about when you're going to use it and how much
- 14:32you will find that it's helpful for doing this
- 14:36research. And of course, one of the things that's
- 14:38happening, we are at this very early stage relatively
- 14:41in the development of AI. And so you have these
- 14:46tools that are getting better and better. From
- 14:49my experience, it's going to be a long time before
- 14:53they can develop these tools that don't produce
- 14:58hallucinations. But I don't know. We're going
- 15:02to have to see. But at least in the immediate
- 15:05future, you have to assume that it will and that
- 15:08your job is to use AI if you're going to use
- 15:13it as well as possible and figure out how to
- 15:16use it. So what do you do if you do use AI? Well,
- 15:22you want to check if the source actually exists.
- 15:25One of the most embarrassing kinds of hallucinations
- 15:28is when lawyers file papers with citing cases
- 15:34that just do not exist at all. So those should
- 15:37be relatively easy to to identify and then looking
- 15:44to see if the sources are current and authoritative
- 15:48and whether they support the proposition asserted
- 15:52and whether they omit important qualifications
- 15:55or ambiguities. These are the sorts of things
- 15:58that lawyers should be doing anyway. So if you
- 16:00get a paper from the other side or you get, and
- 16:05in some ways you might think of an AI as a paper
- 16:10or sites that are as if you're getting them from
- 16:14the other side that you really need to scrutinize
- 16:16carefully as if you're reading a paper from the
- 16:20opposing party. So given the fact that there
- 16:25are hallucinations, is it even worth it? I mean,
- 16:31verification takes time. But Research and verification
- 16:37has always taken time. It's not as if it's an
- 16:43immediate process. Doing research and figuring
- 16:47out what authorities are relevant and helpful
- 16:52for your case is a time -consuming process. So
- 16:57whether the verification of AI responses takes
- 17:01more time, I'm not sure that it does. but you're
- 17:05going to need to do that in any case. And whatever
- 17:09additional time you spend to verify sources may
- 17:15be offset by the gains in efficiency as well
- 17:20as generating brainstorming ideas, helping you
- 17:24draft and finding sources that you might not
- 17:27find so easily. So using AI thoughtfully, can
- 17:33improve both the efficiency and your judgment.
- 17:36And I don't practice law now. I use AI every
- 17:41day because I write a lot. And fortunately, I
- 17:47don't. Well, sometimes I use it to find sources.
- 17:52A lot of the sources that I use are social science
- 17:55sources. And I find that sometimes it produces
- 18:00hallucinations. It cites to articles that just
- 18:04do not exist or don't make the point that it's
- 18:08arguing or that it suggests. But a lot of times
- 18:11it produces good ones that are helpful, just
- 18:15like the hypo that I was talking about. And so
- 18:17for me, it's really worthwhile, it's very efficient,
- 18:21and it gives me ideas of things to look into
- 18:26and follow up, as I mentioned earlier. Now, one
- 18:30of the things is that we've been talking so far
- 18:35about uses where there is a single right answer.
- 18:41So does this case exist or not? Does it say,
- 18:46stand for the proposition that the AI output
- 18:49provides or not? There's a right answer to that.
- 18:54But there are a lot of uses of AI. than involve
- 18:58judgment rather than getting a correct answer.
- 19:01I mean, AI is not necessarily a truth -telling
- 19:06machine. It's not like a machine at a carnival
- 19:09where you stick in a quarter and you're supposed
- 19:10to get the right answer. I mean, that's part
- 19:14of the way some tools work, but AI can involve
- 19:18things that stimulate your judgment or promote
- 19:21your judgment. So brainstorming ideas, helping
- 19:25in drafting and editing, Again, that's what I
- 19:28do most of the time, and it is a fabulous resource
- 19:32that helps me really improve my thinking and
- 19:37writing and expressing ideas. One of the real
- 19:41values of AI is playing a devil's advocate. It
- 19:45can be a coach. And one of the things is that
- 19:48it can be kind of awkward and embarrassing to
- 19:51go to one of your colleagues in a firm and ask
- 19:54for advice about some things. And one of the
- 19:58virtues of using AI is you don't have what's
- 20:01called judgment anxiety, or you don't need to.
- 20:04And you can ask them any time. You can ask them
- 20:06as much as you want. So getting this devil's
- 20:09advocate point of view and addressing the biases
- 20:12that we all have can be very helpful. It can
- 20:17explore alternatives and identify questions you
- 20:20hadn't considered. So here's some resources.
- 20:23that you might want to consider. And again, you
- 20:27should be getting this. And these are links that
- 20:29you can click on. Here's the article that I wrote.
- 20:33Here's the ABA opinion that I referred to. Here's
- 20:36an article that summarizes the proposed rules
- 20:40in California that are going to require you to
- 20:43verify every AI output. This article discusses
- 20:48a range of risks of using AI. and suggest ways
- 20:53to manage those risks. This is the database of
- 20:58the hallucination cases that I talked about.
- 21:01And then down below here, we have this tool that
- 21:04Ala mentioned, this Real Practice System Negotiation
- 21:08and Mediation Coach, which I will say, immodestly
- 21:13is really good. And it's free. And then there's
- 21:20this annotated bibliography of things that I
- 21:23have written. And so with that, I'm going to
- 21:26leave you with this last slide. And if you want
- 21:30to download the surprising value of hallucinations
- 21:35article, you can just use this QR code. And with
- 21:38that, I'm going to stop and see if we have any
- 21:40questions or comments. Susie, do we have anything?
- 21:45Yes, so it looks like I have one here from Douglas.
- 21:51Have you ever heard of a law firm terminating
- 21:54an attorney for incorporating the wrong citations
- 21:58in a legal brief? I have not. I can imagine that
- 22:03they might do so, and that would be understandable.
- 22:09I read one case where the judge where it was
- 22:13a federal appellate case, and the court said,
- 22:18you know, at this point, everybody should know
- 22:22about hallucinations. And so you can't really
- 22:26say, oh, I've never heard of this before. So
- 22:29everybody should be on notice. And I think that
- 22:33that would be an understandable reaction by law
- 22:38firms to possibly terminate. And of course, part
- 22:41of What this also gets to is that law firms have
- 22:47an obligation to supervise their attorneys and
- 22:50to train them. All right, looks like we have
- 22:57another question from Daniel. In your hypo, AI
- 23:01provides four excellent sites and one hallucination.
- 23:04Even though it provides some excellent sites,
- 23:07one hallucination can completely destroy the
- 23:09credibility. Does the cost benefit analysis weigh
- 23:12against using AI? Not in my view. I mean, not
- 23:17if you recognize that it's a hallucination. I
- 23:23mean, that's the whole point. You should be If
- 23:26you do this research and you identify something
- 23:31as a hallucination, then your job is not to use
- 23:36it in filing papers in court or citing it or
- 23:41relying on it in conversations with clients.
- 23:46In my mind, it doesn't destroy the credibility
- 23:50unless you use it without checking and determining
- 23:55whether it's false. I mean, in some ways, that's
- 23:58as if you were using a misleading citation without
- 24:03getting AI. I mean, that's a problem. So the
- 24:07short answer, Daniel, in my mind is the cost
- 24:10benefit doesn't weigh against using AI unless
- 24:13you find that most of the responses you get are
- 24:18hallucinations. And if you're getting a lot of
- 24:21junk, then I would say indeed the cost -benefit
- 24:24does weigh against using AI. Now part of that
- 24:27has to do with learning how to, one, choose the
- 24:31right tools, and two, use prompts that elicit
- 24:38the information that you're looking for. Now
- 24:41again, I don't do research on legal authorities
- 24:47very much. And I know that there are some tools
- 24:51that are specifically designed for that. And
- 24:53I'm guessing that there are a lot that are improving
- 24:58the quality of the responses. And so you're getting
- 25:01a smaller and smaller percentage of hallucinated
- 25:06answers. So a long answer to your question. The
- 25:09short answer, Daniel, is the cost benefit analysis
- 25:12doesn't weigh against using AI. unless you have
- 25:15a very high proportion of responses that are
- 25:18hallucinations. Looks like we have another one
- 25:23from Christine. Do you use AAI to check for mistakes?
- 25:28Have an AI scan your doc for things that sound
- 25:31or seem funky? Not your final doc, but the stuff
- 25:34you get from AI itself? I think that's a great
- 25:38idea. Again, you have to be careful. So, you
- 25:42know, your second AI may also create errors,
- 25:46but it also may pick up some. So let's just say
- 25:51you feed in a document and it identifies some
- 25:55hallucinations and you're confident they're right
- 25:59that those are hallucinations. You can cross
- 26:01those off the list. And then if there's some
- 26:03where they say, oh, they're not hallucinations,
- 26:06then you can check those manually. So I think
- 26:10that that's a useful thing. And I actually Do
- 26:13you know I primarily use chat GPT because that's
- 26:17where my RPS coach is based, but I also have.
- 26:23Claude account and so sometimes I will ask Claude
- 26:26for you know to check on things. So that's a
- 26:31terrific idea. I see that Wilson Winston. I can't.
- 26:39I'm not sure I'll pronounce your last name has
- 26:40raised his hand. I don't know if you can speak
- 26:44in this webinar format, but anyway, Susie back
- 26:46to you. All righty, it looks like I do see a
- 26:58few comments, but they don't. Let's see. Go ahead
- 27:04and check our. Excuse me. Sorry. All right. When
- 27:09have one here. When AI proposes an unfamiliar
- 27:13authority, what is your personal workflow for
- 27:16validating it before relying on it? Well, again,
- 27:20my personal workflow is that I don't do legal
- 27:24research these days, but what I would do would
- 27:26be the same thing that I would do without AI.
- 27:29So if you're using Westlaw or Alexis or whatever
- 27:33other source you use where you can rely on the
- 27:38output or there are other things so that you
- 27:41actually can find a copy of the valid source
- 27:46to be able to check. So that's what I would do.
- 27:50I mean, whatever you would do without AI, that's
- 27:53what I would do. So if you get a site for a case,
- 27:57look up that site. I hope that makes sense. All
- 28:05right. And it looks like we have um oh i see
- 28:08here yeah and some comments here's i related
- 28:13to just what was said nicole says i've run into
- 28:17lexus ai or wesla ai saying the case is not found
- 28:20and the case is real well um you know who can
- 28:24you believe um i mean in some ways you this is
- 28:32the nature of the world we're living in these
- 28:34days and I guess maybe you need to go to the
- 28:42court documents themselves to even double check.
- 28:47But that's interesting. Again, I've used Lexis
- 28:51and Westlaw myself. So Nicole says usually goes
- 28:56to the court records to find the cases. So I
- 28:59mean, unfortunately, this is the world we're
- 29:02living in. And I see Susan says she uses StrongSuit,
- 29:08which is a closed system. Also use Claude and
- 29:11ChatGPT between the three of them. They check
- 29:13each other in citations and they frequently add
- 29:16different reasoning. Claude is the most superior
- 29:19in reasoning. StrongSuit is not well known and
- 29:21is a secret weapon. So I mean, one of the things
- 29:25that Susan's comment reflects is that you will
- 29:28use your judgment over time and find some tools
- 29:31are better than others. Neil says he uses BlueJ
- 29:35for tax analysis and research. It provides citations
- 29:38and links to the source document. And actually,
- 29:42Neil's point is exactly right, which is that
- 29:45you should be asking your AI tool for links to
- 29:50the sources. And some of them will do it routinely.
- 29:52But if they don't, you should ask. And that's
- 29:56usually a good way of determining that there
- 30:00is a that there's a, whether the source exists
- 30:06or not. Nicole includes a link to the search
- 30:11she uses. John says, my article is interesting
- 30:14and downloaded, but I wanted to cite the article
- 30:16in a memorandum. Is it published anywhere that
- 30:19you can cite it? You bet the citation is, it's
- 30:23in, well, one, you can cite it, it's published
- 30:26on SSRN. And there's a citation at the bottom
- 30:30there, but it's also published originally in
- 30:34a magazine, The Alternatives to the High Costs
- 30:37of Litigation, which is published by the CPR
- 30:41Institute for Conflict Prevention and Resolution.
- 30:46So yes, you definitely can do it. It's a terrific
- 30:49source. I'm being facetious here, but I recommend.
- 30:55my writing, because I wrote it, and I think it's
- 30:58good. The term I work for uses Lexis, and I'm
- 31:01a 3L. She has me pull cases she can't find in
- 31:04Lexis, but Westlaw does not always find them
- 31:07unless I use traditional advanced research. Well,
- 31:11again, this just gets to the fact that you need
- 31:14to find out ways that you can really rely on
- 31:21the sources. And what this shows is, I mean,
- 31:25Lexis and Westlaw, you know, I mean, now they
- 31:29have AI built into it, but before, you know,
- 31:33they didn't have AI and you should be able to
- 31:38rely on it, but you may not. And Christine says,
- 31:44AI doesn't peg everything. Westlaw doesn't peg
- 31:48everything. So, I mean, again, what the whole
- 31:55point here is, is that you need to find sources
- 31:59that you can really rely on, which may mean going
- 32:02to the court records, as someone pointed out
- 32:05earlier. I think we may have responded to all
- 32:11the ones that are out there. I appreciate all
- 32:16the questions. Nicole has a new link to share
- 32:21with courtlistener .com. All right. It looks
- 32:28like we are at 32 minutes. Professor Landy, do
- 32:32you have any other closing comments that you
- 32:34would like to make? Only that I first want to
- 32:40thank you for inviting me. I hope you found that
- 32:44this has been helpful and good luck. It's a brave
- 32:50new world out there with AI and hopefully this
- 32:54has heightened your appreciation for the need
- 32:59to be careful and also to appreciate the potential
- 33:04values. Again, we have been talking about looking
- 33:07at things where there is a clear right answer.
- 33:11I have found, I use it for writing, as I said,
- 33:15and as a writing and editing tool, as a copy
- 33:18editing tool, you cannot find a human who will
- 33:22do a better job of copy editing and proofreading
- 33:25than AI. So part of it is figuring out what tasks
- 33:29you're doing and which AI tools can help and
- 33:33what you can rely on and what you can't. So you
- 33:35may just want to think about that graphic that
- 33:38I had at the beginning of you know, the client
- 33:40and the expert and the judge and everyone and
- 33:43you're going to use your judgment about what
- 33:45you can rely on and whatnot and what sorts of
- 33:48clues suggest you really need to be more or less
- 33:52vigilant. So with that, I just wish you well.
- 33:55Again, I thank you for inviting me and I hope
- 33:57you find this useful. This concludes today's
- 34:01presentation. Thanks again to Professor John
- 34:04Landy for an excellent presentation today, and
- 34:06thank you so much to our audience for joining
- 34:08us. Have a great afternoon, everyone.