Latest / CLEs You Actually Want to Hear / AI for Litigators: Leveraging AI
Transcript
- 0:03Hello, everyone. Welcome to today's program,
- 0:06AI for Litigators, Leveraging AI. I'm Alapoli
- 0:10Castro, Director of Education and Professional
- 0:12Development, and today I'm delighted to introduce
- 0:14our speaker, Sam Davidoff. Sam is the CEO and
- 0:18and founder of Align, a digital binder platform
- 0:21he built to give litigators a paperless, secure
- 0:24workflow from intake through trial. Before founding
- 0:27Align, he spent nearly two decades as a trial
- 0:30lawyer handling complex civil and criminal litigation
- 0:32in federal and state court. We're so excited
- 0:36to hear his insights today, so I'm going to turn
- 0:38it over to him. Great. Thank you very much. Yeah,
- 0:44so that is the brief introduction of me. And
- 0:47I was a partner for a long time, a trial lawyer
- 0:50at a law firm, Williams & Connelly, here in Washington,
- 0:53DC, which is where I'm based. But about actually
- 0:57almost seven years ago now, I started a software
- 1:00company, first doing the binders product that
- 1:04was mentioned. And then we actually now also
- 1:07make an AI -based legal research product called
- 1:11Align Research. I'm really excited to talk to
- 1:16folks about AI today, and I'm going to share
- 1:19some slides. But just to say that I pitched this
- 1:24presentation with the idea that it is for litigators
- 1:27who are anywhere from starting to use AI to having
- 1:34used AI and become more sophisticated as AI users.
- 1:42If you're on the latter end of that spectrum,
- 1:44I'm going to go over some basic concepts and
- 1:48bear with me on that. But also get into some
- 1:53more advanced things and one thing I would say
- 1:55is if questions come up during it I think you
- 1:57folks all have access to the chat and can put
- 1:59questions in the chat Please do and I'll keep
- 2:01an eye on the chat and if things that are questions
- 2:03that are topical to what I'm talking about I'll
- 2:05stop and address them and that's not quite having
- 2:08a conversation But it will make this more dynamic
- 2:10and I really appreciate it if folks do that.
- 2:12I love Trying to explain this stuff or address
- 2:15real questions that people have on the ground
- 2:18So feel free to do that as I'm going just put
- 2:20that in the chat. Okay I am gonna share my slides.
- 2:27And hopefully everyone can see that now. Okay,
- 2:35great. Okay, this is what we're gonna talk about.
- 2:38I'm gonna talk about some strategies for using
- 2:41AI as a litigator. And this is going to be. platform
- 2:46agnostic. In other words, the strategies I'm
- 2:48going to talk about are general to the use of
- 2:51AI, specifically large language models that are
- 2:55what really all the AI tools these days, or nearly
- 2:58all of them are using. And these kinds of strategies
- 3:01will work if you are using a legal specific tool
- 3:05like a Harvey or a Legora, or if you're using
- 3:09a, you know, generic commercial tool like ChatGPT
- 3:13or Claude or Gemini or any of those. I'm not
- 3:18gonna speak directly to particular products,
- 3:22except a couple of little places I might offer
- 3:24a couple of tips on things that work. I know
- 3:28Harvey pretty well, I know Legora less well,
- 3:30so I may offer a few tips on that as we go. Okay.
- 3:35So let's start with some basics and just to sort
- 3:39of level set on AI, what AI is, let's start by
- 3:44some basic terminology. So first of all, when
- 3:47we talk about AI, AI these days is used as a
- 3:50term for really what is sort of more technically
- 3:55sometimes called generative AI, or it is really
- 3:59machine learning models. that have been trained
- 4:03on huge amounts of text. And these are called
- 4:09large language models. And the models are sort
- 4:13of the kind of black box secret sauce that you
- 4:17access when you access the chat GPT or... Claude
- 4:22are also when you access one of these legal specific
- 4:26tools like Harvey or Legora. So the models have
- 4:31names like in ChatGPT, now they have names like
- 4:36Sol and Terra and Luna. In Claude, you might
- 4:39have heard names like Opus and Sonnet and Fable.
- 4:43Um, there are other ones, uh, and these are the
- 4:46names of the sort of trained set of, of data
- 4:50or technically, um, vector embeddings that when
- 4:54you give them a prompt, they generate text in
- 4:56response. And that can be as simple as you ask
- 4:59a question and it generates text answer. Like
- 5:01we're sort of familiar with from chachi BT going
- 5:04back to the very early days, or it can be more
- 5:07sophisticated that, you know, you give it. a
- 5:09document and it generates back the text of a
- 5:12new document. But in general, the general concept
- 5:15of the model is you put text into it and it gives
- 5:21text out of it. Maybe with a little qualification
- 5:23that on some of the more advanced ones, it doesn't
- 5:26have to be text, it can also be sound and pictures
- 5:28and video. And they can, some of them can even
- 5:31generate sound and pictures and video. But the
- 5:33idea is you put something in and the model gives
- 5:36you something back. And what it gives back depends
- 5:39on what it's been trained on. But as we've all
- 5:41seen, it's pretty impressive what they can give
- 5:43back. They give back things that look a lot like
- 5:45what a human might give back to you. So those
- 5:50are the models. I want to distinguish the model
- 5:53from what is often called the harness, which
- 5:56is the tools that go around the model. And these
- 6:02are the applications that you actually interact
- 6:04with, like a Claude desktop or a Claude cowork,
- 6:08or Harvey, in the case of Harvey, Align Research,
- 6:12My Product. It could be other things like Copilot
- 6:15in Word. And what these things are all doing
- 6:18is they are wrapping around the model. And they
- 6:22are sort of, in a certain sense, kind of giving
- 6:24the model a, you know, it's like putting the
- 6:28model in a battle suit or something like that.
- 6:30It will take in your question. It will feed the
- 6:33question to the model. But it will also give
- 6:36the model the ability to do things like reach
- 6:39it. Again, if it's enabled, reach out and access
- 6:42other materials. Search the internet. Maybe have
- 6:46the model call another model. Maybe you have
- 6:49the model reach out to a legal research database.
- 6:51Maybe you have the model interact with Microsoft
- 6:54Word and edit documents. These concepts, it's
- 7:02often useful to keep them distinct and understand
- 7:05that the model is the engine, the thing that
- 7:08is able to understand text and generate text,
- 7:11and the harness Is the thing that allows the
- 7:14model to sort of interact with the outside world,
- 7:18the outside world being, you know, getting information
- 7:20from you, getting documents from your system,
- 7:23searching the web and doing other even more sophisticated
- 7:26things like writing programs and running programs,
- 7:28things like that. And so both things are very
- 7:33important. Which model you use is very important.
- 7:36And you hear press releases over time in terms
- 7:40of Claude released a new model. They released
- 7:43Fable 5. They released Opus 5. GPT released a
- 7:47new model. But then also important are the tools
- 7:50that go around them. Harvey is a tool that can
- 7:54work with a number of different models, and you
- 7:56can select different ones when you're in it.
- 7:58And the same is true with Legora, with Code Council,
- 8:04with things like that. They can work with different
- 8:05models to maybe make things even a little more
- 8:08confusing, but also illustrate the point. You
- 8:10may have seen recently that some of these companies
- 8:14like Harvey and Thomson Reuters that have make
- 8:18these tools have actually now started releasing
- 8:20their own models, and that's a whole other subject.
- 8:23But again, illustrating the point that there
- 8:24are these two different things. There are the
- 8:26models, and then there are the harness or the
- 8:30tools that go around them. The prompt is a generic
- 8:34word for the text that you put into it, often
- 8:37the question that you ask or the things that...
- 8:40you are following up with the model. But the
- 8:45prompt can also include the things, documents
- 8:50that you put in or whatever. Context is a more
- 8:55general term that is used to talk about everything
- 9:00that the model has access to when it goes to
- 9:05generate a text. So these models in general,
- 9:08They don't really remember things. When you have
- 9:11a running conversation with the model, each time
- 9:14it gives you an answer, it's actually sort of
- 9:18taking the whole prior part of the conversation,
- 9:22your original question, its original answer,
- 9:24your next question, its next answer. It's taking
- 9:26all of that, looking at it all again, and then
- 9:29giving you the next answer. It might also be
- 9:33looking at more than that. It might be looking
- 9:34at the whole history of your conversation with
- 9:36it. and some files that it's found by searching
- 9:39the web using the harness above, and files that
- 9:42you've provided to it by uploading them to ChatGBT
- 9:47or by accessing them in a Harvey Volt. It's taking
- 9:50all of that and using that to generate the next
- 9:53answer. That whole bundle of material that it's
- 9:57looking at is referred to as the context. It
- 10:00is all the context that the model has when it
- 10:04goes to generate the next thing that it's going
- 10:06to say to you. It's useful to distinguish between
- 10:10prompt and context because one important thing
- 10:13that I'm going to talk about is while it can
- 10:16be important to how you phrase your prompt, how
- 10:21you ask your question to the model, what's actually
- 10:24much more important is the bigger picture. Not
- 10:27only how you phrase the question, but all the
- 10:29other material, all the context. that the model
- 10:31has available to it. So those are some basic
- 10:35AI terminology. I think that's enough AI terminology
- 10:38to certainly get you started and to converse
- 10:41on most things that lawyers using AI will need.
- 10:44So I'll stop there. But I may use those terms
- 10:47as we go through. OK. Let's first talk about
- 10:52things that are risks. uh, in AI usage. So as,
- 10:56uh, as lawyers, um, what are the dangers? What
- 10:59are the problems that we face in using AI as,
- 11:03uh, in terms of, you know, actual legal risk
- 11:06exposure that we may have, um, you know, in terms
- 11:09of sanctions in court or, uh, you know, anything
- 11:14from malpractice claims to just concerns or complaints
- 11:16from our clients, you know, and also to just
- 11:18risks about things that are annoying, things
- 11:20that make us inefficient, things that make us
- 11:21liable to make mistakes and things like that.
- 11:24The first one that, if you've used AI at all
- 11:27or read about AI, you have almost certainly heard
- 11:30about is hallucinations. Hallucinations is the
- 11:35idea that the model will say something to you
- 11:37that in essence isn't true, that is made up.
- 11:42Hallucinations are really kind of an inevitable
- 11:44part of how large language models work. Large
- 11:48language models work by they take in all of the
- 11:51context that you give them, and they then use
- 11:54their model to generate a response that is essentially
- 11:59what their model dictates the response should
- 12:01be based on all of the entire context. There
- 12:07is nothing in that process that has any concept
- 12:10of truth or reality. That's just not sort of
- 12:13a thing in that machine. The machine takes in
- 12:16text and it generates other texts. There's a
- 12:18lot of things that are done to try to make that
- 12:20reliable. But there is no sort of a concept of
- 12:25saying to it, don't lie is not a thing. There
- 12:29is always some risk that when it writes its response,
- 12:32what it writes isn't correct. And in legal, that
- 12:38risk is particularly pronounced in a few places.
- 12:41And one of the things I attach in the materials
- 12:43are some cases that talk about them. And there's,
- 12:46unfortunately these days, hundreds if not getting
- 12:49on thousands of cases that discuss problems with
- 12:53hallucination and briefs that people submit to
- 12:55court. But these can span the range from the
- 12:59obvious things and that have gotten the most
- 13:00press of an AI writing a brief or writing a memo
- 13:04or writing a complaint and citing a case that
- 13:07doesn't exist. But it's really important to realize
- 13:12that hallucinations can go a lot further than
- 13:14that. It can be and be things that are much more
- 13:17subtler and harder to detect. It can be an AI
- 13:20citing the case that does exist, but You know
- 13:24with a quote that doesn't exist right putting
- 13:26a quote in the in the parenthetical that doesn't
- 13:29exist or even more subtler a AI can cite a case
- 13:33that does exist and not have a quote but have
- 13:36a summary in the parenthetical that isn't really
- 13:38what the case stands for or even subtler still
- 13:43it could actually cite the case and Have an accurate
- 13:48quote from the case But in the discussion of
- 13:51the case that it writes in the paragraph, it
- 13:53can make an argument that kind of goes too far.
- 13:56And these are the kinds of things that need to
- 13:59be worried about. And it doesn't only apply to
- 14:00cases. The same thing is true if you ask an AI
- 14:02to summarize a document. If you ask it to find
- 14:05things for you across documents, what it writes
- 14:08back to you has some increment of untrustworthiness.
- 14:14And that is basically, at least in the form of
- 14:18sort of response, whenever an AI writes something
- 14:21from you, there is no way you can guarantee 100
- 14:23% it's correct. And so you need to take that
- 14:25into account as you're working with AI. So that's
- 14:27one risk, the risk of hallucinations. Second
- 14:31is model competence. And what I mean by model
- 14:35competence is just the notion that these models
- 14:38are trained. The way they are trained, in essence,
- 14:42is by having them suck in enormous amounts of
- 14:45data. At this point, the frontier models that
- 14:48are really what we're talking about, the models
- 14:50created by OpenAI, Anthrape, and Google, have
- 14:54read pretty much every piece of thing available
- 14:58that's been written by humans ever. um and um
- 15:02and learned in a certain sense learned a lot
- 15:05from that um but they um but you know they have
- 15:11a they have a cut off on their training date
- 15:12it's only up to a certain point there may be
- 15:14things that were excluded from their training
- 15:16there may be things that they were asked to focus
- 15:17on more so there's kind of a limit to um what
- 15:21they have and you can sort of see that Again,
- 15:23now models search the internet using their harness,
- 15:26so this happens less. But in the early days,
- 15:28you would see this. You could ask a model a question
- 15:30like, who's the president now? And it might not
- 15:32know, especially if you asked it around an election
- 15:34year, because that wasn't part of its training
- 15:36data. Again, this is important for legal, because
- 15:40it means that when you ask even a very sophisticated,
- 15:43the latest, greatest model released by your provider
- 15:46of choice, when you ask it a question, If it
- 15:49doesn't have anything more to go on, if it doesn't
- 15:51search the internet, if you're not giving it
- 15:54other documents, the answer it gives is going
- 15:57to be bound by its training data. In other words,
- 16:00at a certain point, as smart as it is, as smart
- 16:04and fluid as its answer seems to be, the reliability
- 16:08of what it gives you is going to be limited by
- 16:11what was it trained on and what did it choose
- 16:14to access in that training. There are also differences
- 16:17between models. That is something that people
- 16:19focus on. My own view is if we're talking about
- 16:22sort of the top level models that the programs
- 16:25from Claude or Anthropic or from Harvey or LaGaurd
- 16:28give you, those differences are unlikely to make
- 16:30a huge difference to you as a lawyer in your
- 16:33daily work. They're all going to be pretty good,
- 16:35but they're also all going to have these limitations
- 16:36that we're talking about. Another risk is the
- 16:41risk of psychofancy. And what this really is,
- 16:46and anyone who's interacted with these, you've
- 16:48seen this, right? When you talk to these models,
- 16:50they really want to be helpful. They want to
- 16:52be your friend. They want to encourage you. They
- 16:55want to say that your ideas are great. And this
- 17:00is actually a feature of how they are trained.
- 17:03So part of the model's training is that they
- 17:06suck in all of this, you know, enormous amounts
- 17:11of data, but then part is what happens after
- 17:17that is the creators of the models have people
- 17:20go in and sort of score how they answer questions
- 17:22and give them essentially reinforce answers that
- 17:26are desirable and the. suppress answers that
- 17:32are not. The result is the models develop what
- 17:34appears to us to be a personality. The personality
- 17:37of models in general is they want to be agreeable
- 17:39and helpful. This is a problem for lawyers because
- 17:45that's not always what we want when we are trying
- 17:50to do legal analysis. In the days before AI,
- 17:55I think of an example I would always think of
- 17:56when I would hire an expert witness. I didn't
- 17:59want the expert witness who said, yes, of course,
- 18:01your damages theory is right. Of course, your
- 18:04client is correct. Of course, we can show causation
- 18:09the way you've said it. Those experts made me
- 18:12nervous because I thought, are they just agreeing
- 18:15with me because I'm paying the bills? That's
- 18:17not what I wanted. I wanted the expert who said
- 18:19to me, you know, I read your damages theory and...
- 18:22It doesn't actually totally work. And you're
- 18:24going to have to change x, y, and z. That's often
- 18:27what you want in legal. Or if you ask an associate
- 18:30or a colleague to read a case and tell you, can
- 18:34we use this case in our brief? At certain times,
- 18:37you want that person to come back and say, no,
- 18:39we absolutely can't use this case in our brief.
- 18:40It doesn't stand for the proposition we said.
- 18:44AI is not great at that. It has this desire to
- 18:48please. And I've actually done some experiments,
- 18:51which if somebody wants to reach out to me afterwards,
- 18:54I'd be happy to share. I've made them public,
- 18:55where if you give an AI a brief and a bunch of
- 19:00cases and you ask it, can we use these cases
- 19:03to argue a point that we want to make in this
- 19:06brief, its tendency is going to be to tell you,
- 19:09yes, you can, and to try to find some way to
- 19:12make what you want to do work. And that's often
- 19:15dangerous in the law for reasons that we know.
- 19:17So again, this is another risk. And it's different
- 19:19from the risk of hallucination because it is
- 19:22the model being overly agreeable with you and
- 19:25overly trying to lead you in the path that you
- 19:28want, but that may not be right. And then the
- 19:32fourth risk that I want to talk about with AI
- 19:35is what I call context rot. I don't just call
- 19:40it that. This is a term that's used. And it refers
- 19:42to a kind of feature of AI that as their context
- 19:46grows, and let me pause and say, how does their
- 19:48context grow? Well, their context grows in the
- 19:51first place by just you continuing to chat with
- 19:53it, right? So you ask an AI a legal question,
- 19:57let's say. You say, here's my situation. Is there
- 20:01any law in California that would prevent my client
- 20:06from being able to do X, Y, and Z? And the AI
- 20:09kind of gives you, based on its model, an answer.
- 20:13And then you say, well, read this California
- 20:15statute and tell me if that changes your answer.
- 20:17And it does that, and it gives you an answer.
- 20:18And then you say, well, can you put that into
- 20:20a memo? And it produces a memo. And then you
- 20:23say, well, can you edit the start of the memo
- 20:26so that it addresses this more? And the memo
- 20:29doesn't really address that, and so it gives
- 20:30you another version. What's going on is you go
- 20:33through this iterative chat process with the
- 20:35model. Is all of that the context is just continuing
- 20:39to build up, right? So it's not like when a person
- 20:42chats where, you know, we sort of have an idea
- 20:46of where we are in the conversation and we're
- 20:48not thinking back to the first thing that was
- 20:49said every time. The model is literally literally
- 20:52at every stage in the conversation, effectively
- 20:55rereading the whole thing and then giving you
- 20:57the answer. They don't have any concept of state
- 21:00there. It's always every answer is just kind
- 21:03of reading the whole context and giving you back.
- 21:06And so as these chats grow in length, and as
- 21:11more things get added to them, not just your
- 21:12chat, but the documents that you put in, the
- 21:14things they pulled from the internet, there is
- 21:16a phenomena which is that they get worse. They
- 21:20start to forget the things in the beginning.
- 21:23They start to be less good at remembering instructions
- 21:26that you gave them. And this is a phenomena that's
- 21:28been familiar, I think, to anyone who's engaged
- 21:31in a long chat with one of these things where,
- 21:33you know, you ask it to write something, it writes
- 21:35a few paragraphs, you tell it to edit the last
- 21:38paragraph, it does, you tell it to make another
- 21:40change, you know, it does that, you then ask
- 21:42it to edit the first paragraph, and it undoes
- 21:45the edit from the last paragraph, or something
- 21:47like that, right? These kinds of frustrations,
- 21:49what you're experiencing is a form of context
- 21:52rot. And again, what's important to understand
- 21:55about this is there is a part of this that is
- 21:57just inherent to what happens with these long
- 22:01contexts. Again, like the hallucination risk.
- 22:04These are things that are getting better over
- 22:06time and are ameliorated as new models and new
- 22:09tools come out, but they're not eliminated. And
- 22:11so again, it's something to keep in mind that
- 22:13these lengthy chats, lengthy things have a cost
- 22:16to them. So these risks are, I think, general
- 22:22risk of using AI. But again, they all have particular
- 22:26application in legal. And you want to think about
- 22:29them as you are going to work with these tools.
- 22:32So what I want to do now is talk about what are
- 22:35some strategies that I have found to work both
- 22:38in my own work and in work that I do. I also
- 22:43work on a consulting basis with a number of law
- 22:47firms and can share in general things that I've
- 22:51seen and strategies that I've seen to be effective
- 22:53in trying to counter this risks and get the best
- 22:55results out of your AI tools, whatever they may
- 22:58be. And for each one, I'm going to give an example
- 23:01that I hope is reasonably real world in terms
- 23:06of what you might encounter in the day -to -day
- 23:09of litigation. So the strategies I generally
- 23:13put in the category, these five, and let me just
- 23:17go through them one -on -one rather than go through
- 23:19the roadmap. So first, don't chat, plan, and
- 23:23execute. And this is really meant to address
- 23:27the context rot problem. But it also, in general,
- 23:35I think it also helps with some of the things
- 23:37like hallucination as well. Let me illustrate
- 23:45it. So let's take an example where you've received
- 23:49from opposing counsel a set of requests for production.
- 23:52And as we're all familiar with, it's one of these
- 23:55requests for production that, as almost they
- 23:58all do these days, starts with a laundry list
- 24:01of definitions and then a laundry list of instructions.
- 24:05And then each RFP starts with some little preamble.
- 24:09And then there's the little kernel of the thing
- 24:10that you're actually being asked to produce.
- 24:12And then maybe there's more language and additional
- 24:15instructions in there and whatever. And the thing
- 24:17is 80 pages long. And what your client wants
- 24:21to know is like, OK, what are we actually being
- 24:23asked to produce? OK. you know, a common task
- 24:29to do would be to create some kind of summary
- 24:31chart that sort of boils this down to here are
- 24:34the key categories that we're being asked to
- 24:35produce documents in, under each category here
- 24:38are the actual requests and the specific things
- 24:40that we want, you know, here are requests that
- 24:41overlap, things like that. So you go in and you
- 24:45sort of approach you might think would work is
- 24:49you go into the AI and you say, okay, here are
- 24:52these RFPs I just got from opposing counsel.
- 24:56Can you give me a summary of them? And it goes
- 24:59and it gives you a summary. It says these are
- 25:00the categories they're asking. This is what RFP
- 25:02one is about. This is what RFP two is about.
- 25:04And you say, okay, that's not exactly what I
- 25:06want. What I'd really like you to do is rather
- 25:07than just summarize the boiling down what each
- 25:10RFP is, categorize them not by the number that
- 25:13the other side assigned, but organize them by
- 25:17topic. So give me the ones that relate to personnel.
- 25:20Give me the ones that relate to financial statements,
- 25:23et cetera, whatever. And it says, okay. here
- 25:26are the categories of documents, here are the
- 25:29RFPs under each one. And then you say to it,
- 25:33okay, now does this cover every RFP? And it says,
- 25:36well, let me look. Spinning wheel goes for a
- 25:40little bit, and then it says, okay, actually
- 25:42here is all of them. I had missed some, and so
- 25:45actually there's four categories, and now here's
- 25:49the newly organized thing. This is the kind of
- 25:53back and forth chat that you get into with AIs
- 25:55if you approach things this way. And the net
- 25:57result is you end up with something, you have
- 25:59a lot of frustration. You sort of feel like you're
- 26:00fighting with the AI. You get something back
- 26:03that maybe is kind of what you want. And you're
- 26:07not totally sure if it actually covered everything
- 26:09and you need to go back and check it. The needing
- 26:13go back and check it thing is gonna be there
- 26:14no matter what. I'm not gonna give you a, there's
- 26:17no secret sauce for avoiding that. But a strategy
- 26:20that I would recommend in a situation like that
- 26:24is not to engage in a chat with the AI, but instead
- 26:27to follow this plan and execute way. And so instead,
- 26:32how I would approach something like this is to
- 26:34say to the AI, First of all, think through in
- 26:37your own head what is it you actually want and
- 26:39give it clear instructions and say, and in other
- 26:42words, don't think about this as you're going
- 26:43to engage in a conversation with the AI. Think
- 26:46about it as you are giving a task to a junior
- 26:50lawyer, a paralegal, an assistant, and they are
- 26:54gonna go away for two days and do the task and
- 26:57come back to you. That's not what's gonna happen.
- 26:59It's gonna happen very quickly, but think about
- 27:01it in that way. You're trying to give them a
- 27:02set of instructions that they can then go and
- 27:05execute on their own. So write it out like that.
- 27:07Here's what I want. I want an organized summary
- 27:09of these RFPs by category rather than number.
- 27:11It must comprehensively cover all the RFPs. In
- 27:15each category, identify the numbers that, RFP
- 27:17numbers that fall into that category as well
- 27:19as the description of what they cover. And then
- 27:22here's the key point. Instead of just saying,
- 27:25now go do it, tell it, give me a plan of how
- 27:28you're gonna do this and what it will look like,
- 27:30and then come back to me for approval. And what
- 27:34it will do is it will then take all this, it
- 27:36will read the document, and it will write out
- 27:38for you a plan. Here's how I'm going to go about
- 27:40it. And here's the output I'm going to give you.
- 27:43You'll read that plan. And that may lead you
- 27:47to think of some improvements. I actually don't
- 27:52like the way it's approaching this. Or I forgot
- 27:53to tell it what to do with the instructions.
- 27:56Or I forgot to tell it what to do with. But you're
- 27:59going to be much better off revising that plan
- 28:02with it. and then telling it to go execute that
- 28:05plan, then sort of trying to iterate back and
- 28:07forth on the work product that you're having
- 28:09it create for you. I would say that iteration,
- 28:13too, can be a little dangerous for the same reason.
- 28:15And so if it comes back to you with a plan and
- 28:18you say that plan's pretty good, just, you know,
- 28:20I make sure that the headings are bold, that's
- 28:23probably fine. If you get into a back and forth
- 28:26with it over the plan, you're going to end up
- 28:28in the same problem. But the easy solution there
- 28:31is to get get in the back and forth with the
- 28:34plan, come up with the plan that you want, and
- 28:37then start a new conversation. Take the final
- 28:39plan that you've given it and open up a new chat
- 28:42and say, okay, here's my RFPs, here's what I
- 28:45want, here's the plan I'd like you to follow,
- 28:47and again, I would say, tell me what you're gonna
- 28:49do, let me confirm it, hopefully it will give
- 28:52you back the same plan that you put in, say go,
- 28:55and let it go. This strategy, again, you're still
- 28:58gonna wanna check what it does, you're still
- 29:00going to, It isn't perfect, but I think you will
- 29:05find that this strategy of instead of chatting
- 29:07with it to get to the work product you want,
- 29:10instead organizing your thoughts, giving it an
- 29:12instruction, having it come up with a plan, and
- 29:15then having it execute the plan leads to much
- 29:18more consistent and reliable results. I would
- 29:21also say that for some of these kinds of tasks,
- 29:24for example, a task like this RFP thing, if you
- 29:28end up with a plan that you like, and that you
- 29:31see worked well to get you summary of a particular
- 29:34RFP, that's a great candidate for reuse. You
- 29:36don't need to have that conversation to generate
- 29:38the plan with the AI. The next time you get another
- 29:41set of RFPs, you can reuse the one that you have.
- 29:47A couple of things to say about this. Some of
- 29:50the tools are already starting to encourage you
- 29:52to do this or building in features that help
- 29:56you do it. So they're helping you by sort of
- 29:59trying to plan first. And you're seeing this
- 30:02getting built into the tools part of it, the
- 30:04harness of a lot of these tools now. And that's
- 30:07great. I think that's a great thing. But if they're
- 30:11not in the particular tool that you're using,
- 30:12or it's not doing it the way you want, just do
- 30:15what I'm saying here. In other words, any model,
- 30:18even without any tooling, will be able to do
- 30:20this kind of workflow. Give me a plan. Let me
- 30:23have me approve the plan and then execute the
- 30:25plan. And that's much better in my experience
- 30:27and much more reliable than having the back and
- 30:30forth chatting. Okay, let's talk about another
- 30:34strategy, providing context. This is really meant
- 30:38to address a number of things. The hallucinations,
- 30:41the fact that AIs are limited by what they're
- 30:43trained on and as well as those two in particular.
- 30:51Let's take another example of a common workflow,
- 30:55summarizing a deposition. This is a task that
- 30:58we often have to do for various reasons, including
- 31:01updating our clients as soon as we get out of
- 31:03the deposition. A naive or simple way to do it
- 31:09would be to say, I just got out of the deposition,
- 31:12the court reporter sent me the rough transcript,
- 31:14and I give that rough transcript to my AI, and
- 31:17I say, summarize this deposition. The problem
- 31:23here is the AI does have a certain amount of
- 31:26context. It has the deposition, of course. If
- 31:28you ask it to summarize the deposition without
- 31:29giving it the deposition, it would do very badly.
- 31:34But it doesn't have anything else. And so what
- 31:36you're going to find is you're a little at the
- 31:38mercy of the AI might misunderstand some things
- 31:42about what the case is about. It might not know
- 31:44what some of the key terms mean. It might confuse
- 31:47the names of different witnesses. And as a result,
- 31:52how it chooses to summarize, what it chooses
- 31:54to cover is not going to be necessarily what
- 32:00you want. And you're going to find yourself,
- 32:02again, in this kind of battle with it, where
- 32:04you're going to say, hey, you didn't even mention
- 32:05the partner's background, where he said he'd
- 32:07been an expert for the other side in a similar
- 32:09case, or whatever, things that you thought were
- 32:12important. You can try to tackle this problem
- 32:16using the first strategy, which is give it good
- 32:20instructions, asking for a plan for how it's
- 32:22going to summarize, et cetera. And that may give
- 32:25you somewhat better results. But the problem
- 32:27here is not just a problem of clear instructions
- 32:30and chatting. It's a problem that the AI just
- 32:34doesn't know what you know about the case and
- 32:37what's important in this deposition. But you
- 32:39can give it to it. And so a much better approach
- 32:44would be to give it the deposition, but also
- 32:47something like this. Please review the attached,
- 32:49give it other things too. So for example, in
- 32:52this case, please review the attached copy of
- 32:54the operative complaint. Here's an attached memo
- 32:57on the background of the witness. Here's my notes
- 33:00on what I observed during the deposition. Now
- 33:03review the transcript and provide a summary of
- 33:05the deposition and make sure to provide citations
- 33:07to every page online. So what have I given it
- 33:11now? And I've picked these examples. You could
- 33:13give it other things as content, but I've picked
- 33:14these examples to illustrate a few things. I've
- 33:18said, give it the complaint. In other words,
- 33:20I'm trying to give it some material on what this
- 33:22case is about, what's at stake, to have it have
- 33:26some understanding of what is important and what
- 33:31are the important issues in this case. I'm also
- 33:34giving it some background on the witness so that
- 33:35it has to help it with what might it expect to
- 33:39see in here, what might it what things might
- 33:44be expected to come up in this deposition or
- 33:48might clarify things that the witness is assuming
- 33:51when they talk. Then the third thing is something
- 33:55that I think is very useful, which is sometimes
- 33:58you go through a deposition, if you're the one
- 34:00taking or defending the deposition or an observer,
- 34:03and you may take your own notes about what's
- 34:06important. drove me crazy when I was litigating
- 34:10in pre -AI days was the way I like to take notes
- 34:13was to take notes about what seemed important
- 34:18to me as it was going on. If I was taking the
- 34:20deposition, I like to jot down what were the
- 34:22things that I got, what were the admissions that
- 34:24I got, or what were the things that were unexpected,
- 34:27or what were the things that I wanted to make
- 34:28sure to come back to, things like that. Or if
- 34:31I was defending, similarly, I would take notes
- 34:34on ugh, I wish the witness hadn't said this,
- 34:37or oh wow, they got a good answer on this, or
- 34:39oh, it's interesting that the plaintiff seemed
- 34:40to be going after this theory or whatever. Take
- 34:43notes like that. I didn't really wanna take comprehensive
- 34:45summary notes. To me, that was distracting and
- 34:47a waste of time. With AI, you can sort of bring
- 34:51these two together in a great way. You can take
- 34:53your sort of what's important to me, what do
- 34:55I wanna make sure to relay to the client, what
- 34:57do I wanna make sure that my team focuses on
- 34:58in future depositions. give that to the AI and
- 35:02it makes a summary and have that inform its summary
- 35:05so that you don't just get generic deposition
- 35:07summary where, but you get a summary that highlights
- 35:11the things you'd want it to highlight and that
- 35:14you were seeing live when you were there. Then
- 35:16I'm giving it the instructions and again, also
- 35:21telling it to provide citations. This latter
- 35:24point, I'll say more about this a little bit,
- 35:25but is to also help me do what you're going to
- 35:28need to do no matter what, which is validate
- 35:30what it says. If you've just been in the deposition,
- 35:33fine. But if you are asked to summarize the deposition
- 35:38that you weren't there and you haven't read yet,
- 35:41this is not going to help you get away from reading
- 35:42it. You're still going to need to read it to
- 35:43make sure it's right. But this will help you
- 35:45double check what the AI is doing. So here, by
- 35:50providing context, Even though your question
- 35:53is still ultimately summarized in that position,
- 35:55that's what I want you to do. But by providing
- 35:58the AI with context, you are now going to get
- 36:01a summary that reflects an understanding of what's
- 36:03at stake in the case, an understanding of what's
- 36:07going on with this witness, and your lawyer's
- 36:10perspective on what should be shown in summary.
- 36:14That's going to be a much better summary. Again,
- 36:16this is sometimes called So in sort of the early
- 36:19days of AI, people would talk about prompt engineering,
- 36:22which was sort of getting your question exactly
- 36:25right to the AI so that it didn't hallucinate.
- 36:30And that is, in general, much less important,
- 36:33people think, and often misleading. So for example,
- 36:35if you say to the AI, please don't hallucinate,
- 36:38that's not going to make it not hallucinate.
- 36:40And by the way, it doesn't help if you put it
- 36:42in all caps and tell it that the world's going
- 36:44to blow up if it hallucinates. That's still not
- 36:46going to prevent it from hallucinating. However,
- 36:49giving the AI robust context, giving it a sense
- 36:53of what matters in the case, what's important,
- 36:56not going to eliminate things, but it will make
- 36:58the answers much, much, much better. You can
- 37:00tell this to your needs. Then the tip I put here,
- 37:04which is similar to the tip I gave on the execution,
- 37:06which is you don't have to do this from scratch
- 37:10every time. prompt like this with the complaint
- 37:16and the witness's background, et cetera, and
- 37:17you could reuse that. Another strategy you can
- 37:21use is actually to have AI first create some
- 37:25context documents. And this is a strategy that
- 37:27I've utilized, which is say to the AI, read the
- 37:31complaint, read interrogatory responses, and
- 37:36write up a summary of what this case is about,
- 37:39who the key players are, and whatever. And then
- 37:42for future depositions, you could actually just
- 37:44have it give that AI generated summary to the
- 37:47AI that's going to summarize and the deposition.
- 37:53Again, that's just a way of bookmarking or saving
- 37:56your place. Rather than asking the AI to reread
- 37:58everything each time it summarizes the deposition,
- 38:02you can generate a context module. that you're
- 38:06going to read once, but more importantly, you're
- 38:08going to feed it to the AI every time it goes
- 38:10to summarize a deposition. And you will find
- 38:13and you will sort of develop a sense for how
- 38:15creating those leads to better summaries and
- 38:17you'll improve them over time and end up with
- 38:19better work product. Next, I want to talk about
- 38:25show and don't tell. And to me, this strategy
- 38:29is I think really needs to be used more. And
- 38:35This is actually, in a sense, in my mind, the
- 38:39only way to really reliably cut down on it, even
- 38:43in some cases, eliminate hallucinations. And
- 38:46it sort of all stems from the notion that any
- 38:50time an AI is talking to you, it could make things
- 38:53up. So if I say to an AI, what is this case about?
- 38:59The AI, what it says back to me, this case is
- 39:03about, you know, two parties who in this kind
- 39:06of dispute and the court ruled that summary judgment
- 39:09was appropriate for whatever reason, that's potentially
- 39:12wrong what it's saying to you. Anytime it says
- 39:14something to you, it's potentially wrong. It
- 39:16might not be likely, but it's potentially wrong.
- 39:20On the other hand, if I say to an AI, give me
- 39:24the quote from this case that is, you know, the
- 39:28judge is ruling on X issue and tell me the line
- 39:32that that's on. And it gives me that. And I then
- 39:34go and look in the case, and in fact, that quote's
- 39:36there and that line's there. That's not a hallucination.
- 39:40You may see that what it said, it could still
- 39:43lie to you. You could go and look at that quote
- 39:44and the quote wasn't there. But by having it
- 39:47show you what is there in the case rather than
- 39:50summarize things for you, you're going to get
- 39:52more reliable results. And let me give you an
- 39:54example. By the way. This idea of show don't
- 40:02tell and having it point to things in a document
- 40:04is you see this in some legal research providers
- 40:07now where they put actually a number of them
- 40:09where they were put in citations into things
- 40:12in the citations. You can click on them. It will
- 40:14show you things in the actual document. I think
- 40:16that's great. That's actually what the premise
- 40:18of our legal research tool align research is
- 40:21as well, which is we don't summarize cases. We
- 40:24simply have our AI actually going in highlight
- 40:26the actual cases. So there's really no way to
- 40:28hallucinate. because it's actually just highlighting
- 40:31the text in the underlying cases. But the concept
- 40:34is more generally applicable. So let me give
- 40:36you a couple of examples or an example to walk
- 40:39you through this. I go to an AI and I say, what's
- 40:43the law on the Ninth Circuit on fraud by omission?
- 40:46Okay, there's a lot of problems with asking the
- 40:48question this way. First of all, how do I know
- 40:50the AI has the most current law on the Ninth
- 40:52Circuit, et cetera, right? So that's a bad one.
- 40:55Better is to say, look, here are the cases, either
- 40:58because I've gone and searched and found the
- 41:00cases and put them into the context, or because
- 41:04I have a connector in my harness to some tool
- 41:09that has access to a legal research database.
- 41:11But either way, I'm getting at the context, and
- 41:13now I'm saying summarize the law on fraud by
- 41:16omission. That's better, but because I've asked
- 41:19it to summarize, to tell me something about the
- 41:22case, I'm still at a risk for hallucination,
- 41:25right? Its summary of the current state of the
- 41:29law might be wrong. And I'm going to be at risk
- 41:33for that, even though I've reduced the odds of
- 41:36that. Here's what I would suggest is instead
- 41:40writing your query along the lines of, here's
- 41:43a set of cases of fraud. Review each one of these
- 41:46cases. and just give me the names of the cases
- 41:50and pin sites or citations to those that deal
- 41:52with fraud by omission, and then for each of
- 41:54those cases, the ones that were actually about
- 41:57fraud by omission, give me verbatim quotes with
- 42:00sites to the pages. So what this AI work product
- 42:03will now look like is not a summary, but a list
- 42:07of cases with each case a quote and with each
- 42:10case a page. Now I can now go into the cases,
- 42:14find the page, look at the quote, and see if
- 42:16it says what the AI said. Is that more laborious
- 42:20than getting a paragraph summary? Yeah. But it's
- 42:25less laborious than reading all the cases yourself
- 42:28and reading all the parts of the cases. If what
- 42:31you're trying to do is just zero in on the key
- 42:35parts of cases and the parts that deal with fraud
- 42:38by omission, it's less laborious than that. And
- 42:41you're building into it an easy ability to check
- 42:43by having the AI point you to quotes and citations
- 42:47rather than having it summarize for you. When
- 42:50you get a summary, even with citations to case
- 42:53names, your way of validating this to go through
- 42:58and reread the whole case. If instead the AI
- 43:00gives you a quote and a page number and you want
- 43:02to check that, you go look at the page number
- 43:03and you look at the quote. For certain kinds
- 43:06of tasks, again, I gave legal research as an
- 43:09example, but Finding documents that might be
- 43:12relevant to a particular point that you want
- 43:14to have a chronology on or things like that.
- 43:16Having the AI point you to places in the document
- 43:20rather than summarize things for you is going
- 43:24to both cut down on the hallucinations and maybe
- 43:28more importantly, make it easier for you to validate
- 43:31and also is going to discipline you to go in
- 43:34and actually do the validation. And in my mind,
- 43:39given that AIs have this problem of always having
- 43:43a hallucination risk, it's a discipline that
- 43:45you want. Sure, we would all love to not have
- 43:48to read everything and to get perfect summaries.
- 43:52But that isn't the state of the art. And we should
- 43:54not convince ourselves that it is. The state
- 43:57of the art is that these things can make mistakes.
- 44:05And so given that, I think asking the AI to give
- 44:08you an output like this that is validatable and
- 44:10that forces you to go in and validate. It doesn't
- 44:13give you kind of a shortcut, is good discipline
- 44:16and one that I definitely recommend for cases
- 44:21where they can be made to work. The last example
- 44:27I want to talk about in terms of strategies is
- 44:30categorize don't analyze. And this kind of wraps
- 44:36up a number of the hallucination and the psycho
- 44:38-fancy problems. And it boils down to the fact
- 44:44that, in my experience, even the top -of -the
- 44:48-line AI models, the latest model from Claude
- 44:50or OpenAI or whatever, in my view, they're still
- 44:56not as good as lawyers. We could have a debate
- 45:00about whether they'll eventually get there or
- 45:01not, but The state of the art in my view today
- 45:04is that they're not as good at lawyers, particularly
- 45:06when it comes to analyzing and making legal judgments
- 45:10about things that are important, holdings of
- 45:13cases, the importance, significance of things
- 45:14in documents. And they're okay. It's impressive
- 45:19that they can do it, but they aren't as good
- 45:21as good lawyers. And so if you are relying on
- 45:23them for judgment and analysis, I think you're
- 45:26walking into a trap and you're setting yourself
- 45:28up. I mean, in the worst case, If you don't check
- 45:31it, if you don't validate it, you're setting
- 45:33yourself up for some embarrassing, potentially
- 45:36sanctionable failures. But even if you are, as
- 45:40you should be, good and disciplined about checking
- 45:42and validating work, you're setting yourself
- 45:44up for wasting a lot of time. You say to the
- 45:46AI, what are these documents should I use for
- 45:48the deficit of Mr. Smith? It says, oh, here's
- 45:5110 great documents that you definitely want to
- 45:53use for Mr. Smith. You go and read them, and
- 45:55you're like, wait, these 10 documents don't help
- 45:57at all. You go back to the AI and you say, these
- 45:59ones didn't help. I need you to find documents
- 46:02on x, y, and z. And then it goes back and it
- 46:04finds those. And you read those. And it turns
- 46:06out they were about x and y, but not z. And so
- 46:08you go back and say, well, now, right. So great.
- 46:12You validated things. You didn't make the mistake.
- 46:14You didn't submit a brief or miss a document
- 46:16in the deposition. But you wasted a lot of time.
- 46:19And AI actually made you less efficient. And
- 46:22that's because I think that their analysis and
- 46:26judgment sort of powers are just not where we
- 46:31are as lawyers. By contrast, their ability to
- 46:35categorize things, their ability to answer a
- 46:38simple binary question, is this document about
- 46:42the financial statements or not? Does this case
- 46:46deal with fraud or not? It doesn't have to be
- 46:50binary. It can be, which of the following three
- 46:53topics is this document about? They are much
- 46:57better at that, much better at that. Are they
- 46:59infallible? No, you're still gonna have to validate,
- 47:01but they are much, much more likely to get things
- 47:03right. And so the way I would go about asking
- 47:07a deposition prep question like this is not give
- 47:10the AI... 500 documents and say, which of these
- 47:14should I use? And not even to do that with more
- 47:16context. Here's what's important for Mr. Smith,
- 47:18blah, blah, blah. That's not what I would do.
- 47:20Because you're giving the AI then an analysis
- 47:23task, and you're delegating to it the task of
- 47:26figuring out, making the judgments about what
- 47:30would be a good thing to ask a question about.
- 47:32I don't think that's where you want to be. What
- 47:34I do think is useful is asking the AI to help
- 47:37you. bucket the documents, categorize the documents
- 47:40into areas that may be useful for your deposition.
- 47:44So asking it which of these documents discuss
- 47:46Q2 sales at retail stores in the Midwest, if
- 47:50that's an issue that's relevant. If you ask it
- 47:52that question, it will do a very good job of
- 47:56going through and saying these 10 documents relate
- 47:59to Q2 sales and these 90 documents are about,
- 48:05you know, our marketing departments work in Q4.
- 48:09It will be very good at that categorization decision.
- 48:12And that will make you more efficient. Now, you're
- 48:14still going to have to read those Q2 documents
- 48:17and decide what you want to do in the deposition.
- 48:20But by the way, I think that's where you're earning
- 48:22your money. That's what you should be doing as
- 48:24a lawyer. But it will really help you focus on
- 48:27those documents. Now, is there some chance it
- 48:29missed some documents? Yes. And you're going
- 48:31to have to take that into account and you're
- 48:32going to have to have some double checking strategy
- 48:34or whatever for that. But my point here is if
- 48:37you can break AI problems down into categorization
- 48:41problems rather than analysis, judgment, summarization
- 48:44problems, you are going to get much more consistent
- 48:47and reliable results. A tip I will say is that
- 48:52a number of AI products Harvey's one that I know
- 48:55in particular, but others have this as well,
- 48:57have these kind of tabular review features, which
- 49:01let you put a bunch of documents in a table and
- 49:03then ask a question that will be asked to each
- 49:07document. It's a great way to use this kind of
- 49:10categorization strategy. Put in 1 ,000 documents
- 49:13that you think you might want to use for deposition
- 49:15and ask the question, not the ultimate question,
- 49:18how should I use in the deposition? What topic
- 49:21should I ask about on this? But instead, a categorization
- 49:23question. Which of these documents discuss earnings?
- 49:27Which of these documents discuss communications
- 49:31with Mr. Smith, et cetera, and have the AI do
- 49:35that categorization for you? At a minimum, that
- 49:37will give you a great starting point on which
- 49:39documents to dive into first. I see we have about
- 49:4510 minutes left. I do have one other section
- 49:47that I want to talk about on some other ways
- 49:51of using AI. But given that we have about 10
- 49:55minutes, let me stop now and see if there are
- 49:59any questions on any of what I've discussed so
- 50:02far or anything in general on this topic of using
- 50:09AI and litigation. I'd love to stop and take
- 50:12those questions now if they're there and use
- 50:15the time for that. If there aren't any questions,
- 50:18or if the questions don't eat up the time, I
- 50:20will talk for another 10 minutes about the last
- 50:22topic. But let me pause for a minute and see
- 50:27if there are any questions on this. And let me
- 50:37just make sure I'm looking at the right place.
- 50:39I think I am. Maybe this is the quiet crowd,
- 50:42which is OK. Okay. I'm sorry. I was not looking
- 50:49in the right place. I'm sorry. I missed a number
- 50:51of questions. Great. Let's do that. This will
- 50:53be great. I apologize that I missed these. I'm
- 50:56sorry. Got the wrong spot. First question was,
- 51:03how careful do you have to be about confidentiality?
- 51:08This is a great question. At a certain point,
- 51:14I'm going to give the lawyerly dodge answer.
- 51:16You're going to have to make your own legal ethics
- 51:18decisions. But I want to tell you some important
- 51:22things in my view. First of all, all of these,
- 51:28with a very narrow exception that you're almost
- 51:31certainly not using, all of these AI providers
- 51:35involve the concept of when you ask a question,
- 51:39your question in any other context you provide
- 51:41goes out to somebody else, right? It goes out
- 51:47to the AI provider, to whether it's Anthropic,
- 51:50whether it's, you know, JetJPT, whoever, it goes
- 51:54to them. By the way, this is true even if you're
- 51:57using a Harvey or a Legora or a co -counsel or
- 51:59whatever, at least up until very, very recently.
- 52:04They, even though you would send it to them,
- 52:07and then they would send it on to an anthropic
- 52:09or a Google or a clot. So that's happening. Sometimes
- 52:15you hear people talk about closed -loop AI. That's
- 52:18basically not a thing. You could, in fact, you
- 52:22could buy the hardware yourself and run certain
- 52:25models on that hardware. That would really be
- 52:28a closed -loop system, but nobody's doing that.
- 52:30other than very edge cases. So everything we're
- 52:35talking about AI, it's going out to some AI provider.
- 52:37That's just a fact of life. Is that a confidentiality
- 52:42problem? I think the answer is it's not any different
- 52:47than the same confidentiality concerns we have
- 52:50when we use, you know, Outlook, Microsoft Outlook
- 52:52for our email or Gmail for our email or iManage
- 52:56for our documents. The truth is everything we
- 52:58do electronically these days goes out to some
- 53:01cloud provider. And what are the confidentiality
- 53:03protections we rely on? They're contractual.
- 53:06And so you absolutely do need to, at a minimum,
- 53:11I mean, you need to read the terms of your contract
- 53:12with these providers. And I think what you're
- 53:14looking for there is, number one, that they're
- 53:16not going to train on your data. By the way,
- 53:18if you're paying for your AI, every provider
- 53:21that I'm aware of for paid plans for the AI does
- 53:24not train on the data. That's gonna be an easy
- 53:28one. You also wanna look at, you know, what do
- 53:31they say they're going to, how long are they
- 53:33gonna retain their data? And that, there is a
- 53:36variety among them. Some of the plans, you know,
- 53:39retain them for a lengthy period of time, some
- 53:41retain them for less, some, and this is some
- 53:43of the advantages of some of the legal providers
- 53:45like Harvey and LaGora is, they will offer you
- 53:48abilities to have, you know, what they call zero
- 53:50data retention, which is almost, deleted almost
- 53:52immediately. Those things may matter to you in
- 53:55terms of confidentiality, but these are not technical
- 53:57limitations. These are contractual limitations
- 53:59and they're important and you should look at
- 54:01them. But I don't think that in my view, I have
- 54:05not heard a convincing argument for why this
- 54:07is a new class of problem in terms of a lawyer
- 54:11sending information to an AI provider. To me,
- 54:17it doesn't seem like a different problem than
- 54:19a lawyer allowing their email to go through an
- 54:21email server. Different kettle of fish, and there
- 54:24have been some famous cases on this, including
- 54:26Judge Rakoff opinion on if a client asks a legal
- 54:29question to an AI, that's a whole separate kettle
- 54:32of fish. But a lawyer using AI as a tool with
- 54:35proper contractual restrictions, make your own
- 54:38legal judgment. But I don't see the issue, and
- 54:40I haven't heard a compelling argument why that's
- 54:42a concern. OK. Next question. Westlaw co -counsel
- 54:47hallucination risk? Absolutely. It's great that
- 54:51Westlaw co -counsel and other legal AI providers
- 54:53are grounding their AI, as they say, grounding
- 54:59their AI in their legal databases, their legal
- 55:02precedents, et cetera. That's great. But that
- 55:04is fundamentally not different than the examples
- 55:07we talked about before, where I give the AI the
- 55:10cases and then ask it a question. Does that cut
- 55:13down on the hallucination risk? Absolutely. Does
- 55:17it eliminate it? No, it doesn't eliminate it.
- 55:20And there's, in fact, some famous examples. And
- 55:22I actually, if I didn't include the opinion in
- 55:25the materials, you can find it. Or if you email
- 55:27me, I can send it to you. There's even examples
- 55:30of cases where people have got sanctioned from
- 55:36courts for hallucinations from some of these
- 55:37well -known providers. So it's not a panacea
- 55:42that you're using one of these legal research
- 55:44providers. And they will tell you that as well.
- 55:51Okay, if using Claude, can you set the case context
- 55:55once and then refer to it again and then bounce
- 55:58from case to case? This a little bit gets into
- 56:02some tool specific stuff and I guess I want to,
- 56:07the answer is in general yes, but it depends.
- 56:10It depends how you set it up. So a simplest answer
- 56:14that I hope will answer this question is Claude
- 56:18has a feature called Claude Cowork, which is
- 56:22available on their desktop app. That allows you
- 56:24to point the AI to a folder of materials, which
- 56:27could be cases in this. And you can do that.
- 56:31And then you can ask questions about the different
- 56:32cases. And it will go in and find what it needs
- 56:37and come back and give you the answers. It'll
- 56:39do that for any documents. That's an example
- 56:42of, I think, what you're asking. Google has a
- 56:44product that... They used to have a product,
- 56:47I think it's still out there, called Notebook
- 56:49LM, which had a similar feature. You can also
- 56:51do it by uploading things to a chat and having
- 56:54it within the context of the chat. It can look
- 56:58back at the cases. But I don't think that's what
- 57:02you're asking. I think you're asking about Central.
- 57:04And I would look at Claude Cowork. Chat CPT has
- 57:07a similar thing. Harvey has Vaults. I'm sure
- 57:10Lagora has a similar thing. That's kind of what
- 57:11they're meant to get at. OK, the next question,
- 57:19building on your point about prompt engineering,
- 57:21what's your view on loop engineering and its
- 57:24potential application and litigation? OK, this
- 57:28is a question. And I'm going to say congratulations
- 57:31to this questioner because they're clearly, I
- 57:33think, a sophisticated user of AI. So what this
- 57:42question is about is Sort of a lot of the advances
- 57:46in AI, especially in non -legal fields, but it's
- 57:49coming to legal too, is in what are sometimes
- 57:52referred to as agentic workflows, or as the questioner
- 57:57asked, loop loops. And the idea here is that
- 58:03AIs now are capable of, in effect, you could
- 58:06think about it as them having conversations with
- 58:08themselves. So you could ask an AI a question.
- 58:11It could say, OK, To answer this question, I
- 58:14need to read these documents. It'll go out and
- 58:16read the documents. And then it will say, based
- 58:19on reading these documents, I think I need to
- 58:21have another AI agent go out and look at some
- 58:23other things and give me a summary of that, and
- 58:26sort of chain these things together. And this
- 58:27is really the model and the harness working together.
- 58:30And at certain points, it will know to prompt
- 58:32the human and say, hey, human, I'm not sure where
- 58:35to go at this point. I figured out this, and
- 58:37I figured out this, but what do I need to do?
- 58:42And then the human will respond to the AI, will
- 58:45sort of go off in a different direction. This
- 58:47is 100 % important in the future. But what I
- 58:50would say, so to answer the, to be in my deposition
- 58:53mode and answer the question, this is important,
- 59:01yes, and I think is a way to streamline repetitive
- 59:04tasks. At this point in legal, I would say it's
- 59:07a fairly advanced topic. There are not. in my
- 59:10Sam Davidoff's view, great off -the -shelf, easy
- 59:14-to -use tools for doing this in legal right
- 59:16now. So if you're going to do it, you're going
- 59:20to need to level up on AI. I don't think you
- 59:23need to do it to get a ton of great value out
- 59:26of AI. But if you're asking this question, my
- 59:31best advice to you is, that's exciting. It's
- 59:35fun. you're looking at a couple of weekends of
- 59:39really digging in on how to use Claude Cowork,
- 59:43how to use Claude Code maybe, how to use other
- 59:47tools like that and kind of developing the skills
- 59:49for that. I'm not gonna go beyond that in this
- 59:51CLE. Maybe one day I'll do an advanced CLE, but
- 59:53it's advanced topic. It's definitely important.
- 59:55I think we will start to see more commoditized
- 59:57tools. I think we will see firms building centralized
- 1:00:00tools around this. It's not a pool. You can just
- 1:00:04dip your toe in though. That's definitely that
- 1:00:06is the cutting edge of things right now Okay
- 1:00:12Next question to ask about citations for specific
- 1:00:15cases. Does it need access to Westlar Alexis
- 1:00:18or are there enough cases on fine law? This is
- 1:00:20a controversial topic So, let me start sort of
- 1:00:25with simple uncontroversial things it needs access
- 1:00:28to something it definitely needs some way to
- 1:00:31get cases That's true. Second of all, Westlaw
- 1:00:37and Lexis have great databases. The best, right?
- 1:00:43They don't, by the way, even Westlaw and Lexis
- 1:00:44don't have everything, and Westlaw has things
- 1:00:46Lexis doesn't have, and Lexis has things, et
- 1:00:48cetera, but they're great databases. They are,
- 1:00:51in effect, the gold standard for legal research
- 1:00:56databases. Those two things I think are uncontroversial
- 1:01:01and true. OK, here's the issue. You can't just
- 1:01:08give your AI access to Westlaw and Alexis. That's
- 1:01:11not a thing. They don't make that access available
- 1:01:14to AI. They offer their own AI products. And
- 1:01:18those AI products have access to their databases.
- 1:01:21They have connectors to certain other AI products,
- 1:01:24but those are not actually access to their raw
- 1:01:26databases. They're access to their AI, and then
- 1:01:30their AI has access to their databases. So what
- 1:01:37you are, which is sort of implicit, I think,
- 1:01:40correctly in your question is, can you just say
- 1:01:42to your AI, you go search Westlaw? The answer
- 1:01:44is, there isn't any way to do that. you know,
- 1:01:49within Westlaw's terms of service, unless this
- 1:01:51is terms of service. So really, I think what
- 1:01:55the question boils down to is, is your then your
- 1:01:58only strategy to pay Westlaw or Lexis for their
- 1:02:02AI, or, you know, use one of their connectors
- 1:02:07to their AI that you get through a Harvey or
- 1:02:09a Legora? Or can you use your AI and just connect
- 1:02:13it to a public source like like find a court
- 1:02:20listener, which my view is that your AI connected
- 1:02:28to court listener is going to get you 90 to 95
- 1:02:35% of where you want to be, and in many cases,
- 1:02:37100%. I think it is great. I think it is a great
- 1:02:39resource. And I think the uplift you get from
- 1:02:42being able to connect your AI directly to it
- 1:02:45and just direct searches rather than having to
- 1:02:47go through some AI that a Westlaw or a Lexis
- 1:02:51is trying to sell you to me outweighs the fact
- 1:02:54in most cases that there may be a random unpublished
- 1:02:59district court opinion that you missed. I think
- 1:03:01court listening databases are great and getting
- 1:03:02better every day. The other opinion is no, if
- 1:03:07you're not boiling the ocean on Westlaw and Lexis,
- 1:03:11You're not doing your job. I think in most cases,
- 1:03:15in many cases, that's overkill, but not in every
- 1:03:18case. It depends on the stakes of your case,
- 1:03:19and it depends on how obscure your issue is.
- 1:03:30But unfortunately, there's no great answer here.
- 1:03:32In other words, my perfect world is that Westlaw
- 1:03:35and Lexis would make their databases available
- 1:03:37directly. Not for free, to pay for. And your
- 1:03:41AI could use it. It's not the state of the world
- 1:03:43now. So my own view is that the free law stuff
- 1:03:46is great. The court listener stuff is great.
- 1:03:49Full disclosure. That's what our product aligned
- 1:03:51research is based on. So I'm biased. Take all
- 1:03:53that bias into account. But that's where things
- 1:03:57stand right now. Next question. What's the greater
- 1:04:08danger to our profession, choosing between AI,
- 1:04:10hallucination, or implicit bias? If AI is using
- 1:04:14the internet, does it create the risk of AI becoming
- 1:04:16biased? So I guess what I would say is I think
- 1:04:21these are, I think in terms of our profession
- 1:04:24specifically, I think hallucination is the more
- 1:04:27immediate risk. The greater risk, which I think
- 1:04:31is really not specific to legal AI, the question
- 1:04:35of sort of Is there some things about AI training
- 1:04:40that would sort of make it not good in general
- 1:04:43and maybe not good for legal is a really interesting
- 1:04:47question. And I don't know the answer. And in
- 1:04:49other words, I understand the question. I understand
- 1:04:51the notion that perhaps by the AI reading millions
- 1:04:56of Reddit posts, it's now somehow not good at
- 1:05:00answering contract law questions. I don't have
- 1:05:02an intuition as to whether that's true or not.
- 1:05:04I think it's something to think about when you're
- 1:05:10looking at these. And I'll add a further wrinkle.
- 1:05:14As models created particularly in China are becoming
- 1:05:20more popular and actually are the basis for training
- 1:05:24the models that have been released by Harvey
- 1:05:27and Thomson Reuters, I think that question is
- 1:05:32also relevant. In other words, do we know what
- 1:05:35biases are baked into those models? But I wish
- 1:05:43I had a crystal ball and a great answer to that
- 1:05:45question. I think it's a good question. I think
- 1:05:47the immediate risk for us now is hallucinations.
- 1:05:51But as more and more stuff gets delegated to
- 1:05:52AI, I think the question of what are the subtle
- 1:05:55biases in AI is definitely important. OK, what
- 1:06:04do legal models like Harvey offer that Claude
- 1:06:07doesn't? Can any of them prepare a first draft
- 1:06:08of discovery responses? OK, let me take the last
- 1:06:12one. They could all prepare a first draft. The
- 1:06:14question is, how good is the first draft? And
- 1:06:18I think in both cases, they're not going to be
- 1:06:20good enough for you to just send it off. I think
- 1:06:23if you follow some of the strategies we talked
- 1:06:24about today, they may be good enough to save
- 1:06:26you time, to let you do it. I think they will
- 1:06:31both be good enough to do less time than you
- 1:06:38would have taken on your own. What does Harvey
- 1:06:42offer that Claude doesn't? My honest view on
- 1:06:47this is that mostly what Harvey offers that Claude
- 1:06:49doesn't is a kind of safety and comfort. for
- 1:06:59lawyers that they, and this is at the level of
- 1:07:04individual lawyers, it makes it harder for it
- 1:07:07to make some catastrophic, for example, it has
- 1:07:10this vault feature where you upload documents
- 1:07:12that are in the vault and it does things. Whereas
- 1:07:14with Claude, you can give Claude access to your
- 1:07:16file system and it can run wild. You shouldn't,
- 1:07:19but you can. So Harvey sort of put some safeguards
- 1:07:22around these things that I think are helpful.
- 1:07:24It gives auditability and enterprise level controls
- 1:07:28for law firms that have not yet really come to
- 1:07:30Claude or these other frontier providers. So
- 1:07:35I do think it has some benefits, particularly
- 1:07:37when you're at an enterprise scale, big law firms
- 1:07:39needing to manage lots of players. In terms of
- 1:07:42the performance of the models, the legal reasoning,
- 1:07:46My own view is they're either the same or I think
- 1:07:50that the Clauds and Chachiputis have the edge.
- 1:07:54So, in my view, these products are offering a,
- 1:07:58they're offering a service to you in terms of
- 1:08:00making AI easy and safe to deploy to your lawyers.
- 1:08:03I don't think the technical performance is better.
- 1:08:06And in fact, I think in many cases, at the very
- 1:08:09edge of technical performance, I think the frontier
- 1:08:11models win every time. Let me just ask a question
- 1:08:16to moderate, because I see there's like a stack
- 1:08:17of questions here. I can talk about this all
- 1:08:19day, but I think we probably are gonna hit a
- 1:08:21hard limit at some point. So. Continue for as
- 1:08:27long as you'd like. We can adjust the credit.
- 1:08:31Okay. Good. All right. Anybody drop off if you
- 1:08:34get bored. I find this stuff fascinating. What
- 1:08:42affordable products protect against a breach
- 1:08:44of confidentiality? Cloud Pro versus the free
- 1:08:47version. Are they discoverable? Here's what...
- 1:08:57Okay. And then can AI use be detected? Let me
- 1:09:00take these kind of in a couple. In general, I
- 1:09:09would say that... Cyber security is much more,
- 1:09:13you know, it's always been important, and it's
- 1:09:14even more important in the age of AI. I think
- 1:09:17for two reasons. I think there are new risks.
- 1:09:20I attached a case in here about prompt injection,
- 1:09:23which is where in this case, it was a litigant
- 1:09:27actually tried to add instructions secretly to
- 1:09:29a brief to have the court's AI make a wrong decision.
- 1:09:33Didn't work, but that was the idea. But that
- 1:09:35class of attack is a real class of attack. And
- 1:09:38the danger for lawyers are that AI could go out
- 1:09:42and read something on the internet thinking it
- 1:09:43was finding a case and instead read instructions
- 1:09:46to do something malicious. That's a real risk.
- 1:09:50It's a new risk in the world of AI. Old risks
- 1:09:54too, just bad passwords and things like that,
- 1:09:57are sort of magnified by AI. Because as you may
- 1:09:59have read in the news, attackers have access
- 1:10:02to AI and it scales up their ability to run attacks.
- 1:10:05So cybersecurity is important beyond the scope
- 1:10:09of the CLA, but it's definitely important there.
- 1:10:13In terms of the specifics of your contract agreements,
- 1:10:16pro versus free, I sort of addressed this before.
- 1:10:18Yeah, you definitely want a tier that doesn't
- 1:10:20train on your data, and typically the free ones
- 1:10:23do. So I don't think that those are, I think
- 1:10:26there are confidentiality problems around that.
- 1:10:29I don't know that a court has ruled on that.
- 1:10:31I don't think a court has, but I would not be
- 1:10:34comfortable using that for legal work. On the
- 1:10:37flip side, I think the paid versions that don't
- 1:10:38train and with appropriate restrictions on data
- 1:10:40retention, I do think are usable and I know plenty
- 1:10:44of lawyers and big law firms that are using them
- 1:10:47under those contractual terms. Discoverability,
- 1:10:56I don't think discoverability is really a new
- 1:10:57issue. you know, the discoverability of your
- 1:11:00chats with AI are similar to your discoverability
- 1:11:04of, you know, documents that you put into Dropbox,
- 1:11:07for example. If you're a lawyer and you use Dropbox
- 1:11:10for your business and you have, you know, an
- 1:11:12enterprise plan and confidence elements, it's
- 1:11:14probably fine. If you're a client and you stick
- 1:11:18your incriminating documents into Dropbox, they're
- 1:11:20still just as discoverable as they were in any
- 1:11:22other time. So in general, I don't think there
- 1:11:26are really many new discoverable things. But
- 1:11:31certainly, people are going to start requesting
- 1:11:36AI chat logs in discovery. And some of them may
- 1:11:42be privileged, and some of them may not be, or
- 1:11:44work product, more likely. But certainly, your
- 1:11:48clients who are having chats with AI about all
- 1:11:51kinds of things, and they're not lawyers, and
- 1:11:52these aren't legal. That's going to be discoverable,
- 1:11:55and people are going to ask for it, and they're
- 1:11:56already asking for it. The question of can AI
- 1:12:03detect AI, I'm not a big expert on this. My sense
- 1:12:07is that there are tools that can do it. They're
- 1:12:10not 100 % reliable. And there are times when
- 1:12:16those may be worth using, but they're not 100
- 1:12:18% reliable. in both ways. They can miss things
- 1:12:25and they can, you know, detect things as AI written
- 1:12:27that aren't. So, but again, I really don't, I
- 1:12:31don't have a lot of familiarity with those tools.
- 1:12:35Do you find Claude Trump's JET CPT? I don't think
- 1:12:38there's a binary answer to that. I think, first
- 1:12:40of all, you know, they're both releasing new
- 1:12:41models and new, and updates to their harnesses
- 1:12:43all the time. I think it goes back and forth.
- 1:12:47I, you know, I think Claude sort of, I think
- 1:12:51it bounces back and forth. I will tell you, these
- 1:12:53days, I use Claude, Chetchi BT, and Gemini for
- 1:12:58legal. And a lot of the work I do these days
- 1:13:01is non -legal. And frankly, even within particular
- 1:13:06providers, different models give you different
- 1:13:08results. So I don't think one is better than
- 1:13:10the other. I'm blunt about most things. If I
- 1:13:13thought one was, I would tell you. I also think
- 1:13:19that for like, you know, 95 % of what lawyers
- 1:13:23are doing, it doesn't matter that much. There's
- 1:13:26a difference between, you know, the top tier,
- 1:13:29you know, Claude Opus and Claude Sonnet. Opus
- 1:13:31is better than Sonnet, but the difference between
- 1:13:34Opus and Tachibutti, I think for the vast majority
- 1:13:37of what lawyers are doing is not gonna matter.
- 1:13:40And, you know, using a, you know, a frontier
- 1:13:42model is good and they'll give you good, not
- 1:13:44perfect results. The differences among them,
- 1:13:48I don't think matter that much for lawyers. Haha,
- 1:13:53this one's a great set. Is there a product other
- 1:13:55than from Westlar Lexis that can access legal
- 1:13:57research? Absolutely, there is. It's called Align
- 1:13:59Research. You should go use it now. It is Align
- 1:14:03.Lawyer forward slash research. But we're not
- 1:14:06the only one. uh there are a number of them um
- 1:14:12many of which are using as we are the court listener
- 1:14:14database which for us case law is phenomenal
- 1:14:16uh others have built their own databases um and
- 1:14:20um uh and uh you know i have been at some pains
- 1:14:24to do them there are companies like um describe
- 1:14:27and uh i should know these are my competitors
- 1:14:30um Anyway, they're doing interesting things,
- 1:14:35and they're good things. And frankly, so yes,
- 1:14:37there are definitely other options. And I think
- 1:14:40it makes a lot of sense to explore those. Google
- 1:14:46launched Gemini Enterprise for legal. Anyway,
- 1:14:51yeah. I think you just pointed this out. That
- 1:14:54was a very recent announcement this week. And
- 1:14:57I haven't. used it or tried it or anything. I'm
- 1:15:00eager to see it. I think it's great that they're
- 1:15:03getting involved in that space. I've never used
- 1:15:09Westlaw co -counsel. The next question is on
- 1:15:10Westlaw's co -counsel. The questioner doesn't
- 1:15:13find it very useful. I haven't used it, so I
- 1:15:20don't have an opinion on that. I definitely don't
- 1:15:25like tools that sort of constrain you to, you
- 1:15:31know, just sort of this chat format. Like you
- 1:15:33ask it a question and it gives you sort of a
- 1:15:35memo answer. I like one of the things I like
- 1:15:37about a Claude Cowork or, you know, other tools
- 1:15:41is for general purpose AI tools anyway. I like
- 1:15:45my general AI purpose, general purpose AI tool
- 1:15:49to be general purpose. So if I want it to. you
- 1:15:52know, have a chat with me or generate a long
- 1:15:54written summary, great. But I also like to be
- 1:15:57able to ask them to do other things. And you
- 1:15:58certainly have that flexibility with Claude.
- 1:16:02I don't want to speak to the other providers.
- 1:16:04But in general, I've sometimes been frustrated.
- 1:16:06I'm not talking about co -counsel particularly.
- 1:16:08But in general, I have sometimes been frustrated
- 1:16:10with legal tools that purport to be general purpose
- 1:16:13tools. But in fact, I'm kind of limited in kind
- 1:16:17of what I can do with them. I'm confused that
- 1:16:29it was mentioned that co -counsel sells data
- 1:16:31to Anthropic or Chachi Petit. So just to be clear
- 1:16:34here, they don't sell it to them. That's not
- 1:16:36what I meant to say. I just mean, at least up
- 1:16:39until this week when Thompson announced their
- 1:16:43own model, I don't know how much they're using
- 1:16:45it. But in general, all of these models are using
- 1:16:49Frontier models. So you send something to Harvey
- 1:16:51or co -counsel and they then send it to you know,
- 1:16:59a chat GBT or an anthropic, they don't sell it
- 1:17:01to them. They have contractual restrictions that
- 1:17:04say that they then in fact, the opposite that
- 1:17:06anthropic or you can't use it for anything. They
- 1:17:09have to lead it right away, etc. But, you know,
- 1:17:11they are a provider to them, just like, you know,
- 1:17:14they run their, you know, software on Azure or
- 1:17:17AWS or whatever. So they're a provider to them.
- 1:17:20They're one of their their service providers.
- 1:17:24again, at least until very recently when some
- 1:17:26of them have started fine -tuning their own models.
- 1:17:29But they're not selling their data to them. In
- 1:17:36other words, they're still complying with what
- 1:17:37they've promised to you that they would do, but
- 1:17:44they're using third -party providers, including
- 1:17:47for their AI models. I don't wanna really get
- 1:17:53into selling my product, but yes, there is a,
- 1:17:55I've probably crossed that line, but there is
- 1:17:56a free trial. Feel free to try it. I hope we
- 1:17:58are using CourtListener. My email address, you
- 1:18:02can reach me a couple of places. I think I put
- 1:18:06this on the slides, which are, but anyway, it's
- 1:18:09sam at either mfbtech .com or align .lawyer.
- 1:18:15And, And you can also find me on LinkedIn. Please
- 1:18:19do. Please follow me. I write about all these
- 1:18:21topics ad nauseam. So if after an hour and a
- 1:18:25half here, you're not tired of me, go on LinkedIn,
- 1:18:28and you can read my thoughts on that, many of
- 1:18:31which will sound familiar now that you've heard
- 1:18:33me talk. Again, what are the free legal databases?
- 1:18:38Court listener, I think, is great. I think those
- 1:18:40guys are doing great work. I think their project,
- 1:18:41it's not just case law research. They have access
- 1:18:44to pacer materials and other things. For US law,
- 1:18:48it's really phenomenal. Not really a question,
- 1:18:52but very cool that you have a Vintage Mac on
- 1:18:54your desk behind you. I think it's cool, too.
- 1:18:56And the pathology goes even deeper. I have many,
- 1:18:58many, many Vintage Macs. But the less said about
- 1:19:02that, the better in terms of revealing too much
- 1:19:05about my own mental health. And thank you all
- 1:19:08very much for having been a great audience and
- 1:19:10for all the great questions. I love that. This
- 1:19:13is great. The one part I did not get to in my
- 1:19:16slides, but you'll have the slides, is just I
- 1:19:18had a section that followed that had just a couple
- 1:19:23of really the point I wanted to make is, yes,
- 1:19:27using AI to sort of help you do big. picture
- 1:19:29legal things is great. But don't forget about
- 1:19:31the simple things. And I gave some examples here
- 1:19:33that I think illustrate the point. I think that
- 1:19:36using AI for just automating or avoiding simple
- 1:19:44annoying things in your legal practice is actually
- 1:19:47a lot of benefit, too. So just don't forget about
- 1:19:49those things was really my last tip. And with
- 1:19:52that, I will wrap it up. Thank you all very much.