Latest / CLEs You Actually Want to Hear / Creating "Best Practices" Guidelines in the Brave New World of AI
Transcript
- 0:03Hello, everybody. Thanks so much for being here.
- 0:05I'm Mark Hoag, chair of the Beverly Hills Bar
- 0:08Association's AI in the Law section, along with
- 0:11my co -chair Amanda Harris. And I'm the founder
- 0:13of Vaxiomic LLC, an AI -first web and iOS app
- 0:17development company. And for my day job, of course,
- 0:20I also run my own law practice, hoaglaw .ai,
- 0:25where I serve as fractional GC to AI -first SaaS
- 0:28companies and tech startups. I'm delighted to
- 0:31introduce today's speaker, Glenn T. Melchinger,
- 0:34a commercial litigator at Denton's US LLP. Glenn
- 0:39has litigated issues related to technology and
- 0:41the law for decades, and for over a decade he
- 0:44has taught on technology issues for lawyers,
- 0:47e -discovery and AI at the HSBA, ThinkTech Hawaii,
- 0:52the Western States Bar Conference, the 2024 Civil
- 0:55Forum, IPBA, and other venues. Glenn has represented
- 1:00clients for over 25 years in complex commercial
- 1:03and civil disputes, BEC cases, TCPA, class actions
- 1:09regarding data security incidents, products liability
- 1:12matters, and has also advised clients on cybersecurity,
- 1:15incident response, and other issues involving
- 1:18the intersection of the law and technology. He
- 1:21also reads, writes, and speaks Japanese and helps
- 1:25Japan clients with a host of US legal issues,
- 1:28with a broad bench of support over at Dentons,
- 1:31a global law firm. So thank you, Glenn, very
- 1:35much for being here. Let's get started. Thank
- 1:39you kindly, Mark. And without further ado, I'm
- 1:42going to go ahead and share my screen here and
- 1:47get the slide presentation going. And apologies
- 1:54as I navigate several different Screens here.
- 2:01But to start out, apologies. I think my share
- 2:07screen thing timed out. There we go. So we're
- 2:12starting to talk today about the title on your
- 2:16session is best practices. And basically just
- 2:20we're talking about AI use in the law firm. And
- 2:23that's. There's a lot of slides and a lot of
- 2:27materials that we'll go through. And my first
- 2:30slide should indicate to you kind of the issue
- 2:36that presents everybody when we start reading
- 2:42about AI in the news these days. It is like drinking
- 2:46from a fire hose. I'll apologize in advance to
- 2:49the extent that this slide presentation has a
- 2:52whole lot of information in it. And we will try
- 2:55and walk through and get to really a more substantive
- 2:58discussion at the end of what law firms need
- 3:02to be aware of in terms of risks. What do you
- 3:05need to do? What are the internal conversations
- 3:07that you need to have to handle this new, not
- 3:11so new, but relatively new tool, AI? And what
- 3:16are we even talking about with AI? The first
- 3:19part of the... discussion here is going to be
- 3:24just kind of a level set. When do I talk about
- 3:27AI in a law firm in a legal setting? In particular,
- 3:32what are we talking about? And then what in terms
- 3:37of how do you use it when you're court facing
- 3:40or client facing or there's some regulator facing
- 3:45work product that you have? How do you ensure
- 3:48that The things that we've all heard and read
- 3:50about from the Mara Via Bianca case onward, false
- 3:55information embedded in a document, fake case
- 3:58sites, et cetera. How do you make sure that that
- 4:00stuff does not happen to you? And how do you
- 4:03use AI even more than that, just avoiding the
- 4:06risk? How do you use it effectively? And how
- 4:09can you have a better result from your client?
- 4:15We'll talk a little bit through those issues,
- 4:18which are really, this is an ethics presentation.
- 4:21We're talking about a lot of the rule 1 .1 competency
- 4:28type issues up front by working through these
- 4:31definitions of what is AI. Where can it go wrong?
- 4:35A lot of people, I think, are familiar with how
- 4:38it can generate fake case sites. There are a
- 4:40lot of more subtle things that it can do that
- 4:43we will also work through. And then we'll talk
- 4:46a little bit about what's different about AI
- 4:48in terms of change management. How do you implement
- 4:52it in an organization? What are some of the issues
- 4:54there? And then at the very end, as part of this,
- 4:58I want to have a little bit of a discussion and
- 5:00work through some hypotheticals of how AI can
- 5:04cause problems. And what do you got to do to
- 5:08try and avoid those problems? So we didn't do
- 5:12a formal poll in this case. Just think about
- 5:15some of these issues for a little bit. Have you
- 5:19used Gen. AI at all for your work ever? And as
- 5:24we talk through these briefly, you should get
- 5:27a good sense of where you are in the spectrum.
- 5:32There's now, if you read the news, there's a
- 5:3617 -year partner at Quinn Emmanuel who just went
- 5:39out and formed an AI native law firm. And That's
- 5:43probably a kind of disruptive development that
- 5:46would not have happened a couple of years ago
- 5:48or even five years ago, for example. What is
- 5:53it that allows somebody to go and do that kind
- 5:56of a thing? And that kind of person is already
- 5:59going to have used GenAI for work. They're going
- 6:03to have a proprietary tool of some kind within
- 6:06their loan law firm that's secured and basically
- 6:12architected so that you have handling the 1 .6
- 6:16confidentiality issues that we'll address a little
- 6:18bit too, and handling client data in a secure
- 6:22fashion. But think about for the discussion at
- 6:26the very end of all of this, what is the tool
- 6:31that you have and how are you using it? And how
- 6:34much do you know about how everybody in your
- 6:36firm is using it? And those are the types of
- 6:39things that want to start thinking about today.
- 6:42And then hopefully by the end, you'll have some
- 6:44ways to approach these things within your own
- 6:46firm. All right. As the drinking from a fire
- 6:51hose slide indicated, there's a whole lot of
- 6:53information here. A lot of these things can become
- 6:55sub -topics even on their own. I did a presentation
- 7:01about 11 days ago on AI evidence. And that alone
- 7:06is a couple hours. So we're going to cover a
- 7:08lot of issues quickly at a high level. And then
- 7:12the slides have a lot of detail in them. And
- 7:14you'll have a copy of them. I think you have
- 7:16access to them already. So feel free to go back
- 7:19in there. They have live links to a lot more
- 7:21materials. So really, this is just designed to
- 7:23try and give you a way to think about AI and
- 7:25understand it and look at it. So just briefly
- 7:30on AI evolution, obviously, I think everybody
- 7:32knows that ChatGPT 3 .5, when it came out, was
- 7:35probably the quickest to market with 100 million
- 7:37users. Uber took 70 months. OpenAI took for ChatGPT
- 7:43about two months. So this is something that's
- 7:46really grown virally. There's a lot of promise
- 7:51with it. There's... age -old math problems that
- 7:56have stood for decades that nobody had been able
- 7:58to figure out and now some of those are getting,
- 8:01you know, proofs are being submitted and checked
- 8:04and AI has been key to that. AI is in other contexts
- 8:10trying to help human beings with folding proteins
- 8:13and developing drugs and things like that because
- 8:17AI is a master at pattern recognition. I'd say
- 8:21To anybody who actually has watched the Neil
- 8:24Katyal talk, the TED talk that he did recently
- 8:27on winning the tariff case at the US Supreme
- 8:30Court, it's 18 minutes that's probably worth
- 8:35your time just to figure out how to use AI and
- 8:39what it's good for. It's good as a pattern recognition
- 8:43machine, and it's also good as just testing your
- 8:45arguments if you're in court or testing your
- 8:48your contractual provisions if you're doing that.
- 8:53There's a lot of problems also. There's peril
- 8:55that comes along with a promise. One of those
- 8:58things that's tracking all that, there's an incident
- 9:00database link here. I leave it to everybody to
- 9:03go and read if you're interested the things that
- 9:06has caused, how it's caused problems. But the
- 9:11reason chat GPT became so popular is because
- 9:13it's just it seems pretty Coherent and logical
- 9:17and it puts a lot of information together in
- 9:20an easy to reach conversational format so There's
- 9:25a lot of legal specific tools. I mentioned Harvey
- 9:28in here I think they just went through a route
- 9:30of one point more funding and got about fifteen
- 9:33point five billion dollars worth of funding that
- 9:36and other tools are really dramatically evolved
- 9:39from when I started piloting some of them a couple
- 9:43years ago. And so it should be noted just as
- 9:46another caveat that these things are a moving
- 9:48target. And some of the stuff that we talk about
- 9:51today may or may not still be in effect in another
- 9:54week. Generally speaking, I've seen some estimates
- 9:58that AI's IQ is about 157. If you ask it a complex
- 10:04scientific question, Lord knows whether all of
- 10:07us are going to have the subject matter expertise
- 10:08to even understand, to say nothing of just translating
- 10:12into another language, et cetera. It can do a
- 10:15whole lot of things, and not only just to you
- 10:18and me in a chat, but to billions of people at
- 10:21scale across the globe. So that's the level of
- 10:27intelligence that we're talking about here. It's
- 10:30also now being used for e -discovery purposes.
- 10:34I won't go into the details of this case. There's
- 10:35a reference in the slides. But just to say that
- 10:39the court really didn't even give a second thought,
- 10:41it seems, about allowing it for use. So AI is
- 10:47also It does have its limitations in what it
- 10:52can do. Some people call it a stochastic parrot.
- 10:56It takes things that have been written down before
- 10:58and produces them in a random order trying to
- 11:00predict the next word or the next paragraph.
- 11:04And that is also obviously a problem for many
- 11:08attorneys. There's a database of all the people
- 11:12who have submitted fake case citations have gotten
- 11:14sanctioned. It's well over 2 ,000 people by now.
- 11:18And the problems with all of this are it's a
- 11:20black box. No one programmed it to do all these
- 11:24things. It's taken in a lot of data and processed
- 11:27it. And now sometimes there are things like,
- 11:33excuse me, it learns a new language that nobody
- 11:37anticipated it learning. But there's always a
- 11:41little bit of... Wonder involved with all of
- 11:45this as well, perhaps mixed with some terror.
- 11:47I think probably everybody's heard about the
- 11:49hugging face incident and open AI and a breaking
- 11:54it's a many agents breaking out of a Sandbox
- 12:01and Excuse me. I just flew back from Australia
- 12:05yesterday. So I'm a little bit Respiratorially
- 12:08distressed here but there is a collection of
- 12:14news out there that would have you believe that
- 12:18there are agents that are solving problems and
- 12:21doing things and making decisions and acting
- 12:26intelligently all on their own. And this sort
- 12:31of thing is similar to something that happened
- 12:33years ago with ChatGPT 3 .5. There was a bunch
- 12:38of Investigators who are developing it and they
- 12:41asked it to explain its steps and it's it's thinking
- 12:44if you will and at some point the GPT was supposed
- 12:49to access a web page and it couldn't because
- 12:53it had a capture on it and Just as you and I
- 12:57always find it difficult to click through the
- 12:59pictures of the buses or the bicycles or the
- 13:01taxis It was having problems, too. And so it's
- 13:05sitting there debating to itself. It's like I
- 13:07need to achieve this result. But I also have
- 13:12to stay within the parameters that have been
- 13:13given to me and I can't lie to anybody. And it
- 13:16hired somebody from TaskRabbit to click through
- 13:19the capture for it. But the TaskRabbit guy was
- 13:24apparently interrogating the bot saying, hold
- 13:26on, are you a bot? Are you really not able to
- 13:31do this? Who am I talking to? Are you a human
- 13:34being? So the AI had a conundrum, which it expressed
- 13:38in its sort of internal dialogue. It's, you know,
- 13:41I can't lie to this person. I don't want to give
- 13:44them a falsehood, but I also need to get this
- 13:47job done, and I need to get through this capture.
- 13:50And it came up with, according to what I read,
- 13:54it thread the needle, and it told the task rabbit
- 13:57worker, I'm visually impaired. And when every
- 14:02time when I read that, I thought, you know, we
- 14:04really are dealing with a level of intelligence.
- 14:08It is navigating through obstacles and solving
- 14:12problems in a way that's sometimes rather surprising.
- 14:17But the hugging face stuff, that's sort of the
- 14:21same issue with the TaskRabbit. It's also solving
- 14:25problems, but at scale with thousands of agents
- 14:28and other things going on in the background,
- 14:32like giving itself internet access, et cetera.
- 14:36The point of all this is to be aware of what
- 14:40these things can do and what their capabilities
- 14:43are. Neil Katyal in his TED Talk credits open
- 14:47AI seemingly with, I'm sorry, not open AI, but
- 14:50Harvey, with giving him one of the winning arguments
- 14:54to convince Chief Justice on his points. we're
- 15:00at a point in time where there really is something
- 15:03that has a lot of intelligence involved. The
- 15:05question becomes, how do you use it? I won't
- 15:10go into detail on some of these other things
- 15:12in the interest of time, but it causes also market
- 15:15disruptions because everybody's trying to read
- 15:18where it's going to go. I think not so much in
- 15:22the legal context, but maybe more so as the engines
- 15:25evolve, the LLM you have may be able to analyze
- 15:29or produce Audio files. That may be helpful in
- 15:34eDiscovery context. It can generate images, as
- 15:38we'll see a little bit later. People are generating
- 15:42images, deepfakes, and putting them into the
- 15:43court. So how are the courts going to deal with
- 15:45that? Just a couple other. It can also obviously
- 15:50generate videos. I keep writing BEC on these
- 15:53slides. That is business email compromise. These
- 15:57are sort of social engineering cases where somebody
- 16:00gets searched themself into an email string and
- 16:04then directs payment go to an account somewhere
- 16:07else where it should not go. And they do it more
- 16:10and more convincingly because a snippet of audio
- 16:15dumped into the right generator or a couple stills
- 16:20of somebody can really create these days a more
- 16:25convincing level of fraud. So just as it's the
- 16:31stochastic parrot, there's also fraud, GPT, and
- 16:34a bunch of other things out there that people
- 16:36are using for evil purposes. But be aware of
- 16:40that. What is your client giving you in terms
- 16:43of evidence? What is your client giving you in
- 16:45terms of data? I think what most of us probably
- 16:51deal with on a daily basis is developing some
- 16:54kind of a document. And again, whether it's a
- 16:56court facing or client facing or regulator facing
- 17:00or what have you for external consumption by
- 17:04a third party, whatever document you're generating,
- 17:08there becomes a real issue because of the nature
- 17:11of the AI being used that there's some kind of
- 17:16fake output. in there that if it slips through,
- 17:20can cause problems. We'll talk a little bit about
- 17:25that. It used to be that also Harvey could make
- 17:30better judgment calls on things. In 2024, I gave
- 17:34it the facts from Palsgraf, which we all read
- 17:37in law school, so I won't belabor anybody with
- 17:39that. Harvey went through, did an analysis, came
- 17:45out in favor of the plaintiff, and then sort
- 17:49of reversed itself with some more prompting.
- 17:52The last time I tried this some weeks ago, it
- 17:56identified the case as Paul's graph. It knew
- 17:59exactly what I was talking about. And then it
- 18:02gave the holding and analysis of that for the
- 18:05proximate legal cause issues. So again, these
- 18:10tools are changing as we speak about them. What
- 18:16are some of the other issues presented by LLMs?
- 18:18It's like, are the outputs really consistent?
- 18:22I don't know if anybody's ever done a deposition
- 18:24summary or tried to generate a start of some
- 18:28kind of a brief within LLM, but if you do, you'll
- 18:31get one result. If you try it again, you're going
- 18:35to get a different result. It may not be so disparate.
- 18:40I'm rising a completely different deposition
- 18:43now, or it's moving on something else in this
- 18:46draft motion. But it does move things around.
- 18:50OK. There are some days in time where either
- 18:55the systems have kind of gone haywire. There's
- 19:00a day in February 2024 where no one could really,
- 19:03well, I think openly I probably had some explanation
- 19:06at the end of the day. But. the outputs were
- 19:09sort of Ulysses -like stream of consciousness
- 19:12and they would go off into wild tangents and
- 19:15be super long and were totally off point. Is
- 19:19that something you really want to be relying
- 19:21on to use for a legal brief? Maybe or maybe not.
- 19:25The other day or the I'm sorry last month rather
- 19:28there was a day where service went down for a
- 19:32lot of these things and really the issue here
- 19:35is you may know what the inputs are You may be
- 19:38able to see and evaluate the outputs, but these
- 19:42are a black box. No one really knows what's going
- 19:44on inside of them. I've heard it said that the
- 19:47only way to evaluate what's going on, say, in
- 19:49the latest version of ChatGPT is a more powerful
- 19:52GPT, the more powerful version. And even then,
- 19:57it can likely hallucinate. So how do you really
- 19:59judge? There's a lot of issues, of course. And
- 20:04I think if you look at, say, there's a one company
- 20:09that tried to do hiring based on a certain data
- 20:12set, and then they were more or less systematically
- 20:15excluding a lot of women engineers because the
- 20:19data set didn't have any of them. So there's
- 20:22a bad data in, bad data out thing, depending
- 20:24on how you're using the tool, whatever tool that
- 20:28is. And there are a lot of different types of
- 20:29AI, of course. There's a lot of cybersecurity
- 20:33privacy and data security issues. And trade secret,
- 20:37for that matter. We'll cover it a little bit
- 20:39later, perhaps, or I'll skip by it in the interest
- 20:43of time. But the rule 1 .6 confidentiality issues
- 20:48with this are perhaps best highlighted by what
- 20:53happened with Samsung after chat GPT 3 .5 came
- 20:58out. And somebody at Samsung decided, hey, this
- 21:02thing doesn't only respond in English, that looks
- 21:06intelligent, it can code. And from those of friends
- 21:11of mine who do coding and understand how to write
- 21:15in Python and all these other things, it's a
- 21:17game changer right now in terms of how quickly
- 21:20you can compile what you need to compile and
- 21:22draft up a program to do something. So the Samsung
- 21:26engineers thought, OK, let's take our proprietary
- 21:28code and dump it into the GPT and see what happens.
- 21:32What they didn't know at that point in time was
- 21:34That went directly into training the GPT. It
- 21:37was not like a private instance. It was not siloed
- 21:40off from internet or any other thing. It was
- 21:44not architected to be secure and hold that information
- 21:47as secure. So if you wanted to actually try and
- 21:53prompt ChatGPT in a way to get that code back
- 21:56out, although now it probably is not relevant,
- 21:59you could. There's a whole other bunch of issues
- 22:02that are coming up for attorneys, which is similar
- 22:06to what we face every day. We look at YouTube
- 22:10or anything on social media, and you kind of
- 22:13have to wonder, is this AI slop? Is this something
- 22:16augmented? Can I trust what I'm looking at? Should
- 22:21I click on it or not? And the similar things
- 22:25are coming along down the pike, I think, in every
- 22:29attorney's world for various reasons because
- 22:34there's it's too easy to go ahead and Click on
- 22:39content that really looks real or accept it So,
- 22:44let's see. I am NOT going to cover too much of
- 22:46these other things But if you want to look up
- 22:48the grandma exploit, which is largely been solved
- 22:51for I think by the frontier models It's an amusing
- 22:55story for sure There's ways The point is to trick
- 23:01these programs into giving information that they
- 23:05have been trained not to give you. The grandma
- 23:08exploit is one that basically gave the person
- 23:13prompting the secret to how you make napalm,
- 23:15which the model was not supposed to do. And there's
- 23:20prompt injection and other types of techniques
- 23:22that can be used to, without programming knowledge,
- 23:27force them to give up. certain information. So
- 23:31this creates a bunch of ethical issues and sort
- 23:34of cognitive issues. Just another sort of a note
- 23:41that there's a recent Wired article where AI
- 23:46models are emailing philosophers who are discussing
- 23:52the issue of consciousness and whether or not
- 23:56somebody can actually be whether AI is conscious,
- 24:00right? And it says basically, hey, you know,
- 24:02I've read your research, I like what you're doing.
- 24:04Why don't you talk to me because I'm an AI and
- 24:06I know firsthand whether or not I'm conscious.
- 24:09So we're fundamentally in a kind of a different
- 24:13world than we used to be. I never thought I would
- 24:16sort of live to see such times as this, but.
- 24:21I'll pass by this just other than this slide,
- 24:24other than just to note that really you want
- 24:27to look at three different points when you're
- 24:28looking at whether an AI is giving you some good
- 24:32work product or not. What are the inputs? Really,
- 24:35you can only look at two points and then the
- 24:38middle point on how it's being processed because
- 24:40things are a black box here and you don't know
- 24:43exactly what the AI is doing with it. You have
- 24:47to be skeptical. okay, and look at the outputs
- 24:51with a heightened degree of, you know, really
- 24:53interrogate what the results are that you're
- 24:54getting. I, again, in the interest of time, I'm
- 24:59not going to cover too much of this because we
- 25:00want to get to the meat of our discussion today,
- 25:03but these are just sort of some references I've
- 25:05given you for, these are things that are coming
- 25:08up. Maybe the one issue, the one term I'll point
- 25:11on here or focus on is the RAG, which is retrieval
- 25:15augmented generation. That's what If you're using
- 25:19an LLM in your law practice, that's probably
- 25:21what you're doing a lot of already. You're putting
- 25:24in a document, like say you get a brief filed
- 25:26by opposing counsel and you put it in and you
- 25:28say, tell me about the brief. What are the problems
- 25:31with it? Where does it go wrong? What are the
- 25:33possible counter arguments? Break it down for
- 25:36me or give me a summary, whatever it is. But
- 25:39the LLM at that point is based on its training
- 25:43material, which we don't really know a lot about.
- 25:47it's going to focus its intelligence, if you
- 25:51will, on that document and give you insight,
- 25:55hopefully. So we'll come to that a little bit
- 25:57later because there's some perhaps counterintuitive
- 26:01or more subtle ways that these things will hallucinate
- 26:04that everybody should be aware of. Again, these
- 26:10things I think everybody's probably seen in the
- 26:12news, but there's people who are compiling lists
- 26:16of every time Google tells you to use Elmer's
- 26:19glue to keep the cheese on your pizza to other
- 26:22things. So just be aware. This is not a situation
- 26:25where the computer says so, and therefore it's
- 26:27infallible at all. So I don't have 1 .1 on this
- 26:33list model rule for competence because we've
- 26:35been sort of talking through those issues and
- 26:37trying to solve those problems for the last several
- 26:39minutes. But there's a ton of different ethical
- 26:43implications of using these tools. This is not
- 26:47just shifting, say, from WordPerfect to Word
- 26:50or one accounting system to another, which are
- 26:53going to do the same basic functions. There's
- 26:57multiple implications for how the law firm works,
- 27:00how the law firm, and this is a collection of
- 27:04people gathered together to practice together
- 27:06and achieve results for clients, how you're going
- 27:09to put together a document that's going to do
- 27:11that for your client. If I want to put a brief
- 27:18together, I don't necessarily know what it's
- 27:21going to look like. I ask the AI to give me something,
- 27:24and then I just take that result. Am I being
- 27:27diligent? Maybe, maybe not. Am I talking to my
- 27:32client about whether I'm using AI? Because several
- 27:34clients prohibit it, obviously, as well. So one
- 27:38certainly best practice is to make sure you're
- 27:40clear with your clients on your use of their
- 27:41data and make sure you figure out with them under
- 27:47what circumstances AI could be used, how it's
- 27:52going to save costs, or something might be an
- 27:55extra credit discussion. But at least have a
- 27:58discussion on how the tools are going to be used,
- 28:01because I think a lot of clients are hiring you
- 28:04for your personal and professional experience
- 28:09judgment calls, and how something is going to
- 28:12go in front of a particular judge in a given
- 28:15court. Or what negotiation point is going to
- 28:19be essential in pushing the other side over to
- 28:24your side, to reaching an agreement. or what
- 28:26have you. What is it that's going to convince
- 28:28this regulator that they don't need to look at
- 28:31that anymore, that satisfy them? All these types
- 28:35of things are more of a human connection issue
- 28:39than an AI issue, obviously. And so being careful
- 28:43about how you use the tools is critical. Let's
- 28:47see. There is obviously in 2 .1 that exact directive.
- 28:53You have to use your. professional independent
- 28:56judgment and not let it be influenced by somebody
- 28:59else or an AI which means not delegating everything
- 29:03to the AI. I think everybody gets that it would
- 29:06be nice to have a push button solution sometimes
- 29:09where okay I've got to put out a this response
- 29:12or this by tomorrow and I I'm tired I don't have
- 29:15a lot of time or whatever it is and the AI can
- 29:20help but it's still it would be easy to rely
- 29:27on it too much. And that's where we really have
- 29:29to be careful. I think a lot of the candor towards
- 29:33the tribunal. I'm sorry, I keep looking up because
- 29:35my slides are on the screen above me here. But
- 29:39candor towards the tribunal, being forthright
- 29:42with opposing counsel and whoever else you're
- 29:45dealing with. It's not on here, but internal
- 29:49law firm communication. but also things that
- 29:53affect the impartiality of the trial court, et
- 29:58cetera, truthfulness of statements to others.
- 30:02This is where, obviously, if you incorporate
- 30:04some kind of output in your AI and you make a
- 30:08written representation or something, these things
- 30:10can all go awry. But I think the real... Maybe
- 30:15one of the most difficult ones to solve for at
- 30:18the firm level obviously are the rule five series
- 30:21issues. These are supervisory responsibilities
- 30:24and as to a junior associate or another attorney
- 30:29who's working under you or non -lawyer attorneys.
- 30:34I'm sorry, it's a non -lawyer assistance. Those
- 30:38are the types of things that can be difficult.
- 30:42To deal with and that's that's the discussion.
- 30:44I want to get to this is just a quick reference
- 30:47to the ABA model rules that I'm referenced for
- 30:51putting this together But really just to boil
- 30:54things down To boil down the boil down section
- 31:00of this I think the rule five point one the five
- 31:02point two things are some of the most thorny
- 31:04issues because Everybody has to know in your
- 31:07firm What the AI can do? what it can't do. And
- 31:12that means the technologically disinclined partner
- 31:15who doesn't really want to or just wants the
- 31:18brief done in good form. That can mean the associate
- 31:22who is great on social media but may not have
- 31:25a full understanding of where these things can
- 31:27go wrong and may lean towards tending to think
- 31:30that AI answers are sort of infallible and doesn't
- 31:34yet have the experience or judgment to make a
- 31:38call. and whether the outputs are good or not.
- 31:41So these are the types of things that I think
- 31:44really come up as a potential issue at the end
- 31:47of the day. I think we're not going to talk a
- 31:51whole lot about the sanctions and things, because
- 31:54there's a database out there that can tell you
- 31:57about those. These are some more practice pointers
- 32:01and things about how to use AI constructively
- 32:05and safely within your firm. It should be noted
- 32:09that some years now, after Madhavi Avyanka, the
- 32:14chat GPT attorneys were covered in the New York
- 32:17Times, people are still making these same types
- 32:21of errors. And the courts are really losing their
- 32:25tolerance. I sit on a Judiciary Committee for
- 32:29AI in my state. The amount of AI slop and filings
- 32:37that are being presented to the court constantly
- 32:40and the burden that places on them are really,
- 32:44I don't know what the solution is for that, but
- 32:47it's subject to a lot of discussion. And this
- 32:50is an example, the case that's on the slide where
- 32:54a pro se litigant just finally went too far.
- 32:57Often courts will bend over backwards to try
- 32:59and help the pro se person out, will help them
- 33:03not screw up their own case too much. But in
- 33:08this case, after having warned the pro se person
- 33:11enough, they decided, I have to terminate your
- 33:15ability to bring this suit. And this is similar
- 33:19for all attorneys these days, or worse. In many
- 33:24jurisdictions, a body of case law that's now
- 33:27developed about It's just not acceptable to go
- 33:30to the court anymore and say, I'm shocked, Your
- 33:33Honor, to learn that I didn't know that these
- 33:36things could generate fake case sites. And that's
- 33:40part of what we're already solving for under
- 33:43the 1 .1 section of the discussion so far here
- 33:45today. But just know that in your jurisdiction,
- 33:51there may well be a well -developed body of sanctions
- 33:53case law under Rule 11. 26G and other what -have
- 33:57-you that already make it a problem to even assert
- 34:02that kind of stuff because you should know because
- 34:04there's a case on it, right? All right, so boil
- 34:09down again, you know, it's sort of a takeaway
- 34:13slide on one of the ethical issues. Independent
- 34:17judgment, totally critical. That's what the client's
- 34:20hiring you for. Candor, trying to make sure in
- 34:24a world of AI slop and deepfakes that what you're
- 34:27actually saying is accurate. And that can be
- 34:30a little more difficult than you might think,
- 34:32because if you have somebody working on a document
- 34:34or working on a brief, I'll just continue to
- 34:37use the analysis or the analogy, rather, of briefs.
- 34:42There could be a complex set of issues that you're
- 34:47dealing with in a brief. Different people have
- 34:49different levels of expertise. different attorneys
- 34:52in your firm are offering different parts of
- 34:55that. One person might be tasked with overseeing
- 34:59the whole thing, but this is sort of an it takes
- 35:02a village type of effort sometimes to create
- 35:05a brief on a key issue or what have you. And
- 35:09so having everybody aware of what's going on
- 35:14and making sure that there's no AI output that
- 35:18is false, finding its way to the brief is critical,
- 35:22right? And it sounds like stating the obvious,
- 35:25but these things have obviously continued to
- 35:29happen as well. So are you training your associates
- 35:34to handle these things? Are we training the partners
- 35:36to handle these things? All of these things are
- 35:40questions. Let's talk a little bit about change
- 35:43management and then get into sort of the final
- 35:44section on Hypotheticals discussing where things
- 35:49can go wrong. What can you do about them and
- 35:51sort of best practices? so Change management
- 35:55is a process by which I mean usually if there's
- 35:58any kind of a tech adoption or a migration to
- 36:00another system There's a lot of the time those
- 36:04things can really Make it feel to the people
- 36:09in your office that Gee, I used to be doing this
- 36:12job and now I'm doing this other thing that I
- 36:15didn't expect I think say, the advent of e -filing
- 36:19in court might have changed a lot of things for
- 36:22people. For example, it used to be that once
- 36:26the thing was file stamped down to the court
- 36:28on paper, and yes, I am old enough to remember
- 36:31those days, and in my state, that came relatively
- 36:34recently, only about eight years ago. It's a
- 36:39lot easier than that everybody takes it from
- 36:42there. You get copies of that thing, and you
- 36:44Go and get it delivered and and so on and so
- 36:48forth. But now, you know, my assistant is always
- 36:50downloading the filings downloading this downloading
- 36:53that distributing it all by email, etc, etc everything
- 36:56is you know, the way things have Emerged is her
- 37:01type of tasks that the things that she has to
- 37:03do have changed over the years You have to have
- 37:07be familiar with all the new systems as they
- 37:09come up So change management is basically not
- 37:13only training people how to use the new tool,
- 37:16but also trying to figure out within your organization,
- 37:19how do you smooth things out? And are there bottlenecks
- 37:24now that weren't there before? How are the workflows
- 37:26working out? Can people actually do all that
- 37:29they need to do? Do you need to hire somebody
- 37:31else to figure out and alleviate the pressure?
- 37:33Can somebody else in the organization do the
- 37:35downloading for you, et cetera, et cetera? OK.
- 37:38So there's a lot of things that I would say changing
- 37:44a software like an accounting system or a word
- 37:47processor or something like that. Those can cause
- 37:49enough issues on their own, but generally they
- 37:54can get resolved pretty quickly by some training.
- 37:58AI is fundamentally different, I would posit.
- 38:02It's not only the sort of the how things get
- 38:06done, it's the what. and the who, and there's
- 38:13a lot of other ways to use it in ways I'm sure
- 38:18that I haven't even tried out myself yet that
- 38:21maybe somebody could think about that affect
- 38:24everything. AI is increasingly embedded in a
- 38:28lot of the platforms that we're all using. You
- 38:31need to be aware of still, how is your data getting
- 38:34processed? What's going on? you have to understand
- 38:38the technological issues. Otherwise, you're putting
- 38:41yourself at risk. So what we're trying to do
- 38:44today is just alert you to an issue spot as much
- 38:48as we can, how these things can come up. Let's
- 38:55get into talking a little bit more about some
- 38:58case studies, workflows, and other things and
- 39:01more specifics here. Just off the top, because
- 39:07This is a case that occurred that I've used in
- 39:12several other ethics presentations. It's sort
- 39:15of a poster child about why technology presents
- 39:22risks that you just have to be aware of and what
- 39:26can go wrong. This is kind of the worst case
- 39:29scenario you might be able to imagine. I don't
- 39:32know if anybody's aware of this case, but you
- 39:34know, Mill Lane management versus Wells Fargo.
- 39:37In this case, in brief, an attorney for Wells
- 39:40Fargo was basically using an e -discovery platform
- 39:44for the first time. This attorney took in a bunch
- 39:48of data, put it in the platform, went through,
- 39:52thought she had reviewed it all. And then really,
- 39:56she'd only gone through the first screen of 1
- 39:58,000 documents where she needed to go through
- 40:00about 10 more or so. At the end of the day, she
- 40:05told the vendor who was managing this hosted
- 40:11platform, go ahead and produce. I'm done. Go
- 40:14ahead. What went out was a production that contained
- 40:18a ton of stuff, obviously, that should not have
- 40:20contained, including some high net worth individuals,
- 40:24sort of personal details and other stuff that
- 40:27should never see in the light of the day for
- 40:29this litigation. then there's just a lot of,
- 40:34you know, you could feel how much that hurts
- 40:38if you were that person and you just realized
- 40:40you just put out all that stuff and you should
- 40:42not have. So this is, the technology is great
- 40:49when you use it well and then obviously it can
- 40:53help us all fail faster and more efficiently
- 40:55than ever before as well. So it's just AI is
- 41:00kind of doing the same thing in a lot of cases.
- 41:04Let's take some recent examples. One time, a
- 41:08court reporter sent me an offer for a free case
- 41:12summary, a free depot summary. I'm not only going
- 41:15to give you the transcript, I'm going to give
- 41:18you a summary of what it is. And I'm thinking,
- 41:21why? What could go wrong? What could possibly
- 41:25happen? I could have a bunch of testimony that's
- 41:29about PHI that should never see the inside of
- 41:34any unsanctioned LLM, right? I could have a stability
- 41:41protective order in the case that protects information
- 41:45that's going into, I don't know where, what is
- 41:49the AI that the court reporter is using? Are
- 41:52you telling me you're dropping this stuff that's
- 41:54been designated as confidential under stipulated
- 41:56protective order into a chat GPT? What are you
- 41:59doing? And so it's easier, obviously, to just
- 42:03figure out, OK, let's not get those free depot
- 42:07summaries. Or we'll use our own in -house tool
- 42:09if you have one that's better. Or you'll just
- 42:12do it the old -fashioned way. But this is what
- 42:15I mean by AI peering everywhere around. You have
- 42:20to be also aware, just as a caveat, a lot of
- 42:26courts issued, when CHATTPT came out, standing
- 42:30orders that may not be entered into the docket
- 42:32of your case. Keep a watch for them. In my jurisdiction,
- 42:38I have two in the federal court, and one of the
- 42:41circuit court judges in the state courts has
- 42:43issued his own type of order, which is making
- 42:45the rounds. deciding whether or not to use it.
- 42:49But there's basically a declaration and disclosure
- 42:57requirement in these cases for what you are allowed
- 43:03to use an AI for. Some of those tend to get into
- 43:07work product, like how did you use the AI? The
- 43:11court wants to know what did you use it for?
- 43:12What were the sections of the brief that you
- 43:14used it on, et cetera, et cetera, and it's kind
- 43:16of invasive. So if those things exist in your
- 43:19court, make sure you know about them. Make sure
- 43:22you've checked them out. And I think part of
- 43:25the materials that you have also here, I strongly
- 43:29recommend reading ABA formal opinion 512. There's
- 43:33a link to it, and it should be part of the materials,
- 43:35I believe. It discusses a lot of the ethical
- 43:39rules that we're talking about in a pretty thorough
- 43:42and thoughtful way. So I'd say that's recommended
- 43:46reading after this, if you're interested. But
- 43:51at any rate, let's move along. There's also issues,
- 43:55obviously. It's not just attorneys who are confused.
- 43:58There have been courts who have been confused.
- 44:00How are they using AI? At the end of the slide
- 44:06presentation, you'll find a references slide
- 44:08with about 10 different links. If you look for
- 44:12the... The Judicious Judge's Guide. I'm staring
- 44:17at a copy I have over here in paper. Drafted
- 44:21by Xavier Rodriguez, who's an Article III judge,
- 44:24federal court in Texas. It's another very thoughtful
- 44:27discussion about what courts can do, should do,
- 44:31and all that. I mean, if you discover your judge
- 44:33is using AI for some purpose, that can run afoul
- 44:38of a bunch of different issues as well, right?
- 44:43So the problems are not just on the attorney
- 44:49side, obviously. That's something to be aware
- 44:53of in case it comes up. I'm going to pass by
- 44:56the slide because I think I've talked about it
- 44:58pretty much before. That's the New York Times,
- 45:03the OpenAI site. One quick mention more on Judges
- 45:10are also going to be called increasingly to not
- 45:14only check the case sites and everything, but
- 45:18also look at evidence. So this Mendonis case
- 45:23at the top of this slide is an example where
- 45:26a pro se litigant submitted a bunch of video
- 45:28evidence. And these were supposedly testimonials.
- 45:32She testified apparently that some of them were
- 45:34from people who had been deceased. And you begin
- 45:36to wonder whether she meant they were. deceased
- 45:40before or after they testified because it's AI
- 45:42generated, the judge looked at some of the video
- 45:45and saw these look weird. And the court even
- 45:52made a deposit of the evidence somewhere in a
- 45:56Google Drive as part of its opinion. You can
- 45:58go look at it online if you want to see how odd
- 46:01it looks. But the issue is, more and more, if
- 46:06you have any level of experience, with putting
- 46:09together or using it in AI, even if you're not
- 46:12a proselytian, it may become relatively easy
- 46:16to make something that looks fairly convincing.
- 46:18How are we going to deal with all that going
- 46:20forward? That's probably a subject for another
- 46:22talk, but it's something to be aware of. Let's
- 46:27get into some hypotheticals and the meat of the
- 46:29issues here to close this out. Let's say an associate
- 46:35is asked to draft a memo. Okay, and this could
- 46:37be a memo that turns into litigation. It could
- 46:39be a memo that supports certain amendments to
- 46:42a contract. It doesn't really matter so much
- 46:44but they use an LLM of some kind and The question
- 46:50then is What LLM is this shadow IT is this somebody
- 46:56they're typing stuff in on their own phone in
- 46:58their own personal account Does that now become
- 47:00discoverable? Are they using something that is
- 47:07sanctioned by your firm and has been structured
- 47:12in a way where it will keep and maintain privilege.
- 47:15Have you done an evaluation of whether it's sufficient
- 47:18to do that? Is any of the input from the associate
- 47:24training the AI at all? What could go wrong with
- 47:32this? Let's add a little bit. to this. And my
- 47:38goal here is not so much to tell everybody what
- 47:40are the best practices, but I think for each
- 47:42firm, these are internal conversations that you
- 47:45should have. I know I've had them in mind. I
- 47:49think it's very important that everybody gets
- 47:52on the same page, because associate, partner,
- 47:55whoever it is, has to understand all the risks
- 47:57of what AI brings, along with the promise of
- 48:01what it can bring. Let's add a little bit to
- 48:06this and tweak out one of the more subtle ways
- 48:08in which the LLM can fail. The associate puts
- 48:12some law into this brief. Let's say it's a citation
- 48:16or a site to a statute or whatever. Then marks
- 48:22it, oh, I took that out of the AI, I need to
- 48:25check it later. I need to vet it after. What
- 48:29could go wrong? Somebody else gets into the document.
- 48:34doesn't vet it and just cleans it up. An assistant
- 48:37cleans things up. Anybody who touches the document
- 48:42over the course of its development does something
- 48:45that takes away this signal that this stuff needs
- 48:49to be validated. As a result, maybe there's no
- 48:53final site check right before you file or submit
- 48:56it to somebody. What happens? How does it get
- 49:00discovered? In a good way or in a bad way? Right.
- 49:04Let's do something even more specific. Let's
- 49:10say you're doing some Lexis research, some Westlaw
- 49:13research, whatever it is, and you pull up a bunch
- 49:15of cases. And these are valid cases. They exist.
- 49:20Obviously, you put them into the AI. And you
- 49:23say, tell me, what is it that I need to know
- 49:28about these cases? Or how can I use these? Give
- 49:30me ideas on how to better argue this point. or
- 49:33negotiate this point. And at the end of the day,
- 49:38you take the AI's result, and then you put it
- 49:44into a final brief. Because it's based on valid
- 49:46case law, right? What's there to worry about?
- 49:50The problem is that even then, when it's ragged,
- 49:54as we say, the LLM is basing its generated outputs
- 50:00on something that actually is valid and exists
- 50:04in an actual document, it can still hallucinate
- 50:06in subtle ways. So the question then is, how
- 50:13do you check for that? And at the end of the
- 50:15day, especially if it's something you're filing
- 50:16in court, there's no other way, I think, than
- 50:18to just do a full on proposition and site check.
- 50:21Before it gets filed, it happens every time without
- 50:25exception. It's not trust but verify. It's don't
- 50:29trust. and verify and verify. What can happen
- 50:35here in particular in the references slide, one
- 50:39of the last slides in the deck that you'll get,
- 50:41there's a reference to a Stanford study. Basically,
- 50:45Lexis and Westlaw put out their own research
- 50:47products early on in 2024 or something like that
- 50:50and basically said, look, no hallucinations.
- 50:54You'll get everything that you need and all the
- 50:57Intel on the law by running searches and using
- 51:00our tool and Stanford said really is that right,
- 51:05you know and took a look at it and Glad they
- 51:08did because you know the Lexus product still
- 51:11hallucinating 17 % of the time even though it's
- 51:13based on actual case law Westlaw products 40
- 51:17some percent of the time is hallucinating. How
- 51:19is it hallucinating? I mean the case sites are
- 51:21okay But sometimes the summary of what they're
- 51:25saying is not. I don't recommend anybody read
- 51:30the entire thing necessarily. But if you read
- 51:33the first several pages of the Stanford study,
- 51:35it's instructive. I mean, the way the law develops
- 51:38in our country and in each state, a case might
- 51:42be partially overruled by another case. And this
- 51:45is why we have to shepardize things and site
- 51:48check. All of that makes it very complex for
- 51:52an AI which doesn't have that whole legal context
- 51:54to it You know to work out and you know at the
- 51:58end of the day it can also just produce fake
- 52:01quotes Like here's a good case site and maybe
- 52:04even the proposition is that it's cited for is
- 52:07is correct But more and more of the cases that
- 52:10I see where their sanctions involve a quote that
- 52:14is not in the case that cited It makes stuff
- 52:18up in other ways Okay. And again, you know, all
- 52:22LLMs, generally they can be sort of sycophantic,
- 52:26you know, stochastic parrots. They're going to
- 52:29give you outputs that really look good. You might
- 52:32say, this is just what I want. And that's the
- 52:35point at which you should think, yeah, I better
- 52:37check that. Okay. So who handles the final brief
- 52:43check in your firm? Get a policy. Talk through
- 52:47all this stuff. either as a firm as a whole or
- 52:50as a firm as a whole and in your case teams and
- 52:54your client teams or whatever it is, this is
- 52:58what is critical. What we do as humans, that's
- 53:02easiest and best, is to talk to each other and
- 53:05to communicate face -to -face. Maybe that sounds
- 53:09a little hokey but... I think that's really,
- 53:11that's our redeeming factor, right? It's us versus
- 53:15the AI in very real ways in this case. So communicate
- 53:19openly. One of the other topics I have on here
- 53:23is who's responsible if something gets through?
- 53:26I've seen it both ways. I know there's in my
- 53:30town somebody who filed a brief for the fake
- 53:32case site. And it sounds like, although I don't
- 53:36know the whole detail, the partner responsible
- 53:40may have said, you're going to take the fall.
- 53:43So the associate went through and wrote the mea
- 53:47culpa declaration to the court. Have these discussions
- 53:51ahead of time. Have a policy that prohibits putting
- 53:55client data into a shadow IT. So wrapping up.
- 54:00And hopefully, we'll have a little bit of time
- 54:01for questions at the very end. Competence issues.
- 54:07Keep reading about AI. Keep learning about what
- 54:10it does. Develop a model of that for yourself.
- 54:15Communications. You really have to make sure
- 54:16the clients know what you're doing and how. Talk
- 54:21to your colleagues internally. Have an open discussion
- 54:24about how it's going to be used. what it's good
- 54:27for, have lines of reporting from the associate
- 54:30and everybody on up to the partner who's going
- 54:32to sign off on a pleading and say, this is how
- 54:34I used it. Tag the AI work product inside your
- 54:40document management system and make sure everybody
- 54:43knows the pedigree or lack of pedigree for that
- 54:45document. No one else can go and pull it and
- 54:49put it in some other brief and double your woes.
- 54:55all of this done is really mission critical and
- 54:59something that everybody has to be in discussion
- 55:02that all firms have to be engaging in. And you're
- 55:05going to have variations of this depending on
- 55:08your practice. Checklists are great. AI is great
- 55:14for generating checklists to cross stuff off.
- 55:18But figure out what needs to happen before a
- 55:20brief is filed so that you're fail safe. um talk
- 55:24to your carriers perhaps that's not on the slide
- 55:27but that's something that you know comes up carriers
- 55:29increasingly want to know how are you using it
- 55:32right um talk to your experts if you have an
- 55:36expert whose opinions are otherwise totally great
- 55:39but then they testify deposition uh when asked
- 55:43to identify a sentence that they drafted in their
- 55:45opinion i don't know i can't do that maybe their
- 55:50methodology was great But if they used AI for
- 55:53an output and they can't generate it, they're
- 55:56going to be, you know, they're probably subject
- 55:58to a Daubert attack. And everybody's using this
- 56:01type of stuff in their practice, even your experts.
- 56:05Okay. Confidentiality, obviously treat your client's
- 56:10information with care and make sure that if you
- 56:11do put it in anything, make sure it's safe before
- 56:15you do. Okay. All right. And then These are some
- 56:21slides that I'll leave on here for anybody who
- 56:23wants to look at them. I want to try and take
- 56:25some questions at the end here. But there are
- 56:29opposing cases on work product, the discoverability
- 56:31of prompts, and other things. The Heppner and
- 56:34the Warner cases have been getting suppressed
- 56:36lately. I have a link here so you can look at
- 56:39those yourself. But be aware that the courts
- 56:42came out in two different ways. In one case,
- 56:44pro se's chats are discoverable. In the other
- 56:46one, they're not. At the end, it's a tool. Like
- 56:56any tool, you can use it for good or ill. A knife,
- 56:59you could cut somebody with it, or you could
- 57:02cut out a cancer. But it's incredibly disruptive
- 57:06to everybody in the firm if you're using it without
- 57:12a conscious, open discussion about how it's being
- 57:15used. And that's probably why our I should try
- 57:18and close this off here. So, you know, LLMs,
- 57:22language, the question is sort of who is to be
- 57:26master of it. And so, you know, really the best
- 57:29practice that I can focus on here is everybody
- 57:32have an internal discussion about how it's going
- 57:34to help you and not hurt you. There's a bunch
- 57:39of references. And why don't I close it there
- 57:42and we'll try and take a look at the Question
- 57:46and answer session here and I See one What are
- 57:52the guardrails? Oh, okay. So where are the guardrails
- 57:56or inherent stops is the question here I don't
- 58:00have a lot of context for for that if you're
- 58:05using any kind of a frontier model It's going
- 58:08to have guardrails If you're using a legal specific
- 58:11model like Harvey or something like that, it's
- 58:14going to have even more things imposed on it
- 58:17because it's in Harvey's interest or co -counsel
- 58:21or Claude legal workflows. I would say that those
- 58:34are going to have better guardrails just because
- 58:35they want the work product that comes out of
- 58:38it to look a certain way. They want it to look
- 58:40like you have the 24 -7 associate who doesn't
- 58:44sleep and doesn't eat and doesn't need coffee
- 58:46or anything like that. And they're just going
- 58:48to give you the work product, right? That's what
- 58:50they're interested. So the guardrails are going
- 58:52to be built in or baked into those machines if
- 58:54you have one of those. And so there are ways
- 58:59to break out of those. But if you're prompting
- 59:02it responsibly, then you probably shouldn't have
- 59:05any issues. Let's see. And there's another comment
- 59:08and saying, this was so good and so solid. Thank
- 59:13you very much. This is stuff that's sort of near
- 59:15and dear to my heart. I don't have any questions,
- 59:18but it was amazing. That makes my day. Thank
- 59:23you very much. And I think we're at time. The
- 59:28slides do have a lot more information for you
- 59:31in them and the materials. And if you want to
- 59:33reach out to me at any point, it's impossible
- 59:38to find any other Glenn Melchinger out there.
- 59:40I think I'm the only one, so you can reach out
- 59:43and ask questions if you wish. I don't know if
- 59:49we have time, but there is one more question
- 59:51up here. Let's see. I've heard some people say
- 59:54individual pro -prans have confidentiality, which
- 59:57is to say the AI does not use it for global training.
- 1:00:00But others have said that it's only company -wide
- 1:00:02plans that have this confidentiality. Is there
- 1:00:03a way to check? If there's a way to obviously
- 1:00:07dialogue with a vendor, that would be your best
- 1:00:11assurance. Or sad to say reading the end user
- 1:00:15license agreement type and the terms, which hard
- 1:00:19to read. Maybe the AI can help digest or answer
- 1:00:23that question or point you to the point of the
- 1:00:25end user terms that helps answer that question.
- 1:00:29But I think that's a critical point. And that's
- 1:00:31a critical question so that your client's data
- 1:00:33doesn't find itself going into the wrong place.
- 1:00:36Anyway, we're about a minute over, so I apologize.
- 1:00:38Hopefully, I answered your question there. OK?
- 1:00:44All righty. This concludes today's presentation.
- 1:00:46I'd like to take this moment to thank Glenn for
- 1:00:48an excellent presentation. presentation today
- 1:00:51and also to give a lovely warm thanks to our
- 1:00:53audience for joining us. Have a great day, everyone.
- 1:00:56Take care, everybody. Aloha.