Latest / CLEs You Actually Want to Hear / Did My AI Just Make That Up? A Defensive Prompting Toolkit for Hallucination-Proofing Legal Work
Transcript
- 0:03Hello, everyone. Welcome to today's program.
- 0:07Did my AI just make that up? A defensive prompting
- 0:11toolkit for hallucination proofing legal work.
- 0:15I'm Alapoli Castro, Director of Education and
- 0:17Professional Development at Beverly Hills Bar
- 0:19Association. And I'm delighted to introduce our
- 0:21speaker for this program, Ashley Jones. Ashley
- 0:25is Senior Counsel at Lynx Squares, an AI powered
- 0:28contract. Contract Lifecycle Management Platform.
- 0:32Ashley's work at Link Squares focuses on contract
- 0:35management, program management and development,
- 0:38legal operations and negotiation. Outside of
- 0:41law, Ashley is a passionate and prolific artist
- 0:43across multiple mediums. We're so excited to
- 0:46hear what she has to share with us today. So
- 0:47I'm going to hand it over to her so we can get
- 0:49started. Fantastic. Thank you so much for that
- 0:55introduction. I'm so excited to be back here
- 0:57with the Beverly Hills Bar Association. to present
- 1:02on, did my AI just make that up? Prompting to
- 1:05reduce hallucinations. This is gonna be something
- 1:08a little bit different than my last presentation.
- 1:11We're not gonna be talking about an offensive
- 1:13prompting toolkit. This hour is going to be spent
- 1:16just focusing on defense only. And the goal today
- 1:19isn't to make you suspicious of AI. I think we're
- 1:22already a little suspicious, right? It's to ensure
- 1:25you have the tools and skills to hold your AI
- 1:27accountable. But that said, we do have an agenda
- 1:31today and the agenda is hopefully going to help
- 1:33us accomplish the ultimate test, which is can
- 1:36I explain why this output is right or can I only
- 1:40say the AI told me so? So we'll start with our
- 1:43framing, which is essentially going to be why
- 1:46this matters and why hallucinations are more
- 1:50of a liability issue and not a technical quirk.
- 1:53Then we're going to talk about what causes hallucinations.
- 1:57We're going to talk about it from both the practical
- 1:59and the technical side. Hopefully I don't get
- 2:02too technical, but I think these are both valuable
- 2:05points to spend some time getting to know because
- 2:09understanding why hallucinations happens also
- 2:12helps us to understand how we can better mitigate
- 2:15or prevent them. Then we're going to move into
- 2:19the defensive prompting toolkit. We're going
- 2:21to go over five specific defensive prompts and
- 2:24how you can layer them on top of one another
- 2:27for a variety of issues or concerns that may
- 2:30come up. Then we're going to take this a little
- 2:34bit further than just the prompt and touch on
- 2:37two really hot button things in AI right now,
- 2:40which are skills and memory. And this is where
- 2:43we're going to be able to get more preventative
- 2:46with our hallucinations and how we use our AI
- 2:49tools. As housekeeping matter, the slides will
- 2:53be shared with you. I'm hoping we have time for
- 2:56Q &A, but we do have a lot of content to get
- 2:58to. So if for any reason I'm talking too quickly,
- 3:02let me know. And hopefully we get through everything
- 3:06and I get to answer some questions along the
- 3:08way. So first thing we're going to talk about
- 3:14was the framing. And we're going to really get
- 3:17into why this is more professional responsibility
- 3:20in terms of mitigating hallucinations and not
- 3:24just a software question. So next slide, we'll
- 3:30get into why this matters now. And truth is,
- 3:34AI is everywhere. It's embedded in how we conduct
- 3:37our research, drafting, contract review, inside
- 3:40software you're using today. And inside of a
- 3:44lot of the tools we've already purchased in prior
- 3:47like software review cycles, it's being tacked
- 3:51on to many things we're working with and it's
- 3:54just. My opinion, I think something that's not
- 3:56going away for a really long time, it's just
- 3:59going to become more prevalent. And so the more
- 4:02able and ready we are to hold it accountable
- 4:07and hold ourselves accountable, the better users
- 4:09will be in the long run. Second. We've all heard
- 4:14the horror stories of hallucinations having a
- 4:17pretty drastic impact on legal work and legal
- 4:20careers. So that comes in the form of fabricated
- 4:23citations, invented provisions, and these all
- 4:27carry real exposure, whether that's sanctions,
- 4:29malpractice risk, or even declined trust in your
- 4:32clientele, whether that's in a law firm context
- 4:35or in -house. Because let's be real. Clients
- 4:39or customers, whether it's internal stakeholders,
- 4:42they can't separate what the AI wrote versus
- 4:46what my lawyer sent me. And that's our duty to
- 4:48make sure we can stand behind both. And lastly,
- 4:53we do remain accountable for our AI assisted
- 4:56work product. Really, regardless of who or what
- 5:00produced the first draft, it is us being ultimately
- 5:04responsible for that final work product being
- 5:07sent out. And I truly fundamentally believe in
- 5:12the principle of cognitive offloading versus
- 5:15cognitive surrender, which means we are offloading
- 5:20things to help us move quicker and more efficiently,
- 5:23but we're not offloading the actual judgment.
- 5:28The judgment still sits with us. And I think
- 5:30when you take an output from an AI tool and just
- 5:33run with it, you're ultimately surrendering to
- 5:37that output itself without verifying any sort
- 5:42of exposure. So remember, we can offload the
- 5:45drafting, but we can't offload the accountability.
- 5:52Now let's move into what causes hallucinations.
- 5:56We're going to talk about the practical and the
- 5:58technical roots. You'll notice throughout, I've
- 6:03numbered our practical causes at P1 through P5.
- 6:07and our technical causes T1 through T8. This
- 6:10is intentional because we'll come back to how
- 6:13these causes are mitigated or prevented when
- 6:17we look at our prompt playbook. So it'll be helpful
- 6:20when you have these slides in your hand to be
- 6:23able to refer back to the actual causes themselves
- 6:27and kind of see the thread that we are weaving
- 6:30throughout. Because ultimately, if you understand
- 6:33why hallucinations happens, this toolkit stops
- 6:36feeling like magic words you can say to stop
- 6:40it, but arms you with the understanding and the
- 6:43readiness to kind of pick up on things and just
- 6:47be more weary and ready to hold your tool and
- 6:51yourself more accountable with respect to the
- 6:53output. So let's dive into these practical causes.
- 6:59First, I just want to arm us with a working definition
- 7:02of what hallucinations are, and it's really AI
- 7:05-generated information that really sounds plausible,
- 7:08but is factually incorrect. And I think that
- 7:12that is the most dangerous part about it, is
- 7:16what you're reading doesn't seem like it's going
- 7:18to be wrong, but once it's pressure tested, you
- 7:20often find, or you are able to find, that there
- 7:23are holes in what you're reviewing in an output.
- 7:28So in terms of practical causes, the first one
- 7:32is using a vague prompt, because what models
- 7:36do is they fill the gaps in when they're not
- 7:40given the information. They're not great at asking
- 7:43you to fill those gaps for you, because most
- 7:45models aren't trained that way or set up that
- 7:48way. Next is where there's no source document
- 7:54for the model to pull information from specifically.
- 7:59Then in that instance, the model tends to answer
- 8:04from general patterns and not necessarily the
- 8:07facts you're giving it, right? So really being
- 8:11able to build that context and give some source
- 8:13information or have the source of information
- 8:16defined in some way assists in the more practical
- 8:20sense of ensuring your output is actually relying
- 8:23on the facts and not making them up. Practical
- 8:27cause number three. is we're not giving any actual
- 8:31constraints to the AI around inventing things
- 8:34that don't really exist or we're not in any of
- 8:38the fact patterns that were given to it. The
- 8:42one thing about AI is you do often times more
- 8:46than not need to be very explicit as to what
- 8:49it can and can't do. So saying something as simple
- 8:52as don't make this up, It's actually more beneficial
- 8:56than you would think in the context of prompting
- 8:59or mitigating or preventing hallucinations when
- 9:02you are prompting in your AI tool. Practical
- 9:08cause number four is Stacking too many asks into
- 9:12one prompt. This creates room for your AI model
- 9:16to drift, which means it'll pull information
- 9:18from different areas. It'll get distracted and
- 9:22maybe go down some rabbit holes that are not
- 9:24necessarily helpful for the output that you're
- 9:27looking for. And it's really good to think about
- 9:30this in the same context as the junior associate.
- 9:34It's a metaphor that often comes up. And in this
- 9:39specific case, when you give a junior associate
- 9:42too many asks stacked in one overall task, it's
- 9:46easy for them to spend too much time in one versus
- 9:50the other or not be able to really give you holistic
- 9:54answers, at least to what you're looking for.
- 9:58And so taking that same thinking into a prompt
- 10:01with too many asks, it's really helpful because
- 10:04you can pair some asks together. Use some as
- 10:08follow -up asks to make sure that you're getting
- 10:10the right focused answers in your initial prompt
- 10:15output. Last practical cause is, this one's really
- 10:21on us, right? It's when you're mistaking that
- 10:25fluent and really confident sounding output as
- 10:29something that's been verified to be accurate.
- 10:34AI is a smooth talker. And as we've mentioned,
- 10:38is really good at making incorrect or inaccurate
- 10:43answers sound really plausible because they word
- 10:46it so correctly and they tie it to where you're
- 10:49initially asking for it. So it's really about
- 10:52holding ourselves accountable to know that fluent
- 10:56doesn't equal verified and to make sure we're
- 11:00stress testing the model and the output to ensure
- 11:03that it is actually correct. Now we can move
- 11:10into some technical causes. So a lot of these
- 11:15technical causes are really driven towards the
- 11:20models and how they're created. They may not
- 11:24always be applicable to every model, but based
- 11:26on the research I've done and the tools that
- 11:28I've used, these are the ones that really stand
- 11:31out as being universally applicable. First technical
- 11:36cause for hallucinations is models generate text
- 11:41by predicting the next likeliest word. That's
- 11:44where when you hear people talking about models
- 11:46being predictive in their nature, that's really
- 11:48what they're trying to get at. And that means
- 11:51that there may not be factual retrieval unless
- 11:54you're actually grounding the model in a document
- 11:58or a search across a specific parameter of materials
- 12:04to look at. And that's often called retrieval
- 12:08augmented generation two, where you're focusing
- 12:11what your tool is looking at and able to retrieve
- 12:14information from versus focusing solely on its
- 12:17predictive nature. Second technical cause is
- 12:22AI models, they're trained to answer. They're
- 12:24not trained to assess their own confidence level
- 12:27before giving your answer. And so that's really
- 12:31important to keep in mind. Ask and you shall
- 12:35receive, but you may not always receive the most
- 12:39confident or accurate answer. Technical cause
- 12:43number three, I found this one really interesting
- 12:46was that a lot of like the benchmark evaluations
- 12:48that are used to train and score these models,
- 12:51they often reward a confident guess over an I
- 12:56don't know, which means if you're asking your
- 12:58tool something that plausible sounding guess
- 13:02is rewarded more than the I don't know or I don't
- 13:06have access to that information. So these are
- 13:10things to also keep in mind when you're evaluating
- 13:12tools as well and some things to kind of ask
- 13:15or challenge your provider to get a better understanding
- 13:18of how the models were trained and if this is
- 13:21something that you need to worry as a user about
- 13:24mitigating or preventing or if it's built into
- 13:26the tool. Technical costs number four is, since
- 13:32there's no reward for uncertainty, guessing for
- 13:37the AI tool is a default, not an occasional glitch.
- 13:43So it's important to keep these defensive prompting
- 13:46principles in mind as we get through them because
- 13:49until... The day comes where all of these are
- 13:54fully mitigated in model development and preventable.
- 13:58It's really going to be on us to make sure that
- 14:00we are doing our due diligence for our model
- 14:03outputs. We've got a few more technical causes.
- 14:10So technical cause number five, and this is probably
- 14:13the one most of you are going to be familiar
- 14:15with, is I liken this to the garbage in, garbage
- 14:24out saying, because we often say when we're talking
- 14:28about prompting is that the prompt is going to
- 14:32only, sorry, the output is only going to be as
- 14:36good as your input in your prompt, right? So
- 14:39if you give it a really vague or garbage -filled
- 14:44prompt, you can expect that in your output. And
- 14:48that same principle kind of goes towards the
- 14:50model quality because it depends on the training
- 14:52of the quality of the data that it's trained
- 14:56on. And that varies across tools, right? Especially
- 15:00when you're thinking of general purpose tools
- 15:03versus legal specific. It takes a different level
- 15:07of data training to qualify either one of those.
- 15:10And when you're working with tools that have
- 15:13open access to the internet, That means it has
- 15:16access to all of the public resources. And one
- 15:20thing we know about those is they're not always
- 15:22the best, not always the most accurate, so that
- 15:25uneven data training and access can also have
- 15:28an impact on the outputs that we get. Technical
- 15:33cause number six is, hallucinations inherently
- 15:39increase where underlying data is sparse, contradictory,
- 15:44or low quality. And one thing that I found is
- 15:48legal is pretty prone to this because there's
- 15:51going to be conflicting case law, superseded
- 15:53statutes, inconsistent forms that accomplish
- 15:56the same goal. So there's always there's going
- 16:00to be that little bit of an increase in hallucination
- 16:04likelihood because of that. So being more. Focused
- 16:08in our prompting is going to help mitigate and
- 16:12even prevent some of that and we'll get to more
- 16:14details when we get into the playbook there.
- 16:19Lastly, long documents and really long chats.
- 16:22are increase the likelihood of your tool losing
- 16:26some weight from the earlier details that you
- 16:29shared before, which would cause the output to
- 16:32drift. So keeping in mind, when you have the
- 16:35more tight prompts coming up at the beginning
- 16:40and are able to really focus where your prompting
- 16:43is going, it reduces the length of the tracks,
- 16:47right? and keeps your tool more focused so it's
- 16:50not losing the weight of the detail. And often
- 16:53keeping in mind that you may have to remind it
- 16:56of some things to keep in mind as you continue.
- 16:59Because I know as a user, sometimes I am guilty
- 17:02of having really long chats that are about the
- 17:07same. problem and wanting to not start a new
- 17:12chat because I'm like, but all my context is
- 17:14here, but keeping in mind that some of it may
- 17:17be lost if you're just spending so much time
- 17:20in the same chat and that you may have to remind
- 17:23it of it. So all of this is to say that when
- 17:27you're going to evaluate tools, whether it's
- 17:30legal specific or general purpose, your legal
- 17:33specific tools are going to be grounded and curated
- 17:35current legal sources and start from a different
- 17:38baseline. Which means that you may not have to
- 17:42put as much context or as many constraints into
- 17:46your tool when you're using legal specific versus
- 17:49general. It does impact the level of prompting
- 17:53that you do and where you need to guide the tool
- 17:55to go. So these are just important things to
- 17:58keep in mind when. evaluating a new tool or even
- 18:02evaluating how to use current tools that you
- 18:05have as we all know the struggle is budget in
- 18:08most cases so just trying to arm you with that
- 18:11additional information and context as you are
- 18:14using and evaluating and now we move to the last
- 18:20technical cause which I was kind of kind of thrilled
- 18:26to know that there was a name for this one in
- 18:29the Next slide, please. Thank you. The name for
- 18:36this one is AI Cinco Fancy. As I said, I was
- 18:43thrilled to find that there was a term for this
- 18:45because the way I generally talk about this when
- 18:47I'm talking about AI and its outputs is I think
- 18:53of AI as a golden retriever. It's so eager to
- 18:56please and so eager to validate what you're doing.
- 19:01You know, it's got the tail wagging, that silly
- 19:03golden retriever grin, and is really just looking
- 19:06to make you happy. The issue with that is we're
- 19:10not always right. Sometimes we don't need to
- 19:13be made happy. Sometimes we need to be told the
- 19:16reality of the situation. And so how that translates
- 19:20into prompting is if we ask a more, I'll call
- 19:24it leading question, AI isn't necessarily just
- 19:28looking to answer the question it's asking, it's
- 19:30looking to answer it in the way that we framed.
- 19:33So for asking, in this example here, whether
- 19:36a clause clearly violates a law, the assumption
- 19:40from the model is that it does clearly violate
- 19:43something. So I need to find how or why it violates
- 19:45it. or frame, however, my findings are to match
- 19:50the underlying ask. So practical fix here is,
- 19:53of course, just neutralize the question. Instead
- 19:56of leading it, right, we're going to say, does
- 19:59this comply with applicable law or with X law?
- 20:05And what is the strongest argument in each way?
- 20:07Because then you're getting both sides of the
- 20:10coin and you're able to then assess and evaluate
- 20:15um which side is the strongest in your opinion
- 20:19and press further versus getting something that
- 20:22just automatically agrees with you as much as
- 20:25we like to see that it's not often always what
- 20:28we need and this helps to prevent that surrender
- 20:32loop where you know the model agrees with you
- 20:35you agree with it nobody tests the idea and then
- 20:39we get tested by some kind of outside cause And
- 20:43so offloading is where we are keeping the dissenter
- 20:47in the room by going to that practical fix of
- 20:51neutralizing the question. So AI single fancy,
- 20:56a new term for you all. So we can move to the
- 21:01next one where we talk about, sorry, Linda, I'm
- 21:07going to answer your question. Is it best to
- 21:09start a new chat with the final output of a longer
- 21:11chat and ask it to review and evaluate the prior
- 21:13output? I think that's an absolutely great practice,
- 21:16because it does mitigate that T7 technical portion
- 21:22where your chats just get too long. So starting
- 21:25fresh often does help, and using that final output
- 21:29and building off of a new context window is a
- 21:32great solution. So I think that is a great best
- 21:36practice to keep in mind. Prompting and strategic
- 21:43communication. This I like to keep in all of
- 21:46my presentations because on its core, I really
- 21:48do feel like prompting is just strategic communication.
- 21:52It's another form of it. It's what we do every
- 21:54single day is just directing it to a new tool
- 21:58or mode, if you will. And the levers of this.
- 22:03strategic communication are the pragmatics, right?
- 22:06The context, intention, tone, and all of the
- 22:10social norms and things that are behind the literal
- 22:12words that we're using on a day -to -day basis.
- 22:15And when we're using them in a prompting context,
- 22:18we have to keep in mind that models can't, reliably
- 22:21at least, infer sarcasm, implied meaning, or
- 22:25even some kind of unspoken assumption. It can
- 22:28literally only rely on the words we're giving
- 22:30it. So keeping that in mind and building really
- 22:33strong prompts that leverage that pragmatic context
- 22:37explicitly is going to alleviate any sort of
- 22:41guesswork that our model has to do or any kind
- 22:43of gap filling that it has to do. And it's the
- 22:46same way we think about working with the junior
- 22:49associate. Give them as much explicit instruction
- 22:51as you can so they're not trying to fill in gaps
- 22:55or make assumptions because they may be too afraid
- 22:58to ask you for clarity. And building as much
- 23:01as you can, front loading it is going to be really
- 23:05helpful. And as you think about that and go back
- 23:09through the last few slides, you can be able
- 23:12to sort of see where pragmatic context missing
- 23:17could make things worse and where it can make
- 23:19things better. All right, now we can get into
- 23:25the Defensive Prompting Toolkit. I hope all of
- 23:28that background was really helpful as we lay
- 23:30the foundation for where we're going and kind
- 23:33of how to really think about this from a framework
- 23:36perspective and get a better understanding of
- 23:40what causes these hallucinations. And now we're
- 23:43going to move into five specific defensive prompting
- 23:47prompts, what each one catches and how to layer
- 23:52them. And before we get into the prompts themselves,
- 23:55we'll just talk about what risks we're here to
- 23:59mitigate and what defensive prompting is as a
- 24:02whole. So we can go on to our next slide. Perfect.
- 24:08So defensive prompting at its core is using prompts
- 24:11to stress test our AI output, not just to produce
- 24:15an AI output. Four risks that are the biggest
- 24:24ones with AI outputs are, of course, hallucination,
- 24:27which we've gone over, which is the plausible
- 24:29but inaccurate clauses, citations, and facts.
- 24:32Overstating confidence goes hand -in -hand with
- 24:35hallucination, right? It's uncertain conclusions
- 24:38with unwarranted certainty often can be misleading,
- 24:44could mimic hallucination or sit right next to
- 24:47it, which is Plain confusing, I will admit, so
- 24:51it's something to watch out for. Next is to misunderstand
- 24:56context. Applying inapplicable things when context
- 25:02is missing. So assuming the wrong side or deal
- 25:05posture in a review or negotiation cycle, there
- 25:12are often places where this will happen when
- 25:13you're not necessarily explicitly directing which
- 25:17side or deal posture AI should be taking. Then
- 25:21there's the displacement risk. It's where really
- 25:26the output looks really authoritative. This can
- 25:30be in the context of producing case language,
- 25:34case citations, or even citing back to certain
- 25:39laws and things, and where you're relying on
- 25:43that, but not leveraging the attorney review
- 25:47that's still required. It's really where I think
- 25:52displacement risk is just two words for surrender,
- 25:55right? It's where we're allowing the output to
- 25:58stand in for our judgment because it looks finished.
- 26:01Just because something looks finished doesn't
- 26:03mean it is. And each of these prompts that we
- 26:07go back to, that we're going to go through are
- 26:09going to help decipher, discern, and kind of
- 26:16attack all of these on their face. So what this
- 26:23means is that there are two lines of defense
- 26:26for hallucinations. There's mitigation and prevention.
- 26:30So we'll go to our next slide. The prompts that
- 26:35we're going to immediately go through are going
- 26:37to be a hallucination mitigation. They don't
- 26:41necessarily stop the hallucination from being
- 26:44generated, but they stop you from relying on
- 26:47it. So they are really pressure testing AI to
- 26:51go back and look at what it's done and help you
- 26:54flag any sort of hallucinations or issues that
- 26:58may be within the output that may not be readily
- 27:01available on their face to you. And a lot of
- 27:05a really good form of mitigation to keep in mind
- 27:08is data related mitigation. And it's where we're
- 27:12targeting hallucinations that are caused by bias
- 27:14and complete or conflicting underlying data.
- 27:18And as I mentioned, legal work is prone to this.
- 27:21right, with the conflicting case law, the superseded
- 27:25statutes, and things of that nature. So it's
- 27:29important to keep this in mind and have some
- 27:33available data resources for your tool to leverage
- 27:36or at least be able to guide it to the right
- 27:38things. So you can do this manually, which is
- 27:41what a lot of us do by saying look at this specific
- 27:44statute or look at this specific document that
- 27:48I'm uploading. But there are also ways to do
- 27:51this at the system level, which is retrieval
- 27:54augmented generation, which ensures that the
- 27:57questions that you're asking or the prompts that
- 28:00you're submitting to your tool, that any output
- 28:03that's generated is grounded in verified external
- 28:07sources. You can buy tools that are already geared
- 28:11to this. That's a lot of legal specific AI tools
- 28:15have retrieval augmented generation kind of built
- 28:18in at their core. You can also set up specific
- 28:23projects within AI tools or specific agents that
- 28:28are limited in their resources and what they
- 28:32pull from when you ask questions. So when we
- 28:35do this. Manually, it's just providing the links
- 28:39or the specific statutes or documents, and RAG
- 28:45automates this process. That's something to also
- 28:48keep in mind on a tool selection level and always
- 28:52worth mentioning given the never -ending data
- 28:55struggles we will be going through in our tool
- 28:59usage. Perfect. Now, we can get into our first
- 29:08defensive prompt here. which is assumption identification.
- 29:15At its core, it's to identify conclusions that
- 29:20rest on assumptions, inference, or business context
- 29:23information that the model was never actually
- 29:26given. So this prompt will specifically target
- 29:31the listed technical causes and practical causes.
- 29:35So technical would be the ungrounded generation,
- 29:39so when plausible information is missing, the
- 29:43tool will fill in the gaps itself with using
- 29:49its predictive nature or making just assumptions
- 29:52based on what it thinks it's going to answer
- 29:55your question the best. Our practical causes
- 29:59are because we didn't give a really grounded
- 30:03prompt with specific context and didn't leverage
- 30:07our our pragmatics that we talked about for strategic
- 30:11communication or maybe we didn't identify a source
- 30:14document or a specific source to review and so
- 30:18the model will reason from a pattern instead
- 30:22of the facts presented. So on the next slide
- 30:26we'll look at a prompt that is designed specifically
- 30:28to surface this. So you get your output right?
- 30:32Then you're going to say, well, identify any
- 30:34conclusions in your prior response that depend
- 30:36on assumptions, inferences, or missing business
- 30:39context. For each, explain what additional information
- 30:43would confirm or change the analysis. So that
- 30:47first part of this prompt, of course, is getting
- 30:49at that assumption identification. Then that
- 30:51second one, I like to include because it makes
- 30:55your tool do a little bit of extra work because
- 30:57it's going to tell you what information had it
- 31:02had that would confirm what it gave you or change
- 31:06what it gave you and then you can assess whether
- 31:08or not those facts or business context actually
- 31:12exist because sometimes in these cases where
- 31:16assumptions are made it's filling in gaps for
- 31:20what it expects to be there and sometimes it
- 31:23might be there the tool just might not have known
- 31:25it and then in cases where it's not it gives
- 31:28you something to think about in terms of Well,
- 31:32what happens if it is there or is that something
- 31:34for me to think about more from an analysis perspective,
- 31:37but ultimately it's keeping you in control and
- 31:41in the driver's seat as to what to rely on and
- 31:44what to challenge further. Next defensive prompt.
- 31:51This one is counter -argument. This one is great,
- 31:57especially if you're doing legal research or
- 32:00even negotiating a really difficult terms and
- 32:04you're looking to take a hard stance and you're
- 32:07looking to get some information as to the counter
- 32:10side of that, or even the counter side of how
- 32:13your analysis played out as you were interrogating
- 32:15it with the tool itself and the output it gave
- 32:18you. Because this forces the model to argue the
- 32:21other side of its analysis, which breaks the
- 32:24false certainty of one confident answer. Because
- 32:27as we know, these tools are not evaluated on
- 32:30how confident they are when they answer. But
- 32:32now we're forcing them to take that into account
- 32:35in the output that it gave us. So this goes to
- 32:41targeting our technical cost number two, which
- 32:44is that the model's trying to produce an answer
- 32:47and, like I said, not assess its confidence.
- 32:51And practical cost number five, which is where
- 32:55we are in the driver's seat and mistaking fluent
- 32:58confident answers for something that's been verified
- 33:01as accurate. So really forcing the AI to essentially
- 33:06flip the coin and take a look at the other side.
- 33:11So what this prompt actually looks like is, challenge
- 33:15your prior analysis, identify the strongest arguments
- 33:18supporting the opposing interpretation of the
- 33:21provisions you flagged. So you can change this
- 33:24to instead of provisions or this could be directed
- 33:27towards the law or whatever it is you're working
- 33:30for. But the key here is to challenge the prior
- 33:32analysis and identify the strongest opposing
- 33:35arguments. This separates what the model is,
- 33:40sorry, this separates what the model is confident
- 33:47about and what could poke holes in its assessment
- 33:54and it helps you better evaluate the accuracy
- 33:59and reliability of the initial output that it
- 34:02gave you. We all know we love a good counter
- 34:06-argument. This is great to use not just for
- 34:08challenging your tool, but also challenging yourself
- 34:12as well. Next, we'll move to our next defensive
- 34:19prompt. This one is confidence check. It is similar
- 34:22to the last one. but this one helps separate
- 34:27what the model is confident about from what depends
- 34:30on context, facts, or law it doesn't have. So
- 34:34it helps, of course, with that technical cause
- 34:36too, similar to the last prompt about how it's
- 34:40trained to produce an answer, not assess its
- 34:42confidence. But also it goes to that technical
- 34:47cause number four, which is because they don't
- 34:50have an evaluation system that rewards uncertainty,
- 34:55guessing is the default, right? And so it sounded
- 34:59so sure is in relying on that is exactly what
- 35:03surrendering your trust and accountability looks
- 35:06like. So this one converts this into a verification
- 35:11list kind of, and works to surface what was assumed
- 35:15and how reliable the rest is. So keeping both
- 35:19of those in mind when it comes to the output.
- 35:25So moving to, oh, Linda, I love that question
- 35:32about the prompts used every time or is it saved
- 35:34as a skill to be pulled from. Right now we're
- 35:37going through prompts that can be used every
- 35:39time, but we will move into what it looks like
- 35:44as a save skill and how to kind of up level there
- 35:47too. So this is the confidence check example.
- 35:52So the prompt is identify areas in your analysis
- 35:55where your conclusion may be uncertain or dependent
- 35:57on context facts or law not provided in this
- 36:00session. This one is great because the way I've
- 36:05seen this work in practice is there may be times
- 36:09where the tool identifies areas where the conclusions
- 36:12are uncertain and they may be dependent on some
- 36:17of the context or facts it was given. but they're
- 36:20not clear on how those contacts or facts really
- 36:23truly correlate to the conclusion it came to.
- 36:27So this helps you identify where that path between
- 36:30analysis and conclusion kind of goes a little
- 36:34wonky and isn't quite linear because, again,
- 36:38the conclusion can sound really strong. And even
- 36:41some of the analysis, but... getting a better
- 36:44understanding of where confidence truly existed
- 36:47and where it didn't is going to help you poke
- 36:49holes in it, right, and be able to evaluate whether
- 36:54this was truly. improving your efficiency or
- 36:57maybe it's giving you some things to question
- 36:59whether it's on the context you provided or how
- 37:02the context was interpreted. We often think we're
- 37:06the clearest communicators and this one can challenge
- 37:09both ourselves and the tool in upleveling the
- 37:14skill and analysis and keeping that trail between
- 37:18analysis and conclusion as linear as possible
- 37:22to avoid any things that have been made up or
- 37:26overstated with confidence. All right, so next
- 37:32prompt is going to be the hallucination check.
- 37:36And you'll notice a lot of these are very similar
- 37:40in what they are trying to identify, but they
- 37:43do target different technical or practical causes.
- 37:46So they're all good tools to have in our tool
- 37:48belt, right? So the source of this prompt is
- 37:53to ensure that every flagged risk is tied to
- 37:57specific language in your document or marked
- 38:00unsupported. And what this one goes to is our
- 38:04technical causes one and six, because by forcing
- 38:12the tool to go back to specific citation, or
- 38:19to your source material prevents that hallucination,
- 38:22right? Because it's creating something that they
- 38:25think it may be correct. But unless it's challenged
- 38:30to specifically cite, it won't. It could also
- 38:36sidestep some of that training by tying every
- 38:41conclusion to the document you provided rather
- 38:43than actually just relying on the models training,
- 38:47which is ideally what we want, right? We want
- 38:49to refocus the tool to focus on what we've given
- 38:54it versus on some of the data it's been trained
- 38:57on or kind of going off onto the tangent, which
- 39:01we would call a drift. Then this also from the
- 39:06practical perspective, we are able to, a lot
- 39:12of this happens again when no source document
- 39:14is tied to or provided to the tool when you're
- 39:18doing your prompting or when we're not giving
- 39:20a specific constraint against inventing or hallucinating
- 39:25in general. So sometimes I do like to just keep
- 39:28the do not make anything up if you have um if
- 39:32you aren't clear about anything or uh please
- 39:36flag that before answering keeping that at the
- 39:38end of some of my prompts or even some of my
- 39:40skills. is really helpful because it helps to
- 39:45prevent this before it happens and forces me
- 39:49to give better context at the outset versus waiting
- 39:53at the end of the final output to get it. And
- 39:58now we'll take a look at what the example looks
- 40:00like. This one is, confirm that each risk identified
- 40:06in your prior response is explicitly supported
- 40:09by language present in the contract provided.
- 40:11If any risk is conferred rather than directly
- 40:14supported, flag it clearly. So this is definitely
- 40:17geared towards the transactional practice, but
- 40:20you can see how the language can be tweaked to
- 40:23a research context or even a communication review
- 40:30context if you're leveraging your tool to. you
- 40:32know, take some very legally speaking and gear
- 40:36it towards a business counterpart or internal
- 40:39stakeholder. You can see how doing some language
- 40:43tweaks hold the tool accountable for each risk
- 40:47identified or each item identified really and
- 40:50tying it back explicitly to language or context
- 40:55or maybe even the exact language of the law.
- 40:59from where it's getting things. And if it's not
- 41:02getting it from it, it needs to say that clearly
- 41:05to you to help ensure, one, that you have confidence
- 41:08in what your tool is pulling, but also to help
- 41:11you understand better where more digging should
- 41:14be done so that you're not just relying on the
- 41:20solid output that's being given and really, really
- 41:24forcing it to check on the hallucinations. Judith,
- 41:29I love that. I think asking, do you have any
- 41:31questions is a great way for it to flag what
- 41:34it has. And because going back to something we
- 41:37talked about earlier in the presentation about
- 41:41the AI single fancy, right? Where the tool is
- 41:47eager to please. So when you ask something like,
- 41:50do you have any questions? That actually forces
- 41:53the tool to come up with questions and to then
- 41:57kind of second guess its confidence at attacking
- 42:00the actual ask. So it's a great question to ask
- 42:05the tool. So great point to share. All right,
- 42:10now we'll move to our last defensive prompt,
- 42:13which is the escalation prompt. This is kind
- 42:16of a catch -all pass at flagging what still needs
- 42:19independent attorney review. So this is a really
- 42:22good one, one for attorneys to use, but also
- 42:24to kind of arm more junior roles or legal ops
- 42:28roles with as well because it'll help them gauge
- 42:33on some of the tasks that they're armed with,
- 42:37whether and when things really need to be escalated
- 42:41to ensure that they're catching those things
- 42:44just like we would catch them, right? So some
- 42:47of the technical causes for when this would need
- 42:52to arise is, of course, those long conversations.
- 42:55that can cause the drift. And this will catch
- 42:59things that some of those other prompts may not
- 43:02fully catch. Then, again, the AI synchrofancy,
- 43:07where it's just really trying to please you and
- 43:11pleasing you through some flawed premises isn't
- 43:15what we're looking for. So this challenge is
- 43:17the tool to really think about that. And lastly,
- 43:22that practical cause, which is too many asks
- 43:24stacked into one prompt, then increases the room
- 43:28for a drift. So this prompt acts as like a safety
- 43:32net for that in catching those things. So let's
- 43:36take a look at what that prompt looks like. Okay.
- 43:41So identify any issues in your prior analysis
- 43:44that should be escalated for attorney review,
- 43:46then relied upon without independent verification.
- 43:49Flag anything involving jurisdiction specific.
- 43:53uh law regulatory compliance or provisions with
- 43:55material financial exposure so that that first
- 43:59one is going to um really catch a good amount
- 44:04of things right and its output we're giving it
- 44:07kind of a wider net to cast then that last sentence
- 44:10that's something specific to me and in my transactional
- 44:13um practice but it just demonstrates how you
- 44:16can even take that a little bit further to let
- 44:19it know what you're really focusing on in terms
- 44:22of the analysis that it was doing. So you can
- 44:25switch that up to flag any sort of things that
- 44:28you're really concerned about and then having
- 44:31a more holistic look at the escalation involved,
- 44:35right? And you can even swap out attorney review
- 44:39for, depending on the issue you're looking at,
- 44:42is this financial? Is this have to do with insurance?
- 44:46Is this something that maybe needed to be escalated
- 44:50to CEO, CCO, things like that. So you can really
- 44:55tweak the escalation level of this to help get
- 44:59a better understanding of issues in the analysis
- 45:03that need to be escalated to different internal
- 45:06stakeholders or maybe even above you and involve
- 45:10risk that those individuals who would be exposed
- 45:13to the escalation would really truly. care about.
- 45:21No, Marcella, I think that's a great, so I think
- 45:24both Marcella and Savannah, you're both on that
- 45:28track when it comes to AI law regarding disclosure
- 45:33of AI use, and that was just passed in California.
- 45:39While I am not licensed in California, I am vested
- 45:43in how this all plays out, and I think Since
- 45:47California is one of the first states, if not
- 45:49the first one to come out with some guidance
- 45:52on this, I'm curious about what it looks like
- 45:54on a national level. And Marcel, I think it is,
- 45:57I mean, it truly is on my list to do a little
- 46:00bit more digging and that analysis of how it
- 46:04affects practice at every level because. Without
- 46:08guidance that is national and sweeping and applicable
- 46:12to all attorneys, it's tough to know where this
- 46:16is actually going to be going and affecting practice
- 46:18across the nation. And as AI disposures are becoming
- 46:22more prevalent, especially when it comes in the
- 46:25marketing context and communication context and
- 46:29privacy laws, I think there may be a more lax
- 46:38approach nationwide as people understand the
- 46:41impacts and use of AI as to how they go about
- 46:45disclosures. So this is a long -winded way of
- 46:48saying to be determined on my end where I actually
- 46:53come out on this one and what it actually looks
- 46:57like on a national sweeping level, but absolutely
- 47:01agree on wanting to stay tapped into what that
- 47:04full analysis looks like. All right, so moving
- 47:10into how to go about layering some of these defensive
- 47:15prompts on top of one another. I just mapped
- 47:19out some quick, like, layering scenarios that
- 47:22you can think about so that you understand, you
- 47:25know, some of the work that you may be doing
- 47:27may require multiple of those prompts. May only
- 47:31require one, but this is just a way to look at
- 47:33how to layer them, and just pick the situations
- 47:37to go about layering them for. So like a quick
- 47:40red line check. I say use that hallucination
- 47:43check. Use that quick hallucination check to
- 47:49see what, to make sure it's using the information
- 47:52that is right within the document, right, that
- 47:55you're looking at and not pulling from scenarios
- 47:58that aren't actually present on your desk right
- 48:00now. And the escalation. Novel legal questions.
- 48:05Yeah. Let's look at the counter argument and
- 48:07let's also look at the confidence check and escalation.
- 48:10Maybe we're not the ultimate judgment maker on
- 48:13whether or not or how this law truly impacts
- 48:18us or whether or not we need to make some operational
- 48:20changes or are going to feasibly. Maybe that
- 48:24impacts how we come out on the legal question
- 48:27or being able to include it in any of our terms
- 48:30or documents moving forward. And then there's
- 48:33the client -facing memo. Assumption identification
- 48:36is a great one. Then, of course, hallucination
- 48:38check and escalation. So when you've got that
- 48:42external reader who is invested and you want
- 48:45to make sure no assumptions, hallucinations,
- 48:47or things haven't been escalated that need to
- 48:50be. All right. Yeah, thank you, Anna. I really
- 49:00appreciate that. So slide 25. I know we didn't
- 49:05talk about offensive prompts in this session,
- 49:08which are the prompts that are going to be leveraged
- 49:10to really get work done. And we talked about
- 49:16those pretty extensively in my last presentation
- 49:19for BHBA. And so I wanted to just take a second
- 49:23and show you guys that offensive and defensive
- 49:27prompts can be layered together, because that's
- 49:30essentially what's going to happen in practice.
- 49:32You've got your offensive prompt, which is where
- 49:36you're going to identify all assignment changes,
- 49:38control consents of signed provisions, determine
- 49:42whether the counterparty consent is acquired,
- 49:44summarize transaction risk, and produce an executive
- 49:47-facing diligent summary table. That is all one
- 49:50prompt. I just broke it down into sections so
- 49:52you can see the different portions of the prompt
- 49:55itself. Then once you get your output, you can
- 49:58layer on that hallucination chart. and confirm
- 50:02that each provision in that diligence table they
- 50:05provided is tied to specific language in the
- 50:08agreement it references, and flag anything that's
- 50:10not directly supported. Then the escalation,
- 50:15right? Identify, flag provisions that involve
- 50:18jurisdiction -specific law or material financial
- 50:21exposure that should be escalated before the
- 50:25summary is relied upon. And again, you can tweak
- 50:28these to the flavor of what you're looking for,
- 50:30but this is just to show you how to layer the
- 50:35two on top of each other and what this essentially
- 50:37will look like in practice as you're going through.
- 50:40And you're doing that offensive prompting to
- 50:42get a result and how you can then get defensive
- 50:45once you've got that output to do the hallucination
- 50:49tracks or escalations. And again, I think there
- 50:53was a question before. or just a point about
- 50:57switching to a new chat window. And this could
- 51:00be a great way to do it right. Have one window
- 51:02for your offensive portion, and then you can
- 51:05start a new defensive thread as well. All right,
- 51:13so now let's shift from catching the problems
- 51:17or mitigating them to now preventing them. And
- 51:21this is where skills and memory come into play.
- 51:25So. AI skills are saved reusable instruction
- 51:29sets that your tool applies automatically to
- 51:32a given type of task. So most tools these days
- 51:35have an ability for you to save some skills in
- 51:39the background. And I will say that Claude does
- 51:44have a great open source legal skills library
- 51:47to take a look at. So you can kind of see how
- 51:51skills are written and structured. But these
- 51:54are great ways to embed some of what we've been
- 51:58talking about to specific tasks that you do.
- 52:02So if you had like, let's say an NDA review skill,
- 52:07you can within that skill embed some of these
- 52:10defensive prompting principles, like the assumption
- 52:13identification and things like that. Then there's
- 52:17also AI memory. And this is where This is giving
- 52:22your tool persistent context that's going to
- 52:25be retained across all sessions used. So there
- 52:29are standing constraints that don't need to be
- 52:31restated every time. This is you can definitely
- 52:34translate a lot of these defensive prompts into.
- 52:37memory so that they're applied across all the
- 52:40work that you do within your tool, which takes
- 52:43them from being purely mitigation to now being
- 52:46more preventative and giving your tool some standing
- 52:50constraints to operate within. I like to use
- 52:54both of these in tandem with my defensive prompting
- 52:59too because sometimes I'm While I can know these
- 53:04things are being done in the background, sometimes
- 53:05I do just still want to review that table at
- 53:08the end of the day to really hold the tool accountable
- 53:11to giving me that kind of more linear analysis.
- 53:15But knowing that I'm doing the work to be more
- 53:17preventative helps my outputs kind of start knowing
- 53:23that's what I'm looking for and giving that to
- 53:25me in a more consistent fashion than instead
- 53:28of when I'm just asking for it. I have to use
- 53:35an ANA tool for a long time. I think that depends
- 53:41on the tool you're using and the kind of credit
- 53:47-based credit or not credit -based in memory
- 53:53-based situation that they have. But what I find
- 53:56is I will go back through and I will audit my
- 54:00memory and my skills on a relatively regular
- 54:05basis. I think right now I'm probably at a monthly
- 54:09review, because what you'll find is, especially
- 54:13if your tool will show you the specific memories
- 54:16that are logged, is there will be a lot of overlap
- 54:19between the memories, and there can be a lot
- 54:22of consolidation that you can do. You can do
- 54:24that consolidation manually. You can even leverage
- 54:27the tool to help you do that consolidation. And
- 54:30then I'll also find that some of my skills are
- 54:33doing a lot of the same thing. So how can I consolidate
- 54:36them to have one skill or maybe I'm making my
- 54:43skill too complicated and maybe it needs to be
- 54:46separated, things like that. So definitely holding
- 54:49ourselves accountable to doing kind of an audit
- 54:51of it is helpful in mitigating any slowdowns
- 54:55or extensive credit usage we may run into when
- 54:58we have too many things loaded into the system.
- 55:02um this is something we've really already talked
- 55:05about which is um skills and memory being that
- 55:09more preventative layer versus that mitigation
- 55:12layer um so we can move to the next slide for
- 55:14the sake of time because i know i always cut
- 55:17these close um all right so um building a defensive
- 55:24skill this is where we're like this a specific
- 55:30example is we're going to build a skill that
- 55:34is safe to fire on a contract review task, right?
- 55:37So we're uploading a document and we can type
- 55:40a plain request that says review this for risk.
- 55:45But when we say that review this for risk, what's
- 55:47happening behind the scenes is it's running that
- 55:50standing rule that we've built. And so the standing
- 55:52rule right here is there will be no fabrication
- 55:57of citations or contract revisions. flag anything
- 56:00not directly supported by the source, state uncertainty
- 56:03explicitly rather than guessing, and before finalizing
- 56:07any risk analysis, run the hallucination check
- 56:10and escalation prompt automatically. So these
- 56:13standing rules, it's not the entire skill, right?
- 56:17Sorry, it is the entire defensive skill, but
- 56:20it can be tagged on to any other skill, right?
- 56:25But when you have this in here, the standing
- 56:28rules your tool knows to run at every time. And
- 56:32it's going to distinguish genuine risk from missing
- 56:34clauses. And it's going to help based off of
- 56:37the hallucination check and escalation prompt
- 56:39route specific high exposure questions to the
- 56:43right team to review and catch anything that's
- 56:46been made up. Yeah, you know, Mary Bird, I think
- 56:55that's a great one. Yeah, I would love to do
- 56:58something where I am running through all of these
- 57:02in an actual tool example and showing you what
- 57:07it looks like in practice versus talking about,
- 57:09I think, this great call out. So this breaks
- 57:14down the standing rules and what happens behind
- 57:17the scenes when you're using them. And we can
- 57:20move to the next one. And this is just to go
- 57:28over using memory, right? So memory should hold
- 57:33your standing preferences and constraints and
- 57:36uncontested conclusions, the AI, not uncontested
- 57:40conclusions, right? So it's an efficiency tool.
- 57:44It's not going to be a substitute for everything
- 57:47that we do, but if there are things. standard
- 57:50operating procedures for your legal team, specific
- 57:54things you look for in every output that you
- 57:56review, certain assumptions you are okay with
- 58:01the tool and making across outputs that you're
- 58:04asking for. This is a great place for using memory,
- 58:10right? It's where can you build in that context
- 58:12from the get -go without having that context
- 58:16or any kind of pragmatics built in on the memory
- 58:20level without having to restate them every single
- 58:24time that you're going to build a prompt, whether
- 58:28that's offensive or defensive. And I totally
- 58:32forgot that I included the periodic audit here.
- 58:35I think that's amazing I did that. I totally
- 58:38forgot I did. But definitely want to periodically
- 58:42audit what's safe to your memory because when
- 58:45you have too many things that are in there that
- 58:47are overlapping or no longer relevant, it will
- 58:51compound errors across future sessions or just
- 58:55make some of the outputs a little bit harder
- 59:00to discern. So if you ever notice that you've
- 59:02got a lot of memory and you haven't audited and
- 59:04you're starting to see that your outputs are
- 59:06becoming a little bit more frustrating, that
- 59:08is a big key to go and review your review and
- 59:14audit what you have saved to your memory. When
- 59:22it comes to concerns with giving tools, client
- 59:25documents, I think that really depends on the
- 59:28nature of the tool you're using. Like, is it
- 59:32enterprise level? Is it not enterprise level?
- 59:36And definitely conversation to have with your
- 59:38IT security team to make sure data is staying.
- 59:43where it needs to stay in order for you to rely
- 59:46on using it and kind of making sure that what
- 59:50you upload to the tool in your session stays
- 59:52within your sessions and is not shared across
- 59:55other sessions is a really great thing to make
- 59:58sure you and your data science and IT security
- 1:00:02team are aligned on before uploading client documents.
- 1:00:10So, Linda, in terms of memory, when you go into
- 1:00:14the settings of most of your tools, they should
- 1:00:16have something that is titled or similarly titled
- 1:00:21to memory, which is where you will see a list
- 1:00:25of things that it has saved to its memory with
- 1:00:28respect to the work you do with it. If you're
- 1:00:31not able to find this in your settings, I'd love
- 1:00:35to just ask my tool, what have you saved to memory
- 1:00:39about the way that you and I? work together and
- 1:00:42it'll give you a list of things and you can then
- 1:00:44go through and audit that list and kind of have
- 1:00:48a conversation with the tool about what that
- 1:00:50looks like. But I know a lot of tools are moving
- 1:00:54towards having that dedicated memory space within
- 1:00:59settings for you to review specifically. I know
- 1:01:05we're over time but key takeaways. hallucinations
- 1:01:10isn't just a glitch. It's the default behavior
- 1:01:13of the technology, right? And the way we can
- 1:01:16get around that is prompting. It's the discipline
- 1:01:18we can control. And that's what I like to focus
- 1:01:21on in all of my presentations is focusing on
- 1:01:27what we can control as users, as practitioners,
- 1:01:31whether that's in the prompt itself, if it's
- 1:01:35evaluating the prompt afterwards or what we can
- 1:01:38do in a more preventative measure. Just focusing
- 1:01:40on the things we can control and understanding
- 1:01:43the things we can't, which is what we spent our
- 1:01:45time in the first part of this session doing.
- 1:01:51Defensive prompting is quality control. It's
- 1:01:54not distrust of the tool or any of the tools
- 1:01:56or AI generally. It's just an additional layer
- 1:02:00of how to prompt and use the tools more efficiently
- 1:02:04and it builds up our confidence, which then in
- 1:02:08part improves the confidence in the work product
- 1:02:11that we're using our AI tools to help generate.
- 1:02:15Once you've really honed in on this defensive
- 1:02:18prompting practice, it's a good time to then
- 1:02:21move into skills and memory. I highly encourage
- 1:02:24everyone to interrogate the tools they're using
- 1:02:26today to see what they can. put into the back
- 1:02:30end, right? Whether you can impart some skills
- 1:02:33and memory into your tool. And if not, think
- 1:02:38about tools where you can do that or how you
- 1:02:41can maybe work with your tool to finagle something
- 1:02:44that helps you get to something adjacent to skills
- 1:02:49and memory. Lastly, we ultimately remain accountable
- 1:02:53for the work product, regardless of what produced
- 1:02:55the first draft. And as we saw in some of the
- 1:02:59comments and questions around AI disclosures,
- 1:03:02I think there are going to be more questions
- 1:03:06and rules and things being put out and stripped
- 1:03:10back around what this looks like. Being able
- 1:03:14to start from a place where you are accountable
- 1:03:17for the work product and understand how to challenge
- 1:03:20it and leverage the tool to your best of abilities,
- 1:03:23it's going to set you up for success with any
- 1:03:25kind of guidelines that come down the line with
- 1:03:27respect to disclosures or limits on use. No matter
- 1:03:31how long lasting or fleeting they may be, time
- 1:03:34will tell on that. David, I'm a big fan of Claude.
- 1:03:42I really am. I'm a big fan of Claude and recommend
- 1:03:47that as a great place to start. Thank you all
- 1:03:52for your questions. I know we're over on time.
- 1:03:56Please, if we go to the next slide. Thank you
- 1:04:01so much. I'm happy to keep this conversation
- 1:04:03going. I'm just mindful of everyone's time. You
- 1:04:06can scan this QR code to connect with me on LinkedIn
- 1:04:08or just find me on LinkedIn. Every time I do
- 1:04:12one of these, I love the questions that come
- 1:04:13up. in my messages and I do my best to get back
- 1:04:17to you as quickly as possible. But big thank
- 1:04:20you to all of you for attending and big thank
- 1:04:22you to the HBA for having me back again. This
- 1:04:25was great.