Latest / Coopers' Code / Human Instinct vs. AI: What do Berkeley and Cornell Say? This Month in Legal AI

Transcript

Human Instinct vs. AI: What do Berkeley and Cornell Say? This Month in Legal AI · Coopers' Code

0:00
Hello and welcome to another episode of Cooper's Code.
0:03
This is our occasional episode where we talk about AII say
0:08
occasional because it's typically monthly and because AI
0:11
has been moving so fast, we try to get all the smart minds
0:14
together to talk about things with Trod, Millenfar, Marshall,
0:17
Pole. And today we are joined by
0:19
special guest Spencer Palki. Spencer is a partner at Walk Up.
0:23
Spencer also has a niche focus on mock trials with his own
0:28
podcast and teaching and developing future lawyers.
0:32
Spencer, since everyone knows a bit about Marshall and Sherrod,
0:36
you want to tell us a little bit about you, your podcast, your
0:38
focus, so that people understand a little bit more about you as
0:42
we dive into this episode. Sure, plaintiffs trial lawyer
0:46
walk up have been for 18 years. Love that.
0:49
But on the side, a very large amount of my time is spent
0:52
coaching, teaching trial advocacy at the law school
0:55
level. I Co direct Berkeley Law's trial
0:57
competition program and one of the various things I've done in
1:00
recent years is Co host a podcast on law school trial
1:03
advocacy called Unscripted Direct with my friend Justin
1:06
Bernstein. It's a ton of fun.
1:07
Lot of nerds. And we have you here.
1:11
I think we're all, all nerds if we're going to be on, on these
1:14
subjects, We have you here in part because of some work that
1:18
you've done related to AI and using it in productive ways with
1:25
mock trials of other pieces. And, and so with that kind of
1:27
as, as a high level platform, I'd, I'd love for you to kind of
1:30
kick us off in terms of some of the, the research and, and other
1:34
work you've done so that we can digest it and, and, and see how
1:37
it may play out for people. I mean, of course I use it in my
1:40
practice etcetera, but I wanted to have an opportunity to see
1:45
how it would be used in a trial advocacy setting.
1:48
And every year, Berkeley Law along with Walk Up and other
1:52
sponsors host the national Flash trial competition.
1:55
Our fourth of those 4th annual competition was this was this
1:59
fall here in San Francisco. And that was like a perfect
2:00
opportunity to test out AI. How did you test it?
2:05
So we have 10 teams in the competition.
2:07
I should say that the competition's format is it's
2:09
called flash trials for a reason.
2:11
So in each round you get 45 to 60 minutes depending on if it's
2:15
a one-on-one or A2 on two competition to prepare.
2:19
So the students go from knowing nothing about the case at at
2:23
minute 0 to trying the case 45 or 60 minutes later.
2:28
Very useful trial training method that we've used at
2:32
Berkeley for many years, I will say.
2:33
But it presented a special challenge as generative AI
2:38
became more and more, you know, accessible, available, because
2:42
we had to think about, well, what's our rule going to be for
2:44
how we use AI in the competition?
2:47
And did you come up with a a way of handling things?
2:51
We did you know the question was on the spectrum, where do we
2:54
want to be? Outlaw it or or let it be used
2:57
because for better or worse, I'm actually quite agnostic about it
3:01
from like a moralistic perspective.
3:03
But for better or worse, it's here and it is only going to
3:06
become a larger part of practice.
3:07
We decided to not only say you can use AI more or less in any
3:11
way that you want to, to prepare.
3:12
We also teamed up with Eve Legal to give everybody, you know,
3:17
full enterprise access to the platform, with the condition
3:21
being that we'd be able to look at how they used it on the back
3:23
end. As far as that's concerned, can
3:27
you give us an insight into, I know there are a variety of
3:29
products out there and we're not promoting one over another.
3:32
Eve has not done a product placement.
3:35
However, if Eve is making a tool available for your trial teams,
3:38
that's that's very generous of them.
3:39
So perhaps you can tell us a little bit about what Eve is
3:42
relative to other products out there?
3:44
Yeah, and candidly, I also don't use Eve in my practice.
3:49
I use other AI platforms. However, a a law school friend
3:53
of mine is, you know, in the higher ups or among the higher
3:58
ups there, and I reached out to him.
4:00
It is, I understand it is an AI platform that is focused on
4:03
plaintiff side personal injury cases.
4:06
I used it as part of the competition and, you know, I was
4:08
impressed with it. But you know, they were a great
4:10
partner to share for free access to their platform to all these
4:14
students. The next question of course is
4:17
what's the back end meaning you that you partnered with them,
4:20
but the exchange was that you got to see what was going on in
4:23
the back and then and this is where I think we might start
4:26
seeing some of the the utility and other pieces.
4:29
Yeah. So I mean, the deal with the the
4:31
law students was like, you know, you can use AI really more or
4:35
less in any way you want. Can't use it during the trials,
4:37
but you can use it in prep. But we're going to look at what
4:42
you did just to get a sense of how you used it and that kind of
4:45
thing. And so I do love math, but I am
4:48
no specialist in math. And so we have some folks
4:50
helping us crunch the numbers. And it's not completely
4:52
crunched, but we have some sort of, you know, preliminary
4:56
insights into how it was used and that kind of thing in the
4:59
competition. Looking for the big reveal and
5:02
what types of insights have you have you gotten?
5:05
I would say in, in, in an overarching respect, it was,
5:09
thank goodness, at least at this juncture, not life altering.
5:13
You know, I can go through my list of I guess I've got five
5:15
points here that are I think worth talking about.
5:18
But the most heartening thing was it didn't appreciably change
5:22
what was being done at the competition as compared to prior
5:26
years. And that was, again, like I
5:29
said, heartening because I am agnostic about it.
5:31
It scares me and it also excites me, but to have the students
5:37
still do what they're supposed to do, you know, advocate on
5:39
their own, think on their own, even though they had access to
5:42
the AI, was I think, a good sign.
5:44
I'm bouncing around a little bit here, but going back to E for a
5:47
second. I'm curious, Marshall Sherrod,
5:51
have either of you played around with that relative to any of the
5:53
other tools? Any any thoughts on that
5:56
particular product versus other products?
5:59
I haven't played with that particular particular product,
6:03
but I have seen it demonstrated before and it does seem to have
6:07
some great utility in terms of what it can do in the
6:10
functionality. There are other products that
6:13
are out there and even our practice management system, they
6:16
just released AAI module that they want to integrate that's a
6:21
more advanced AI module that they want to integrate into our
6:23
practice management. So it is something that there is
6:27
now more competition in the industry.
6:31
And I think a lot of the lower level companies are getting
6:35
weeded out. And now there's going to be some
6:38
companies that are becoming more emergent in terms of their
6:42
capabilities and their abilities based on the amount of money
6:45
that they have pumped into them to provide a product that's more
6:50
in line with what we would need. And part of the reason I'm
6:53
taking this tangent is because I know our listeners have kind of
6:56
a wide variety from highly technical to want to just, you
7:00
know, tread water and keep up with what's going on.
7:03
A product like Eve, how does that differ from say, Westlaw AI
7:07
or Claude Jet, GBT, Harvey, the the, you know, kind of the, the
7:11
very high end legal piece. Any insight on on where this
7:15
fits in in the in the pantheon of product?
7:19
Glad to give you what I understand the the main
7:21
difference to be. I mean, so, and it's so
7:23
interesting Shroud, like you say.
7:24
I mean, this is definitely survival of the fittest sort of
7:27
space right now, right? I mean, and, and sadly, the
7:29
fittest is the richest because they have the most wherewithal
7:32
to stick around, right. So query what that means in a
7:35
year or, or you know, 18 months about who's still, you know,
7:38
what's the tide has gone out, you know, who is swimming with
7:40
what amount of clothing on. But Eve's pitch is it's personal
7:44
injury focused. And there and by that they mean
7:47
they have a whole series of playbooks.
7:51
They call them sort of pre designed prompts that are meant
7:57
to help you throughout a case. And again, I'm not, I'm not like
8:00
advocating for them or against. I'm just sort of saying what
8:02
what I learned in working with them.
8:04
And I was impressed by the playbooks.
8:05
In fact, we, we turned them off for purposes of the competition
8:08
because we didn't want, you know, we didn't want to have an
8:10
uneven playing field in that respect.
8:12
So students and coaches couldn't use the playbooks.
8:16
But I think that's the thing. They've, they've given some
8:18
thought to what it looks like to run a personal injury case and,
8:20
you know, created some, I think playing field leveling
8:23
opportunities for real lawyers by having like, here are your
8:25
best prompts, that kind of thing.
8:27
So that's, that's my understanding of its main
8:29
difference. Whereas a Harvey or you know,
8:31
Harvey, we do use that at our firm, much more varied and you
8:34
know, it's used not just by lawyers course you could create
8:36
your own, write books there too, but sort of more generalized I
8:40
would say than what Eve's doing. I know before we started,
8:43
Marshall was talking a little bit about how sometimes he'll
8:45
he'll queue up a clawed window so that live in Depo as people
8:50
are throwing things out and he's able to throw stuff back.
8:53
I should let you speak to that Marshall.
8:55
I'll, I'll return to my question for Spencer, but, but here's a
8:58
tangent on that piece. Yeah, yeah.
9:00
And and Spencer listening to you start out and and I'm very
9:05
curious to see the final results of how the students used AI
9:10
because they're not. I'll, I'll use the word jaded,
9:14
but it probably not the right word.
9:16
But you know, as having been a lawyer for a lot of years, you
9:20
get to see different things. You've deposed people in real
9:23
life, right? Like you, you can see you're not
9:26
just giving a script of, of here's what it is.
9:28
And then how can you, how you want to use AI?
9:31
How are you thinking about using it?
9:34
And so I was really interested to see how creative they are
9:38
because it's interesting as being a lawyer and that how we
9:42
use it and in depositions and leading up to depositions,
9:45
especially with experts. And I probably mentioned it
9:47
before, I think it's really helpful to help you prepare to
9:51
depose an expert. Like I've got a real estate case
9:54
going on I was talking about and I was throwing in a bunch of pro
9:59
formas with respect to various properties and dealing with cap
10:02
rates and trying to see variances with cap rates.
10:05
And obviously, if you just plug and play numbers, your cap rate
10:08
in a real estate transaction can dramatically differ.
10:12
And I was using Claude with respect to inputting all this
10:16
data. And it was really, really
10:18
helpful for me to teach me about more about cap rates and rent
10:23
rolls and how the various documents that I was using and
10:28
we're going to be used in this case were skewed in certain ways
10:32
depending on various factors that this expert was using.
10:36
And so it's going to be really helpful for me and obviously to
10:39
be verified with our own expert. But certainly from a cost
10:42
perspective to enable me to get all of this information and Army
10:48
with great questions to ask without having to have consulted
10:51
an expert right now is great. And certainly when the time
10:55
comes, I'll loop he or she in and talk about all the things
10:59
that I've already done and confirm the accuracy or not.
11:02
But because what I'm doing isn't necessarily I'm not seeking like
11:05
answers to these questions, I'm just trying to understand data
11:09
that the variances between an 8 cap and A7 cap 6 cap property
11:14
are helpful for me to be able to ask questions for this expert to
11:19
understand whether they've considered all of these factors.
11:21
And so I've just, I, this was in the last week and I've loved it.
11:24
I've spent a lot of time dealing with it and it's been very
11:28
helpful for me. It seems to me that as a
11:31
consumer that who is constantly bombarded with five emails a day
11:36
from different AI vendors and other vendors, we need to be two
11:40
things in our approach, curious and suspicious.
11:45
Because what I found is that a lot of times the vendors and
11:50
they're specifically their salespeople will tell you very,
11:55
very lofty things that will not pan out when you're actually
12:00
using that product. And that's where we have to
12:03
understand that ultimately these products can help us do great
12:10
work for our clients, but there is work that we have to put in
12:14
it initially in order to get those results.
12:17
And it's not simply, hey, it's a plug and play jump in and and
12:21
you're done. You have to learn how to use it.
12:23
And sometimes we have to know which system is the best system
12:27
to go to for the particular question that we have.
12:30
And those are important considerations when you're
12:33
looking at it from a big picture standpoint.
12:35
And I think curious suspicion and and the over promising it is
12:39
definitely something to be aware of and the survival of the
12:43
fittest because we were looking at products a year ago who were
12:51
promising some pretty amazing things.
12:53
And some of those people have already been slimmed down that
12:57
they're, there's one entity in particular I'm thinking of that
13:00
that no longer really even talks about their AI wrap on their
13:03
product anymore, whereas others have have continued to
13:05
accelerate. So I'm bringing this back to
13:08
you, Spencer, because part of the reason I, I did this
13:10
divergent is is it only Eve that the flash team was able to use
13:14
or was it a look you you, it's just like you're in a courtroom
13:19
and you have access to whatever tool you need.
13:22
It was, and there's some honor code stuff happening here,
13:24
right? But I mean, the rule was that
13:26
for using generative AI, it would only be EVE that would be
13:29
allowed because we wanted to be able to look at it on the back
13:32
end, not because it's, you know, better or worse.
13:34
In fact, it's just built on, you know, it's using other, you
13:36
know, engines, but we wanted to be able to look at it at the
13:38
back end. So it it was only Eve for the
13:41
competition. And I apologize, I, I, I warned
13:45
you could just kind of pack in a few things along the way.
13:48
And, and I feel like I interrupted the flow in terms
13:51
of, of the learning that you got from it.
13:53
So I, I know you had a number of points that, that you, you had
13:56
from the back end of, of, of even terms of what they were
13:59
doing, what the students were doing.
14:01
Yes, but I'm just delighting being interviewed by a podcast
14:04
interviewer, having sat in that chair so many times and thought
14:07
about the different tangents that are happening and whether
14:09
you should go down this route or not.
14:10
It's interesting, but nobody doesn't say too much about it.
14:13
Feel your pain. Perhaps we should have another
14:16
like a a a meta podcast where we do a podcast podcast.
14:20
How's that That's. A great idea.
14:22
Yeah, back to you, Spencer, in terms of of the takeaways.
14:27
So first, people did not wholesale tell AI to do all the
14:32
work for them. Second, teams most often use the
14:36
tool for brainstorming. Third, almost all, but not all
14:41
nine of ten teams used AI to some extent. 4th, there was an
14:46
inverse correlation between the ratio of unique prompts to
14:50
follow-ups as as it relates to overall success in the
14:52
tournament. I can explain that more.
14:54
And then 5th. For what it's worth, the team
14:57
that won the tournament also had the most sophisticated AI
14:59
approach. OK.
15:01
Can you unpack numbers four and five?
15:03
Since you said in particular with #4, you could explain that.
15:06
Yeah, I guess my point is that looking at people's use of the
15:10
AI platform there, we we categorize their prompts in two
15:15
categories. 1 is, is it a unique prompt?
15:17
Is it like a new topic basically or is it a follow up as part of
15:21
a prior topic and the teams that did the best.
15:24
And again, this is like, you know, a very overbroad and I got
15:27
to say it's only 23 trials that this happened in and you know,
15:30
it's it's not overall representative and I'm not a
15:32
statistician and etcetera. But point being, the team teams
15:36
that had a unique prompt and then dug down within that prompt
15:40
tended to do better than teams that had lots of unique prompts
15:44
and didn't dig down, which, you know, makes a lot of intuitive
15:46
sense because it's sort of more scatterbrain that that second
15:49
approach. It's like this, that this, that
15:51
the first approach where it's like, here is my unique prompt
15:54
and then my follow-ups is much more, in my view, a part of
15:57
being having a plan. Here's I'm going to use it.
16:00
Here's what my primary prompt is.
16:02
And that's again only by correlation tended to correlate
16:06
with more success in the tournament.
16:08
What about #5? For what it's worth, I say
16:11
again, because of the limitations of the number,
16:13
amount of data and all that stuff, the team that won spent a
16:16
considerable amount of time prior to the competition honing
16:21
a primary prompt. And we have got a, we've got a
16:25
bunch of, I should say flash trial problems are 20 to 30
16:28
pages long each of them. We've got a whole bunch of them
16:29
accessible to students to look at what it's like to practice
16:32
and prepare. They went through a lot of those
16:35
problems and used the prompt and then changed the prompt, used
16:39
Eve to improve the prompt. And once they got to the
16:42
competition, they had their sort of primary prompt.
16:45
I think they're the only team that had like here is our prompt
16:48
and they used it right away when the problems dropped each time.
16:51
And I again by correlation, that seemed to have served them well
16:55
because, you know, they sort of knew what they were asking for.
16:58
They got it. And you know, you could
17:00
certainly tell a story when that makes sense to me that having
17:03
done that, they were ready to jump into the trial, jump into
17:05
their prep, because they sort of had a second layer of here's the
17:08
fact pattern. Here's our here's our analyzed
17:11
understanding of it. Now we can jump off and get our
17:13
theme and get our theory and all that stuff.
17:15
So that's that's number five for you.
17:17
I. Recognize the the question I'm
17:20
asking is more anecdata than data data.
17:24
Any perception on whether the teams that did better with these
17:28
tools were teams that were more likely to be able to embrace
17:33
technology? More engineering, IP lawyer
17:36
types versus that's interesting. Are you know liberal arts types?
17:41
That is a fascinating question which I do not know the answer
17:44
to. I apologize.
17:46
I don't know. I don't know what the
17:48
backgrounds of the students were aside from, you know, I do know
17:51
that, you know, the sort of the primary question about prior
17:54
experience at the law school trial at C level as if they've
17:56
done it before, for example, in college.
17:58
And I know that in the final round, all four of the
18:00
competitors had a lot of had a lot of prior mock trial
18:04
experience, but I don't know about their engineering IP
18:07
interests. And now that prompts me and this
18:09
is outside of what you've been doing also an anti data
18:12
question. Have you noticed any trends in
18:14
the people that you've because you've been doing this a long
18:18
time? Yeah. 18 years.
18:20
Yeah. Yeah.
18:20
Ever since I graduated, yeah. So in terms of the people that
18:23
you've taught, have you seen any trends with the people that you
18:27
who've who've been with your teams in terms of where they end
18:30
up in their careers? Lots of them end up in, you
18:34
know, P DS and DA's offices. They want to try cases, you
18:37
know, So that's a primary outlet for folks also.
18:41
And I do a lot of work with the National Plaintiffs Law
18:42
Association. Plug to them.
18:44
By the way, there's a huge flood of amazing law students out
18:47
there who are spurning big law and saying they want to go work
18:49
with folks like us. So open your doors and open your
18:52
arms to them because they are amazing and there's incredible
18:54
energy in it. But in keeping with that, I have
18:57
certainly seen more of my students going off to the
18:59
plaintiff side. I'm a huge, huge advocate for
19:02
that. And so plug, plug for that.
19:04
So that's a trend. And of course, big law is big
19:06
law and it gets ever more eager to, to recruit.
19:11
And I don't know if you've noticed, but the recruiting
19:12
practices have have changed dramatically on big loss side
19:15
they are now. So we're in the fall, late fall
19:18
of, of 2025. If you're a 1L at a, you know,
19:21
at a law school, they're recruiting you for your two L
19:24
summer right now. So not next summer for your two
19:26
L summer before you've got grades before you have any idea
19:28
what you're doing. And in some cases they're paying
19:30
very large bonuses for you to accept your two L summer, which
19:33
then turns into a, a job offer after your three after you
19:36
graduate. And so there is a huge effort on
19:40
their part to to force people to go to big lot.
19:42
And you know, to me it tells me that like we're doing something
19:46
right if it's if it's scary to the other side, that more people
19:48
are going to our side. So.
19:50
To follow up on that piece, because I, I, you wear many hats
19:53
in terms of the things that you do.
19:55
I know you're also involved with getting the word out for
19:58
plaintiffs lawyers nationally. For those who want to be
20:00
involved or want a little bit more visibility on that.
20:02
Can you give a shout out or a website or a how do I do what my
20:08
part in terms of of getting the message out in the plaintiffs
20:10
world? Thank you.
20:11
What a great question. So absolutely.
20:13
So there is a new organizations about four years old called the
20:17
National Plaintiffs Law Association and that's their
20:19
website, nationalplaintiffslawassociation.org.
20:23
It is law students across the country who have created an
20:27
umbrella organization over the top of the plaintiffs law
20:29
associations that exist at many law schools now.
20:32
In fact, as of last count, there are 39 active plaintiffs law
20:35
associations across the country with 17 more coming on board
20:38
here very shortly. So the eagerness is palpable.
20:41
I just actually spent the last 10 days.
20:44
Actually, I just spent the last seven days.
20:47
In seven days, I went to 8 law schools on the East Coast and
20:49
talked about what plaintiffs practice is, talk to 150
20:51
students about it. So check out National Plaintiffs
20:54
Law Association and honestly, if you're interested in having
20:57
summer associates or looking for a way to tap into this energy,
21:02
please e-mail me spalki@walkuplawoffice.com.
21:04
I would love to help open the door to more law students to
21:07
plaintiff side. And just to keep.
21:09
Unbiased. I think we have to give a plug
21:11
to the defence side too here, you know, there's, there's just
21:15
as many organizations and recruiting just as heavily.
21:18
So you can just go to nevasec-cole.com and find all
21:23
you need to know. Enough of that.
21:26
No, I'm curious. I I.
21:27
No more. We need more we.
21:28
Didn't I didn't lie totally and I didn't mean to paint you with
21:32
the same brush I was I was using, I was using the big law
21:35
brush and I don't think you're that.
21:36
Don't sweep me with that brush. There you go, and my apologies.
21:40
With every Ying there is a Yang, and if we don't have defense
21:43
lawyers, then they're in. There's no one to play with,
21:45
right? Yeah.
21:47
But it's interesting. It's a big, I mean, like when I
21:48
say big law, you know what we're talking about.
21:49
We're talking about, you know, these AM Law 100 firms and all
21:53
that stuff. That's a different animal than
21:55
the people we're typically fighting with, you know, that
21:57
kind of thing. So.
21:58
For those who are following along, we'll we'll drop links in
22:00
the show notes so that people can can get these resources
22:03
without having to to pause and and scribble stuff down.
22:06
I think it's great that that law students coming out of law
22:10
school aren't necessarily just going for those the big law jobs
22:14
and are really understanding that the way to get in the
22:16
courtroom. It is honestly on the plaintiff
22:18
side. I mean, that's where you're most
22:20
easily going to get trials outside of criminal.
22:23
And I think it's, I think it's great.
22:25
I really do. Well, let's be realistic.
22:27
Nobody decided to go to law school because they said I want
22:30
to be corporate counsel for, you know, name whatever company you
22:33
want to name. They're going to law school
22:36
because they have a purpose. And I think for a while that
22:39
messaging on the plaintiff side has been lost as to why we do
22:43
what we do. And I think the work that
22:45
Spencer and the other law students are doing is now
22:49
bringing it back to the core values that really get us up in
22:53
the morning and actually get us to do the work that we do.
22:57
And plaintiffs lawyers are cool at law schools now.
22:58
Like we're cool there, you know, like we're not like chasing
23:01
ambulances. I mean, it's like you get
23:02
questions about like, aren't you know, I've heard it's not like,
23:04
it's not like aren't you guys all playing ambulance chasers?
23:06
It's not like that now. It's like I've heard, you know,
23:09
people's describe plaintiffs lawyers, ambulance chasers.
23:11
How do we fight that, that image?
23:13
Like it's like they are like they're on our team.
23:16
I think you just put up $100 million verdict and then you're
23:19
like, man, I think it's a little bit different than I.
23:24
I'm, I've seen a good segue here, Spencer, because one of
23:27
the things that we rounded with the group just before we started
23:30
was an article that Sherrod put forward by Nick Schweitzer and
23:34
Jessica Salerno at Cornell University.
23:36
If I'm getting it right, you recently had coffee with Nick.
23:40
And this is an article on the disutility of large language
23:43
model AI and predicting juror decisions.
23:46
And so I think this is a good corollary to the information
23:51
that you just provided us and and and a launchpad for us to
23:54
talk a little bit about how people are using focus groups
23:59
and AI and actually going to verdict to see what's happening
24:03
there. I don't know what my question is
24:05
but that was the Segway it. Was a great Segway.
24:08
Thank you. Actually, well, I, I will kick
24:10
it to you. Spencer, since you had an
24:12
opportunity to sit down with one of the authors of of this paper,
24:15
any insight from you as how it relates to what you're seeing on
24:20
the other side with the mock trials?
24:24
That's interesting. So I will say just as to the
24:26
paper itself, I've, I've not read the paper, but I watched
24:28
Sharad and I, as it happens, watched Nick give this
24:31
presentation not long ago at a conference.
24:34
And it was, it's completely fascinating because one problem
24:38
with AI is it's, it's like got this like positive confidence
24:43
all the time. You know, it just acts right all
24:44
the time, which is one reason it's so dangerous.
24:46
And it's so easy to think, well, it's, you know, it's analyzed
24:49
all the data in the world. And why can't I just, you know,
24:51
describe my focus group and tell it to like, you know, give me
24:54
Mecklenburg County or, you know, Alameda County or whatever and
24:57
run it with 100 juries, which people do all the time.
25:00
And it gives you this very clean bullet pointed, you know,
25:02
response. Here you go.
25:03
Here's your answer to your question.
25:05
And because it's so good at the answering other questions which
25:07
we can trust, it's answer to it. It you worry that it's going to,
25:10
you know, it's overly trust. You know, it's give it too much
25:13
trust with focus groups. So Nick's paper and Nick's
25:16
presentation is all about how like he checked.
25:18
He's like, OK, well, let's let's look at outcomes as predicted
25:21
by, you know, AI and by the big data focus groups that they do
25:25
with jury ball. And like, it's stunning to look
25:28
at some of his graphs. You're like, you know, here's
25:30
what AI predicted and like, here's the outcome.
25:32
It's like orders of magnitude in each direction, you know, and
25:35
that was stunning and also very heart warming to see
25:39
shortcomings in AI documented. So clearly.
25:42
I'm looking forward to reading the paper, but the presentation
25:44
was awesome. One observation I had in
25:47
response to this paper and I I've shared this in various
25:50
places for a while. We used a Monte Carlo simulator
25:55
to help us get some information and insights on our our case
26:00
projections. So what when we take a case in
26:03
early on, we rate what the case is worth and we try to be
26:05
conservative. Similarly, we, we rate how long
26:08
do we think it's going to take us to get to conclusion and we
26:11
try to be conservative. And what I found when a finance
26:17
person used a Monte Carlo simulator, which was we'll have
26:20
this thing run multiple, multiple times and use past
26:22
data, etcetera, complicated stuff that I is beyond me.
26:26
Quite honestly. What we found is very similar to
26:29
this. This article, which is no data
26:35
combustor is as good as the gut instinct of say a Marshall or a
26:42
Sherrod or a Spencer or a me who's done this for many, many
26:45
years and has kind of that limbic system sense of, oh, you
26:49
know, a collarbone fracture from a right hook in Alameda County,
26:54
non surgical. That's going to be 175 to 250
26:59
and it'll take, you know, 18 months.
27:01
For me. It's it's a a reaffirmation that
27:04
there's a human element that will continue in our work.
27:10
Well, I mean, let's be realistic.
27:12
There's a reason it's called artificial intelligence, because
27:15
there is no human aspect to it. Everything that it tries to do
27:20
is basically a simulation, and it's a word predictor.
27:24
It is not a human predictor. And as good as all of us may be
27:28
in valuing our cases, you still think about if you have a group
27:32
of 12 jurors sitting in a trial, we switch out some of those
27:35
jurors, the result could be different, which is why we all
27:39
want a bit more certainty. And I think the point of this
27:44
paper is that artificial intelligence doesn't provide you
27:47
certainty, but having more of a statistical sampling of human
27:54
beings will provide you a more accurate understanding of what
27:58
12 people may do in a jury setting.
28:01
And even with that, as I, you know, as close to accurate as it
28:05
is, I think their documents showed that they routinely
28:09
underestimate, not overestimate their verdicts by 20%.
28:14
And there was one example that they estimated that the verdict
28:18
was going to be $133 million and it actually turned out to be
28:22
$145,000,000 verdict. Yeah, I think it's like when I
28:26
looked at it, if someone's going to do this and really they do do
28:30
it for trial, it's almost like putting in the prompt, how do I
28:33
win at the at this point? Because, you know, in the actual
28:38
samples that were used, those lawyers tried a case to that
28:42
jury that they picked, certainly with whatever focus groups they
28:47
did and whatever knowledge they had about that particular
28:51
county. But they certainly struck jurors
28:54
who they thought were problematic, who may not have
28:57
been problematic for another lawyer, right?
29:00
The same person could have found themselves on the jury for the
29:04
same case with a different lawyer who got booted by someone
29:07
else. You know, So that's where like
29:09
the the trial lawyer aspect comes right into the fold.
29:13
Obviously in trial, when you see those actual jurors and
29:19
determine, is that particular person going to be sympathetic
29:24
to my client, harsh on my client, What's their direct
29:27
background, their employment history, maybe some family
29:30
questions came up during Drew, whatever it might be, none of
29:34
that can be sort of inserted into the prompt.
29:39
And so I, I, I was not really surprised by the fact of the
29:43
huge discrepancy and certainly with the fact that the actual
29:47
verdicts exceeded the AI predicted potential verdicts.
29:52
Because I think that just shows the true understanding that
29:55
those particular lawyers had a of their case and their client
30:00
and the actual jurors that they ended up with.
30:04
Also, how many times have we had a situation where you say to the
30:09
AI after it gives you the response?
30:10
Are you sure? And oh I'm, I'm, my apologies.
30:14
I'm so sorry, I got it all wrong.
30:17
So that's, that's one way of vetting.
30:20
It is like, are you, Are you sure?
30:21
I mean, where'd you get this from?
30:23
And I, I've done this before and three or four times, you know,
30:27
if I know the answer to the question, you ask it to give you
30:30
the answer, it's clearly wrong. And then you have to kind of
30:33
prompt it to finally get to the right answer.
30:37
And the reason is that it's trained to get to an answer as
30:40
quickly as possible. And even with the deep dives
30:43
that it does, it takes time to develop that.
30:47
And I think that goes back to what Spencer was saying at the
30:49
beginning, is you have to refine the prompt and you have to
30:53
refine your approach. But predicting what human beings
30:57
are going to do, I think that's a ways off.
31:00
In my real estate one, it was pretty funny when I I changed
31:03
one piece of data and it did actually come back and say, well
31:06
this changes everything and. I thought that.
31:10
I had never seen that happen for and there was like a red siren,
31:14
like a little emoji, and I thought that that was pretty
31:17
funny. I had never seen such a harsh
31:19
correction by a change in a piece of data.
31:22
I will say that, you know, one thing I take hard in is that
31:25
there's so much data that for now AI can't use in all of its
31:29
analysis. You know, for example, there's a
31:30
ton of proprietary data data out there that just it doesn't have.
31:34
There's a lot of data in our heads as you're talking about
31:36
miles that that it, as you know, as yet does not yet have about,
31:39
you know, values of cases and that kind of thing and then
31:42
interpreting emotions and circumstances.
31:44
So thank God for at least a while won't have that.
31:47
And also the flip side is sort of what it does have is like
31:49
it's got a lot of human laziness in it too.
31:51
You know, it's reading the whole Internet and everything.
31:53
Like there's a lot of laziness out there and a lot of wrong
31:56
stuff. So, you know, hopefully us
31:59
humans got a few more years in US.
32:01
Kicking it back to you, Spencer, I know that that this is what I
32:05
would describe as the, the qualitative discussion of the
32:07
information that you've gathered from this evolution.
32:11
It sounds like your team is also going to be doing some more work
32:15
for a, a quantitative analysis and, and may have more
32:19
information down the road. Can can you forecast what our
32:23
appetite of what that might look like and and and whether we
32:25
might be seeing you again? I hope so.
32:28
We'll see. So one beautiful thing about law
32:31
school trial advocacy is it is it is a good place for a lot of
32:35
testing because, you know, there are lots of repetitions, maybe
32:39
not, you know, thousands, but there's way more repetitions of
32:41
these trials than there are real trials.
32:43
Plus every aspect of the trial is scored by, you know,
32:46
practicing attorneys, which have their limitations.
32:48
But also like there's numerical, you know, numbers that, you
32:51
know, numerical analysis that goes with people's thoughts
32:54
about how good was that cross, how good was that closing, that
32:57
kind of thing. So we've got all that data and
32:59
who won each round? Did they win?
33:00
You know, all three judges, 2 out of 3 judges, that kind of
33:03
thing. And we want to look at the
33:05
amount of and the nature of the AI use as compared to who won in
33:10
trials to see if there's any correlations that can be pulled
33:12
from that. And frankly, I am quite cure.
33:14
I would be I would be fascinated to learn that there is no
33:16
correlation or that there is AII would think any of the following
33:19
any of the answers would be interesting to me.
33:22
But we do want to break that down.
33:25
We've got some folks helping us. So I would love to report back
33:27
so the opportunity arise. I'm curious and you if you can't
33:32
answer it, I understand. Do you believe that the team
33:36
that won and had the most effective use of AI would have
33:40
won without if nobody was able to use AI?
33:44
You know, if you just remove that based on what you know and
33:46
having seen them as trial lawyers and put on the case, do
33:50
you think that was even a difference maker or were they
33:52
going to win anyways? It's really interesting.
33:55
Potentially, potentially would have made a difference because
33:58
the team that they beat is a historical powerhouse in mock
34:04
trial and didn't use AI very much.
34:06
And you know, again, I don't know if that's good or bad.
34:08
I'm not sure, but if you were to ask who's going to win that
34:12
round, knowing nothing else in our community, you would not
34:15
have picked the team that won so.
34:19
That's, I think that's really interesting.
34:21
Any other questions for Spencer? Want to be a defense player?
34:28
You, Sir, you're, you're defending, you know, plaintiff
34:32
lawyers and in part who you know are.
34:35
Really needing your help and I'm the only defend.
34:37
God bless you, you guys like I'm the only defense lawyer that.
34:40
Marshalls Marshalls defending the wrongly accused.
34:43
Bless him for it. A final piece, and to the extent
34:46
that anyone here has anything unrelated to what Spencer been
34:49
talking about, but we only get together about once a month or
34:52
so. Anything cool and neat fun that
34:55
you stumbled across in the last month that you feel our audience
34:58
should get a a snippet insight into?
35:00
I did have a pro per plaintiffs use AI on me.
35:05
And it's going to be interesting to see what the judge does
35:08
because we have obviously and actually the the hearing is
35:11
tomorrow where this is going to come to light.
35:16
It's in the relation to a motion for protective order.
35:18
Not a huge deal, but the pro per clearly used AI.
35:23
All of the cases either don't exist or are miss cited and the
35:27
propositions don't even remotely stand up.
35:31
So I'm curious because obviously as a lawyer, you know what's
35:33
going to happen. You're going to get sanctioned,
35:35
you're going to get referred to the State Bar, whatever.
35:37
But but here you got a pro per and the judge doesn't
35:41
necessarily have a ton, ton of options for what he can do to
35:46
this person. And so I'm really interested to
35:49
see how it's going to impact a the case as a whole.
35:51
It's obviously not a big case, but I haven't seen yet and I'm
35:56
glad I get to see it first hand. A judge deal with a pro per who
36:00
is using AI and I, I presume he may kick it over to us, just
36:06
seek some type of sanction, but we'll see.
36:08
I, I think it's interesting because it's clearly it being
36:10
used by proper plaintiffs all across the country and proper
36:13
defendants. And so I'm this thing, it's
36:16
really interesting on my end to see how it's going to work out.
36:19
Well, and I thought that it was an open AI that in said that
36:25
they're no longer going to let their systems give legal advice
36:31
recently. I don't know if I where I read
36:33
that, but that'll be an interesting development to see
36:36
how that works. And the reality of it is there
36:40
is a longer conversation to be had about access to justice for
36:43
people who simply can't hire a lawyer and how those systems are
36:48
going to be used to operate within the court system.
36:53
And whether or not any of the free legal clinics that are out
36:57
there are going to be able to tell people how to use AI to
37:00
protect themselves or to advance their cases in court.
37:03
But that's obviously a much longer conversation.
37:06
I will say that in a connection between mock trial and our
37:10
practices, one of my friends who's a trial lawyer about my
37:13
vintage in South Carolina runs University of South Carolina's
37:18
very good trial competition team.
37:21
He had a trial against a proper plaintiff who was, you know,
37:27
very clearly using I don't know what platform it was to try the
37:31
case. And, you know, I don't think it
37:34
was pretty. But he, you know, he got through
37:36
his case in chief and he had left out some, you know, very
37:38
important, like jurisdictional or other, you know, elements in
37:41
the case and got non suitors at that point.
37:43
But like that is far down the road right there that, you know,
37:46
to get that far. So wild story.
37:49
Yeah. We'll look forward to getting an
37:52
update on that Marshall next time we next time we meet.
37:56
For sure. Well, anything else that anyone
38:00
use Spencer, particularly since since you're a guest, feel that
38:03
we should cover? I can only give my two cents to
38:05
say that like it, it scares me and but it also excites me.
38:08
And of late, I've been feeling a little bit more optimistic.
38:11
Like it is a powerful tool that is a tool.
38:15
And much like the spreadsheet was 40 years ago, except for 100
38:18
times or 1000 times more powerful.
38:20
And you know, God bless humanity.
38:23
And I hope we're still necessary for a very long time into the
38:25
future, despite our helpful tools like AI.
38:28
From your lips to Chad GB TS ears.
38:36
Well, thank you Spencer for joining us and and Marshall
38:40
Sherrod. It's always a pleasure.
38:42
I enjoy these conversations. It helps keep my mind stimulated
38:46
on on the AI front. So love.
38:49
It thanks for having me guys. I really appreciate it.
38:51
For all you listening out there, thanks for listening.
38:54
If you have any questions, comments, feedback on how you've
38:57
been using AI or any trends that you've seen, please e-mail us at
39:00
podcast at Coopers dot law. For all of you doing justice out
39:03
there, happy hunting.