Latest / AI Tools for Practicing Lawyers / Field Note: Tiers Of The Clown

Transcript

Field Note: Tiers Of The Clown · AI Tools for Practicing Lawyers

• 0:03
  SPEAKER_01: Welcome to AI Tools for Practicing Lawyers.
• 0:07
  SPEAKER_01: Practical, no-nonsense guidance to help attorneys put AI power to work in their practice right now.
• 0:15
  SPEAKER_00: Welcome back to AI Tools for Practicing Lawyers.
• 0:19
  SPEAKER_00: I'm Ron Dresher, and today's field note with apologies to Smokey Robinson is called Tears of the Clown.
• 0:29
  SPEAKER_00: And it started with a Reddit thread.
• 0:31
  SPEAKER_00: Someone asked a simple question: if I pay for the higher AI tier, do I get smarter AI?
• 0:38
  SPEAKER_00: The responses were exactly what you would expect.
• 0:41
  SPEAKER_00: Some people said yes, some people said no.
• 0:44
  SPEAKER_00: Some people said the model has gotten worse.
• 0:46
  SPEAKER_00: Others said, no, no, no, the model's gotten better.
• 0:49
  SPEAKER_00: And as I read through the discussion, I realized something.
• 0:53
  SPEAKER_00: Everybody was arguing about a label.
• 0:56
  SPEAKER_00: And the label was smarter.
• 0:59
  SPEAKER_00: Now, years ago, when my son and daughter were teenagers, and this was very irritating to them at the time, I tried to teach them to be careful about labels.
• 1:09
  SPEAKER_00: Don't rush to label people, don't rush to label situations, don't rush to label ideas.
• 1:16
  SPEAKER_00: Because what happens when you apply a label is you short circuit analysis.
• 1:24
  SPEAKER_00: And the labels become a shorthand for something that is usually much more complicated.
• 1:29
  SPEAKER_00: So when someone says that person is dishonest or that golfer is talented or that AI is smarter, I'm always feeling the same way, which is compared to what?
• 1:41
  SPEAKER_00: Under what circumstances?
• 1:43
  SPEAKER_00: For what purpose?
• 1:44
  SPEAKER_00: What's your evidence?
• 1:46
  SPEAKER_00: The label is not the answer.
• 1:49
  SPEAKER_00: It's the beginning of the analysis.
• 1:52
  SPEAKER_00: And I think that's exactly what's happening when people argue or really become confused about AI tiers.
• 2:00
  SPEAKER_00: The question isn't which tier is the smartest.
• 2:03
  SPEAKER_00: The better question is what capabilities does this tier provide?
• 2:09
  SPEAKER_00: Now, before I go further, I prepared a companion handout for this episode.
• 2:15
  SPEAKER_00: Let me give you a warning.
• 2:17
  SPEAKER_00: The handout may be partially obsolete by the time you're reading it.
• 2:23
  SPEAKER_00: In fact, even by the time you're listening to this, AI changes that fast.
• 2:28
  SPEAKER_00: Pricing changes, features change, the models change, tier names change.
• 2:35
  SPEAKER_00: So the handout, which I still feel is very valuable, is really it's just a snapshot for this moment in time.
• 2:44
  SPEAKER_00: But it's the framework that really matters.
• 2:47
  SPEAKER_00: The framework survives.
• 2:49
  SPEAKER_00: All right, now let's talk about how I got here.
• 2:52
  SPEAKER_00: Some of you may remember my folder mania experiments and what I called the file cabinet test.
• 2:59
  SPEAKER_00: The basic idea was simple.
• 3:01
  SPEAKER_00: Can an AI understand the contents of a folder full of documents?
• 3:07
  SPEAKER_00: Can it find information?
• 3:08
  SPEAKER_00: Can it organize information?
• 3:10
  SPEAKER_00: Can it summarize the information?
• 3:13
  SPEAKER_00: Can it work with information the way lawyers actually work?
• 3:18
  SPEAKER_00: During those experiments, I had access to an enterprise environment through my co-host Heather Gardner.
• 3:25
  SPEAKER_00: And the enterprise environment clearly demonstrated capabilities that my ChatGPT Plus account did not have.
• 3:34
  SPEAKER_00: In some situations, it performed better.
• 3:37
  SPEAKER_00: Not marginally better, materially better.
• 3:40
  SPEAKER_00: For example, the enterprise environment was able to pass versions of the file cabinet test that my plus account just couldn't handle.
• 3:51
  SPEAKER_00: That was a real difference, a real capability advantage.
• 3:55
  SPEAKER_00: You know, I was able to upload a zip file to the ChatGPT enterprise version and have it read it perfectly, and the plus version just simply couldn't handle a zip file of 48 sub files the way the enterprise tier could.
• 4:15
  SPEAKER_00: But here's where the story gets kind of interesting.
• 4:18
  SPEAKER_00: When I sit down to brainstorm a podcast episode like this one, I almost always go back to my plus account.
• 4:26
  SPEAKER_00: Why?
• 4:28
  SPEAKER_00: Because it's mine.
• 4:30
  SPEAKER_00: Not because it's smarter, not because enterprise is worse, because this environment knows me.
• 4:36
  SPEAKER_00: It knows the podcast.
• 4:38
  SPEAKER_00: It knows the Flintstones, Simpsons, Jetsons framework, knows the three-legged stool, it knows folder mania.
• 4:46
  SPEAKER_00: It just knows the kinds of questions that I ask.
• 4:50
  SPEAKER_00: It knows the listeners.
• 4:52
  SPEAKER_00: The enterprise environment is optimized for Heather's world.
• 4:56
  SPEAKER_00: This environment, over time, over all of the different brainstorming sessions that I've gone through, has become optimized for me.
• 5:06
  SPEAKER_00: And that taught me something really important.
• 5:09
  SPEAKER_00: The best AI tier is not necessarily the smartest tier.
• 5:13
  SPEAKER_00: The best AI tier is not even necessarily the most capable tier.
• 5:18
  SPEAKER_00: The best AI tier is the one whose capabilities and accumulated context.
• 5:25
  SPEAKER_00: AI wrote that sentence.
• 5:31
  SPEAKER_00: I wrote that part of it.
• 5:32
  SPEAKER_00: Once I realized that, I stopped asking which tier is best.
• 5:36
  SPEAKER_00: And I started asking, what capabilities am I actually buying?
• 5:42
  SPEAKER_00: As I looked across ChatGPT, Claude and Gemini, I realized that vendors keep packaging the same handful of capabilities in different ways.
• 5:53
  SPEAKER_00: And to make this field note really a little bit deeper and a little bit more meaningful, I want to briefly walk through them.
• 6:01
  SPEAKER_00: First of all, there's the concept of context window.
• 6:06
  SPEAKER_00: Think of it as the size of the AI's desk.
• 6:09
  SPEAKER_00: The larger the desk, the more information it can keep in view at one time.
• 6:16
  SPEAKER_00: Then there's the concept of retrieval.
• 6:19
  SPEAKER_00: Can the AI find the information it needs?
• 6:23
  SPEAKER_00: My file cabinet test taught me that retrieval can be every bit as important as reasoning.
• 6:29
  SPEAKER_00: Because if the AI can't retrieve all of the documents that you needed to retrieve, it's not going to satisfactorily accomplish that task.
• 6:38
  SPEAKER_00: All right, then there are usage limits.
• 6:41
  SPEAKER_00: How much fuel do you get?
• 6:43
  SPEAKER_00: How many questions?
• 6:44
  SPEAKER_00: How many uploads?
• 6:45
  SPEAKER_00: How much processing?
• 6:47
  SPEAKER_00: That's all going to determine how satisfying and how complete your session is going to be with the AI.
• 6:56
  SPEAKER_00: Then we get to connectors.
• 6:58
  SPEAKER_00: Can the AI access information where it already lives?
• 7:02
  SPEAKER_00: For example, Google Drive, Gmail or Outlook, SharePoint, Slack.
• 7:08
  SPEAKER_00: Can the AI reach the different tools that you're using that are outside the boundaries of that AI tool?
• 7:17
  SPEAKER_00: Then there's memory.
• 7:19
  SPEAKER_00: Can the AI remember useful information across time?
• 7:23
  SPEAKER_00: And finally, there's governance.
• 7:26
  SPEAKER_00: Now, if you've listened to my three-legged stool episode, you'll recognize that governance is essential.
• 7:34
  SPEAKER_00: But governance doesn't necessarily make a tool good.
• 7:38
  SPEAKER_00: A tool that is compliant can still be terrible.
• 7:43
  SPEAKER_00: It can still not accomplish what you're trying to accomplish with that tool.
• 7:49
  SPEAKER_00: So governance simply means the tool has the controls necessary to be used responsibly in a professional environment.
• 7:58
  SPEAKER_00: Capability and governance are different things.
• 8:01
  SPEAKER_00: You need both.
• 8:02
  SPEAKER_00: And that's where the handout comes in.
• 8:05
  SPEAKER_00: The handout tries to break these tiers down into their component parts.
• 8:09
  SPEAKER_00: Not because the table that's in the handout is perfect, tables typically aren't.
• 8:16
  SPEAKER_00: They're just convenient ways of looking at data in one quick look.
• 8:22
  SPEAKER_00: And not because every feature listed there will still be true six months from now.
• 8:26
  SPEAKER_00: They won't.
• 8:27
  SPEAKER_00: The value is in learning how to think about the tiers.
• 8:32
  SPEAKER_00: When someone says Claude is smarter or Gemini is better or ChatGPT got worse, the lawyer's response really should be, what exactly are we talking about?
• 8:43
  SPEAKER_00: Context, retrieval, usage, connectors, memory, governance, model access?
• 8:52
  SPEAKER_00: Is it something else?
• 8:54
  SPEAKER_00: Because once you unpack the label, the analysis becomes much more interesting and much more useful.
• 9:00
  SPEAKER_00: So if you take one thing away from today's episode, let it be this.
• 9:04
  SPEAKER_00: Stop asking which AI tier is smartest.
• 9:08
  SPEAKER_00: Start asking which capabilities matter for the task at hand.
• 9:13
  SPEAKER_00: Because the answer will be different for brainstorming a podcast, reviewing discovery, managing a law firm, or handling confidential client information.
• 9:25
  SPEAKER_00: The best tier isn't the smartest tier.
• 9:28
  SPEAKER_00: It's the tier whose capabilities and context best match the job you're trying to do.
• 9:35
  SPEAKER_00: Thanks for listening.
• 9:37
  SPEAKER_00: This is Ron Tresher, and I'll see you next time on AI Tools for Practicing Lawyers.
• 9:48
  SPEAKER_01: That's it for today's episode of AI Tools for Practicing Lawyers.
• 9:52
  SPEAKER_01: Thanks for listening.
• 9:54
  SPEAKER_01: We'll see you next time.