Latest / Herbert Smith Freehills Kramer Podcasts / Legal Tech Deciphered Trust, Truth and Hallucinations EP3: Innovation and Defensibility: Why hybrid solutions can be the new standard for AI assisted review
Transcript
- 0:00SPEAKER_00: Every matter has its unique risk profile, scale, and data considerations.
- 0:06SPEAKER_00: Hybrid models give teams the flexibility to adapt without compromising defensibility or efficiency.
- 0:13SPEAKER_00: The most effective reviews we've been involved in are the ones where there's been genuine thought at the outset about what the data looks like, what the issues are, what level of defensibility is required, and which tools, particularly in which sequence, best serve that particular exercise.
- 0:43SPEAKER_01: Welcome back to Legal Tech Deciphered: Trust, Truth and Hallucinations, Herbert Smith Free Hill Scramer's podcast cutting through the noise around legal tech and gen AI.
- 0:54SPEAKER_01: In today's episode, we focus on when Gen AI genuinely adds value.
- 0:59SPEAKER_01: I'm Stephanie Barrett, a director for legal technology in the Digital Legal Delivery Practice Group.
- 1:04SPEAKER_01: I work at the intersection of e-discovery, legal workflows, and emerging technology, bridging the gap between the legal world and digital innovation.
- 1:12SPEAKER_01: In this episode, we'll be exploring how Gen AI and hybrid review workflows are fundamentally changing the way legal services are delivered.
- 1:21SPEAKER_01: Today I'm joined by two colleagues from our digital disputes practice group.
- 1:24SPEAKER_01: Firstly, Craig Glover, head of disputes.
- 1:27SPEAKER_01: Craig works closely with clients on complex litigation, arbitration, and investigations.
- 1:32SPEAKER_01: He collaborates to design disclosure workflows that are not just efficient but robust enough to stand up to scrutiny from courts, regulators, and opposing parties.
- 1:40SPEAKER_01: Completing our trio today is Kiva Powell, a director in the digital disputes team.
- 1:45SPEAKER_01: She works closely with clients on complex, contentious matters.
- 1:49SPEAKER_01: Kiva brings a strong focus on designing disclosure workflows that balance efficiency, defensibility, and practical delivery.
- 1:56SPEAKER_01: Welcome both.
- 1:57SPEAKER_00: Thanks, Steph.
- 1:58SPEAKER_00: Good to be here.
- 1:59SPEAKER_01: Thanks, looking forward to it.
- 2:02SPEAKER_01: Excellent.
- 2:02SPEAKER_01: And it's really helpful to have both the delivery and disputes perspectives in the room for this.
- 2:08SPEAKER_01: In a previous episode, we explored how generative AI was deployed in a large-scale high court disclosure exercise, the practical decisions made, the governance around it, and the genuine value it delivered.
- 2:19SPEAKER_01: Today's conversation is going to build directly on that foundation.
- 2:22SPEAKER_01: But what's most important about this episode is we're going to talk about the application of Gen AI and the impact it has from the people who are at the coalface, the review team.
- 2:32SPEAKER_01: Rather than just talking about the tech, we're going one step further and asking how do you actually integrate Gen AI into real review workflows in a dispensible way at scale over the course of a complex matter?
- 2:44SPEAKER_01: And we're going to come back to what defensible really means in practice a little bit later in the episode.
- 2:50SPEAKER_00: And to your point, Steph, this isn't a conversation about Gen AI being the answer to everything.
- 2:56SPEAKER_00: The headline, I suppose, if you want one, is this the future of document review isn't Gen AI or Cal.
- 3:03SPEAKER_00: It's often both working in concert within the same workflow.
- 3:07SPEAKER_00: That's the shift we're seeing in practice.
- 3:10SPEAKER_00: And it's what we want to unpack today.
- 3:14SPEAKER_02: I agree.
- 3:14SPEAKER_02: And from a delivery perspective, that distinction really matters.
- 3:18SPEAKER_02: There's still a lot of hype around Gen AI and document review.
- 3:21SPEAKER_02: And our aim today is to cut through that and grind the conversation and what we're actually seeing in live delivery, what the tools each bring, where they fit together, and what good looks like when you get the combination right.
- 3:34SPEAKER_01: Excellent.
- 3:34SPEAKER_01: Well, Craig, to kick us off, you mentioned a shift in perception around AI-assisted document review.
- 3:40SPEAKER_01: Something interesting does seem to be happening in the market.
- 3:43SPEAKER_01: Gen AI is actually making clients more comfortable with the concept of AI-assisted review generally, including TAR.
- 3:50SPEAKER_01: And to keep things simple, when we're talking about TAR, let's assume we're referring to continuous active learning or Cal.
- 3:56SPEAKER_01: Do you think that's right?
- 3:59SPEAKER_00: Thanks, Steph.
- 4:00SPEAKER_00: It is.
- 4:00SPEAKER_00: And it's one of the more uh welcome developments we've seen.
- 4:05SPEAKER_00: For years, there was a degree of skepticism around Cal courts and clients were cautious.
- 4:11SPEAKER_00: And explaining the methodology was sometimes an uphill task.
- 4:16SPEAKER_00: What we're observing now is that Gen AI paradoxically is helping because clients are engaging with AI in other parts of their lives.
- 4:24SPEAKER_00: The concept of AI assisted document reveal feels less foreign.
- 4:29SPEAKER_00: That halo effect is real and it's created space for a much more nuanced conversation about tool selection.
- 4:38SPEAKER_02: And that shift shows up very clearly in the questions we're getting.
- 4:41SPEAKER_02: Rather than Gen AI creating competition, do we use this or Cal?
- 4:46SPEAKER_02: It's actually enabling a more sophisticated discussion.
- 4:48SPEAKER_02: Clients are asking all the right questions and they want to understand the workflow, not just the outcome.
- 4:54SPEAKER_01: And it's interesting, isn't it?
- 4:55SPEAKER_01: Because one thing we hear a lot is that Gen AI is so powerful, it simply replaces everything that came before it.
- 5:03SPEAKER_01: But based on what you've just said, would you say that's actually an oversimplification when it comes to document review?
- 5:10SPEAKER_00: Absolutely.
- 5:11SPEAKER_00: It's an oversimplification for a very practical reason.
- 5:15SPEAKER_00: Document review isn't one thing, it's a series of distinct tasks identifying the population, legal analysis, classification, quality control, privilege review, redaction, production.
- 5:30SPEAKER_00: And they don't all have the same risk profile or the same validation requirements.
- 5:35SPEAKER_00: No single tool covers all of those needs.
- 5:38SPEAKER_00: Different technologies excel at different tasks.
- 5:41SPEAKER_00: Gen AI is extraordinarily capable in some of those areas.
- 5:45SPEAKER_00: In others, different tools remain not just relevant, but essential.
- 5:50SPEAKER_02: And I think it's worth being honest about where the hype comes from.
- 5:53SPEAKER_02: Gen AI is genuinely impressive.
- 5:56SPEAKER_02: The speed, the contextual reasoning, the ability to surface themes across enormous data sets, these are real capabilities, but impressive isn't the same as sufficient.
- 6:04SPEAKER_02: And in the context of contested litigation, sufficient means defensible.
- 6:09SPEAKER_01: So, in both of yours experience, is it fair to say Gen AI actually works best when it's not being asked to do everything?
- 6:18SPEAKER_00: Yes, that's exactly the right way to put it.
- 6:21SPEAKER_00: Genai works brilliantly as part of a workflow that's been designed thoughtfully, one that pairs its intelligence with appropriate validation, which can be done in a number of ways, including via the use of Cal and human oversight.
- 6:37SPEAKER_00: Hybrid models allow teams to tailor workflows to the specific risks and complexities of each matter.
- 6:44SPEAKER_00: In high-stakes context, like litigation, having proportionate controls and robust validation in place makes Genei outputs far easier to rely on and defend.
- 6:56SPEAKER_01: And we we've touched on this already, but I think it's worth calling out directly.
- 7:00SPEAKER_01: We've all been working in this space for a long time.
- 7:03SPEAKER_01: And I know from an e-discovery perspective, it's been fascinating, um slash possibly a little bit perplexing to see how quick mindset seems to be shifting from this quite cautious approach to AI assisted review to jumping straight in with let's use Gen AI for the whole workflow.
- 7:18SPEAKER_01: Let's go.
- 7:18SPEAKER_01: Um, where do you think that that's come from?
- 7:22SPEAKER_02: I think a lot of it comes down to just how impressive Gen AI is to use.
- 7:26SPEAKER_02: You throw a complex question at it, point it at a big data set, and it comes straight back with a confident, intelligent sound in answer.
- 7:33SPEAKER_02: And that can make it seem like it's doing everything for you, start to finish, finding, classifying, validating, explaining.
- 7:40SPEAKER_02: And in some settings, that's close to true.
- 7:42SPEAKER_02: But document review and litigation is a very specific context.
- 7:46SPEAKER_02: One size genuinely doesn't fit all.
- 7:49SPEAKER_02: There's no single model that works for every case.
- 7:51SPEAKER_02: What you need really depends on the data, the deadlines, and the risk profile of the matter.
- 7:57SPEAKER_00: I agree, Kiva.
- 7:58SPEAKER_00: This the specific problem is this.
- 8:00SPEAKER_00: Courts, regulators, and opposing parties need to be able to understand and may challenge your methodology.
- 8:08SPEAKER_00: They need to know not just what the decision was made, but why and how you've validated it at scale.
- 8:15SPEAKER_00: Gen EI produces outputs.
- 8:17SPEAKER_00: Without the right framework around it, these outputs are difficult to audit and difficult to defend.
- 8:23SPEAKER_00: Hybrid approaches address that directly.
- 8:27SPEAKER_00: They may improve transparency and demonstrate to regulators and courts that teams applied the right tool for the right task.
- 8:34SPEAKER_01: So are we saying that rather than Gen AI replacing tools like Cal, the argument is that it genuinely enhances them?
- 8:41SPEAKER_02: That's the model we're seeing in practice.
- 8:44SPEAKER_02: Gen AI and continuous active learning aren't competitors, and it's important to understand what each distinctly brings.
- 8:50SPEAKER_02: Gen AI brings contextual understanding, rapid summarization and theme extraction, semantic exploration and enhanced early case assessment.
- 8:59SPEAKER_02: Continuous active learning brings defensible classification, consistency and audibility, statistical validation, and a methodology that's ideally suited to large-scale high-risk review.
- 9:11SPEAKER_02: Together, they're significantly more powerful than either on its own.
- 9:15SPEAKER_02: Hybrid workflows reduced reliance in any single technology, balance in innovation with proven methodologies.
- 9:22SPEAKER_00: And there's a broader point here.
- 9:24SPEAKER_00: The teams we see getting the best results are asking which tool we should use.
- 9:29SPEAKER_00: They're asking what does this particular matter need at each stage and which tools deliver that?
- 9:35SPEAKER_00: That's a fundamentally different and more productive question.
- 9:40SPEAKER_01: So the title of this podcast focuses on where Gen AI actually adds value.
- 9:45SPEAKER_01: So where are you both actually seeing Gen AI delivered the most value in a real review workflow?
- 9:52SPEAKER_00: Yeah, thanks, Steph.
- 9:53SPEAKER_00: That's a question that comes up a lot.
- 9:55SPEAKER_00: Um, so one area that stands out is that um it it consistently delivers first-level review acceleration.
- 10:03SPEAKER_00: When you have a well-engineered prompt and a well-defined relevance framework, Gen AI can work through very large document populations at speed that changes the economics of the exercise entirely.
- 10:16SPEAKER_00: We saw that in the case study from the previous episode.
- 10:19SPEAKER_00: 97,000 documents processed in 12 hours.
- 10:24SPEAKER_02: I'd add a second related area.
- 10:26SPEAKER_02: We're also seeing Gen AI deliver value early in the workflow around sense making rather than final judgment.
- 10:32SPEAKER_02: It's particularly strong at helping teams understand large, unfamiliar data sets quickly, surfacing themes, identifying patterns, and highlighting material that warrants closer attention.
- 10:43SPEAKER_01: Just to pick up on that last point, you mentioned surfacing themes more quickly.
- 10:48SPEAKER_01: Why is Gen AI better at this than the earlier technology we've been using?
- 10:53SPEAKER_00: Good question.
- 10:54SPEAKER_00: Um, the key difference is rationale.
- 10:57SPEAKER_00: Gen AI tools can provide rationale, and the explanation matters as much as the decision.
- 11:04SPEAKER_00: When the lawyer is reviewing a document and dealing with a genuinely marginal call, understanding why it's been flagged is genuinely useful.
- 11:13SPEAKER_00: That's where Gen AI adds something new.
- 11:16SPEAKER_00: It can articulate its reasoning in a way earlier technology simply couldn't.
- 11:21SPEAKER_00: And critically, that early intelligence has a knock-on effects, accelerates early understanding of the issues, concepts, and themes in the data, which in turn supports more informed and better targeted human judgment later in the process.
- 11:36SPEAKER_02: The ability to apply a consistent articulated standard across large populations for quality control purposes is a significant capability.
- 11:44SPEAKER_02: Rather than sampling randomly and hoping you've caught what matters, you can use GNAI to interrogate specific populations, test consistency, and focus human review on the areas where it adds most value.
- 11:56SPEAKER_01: And I don't think we can get through this podcast without me asking the cliche question.
- 12:00SPEAKER_01: And it was one that was asked when tar was first introduced back in the day and conferences, everyone was panicking, convinced reviewers were going to be replaced by robots.
- 12:09SPEAKER_01: Um, in your opinion, do you think Gen AI does actually remove the need for human reviewers, or is it simply changing where their time is spent?
- 12:18SPEAKER_00: Thanks, Stephanie.
- 12:19SPEAKER_00: It's a question we actually get a lot.
- 12:20SPEAKER_00: Um it changes where their time is spent, and that's a meaningful shift.
- 12:26SPEAKER_00: The goal was never to remove lawyers from the process, it was to ensure that the time lawyers spend is concentrated on the decisions that generally require legal judgment.
- 12:36SPEAKER_00: Gen AI handles the volume and surfaces the complexity.
- 12:40SPEAKER_00: Lawyers handle the complexity.
- 12:43SPEAKER_00: Combining Gen AI with human QC layers is precisely how you elevate accuracy while maintaining the oversight that defence ability requires.
- 12:53SPEAKER_02: I'd agree, but I go further and say that human oversight becomes more important, not less, when Gen AI is introduced.
- 13:00SPEAKER_02: You still need experienced lawyers, you understand the issues, you can interrogate the prompt, and you can recognize when the outputs need to be refined.
- 13:07SPEAKER_02: Gen AI performs best when it's used by engaged and performance practitioners, not as a substitute for judgment, but as a tool that sharpens it.
- 13:15SPEAKER_01: Yeah, that will make sense.
- 13:17SPEAKER_01: And let's come back to Cal for a moment.
- 13:19SPEAKER_01: Um, we've spent a lot of time talking about Gen AI, but technology-assisted review in all of its iterations has been around for a long time.
- 13:26SPEAKER_01: Um, based on what we've discussed today, it sounds like Cal is is going to live to fight another day.
- 13:31SPEAKER_01: It's clearly not going away anytime soon.
- 13:33SPEAKER_01: Um, why is it still so important?
- 13:36SPEAKER_02: I think it's because defensibility has a specific meaning in this context.
- 13:41SPEAKER_02: It doesn't just mean I think the outputs are right.
- 13:44SPEAKER_02: It means being able to demonstrate with statistical grounding what was found, what wasn't found, and the level of confidence you can stand behind.
- 13:51SPEAKER_02: Cal remains a highly reliable method for classification at scale, particularly where courts or regulators expect and demand defensible, repeatable methodologies.
- 14:00SPEAKER_02: Cal provides exactly that framework: precision and recall metrics, statistical sampling, documented audit trails.
- 14:08SPEAKER_02: These aren't bureaucratic requirements.
- 14:10SPEAKER_02: They're the evidence that allows you to explain and justify your methodology to a court or to an opposing party.
- 14:17SPEAKER_00: And this becomes especially relevant as Gen AI is introduced into review workflows.
- 14:23SPEAKER_00: Genai produces outputs that are contextually intelligent and extremely useful.
- 14:30SPEAKER_00: But as volumes and stakes increase, teams often need additional ways to demonstrate consistency and confidence across the full data set.
- 14:39SPEAKER_00: Cal provides one way of doing that.
- 14:41SPEAKER_00: Rather than being replaced by Gen AI, it complements it, adding structure, transparency, and statistical support where the risk profile of the matter calls for it.
- 14:53SPEAKER_01: So all of this is ties back neatly to what I referenced at the start of this episode in relation to defensibility.
- 14:58SPEAKER_01: So when we are talking about defensibility, what does Cal add to the picture that teams often will be relying on in those higher risk matters?
- 15:07SPEAKER_00: Um, simply metrics, a documented training history, the ability to demonstrate what the model learned, how decisions were applied consistently, and what statistical confidence you have in the recall of relevant documents.
- 15:23SPEAKER_00: In very contested disclosure, these aren't nice to have.
- 15:27SPEAKER_00: They're what you rely on when the methodology is challenged.
- 15:32SPEAKER_02: And in hybrid workflow, the interaction between the tools is key.
- 15:36SPEAKER_02: Genai can contribute insight, context, and early understanding, while Cal could be used to test, corroborate, or challenge those insights at scale.
- 15:43SPEAKER_02: That's not about one tool propping up another.
- 15:45SPEAKER_02: It's about creating a workflow where different technologies validate each other.
- 15:49SPEAKER_02: So the overall approach is both efficient and offensible.
- 15:54SPEAKER_01: Now I'm the uh the technical voice in this discussion today.
- 15:57SPEAKER_01: So stepping back slightly, I think it's important to remember that whilst the focus we've just spent on talking about Gen AI and Cal, they are sophisticated tools within our toolkit.
- 16:07SPEAKER_01: It's just that they're part of a broader toolkit.
- 16:10SPEAKER_01: So given you're now running reviews which are supported by Gen AI and Cal, um, I have to ask, what about the sort of the more almost to be just described as old-fashioned, um, but I would say established techniques like conceptual analytics, email threading, neo-duplicate analysis.
- 16:27SPEAKER_01: Uh, do they still matter?
- 16:28SPEAKER_01: And please say they do.
- 16:30SPEAKER_02: Yes, they absolutely still matter, Steph, because they do work that makes everything downstream better.
- 16:36SPEAKER_02: Take email threading, for example, reviewing conversations as coherent threads gives you a proper context, helps you identify privilege accurately and consistently, and reduces the risk of missing significance because you're looking at a single message in isolation.
- 16:51SPEAKER_02: Neo-duplicate analysis plays a similar role.
- 16:54SPEAKER_02: It avoids duplicated effort, promotes consistency and decision making, and helps teams focus their attention where it's needed most.
- 17:02SPEAKER_02: These tools aren't always the headline act, but they're the foundations of an efficient, accurate review.
- 17:07SPEAKER_02: And when they're used properly, they make Gen AI and Cal all the more effective.
- 17:11SPEAKER_00: And functionality like clustering, gap analysis, conceptual analytics matter enormously at the strategic level.
- 17:20SPEAKER_00: Understanding what the data set actually contains, the key custodians, the date ranges, the communication patterns, the data types, that informs every subsequent decision about how to configure the review.
- 17:34SPEAKER_00: Those insights feed directly into GNAI prompting and TAR prioritization models.
- 17:40SPEAKER_00: Skipping that step is, in my opinion, a false economy.
- 17:44SPEAKER_01: Yeah, and I think what's really important and interesting actually about that part of the process, and in e-Discovery, it's what we would refer to as being on the left-hand side of the EDRM.
- 17:53SPEAKER_01: We're seeing e-Discovery software providers are really focusing on making that part of the process Gen AI enabled as well.
- 18:00SPEAKER_01: So I think going forward it's going to be fascinating to see how our workflows develop in that space.
- 18:06SPEAKER_01: Um, now to stick with my old-fashioned um technical methods, let's really go um back in time.
- 18:11SPEAKER_01: What about search terms?
- 18:13SPEAKER_02: Good question.
- 18:14SPEAKER_02: And one we're hearing a lot recently, search terms still have a role to play.
- 18:19SPEAKER_02: Used properly, they're an effective way in shape of shaping an arrow in the initial document population, creating a more focused starting point before the more advanced techniques can be applied.
- 18:29SPEAKER_02: The point isn't that any individual approaches become obsolete, it's that outcomes are consistently stronger when tools are deployed in a deliberate and complementary way, each doing what it does best as part of an overall and coherent workflow.
- 18:44SPEAKER_01: And I think again, just to I guess reassure everyone, I'm not I'm not proposing that we should stay in the dark ages with these technical processes.
- 18:50SPEAKER_01: Um, it's interesting for me to see where Gen AI is playing a role in search term creation.
- 18:55SPEAKER_01: Um, our e-discovery team is actually looking at where Gen AI tools can assist with refining search terms, making use of the best syntax, being as sophisticated as possible in terms of how those underlying search terms are structured.
- 19:08SPEAKER_01: So I think again, what we're really saying is it's about choosing the right combination of tools and that it is a combination of tools.
- 19:16SPEAKER_00: Absolutely.
- 19:17SPEAKER_00: It's always the right combination.
- 19:19SPEAKER_00: There's no universal right answer.
- 19:22SPEAKER_00: There is a thoughtful answer for each matter.
- 19:25SPEAKER_02: And the message we'd want clients and practitioners to take from this is that the tools work better in harmony than in competition.
- 19:32SPEAKER_02: Clients are increasingly open to blended approaches, partly because, as we said earlier, Gen AI is demystifying AI assisted review as a whole.
- 19:39SPEAKER_02: That openness is creating real opportunity to deploy these workflows more effectively, and the results are bearing it out.
- 19:47SPEAKER_01: Fantastic.
- 19:48SPEAKER_01: Well, before we wrap up, I'd like to ask each of you one final question.
- 19:52SPEAKER_01: If the listeners take one thing away from this episode, what should it be about using Gen AI defensibly in document review?
- 20:00SPEAKER_00: From my perspective, that there's no one size fits all workflow.
- 20:06SPEAKER_00: And the most defensible approach isn't the one that relies on the most advanced single technology.
- 20:13SPEAKER_00: It's the one that deploys the right combination of technologies with rigor and transparency at every stage.
- 20:20SPEAKER_00: As both TAR and Gen AI continue to advance, hybrid approaches will become the expected norm.
- 20:27SPEAKER_00: Teams will rely on layered proof, diverse tools, and strategic orchestration, not a single AI tool acting in isolation.
- 20:37SPEAKER_02: And what's encouraging is that this isn't a feature aspiration.
- 20:41SPEAKER_02: It's already happening now.
- 20:43SPEAKER_02: The hybrid approaches we've been discussing today are being applied in live matters, and the results are burning out.
- 20:48SPEAKER_02: Teams are working faster with greater confidence in the quality of what they're producing and with methodologies that they can explain, justify, and stand behind when they're challenged.
- 21:00SPEAKER_01: I think that's a strong note to end on, and perhaps the banner for this episode can be that the hybrid workflow isn't a compromise or a fallback, it's an upgrade.
- 21:08SPEAKER_01: And as data volumes grow and regulatory expectations increase, that layered, flexible approach is set to become the standard for faster, more defensible disclosure.
- 21:17SPEAKER_01: Craig, Kiva, thank you both, and thank you to everyone listening.
- 21:21SPEAKER_01: If this episode has raised questions about how a hybrid review workflow might apply to your own matters or how to think about the governance around Gen AI assisted disclosure, Craig, Kiva, and I will be very happy to explore that with you.
- 21:33SPEAKER_01: In the next episode, the misdiagnosis effect, finding the right solution to solve the right problem, we will be looking further into the gap between what clients think they need and what actually solves their problem.
- 21:47SPEAKER_02: Thanks, everyone.