Latest / The Adults In The AI Room / Earley AI Podcast - Episode 95: Contract Intelligence, Context Engineering, and Building AI That Scales with Deepak Bapat

Transcript

Earley AI Podcast - Episode 95: Contract Intelligence, Context Engineering, and Building AI That Scales with Deepak Bapat · The Adults In The AI Room

• 0:01
  SPEAKER_00: This is the Early AI Podcast hosted by Seth Early.
• 0:04
  SPEAKER_00: Join us as we delve deep into the passions, expertise, and experiences of thought leaders and practitioners to talk about what's possible with artificial intelligence.
• 0:14
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science for a digital transformation journey with design and deployment of innovative technology solutions, as well as Fector.
• 0:26
  SPEAKER_00: Now enjoy the show.
• 0:28
  SPEAKER_01: Well, welcome to the Early AI Podcast.
• 0:31
  SPEAKER_01: I am your host, Seth Early, and in each episode we explore how artificial intelligence and data are reshaping business strategy and operations.
• 0:39
  SPEAKER_01: Today we're going to be talking about one of the more technically demanding applications of AI in the enterprise, and that is extracting structured accountable data from complex legal agreements and contracts at scale.
• 0:51
  SPEAKER_01: It sounds straightforward, but it really is not.
• 0:54
  SPEAKER_01: There's a lot of detail and nuance today.
• 0:56
  SPEAKER_01: Joining me is Deepak Bapat.
• 0:58
  SPEAKER_01: He's co-founder and CTO at TABS, which is a revenue and accounts receivable management platform built for B2B companies.
• 1:05
  SPEAKER_01: Deepak brings a deep technical background spanning high-frequency trading infrastructure, hardware, software systems at Latch, and now AI-powered contract intelligence at TAPS.
• 1:15
  SPEAKER_01: Deepak, welcome to the show.
• 1:16
  SPEAKER_03: Thanks for thanks for having me, Sat.
• 1:18
  SPEAKER_01: So let's start off with misconceptions and we'll get into some deeper misconceptions specifically about your tools.
• 1:24
  SPEAKER_01: Of course.
• 1:25
  SPEAKER_01: But give me what do you run into when you talk to executives, generally speaking, about AI?
• 1:32
  SPEAKER_01: And it can be about your particular problem space as well.
• 1:35
  SPEAKER_03: Yeah, I I think the biggest thing that we run into consistently on our side is actually this idea that difficult problems can more and just more and more be easily solved by throwing something into Chat GPT and having the answer come out the other side.
• 1:52
  SPEAKER_03: So for in our case, we get a lot of misconceptions around hey, why can't I just take all my contracts, throw them into a chat GPT or a Claude, and just get the outputs that you all have?
• 2:04
  SPEAKER_03: And it is a it is a it is a fair question to ask.
• 2:07
  SPEAKER_03: But I but I think one of the things that we always come back to, and our mission statement here at TABS is to make complex revenue simple at scale is the at scale piece.
• 2:17
  SPEAKER_03: And and explaining this idea that the intelligence tools that we have today, especially the ones that make it really easy for users to use, Claude, ChatGPT, even at the organizational level, are still individualized tools.
• 2:30
  SPEAKER_03: And to do things at scale for an entire enterprise, my personal opinion, still takes an amount of specific tooling and specific thought and specific uh expertise compared to just like horizontal generalized model.
• 2:46
  SPEAKER_01: I think it's really critically important to be integrated into the workflows.
• 2:50
  SPEAKER_01: And that's one of the things you're saying.
• 2:51
  SPEAKER_01: It's like if you have all of these uh initiatives and all of these people using uh a large language model or Cloud or Gemini or ChatGPT, they are doing it at a at a fragment, fragmented level where you're not collecting the enterprise intelligence and you're not detecting those patterns at scale, and you're not integrating with your other tools and systems.
• 3:11
  SPEAKER_01: It's uh it is interesting.
• 3:13
  SPEAKER_01: And you know, we'll we'll talk a little bit more about some of that.
• 3:16
  SPEAKER_01: So the so that's one misconception.
• 3:18
  SPEAKER_01: We've also we also hear that people are uh thinking that software is going to disappear uh into your into the chat interface.
• 3:27
  SPEAKER_02: Yeah.
• 3:27
  SPEAKER_01: What are your thoughts about that?
• 3:29
  SPEAKER_01: That's the Sastocalypse claim, right?
• 3:32
  SPEAKER_03: Yeah, I think I think there's a couple thoughts here.
• 3:35
  SPEAKER_03: So I I do believe that whereas many years ago you could get by with a system of record with simple intelligence sitting on top, it is really important that now you embed intelligence throughout the entire plaque.
• 3:48
  SPEAKER_03: So while I don't believe that a chat interface is necessarily the correct way to be interacting with your software, I do understand the allure of being able to do more complex querying, reporting, or bulk aggregated actions across the platform more simply than what previous systems of record allowed.
• 4:11
  SPEAKER_03: So when you think about like, you know, your early billing tools or even your Salesforces, your HubSpot's pre-agents and chat and harness, someone had to go in and click a bunch of buttons in order to get something done.
• 4:23
  SPEAKER_03: The ability for us to start to understand those actions, understand the outcome that someone's trying to drive, and then preempt that is actually the more interesting thing that we need to go solve for.
• 4:34
  SPEAKER_03: But it but I I again I understand why people believe chat is a better experience in some cases than a classic UI.
• 4:44
  SPEAKER_03: The other thing I'll call out is a chat interface has its uses, like I said, for things where you're trying to get information or you're trying to spin together a series of simplified tools to achieve some output.
• 4:56
  SPEAKER_03: But in in finance, in in the domain that we're in, provability on screen is actually really important.
• 5:02
  SPEAKER_03: And on screen, being able to visualize an entire history that is both provable and accurate is is incredibly important for every one of our stakeholders.
• 5:12
  SPEAKER_01: Yeah, yeah.
• 5:13
  SPEAKER_01: I I I it's interesting when you start thinking about the the little window that a chat interface will give you versus giving you that holistic view of your information or your process.
• 5:23
  SPEAKER_01: So I I totally agree.
• 5:25
  SPEAKER_01: I do tend myself to start asking Claude for uh answers to HubSpot rather than going into HubSpot.
• 5:31
  unknown: Right.
• 5:32
  SPEAKER_02: Exactly.
• 5:33
  SPEAKER_01: But but you know, I think there's still obviously the the view of of what that application has been designed to do and what that process is designed to do.
• 5:42
  SPEAKER_01: So give us a grounding in what Tabs is built to do and what problem looks like for B2B companies that that you serve.
• 5:49
  SPEAKER_03: Yeah.
• 5:50
  SPEAKER_03: So the like I said, our Tabs is a is a is a revenue finance and billing platform, right?
• 5:56
  SPEAKER_03: So fundamentally, what we're trying to do is for companies that have complex revenue, we are trying to make it for them simple and easy at scale.
• 6:04
  SPEAKER_03: What that means is basically automating everything from closed one contract through cash and bank and through audit.
• 6:12
  SPEAKER_03: So the way I put it is contract to cash and contract to audit.
• 6:16
  SPEAKER_03: Those life cycles are what we are built around and what we are trying to do.
• 6:20
  SPEAKER_03: Everything reconciled back to the source of the source contract, the source of truth, and then any any additional cut change to that customer relationship, monitored, observable, audited well in platform, and then all of the workflows around that just happening automatically for you.
• 6:37
  SPEAKER_03: So interesting.
• 7:01
  SPEAKER_03: And in those cases, you are spending a tremendous amount of time and energy, energy trying to mold those platforms to what your business looks like today.
• 7:10
  SPEAKER_03: And the problem with doing that is that when your business inevitably changes either a month from now or six months from now or a year from now, you have to do it all over again.
• 7:19
  SPEAKER_03: And so if we can build an incredibly elastic platform that can handle any sort of business case through both understanding aggregated data in the system as well as your specific preferences, we can almost automate away a lot of billing and revenue compliance for your company.
• 7:55
  SPEAKER_03: Both the complexity and ambiguity of the jargon in the contracts themselves.
• 8:00
  SPEAKER_03: So I I think part of the reason that LLMs became a really the reason we we we kind of, and I'll be very honest, like our company was founded, was going to be a billing platform.
• 8:11
  SPEAKER_03: You don't have to be very honest, just honest is kind of very honest, you know, yeah, yeah.
• 8:15
  SPEAKER_03: That's a fair point.
• 8:17
  SPEAKER_03: Our company started out as a billing platform, anyways, specifically for hardware, hardware and software devices.
• 8:22
  SPEAKER_03: And we actually came about about the same time as Chat GPT launched for the first time.
• 8:28
  SPEAKER_03: And so in that moment, there was like a very specific aha moment where we we recognized that in the past, because reading a 400, 500-page document is incredibly complex using OCR and old school machine learning tools, humans still needed to be the ones to manually input all the information into the platform itself.
• 8:49
  SPEAKER_03: Well, now you could have an LM, and over time these models are gonna get better and better and better.
• 8:53
  SPEAKER_03: You could have an LM actually reason over the words in the document and understand what it meant and understand what the output should be given a series of instructions.
• 9:05
  SPEAKER_03: And that's kind of where this, the whole impetus for the the problem and solution came about.
• 9:11
  SPEAKER_03: And actually, our initial product was just a contract processing product.
• 9:15
  SPEAKER_03: We processed your contract, pulled out key information, and then we realized billing, like we could combine this billing idea with it, and we took all.
• 9:22
  SPEAKER_01: And you started as a hardware software team and pivoted to receivables.
• 9:26
  SPEAKER_01: What was the what told you that CFOs cared enough to build a company around that?
• 9:30
  SPEAKER_03: It was so I when I joined, I was brought in obviously for my technical expertise, and I didn't write a line of code for six months.
• 9:37
  SPEAKER_03: All we did was take customer and prospect calls, right?
• 9:41
  SPEAKER_03: It was all design and development, talking to CFOs, talking to VPs of finance, controllers, et cetera.
• 9:46
  SPEAKER_03: And just over time you realize they're all saying the same thing.
• 9:49
  SPEAKER_03: It is not just after the invoices are generated, or it's not about the any one specific problem.
• 9:57
  SPEAKER_03: The problem is a customer relationship problem.
• 9:59
  SPEAKER_03: How do I understand from a financial perspective what the relationship is between me and my customer?
• 10:05
  SPEAKER_03: And that all starts from the contract.
• 10:07
  SPEAKER_03: And so Tabs right now is primarily used by sales-led growth companies because the complex contracts are the basis of the customer relationship and they are the thing that are hardest to administer.
• 10:18
  SPEAKER_01: Interesting.
• 10:19
  SPEAKER_01: That's great.
• 10:20
  SPEAKER_01: And and so uh when you mentioned GPT 2 and and uh there was a wave of PDF extractors that were just arriving as you were creating these ideas, extraction accuracy was not that good, right?
• 10:33
  SPEAKER_01: Yeah.
• 10:34
  SPEAKER_01: Around 50% of your golden data set.
• 10:36
  SPEAKER_01: What what why why put the company on a capability that was not good enough yet?
• 10:40
  SPEAKER_01: You just had that vision of where it was going.
• 10:42
  SPEAKER_01: What did you do to kind of uh compensate for that?
• 10:45
  SPEAKER_03: Yeah, so I I I think it's fair to say that like when the initial uh Chat GPT launched and OpenAI created their first APIs around GPT, um, it was it was very impressive, but it was limited in what it could do, right?
• 11:00
  SPEAKER_03: The context window was was smaller, the outputs weren't perfect, and so we actually didn't rely heavily on in the early days, we didn't rely heavily on GPT too much.
• 11:12
  SPEAKER_03: What we did is we broke down all of our all of the things we were trying to extract and broke down the pipeline into a series of atomic jobs.
• 11:19
  SPEAKER_03: And and that is actually carried on throughout the entire lifespan of what we have done.
• 11:24
  SPEAKER_03: So at the time, it might be hey, the first thing you need to do is obviously chunk uh the contract.
• 11:30
  SPEAKER_03: Because at the time, context windows were much smaller.
• 11:32
  SPEAKER_03: And then what we're gonna do is we're gonna label the types of information we find in each chunk, and then we're gonna do the next step, and then we're gonna, and we broke it down tremendously, but at the end of it all, we still had a human in the loop early on.
• 11:45
  SPEAKER_03: Where we are at today is by the end of this year, somewhere between 95 and 99 percent of our contracts will have no human in the loop involvement because both the models and our understanding of how they work for our use case have gotten so good.
• 11:58
  SPEAKER_01: Interesting, very interesting.
• 12:00
  SPEAKER_01: So uh I want to kind of go back to your background and how you went from you know HFT infrastructure to YX to Latch to you know what sort of crypto to tabs.
• 12:13
  SPEAKER_01: So there's been a lot of kind of resets in a way of different areas, high frequency trading to hardware security to finance automation.
• 12:20
  SPEAKER_01: What made you keep changing disciplines and what transfers from one to the next?
• 12:24
  SPEAKER_03: Yeah.
• 12:24
  SPEAKER_03: So I I I think part of it was, and again, in in in the with in full transparency, like part of it was searching for the interesting problem.
• 12:35
  SPEAKER_03: So early on, it was the the HFT thing was kind of interesting because working in low level C, C, sometimes assembly, trying to optimize for very specific right instructions at that level, and then moving on to trying to build software for in the B2B space, but that was elegant and could be used by lots of different people to a hardware software product that allowed people to access their, you know, their buildings and their their living space.
• 13:06
  SPEAKER_03: Like every problem is hard and at the time for me, and every problem was unique and interesting in its own way.
• 13:13
  SPEAKER_03: And so it it was almost like when I started to lose, when I started to maybe lose the inspiration that got me to where that place was, I would, I would figure out the next thing to go do.
• 13:25
  SPEAKER_03: And as you can see, like it wasn't, it wasn't like one year, one year, one year.
• 13:29
  SPEAKER_03: Like I spent a good amount of time at each place because I did want to build up some level of of skill at the thing.
• 13:34
  SPEAKER_03: But I think what has carried on throughout the entire process is like always trying to bring everything back to first principles, always trying to bring a systems level approach to things.
• 13:42
  SPEAKER_03: Like that's what you get taught in engineering school and whatnot.
• 13:45
  SPEAKER_03: And I and I would like to think that I have been able to do that relatively successfully.
• 13:50
  SPEAKER_03: Um, some might say different, but relatively successfully across across these different domains.
• 13:54
  SPEAKER_03: But honestly, it's it's just been like, what's an interesting domain?
• 13:58
  SPEAKER_03: Screw it, like life's too short.
• 13:59
  SPEAKER_03: Let's just go.
• 14:00
  SPEAKER_03: Right.
• 14:00
  SPEAKER_01: That's great.
• 14:01
  SPEAKER_01: That's great.
• 14:02
  SPEAKER_01: And and I do want to dig in a little bit at Latch because you ran that entire back end stack for the correct connected buildings and walk us through what smart access led to in terms of a bigger vision where widespread adoption really changes delivery, logistics, uh, even traffic patterns across the city.
• 14:21
  SPEAKER_01: So it starts off with this very specific problem, all these downstream ramifications and implications.
• 14:27
  SPEAKER_01: And, you know, so talk a little bit about that.
• 14:30
  SPEAKER_03: Yeah, I think for I think for a startup to be exciting and enticing, like you need a goal and an outcome bigger than whatever it is that you're doing, right?
• 14:40
  SPEAKER_03: Like I think that's true of why, you know, investor relations is is such an important part of a part of the game.
• 14:46
  SPEAKER_03: But at Latch specifically, I remember the goal that we always had was if we are in enough buildings in a city, what becomes possible is the ability to change delivery and logistic patterns because access is no longer the gating factor for the majority of people in that in that city.
• 15:03
  SPEAKER_03: So you can imagine.
• 15:05
  SPEAKER_03: Talk about the problem.
• 15:06
  SPEAKER_03: No.
• 15:06
  SPEAKER_03: So you can you can imagine that like in most cities, the reason that UPS and FedEx only delivers between certain time periods or your uh your like delivery trucks that are doing food to restaurants only deliver at certain time periods is because their ability to access the places to hand off packages or drop off food is limited by the time period in which someone can monitor it or someone can be there to allow them to do so.
• 15:33
  SPEAKER_03: And so what does it look like for us to remove these constraints?
• 15:37
  SPEAKER_03: And I think that this is really where it comes from.
• 15:39
  SPEAKER_03: It's like, can we fundamentally remove people's constraints in order to just allow them to prosper or flourish in whatever it is that they're doing?
• 15:49
  SPEAKER_03: And what this would have allowed for in a in any city that at the time, right, when we were thinking about latch 2016, if you can have every latch and a camera and whatnot on every single door or every single space, what that allows for is the level of trust.
• 16:04
  SPEAKER_03: So FedEx can go deliver in the middle of the night and your food trucks can go deliver at two o'clock in the morning.
• 16:10
  SPEAKER_03: And now they're not coming, they're not they're not clogging up the streets between nine and five or nine and six.
• 16:15
  SPEAKER_03: And and it fundamentally changes traffic patterns in in a lot of places.
• 16:20
  SPEAKER_03: And so high level, like that was the grand vision of what we were trying to achieve at Latch.
• 16:24
  SPEAKER_03: And it's it's it's always fun to work towards that sort of vision and see how everything you're building moves towards that.
• 16:31
  SPEAKER_01: And that is so interesting, again, because you wouldn't really think of that as you know, the you're solving a short short-term problem that has all these great implications and and and very fascinating.
• 16:42
  SPEAKER_01: So one of the things that you learned, or you can't became convinced that that startups can't truly succeed remotely.
• 16:48
  SPEAKER_01: Why why does building in person matter so much?
• 16:51
  SPEAKER_03: Yeah, I uh I'm not saying that startups can't see remotely.
• 16:54
  SPEAKER_03: I know that I, as a startup founder, would not be able to succeed with my startup remotely.
• 16:59
  SPEAKER_03: I do not have the ability to administer a team of people remotely.
• 17:03
  SPEAKER_03: And that's because I I believe I can pull the best out of people and pull the best ideas out of people when I'm in a room with them on a whiteboard.
• 17:11
  SPEAKER_03: Like the energy that we get from being in the office five days a week here at Tabs is incredible.
• 17:18
  SPEAKER_03: It is like the, it is truly the like the thing that wait gets me up in the morning and gets me excited to go to work is the idea that I'm gonna get to be in a room with a bunch of people working on really hard problems.
• 17:30
  SPEAKER_03: And I don't and throughout the entire, you know, COVID era where we were remote, I just that didn't happen.
• 17:36
  SPEAKER_03: That it was it was very much like uh we used to have a joke where it was like I would I would hire people, I would talk with them every single day, and I'd have no idea how tall they were, right?
• 17:45
  SPEAKER_03: Like, like that idea is just fundamentally.
• 17:48
  SPEAKER_01: Yeah, it's funny.
• 17:49
  SPEAKER_01: When I met one of my folks who I've been working with remotely, I was like, wow, you're really tall.
• 17:53
  SPEAKER_01: I did not know that.
• 17:54
  SPEAKER_01: So when you think so, let's talk a little bit about uh architecture.
• 17:59
  SPEAKER_01: Well, you know, you needed to move from 50% accuracy into 90s, in the 90s of accuracy.
• 18:05
  SPEAKER_01: You could fine tune, you could engineer context, you could build UX for correction.
• 18:10
  SPEAKER_01: What why did fine tuning not live up to your needs?
• 18:14
  SPEAKER_01: And why did context engineering win?
• 18:16
  SPEAKER_01: You want to talk a little bit about that?
• 18:17
  SPEAKER_03: Yeah, I think the models just improved so quickly in that in that period of a couple like a couple, like that year and a half where we're experimenting and trying a bunch of stuff, that we quickly realized like, look, fine tune fine tuning is awesome because it optimizes on the data that you have.
• 18:34
  SPEAKER_03: Yeah, but a lot of what we're trying to do is be manipulatable and elastic for data that doesn't yet exist.
• 18:40
  SPEAKER_03: Yeah.
• 18:41
  SPEAKER_03: And so for us, when we have a new merchant that brings new contracts or an existing merchant that has a new pricing model or whatever, like instead of baking that into an expensive fine-tuning process, like being able to just use an existing, generalized, really good smart model and just engineering the context around it is a much faster avenue to the same outcome.
• 19:02
  SPEAKER_01: Why don't you just define for people who may not be as familiar with the the details of what fine tuning is and what context engineering is?
• 19:12
  SPEAKER_01: Like fundamentally, what are they?
• 19:14
  SPEAKER_01: Because again, you get a model that's tuned for a particular task.
• 19:18
  SPEAKER_01: And uh, you know, one of the ideas is a GPT, generalized pre-trained transformer, right?
• 19:23
  SPEAKER_01: Yeah.
• 19:23
  SPEAKER_01: Meaning it's it's it's trained on certain tasks and then it's it's being able to transform and be able to be applied to other tasks.
• 19:29
  SPEAKER_01: And so tell me a little bit about what your thoughts are there.
• 19:32
  SPEAKER_01: And I have my concepts, but but go ahead and talk a little bit about your thing, your thinking there.
• 19:37
  SPEAKER_03: So the idea behind fine-tuning is you start with a generalized model and then you apply more data to it such that you can change the weights inside of the model based on whatever you know data set you provide.
• 19:50
  SPEAKER_03: So what it should do is it should increase the weights of edges in the model and that will allow an outcome more closely resembling the pattern that you are fine-tuning on.
• 20:01
  SPEAKER_03: All context engineering is is the ability, you can think of it as, in my opinion, you can think of it as a more mature form of prompt engineering where you are where you are taking, it's it's not just the prompt you write, but under allowing the model as part of its input to understand the context under which some variable data is being provided.
• 20:25
  SPEAKER_03: And therefore, at the generalized state, for these large language models, especially the the new the the new ones, they are very good at taking all of that context and understanding how it applies to your to your the input that you're actually trying to drive an output on, because the windows, the context windows are so so large.
• 20:45
  SPEAKER_03: And so both so large and the you know, there's so many uh parameters in the models themselves.
• 20:50
  SPEAKER_03: And so what we found is over time, if you have a small enough job and you have a good enough model, it doesn't have to be the model that was most released that's on the frontier, but a good enough model to do that small enough job, you really don't need more than just the right information going into the platform in order to provide the output.
• 21:12
  SPEAKER_01: And and and when you think about fine-tuning a model or training a model, that is that is a large, larger initiative because you have a larger data set.
• 21:20
  SPEAKER_01: And what you're doing is you're trying to change the actual functionality of what that model is.
• 21:25
  SPEAKER_01: When you're changing weights, you're saying, let's let's adjust this to uh this particular data.
• 21:31
  SPEAKER_01: So you're in the life sciences space, exactly a whole lot of life sciences content and lots of databases and so on.
• 21:38
  SPEAKER_01: It's gonna perform at an overarching level more effectively than a generalized model.
• 21:43
  SPEAKER_01: For that specific for that specific domain.
• 21:46
  SPEAKER_01: Yes, exactly.
• 21:47
  SPEAKER_01: And then on top of that, what you could say is well, rather than retrain this whole entire model, right?
• 21:53
  SPEAKER_01: I'm gonna just provide the context in terms of in Claude, it's a project and product and knowledge, right?
• 21:58
  SPEAKER_01: Project knowledge.
• 22:00
  SPEAKER_01: Or or in terms of retrieval, retrieval, augmented generation, where we're planning the data.
• 22:05
  SPEAKER_03: Getting the right data, structuring it in the right way so that the model can understand it most effectively.
• 22:11
  SPEAKER_01: That's right.
• 22:12
  SPEAKER_01: And that is kind of a they're they're achieving similar things through different mechanisms.
• 22:17
  SPEAKER_01: And then in some and industries, you're doing both, right?
• 22:20
  SPEAKER_01: You're using a specialized model that is fine-tuned for the domain, and you're also providing this contextualized data sources to provide the context.
• 22:29
  SPEAKER_01: So when we provision a claw project, we have all our IP that has been harvested and structured in certain ways for that project to do things, right?
• 22:38
  SPEAKER_01: And then, you know, for decades worth of IP, right?
• 22:41
  SPEAKER_01: So, you know, so what it does is it knows enough about those processes to be able to solve that particular problem by ingesting all of that knowledge and expertise.
• 22:50
  SPEAKER_01: And that does kind of bring you to a different space in the vector world, in the vector space to be able to handle that particular type of content.
• 22:59
  SPEAKER_01: It's just like saying you were playing this role, you're doing this task, you're doing the more specific we can be with those prompts.
• 23:05
  SPEAKER_01: Go ahead.
• 23:05
  SPEAKER_03: And in my opinion, that works really well if you have a ton of aggregated data and the output across any new input generally is the same.
• 23:13
  SPEAKER_03: The problem for our space is that at the merchant level, preferences matter so much that it's very difficult to fine-tune across a lot of different merchants like data simultaneously.
• 23:25
  SPEAKER_03: So if you're gonna fine-tune it has to be merchant by merchant, which becomes almost cost ineffective.
• 23:30
  SPEAKER_01: Exactly.
• 23:31
  SPEAKER_01: Exactly.
• 23:31
  SPEAKER_01: So that's why the con you provide the context of that merchant.
• 23:34
  SPEAKER_01: Here are my procedures, here are my guidelines, here are my preferences, here are my terms.
• 23:38
  SPEAKER_01: Exactly.
• 23:39
  SPEAKER_01: This is all about my world.
• 23:41
  SPEAKER_01: Now make the decisions in that context.
• 23:43
  SPEAKER_01: Exactly.
• 23:44
  SPEAKER_01: Yep, yep, yep.
• 23:45
  SPEAKER_01: Great.
• 23:46
  SPEAKER_01: And uh so, so, so, you know, walk us through the pipeline.
• 23:50
  SPEAKER_01: You you validate a contract against prior ones with embeddings, you classify it, then you extract.
• 23:55
  SPEAKER_01: And you talked about this before, breaking into small atomic jobs rather than one large model call.
• 24:00
  SPEAKER_01: And you had a very specific reason for doing that before because of limitations of the tools and the platforms and the models, but you've carried that on because there are some benefits to that.
• 24:09
  SPEAKER_01: So talk a little bit about how that works.
• 24:11
  SPEAKER_03: Yeah.
• 24:12
  SPEAKER_03: So a good example of this is there's different types of jobs that we need to do through the pipeline, right?
• 24:17
  SPEAKER_03: There are classification jobs, there are discovery jobs.
• 24:20
  SPEAKER_03: So, like read through the contract and and annotate the contract.
• 24:24
  SPEAKER_03: There are jobs around what is the for an amendment, for instance.
• 24:28
  SPEAKER_03: It's like, what are the set of things that are being amended versus, okay, now that we've decided what is being amended, like now tell me how they're being amended.
• 24:37
  SPEAKER_03: Each of those specific jobs, look, there will come a point, I'm sure, where someone will build a model that is good enough where you can pass it in and say, do all thousand of these things put into this giant schema and deliver it uh in Providence and like we're good.
• 24:55
  SPEAKER_03: And right, I we just aren't there yet.
• 24:57
  SPEAKER_03: Like, in fact, what we have found that a lot of the models, especially both the open source and the closed source models that are being delivered today, are actually getting less, their size is getting bigger, but they're actually getting less and less efficient at non-coding related tasks, surprisingly.
• 25:15
  SPEAKER_03: Like there are models from six months ago that for any of these jobs actually perform better.
• 25:21
  SPEAKER_03: We have a bunch of eval sets that we run than newer versions of those same models.
• 25:26
  SPEAKER_03: Yeah.
• 25:26
  SPEAKER_03: I go ahead, Pete, keep going.
• 25:28
  SPEAKER_03: Yeah.
• 25:28
  SPEAKER_03: And and I think part of that is because this is not like truly, when we think about what these models are training, being trained on, what people want these models to be used for, I really believe that there's an overfitting on coding that some of these models are undergoing.
• 25:43
  SPEAKER_01: Well, I have found uh, you know, I use uh clawed projects, you know, with our methodologies, right?
• 25:48
  SPEAKER_01: So I will say, here's our frameworks, here's our you know, our approaches, here are examples, here are templates, you know, here's deliverables, here are definitions, here's instructions, right?
• 25:57
  SPEAKER_01: And they're very, very precisely provisioned, right?
• 26:00
  SPEAKER_01: So I have, you know, SharePoint and Dropbox sites that I use to be able to provision a new project, right, with methodologies.
• 26:08
  SPEAKER_01: But I still need to go through multiple steps to that process, right?
• 26:12
  SPEAKER_01: I still have about six different major operations that need to be done in order to take process interviews and be able to have an output of a maturity model and a roadmap and AI readiness scores and all of that.
• 26:24
  SPEAKER_01: That's right.
• 26:25
  SPEAKER_01: It still saves weeks and weeks of time, right?
• 26:28
  SPEAKER_01: Yes.
• 26:29
  SPEAKER_01: But I found that three months ago, six months ago, to your point, I could take those interview uh transcripts, input them into one location, into in one project with my templates, and I would get the answer, right?
• 26:42
  SPEAKER_01: And now it's requiring that I that I do break it down in in these interesting ways.
• 26:48
  SPEAKER_01: And I'm finding that yes, because because for whatever reason it just said this chat is too long because it looked too much stuff.
• 26:55
  SPEAKER_01: Exactly.
• 26:56
  SPEAKER_01: The context window, it chewed up my whole context window, and then I could not load everything and do it in one operation.
• 27:01
  SPEAKER_01: I had to break it down.
• 27:02
  SPEAKER_03: Yes, that's that's a good point.
• 27:04
  SPEAKER_03: Like I do think that for a lot of cases, like the overloading, like just because you have a million tokens that can fit into a context window doesn't mean you want to use all million tokens as as part of your input, right?
• 27:15
  SPEAKER_03: Or like input and output.
• 27:17
  SPEAKER_03: So like there is some of that.
• 27:19
  SPEAKER_03: One thing, yeah, exactly.
• 27:21
  SPEAKER_03: No, it's just like it's impractical.
• 27:23
  SPEAKER_03: It's impractical and it's like it actually doesn't perform as well.
• 27:26
  SPEAKER_03: Like I think there is a there's an upper if you look at what how like the trajectory of of efficiency, like there's definitely a a peak in terms of of context window percentage.
• 27:37
  SPEAKER_03: And I think it changes per model, but fascinating when you think about it, right?
• 27:40
  SPEAKER_01: Yeah, it's it's like they've gotten better in some ways, but then they've they've they've changed.
• 27:45
  SPEAKER_01: I guess you can't say they can say they're better, they've improved in some ways, but that's right, but regressed in other ways.
• 27:51
  SPEAKER_03: The other thing to call out, right, is also like the price of the model may be the same per token, but newer, like if you look at the opuses, for instance, if you put the same problem into 4.5 and 4.8, 4.8 might actually use more tokens in order to solve the problem, and therefore it is more expensive.
• 28:08
  SPEAKER_03: And so, in fact, sometimes it is better to use smaller models at a more atomic level, it's actually more cost efficient.
• 28:15
  SPEAKER_01: Yeah, totally agree.
• 28:17
  SPEAKER_01: Yep, I found that completely to be true.
• 28:19
  SPEAKER_01: That's so interesting, isn't it?
• 28:21
  SPEAKER_01: So it's really fitness to purpose around the size of the model, the task, and the context, which is incredibly fascinating when you think about all those different variables, right?
• 28:32
  SPEAKER_01: Yeah, I'm I'm finding that you know, I'm fragmenting my knowledge in Claude projects, right?
• 28:37
  SPEAKER_01: Because I'll be in one context of you know, writing a discovery letter, and then I'll say, Well, I really need this to now be a state of work.
• 28:44
  SPEAKER_01: Now I have to switch projects and I have to pull the context from the first project to the second project, or then I'm saying, well, wait a minute, I really want to be able to build the methodology and the work plan from this.
• 28:53
  SPEAKER_01: Well, wait a minute, do I have to go into another project for that?
• 28:55
  SPEAKER_01: Because this one doesn't have all that, and it can't handle it in terms of the size of the chats or the context windows, right?
• 29:02
  SPEAKER_01: So you absolutely have to do that.
• 29:03
  SPEAKER_01: Yeah.
• 29:03
  SPEAKER_03: And I think this is why drop-in tools or like platforms are still and SaaS is still important.
• 29:10
  SPEAKER_03: Yes.
• 29:10
  SPEAKER_03: Because because our my my ability in my platform to be non-fragmented across an entire set of jobs to be done for a finance professional is incredibly important for that finance professional.
• 29:23
  SPEAKER_03: Yeah.
• 29:23
  SPEAKER_03: Yeah.
• 29:24
  SPEAKER_03: Versus, you know, if they were to do this in Claude with Zapier and X and Y or automations or scheduled tasks, like they would be, they would have like 50 different things that projects and tasks and artifacts that they would be using.
• 29:36
  SPEAKER_01: Yeah.
• 29:37
  SPEAKER_01: And and and that complexity continues to multiply.
• 29:40
  SPEAKER_03: Exactly.
• 29:41
  SPEAKER_01: As you build more capabilities, as you have more projects, as you enable more processes.
• 29:45
  SPEAKER_01: So it really does argue for that platform approach because the platform is integrating these things.
• 29:50
  SPEAKER_01: It's not fragmenting the knowledge, it's not fragmenting the processes, and you're able to have this in a more holistic approach with greater efficiencies through token usage, greater efficiencies through processes, and all of that.
• 30:03
  SPEAKER_01: So that really is a very strong argument for the platform and against the SESPOCLYPS, right?
• 30:08
  SPEAKER_01: Yes.
• 30:09
  SPEAKER_01: Yeah, exactly.
• 30:10
  SPEAKER_01: I think that's a great point.
• 30:11
  SPEAKER_01: So you bucket merchant's contracts using principal component analysis.
• 30:15
  SPEAKER_01: Then so what do they what do these things have in common, right?
• 30:18
  SPEAKER_01: And then you measure how far a new contract sits from the nearest bucket and pull a human in when it falls too far outside.
• 30:24
  SPEAKER_01: How do you decide where that line lives?
• 30:26
  SPEAKER_03: I mean, it's it's just uh we use quantile sampling and it's just in some ways, it's an arbitrary decision line that that we put in, right?
• 30:33
  SPEAKER_03: And so it is this is where being able to generalize is incredibly important.
• 30:38
  SPEAKER_03: Like we need to build it in such a way such that every merchant can use it.
• 30:43
  SPEAKER_03: And what we're trying to aim for is like P95 merchants don't feel that it's noise.
• 30:48
  SPEAKER_03: And if 5% feel that it's noise, like yes, we may have to do something special for them, maybe turn it off or whatever, but but this is the hard part.
• 30:55
  SPEAKER_03: This is the the part that is uh that requires a ton of data.
• 30:59
  SPEAKER_03: You have to constantly be taking in the feedback from the users in order to understand and be able to aggregate that data and make decisions off it.
• 31:06
  SPEAKER_03: I mean, that is at the end of the day, really just product engineering.
• 31:09
  SPEAKER_01: Yep, yep, yep, that makes sense.
• 31:11
  SPEAKER_01: Yeah.
• 31:12
  SPEAKER_01: So you don't necessarily pair a schema with every contract, you keep a generic schema, then you generate a context object per bucket that helps the model extract the data into.
• 31:22
  SPEAKER_01: Why is that more scalable than fitting contracts into its own schema?
• 31:26
  SPEAKER_03: Yeah, a single generalized schema allows you more easily to run deterministic workflows without significant overhead per merchant, right?
• 31:36
  SPEAKER_03: And so our goal is to build the most flexible schema and then make sure that we have enough context engineering in place that the schema can be fit correctly.
• 31:45
  SPEAKER_03: I think it's a I think it's a fair question.
• 31:47
  SPEAKER_03: I think at some point there, the the world may move to individualized, like you know, how people talk about like gener generative UI.
• 31:54
  SPEAKER_03: Like people might, we might start building generative data models and individualized data models per your use case.
• 32:00
  SPEAKER_03: But I think we're still a little bit far from that.
• 32:03
  SPEAKER_03: And I actually find that really, again, what we're solving for is that P95.
• 32:07
  SPEAKER_03: Like if we can get 95% of merchants working with a generalized schema, like we have still done our job.
• 32:14
  SPEAKER_03: And it makes that like hitting 95% of merchants' schema such that their lives become easier is significant, significantly better and significantly less uh costly than trying to optimize the schema at a per-merchant level.
• 32:27
  SPEAKER_01: Yep.
• 32:28
  SPEAKER_01: Totally makes sense.
• 32:29
  SPEAKER_01: So, you know, what you've described, I mean, a lot of what we talked about is, you know, there's no AI without IA, right?
• 32:34
  SPEAKER_01: There's no artificial intelligence without information architecture.
• 32:37
  SPEAKER_01: And so what you're talking about is you're saying you're describing classification, controlled structure, progressive enrichment, feeding the model.
• 32:43
  SPEAKER_01: How uh much of the accuracy comes from the models and how much comes from the information architecture underneath it?
• 32:50
  SPEAKER_01: That's exactly what we're just talking about.
• 32:51
  SPEAKER_03: I think it's I think it's yeah, I think it is a lot of look, I think it's a lot of information architecture that leads to the context engineering and the the reinforced learning that we we do internally.
• 33:02
  SPEAKER_03: Like, you know, our we have a generic extraction process, and that generic extraction process on a on a set of eval, like on an eval set that is quite expansive in what it covers, hits about 87%.
• 33:15
  SPEAKER_03: And if for each merchant we can get above 95%, additionally, that 87% is not on the newest model, which the newer models fare much worse than older models.
• 33:27
  SPEAKER_03: Interesting.
• 33:28
  SPEAKER_03: So my general take is our ability to get the right data, to structure the right data, and to feed that into the model and to like classic ML systems and deterministic logic, such that a single contract comes out the other end at a 95% plus accuracy.
• 33:47
  SPEAKER_03: I think it is it is a ton of data engineering and information engineering for all intents and purposes.
• 33:53
  SPEAKER_01: Yeah, yeah.
• 33:54
  SPEAKER_01: So that is uh it really does point to the the fact that this is not automatic and this is not exactly happen without those reference models.
• 34:03
  SPEAKER_01: And those reference models are gonna be super important moving forward.
• 34:06
  SPEAKER_01: So, you know, there's also there's also the perspective of uh, you know, engineers saying, Oh, you know, let's look at this contract extraction and I'll use clawed code and and I'll I'll build it myself.
• 34:20
  SPEAKER_01: And what what if what do they not understand, right?
• 34:23
  SPEAKER_01: Because again, you you can build an application.
• 34:25
  SPEAKER_01: So if someone says, okay, yeah, I make sense.
• 34:27
  SPEAKER_01: I don't want to just do it as a one-off, I want to build an application and I can do that much more quickly.
• 34:32
  SPEAKER_01: So what what are they missing there when and uh obviously a lot, yeah, but just articulate something.
• 34:37
  SPEAKER_03: Yeah, there's a couple things.
• 34:38
  SPEAKER_03: So I think look, if you have simple contracts that are like and you don't have that many of them, I think you get your engineering team or a semi-technical person can just go and build this.
• 34:49
  SPEAKER_03: If you don't plan on changing gear contracts, I think a semi-technical person can go build this.
• 34:53
  SPEAKER_03: But if you once you start getting into complex finance and financial compliance, it starts to get difficult because what you're actually trying to do is you're not just trying to say, is the the model produces the JSON and it's correct.
• 35:06
  SPEAKER_03: What you're trying to say is I can prove that this is correct.
• 35:09
  SPEAKER_03: And so half of what we are building is a system that allows what we call our merchants to prove that invoices and revenue are as they are expected to be based on rules of engagement that they have defined, data that is available on, you know, ASC 606 compliance for revenue.
• 35:28
  SPEAKER_03: Like all of the data that is that is necessary to inform provability of our outcomes is part of what the system is built for.
• 35:37
  SPEAKER_03: The other thing the system is built for, right, is like we handle amendments, we handle renewals, we handle versioning over the life of the customer relationship, which is a much more difficult problem to solve.
• 35:47
  SPEAKER_03: Because if you take an amendment and you pop that into a into a into an engine that you've built, and then someone comes along and says, hey, I want to do amend something completely differently, and now you're trying to pop it, it doesn't work as well.
• 35:59
  SPEAKER_03: If you have a, like I said, if you have a standard set of contracts, every field could you could take that contract and reasonably turn it into a standard set of fields, and you know that someone will go fill those fields out correctly every time.
• 36:13
  SPEAKER_03: I think you can go and do it.
• 36:15
  SPEAKER_03: But but my guess is, and I've seen we've seen this for almost every one of our customers, that is just not the case.
• 36:21
  SPEAKER_01: Right, right.
• 36:22
  SPEAKER_01: The larger the enterprise, the more complexity and variability there will be, the more diverse the enterprise, different business units, different departments, different, you know, functions will have different requirements, and and it's going to very quickly become something that is not sustainable from that, you know, I'll I'll code it with, you know, clawed code, right?
• 36:42
  SPEAKER_01: Perspective.
• 36:42
  SPEAKER_01: And what about the accounting packages that, you know, people are saying, well, existing systems handle contracts, revenue recognition.
• 36:49
  SPEAKER_01: What are they missing when they make that assumption that their accounting packages can handle these things?
• 36:53
  SPEAKER_01: And of course, those are we're gonna get better and more, you know, had more capabilities, but where are they and and tell us a little bit about that, those gaps?
• 37:01
  SPEAKER_01: Sorry, what do you mean accounting packages?
• 37:03
  SPEAKER_03: Well, you know, my accounting software, my ERP system.
• 37:05
  SPEAKER_03: Oh, I see, I see, I see.
• 37:06
  SPEAKER_03: Yeah.
• 37:07
  SPEAKER_03: A lot of those are still built around structured data.
• 37:10
  SPEAKER_03: So it's you must have structured data.
• 37:13
  SPEAKER_03: We have built you specific, we've written specific code for your current use case that allows us to take this and say, yes, of course this is provable over here because here's the code that's been written.
• 37:25
  SPEAKER_03: But you have an exception.
• 37:26
  SPEAKER_03: You want to change that.
• 37:27
  SPEAKER_03: Now we got to do that whole process over again.
• 37:30
  SPEAKER_03: The whole idea behind tabs is if you if you if you assume that a customer relationship, revenue data or revenue rules, if you assume that a lot of this is in natural language and unstructured, and you must move from an uns a place of unstructured data to provable and correct structured data, like that is what tabs is built around, which a lot of these other systems cannot handle.
• 37:51
  SPEAKER_03: Sure, sure.
• 37:52
  SPEAKER_01: So I think we're in good shape here is if if for finance or data leader wants to wants trustworthy AI running on their contracts, where do they start and what uh foundational work cannot be skipped, would you say?
• 38:06
  SPEAKER_03: Yeah, it's it's the basic stuff.
• 38:08
  SPEAKER_03: It's understanding what you want your outcomes to be and being able to reasonably articulate them, right?
• 38:13
  SPEAKER_03: One of the things that we find is that as part of our implementation process, oftentimes we're going in and as we are starting to pull in contracts and cal like we have an entire calibration flow where we take all your contracts, we figure out the representative set, we run calibration on those, et cetera, et cetera.
• 38:31
  SPEAKER_03: As we're going through that process, we realize the other side needs to clean up how they think about contracts.
• 38:37
  SPEAKER_03: They need to, they need to coordinate amongst themselves how what outcome they're trying to drive.
• 38:42
  SPEAKER_03: On if you have two contracts and they have very similar language, there are there are companies that we work with today where you will get, if you ask two different people on their team, you'll get two different outcomes.
• 38:51
  SPEAKER_03: And every and so part of part of the job is just internally standardizing on what you want.
• 38:58
  SPEAKER_03: And this is just the case in any in anything, right?
• 39:00
  SPEAKER_03: Like as companies scale, half of our job is to make sure that you are you are thinking about scale outside of just throw it into an LM, get whatever output it is, and go do something with it.
• 39:10
  SPEAKER_03: Like you were thinking about it from a systems perspective.
• 39:12
  SPEAKER_01: Yeah, that makes great sense.
• 39:14
  SPEAKER_01: Well, listen, I think that's all the time we have for today.
• 39:18
  SPEAKER_01: So I really want to thank you for spending your your uh your time with us.
• 39:22
  SPEAKER_01: I appreciate your insight.
• 39:24
  SPEAKER_01: Thank you very much for uh being on the on the podcast.
• 39:26
  SPEAKER_01: Thanks, and thank you to all of our listeners uh for tuning into the Early AI podcast.
• 39:31
  SPEAKER_01: Be sure to subscribe for more conversations on how AI is shaping the future of business.
• 39:38
  SPEAKER_00: Thank you for joining us on another deep dive into AI innovation.
• 39:42
  SPEAKER_00: Tune in next time when we introduce another industry expert and discuss how to maximize AI in your world.
• 39:48
  SPEAKER_00: The Early AI Podcast is sponsored by Early Information Science and Vector.
• 39:54
  SPEAKER_00: To learn more, visit early.com.
• 39:56
  SPEAKER_00: That's E A R L E Y dot com.
• 39:59
  SPEAKER_00: Thank you for tuning in.