Latest / Agentic Conversations (formally mlops.community) / RAG Quality Starts with Data Quality // Adam Kamor // #262

Transcript

RAG Quality Starts with Data Quality // Adam Kamor // #262 · Agentic Conversations (formally mlops.community)

0:00
Hey everyone, my name is Adam K Moore.
0:02
I'm the Co founder and head of engineering at Tonic dot AI and
0:06
I take my coffee black using a normal coffee machine like with
0:14
the Philtre, the drip. The drip coffee I think is what
0:17
it's called and I use Folgers because that's what they have at
0:21
Costco. Welcome back to the MLF
0:25
Community Podcast. Surprise, surprise, today we're
0:27
talking about data, not just any kind of data, the data that is a
0:31
little bit more sensitive that PII data and how when you are
0:36
dealing with rags, when you've got that chatbot that you think
0:39
is going to change the whole direction of your company, it's
0:43
going to inform all of your leaders on the status of
0:47
projects. How do you make sure that all
0:51
that sensitive data does not get leaked by the chatbot?
0:55
Well, we get into it today with Adam.
0:58
Huge shout out to the folks at Tonic for supporting the
1:02
community on this one. If you enjoy this episode, feel
1:05
free to share it with one friend.
1:08
And let's get into the show. So we got to start with the name
1:20
of the tool. Did you guys know what you were
1:24
doing when you called it textual?
1:26
Because in my mind, getting textual with somebody is
1:29
something totally different. We, we realised that connection
1:35
after we came out with the name. It's like, you know, we come up
1:39
with the name and I start using it in all of like the sales
1:41
calls and for like the third or fourth sales call, I'm like,
1:45
wait a minute. And then it kind of hit me and
1:48
it has been an ongoing annoying joke and within the company
1:53
since then. I, I think it, you know, there's
1:57
that saying among like, like in software development, like the
1:59
two hardest things are like, like cash problems and variable
2:03
naming. Well, product naming is way
2:06
harder. And by the way, the biggest
2:09
constraint in naming a product is figuring out what domains are
2:13
available. Of course, like that, Kate, when
2:16
we started Tonic AI, like before we, we were like naming our
2:18
products like specific names. We went with Tonic AI because
2:22
like everything else is taken, man.
2:24
There are there's just no domains out there.
2:27
And unless you're willing to just like, like really, really
2:31
misspell the name of your product.
2:32
Oh yeah, then you just can't get a good domain.
2:35
And I refuse to. Yeah, I hate the miss the
2:37
misspelling of products as it's impossible to search for them.
2:40
Exactly. Or you go with some random dot
2:44
like that and the dot iOS got popular the dot SOS are pushing
2:48
it. Or then you have I've seen you
2:51
can go with the.ca or the.co and.
2:55
That's right. That's right.
2:56
I, I mean, we're dot AI and I mean, so yeah, I you're loud and
2:59
clear that that was an issue that we had as well.
3:02
Well, good call on Tonic because that gives an aura of sleekness
3:10
and like a gin and tonic type thing.
3:12
Yeah, and you know. We actually had a product for a
3:14
while named Gin, but we spelled it.
3:18
We didn't spell it. GIN actually going to not
3:21
spelling things correctly. We spelled it DJINN like, you
3:24
know, like a gin, like a genie. Oh yeah, yeah, it was.
3:27
Kind of like a clever play on that.
3:28
But I, I, I think it was perhaps too clever.
3:31
And no, really. Got it.
3:32
Yeah, yeah, sometimes you can outdo yourself and yeah, that's
3:37
right. So the but Speaking of textual,
3:39
let's get textual right now. Yes, I did do that.
3:43
I I can only imagine how much you hear that that's.
3:45
Actually a great I might put that on T-shirts for
3:47
conferences. Let's get textual.
3:49
There you go. That's great textual.
3:51
Textual healing, perhaps? Exactly, that is 100% what I was
3:57
thinking of so. All right.
3:59
Thank you that. Is not exactly the road we're
4:02
going to go down right now though.
4:03
I will switch it up because yeah, yeah, I mean, we could go
4:08
PG13 if you aren't, but we've been known to go R sometimes.
4:14
But for this very thing, I think what you all are doing with
4:19
textual is, is really cool and it's going to the data layer and
4:25
making sure that some of these challenges that you face in the
4:30
data layer can help you downstream when you're working
4:33
with AI and ML. And so can you just give us this
4:38
overarching idea of what textual is real fast so we can
4:42
understand it? And then I'll I'll dive into
4:44
some of the challenges that come off of it or it helps to curb.
4:48
Yeah, absolutely. So Tonic Textual is a tool that
4:53
is used to build top quality data pipelines for people that
4:58
are building RAG systems. We, we, we got started on this
5:03
because we realised quickly when we were building our own like
5:05
internal RAG solutions, that data quality really Trump's
5:09
everything else when it comes to getting good quality answers
5:14
from your RAG system. If you're not providing your LLM
5:17
with the right context, then especially like when you're
5:20
asking questions about like private data, you're, you're
5:23
just never going to get the right answer.
5:24
So building a pipelining tool that actually gives you like
5:29
good quality chunks where you're kind of like optimising for RAG
5:31
retrieval is really the best way to building a good RAG system.
5:37
So the obvious statement here is you got to have the quality data
5:46
before you can do anything with it, right?
5:49
And I've heard a lot of people complain about how well my
5:52
knowledge assistant would be much better if people actually
5:56
documented properly in my company.
5:58
Yeah, that's, yeah. Well, and yeah, you have to have
6:02
the data that that's definitely the prerequisite.
6:03
I, I, you know, I, I can't do anything about that for you,
6:05
unfortunately. I'd say getting people to
6:07
document is not something that that anyone is going to ever
6:10
solve. Yeah, but yeah.
6:12
Doesn't that scare you? Doesn't it feel like you're
6:14
built on found a House of Cards there?
6:17
Because it's like documenting, getting people just fighting.
6:19
Human nature every step of the way.
6:21
No, I mean, so like when, when, when you look at like most
6:24
enterprises, they're not even enterprises, just really
6:26
companies of any size. They, they, they always have,
6:29
you know, very large amounts of data.
6:32
I mean, sometimes too much data, honestly, where, where like you
6:35
actually don't even need all of it.
6:37
And a lot of it can be a distraction for what you're
6:38
trying to do. And especially as organisations
6:43
are starting to like, like what we, what we were seeing early
6:45
on, and it's still mostly true today is that companies are
6:49
building rag systems to solve internal problems.
6:52
Like here's our HR chat bot that, you know, employees can go
6:55
to to ask HR questions or you know, here, here's my chat bot
7:00
where people can ask about the company handbook, you know,
7:02
things of this nature. So like internal chat bots that
7:05
were mostly, I don't want to say that they were there for
7:08
entertainment. They're certainly solving a
7:10
need, but people were doing things that were, I think
7:13
relatively straightforward to begin with to kind of figure out
7:15
what's going on. Because this is new to
7:16
everybody. And we're starting to see like
7:19
organisations starting to like build more serious RAG systems
7:23
as as time progresses. And then we're now starting to
7:26
see companies building externally facing chat box or
7:30
RAG systems. And then when you get into the
7:33
externally basing ones, it's typically being used by their
7:35
customers and it's incorporating customers personal data into the
7:40
experience. And that's where we really come
7:43
to play when we're dealing with information that isn't just
7:46
publicly available and like a company's policies or on some
7:49
like internal ocean page, like we're getting into like the
7:52
actual data of the enterprise that involves their customers.
7:56
And then that's where we prefer to focus.
7:59
Yeah. And I know one of the huge
8:03
benefits of this is you don't get that leakage.
8:06
And the I have a fairly full understanding of how you can
8:12
swap out a lot of this PII because I've looked at what you
8:15
all are doing before and I've gotten textual we could say.
8:19
But maybe you can give me a quick overview or the rest of us
8:24
a quick overview of like how you make sure what data stays and
8:29
what data goes and what is not safe versus safe.
8:32
Sure. So behind Tonic Textual is a set
8:37
of proprietary like built in house NER models.
8:41
So NER is a subset of NLP, which is natural language processing
8:44
and NER means named entity recognition.
8:47
It's essentially a machine learning solution that
8:51
identifies entities within unstructured text.
8:54
So I say, hey, my name is Adam, I work at Tonic.
8:58
You know, presumably the entities that we would pull out
9:00
from that are Adam being a name and Tonic being a company.
9:03
So that that would be an example of named entity recognition.
9:07
All of the data that you bring into Tonic Textual, you know,
9:10
for the purposes of building your data pipeline, run through
9:14
our NER models to identify entities that that are A
9:18
sensitive so that we can kind of address any sensitive
9:21
information to prevent data leakage and B entities that are
9:25
interesting, like not all entities we find are sensitive.
9:29
Like, yeah, OK, sure. We can find SSNS and credit card
9:31
numbers. Those should probably never land
9:33
in your RAG system. But we can also identify things
9:36
like what product is being discussed in this customer
9:39
service transcript or like, OK, what, what company does this
9:44
person work at that might not be sensitive, right?
9:47
And then we can use the entities that we find to actually build
9:51
for you better chunks that can optimise your RAG retrieval to
9:55
get better quality answers. So all of the data goes through
9:59
these NER models and we address anything that's sensitive
10:01
through a variety of factors or a variety of techniques rather.
10:05
And then we also kind of augment your chunks with additional
10:09
entity metadata that we find. So, you know, we talked about
10:12
both quality and safety of the data in general.
10:15
OK, and I know that chunking is one of the hardest things,
10:20
making sure that a you don't have too many chunks, but you
10:24
give it enough context, right? And have you figured out some
10:28
tricks around that? I mean tricks, yes, you know,
10:32
general solution that works for everybody perfectly every time.
10:35
No, of course not. I don't think such a thing
10:37
exists. It might not even be possible.
10:41
What we our approach is of course our solution ships with,
10:45
you know, different chunking techniques that that we've
10:47
developed. I, I wouldn't what to claim any
10:50
of them are state-of-the-art or revolutionary.
10:53
Our approach instead is that our system plugs very well into the
10:58
existing work you've already done and you can really
11:00
integrate into like the workflow that we provide to you any
11:04
chunking algorithm you like. So, you know, you might already
11:06
have your own like purpose built chunking algorithm that you,
11:10
that you think is ideal for your data that you developed in
11:12
house. And you can, you can work that
11:14
into your, you know, textual pipeline without issue.
11:17
I'm so, you know, we basically take this approach
11:19
out-of-the-box. We have something for you if you
11:21
think you can do better. And maybe you can, you could
11:24
also bring in your own algorithm.
11:27
Nice. I, I think that the idea that
11:29
like there's going to be like 11 chunky algorithm to rule them
11:32
all. I, I don't think it'll, I don't
11:36
think you know we're there yet. No, I like how you say that it
11:41
is very case by case basis because we don't really
11:46
understand until you have lots of cycles and lots of iterations
11:50
on what this looks like and how it performs on this specific use
11:55
case with this set of data or this embedding model that we're
12:00
using, whatever it may be. You've got a lot of different
12:02
variables in there. That's right, I'll give a real
12:05
trivial example. Let's say you are building a
12:08
rack system where some of the documents are like FAQs, right?
12:12
You know, typically for an FAQ, what I've seen work very well is
12:15
you want to chunk each question answer pair separately.
12:18
So you kind of like don't really look at like length as much, but
12:20
it's just like, OK, here's the question, here's the answer.
12:23
That's one chunk and then go to the next.
12:25
Like that. That's a very simple solution,
12:28
but it's also incredibly bespoke and does not apply to like any
12:31
other document out there, right? So like, you know, it's like you
12:35
might not even have one chunky algorithm for your entire
12:37
corpus. It might be like, this is what I
12:39
do with these types of documents.
12:40
This is what I do with these other types of documents.
12:42
It could really be all over the place.
12:45
Dude, what a good point. That is how you don't need to be
12:49
stuck with one chunking algorithm.
12:51
And no, no, figure out what the best algorithm is for the
12:57
different documents that you're working with.
12:59
That's right. And I think this good, like that
13:02
idea, like, I don't know, we, we, you know, we've been, we've
13:05
been doing rag stuff now for a while.
13:07
And, and we, it, it's, it gets more complex every day.
13:11
I'm not saying like that's like the most complex concepts in the
13:14
world or anything, but like it's one example of like, you know,
13:17
things are like ramping up, you know, quickly in this space.
13:20
Yeah, I always laugh because it's a little bit to me, like we
13:26
are engineering band aids, but not really going to the root
13:31
cause of what the problem is that these LLMS can spit out
13:38
things that are absolutely made-up sometimes.
13:40
And so we have to try and do all this engineering work to make
13:44
sure that we cover our asses so that it doesn't happen.
13:48
That's right. It's, it is, it's a very
13:51
challenging, I mean, it's all, it's all very challenging.
13:54
Like I, I think you, you can get to like a decent quality system.
13:57
I think, I'm not going to say with ease, but like you can, you
14:00
can get something that mostly works out there quickly and then
14:04
dealing with the edge cases and like the occasional
14:07
hallucination. Like it, it just the, the
14:10
complexity goes high quick. It's like, you know, the last
14:13
10% is the hardest that that that type.
14:15
Exactly. Couldn't.
14:17
It's so true. So aside from scrubbing the PII
14:22
or the entities that are actually worth scrubbing versus
14:26
ones that can give you insight and making sure that you're
14:31
chunking properly depending on documents and your algorithms
14:35
you're using, are there any other things that you feel like
14:39
are essential for this data preparation before you're
14:42
sending it to the RAG or when you're using the data for RAG
14:47
use cases? So I like, let me give you the,
14:50
the common workflow that we see among our, our users.
14:53
And I think that that paints a pretty good picture of both of
14:55
our capabilities. And really, like, I think what
14:58
should be done when building these pipelines.
15:00
The first is like, you kind of need to understand, OK, well,
15:03
what kind of documents do I have and where do they live?
15:06
Like every, every document type, you know, there's a different
15:08
way of extracting out the relevant information.
15:10
Like in a Word document, a Word document actually is just a zip
15:14
file. It's a zip archive.
15:16
And inside the zip archive is a bunch of XML.
15:19
So programmatically you can actually traverse through like
15:21
Word documents, Excel sheets, PowerPoints, etcetera.
15:24
You can programmatically traverse through these files and
15:27
extract out all of the content and material you need.
15:29
Like within these XML files is all of the information you need
15:32
about these documents. So for for office documents, you
15:36
know, extraction is just a, a programming problem.
15:39
PDFs on the other hand, typically you're using some type
15:41
of OCR process to extract information and then you have
15:45
other file types like images, which is also OCR plain text,
15:50
which is relatively straightforward.
15:53
But you know, for every file, it's like you got to consider
15:55
how am I going to extract out this information?
15:57
Then you have the question, OK, where does my data live?
16:00
You know, we commonly see data being stored in like S3 and the
16:04
and like, you know, Azure BLOB Store and GCS, but it's also
16:07
going to be stored in like Confluence pages and Jared
16:10
tickets and. Notion docs.
16:13
Etcetera, right. So how are you going to access
16:14
this data? And then you have this question
16:17
of, OK, well, in all of these file systems, and I'm using the
16:20
Word file system loosely here, how often are files changing?
16:24
Like what process do I need to go through to ensure that like
16:27
as files are being added, modified and deleted, how is how
16:30
are those changes like, you know, percolating into my vector
16:32
database? OK.
16:34
And obviously I'm bringing this up because, you know, we have a
16:36
solution to it, which I'll get to momentarily.
16:39
So then, OK, you've identified like where the files live, what
16:41
the files are, how frequently they're changing.
16:44
And then what essentially you want to do is begin extracting
16:48
out all of the information that you need, which typically means
16:50
just like converting all of these documents into some
16:53
version of plain text, we prefer to use Markdown.
16:56
So what we do is we convert all of your complex file formats
16:59
into Markdown, which is plain text with some, you know, syntax
17:04
sugar that you could add on top, right?
17:06
And it's nice because, like you can represent complex file
17:09
formats like PDFs and Word documents in plain text and
17:12
preserve a lot of the layout and formatting still.
17:15
Like you get like, you know, lists, tables, headers, titles,
17:19
you know, things of that nature. OK, let me stop you right there
17:22
because you are bringing up such a great point that goes right in
17:25
line with the idea of how important data quality is for
17:30
your AI systems. How are you confident or how can
17:37
you know you as a a user or an engineer that's creating these
17:42
systems know that when I just ingested that PDF and it
17:45
references, there's a paragraph in there that references table
17:49
one. Table 1 is in there labelled
17:53
table one. All that fun stuff, you know,
17:55
which is, I've heard an incredibly challenging problem
17:58
when you're trying to do it at Yale.
18:00
Yeah, it, it, it is. So I mean, the, the, the first
18:04
step is you need to be able to extract tables from your
18:07
documents. And that means like identifying,
18:10
you know, something, some section of the PDF actually has
18:13
a table in it. And, and especially like in, in,
18:16
you know, in an Excel sheet, it's, it's pretty obvious in
18:19
Microsoft Word, you can make a table object and add rows and
18:21
columns. That's all relatively
18:23
straightforward. But in APDF the, the table could
18:26
be anything. It could even be like very like
18:30
artistic in nature where it's like like a designer crafted a
18:33
table like that that is highly stylized.
18:36
So there's like, so the first step in what you're asking is,
18:39
OK, well, how do you identify tables and documents and PDFs?
18:43
That alone is pretty challenging.
18:46
There's a few techniques out there and there's a few
18:48
libraries that'll just kind of solve it for you.
18:50
And I say solve meaning like they do decent like a comment
18:54
OCR system that most folks know is Tesseract.
18:57
Tesseract out-of-the-box doesn't support like table
19:00
identification. You can use.
19:02
There's some other open source OCR systems that do a better job
19:05
like paddle OCR is one that I've looked into recently that
19:09
actually does support table detection.
19:11
And then there's also like cloud providers like you know, Azure
19:14
has the document intelligence service and AW is text tracked.
19:17
And these OCR providers actually support table identification and
19:20
like extraction out-of-the-box. So, you know, you could just go
19:24
with one of these, you know, cloud providers and get this
19:26
table extraction. There's also cool techniques
19:28
actually that combine using tesseracts like this open source
19:32
OCR engine along with an LLM and kind of combining the two to
19:37
identify tables, like extract all the text with text tract,
19:40
send it to the LLM, ask it to kind of format this in the way
19:44
that you might expect. And it'll typically do a pretty
19:47
good job at identifying tables for you.
19:49
So that's another approach you have, but you got to figure out
19:52
how you're going to do your table identification.
19:53
And of course, you know, this goes without saying.
19:56
Tonic textual supports the identification and extraction of
19:58
tables as well as key value pairs, which is another like
20:01
pretty common concept in, in, in OCR documents.
20:06
Like imagine a tax form. Like it's got the name box, you
20:08
put your name, it's got the SSN box, you put your SSN.
20:11
Like, you know, the, the, the keys would be name in SSN and
20:14
the values would be the name in SSN itself.
20:17
Yeah. And then, you know, the second
20:20
point you're asking, OK, well, like there's a paragraph that
20:22
references the table. How do you tie them together?
20:26
Yep, typically it requires like a second pass be done with some
20:30
additional model that can kind of help associate things
20:33
together. And, and it's, it's not easy,
20:36
but like what a lot of folks will do is they will use like
20:41
they're, they're like they'll, they'll typically take like
20:43
follow up passes with like large language models.
20:46
It could be open AI or it could be like a lower weight or a, a
20:49
lower parameter count model that they deployed locally.
20:52
But they're going to have to do a second pass with an LLM to
20:54
kind of like draw those associations.
20:56
Yeah, that's awesome. All right, continue back with
21:00
the rest of what you were talking about, where I just had
21:03
to ask because, dude, that is such a difficult problem to get
21:08
into and really have the confidence that, all right, I
21:13
just ingested two gigs of PDFs. What?
21:17
Or let's say 2 terabytes of PDFs.
21:20
Yeah. How do I know that all these
21:22
tables are A showing the right data, B, being referenced and
21:27
connected correctly. And so I like this, a very
21:31
robust way of going about it and just throw another model at it
21:35
when in doubt basically. I know I, I, I don't like
21:39
saying, oh, just run it through an LLMA second time.
21:43
I don't, you know, but like they're incredibly effective at
21:46
those types of challenges. Yeah.
21:48
And like, look, in an ideal world, you have your data
21:51
science team, you guys go train your own birth specific model to
21:53
do it. And you can do that and it'll
21:56
run faster and more efficiently and you can run it for cheaper,
21:59
but it's gonna take time to develop.
22:01
And at the end of the day, it might be faster just to go send
22:03
it to like your LLM of choice. Yeah.
22:06
And it all comes down to like, you know, the Co pros and cons
22:10
and like the opportunity cost for your specific situation.
22:14
Yeah, right. So for example, like in Tonic
22:17
Textual out-of-the-box, we don't use any large language models.
22:21
All of the models that we kind of had developed and that we
22:23
ship with, they're all proprietary models trained and
22:27
fine-tuned by us. Typically we focus on Bert or
22:30
Roberta models as like our, our like core architecture and then
22:33
we fine to it and improve on top of those.
22:37
But we found those to just be more cost effective and better
22:41
at like very specific tasks that we're doing.
22:43
Like specifically like our NER models, there's no LLM in use
22:47
for any of our NER detection. Yeah, I was going to say
22:51
potentially what you're looking at is better, cheaper, faster,
22:53
right? Yes, it is.
22:55
It's, it's more expensive to develop on the front end because
22:58
we have to go train, we have to go manually annotate large
23:03
bodies of text so that we can kind of feed these models.
23:06
But once you got to build out that machinery, it becomes
23:09
actually relatively straightforward and quick to
23:11
like improve, add new entities to detect things of that nature.
23:15
And that's kind of where we're at now, yeah.
23:17
I imagine once you developed one and you saw the payoff, it's
23:22
hard to go back to or do anything else besides that, even
23:25
though you have that initial upfront cost.
23:28
Yeah. And as you're going through it,
23:30
you're thinking to yourself, I don't know, should we just use
23:34
an LLM? It might end up being cheaper.
23:37
But after you see that this actually works, no, I can see
23:42
why you would only want to use that.
23:44
Yeah, that, that, that's right. I mean it's essentially like it.
23:47
It also really pays off for our customers in terms of like cost
23:50
of operating both in terms of like how long it takes to run
23:54
like, you know, a, a, a ARNER models like running on like
23:58
commodity hardware. You know, it's 10s of thousands
24:01
of words per second that are being scanned.
24:03
And you can scale that out like, you know, to whatever desired
24:06
throughput you want. And like, you just won't get
24:08
that from it's just, no, it's not going to happen at this
24:13
point. And if you could, it would
24:14
probably be very expensive. So back, back to that question
24:18
about like, OK, you know, what are folks doing to build these
24:20
pipelines? So, you know, they, they,
24:22
they've begun extracting all the data.
24:23
Our approach, and I think this works well is we give her all of
24:26
your documents into markdown. And we do this in like I, I
24:29
think a really high quality way where the markdown is really
24:32
representative of the original document.
24:34
But then we also provide our users with essentially a
24:37
structured output of their document.
24:39
So you know, think of our solution essentially as taking
24:42
every unstructured document you have sitting in S3 for example,
24:46
and creating a structured counterpart to it.
24:49
Our counterparts are typically Jason.
24:51
So essentially every document you have gets converted into a
24:54
Jason document regardless of the file type and make the details
24:58
of the file. Every Jason document has the
25:00
same like structure and schema to it.
25:03
So you essentially end up now with a huge body of Jason
25:07
documents that are all schematically structurally the
25:10
same that you can interact with for building your back system in
25:13
lieu of the original files and the different file formats that
25:17
you have. And then on top of this, you
25:19
know, body of Jason that we give you, we provide like nice
25:22
utilities and SDKS for like kind of interacting with it and then
25:26
like quickly and efficiently getting the Jason into your data
25:29
pipeline. So that that is typically the
25:31
approach we take. But the data isn't.
25:34
So you basically transform the data, but then you throw it back
25:39
into S3 or you have that in the database.
25:44
Where does that? It's really like up to the user.
25:47
Like by default, if the user, for example has all of their
25:50
PDFs or Word docs or whatever in S3, then we're gonna dump a
25:54
bunch of Jason documents into S3 as well.
25:56
Like you kind of specify where, but they'll be sitting in S3 and
26:00
it's very easy then to programmatically like interact
26:02
with these documents via our like SDK.
26:05
And then from there you can do whatever you like.
26:07
So for example, our solution is also available in the Snowflake
26:10
native app Marketplace. So for customers that want to
26:13
keep all of their compute within Snowflake, you can run our
26:16
entire solution on your Snowflake compute.
26:18
Everything stays within Snowflake all like all the web
26:20
calls, all of the models are running in Snowflake, etcetera.
26:23
And in that solution, we don't dump into S3.
26:26
That doesn't make as much sense. We're writing these Jason blobs
26:29
directly into a Snowflake table. Snowflake has a column type
26:32
called a variant, which is basically, it's not exactly
26:37
Jason, but it's very analogous to Jason.
26:39
And we're just writing all of the results into a variant
26:41
column in a Snowflake table, and then you can programmatically
26:44
access it that way. Nice.
26:46
So you're ingesting and then transforming this data and
26:52
making it real pretty for anybody to use with I I can
26:58
imagine the next steps are all right.
27:00
How do we throw this at the embedding models?
27:04
Yeah. Well, how do we chunk it and
27:05
then how do we throw it at the embedding models, which is,
27:06
yeah, it's sorry, that's what you said.
27:08
So yes. Yeah, yeah, exactly.
27:11
And so, so then once you're there, there was something that
27:14
you mentioned before, which I know is another very sticky
27:19
problem. And how do you know those
27:24
documents like you, you reference JIRA tickets, right?
27:27
Those are getting updated all the time, right?
27:30
How can you ensure that if somebody is an end user of that
27:35
internal chat bot and asking a question about a project that is
27:40
continuously being updated on JIRA, they're getting the most
27:44
up to date? Information that's right.
27:46
So for all of like the different look like data locations that we
27:50
support, like on the input side, there's, there's this notion
27:53
that we have of like, I want to say this in a generic way.
27:59
You know what, I'm going to stick with S3 as my example for
28:01
now because I, I think most folks know S3 or Azure BLOB or
28:05
GCS, they're all roughly equivalent.
28:07
So let's say I upload a file to S3.
28:10
There's, there's, there's two things that can happen, right?
28:13
Maybe I upload a new file. So we want to be, we want to
28:16
know our system needs to know when a new file has been added.
28:19
Meaning we're we're reprocessing and we encounter a file we
28:22
haven't seen before. We also want to know when a file
28:26
has been changed. Meaning yeah, we've processed
28:28
this file before, but wait a second, it's checksum isn't the
28:30
same as it was the last time. And then there's the issue of
28:33
when a file is deleted and we need to know that as well.
28:35
Meaning we've seen it before, but we're not seeing it this
28:37
time. So those are kind of the three
28:39
situations that we find ourselves in.
28:40
And and if we wanted to have JIRA tickets, like the
28:43
equivalent would be like JIRA will have a notion for a ticket
28:46
of like last, last modified or last updated.
28:49
And we'll be hooking into like columns of that type in JIRA.
28:53
But regardless, whenever you process data using tonic
28:56
textual, whether it's an S3 JIRA or somewhere else, we are
28:59
keeping track of the three things that I just mentioned.
29:02
And every time we process, it's just running a delta.
29:04
So like everything we do is like a delta operation.
29:06
So the first time you run, it's the whole thing, right?
29:09
Like we have to, we have to process the entire backlog
29:12
that'll take time. The next time you run, we're
29:15
only processing the things that have changed since the previous
29:17
run. So like because of that ability,
29:20
you can be running our pipelines like as frequently as you like.
29:23
Like the more frequently you run them, the smaller the delta will
29:27
have will be each time. And then they just complete very
29:30
quickly. Like, you know, you run it every
29:31
30 seconds. Well, most of those runs the
29:34
delta is 0, nothing has happened.
29:36
And then occasionally like a file's been added, modified,
29:38
deleted, etcetera. And then we pick up on that.
29:41
And then we just provide nice hooks to the SDK user to allow
29:48
them to easily say, OK, compute the delta in files between the
29:52
the current run and the previous run.
29:53
Give me all files that have been changed and then you can kind of
29:56
like, you know, do your pipelining based on that change.
29:59
Set is typically how we approach it.
30:02
And what have you seen once you know OK, this has changed, I I
30:07
think the real sticky part happens in the vector database,
30:11
right? Oh, this file has been deleted.
30:13
Now how do I go and RIP that out of the vector database?
30:16
Yeah, that's right. So you know whether so most
30:20
common situation is the files hasn't been modified.
30:22
So in that case, skip it, Do nothing.
30:25
If the file, if a file has been added, then you're gonna go
30:27
through your chunking and embedding and you know,
30:29
inserting process just like you did for the initial backlog.
30:33
If the file's been modified, you're kind of having to do
30:36
initial initially, you'll do the deletion.
30:38
You got to delete the original chunks.
30:40
Like it's better just delete those chunks as opposed to like
30:43
trying to do like a delta on a chunk level.
30:45
Just do it on the file level, delete all the chunks for the
30:48
file and then you're gonna go and re chunk it using, you know,
30:51
whatever, you know, strategy that you have already kind of
30:54
selected. That's typically what we see.
30:58
And so, so basically, if if a certain file has been deleted,
31:02
you just go and find those chunks and delete those chunks.
31:06
Yes, that's right. And.
31:08
Yeah. And you know, it's, it's like
31:10
how to identify those chunks in the database.
31:12
It's that's left how just, you know, quoting like a textbook,
31:17
you know, it's left as an exercise to the reader.
31:20
So they they'll right, so they'll, you know, you know, you
31:23
need to set up your vector database in a way that kind of
31:25
like makes this easy and straightforward.
31:28
Like, you know, typically what we'll see is like, you know,
31:30
folks will include like a, a pointer to the file alongside
31:34
the chunk that the chunk, you know, alongside the chunk, you
31:38
know, to kind of show where the chunk came from.
31:40
So you know, some approach like that, where it's just easy to
31:43
identify which chunks came from which files makes this a a
31:46
straightforward process. Yeah, I've also heard trying to
31:49
link up the creation dates and then inside the vector database
31:54
you can also philtre for creation dates and say, OK, this
31:58
one goes here and it's true. Seems like it can get a little
32:02
bit messy. Yeah, I.
32:06
Mean it's all like a vector database is a database first and
32:10
foremost, right? Yeah, like these are very
32:11
standard database operations that like no one would bat an
32:15
eye at doing in like a traditional like RDBMS. And it's
32:21
you gotta play the same games here like, yeah, OK, you have
32:24
chunks and that's cool and you have your embeddings that's
32:26
neat. But there's other columns too.
32:28
And you know, it's still software engineering.
32:31
Yeah, nothing's changed just because if we call it a vector
32:34
database, yeah, that's great. It's.
32:36
Still a database. It falls under the umbrella of
32:38
databases, so treat it like a database, not some new fancy
32:42
thing that is specific for AI. Although it is.
32:46
But it's not. It doesn't change everything.
32:48
Yeah, that's right. I like that.
32:50
You know a good example of this. You can create your vector
32:53
database in Postgres. Just go install the PGA vector
32:55
extension. Exactly.
32:56
You get all the power in Postgres and you have your
33:00
vector columns as well. So many, yeah, I, I just saw a
33:04
company that was doing that basically their, their whole
33:10
thing was, hey, we're Postgres. We're managed Postgres.
33:13
And one of the big selling points was we can give you a
33:16
managed. Postgres database, vector
33:18
database. Vector database.
33:19
Yeah, we, we create. We when, when we were initially
33:23
building rat systems, we were just using God, I hope, I hope
33:27
this is correct. We were using just Postgres in
33:29
RDS, you know, the managed service in Amazon and I, I think
33:32
we were just installing PG vector on the boxes.
33:36
I think that, you know, I think that's what we did.
33:40
I was debating if PG vector was available in RDS then and I
33:42
believe it was. Well, I think that's best
33:45
practise, right? Like use it until you outgrow it
33:50
and then you need something more specific.
33:53
OK, make the transfer. Get one of these like the Milvis
33:58
or the Quadrant type or Lance DB, whatever it may be.
34:03
That is when you hit your limitations with what you're
34:09
already very good at. Because if you already know PG
34:14
or Postgres and you've been working with PG vector, then
34:19
that is probably going to be the best tool that you can work with
34:22
until you can't work with it. Yeah, until you, until you
34:24
can't, you know, that might never come.
34:27
So you know. Yeah, if you're live.
34:29
Advice to everyone building companies is start simple, make
34:34
no assumptions about the future, just focus on today.
34:38
I think that's. That's focus on good names and
34:41
dot AI's and dot coms. That's what you want, folks.
34:45
Thanks for including the AI in there.
34:46
I appreciate that. Yeah.
34:49
So there, there was another thing that I, I wanted to
34:52
mention when it comes to like the data prep side.
34:56
So we before we hit record, one thing that we were talking about
35:00
was role based access control and allowing certain things to
35:06
come into the database or certain people to look at the
35:09
database. And this feels like again, it
35:11
goes back to that same theory of this is a database.
35:16
So database things should happen at the database level.
35:21
And I was asking you, OK, so do you allow with text rule, do you
35:28
allow certain people to have certain access to certain files?
35:32
And you kind of push back on that.
35:35
So I would love to hear, let's like try and rehash that
35:39
conversation and act like it's the first time that we're having
35:41
it. Yeah, that's good.
35:43
That's good. So I think that, you know, the,
35:46
the question that I was supposed to be before we started
35:47
recording was does tonic textual help with, you know, granting
35:53
access, you know, on an individual basis or on a user
35:56
basis to the different portions of like the, the vector database
36:00
tables? Like, you know, chunks basically
36:02
like, can you, can you, you know, ensure that user A only
36:06
has access to this set of chunks for as user B has access to this
36:09
other set of chunks? And my response is, no, we don't
36:12
do that. I, I, I see that more as the
36:14
responsibility of the database system.
36:16
Like to me that is a very, you know, vanilla use case for row
36:19
based access control, You know, given users have access to given
36:23
rows. And, and that's really the end
36:24
of it where where we come into play more is an organisation is
36:30
going to have certain requirements on it as in terms
36:33
of like what types of user data can land in the vector database,
36:38
you know, and, and the common fear that these organisations
36:40
have, and rightfully so, is data leakage, right?
36:44
Yeah. Anything you put into the vector
36:46
database is fair game for the chat bot to regurgitate to the
36:50
end user that's interacting with it.
36:51
Yeah. So how do you deal with that?
36:53
That's really where Tonic Textual comes in.
36:55
Yeah. We give you very high quality
36:58
data to make your VAC system better.
36:59
And then this is empirically proven.
37:01
And I'm, I'm, you know, happy to go into those details, but
37:05
really what we do is we ensure that an organisation that says,
37:07
hey, these types of entities cannot land in the vector
37:11
database. They are too sensitive.
37:12
We cannot have them regurgitated.
37:13
We don't want that data leakage. That's where we come into play
37:16
with our NER models to ensure that that data is either never
37:20
included in the vector database, or it is it is somehow
37:24
rectified, typically by either redacting certain pieces of
37:27
information or by synthesising it in order to get, you know,
37:31
high quality chunks. So there's a fascinating kind of
37:37
screw up recently that happened with Slack, I don't know if you
37:40
saw it, how Slack AI can give data leakage for private
37:47
channels. So you can ask Slack AI, did you
37:49
see that? I I think I saw it on like
37:52
Hacker News or something. Honestly, I should have probably
37:55
read the whole thing because it's incredibly pertinent to
37:57
what we do, but I must have been busy that day.
38:00
So, so you know, I, I, I have the high level, but I don't have
38:03
all the details which maybe you could share.
38:05
Yeah, that so it, it's basically saying the long and short of it
38:09
is that you can do some prompt injection and get information
38:15
from the private channels out in the open when you're talking
38:21
with Slack AF. Yeah, OK.
38:23
And I, I just wonder from your approach, how would you combat
38:30
something like that? Because it's not necessarily
38:32
like you can say this channel is 100% entities that we don't
38:39
want. Maybe it's certain things that
38:41
come up we don't want others to see, but actually maybe
38:45
everything is we don't want to even touch that or put in the,
38:50
that's right, vector database. Like would you just, yeah, just
38:55
not put it in or I? Mean, yeah, certainly you can do
38:59
that and then you have a worse system because of it.
39:01
So that's not ideal, that specific problem.
39:04
And I, I, like I said, I, I don't know, you know all the
39:07
details. I, I, what you said, I'm, I'm
39:10
hearing for the first time on the surface, it seems like
39:14
that's more of like access control issue where like how,
39:19
how could I, how should I be or why would I be given access to
39:23
chunks that were generated by other users in private DMS?
39:29
So to me that that's that on the surface, it sounds like a row
39:32
based access control type deal. Let let me give you an example
39:36
of where you would want to use our solution for kind of, you
39:41
know, removing sensitive information.
39:43
Like let's say you are building a customer service chat bot that
39:48
customer service users, customer service agents can use to
39:52
quickly answer questions that come in from users, where users
39:55
might interact with themselves directly on the website.
39:59
OK, now one solution could be a user comes in, they're
40:03
authenticated. So the chunks that they can act,
40:06
the chunks that you know they can retrieve are only the only
40:09
going to be chunks that have been generated from previous
40:12
conversations of that user with the customer service system.
40:16
OK, well you could do that, but every user's probably going to
40:19
have a very small amount of data, like maybe some user only
40:22
ever calls customer service once.
40:24
So it's kind of not very helpful.
40:26
So really though, you know that like customers called in with
40:30
the same issues, like if one person's confused, then 10,000
40:34
people are confused, OK, so you want to basically use all of the
40:38
data generated by everyone to answer any individual's problem.
40:42
Well, OK, I call, you know, customer support for some place.
40:45
First thing I do. They're like, OK, what's your
40:46
name? OK, give them my first and last
40:48
name. What's your address?
40:49
What's your credit card number? Like what's your what's the last
40:51
four of your SSN? Like all of these things they
40:53
got to do to identify, you know, you are who you say you are.
40:55
You are right in that situation. Those are the types of chunks or
41:00
entities rather that we need to remove prior to like building
41:04
our RAG application and exposing this chatbot.
41:07
Because that type of information like my name has no bearing on
41:11
like the actual problem on basing or the solution or
41:15
whether or not it's going to be useful to someone else.
41:17
We need to remove that. We need to remove obviously the
41:20
last four of my SSN or my like my, my 3 digit credit card code,
41:24
you know, things of that nature. So that's more where we come
41:27
into play. So previously an organisation
41:29
just couldn't use this data. It's like, oh man, we there,
41:32
there is literally nothing we can do.
41:34
We can't touch this data because anything that we put it into the
41:37
vector database, we are just going to be in a load of pain
41:40
and trouble, right? But now they can do that because
41:43
we will go in, we'll identify every entity, every utterance
41:46
said that is potentially sensitive.
41:48
And then we can either redact it, meaning we just kind of like
41:51
strike it from the record, replace the the utterances with
41:53
tokens, or we can synthesise it. You know, given a chunk that has
41:57
a bunch of sensitive entities, we will create for you a new
41:59
chunk that is semantically identical.
42:01
But you know, Adam is replaced with John, my SSN is replaced
42:05
with a fake SSN, you know, things of that nature.
42:07
And of course it's sort of the very intelligent way where even
42:10
like the semantics across multiple chunks is still
42:12
maintained so that that's the approach we take.
42:15
So previously an organisation that couldn't use any of this
42:17
data now gets to use it safely and in a high quality way to
42:21
build their experience. Yeah.
42:23
So that first of all, I do like that you can kind of do a hot
42:30
swap so that you still retain the information that is being
42:36
said. It's just has no meaning because
42:40
it's not the person's name, it's not the company, it's not their
42:42
screen or their Social Security number.
42:45
None of the actual valuable stuff is there, but the whole
42:48
meaning is there and so you can understand it.
42:51
It hot swapping as you call it. We call it synthesis, but hot
42:55
swapping, it's actually safer. Let's say I'm given a corpus of
42:59
text and I think, OK, let's replace every name with a token.
43:03
You know, that's like redacting it essentially.
43:05
OK, well, you get now the safe text and you see Adam.
43:10
You're like the model missed Adam.
43:12
Adam's a real person in the data because every other name is
43:14
tokenized. So you know, the model missed.
43:17
And these NER models, by the way, and this is true for any
43:20
NER model out there, they miss. They're not 100%.
43:24
I mean, this is true for, yeah, any ML model, that's right.
43:29
Folks, they will miss downer. You have to consider that when
43:31
building the system. Yeah, no.
43:33
But instead, if you do this synthesis or the hotswapping
43:35
that you called it, well, OK, Demetrios goes to John Adam,
43:40
though is missed. The attacker or the end user,
43:43
whoever sees this data, you don't know.
43:47
You don't, you know you have some more plausible deniability
43:49
there. It actually gives you a safer
43:51
footing. So I just pulled up the report
43:56
on Slack AI just because I wanted to scratch my own edge.
43:59
And you really should have read this because it is.
44:03
It is talking about how it is leaking API keys, which is
44:07
totally what you could have done.
44:08
You could have saved Slack. I mean, it looks like they would
44:12
be a good customer for OK, you, you know, you guys, somebody
44:16
from the sales team should reach out to the Slack AI team because
44:20
in private channels someone set up API keys and then they put in
44:26
a public channel basically some prompt injection.
44:30
And from that, they were able to leak the API key when talking
44:34
with the LLM. And so the, The funny thing here
44:40
though, that I, I think is the sticky problem with Slack or the
44:45
the messy problem, I think we should call it, is that private
44:48
channels you, you kind of think they're going to be private.
44:51
So you might be talking shit about Karen.
44:54
And now if all of a sudden the LLM can leak that data.
45:01
Yeah, yeah, no, I can't talk shit on Karen.
45:04
That's right for her. That's right.
45:06
Like I got to do it on. Signal the work.
45:07
If you can't do that, you know it's not under anymore.
45:11
Yeah, now it's coming out. That is, that's interesting.
45:16
I mean, yeah, like I, I would expect my DMS on really any
45:22
system I use, not, not Slack in particular to certainly stay
45:26
private. And, and the thought of them
45:28
going into a, a vector database is certainly concerning.
45:32
And not even the ones where you're just like, you know, was
45:35
it, is it wait, is it spilling tea?
45:38
Is it? That's the phrase for gossiping.
45:40
Yeah, I'm showing my. I think so.
45:42
That's what the cool kids are calling it these days.
45:44
So not even those situations, but imagine it's like the the
45:47
private channel of like leadership and like they're
45:49
talking about like, you know, presumably like important
45:52
things. Yeah.
45:53
Or yeah, just important stuff. Yeah, that's right.
45:57
Important stuff. That's right.
45:58
I don't know. You're you're on the leadership
46:00
team, right? You know what those
46:01
conversations are? Yeah, yeah, absolutely.
46:03
It's not just spilling tea. There's other things discussed.
46:06
Too, Yeah, there's there's a quote unquote important stuff
46:08
happening like the direction of the company and you want to be
46:12
able to have those conversations in private before you bring them
46:15
out to the team. But if somebody can prompt
46:18
inject the LLM, then that's a little bit dangerous.
46:22
And this, this is true of this, this is going to be true of
46:25
like, you know, a company comes in saying, OK, we're building a
46:29
rack system on our documents in Notion or in Google Drive or
46:33
really in any system where like there's this notion of like
46:36
shared private, you know, this group can see it, this group
46:40
can't. You know, you have to, like, you
46:42
have to like, you know, when I create a document in Google
46:45
Drive or in, in like in Google Sheets or whatever, like I have
46:49
faith that the sharing functionality works and, and
46:53
just cuz I've used it for years, right?
46:55
And but it's complex and like, if you're going to be like
46:58
building something on top of that, you need to be as good as
47:01
Google Drive is in that situation.
47:03
Well, what a great point because it's not only Slack that has the
47:08
notion of private channels and the public channels.
47:11
It's everything that we're using any documents, If you're on
47:14
Notion and you have your private workspace or you have your
47:17
private documents in there that you've shared with a few people
47:21
to socialise around, do you want that to be publicly questionable
47:27
through the QA bot, like the chat bot?
47:30
And how, how have you seen best practises on what to do?
47:35
So what's private stays private, and what's public is public.
47:40
So we, we don't take a like our, our product doesn't take a
47:45
position here. It's like our, our product
47:48
typically does not it, it, it, it doesn't like hook into like,
47:54
you know, the sharing and like the, the access controls of
47:56
given documents. You can certainly feed that into
48:00
our system like as metadata as you're processing documents and
48:02
like we can assist in, in landing the metadata in the
48:05
right place. But in terms of like what, what
48:09
happens once the data goes into the vector database and who can
48:11
access it that, that it's, it's really up to the end user.
48:15
And really, in my opinion, like the responsibility of the
48:18
database system to provide like reasonable, like RBAC.
48:22
I think the problem that you're describing though, like the
48:25
ultimate form of this problem is we have tonnes of data, we need
48:30
to make it shareable among everyone, right?
48:33
Like the customer service example I gave her, I would have
48:35
trained my model on all customer service engagements, not per
48:37
user. So how do I, how do I basically
48:40
in a world where you have tonnes of documents and there is
48:43
implicit sharing in the sense that like, you know, everyone's
48:47
discussing their own private stuff, but you have no
48:49
structured way of knowing what's what, how do you deal with that?
48:52
And and that's where we shine just because of like what we're
48:54
able to do in terms of like identifying in information
48:58
within documents in an unstructured way.
49:01
Yeah. And in the ways that you've seen
49:03
customers building their chat bots, have you seen much of
49:08
these design patterns of like trying to have some kind of
49:14
guard rails for the question or guard rails on the output?
49:18
And or is it mainly around just hey, the database has got to
49:22
take the brunt of this one and it's got to be the roles on
49:26
that? Well you know, I thought you
49:27
were going a different direction with this.
49:29
When I hear guard rails, I, my mind immediately goes to guard
49:33
rails around like answer quality and preventing hallucinations
49:39
and ensuring that like the answer given isn't just like
49:42
complete and utter nonsense. I, I, I the, the guardrails in
49:49
terms of like who can see what? I, I, I don't, I'm not really
49:54
Privy to those like portions of like the setup of our customers.
49:57
Like what we try to do for our customers is help them easily
50:02
build a safe and like high quality data pipeline where we
50:06
try to optimise your retrieval metrics.
50:09
Meaning for a given question, you're always retrieving the
50:12
optimal set of chunks or the optimal collection of chunks.
50:16
Like that. That's where we typically draw
50:18
the line and down the stream of that tonic textual doesn't
50:23
doesn't kind of go further. Yeah, you don't have the.
50:27
There's like things on our road map that we're doing that that I
50:29
think actually will end up going further over time, but beyond
50:33
right now, like the data layer, we we kind of stop playing at
50:36
the moment. Yeah.
50:39
And since it feels like in the last six months Graph Rag has
50:44
have an explosion. Do you have thoughts on that?
50:49
I am interested to see where it goes, it is I guess my only
50:54
qualm with it at the moment and I don't even know if it's really
50:57
a a problem I have or just just an.
51:00
It's more of an observation really.
51:02
It, it, it is relatively expensive right now to build out
51:05
your graph. And it's expensive because it
51:07
typically requires an obscene number of LLM calls in order to
51:12
like kind of like build out the connections between documents or
51:15
chunks or just depending on what level you're working at.
51:18
We're, we're working on our own technology here at the moment,
51:21
kind of powered by Legar entity recognition to help like
51:25
identify connections between different documents and between
51:28
different chunks. I think the first version of
51:30
this we're, we're aiming to kind of release over the next week or
51:33
two. And it's, it's not going to be
51:35
as powerful as like as a graph rack approach at the moment, but
51:38
it'll eventually get there. And it, it's more like a
51:42
technique for grouping documents together that all pertain to a
51:46
same or similar topic. So it's more of like a document
51:49
level topic analysis that's being done.
51:52
And I, I think from there we'll, we'll likely move to finer grain
51:55
like per chunk and then building in like, you know, ways to
51:58
connect documents and chunks together.
52:01
But I'm, I'm excited by graph rag and approaches like it.
52:04
I, I, I think there will be a way to do it that is like
52:08
reasonable for like large enterprise sets of data.
52:12
And I, I, I think hopefully you know, we're the ones to do it,
52:15
but you know, that will tell. Nice, I cannot wait to hear what
52:20
name you come up with for the. Yeah, I mean, you, you have a,
52:24
you have a, you have a good take on all that.
52:26
Maybe you'll if you think of something funny, you know puns
52:28
allowed, then let me know. There we go.
52:31
All right, you where? If anybody listening has some
52:34
good names we would love to hear them too.
52:36
Yeah, that's that's. Awesome.
52:38
We can give. Yeah, we can have it.
52:40
You know, when I was in 3rd grade, they had all the third
52:43
graders and all the basically all the elementary kids in the
52:48
city that I was from try and compete to create the new mascot
52:53
for the the recycling trash tick up.
52:57
Yeah, yeah, yeah. And that's good.
52:59
A kid in my class actually won. So from that point in time, I
53:03
believe that anything is possible.
53:04
All right, that is the big learning in my top takeaway is
53:08
like that kid created the mascot that I now see driving all over
53:12
town and that's my buddy right there.
53:15
And no. That's that's incredible
53:16
actually. Did the kid win like an award or
53:19
anything? So these days you would think
53:22
like, wow, he probably should have gotten like 10 grand or at
53:25
least free trash for a year or something.
53:27
Yeah, they didn't get anything, man.
53:29
And they just used that. They they blatantly stole it.
53:32
They were like, hey, thanks. You applied to this, which we
53:35
didn't really have a choice, right?
53:37
We had to do it as part of school.
53:39
It was like, now we're going to submit this to the trash, the
53:43
recycling folks. I mean, that's.
53:45
That's if my kid won that, I'd be very pleased that that's
53:49
going on the college application for at least you know that out
53:54
of it at least. Exactly.
53:57
You got a lot. You could juice that for a lot
53:59
of miles. Yeah, yeah, 100.
54:00
Percent for sure, yeah. Anything to get ahead.
54:03
So the like that very same use case if someone out there is
54:09
listening and they have a great name and you end up using it.
54:13
Yeah, yeah. Just like your friend, we won't
54:15
pay for it, just give it to me. And put it on your college or
54:22
your MBA or PhD application. Yeah, yeah.
54:25
Yeah, we're going to pay you an. Exposure.
54:28
Perfect. People love that.
54:30
Or at the very least, you might get a promotion.
54:35
We'll let your boss know that, right?
54:37
You should, you deserve, you deserve a pay bump.
54:41
So all right, last one before we jump and we'll stop bullshitting
54:44
on this retrieval evaluation. Let's practise it.
54:51
Oh man, that's a hell of a question to end on.
54:55
That's tough. So look, I mean, I'll say
55:00
something at first that I think is obvious to people, but I'm
55:03
going to say it anyways. Like when you're building a
55:05
system, you write tests in general and you know, given,
55:09
given this input, this is what I want the output to be.
55:11
Well, OK, that that paradigm doesn't work as well when you're
55:14
working with large language models because they're not like
55:17
deterministic in the way that like a normal, you know,
55:20
software system that you build would be.
55:21
So how do you test the quality of the system you're building,
55:26
the end to end quality? Given this question, I want an
55:29
answer that touches on these points or contains at least this
55:34
and it's answer things of that nature.
55:36
How do you test this? There's different approaches out
55:40
there. We actually have our own open
55:41
source solution in this space. It's called Tonic Validate and
55:45
it's a it's a rag. Performance.
55:47
I was hoping for a better name than that honestly, after all
55:50
this talk. I mean, it's actually like, you
55:54
know, traditionally it's a good name.
55:56
It says what it does. Because people understand it.
55:59
You can, you can spell it. There's no like there's no jokes
56:02
to make. It's just, you know, tonic
56:04
validate. So the tonic.
56:06
By those standards. Yeah, it's, yeah, very, very
56:08
vanilla. So Tonic validate works
56:12
essentially, but like by treating your RAG application or
56:16
really any LLM backed application as a black box, you
56:20
know, given this input, we're going to evaluate both the
56:24
context retrieved and the answer given to determine if it's a
56:28
good answer. And there's different metrics
56:30
that we support that that we support out-of-the-box that you
56:34
can kind of use. And the metrics themselves
56:37
aren't perfect. Many of them require the use of
56:39
like a large language model to do the evaluation.
56:42
But what they get you is a consistent and easily
56:46
automatable observer, which which it is.
56:49
It's worth a lot, right? Because it allows you to see in
56:52
a consistent way like trends over time, like trends in your
56:56
scoring. So then I'll give you an example
56:58
of some of the metrics that that we see as being like the more
57:01
popular ones. One is called answer similarity.
57:06
This is probably the best metric, but it's a very
57:08
expensive 1 to compute because it requires every test question
57:12
to have a human provided reference answer.
57:15
And essentially we evaluate the RAG answer with the reference
57:18
answer and we check for similarity.
57:21
That's a very powerful metric, but again, expensive.
57:24
And then we have other metrics, for example, like one metric, it
57:27
looks at the question and it looks at the context retreat and
57:31
it kind of tries to determine, you know, how relevant is the
57:35
context to the question. And then on the flip side, we
57:38
look at the context retrieved and the answer, how much of the
57:40
answer is derived from the context.
57:43
So, you know, we're doing metrics like that to kind of
57:46
help you understand both answer quality, retrieval quality,
57:51
answer generation quality, etcetera.
57:54
And then there's other metrics that that that don't require LO
57:56
Ms. Like does the does the context contain Phi?
57:59
Does the answer contain PII, you know, things of that nature?
58:02
Or what's the latency? What's the response time?
58:05
How much did the call cost? How many tokens were used?
58:07
Did the answer contain this exact substring?
58:10
You know, simple things like this.
58:11
And we make all this available in a open source Python SDK and
58:15
then we provide a web UI to help you visualise and understand
58:18
metrics over time. But the, the, the problem is not
58:21
solved. It's very challenging,
58:24
especially as you get into like more complex rag systems coming
58:28
on the market that, that become that, that are, they're just,
58:31
they're complex and they're hard to evaluate and they have like
58:33
multiple steps and they're agentic and then they're
58:36
multimodal. And like, you know, you can't
58:39
just do like a black box evaluation at that point.
58:41
You want to do like careful testing of each step in the
58:43
process. And I, I don't think there's a
58:45
great answer yet. I, I think there's like there,
58:49
there's no, like there's no great solution out there yet.
58:51
I think Tonic Validate does a decent job.
58:53
It does a good job, but it, it's a very challenging problem.
58:57
And I, I think it's one that's going to get more and more
59:00
relevant, especially as rag systems go from being just like
59:03
internal tools to externally facing production, you know,
59:08
deployments. Yeah.
59:09
And they leak data. And they leak data.
59:13
And let's and they have more agency.
59:15
Yes, that's right, all all of the above.
59:18
Yeah, dude, this has been great.