Latest / Agentic Conversations (formally mlops.community) / [Exclusive] Zilliz Roundtable // Why Purpose-built Vector Databases Matter for Your Use Case

Transcript

[Exclusive] Zilliz Roundtable // Why Purpose-built Vector Databases Matter for Your Use Case · Agentic Conversations (formally mlops.community)

0:06
Hey Ohh, what is happening? Everyone, welcome, welcome,
0:14
welcome to another ML OPS community roundtable today who
0:21
got something special in the works talking all about vector
0:26
databases and hopefully gonna demystify a bit of what you're
0:33
looking at when you're thinking and you're testing out your
0:36
vector databases. I figured you know what?
0:39
As we like to do around here, I might as well bring out our
0:43
guest of honours with a little diddly.
0:48
And so I brought my guitar And if you will permit me, in this
0:51
moment, I'm going to start playing that guitar because
0:55
we've got three incredible speakers or panellists, if you
1:02
want to call them that. And it's only right that we give
1:08
them a warm welcome. So while I am jumping in here
1:14
and making some noise or what some people might call music, I
1:20
would love to hear in the chat where you all are calling in
1:24
from. Let's get this started, huh?
2:41
AO. Let's bring out our guests of
2:45
honour. Now where they at, Mr Frank?
2:49
There is, for everybody that do not know.
2:53
Head of AI and ML at Zillas. I see you laughing.
2:57
You were not expecting that kind of intro, were you?
3:00
I I was not expecting that kind of intro, but I'm very
3:02
pleasantly surprised as I have always.
3:05
When someone serenades you, we've also got Eugene.
3:08
Where you at, man? Hey, there you are, these
3:10
developer advocate at Zillas. And last but not least then
3:15
where are you that there you are, Head of AI platform and
3:19
Ecosystem. So today folks, we're going to
3:23
be talking all about why purpose built vector databases matter
3:29
and if you should be thinking about them for your use case.
3:34
We really want to get into a deep dive on the vector database
3:41
scene because right now it feels like just about every database
3:46
out there has bolted on a vector offering.
3:52
And so I think there is this huge question in people's eyes
3:57
as far as when do I just use the vector database that I've got
4:03
and use these vector offerings from whatever my favourite
4:11
vector database or favourite database flavour is versus when
4:15
do I just totally look at going all in on a vector database and
4:20
why I would want to, what are the trade-offs that I'm thinking
4:24
about? And so I'm excited.
4:27
I can't believe we got all three of you on here on this call at
4:30
the same time. It's an absolute delivered
4:33
privilege and an honour. And so I think what we should
4:38
start off with And Frank, because I brought you up first,
4:40
I'm going to direct this first question at you Shit.
4:44
When it comes to vector databases, there are always the
4:48
questions of if you need it or not, because these LLM context
4:55
windows are increasing right? And most specifically, when we
4:59
talk about rags, I think a lot of people as Google put out
5:05
their their paper like, ohh do we need vector databases
5:09
anymore? Because now the LM month ends
5:13
like that. It's just you're gonna grow the
5:15
Bible out of it and you can throw the Koran at it and you
5:18
can throw the Tibetan Book of the dead at it.
5:19
Whatever you want to throw, you can throw it all and more at it.
5:23
So where you sitting in this? How do you look at it?
5:28
Well, yeah, you could throw the Bible at it.
5:29
You know, you can throw the crown at it, whatever.
5:31
You could throw the, but can you?
5:34
I think the real question is, there's a couple of key factors
5:37
here I think about. First is, do you really want to
5:40
pay? If you're gonna query over,
5:43
let's say, the Bible, do you really wanna pay a dollar per
5:45
query? Right.
5:45
Do you? Do you really wanna say, OK, I'm
5:47
gonna have all this stuff into the context window for my large
5:51
language model and I'm gonna pay a dollar to ask a question right
5:55
now. If I have 100,000 questions, I
5:58
have to pay $100,000 just for for, you know, just for all of
6:01
that. So that's one of the questions.
6:02
So it adds to the context. As you have more and more
6:04
context, the the price of what you're charged becomes more and
6:09
more expensive. And not just from a dollar
6:12
perspective, but also from a compute perspective as well.
6:15
So large language models, it's more based at the end of the
6:19
day. So they have, they really are
6:22
are quadratic when it comes to the token length in terms of
6:26
compute complexity, right, in terms of runtime complexity.
6:28
So it's both factor of time as well as cost.
6:31
The other thing to think about is, OK, you have a million
6:33
tokens that you can feed your large language model.
6:37
But if I'm trying to index, let's say you know the San
6:39
Francisco Public Library or if I have a tonne of books that I
6:43
wanna index, it's not all going to fit into the context.
6:45
Whether it's still, I think it's still.
6:48
There's these two factors. Irrespective of how long your
6:51
context window is, irrespective of whether or not you have a
6:57
language model that how large it is, you will still need vector
7:02
databases to really retrieve to store and retrieve all contacts
7:08
that you really. Yeah, I see you and I I see that
7:13
you were like thinking maybe ohh, should I go?
7:16
Should I step up to the mic? I'll just be I mean like the
7:21
other thing that I would say with this is just like you know
7:23
there's still they still have this kind of like lost little
7:25
problem and you know I don't know what like clause sorry clot
7:33
threes and new cost is like $75.00 for a million tokens and
7:39
that's that's that's a lot you know.
7:42
So just to kind of add on to what Frank was saying, like a
7:46
dollar per per question might actually be underestimating it
7:49
if you're gonna be talking to the.
7:50
Bible, you know. Yeah, and it does feel like I I
7:53
often wonder about the performance on if that is the
7:57
best way to do it. So there is a clear piece around
8:02
speed not being up to par. And then the other side is like
8:08
you're potentially going to burn a hole in your pocket, but like
8:13
is this actually going to give you the most relevant answer is
8:18
probably the most important thing because maybe people will
8:20
pay at the end of the day and they will wait.
8:24
But if it also doesn't give you that the most relevant answer or
8:29
the best answer that you could get, I think then it just makes
8:33
no sense. So that's another piece that I I
8:36
would love. I haven't seen much around that.
8:38
I don't know if any of you guys. Have I think that's an
8:41
interesting question. I think a lot of the a lot of
8:42
the research at least I've seen today when it comes to retrieval
8:46
is is very much what people what's called needle in the
8:49
haystack retrieval. So they'll have you know, you'll
8:51
have, I think one of the earliest examples is you have
8:55
Harry Potter, you know you have the Harry Potter series.
8:57
They add it what, Harry Potter and the Chamber of Secrets.
9:02
And you take this entire book, all of its tokens, and you fit
9:05
that into the token window. And then right in the middle you
9:08
have a sentence like and Ujan is a developer advocate as it is.
9:14
Or something silly like that, right?
9:15
And Eugen loves machine learning.
9:17
Something like that. But to me it's a really, really
9:20
unfair comparison. First, because how many times
9:22
does the phrase machine learning appear in the Harry Potter
9:25
right? How many times does the word
9:26
Ujan appear in the Harry Potter series?
9:29
Probably no tie like it doesn't appear at all.
9:33
So it's very, very easy. You know, it's very, very easy
9:36
for. It's very, very easy for a large
9:38
language model for these attention mechanisms that are
9:41
elements to pick out that that subset of those phrases.
9:45
And the other thing is, I think there's also recent research.
9:48
I was reading a paper recently. It talks about the Holy Grail of
9:52
sort of large language models. Is reason and generalisation
9:55
right across you know different? You know across different tasks.
10:00
And if you try to do reasoning across you know.
10:04
If you try to, you know you have long context and you add these
10:07
sort of bits and pieces and you try to do reasoning across all
10:09
context, The performance becomes much worse as you increase the
10:13
likelihood, as you increase like the amount of tokens you're.
10:16
So think of the day it's still relevant and retrieval is still
10:19
very, very relevant wise. And I think it will continue to
10:22
be now maybe one day. I think one day really, really
10:26
far down the road, it doesn't really make sense to use rag I I
10:30
do see that potentially happening, but I think in the
10:32
short term doesn't matter how long these contacts.
10:37
Yeah. The other thing that I often
10:38
think about too is like how much goes on outside of just a prompt
10:44
and the context window and how many other pieces you need to be
10:48
thinking about. That is not just like, oh cool,
10:52
I'm going to put all this data into a context window and then
10:56
it's going to make my life easier.
10:59
And really, it reminds me of that famous Google paper back in
11:03
the day where it had the model and that was like just one part
11:07
of the entire system. And so it still feels like this,
11:11
like the context window is just one part, like, how clean is the
11:14
data? How are you ingesting that data?
11:16
How are you quality controlling? Like if you're getting tables
11:19
from PDF's, how do you make sure that those tables are actually
11:24
water in the PDF? And there's there's just so many
11:28
other questions that are beyond the the context window and so
11:32
that that's another huge piece. Yeah, absolutely.
11:36
I think I want to, I want to give Jack Jack some time to
11:39
speak about this as well because he, he does, he does those
11:41
pipelines and he's got tonnes of experience.
11:43
You mentioned PDFs, tables, all these kind of different
11:46
modalities of data, but the the analogy that I like to use is
11:51
that the context window. I saw this somewhere, I don't
11:54
remember where I saw this, but I thought it was a great analogy,
11:57
right? It's the context windows like
11:58
your it's like ramp or it's like it's like it's like an L1L2L3
12:02
cache and your database is is like this.
12:07
Now can you store everything your database?
12:08
Probably not unless you want your costs to be astronomical.
12:11
But it is meant to give you, it's meant to give you different
12:15
different layers of storage so to speak, right.
12:18
And you don't want to just as you don't want everything to be
12:22
in your RAM all the time and continue to ask questions over
12:25
that, you don't necessarily you want to offload those
12:29
capabilities to disc to your vector database.
12:33
So anyway that's just my two cents.
12:35
I think, I think perhaps maybe we haven't done as great of a
12:38
job of educating community as why word search is so relevant
12:41
with these long context models that can do needle haystack
12:44
retrieval. But that's what we're doing
12:48
here, right? And so it's like, you know,
12:51
hopefully in the future, you know things will, things will
12:54
change. Talk to us.
12:56
Yeah. Talk to us about pipelines.
12:57
Yeah. Yeah, so coming coming from a
12:59
background of search indexing, I I do have a lot to say about
13:03
this. This kind of, I would say a hype
13:06
of lung contacts will take over everything.
13:10
To me that doesn't really make sense from a production
13:13
perspective because well, there are just so many things to
13:17
consider where our serving user queries in a real world are
13:21
considering the latency or considering the user experience.
13:24
You're considering the cost like all sorts of of aspects are not
13:30
considered when when doing this more abstractive like academic
13:34
research on needle in haystack experiment.
13:38
It's more of a stress testing of the capability of large language
13:43
model rather than promoting. While you should use this in
13:46
your production setup. I think that that makes no sense
13:50
but why? So I can I can give some
13:53
examples or some throw in some some ideas here.
13:56
So one thing is on we like in in the in the search world we
14:02
always have the analogy that the serving time like serving user
14:06
queries, just people, the expert, there's a whole amount
14:10
of efforts being spent in the offline indexing time.
14:15
Doing so is because a lot of the analysis of the understanding of
14:20
the content are like really, really expensive.
14:24
They are like they involves running many machine learning
14:29
models and they involve like doing very lengthy analysis on
14:34
those content and you cannot simply cannot afford doing that
14:38
when the user is sending a query to you, right, you are supposed
14:41
to give back an answer within like 10 minutes, seconds or
14:44
something like that. So that there's a, there's a a
14:48
huge principle of doing all those kind of heavy lifting
14:52
stuff offline. And that's I think that's that's
14:55
one of the main principle of RACK as well, right For for RAG
15:00
you you do retrieval to augment or to enhance the large language
15:05
models and a huge part of retrieval is that you need to
15:09
understand the content and prepare all those efficient
15:12
index for those content before the user query becomes.
15:17
So I think that the pattern is a great advantage or value of of
15:22
right. There are also other
15:24
considerations because by doing this offline you are saving
15:28
time. Not only time but also cost
15:32
because if you are not doing that then every time the user
15:36
query comes, you need to do that once more like for for that
15:40
particular query and that's not affordable.
15:43
So even though like say understanding this whole Bible
15:47
takes say $3, right, like spending $3 once is whole
15:52
different story, that's spending in for millions of times when
15:56
the user query comes. So all those sorts of things
15:59
we're now considered in this experiment.
16:01
I'm I'm like personally excited about this experiment because
16:04
that's really extending the boundary of the capability of
16:07
large language models. But like looking back into what
16:11
kind of production system we want today, I think RAG is still
16:15
relevant. Is is never more relevant than
16:18
than right? I really enjoy this idea that
16:21
you're talking about about like what the needle in the haystack
16:25
should be looked at is more of a stress test, not like a daily
16:29
driver type thing. It's not a a best practise that
16:33
people are advocating for. It's just like how far can we
16:35
push this, and at least not today in we shouldn't be
16:41
thinking that's how we we wanna architect our systems and we
16:46
wanna like leverage that. It's just that, OK, this is
16:48
another tool in our arsenal. I also don't want to let anybody
16:54
think that I did not let it go over my head.
16:58
We are using the Bible as our reference here, which some
17:05
people on YouTube have said I bear striking resemblance to a
17:09
person in the Bible. And so we can kind of book now
17:14
if you want. We could talk about maybe like
17:17
encyclopaedia for you. Know the encyclopaedia worse as
17:20
well? Yeah, the encyclopaedia or Harry
17:24
Potter was great. Yeah, Harry Potter's good.
17:27
Harry Potter is probably more culturally relevant than the
17:30
encyclopaedia. So now that we got that out the
17:35
way, I, Frank, were you able to find what the name of that paper
17:41
was that you you referenced a minute ago?
17:43
And if not, I'll let you look for it, Yeah.
17:45
Unfortunately, haven't I? I need a little bit more time.
17:47
There is. And I'm wrote in the chat.
17:49
When there are so many papers coming out that it just kind of
17:53
drives me crazy and you know, I I I try to go through these days
17:56
I only read the the intro and the conclusion.
17:59
He's just like, smart, man. You are smarter man.
18:03
Yes, I try and understand it and then if it's really interesting,
18:07
then I'll try and dive into it. But all right, there is one
18:11
thing that I want to talk about. So we talked about this rag idea
18:14
and we talked about hey is, is there still room for vector
18:20
databases there. But let's move on a little bit
18:23
more because as we are talking in this the general setting of
18:28
vector database, it's a purpose built vector database versus
18:33
like just vector database bolted on the normal database that I'm
18:37
using whatever my flavour is And before we get there maybe can we
18:43
set a bit of a scene because are there other things.
18:46
So if you have another database that is specialised in certain
18:53
areas and then it throws on the vector database aspect to it,
18:58
that's one thing. But with like vector databases,
19:01
are there other types of data that you would think about
19:05
storing in a vector database and why?
19:08
If yes, like why would you do? That I guess I can, I can take
19:12
this one to start with and then Frank and John can give their
19:16
opinions on this. So I've been saying this for a
19:19
while because I I heard this from James and I was kind of
19:21
curious to look into this. But vector databases are really
19:26
compute engines and the whole like name vector database is
19:29
actually just a miss nowhere. It's just that it's easier to
19:33
tell people it's a database because you're storing
19:35
something, but the reality is you're actually storing
19:37
something in you are storing vectors in memory, but most of
19:41
the data that you store is actually held into permanent
19:44
storage, so like S3 or mini or something like that.
19:47
So you can store basically any types of data that you want in
19:50
investor database, just like you could treat it just like a Nosql
19:54
database and you would store things like text for rag, or if
20:00
you're going to be working with multimodal rag, you're gonna be
20:02
storing images perhaps, you know, probably links to images,
20:05
videos, things like that. And you should store this stuff
20:08
and you just store the metadata and maybe it tells you like,
20:10
hey, this is when this was published, this is who wrote it,
20:13
this is, you know, XYZ about the the information that you're
20:17
storing. And this will basically let you
20:19
use something that was originally built as the Compute
20:22
Engine to do vector search on top of you know.
20:27
A permanent storage layer, as if it were a real database.
20:31
And this is basically kind of how let's say Nosql databases
20:34
work and the difference between the way that a purpose built
20:40
back to database works and something that you just tack on
20:42
to let's say no SQL database or SQL database is in the way that
20:47
it's designed. And you know, we all kind of
20:52
have this intuition that things are best used for the purposes
20:59
that they're designed for. And so vector databases are
21:03
designed to be able to do this high amount of compute
21:06
efficiently and effectively at scale.
21:08
I mean let's just say open source for example.
21:10
If you just look at some of the open source projects, you can go
21:12
in and you can look at how much work was really put into this.
21:15
And I'm not going to say that that is the only indicator of
21:19
how many lines of code maybe is not the only indicator of how
21:22
robust or feature rich system is.
21:25
But it is definitely something where you can kind of look at
21:28
and be like OK, like you know, there's a lot of, you know,
21:32
features here that kind of help you do things perhaps on an
21:35
enterprise scale or at a most or at a more robust level.
21:39
Yeah, I think if you if you think of vector database in the
21:43
ecosystem of search or information retrieval, it is
21:47
supposed to be the most efficient way of retrieving some
21:50
information that has been offline, index and and
21:53
organised. However, when you organise the
21:56
information, you probably don't want to want to organise the say
22:01
the IDs or the the the index of it.
22:04
You also want to kind of put the original content as close as
22:08
possible to it, because that's what you really want to retrieve
22:12
and return back to your customers or clients, right?
22:16
So with this mindset we we do design ways of storing not only
22:21
vector data but also other kind of unstructured data which is
22:25
the source of the information where those vectors are
22:28
generated from. We by placing them closely with
22:33
vectors and returning them at the same time or together, we
22:38
can help the developers to build more efficient applications
22:41
because they do not need to retrieve these raw data from
22:47
somewhere else again and they they can achieve the best
22:51
efficiency from this pattern. Yeah, I I was sorry.
22:56
I was sort of busy searching for that paper.
22:58
I have not about it yet. So I I apologise to the as as as
23:04
soon as they are able to direct places I will find it.
23:07
Yeah, you know, I don't find it by.
23:08
Better search it. I don't find it by the end of
23:11
the session. I will.
23:13
I will. Build the new search.
23:15
On Twitter, I'll build a new search.
23:16
Yeah, I'll post it. On Twitter, I'll tag you choose.
23:19
Yeah, archive. You're gonna build a whole new
23:21
archive search function so that you can find it next time.
23:25
Yeah all these papers I mean cause the way the way I'm
23:30
searching for it now or which you know it sounds really it's
23:33
it sounds realistic. It's pure keyword based on I'm
23:35
searching like you know, reasoning capabilities as
23:38
context window increases and it's just not popping up.
23:41
There's actually there's actually some diagrams in there
23:44
which I actually remember very very fondly.
23:48
And if I find those images I'll sort of share them with you as
23:51
well. Dimitrios, maybe you can nice
23:53
post them, but. Yes, please.
23:55
We'll send an e-mail out to everyone to the The other thing
24:00
that I was going to mention unless did you have something
24:02
you want to attack on there? I imagine you were.
24:05
I think, I think Jane and Jane said it really, really well,
24:09
right at the end of the day, you know this about your database is
24:12
a compute engine. But the one thing that the one
24:16
thing that I do want to say is my personal opinion.
24:18
I do think Jiang disagrees with me here, so it would be
24:21
interesting to just chat about, chat about a little bit more.
24:24
If you look at, look, you got guys like manga to be coming in
24:29
and data, sex and and you know, elastic or whatever.
24:32
I'm not, I'm not trying to throw shade on them.
24:34
I think these are really great systems, but what what is their
24:39
marketing team you say, right? They say hey you should just
24:41
come use us because you can store structured or semi
24:46
structured data in in in here. You can even store JSON
24:49
documents in MongoDB and you can store vector data too, right?
24:53
To kill two birds with one stone, right?
24:55
Just to just use us and I think they really missed the point in
24:59
that these databases at the end of the day, you know vector
25:02
databases were built for different things.
25:04
Vector databases are built from the ground up to support vector
25:06
search and filtered search and and hybrid sparse does search.
25:11
It doesn't mean that vector databases don't support JSON
25:13
Data, right. You could have a blog post
25:16
that's coming up that's all about just storing using your
25:19
vector database as a pure Nosql data store.
25:22
You can just store JSON Data without vectors.
25:24
You can do that if you want. It's like, hey, if MongoDB wants
25:27
to play that game, we can play that game too.
25:29
It's just that we don't because I think we've always been, at
25:33
least in my eyes, we've always been very much about
25:36
accelerating the adoption of vector search and accelerating
25:39
the adoption of of of embeddings and embedding based retrieval.
25:43
That is my two cents. I know there are, I know that
25:47
there are differing opinions out there And I'm not saying that
25:49
you should store JSON Data injector database.
25:52
I'm not saying that all The thing is that if you are, if you
25:55
don't want to manage multiple databases, if you don't want to
25:57
manage multiple sources of data, start with the vector database.
26:01
Right future pretty yourself, right?
26:03
Don't, don't. You know, don't listen, you
26:07
know, don't listen to the, you know, don't drink the kool-aid,
26:10
so to speak. Or what is it?
26:12
What is the phrase that folks use for this kind of stuff these
26:14
days? What are all the cool kids
26:15
saying? They What are?
26:16
The cool kids saying, you know, I don't think.
26:18
He doesn't want to crochet, but I'll throw shade, try to store
26:21
1,000,000 a million vectors in MongoDB, retrieve them and come
26:24
back to me. Yeah, I mean, I think, I think
26:28
Mongo DB is great. It's excellent, right.
26:30
But at the end of the day it's it's, it's you're doing
26:33
different things. It doesn't mean that we don't
26:35
support JSON Data which don't support it as well as MongoDB
26:38
does, just like MongoDB doesn't support vector data as well as
26:40
we do. That's and that's exactly it.
26:43
It's like, why are why is there a whole market of so many
26:47
different databases? Because each database has their
26:50
specific flavour that they do things really well at, and for
26:54
use cases you need these different flavours because it is
27:00
what you're optimising for. Yeah, as a matter of fact, we
27:04
are actually building this data connectivity with all sorts of
27:08
connectors. So even if some of the data,
27:11
while they are more meant to be stored and managed by another
27:18
purpose to back a database or data management system, we do
27:23
have a valid path for those data to become vectors.
27:28
Like being generated as vectors and then being send to vector
27:34
database for efficient and timely retrieval.
27:37
So there there's some questions coming through in the chat that
27:40
I want to ask and then I want to get into a few ways to future
27:44
proof yourself and how to like use specific features in the
27:50
vector databases that can help you.
27:54
But we've got one question coming through since we are
27:57
talking about like pros and cons of different vector databases.
28:02
So someone is asking about Novis versus Pine Cone.
28:07
I think one piece is, but I'll start Mill, this is open source.
28:12
Pine cone is not right. I think that's probably the the
28:16
biggest thing in my mind. But you all have fielded this
28:20
question many more times than I have.
28:22
So is there anything else that you can say?
28:25
I'm looking at Eugene looking at licking his lips.
28:27
He's like, let me add it, hold on, let me turn myself off mute.
28:31
I got other stuff to say. I was already talking about
28:33
Mongo. Now Pine Cone, this is the best
28:34
day of my life. You got centre stage, dude.
28:38
I I don't wanna, I don't wanna be too inflammatory here.
28:42
But go on Pine Cones website, see how big you can, how many,
28:48
how many, how much data you can really store.
28:50
We've got some. We've got some benchmarks for
28:52
this. They're open source, as is the
28:55
ethos here as Ellis and you can really take a look at the data
29:00
sets we use and the way we test the data and benchmark the data
29:05
and just look at just you can just text the benchmarks
29:08
yourself. Like, I don't think I need to
29:10
say too much about this. I'm just going to say look at
29:12
the benchmarks. It's all open source.
29:13
You can see exactly how it's done.
29:15
All the data is out there. You can bring your own data,
29:17
just benchmark it yourself. Yeah, I think on top of that
29:22
other than the open source malware will, we also have
29:25
managed meals which is series called and in silence cloud we
29:29
added 10s of features On top of that we have a even more
29:34
performance even though like Melissa is already performing.
29:37
We have even more performant core or like indexing engine
29:43
until it's out. And we have this this feature
29:47
called Silk called Pipelines, which provides the streamlined
29:52
fashion of generating vegetarian bindings and then storing and
29:57
retrieving them in vector database.
29:59
And we also have the data connectivity that we mentioned.
30:02
We are building data connectors with all sorts of data sources
30:06
in this ecosystem and all those features.
30:11
I think those provide some differentiation between US and
30:15
and pine cone if if open source is not only the the
30:20
differentiator. Excellent.
30:22
The the next question coming through in the chat is asking
30:26
I'm vector DB curious but very ignorant on the topic.
30:31
How flexible or feasible would it be to use the same vector DB
30:35
of images with annotations for data loading during training as
30:41
part of the inference deployment?
30:44
So you're looking to use vector databases in your training loop
30:46
and I think that's it's it's actually not a common use case
30:50
that we see today just across the board but it will be very
30:55
very common for it And we've already got you know I won't
30:59
necessarily name who but we already have folks that actually
31:01
use build this in the training loop not just for large language
31:06
models but for other models as well.
31:08
And one of the things that they use it for is for example the
31:12
duplication of your training data.
31:13
And one really interesting way that you can using actually just
31:17
say hey, I'm going to I have this model that I've trained.
31:21
I'm going to use it to fetch items that are make sure that I
31:26
fetch items for training my new model that I know are are not
31:31
duplicates of what I've already used.
31:33
And there's also other ways other really interesting ways to
31:36
use your back to database in the training loop as well which I
31:38
won't get too much into some of the details.
31:40
But specifically as it comes to your use case, when it comes to
31:43
image, images with annotations, if you have, let's say you're
31:47
trying to train, let's say a very large multimodal model, you
31:51
can take some of these smaller embedding models that you have
31:53
and you can especially if you have these multimodal models and
31:57
then you can say, hey, I want to maybe for this particular batch
32:00
or I want to shrink this particular expert on some, you
32:04
know, one particular cluster of images or one particular, one
32:08
particular set of images, right. And that's one of the really,
32:11
really interesting, just one of the large subset of ways I think
32:15
you can use this. You can use not just Milkis,
32:17
right, other vector databases as well inside of your.
32:20
I think we're going to see more and more of these sort of use
32:24
cases for vector search in the future.
32:28
Really ties in very, very nicely with ML OPS.
32:31
Yeah. But we did see some use cases of
32:34
using vector database as retrieval and for deduplication
32:38
of images during the training phase.
32:40
I don't think we have seen like kind of combining this workload
32:44
with the online serving workload because one is like one comes in
32:49
bulk and it will probably take over some of the computing
32:53
resource of the of your collection.
32:55
So even though both of the use cases are are being seen in your
33:01
in your clients use cases, it kind of makes sense to to with
33:06
them into two different collections or even two
33:08
different physical setups so that you don't hurt the
33:12
performance of the online serving with your like offline
33:15
training use cases. Yeah, there's also like we can
33:19
kind of actually see some people are already doing something
33:21
similar to the the, the question askers question.
33:28
So you go look at. Your question askers question.
33:31
Right. But yeah, if you look at you
33:33
know like ARISE and VASSAL 51 and Galileo and like some of
33:39
these companies are doing this kind of thing where they're
33:41
using effective database to kind of show you the data quality and
33:45
the way that your data is clustered.
33:47
And I know I just worked on something with Jacob from 51
33:52
where you know, we use Milvus to store some data and to load some
33:59
similar data and to to observe some of the vector embeddings
34:03
and and how they look like. And so I think this is
34:05
definitely something that people are looking at doing and and in
34:08
fact already being done by a bunch of startups.
34:13
Great question, Burhan. I really appreciate that one
34:16
because I did not think of that at all.
34:18
And so it's almost like, yeah, that beginner's mind came
34:22
through and who would have thought that this is actually a
34:26
pattern that you're starting to see emerge and probably will be
34:29
something as we move forward. So now before we jump into
34:33
embeddings, because I feel like we can't talk about vector
34:36
databases without talking about embeddings, I do want to talk
34:39
for a minute about the different like ways to future proof your
34:47
AI applications and especially when you are using a purpose
34:50
built vector database. Like what are things that we
34:55
should be thinking about as we are moving through our life
35:00
cycle? And like how we're setting U our
35:02
system? And when it comes to most
35:06
specifically like the vector database, are there features
35:09
that we need to be keeping in mind?
35:10
Is there metadata that we need to be thinking about?
35:12
All of that fun stuff? There's a lot of new features
35:15
coming in Millis 24 like hybrid search.
35:19
And by the way, I'm gonna define hybrid search here.
35:21
Hybrid search is when you do metadata filtering on your
35:26
vector US hobby or vector search pre filtering on your metadata
35:29
for your for your vector search. OK that will hybrid searches.
35:32
Anybody who says otherwise is wrong.
35:34
And then there's also a multi vector search coming.
35:37
So that's when you're gonna be able to search multiple vectors
35:40
and you're gonna be able to rerank based on how the vectors
35:43
are. So for example a use case for
35:46
this might be that you have both a semantic similarity aspect
35:50
that you would like as well as maybe a visual similarity aspect
35:54
that you would like. In which case you would want
35:56
that vectors to do the semantic similarity search to get your
35:59
traditional or what we call vector embeddings right now to
36:02
do your semantic search. And then you're going to want
36:04
maybe something like full embeddings to do your visual
36:06
search. So there's definitely these use
36:09
cases that we start seeing come up for multi vector search and
36:13
maybe even you have text and you want to compare it and rerank
36:16
based on the text or things like that.
36:18
So there's a lot of these kinds of different new use cases that
36:22
are a little bit more, let's say, advanced that are coming in
36:25
the last two, four. Yeah, this this term of hybrid
36:28
research. It's really funny.
36:29
It's like everybody's using it for different meetings to my
36:34
understanding. Well initially I was thinking
36:36
like hybrid search is probably you're you're searching the same
36:39
thing through different modality or like different different ways
36:43
of while to to avoid the confusion I will just I will
36:47
just describe that way of which directly rather than using this
36:51
fancy terms. So yeah, genocide.
36:55
In the upcoming 2.4 release we are supporting, we're supporting
36:58
the while last hybrid the the mix of surf with with dense and
37:06
sparse inviting because there are definitely some great
37:09
features or properties of sparse embedding baby graphs.
37:13
The they the graphs the the individual concepts of a long of
37:18
a lengthy documents a bit better or like in a more understandable
37:22
way than the dense embedding. So that there are definitely
37:26
some developers fearing sparse search if not I stand alone but
37:31
also like combined with the the the dense embedding retrieval so
37:37
that yeah we we told the echo that requirement we we build
37:41
that feature really really fast and we are releasing it very
37:46
shortly. There is also while we're still
37:49
enhancing the other aspects of the of the Mills offering.
37:55
Just internally we are working on providing a a set of like how
38:02
easy to use utility functions to generate the embeddings from the
38:07
client side. So that you don't need to play
38:10
with all sorts of frameworks and get confused in between them.
38:14
If you have a relatively simpler requirements regarding which
38:19
kind of actually embedding you want to generate.
38:21
For example if you want to generate the sparse embedding
38:23
from split model or BM25 which is more like old school way of
38:29
representing text or definitely the hugging phase, sentence
38:35
transformer and open eye and other the game writing services.
38:39
So you bring up a very good point and I think it is the
38:43
perfect segue into embeddings in general.
38:48
And there was a question that came up in the MLS community
38:51
slack and it was kind of around the lines of So what are some
38:55
good practises around generating embeddings?
38:57
What are the best models? And the consensus was it's kind
39:01
of just like throwing spaghetti at the wall right now.
39:04
You see what works and hopefully you get lucky.
39:08
Go on to the leaderboard, grab a few of those models and then
39:13
does it work for your use case? Cool.
39:15
You're lucky it doesn't not keep searching.
39:17
You know, Maybe you guys have seen better practises than that
39:21
because it feels like there's got to be a different way,
39:24
right? Well first I want to say this
39:26
whole story of machine learning and artificial intelligence is
39:30
like throwing sky onto the wall and see what works and so system
39:35
of technology are are being developed but we don't want
39:40
every single developer to do that again and again we we do
39:43
want to provide some insights on that.
39:46
Yeah, I would let Frank to to speak on our left.
39:49
Frank is that while the great work in this area.
39:53
OK, my my, my personal opinion is so I see, I see a lot of
39:57
folks I talk to a lot of devs out there and you know when they
40:00
when they're building out their application that leverages Lambs
40:03
or leverages you know? Generative AI.
40:07
They just say I'm using Open AIS API or I'm using opening eyes at
40:11
an endpoint, which is great. Like I don't wanna, I don't
40:14
wanna throw shit on them. I think it's a great embedding
40:15
model, but like there's not one like it.
40:20
There's no way that there's one embedding model that fits your
40:23
use case right? If you look at I have this, I I
40:27
wrote this, I wrote this blog and there's this great example
40:29
on there where I I don't remember where I got this
40:34
example from. Again, I I get a lot of these
40:36
from different locations, different locations around the
40:39
web, from different people, and I sort of repurposed them.
40:41
So, you know, this is a great example where one sentence is
40:47
let's eat, I'm going to say let's eat.
40:50
Eugene here, let's eat, let's eat, right?
40:53
And the other sentence is let's eat comma Eugene.
40:55
And these mean two very, very different things right now for
40:59
some applications, let's say you know for some applications,
41:04
let's say you're you're you're talking a legal context or
41:06
you're talking in that type of you're talking in things that
41:12
are a little bit fuzzier. You probably want the embeddings
41:14
for these two sets to be very, very close to each other.
41:16
They're very related. They have almost the same tokens
41:19
with the exception of some punctuation.
41:21
And the people, the person that I'm talking about football,
41:24
these are this case is is huge. It's to set it right.
41:28
And in other you know, if you if you want to, if you want to talk
41:31
about these two sentences from a the perspective of what does it
41:34
actually mean. These mean very, very two
41:36
different things, Two very, very different things.
41:39
In one sentence I'm saying let's actually, you know, physically
41:42
Yugen, which, you know, let's not do that.
41:45
And then the other sentence I'm saying hey you Jen, let's go
41:48
get, let's go get lunch or let's let's go out to eat.
41:51
These mean two very different things.
41:53
For some you know, it depends on what you want to do.
41:56
You know there's there's no one-size-fits-all embedding
41:59
embedding all. And if you look at a lot of the
42:01
metrics out there of how, oh, you know, weddings aren't as
42:04
good as be of 25, it's it's because you're not using
42:07
something that is tailored to your application, right?
42:12
That is tailored, you know, that is semantically relevant to your
42:15
application. And I think try.
42:17
My recommendation if you're looking to do if you're looking
42:19
to get started with vector database directory search, try
42:22
different models. I know it's A and you know
42:25
hugging face and sense Transformers, and you know
42:28
there's Voyage as well. They make it, you know these.
42:30
They make it very easy for you to try different things, try
42:32
different models and see what works best for your application.
42:35
Use the LDM attend test, right? You know if I try, try tendering
42:41
examples. Let's see which one looks the
42:42
best, or try 20 different examples and see which one looks
42:46
at best. At the end of the day, there's
42:48
you know humans are still the best evaluators, right?
42:52
And you should, depending on what it is that you're trying to
42:55
build, evaluate your embedding model accordingly.
42:58
I'll give a slightly different answer here.
43:02
You should fine tune your embedding models.
43:05
You should fine tune your embedding models with like that
43:08
of what you're going to be doing.
43:10
And actually this was. I talked about this in my paper
43:14
with voxel 51 as well. We took some examples with Clip,
43:19
VIT and the CFAR 10 data set and we searched 3 words Ferrari,
43:25
pony and Mustang. And so you know, you can see
43:28
Ferrari, he's gonna give you cars back, pony is gonna get
43:31
animals back, but must say is going to give you both horses
43:35
and cars back. And so, depending on what you're
43:38
working on, let's say for example you're working on cars,
43:41
you don't really want pictures of horses back, and so you have
43:44
to fine tune your embeddings models to kind of get back the
43:47
right next. Yeah, I think there are are just
43:52
just so many embody models available on the market and
43:57
unfortunately most of the developers just choose, say open
44:01
eye embedding not because of this quality, but because of the
44:04
name of open. So that's definitely not the way
44:08
I would recommend. I think surprisingly just with a
44:13
very small data set, you can probably test the effectiveness
44:18
of the embedding model on your particular use case.
44:21
So that my top recommendation is that if you have this with this
44:27
capacity of doing some testing and evaluation, definitely build
44:30
your own data set which can speak best for you or your use
44:35
case than any other open data site such as Ms Macro or Beer.
44:40
And also this, this, this public public leaderboard from
44:47
packaging base. I'm dead right.
44:49
This I'm tab leaderboard has been overfitted by many, many
44:54
models already, so it's really hard to simply judge.
44:58
I mean, that's still instructive, but we we should
45:02
then simply judge by by their ranking on this leaderboard.
45:06
However, that being said, if you don't have the capacity of doing
45:10
an evaluation, then choosing from some big name is still a
45:14
practical way of making the job. So there there's something
45:18
fascinating that you said there, like creating your own data set.
45:22
And I think I just saw some posts on one of the social
45:30
medias talking about like how a big question and open question
45:34
is how much data is enough to be able to say like, yeah, this is
45:40
good And especially if we're doing for taking your Jane's
45:43
recommendation and saying, all right, we'll fine tune our
45:46
embedding model. Have you seen best practises
45:49
around that? Like what is a good amount of
45:53
data look like and how can you assure the quality of that?
45:57
Also like how can you make sure that it is nice and diverse,
46:02
robust, all the fun stuff in there too.
46:06
So there was a recent paper from 2023 or maybe it's 2022, I think
46:11
it's 2023. It's called neural priming and
46:14
it comes out of. Ohh man I don't know what
46:18
school. It comes out of, but it's it's
46:20
Ali Farhadi is the, the lead researcher for that.
46:22
He's currently the CEO of AI 2:00.
46:24
And basically what they show is that if you prime a model with
46:30
some images. So this isn't even fine tuning.
46:32
This is I mean this is like kind of fine tuning.
46:33
It's like very you know it's a very small amount of fine tuning
46:36
with the right amount of with the right contacts, you will get
46:39
some decent percentage improvement in results.
46:42
And of course this will vary from data set to data set.
46:45
But for images, you're really only looking at 2025 images to
46:50
kind of skew the to skew the model towards what you're
46:54
looking for. And for texts, you're looking at
46:57
maybe like 100 to 120 sentences. From my just my experience of
47:02
working with the evaluation, I think the bare minimum is
47:05
definitely tends to hundreds of of examples to give you a
47:10
meaningful indication of how well this model performs.
47:15
If you have thousands of examples, you are probably in
47:17
good shape as a general rule of. So basically it to try and
47:23
summarise this so that it's not throwing spaghetti at the wall,
47:26
right. Well, what we're going to do is
47:28
we're going to go on the leaderboard.
47:30
We're going to grab a few models, test those out with
47:33
maybe 100 if you can, 100 different types of of examples.
47:41
And then if you see that there's one that's performing well, go
47:45
ahead, add that extra tuning, Fine tune it data as you can.
47:50
But you probably don't need to be thinking more than like 1000
47:53
examples. You probably should be thinking
47:56
more in the hundreds of examples and may be able to get away with
48:00
the 10s of examples. The data has to be really high
48:03
quality. Though, right?
48:04
So you don't want to what large language models are really, or
48:09
what large language models are really good at is, you know,
48:11
they have just a huge wealth of pre training data.
48:15
And if you want to, if you want to use very, very little data,
48:19
that's OK. But you just have to make sure
48:21
that it's really, really relevant to your data set.
48:24
It's very It makes sure that it's a good representation for
48:29
what things are actually going to look like when you go to
48:31
product. Yeah, that's so true.
48:33
I and that was a another great paper, the Lima paper like less
48:37
is more. It's alignment.
48:39
I I remember that one. I'll drop that one in the chat.
48:42
That one I can definitely find because it's an easy name,
48:45
right. Eugene, if you have the link to
48:48
the paper that you've referenced, throw it over.
48:50
So before we jump folks, I want to get a few cool maybe hot
48:57
takes. I don't know.
48:58
We'll see mild takes to her, but I think there's, there's
49:04
something that we could talk about which has been taking
49:08
Twitter by storm, right, which is the Colbert model and it
49:13
feels like these late interaction models.
49:16
Maybe you, Jen, can you talk to us about what those are and why
49:19
that's interesting and what what you think the reason for it to
49:24
become more popular right now is?
49:27
Yeah, so this this whole bear. Paper and the late interaction
49:31
model behind it that's that's really interesting and that's
49:34
catching the eyes be because right now people kind of realise
49:39
that this paradigm of embedding model plus after retrieval is a
49:44
perfect thing for like for for retrieving a bunch of like
49:49
candidates of the final result. But if you add another layer
49:54
which is called rerun car model on top of this to kind of refine
49:58
the results and select the very best from the best candidates
50:03
then you can probably achieve better results.
50:06
We did some quantifiable analysis on this.
50:09
Yes you you can improve the like the recall numbers or a all
50:16
those quality metrics by a few percent.
50:19
So that's that's that's meaningful difference.
50:22
However, this Republican model is super expensive.
50:26
You are probably doing. Something.
50:29
That you should do you know, offline indexing at online query
50:33
serving time just because you're using this regular because they
50:37
they use the cross encoder model which needs to basically just
50:42
look at the, the, the, the whole query and all the candidate
50:46
documents rather than just computing the cosine similarity
50:50
score, which is quite cheap computational wise, right?
50:53
So to solve this problem, there is this late interaction model
50:59
being proposed which kind of combines the good things about
51:04
embedding vector retrieval and this crossing coder rank.
51:09
It generates a whole set of embeddings from one single query
51:14
or document and then do the late introduction.
51:19
By. Computing the, I would say the
51:21
the similarity score between them at query time.
51:25
So that's cheaper than regular, but that represents more raw
51:29
information than the single dense bacterial bag.
51:34
So I think that's that's a great idea.
51:37
But again putting it into production that's that's
51:40
questionable to, I mean at least per se because it it definitely
51:47
used more like space to encode this information.
51:51
So that creates a new challenge during the online retrieval
51:55
time. And moreover like a lot of the
51:59
efforts from those community has been put on single vector data
52:03
embedded so that the embedding model is being refined and being
52:07
like being trained over and over with better data so that it does
52:12
achieve better performance than the Colbert embedding which has
52:16
been only trained by the academia community.
52:21
However, we are seeing like more and more efforts on this.
52:24
So it's hard to say what will happen in the future.
52:26
We are looking into this and we are also thinking of we start to
52:33
think about how to how to integrate this pattern into the
52:37
vector retrieval facilities that we are building so that in the
52:42
future this becomes the, I would say a popular choice among
52:46
developers will have a solution that pairs it.
52:49
Here's your hot take for this, OK?
52:50
Forget Colbert. The only Colbert you need to
52:52
know is Stephen Colbert. What you should really be doing
52:55
is you should be using Millis's multi vector search with re
52:58
ranking to produce the same results at the at at a better at
53:02
a better price. And it's easier.
53:04
We all love cheaper. That is very, very useful to
53:09
know. The cheaper the better I think.
53:12
And that's cool that you all have been inspired and taken
53:17
that into account. I know we talked a little bit
53:20
about I I want to take the next like 2 minutes just to shine a
53:25
light on some of the cool stuff. You all are doing it now this
53:28
and Zillas because we talked about the pipelines.
53:32
We also talked about the the Rancour, what other?
53:38
Features out there I. Know that we said that there's
53:41
in the pipeline. We've got a few features that
53:43
are coming out, like the is it sparse embeddings and and a few
53:50
others, but by this time the reader is coming out.
53:57
So the either we have just supported the reranked model in
54:01
the cloud pipelines and for the open source milvus we are about
54:07
to support the sparse and dense vector hybrid retrieval.
54:11
I cannot avoid using the word hybrid in the next release.
54:17
Frank loves it. Frank's like, oh, we got to get
54:20
a better naming convention around this.
54:23
Well. Fellas, this has been awesome.
54:25
I really appreciate you doing this One thing as we end, I
54:30
would love to ask you a question that I tend to ask when it comes
54:36
to vector databases. It's not like doesn't really fit
54:39
into the specific. Is this a specific vector
54:43
database or like kind of catch all database that has vectors
54:48
support theme but I do. I was talking to some data
54:55
engineers two weeks ago from Quantum Black and they were
55:00
saying how hard it is to make sure that when you put
55:05
information into a vector database for your rags, how do
55:10
you ensure that that information, if it gets updated
55:15
it is the one that is being retrieved when you want to go in
55:22
ask answer questions O they gave me an example of like an HR
55:27
policy being updated and now all of a sudden you have less
55:30
vacation time and so if you have a chat bot that gives the old
55:34
vacation time, that's not good because then the company all of
55:38
a sudden is like hey no actually you only have 10 days not 20 and
55:44
you could get some pissed off employees.
55:46
So how have you all seen best practises around that?
55:51
Yeah, it should treat your. Vector database as like a living
55:55
entity, right? And it may as you as you know
55:58
create these. And this is one of the great
56:00
things about what some of the subject has does had Jiang and
56:04
his team have done, which is as you have, as you you have all
56:08
these all this data, you have all these embeddings inside your
56:11
vector database and create these real time sources, these real
56:14
time syncs and make sure that it's up to date at all times.
56:17
I know that sounds, I know it's like a really, really simplistic
56:20
answer, which it is, but I think at the end of the day, that's
56:24
really. Your vector database you'd want
56:28
to treat. It almost as other other as you
56:31
do other databases, as well as a single source of truth for
56:35
semantic information for semantic representations, and
56:38
that shouldn't change just because you're only using it
56:41
just because you want to be able to do.
56:42
Just because approximates never search is different from the way
56:46
you typically query other databases.
56:48
Yeah, see, it all comes down to. Manage the data carefully,
56:52
carefully crafting the business logic that identify each piece
56:55
of information. For example, the HR the policy
56:59
is in some document, then once you update the policy,
57:02
definitely have updated the document.
57:04
Then just do like purging and reinserting or better off of the
57:10
dust, single dating. Or if there's a single piece of
57:13
text chunk, then just identify the text chunk and then update
57:17
it. Treehouse use.
57:20
Triggers. Yeah, yeah, That's.
57:22
I So that's awesome to think about like for a for that use
57:27
case. And I think the other one is
57:31
like a little bit more grey where you're like, OK, we
57:34
actually want to see how this document has evolved over time.
57:38
So we don't necessarily want to do a find and replace, but it's
57:42
100% like what you're saying. Data engineering has been
57:47
working on these types of problems for a long time.
57:50
It's typical data, data OPS, database stuff.
57:55
And it's not like all of a sudden we should forget about
57:58
all of that, because now we're dealing with vectors, we can
58:01
still bring some of those best practises into the mix.
58:05
Yeah, of course. I think that is it.
58:09
Our time here is sadly coming to an end.
58:14
I want to give you all a huge thanks and for everyone that
58:19
joined us virtually, this was super cool.
58:21
Thank you for all the questions. For anybody out there that is
58:25
looking for a new piece of cool clothing, I will remind you that
58:32
we have this I hallucinate more than Chad GP T-shirt.
58:36
Yeah, it is hot off the press where they're selling like hot
58:41
cakes. So maybe you can get yourself
58:44
one by scanning that QR code. Before I end this stream.
58:49
I'll let all these guys go. This has been awesome.
58:52
Thank you all.