Latest / Agentic Conversations (formally mlops.community) / Information Retrieval & Relevance // Daniel Svonava // #214

Transcript

Information Retrieval & Relevance // Daniel Svonava // #214 · Agentic Conversations (formally mlops.community)

0:00
So I'm Daniel Sonara. I'm one of the cofounders and
0:04
the CEO of Superlink and Blake. No sugar, nothing straight like.
0:13
Hey yo, we are back with another MLB's Community podcast.
0:18
What a doozy. This one was talking to Daniel.
0:20
I gotta say before we go anywhere, thank you Daniel for
0:25
being so patient with me. Thank you so much because he
0:29
explained it to me like I was five.
0:32
I felt like I was inside that Reddit subreddit community and I
0:36
had to ask him and re ask him because I really wanted to make
0:40
sure that I understand what exactly he is talking about when
0:44
he mentions vector compute and how it can enhance these
0:51
different systems that are using vectors and embeddings.
0:54
And so he broke it down for us and it was a little bit
0:58
reminiscent of DSP I when I talked to Omar, the creator of
1:02
DSP I. Which by the way, has anybody
1:05
else noticed that? That feels like it's absolutely
1:07
blown up. If you are using DSL in
1:10
production, hit me up because I would love to hear about your
1:13
story and what you're doing. I don't know anybody using it in
1:15
production, but I feel like it's soon.
1:17
It's coming and with all the attention that it's getting, it
1:20
should be happening. But who knows, especially
1:24
because the creators and maintainers are university
1:27
students. So I'm not sure if anyone feels
1:29
comfortable using it in production yet.
1:31
But if you are tell me okay side note, little tangent over Let's
1:36
get back to Daniel and the Vector Compute.
1:40
This is something that I felt like I needed a metaphor before
1:45
I understood it, and he broke down the metaphor very nicely.
1:48
He said. You know what?
1:50
Think about when you are shooting photos with a DSLR and
1:54
you get a RAW file. That file is not the most
2:00
beautiful image, but what it has is all of the information
2:04
contained in that image. O What you're able to do when it
2:09
comes to tweaking that image is bring out the pieces that you
2:15
think are best in that image. And the metaphor there is that
2:20
when it comes to embeddings, throwing them into your vector
2:25
database with the vector compute, he wants all of the
2:31
embeddings to be flattened and he wants the vector compute to
2:36
be that is. That allows you to bring out
2:41
what is most important when it comes to your use case and what
2:46
you're optimising for and what you want to show people.
2:49
O In the case of a recommender system that you want to show
2:53
people some for you page type stuff, you're going to reference
2:58
certain features and certain parts certain embeddings more
3:03
than others. And when it comes to a rag
3:06
system, you're going to reference different embeddings.
3:12
Now if that ain't a good metaphor, I don't know what it
3:17
is. I'll let him explain it more.
3:19
Hopefully. You all aren't thrilling things
3:23
at your screen or phone. Whatever you're listening to is
3:27
on because it took me so long to understand what the hell was
3:31
going on and hopefully after this you two are able to
3:36
understand what is going on. If you end up using Super
3:39
Linked, let me know. I would love to hear what your
3:41
experiences is with it. If you have strong opinions
3:45
about why or why you shouldn't use it, I would also love to
3:48
know that. I know there are opinionated
3:49
people in the community and so that's cool too there This
3:54
episode I wanna preference this with.
3:56
It is not sponsored. I just love Daniel and what he's
3:59
doing and wish him the best. Go check out superlink.com if
4:03
you get the chance. And also he is creating Vector
4:07
Hub. So that's a way for you to
4:10
compare all of the different vector databases and what their
4:14
features are. O.
4:16
That's cool because it's basically like nonbiased third
4:18
party able to look at all the different vector databases.
4:23
Not just the specific vector databases that are only vector
4:28
databases, but everything. I mean everything from the
4:31
Vector database that is something like a Cassandra that
4:35
bolts on a vector database to like a quadrant which is only
4:39
for Vector database or vectors. Ohh, I said Vector database way
4:44
too many times in that last sentence.
4:47
So let's get into the conversation with Daniel.
4:50
Check it out superlinked.vectorhubithinkiswhathesaid.com
4:55
and we'll leave it all links in the description for you anyway.
4:58
And if you enjoy this, if you think that it is worth talking
5:03
about it more, hit me up or share it with a friend.
5:06
Let's get into the conversation and I hope you enjoy my new song
5:11
prompt templates, which you can stream anywhere, especially on
5:14
Spotify right you went from being a recommender system,
5:28
Basically, what was the V1 idea? And then how did you realise
5:35
that you needed to pivot? Yeah, it was a, it was a
5:39
journey. I would call it kind of
5:41
incremental steps to success, not not like full periods had
5:47
many, many years ago. We wanted to build a social
5:51
graph across the Internet, right, Kind of social login
5:56
basically there's those kind of COVID was just kicking off and
6:02
everybody was online wanting to connect scattered across
6:05
communities. So we built various apps for
6:08
professional communities with the eye towards shared login and
6:14
kind of shared user pressurised model under the hood, right.
6:18
I have been at in the YouTube ads for six years.
6:21
So user modelling is kind of the particular hammer I have to hit
6:26
every nail. So, OK, so that that was the
6:31
plan. And then we realised that, OK,
6:34
all kinds of other products want to build personalised
6:39
experiences, right? Understand what the user wants,
6:42
give it to them quick, right. You know, around the time TikTok
6:47
also came out with the Monolith paper where they kind of
6:51
cemented, you know some of their ideas into real time
6:55
personalization and our system was kind of structured quite
6:59
similarly. So that was like the initial
7:02
validation and we kind of opened up the platform and started to
7:07
onboard people beyond social like jobs marketplace and and an
7:13
ecommerce company and and so on. And then you know we we we
7:20
raised some memory and the second time we tried to do that
7:24
and really was like ohh recommend the system as a
7:27
service. We see that you know five times
7:29
a year what's so special about this one.
7:32
And we had some ideas for how not to be stacked into like a
7:38
software consultancy mode, building ranking models, nuance
7:42
for each client because they typically have those companies
7:45
go right you kind of speech people on KPI list and then you
7:49
go building these ranking models based on the behaviour of those
7:54
specific users in that specific context.
7:56
We didn't want to do that. We wanted the product and so we
8:00
we basically said, OK, in this whole recommender system stack,
8:04
you typically have retrieval and ranking, right?
8:06
And our goal was to basically reduce the need for ranking as
8:13
much as possible and do it by making the retrieval smarter,
8:16
right. And we did that by, you know,
8:20
carefully constructing these complicated vector embeddings of
8:23
both the users and the the recommended items, and then
8:28
having our own query language into the vector search vector
8:32
retrieval. That would allow you to express
8:35
all these different objectives that you have when you do
8:38
recommender systems and even add the behavioural feedback loops
8:42
on top without building this kind of monstrous ranking model.
8:48
So I'm just wondering the basically the Rexus as a service
8:55
feels like when you say it, Ohh yeah everybody needs recommender
8:58
systems and it should be something that is important to
9:03
people. But none of it took off because
9:06
of this consultancy angle or this basically white gloved
9:10
angle that you have to take and you realised, OK, if we're gonna
9:16
have to make custom models for everyone or if we're going to
9:20
try not to make custom models for everyone, then what can we
9:25
do to make this easier for us? And presumably that's where you
9:29
came up with the idea. Ohh, wait, there's other pieces
9:33
to the puzzle that are harder problems that maybe we can go.
9:37
It's like you're going a little bit lower down the stack, right?
9:40
Exactly. You know, by kind of asking
9:43
ourselves the question of can we make the underlying retrievers
9:48
retrieval smarter, right? We spend a lot of work in the
9:54
vector embedding calculation for these complex data objects and
9:59
then they realised that just that alone is is an interesting
10:04
problem to solve that repeats across many different use cases.
10:08
And that's where we basically a couple years ago started to
10:11
build the vector computer, right?
10:15
You know, kind of as productized as possible way to turn complex
10:19
data into vector embeddings. For her.
10:23
And and you told me about this before, but basically it's like
10:31
in software, especially recently we have vector, well we have
10:37
compute and storage. And one thing that has been
10:41
very, very popular and it's been almost like a a boon to software
10:46
developers is decoupling compute and storage.
10:49
And right now with vectors, we have vector storage, but we
10:55
didn't really have anybody that was thinking about vector
10:57
compute, and judging by how hot the vector space is, it seems
11:05
like it's ripe for compute now. Why?
11:10
I guess, Why is it different than just like regular compute,
11:15
right? That's probably like the first
11:17
question that you got asked by every VC that you tried to raise
11:20
from on this second time around. Right, right, right.
11:24
So, so as you're saying you know the there is the, there is the
11:28
storage part and you know I would also add to the the
11:34
problem of organisation, right. So building the vector index,
11:37
doing all the things the database has to do, sharding,
11:41
reliability, access controls, right.
11:45
So there is a bunch of people working on that And then how is
11:50
the compute different? At the end of the day you are
11:54
spending, you're burning the same electricity running on the
11:57
same hardware. So at the end of the day, the
12:00
the underlying operations are the same and the name of the
12:04
game is always about how do you control this thing, right?
12:08
Do you have, are you building models on strategy in charge?
12:13
Are you running data pipelines in ARK?
12:17
Right? Are you using these kind of
12:18
completely generalised computing models to solve the problem or
12:24
is it the worse thinking about a an abstraction that's specific
12:29
to turning data into vectors, right.
12:31
And and if you basically say OK, how would that move right?
12:34
Like what would be the properties that you want out of
12:37
such abstraction? Then?
12:39
That's when you start to think about kind of the data
12:42
engineering problems. How?
12:43
How do you keep things synchronised?
12:45
How do you do backfills? You think about the machine
12:47
learning problems. How do I, for complicated data
12:52
object with million different properties of different types?
12:55
How do I somehow boil this together and make the vector
12:58
right? What's the?
12:59
What's the highest level abstraction that allows me to do
13:03
that but still gives me control, right?
13:06
So it's not like I just turn everything into string and then
13:09
send it to a language model, because there's literally the
13:12
opposite of control, right? So how do we do this?
13:16
How do we put that one to? Help me tell me more about this,
13:21
turning it into a string because I feel like you're talking
13:26
directly to me. That seems like something that a
13:29
lot of people do, and then what? The big biggest problem there is
13:33
that we can't wait this string in any way shape or form.
13:37
Right, right. So, so basically, you know,
13:40
let's take an example, right? You have, let's say, a movie
13:44
database and you know all kinds of things about your movies,
13:48
right? The movie names and the
13:51
categories and the descriptions. That's the easy stuff.
13:55
And then you know things like viewership, popularity, right?
14:01
You know specific click patterns of your users if they interact
14:06
with these movies, right? These are all the different
14:09
signals that you have about the movie.
14:12
Maybe the lounge date? When was it released?
14:15
Right now when people do rag and they do recommend the systems
14:19
and and and kind of retrieval unless they train their own you
14:23
know, 2 tower reckless model which often still doesn't eat
14:28
all the data because some of it is just difficult to eat.
14:32
Basically the the kind of language model first approach
14:36
here right is to stringify the whole movie right?
14:40
So you literally concatenate all the different properties and and
14:45
then you take embedding and the embedding is like God knows what
14:49
right? Thousand floating point numbers
14:52
right. And and then you know some
14:55
movies kind of get clustered and and some movies are near each
14:59
other that that that really shouldn't like you look at it as
15:02
the you just eyeball right. You didn't get to quantitative
15:05
test you just eyeball this thing and there is something weird.
15:09
Every once in a while something gets slotted somewhere where it
15:12
doesn't make sense and then you'll notice that ohh it's
15:15
picking up on something in the description that sort of like
15:17
makes it think that is related to this other thing but but
15:21
really shouldn't. And that's where prompt
15:23
engineering something, which is my both most and least favourite
15:27
part of this because because people literally start to say
15:31
like, you know movie name column, movie name, comma and
15:37
then like here's the description, but it's not so
15:39
important. But you know, FYI Callum and
15:42
then the description, right. And then they're like just kind
15:45
of tweak just how you know that's for heavy-handed.
15:51
They should be about processing the description and there's just
15:55
not a way to build an actual system, right?
15:59
Yeah. Right.
16:01
So, so it's any time, either way to kind of put the constraint
16:05
over it, like how do you want to create this pressure to for
16:08
certain things to be close together and certain things to
16:11
be far apart in a way that makes sense to your specific problem,
16:15
right. So maybe you Add all about like
16:18
freshness, right, recent news, right.
16:21
And so then you would want to kind of really boost the, the
16:24
aspect of how long has the news been out, right and and you know
16:29
popularity and relevance to your query, whatever it is, right.
16:33
So we spend a lot of time thinking about ways to express
16:37
these different, often competing objectives and then have the
16:43
underlying system kind of navigate the trade off which you
16:46
know like OK in amongst engineers which means basically
16:52
you know you're putting all these things to the embedding,
16:54
right. Then you are constructing the
16:55
search vector to tap into all those different properties of
16:58
your movie and it had. It helps if the video index is
17:03
kind of unit normalised, it's not biassed to any of those
17:06
signals. And then when you search you can
17:08
create the biases or awaits different ways to different
17:13
aspects of the vector space. Basically, so I like this.
17:18
I think the idea here of you saying this is how you can get
17:24
more control when it comes to dealing with vectors and
17:29
especially on things like recommender systems or rags
17:32
where you have all of this data but you don't necessarily know
17:36
which data is important. Or you do know which data is
17:40
important, but you don't have a way to explain to the model how
17:45
ohh, this data is more important except for through prompt
17:48
engineering and you never really know if that is working or not.
17:52
And so now you're saying, If I'm understanding this correctly,
17:56
with the vector compute, you're able to take all of the
18:01
different things that the vector or the item is made out of.
18:12
So it is made of a photo and some description and some
18:18
features about how long people watch the video and the
18:22
popularity score. Whatever, there's metadata,
18:26
there's the actual title and there's the images.
18:30
So you have all these different pieces of the puzzle, and what
18:35
are you doing with all that information?
18:38
Exactly. Exactly.
18:39
How are you tapping into, you know, expressing your desires?
18:44
Yeah, in in, in the language of those different properties,
18:48
right. And how are you sort of doing it
18:52
in a way that everything has a type you know, so there is no
18:57
random Python lambdas flying around and you have a very clear
19:03
path from whatever you cook up in your notebook, right, to
19:08
actually launching stuff. Because yeah I I think that's
19:12
going to be sort of important this year is to already talks
19:17
about getting things to production and not everybody is
19:21
very clear on what that means. You know that's the
19:24
responsibility of the vendor kind of stop at you being able
19:27
to build a docker container or is it sort of reasonable to
19:32
expect that you will get help setting up all the resources in
19:38
your in your cloud for example which I believe it is.
19:42
You know if I give you a docker container, you still have to
19:45
figure out all the serving issues right back feels.
19:50
How do you like shared the workflow?
19:54
How are you connecting this to the destinations?
19:56
Because you know as as we kind of established, there is the
19:59
compute and the indexing or management of the vectors,
20:03
right? And so that means we work with
20:07
the vector databases to make the overall offering better, right?
20:11
And so now it also has to reflect all the sort of
20:14
production implications of what the vector database wants out of
20:20
the workflow, right? Like are you dropping a million
20:22
vectors in it all at once? And have it, you know, is there
20:27
an indexing parallelized like all of those considerations,
20:30
there are part of the part of the problem and then you have,
20:34
you know, you have the batch and you have the real time kind of
20:37
use cases, right? Yeah, So, so, yeah, but there is
20:41
a, there is a lot around it all kind of starting the notebook
20:46
and then get it to production, the get it to production people
20:49
kind of gloss over it, but it's, it's a lot of work, right.
20:54
Yeah. We know that, definitely know
20:57
that. So what does it actually look
20:59
like in practise when you're setting up a system, how do how
21:03
do you have like a vector compute deal or communicate with
21:07
a vector database and all the other pieces like the embedding
21:11
model and then the LLM and everything else?
21:14
If you're doing some kind of RAG.
21:16
If it's a Rexus, I can imagine it's a little bit different of a
21:20
system design, so maybe you can break down those two and how you
21:23
see the vector compute or the vector engine.
21:27
What are you calling it? I don't even know.
21:29
Is it a vector engine? Is it a vector compute engine?
21:32
Is there a word yet? So we call it the vector
21:35
computer and we kind of take it to the computer place just to
21:40
illustrate that there'll be electricity spent on the CPU
21:46
files, GPU type situation. And.
21:52
Yeah, right. So, so in terms of kind of how
21:54
does this look when you just want to solve the problem, just
21:57
build the right system that actually returns reasonable
22:01
results, right? This is what everybody wants.
22:04
And you know the the sort of the end goal here should be
22:08
something like, you know, you go to your cloud provider, anyone
22:13
you choose and these people have marketplaces, right?
22:17
And that's where you already probably provisioned your vector
22:21
database. Unless you have one of the
22:23
managed solutions, which is also fine, right?
22:25
But for many people, they go to their cloud, they provision
22:30
their whatever, XYZ database and and then it's just running
22:37
there. You know you kind of pay through
22:39
the same billing account and and then the the same way you should
22:43
be getting your vector compute sort of system ray.
22:49
You should be pointing it on one hand at the vector database for
22:53
all the vector keeping vector indexing needs right?
22:57
And you should be pointing it as where the data comes from right?
23:00
Because as we said, right, if you, let's say you are a
23:05
marketplace with houses, you know all kinds of things about
23:08
those houses, right? And those these data lives in
23:13
different systems, right. So you know their data
23:17
warehouses, their databases that is kind of the BLOB storage
23:22
where you might have other types of features that you compute
23:25
with some other system. Typically.
23:29
You know there is kind of I like to separate this computer
23:32
problem from ETL. So I like to think about it as
23:37
kind of ETL is getting the data from outside of your system to
23:42
your system, right. So you pull it from your mail
23:45
teams and from your CRM and all of those million different
23:51
places. I think the modern data stack
23:54
kind of movement last couple of years pushed forward the idea
23:58
that OK, you should somehow like pull from all these different
24:01
tools and centralise it and somehow organise it in your core
24:06
infrastructure. And it could still mean a few
24:08
databases, but you know, it's like something sane and and then
24:13
you know, compute problems, any kind of machine learning
24:16
workload should kind of start from those core pieces of the
24:20
infrared and take it from there, right.
24:22
So that's why I usually say kind of ETL is kind of the step one
24:27
and then any kind of pipelines, model training, all of that work
24:32
is kind of on top of it. And then you mentioned one also
24:35
important thing which is kind of the model inference.
24:39
So normally you have this kind of train, infer loop, right.
24:43
And with the language models now maybe that should be renamed
24:46
signed to infer, right. So there is usually kind of Sam,
24:52
Sam, some Sage Loop. And yeah, like you have to
24:58
figure out when you are building your own, you know, vector
25:02
computer, you have to figure out, OK, we're not winning the
25:06
models and how are we making those, let's say inference
25:09
services accessible from our pipelines, right.
25:13
This is the kind of these were that advertised basically,
25:16
right. We have some Kafka Streams
25:19
coming in. We have some Spark pipelines
25:23
shovelling data around and somehow from those pipelines and
25:27
all of those workers and those pipelines we have to either run
25:31
the models on the same nodes or we have to create an inference
25:37
service that we call from those pipelines.
25:39
And then kissing and not embedding things multiple times
25:44
or actually re embedding things when the model version changes.
25:48
I do this kind of then the intersection of data
25:51
engineering, machine learning, engineering, right.
25:54
So those are some of the things you need to think about when you
25:57
build vector compute. I gotta tell you man, because
26:05
I'm not sure how I feel about the vector computer name, but
26:11
you said so many incredible things in the last five minutes.
26:14
I'm gonna let it slide alright, but the vector computer, we're
26:17
gonna have to think about a different because I'm not sure
26:22
it has that that spiciness, that touch that I'm looking for.
26:26
But whatever. Who am I to tell you how to do
26:28
your thing? So the marketing guy you know,
26:30
you tell me I I'm just an engineer trying to you.
26:33
Know. Yeah.
26:35
Goodbye. There's so much incredible stuff
26:38
that it's doing. I wish the name could encompass
26:41
that more than Computer Vector Computer, but we'll figure that
26:46
out. Maybe by the time this airs
26:47
we'll have a different name for it and I'll put it in the
26:50
beginning as like an A dumb addendum, however you pronounce
26:53
that word. So OK, there's.
26:55
There's a lot to unpack there though, when it comes to
26:58
everything that you just said. And I'm gonna try and just goes
27:02
through it in my head where it's like, right.
27:04
Yeah, you have the ETL that comes through where you're
27:07
grabbing all this data from outside.
27:09
You're throwing it into whatever your S3 buckets, your data
27:12
warehouses, all that fun stuff. There's nothing new there when
27:14
it comes to ETL. I think everybody's fairly
27:17
familiar with that. Then you have the data in your
27:22
ecosystem, and from there you're going to be taking that data and
27:28
throwing it at the vector computer so that the vector
27:32
computer can create different and.
27:35
And you're not creating embeddings from this like the
27:40
embedding piece is a whole different area.
27:43
You're taking the data and you're creating embeddings in a
27:47
different route, like. Can you breakdown how does this
27:49
work with embedding models and all that fun stuff?
27:52
Yeah, so you are taking everything you know about the
27:57
entity that you want to process, right?
28:00
So if it's your user, then it's all their behavioural data, all
28:03
their metadata, everything you know, right?
28:08
And you are somehow either training a custom embedding
28:14
model that can eat all of those different types of data that you
28:17
have about your user and produce the perfect vector that encodes
28:22
everything you know about them, right?
28:25
So you either do a custom model and then you bring out your
28:29
Pythor to create all the layers that can process feature
28:33
engineer. Because some of these inputs
28:36
will be like variable length click sequences and whatever,
28:39
right? So you you either build a custom
28:42
model and then you give it some sort of objective for a
28:47
downstream task and then kind of back propagate the error and the
28:50
typical kind of machine learning work that we all do.
28:54
Or you find, let's say on the opposite extreme, you have your
28:58
language model serving in some entrance service in your own
29:03
cloud, or use one of the, you know, providers that can help
29:06
you serve open source model, let's say, and that they'll
29:10
have. Fortunately, people stringify
29:13
everything they know about the user and send it to the language
29:16
model and make the vector that way, right?
29:18
Those prompt and blitz, yeah. So I think in an ideal scenario,
29:25
you have figured out the way to reconcile those two worlds,
29:28
right? Because there will be some bits
29:31
of text that you have on your user, right?
29:34
And so for those using some sort of language encoder, is is is a
29:41
good idea right? Because this has been trained on
29:44
a lot of data, it's good, right? So, So for those isolated bits
29:49
and pieces of text, if you have some way to put some framework
29:52
around it that doesn't require you to first flatten all the
29:56
string bits together but somehow keep it separate, it's a good
30:01
idea to use a language model, right?
30:03
But for for some of the other stuff, yeah, you might need an
30:07
image embedding, you might need the GRASS embedding, you might
30:10
need the time series embedding. Maybe you just want to encode
30:14
the timestamp, but then do it in a way that when you do cosine
30:17
similarity afterwards, it's like a time delta calculation, which
30:21
is like another kind of math puzzle so, so you know.
30:26
Long story short, you are collecting from your internal
30:31
data infrastructure everything you know about an entity.
30:35
And you want to have some process that runs through all of
30:38
the instances of that entity and turns them into vectors that
30:43
describe that encode and compress all your organisational
30:48
knowledge. Because then when you do
30:50
retrieval on top of that thing, or when you do any kind of
30:53
modelling on top of it, the input will be as rich as
30:56
possible, right? You have the highest chance to
30:59
actually make this thing to to work, so that's one of the goal.
31:04
So I'm starting to wrap my head around it and excuse me for
31:08
taking a little bit longer than expected on this because I was
31:13
too caught up with the name and I wasn't focusing on the actual
31:16
important part. But the idea here then if I'm
31:20
understanding it as you're saying, being able to fine tune
31:25
and embedding model is a superpower and it's very
31:29
important when it comes to Rags. So let's just take Rags as an
31:31
example, right. And so you want to make sure
31:35
that that fine-tuned model is as accurate as possible.
31:39
The way that you can make it more accurate is by creating the
31:43
right pipelines with the data coming from.
31:46
So it gets ETL, then it's in your data warehouse or S3
31:50
bucket, whatever. Then you're going to create a
31:53
pipeline. It's going to go through the
31:56
vector compute and it's going to be used because the vector
32:00
compute is able to weigh what's important, what's not, and then
32:03
you're able to fine TuneIn an embedding model off the back of
32:07
that in the pipeline, kind of looking at it sequentially.
32:13
I mean, that's like we're getting there, we're getting
32:15
there. Keep it going, then.
32:19
Keeping yeah. Tell me where I'm wrong.
32:21
Tell me what else? So it was all correct except
32:24
when one aspect, which is when you make this vector for every
32:29
user. Think about it as a problem of
32:32
just kind of collecting all the available information but not
32:36
yet putting in the biases. So you want kind of raw vector,
32:41
you want you want the vector that encodes everything you know
32:44
about the user, but kind of normalises the components so
32:48
that at the inference time or at the, you know, when you actually
32:53
do the querying. For example, right here in first
32:56
time is partly because is it the inference of the embedding model
32:58
in the English timeline or is it the inference when you actually
33:02
query and you want to have your access to answer the question,
33:06
right. So when when we talk about this
33:09
sort of like end inference, right, so we are training a
33:11
model on top of the vector, we are doing retrieval on top of
33:15
the vector. That's the time when you want to
33:18
start playing those biases, right?
33:20
Because then you can abit as you can be kind of fast and nimble.
33:24
You can converge to biases that now you know hit the sweet spot
33:29
for your different for example, screens where you display the
33:32
recommendations, all of that stuff, right?
33:34
So that you want that to happen as late as possible that there
33:38
and then in terms of the vector formation, you know, how do I
33:41
make thousand float numbers that encode everything I know about
33:44
my user or my document chunk or house or whatever.
33:51
You want to index these vectors into your vector database, but
33:55
you want to make them not opinionated at all, right?
33:59
So somehow you want to normalise the strength to which all these
34:04
different parts contribute, right?
34:06
Do you care about freshness or do you care about relevance?
34:09
Well, 5050, that's the index, right?
34:12
And then and then when it comes time to OK, we have the new
34:16
page. OK, so there we want to boost
34:19
freshness, right? We have the 4U age, OK, little
34:23
bit more relevance, right. This is a concern that happens
34:26
kind of at the final moment of just pulling the data together
34:31
and putting it in front of the user or clustering your machines
34:35
and manufacturing like, OK, in this moment, in this context,
34:39
which machines will break, right.
34:41
And and that's the time to put in the the biases all the way up
34:45
to them. You just want to take everything
34:47
about your machine and boil it down to the vector to just
34:52
encode all your observations in kind of impartial way, let's
34:57
say. OK.
34:59
And so then the vector computer comes in in that last moment to
35:04
make sure that it weighs out everything properly.
35:08
Both in the ingestion and in the last moment.
35:10
So basically our component basically works with the vector
35:15
database both for the industry and for the retrieval and and
35:23
you know then for the using the vectors as the feature vectors.
35:29
So you are training another model on top of the output of
35:31
the vector computer. That model can learn its own set
35:36
of biases, right? So in that workflow, the
35:41
workflow you you, you don't necessarily need the vector
35:44
computer anymore. But what will happen is that the
35:46
model that uses the vector as an input instead of click sequences
35:52
for users will be much simpler, right?
35:55
So now you have. Now we have this model that you
35:57
are training on labels and vectors, and you know like a
36:01
floating point vector is, you know it's.
36:03
For a language model, it's That's exactly what it wants to
36:08
eat, right? OK.
36:10
So then if I'm understanding this again, I'm just going to
36:13
keep like throwing it back at you to make sure I understand
36:16
because I really want to wrap my head around it and I think I'm
36:19
there, but I'm not sure if I'm fully there, so feel free to
36:23
correct me. Again, the reason you're
36:24
flattening it out is because you want the vector computer to do
36:29
all the heavy lifting. You want all of this to be as
36:32
flat as possible, so that later you can add in some kind of
36:36
logic and the vector computer and let that give you the
36:39
control that you were talking about before.
36:42
Yes, yes. Alright.
36:44
If it's maybe like you take A roll photo, right.
36:48
That's like a bit kind of greyish, right?
36:50
Like underexposed or whatever they say raise.
36:52
Like, yeah, you know, it kind of looks boring, but it preserves
36:56
the most information, right. It preserves the most
36:59
information about the scene. And then.
37:02
And then it depends. OK, do I want this for wedding?
37:05
Do I want this for social? Right.
37:06
You like, push the curves this way, that way.
37:10
And that's kind of the ingestion and the OK, I'll leave this, I
37:14
mean rap for my. There we go.
37:18
That is perfect because now I understand.
37:20
It's like I want to capture as much information as possible and
37:25
then I want to be able to do what I will with it.
37:28
However, I like to show that information to people.
37:32
I should be able to turn the knobs in the way that I want to
37:35
turn the knobs and then it gets outputted in something that I'm
37:40
happy. With.
37:41
Yeah, yeah. That's that's exactly right.
37:44
OK, All right. So now it's starting to come
37:47
together for me where I'm seeing, yes, this is giving you
37:52
more control because you're making the decisions with the
37:57
vector compute as opposed to what you would traditionally see
38:02
in the RAG pipelines or even Rexus.
38:05
And it's like just going straight from the vector
38:08
database and maybe you've got your embedding model that throws
38:12
the embeddings into the vector database and then the retrieval
38:15
and it goes to the LM. And maybe what goes to the LM is
38:19
right, maybe it's not. And you've also got your prompt
38:22
templates. And.
38:23
So now what you're saying is before the embeddings go to the
38:30
LLM, they're hitting the vector compute, and then the vector
38:33
compute is turning the knobs to say this is more important, this
38:37
is less important, and we feed it to the LM in the way that we
38:42
want. It's like our secret recipe now.
38:44
Yeah, exactly. And very importantly in that
38:49
sort of progression that you mentioned, when you add the
38:53
query time, you are for example learning or or AB testing,
38:57
what's actually important use case, you don't just use that to
39:02
reorder the candidates that you already retrieved, right.
39:07
This is, this is the one of the problems that exist in the space
39:10
is that people do that initial retrieval kind of them kind of
39:13
heuristic, right. Throw in a sub hybrid search, do
39:17
just kind of like simple embedding of the query and just
39:21
kind of see what gives them the text embeddings that come back
39:26
And then they try to be clever about, OK, let's reorder this
39:30
thing to match whatever objective we have.
39:33
But now you are reordering oh point 1% of your database,
39:37
right? You have retrieved a small
39:40
fraction of all the stuff you have and now you are trying to
39:43
be smart on top of that. The obvious problem with that is
39:47
that there is all these other stuff in your index that maybe
39:50
you didn't even surface right. And so when you edit the query,
39:54
time when you are deciding okay, freshness is this important,
39:58
popularity this important, relevance this important.
40:02
It's important to express them in a way that it informs the
40:04
underlying retrieval, not just I'll retrieve thousand things
40:09
and then I'll have my sort of kind of linear combination
40:13
little model that figures out the preferences and then I'll
40:16
reorder because that misses kind of the point.
40:20
So you know this kind of retrieval time vector computer
40:26
objective is to express these different goals and then
40:30
formulate the search vector such that when you go and do the
40:33
nearest neighbour search in the vector database, you are already
40:37
in the sort of step boosting certain aspects and so on.
40:40
And finding the globally optimal set of items that have the right
40:45
trade off between how far is the house, how big is the house you
40:49
know is the like price history similar to how is that clicked
40:54
on before right like you need to be able to express.
40:57
This the add yeah? Yeah, and send it to the index,
41:02
right? And then the index should be
41:04
globally finding you the 10 houses that already kind of
41:09
fulfilled that combination of objectives and then you just
41:13
returned it to the user, right. So at the start of this
41:16
conversation, we talked about how our objective for a couple
41:19
of years was to eliminate the need for the re ranking by
41:24
making the underlying search smarter.
41:26
So while this is how that works, right, that you kind of express
41:30
the objective, we construct the third vector in a way that's
41:35
where the boosting happens, so that when you send it to the
41:39
nearest neighbour index in the in the vector database, it's
41:43
it's already doing that kind of opinionated combination of
41:47
factors, right? And the underlying service,
41:50
cosine similarity we don't have. I don't think you need something
41:53
like special scoring functions in the index itself, because
41:58
that's slow, right? You have to be building an index
42:01
without user defined functions. That really goes well.
42:05
And so you need some way to do this with like vanilla nearest
42:11
neighbour search with cosine or dot product or something.
42:15
And yeah, it's possible. Like we have been discussing
42:20
like a hypothetical system so far but but you know like the
42:26
the the company is is is building so super linked is is
42:31
building this this this vector computer actually by the time
42:36
this goes live we might have sort of done the initial kind of
42:41
public relief already so. There we go.
42:45
Just the thing that. Everybody go to
42:47
whatsuperlink.aior.com. Right.
42:54
Everybody just listening. He just gave me the biggest mean
42:58
mug ever, first saying dot AI? No, no.
43:02
Well, we we have AI as well I think, but but the Ohh even
43:07
better. So you can go to superlink dot
43:09
AI and get redirected to superlink.com.
43:12
OK, cool. OK, yeah, so for anyone there
43:14
after the recording, but. OK, we'll cut it out if you
43:18
don't. People should go to
43:22
superlength.com and learn. You know, learn about other ways
43:28
of building vector computers, but I think this will be like a
43:33
whole another field, right? So you should also kind of, you
43:37
know, hopefully we kind of covered between ideas for how
43:41
you can build a vector computer of your own.
43:45
And I think for some people that definitely might make sense,
43:50
have been thinking about this for a while and you know we're
43:52
happy to to chat and kind of share what we learned and maybe
43:58
just sort of like as we come up on the on the wrap up.
44:04
I mean, wait, don't go anywhere. Yeah, I still got so many
44:06
questions for you. Do you have a hard stop?
44:09
You do. I can go.
44:10
I can go a little bit overtime. OK.
44:12
I can give them 10 minutes. Ohh.
44:15
Man. Alright then, speak fast, man.
44:17
You talked about speed. Yeah.
44:19
Yeah, How does the, what have you seen when it comes to speed?
44:23
Like, does this speed things up? Does this slow things down
44:26
because it's one more step? Is it negligible?
44:31
You have to do this step anyway. You have to turn your data into
44:34
vectors and typically the language model which is kind of
44:39
taking the brand of these workloads right now is the
44:43
biggest possible model that you can be using to turn data into
44:46
vectors. There's for different types of
44:49
data, there are specialised models that do it much more
44:52
efficient. Play long train, let's say graph
44:56
embedding for for the graph nodes to encode the structure.
45:01
It is much smaller model than the language model.
45:04
So actually kind of, you know, only using language model where
45:11
you absolutely have to and then using specialised models is kind
45:14
of one of the themes of how you make all these systems more
45:16
efficient. So it's definitely there is
45:19
there is a part of that being reasonable around the
45:23
infrastructure, how you organise the pipelines, Is the model
45:27
collocated with the pipeline workers or is it a separate
45:29
service, You know, you know, you need people thinking about that,
45:32
that type of stuff when you build this yourself.
45:36
That's where the games come from, right.
45:40
So yeah, I think for, for like if you have a scale of kind of
45:45
single millions or maybe low 10s of millions of data points, your
45:50
main problem will be how do we experiment, right?
45:54
How do we sort of try different ingestion policies and and query
46:01
policies in some environment where we can eyeball or throw
46:04
things in front of the user? Quickly iterate right step one
46:09
and then step two? Yeah, like how do we end the
46:11
back field? How do we get the service to be
46:14
reliably running in our in our cloud?
46:17
And how do we manage the lifecycle of the underlying
46:19
vector database? The.
46:22
These are kind of like basic concerns, but there is a bunch
46:25
of moving pieces right? And and then and you don't
46:30
really care that much about OK for everyone model ran, you know
46:37
how many milliseconds per vector does it take, right.
46:41
It's more the just kind of covering all the bases of the
46:44
process and and just kind of getting good at launching things
46:48
and then when it goes to sort of like 100 million plus data
46:52
points that's where you really start to care about OK, are we
46:57
serving this model as efficiently as we can, right.
47:01
And you know, each cloud now has some managed language model
47:04
serving offerings or their third parties and so then it starts to
47:10
make sense. But you know, just to give you
47:13
some like specific number A-100, for reasonably small chunks of
47:18
text with reasonable embedding model can make thousands of
47:22
vectors per second. So did you lose the belief?
47:30
And it's not gonna be with the GT4 or with like Lama 2, but
47:35
but, you know, everybody knows the massive embedding table and
47:39
on that table you can find models that have one or 200
47:44
megabytes basically. And then yeah, those are those
47:48
are fast. Sweet.
47:49
So there is something else that I wanted to call out which I
47:52
think is worth everyone that is listening.
47:54
Go and check out some of the great work that you're doing.
47:57
Vector Hub too, if they haven't seen it already.
48:00
If anyone wants to compare vector databases, you've done a
48:04
great job because you have no dog in the fight basically.
48:08
I really like that you're a vector compute, so you don't
48:12
have to worry about the vector databases.
48:14
You're compatible with all vector databases, I imagine.
48:17
And that gives you a nice position to be like, OK, here's
48:21
what each vector database does and doesn't do, and here's a
48:25
table so that if anyone wants to see what vector databases do
48:32
what, then they can go to vector hub dot a icom.
48:37
Superlink. No, no, it's it's Hab.
48:40
Ohh God, I got it so wrong. We'll leave it in the show
48:43
notes. Have have looked superlength.com
48:46
is or the maker have where we have basically contributor
48:53
provided deep dives on Information Retrieval
48:57
reflectors. So Rex is Semantic search rag to
49:02
do this stuff in browser introduction we try to be quite
49:07
practical and little bit advanced, right?
49:11
Not your totally basic thing, a little bit more advanced tricks
49:15
and tips and so that's the vector have have got
49:19
superlink.com and then yeah we have basically partnered with
49:25
the different vendors in the vector database space to try and
49:29
map out the different offerings, right.
49:35
For this is one of the problems that we hear when we chat with
49:38
new folks coming into the space is they're trying to wrap their
49:41
head out. OK.
49:41
When do I use what right? Which one is good for?
49:46
If I have many small indices or few big indices or how does the
49:51
pricing work with that? Or is it running bundled with my
49:56
own app or is there a managed offering or which hybrid search
49:59
features, right. So we have basically for 36 or
50:02
so different vendors, we have a list of something like 30
50:09
different features. And then what we tried to do
50:12
with the with the vendors is not just say, OK, the database
50:15
supports the feature or partially supports it, but also
50:19
link to the documentation for that specific feature, right.
50:24
So when you are kind of learning about the space and you are you
50:27
clear about certain things, it's very easy to go and see OK,
50:31
which database supports the functionality I need and then go
50:34
to the documentation to kind of double check does this actually
50:37
mean what I think it means. And and you know the the vendors
50:42
kind of send us it's it's backed by GitHub, so it's GitHub repo
50:46
with Jason files for each for each vendor and the vendors send
50:50
us pull requests to keep the data up to date.
50:53
So this has been working quite well and it gave us a way to
50:58
kind of chat with all these people and you know, build that
51:01
relationship as well. So as you said, you know we
51:03
don't have a horse in the race. It's just that we need databases
51:09
that that work well and that people, you know a kind of
51:13
comfortable with and for us that made total sense to kind of put
51:17
together the data set and just help everybody understand this a
51:20
bit better. So lucky for you, you get to
51:23
benefit from the hundreds of millions of dollars that have
51:27
been invested into vector databases in the last year
51:30
because you can partner with all of them.
51:32
So just jump on the hose marketing trends and get after
51:38
it. Yeah, exactly.
51:39
And and in some way help them deliver on that kind of
51:43
overarching promise, right. So just basically everybody is
51:50
excited about deep learning based retrieval and feature
51:54
engineering work, right? This is the top of mind for
51:59
people, and a lot of the sort of materials out there make it look
52:04
super easy, right? So you can just you know then
52:09
something through open the eye and retrieval quality will
52:13
happen and then in practise folks get into this project and
52:19
it takes quarters to get something like really, really
52:23
working and we want to sort of remove the disappointment,
52:27
right. So like everybody is excited and
52:30
we want to keep the happy train going and people out existence.
52:34
I think, yeah, it's going to take both of those pieces of the
52:38
compute and the the the vector indexing and manage and those
52:44
pirates have to work together basically.
52:46
So that's that's what we want to do.
52:49
Well, I love what you're doing. I also want to call out that
52:53
it's not just for rags, right? Like I think we made it pretty
52:57
clear, but just to reiterate that anyone doing anything that
53:01
deals with embeddings, which of course recommender systems are
53:05
huge when it comes to that. And that is more of a
53:08
traditional ML type of system as opposed to the LM type system.
53:15
Vince, Vector Compute feels like it is also really interesting
53:20
for that and it can help your system perform better.
53:24
Yeah. And then another kind of big one
53:26
that we didn't really discuss that much analytics, you know
53:30
you want to find customers that behave similarly.
53:33
You want to find the your your machines in a factory that will
53:38
break at the same time. So doing kind of clustering on
53:42
top of these vectors or training other models or just doing OK,
53:45
show me similar products that sell similarly things like that.
53:52
You know, I think by the end of this year, you know if we do a
53:56
good job on partnerships, people will be doing these kind of
54:00
queries in their business intelligence and they will not
54:03
necessarily know that actually they ran a kind of super linked
54:07
workload under the good. But they will be tapping into
54:11
the this kind of dream of deep learning based retrieval, right.
54:17
That's the goal is to kind of make it accessible to as many
54:20
people as possible when they ordered the work today, right.
54:24
So the hype is awesome, but you know, I I really love to work
54:29
with folks who have the ordinary problems and they just want to
54:33
get their job done and kind of meeting them there, right?
54:37
Yeah. Ohh man, that's so cool.
54:39
I'm getting goosebumps just thinking about it.
54:42
I love what you are working on. I appreciate you coming on here
54:46
and explaining it to me like I'm five.
54:48
I know it took a little bit longer to get it, especially
54:51
because this is like the second or third time that I've talked
54:53
to you about it then which I didn't want to mention that
54:56
while we were talking about it. I should know by now, but I was
54:59
just, you know, going a little slower so that all the listeners
55:04
out there could go stay with me and.
55:06
Hopefully everybody along exact. I wish it was like that, man.
55:12
I have a bit of a baseball moment for me, but.
55:16
I will say we go, there we go. There.
55:18
Yeah, exactly. Well, hopefully I I do have a
55:21
better understanding. I will encourage everyone that
55:24
is listening to this. If you're interested in it, go
55:27
to superlink.com. Also, if you want to partner
55:30
with Daniel and the Superlink team, get after it and make sure
55:34
that you hit him up now because dude, I think you all are going
55:37
to blow up. I will say that right now I'm
55:39
going to call it and I I really hope so.
55:43
Alright, thanks a lot for having me and it's been a pleasure to
55:47
dive into this esoteric topics of vector embeddings.
55:52
And yes, around in the community and on the upcoming event.
55:58
There we go.