Latest / Agentic Conversations (formally mlops.community) / The Future of Information Retrieval: From Dense Vectors to Cognitive Search

Transcript

The Future of Information Retrieval: From Dense Vectors to Cognitive Search · Agentic Conversations (formally mlops.community)

0:00
Yeah. For image, I think that's we can
0:03
convert that into a vector. But for, I think for a movie,
0:09
what we can do is let's say we extract some of the frames of
0:13
the movie because it usually have a lot of frames and just we
0:18
can convert those into vectors. And then again when we want to
0:21
search again, yeah, we can match those vectors which with
0:24
whatever we have in database. So that can speed up here the
0:27
search query that we have. People found out real quick in
0:36
Rag that the R was the most crucial part of Rag and then.
0:43
Yeah, I think RAG, yeah stands for Yeah, Retrieval Augmented.
0:47
So if Yeah, RAG is not doing retrieval, then yeah, I'm not
0:51
sure what RAG is actually doing. Yeah, you just are entering a
0:55
wild card into classical search when you throw the generation
1:03
into it. And so it can be, unless you
1:05
really got that retrieval and that search down, you can have a
1:10
hard time. It's not like, oh, we're going
1:12
to add another limb to it and it's magically going to change.
1:16
Yeah, I think LMS are part of it.
1:18
Like they can try to enhance the results like like the the what
1:24
the A stands for, like they can try to augment the results what
1:27
the retrieval layer has already given.
1:29
But yeah, unless you have a retrieval layer and that is
1:33
functioning properly, then yeah, the G like the generator here,
1:37
it cannot help much. So retrieval is yeah, definitely
1:41
the most important part of a RAG system.
1:44
Well, dude, you've been knee deep in search for a while now.
1:48
Basically, you are the coveted search engineer and I want to
1:55
learn all about it. I've been loving talking to
1:58
folks about the new ways that we're doing search, and search
2:03
is so topical right now since context is so important for
2:08
agents. And how do you get that context?
2:12
Well, you got to search for it, right?
2:15
Maybe you can start by just giving me a bit of the evolution
2:19
of search. OK, yeah, so search, I think so
2:25
in the traditional like information retrieval systems or
2:28
search systems like what we used to do is that we just use let's
2:32
say type queries and the search system will match your query to
2:37
some document. That document might have your
2:40
the query that you're looking for.
2:42
So it will just try to look up OK which documents have the
2:45
queries that the user wants and justice based on that keyword
2:48
matching using some data structure like mostly it's
2:51
inverted index, it will return like some document that these
2:55
are the top links that you are looking for a top documents.
2:59
So now that was the traditional IR.
3:02
So now from that it evolved to dense retrieval.
3:06
So by dense retrieval I mean that now like your queries and
3:10
documents, now they are converted into embeddings now
3:14
with all sort of like language models, pretend language models
3:17
that have come in like we generate embeddings from the
3:20
query and also the document. So now then the search system
3:25
they don't like they didn't only match only the keywords, they
3:28
also try to match the semantics like the meaning behind it.
3:33
So you can like for example type synonyms.
3:36
If let's say you type a query, it will also try to match the
3:40
the words which are synonyms and which are similar meanings.
3:43
So then we wall from that and just the keyword based matching
3:47
to more like embeddings based matching and like bird
3:50
Transformers all of this they help us to create embeddings
3:54
fast and then match these embeddings.
3:57
And now we have seen that now from dense retrieval now we are
4:02
going towards the next phase which is the cognitive search.
4:05
So in this phase, like it's not just about like matching the
4:10
meanings or understanding the, the similarity between the
4:15
words. It's also trying to like
4:17
understand what's the context behind the search.
4:21
So in the search system now, they like try to solve the
4:24
problem, try to understand the goal, what a user has in terms
4:29
of what they're looking for, like what's the eventual goal
4:32
versus just matching some documents or like I'm giving
4:37
them some answers, we just made their queries.
4:40
So, and this is what like cognitive search has become now.
4:45
Wait, I'm not sure if I fully understood cognitive search and
4:48
I know the semantic search got huge, right?
4:50
We're generating embeddings and we've, you've also got folks
4:54
that are saying, all right, well, it's actually super
4:57
effective if we do keyword search and semantic search and
5:00
we get that hybrid thing going on.
5:02
But cognitive search is trying to just understand the intention
5:09
behind the search and then figure out from there.
5:13
Yeah. So for example, if you want to
5:16
search something and I think your ultimate goal in that
5:20
search is not only to just get some documents you want to like
5:23
solve some problem. So semantic search tries to
5:26
understand what's your eventual goal is like after, let's say
5:29
searching this first of all, what's your context behind this
5:33
search? And after searching that query,
5:36
like where do you want to go next with that?
5:39
So it also like in terms of it can do a series of actions for
5:42
example return answer to 1 query based on that take another
5:46
action. So it will help user to
5:48
eventually reach to their goal. And also based on like now we
5:51
have LLM so it can also add like personalization to search and
5:56
whatever information that is generating or like retrieving
6:00
for the user. And how is this different under
6:03
the hood? Yeah.
6:06
So under the hood now we since I think I mentioned we already
6:10
have LLMS in place. So LMS is what they, they help
6:15
to like first of all, they, they also help to reason what the
6:20
query that has been returned by the search system.
6:23
Like they don't just return for example, this is what's your
6:27
answer to the search query. They also try to reason like and
6:31
also try to think as a system that this is what I return and
6:34
this is what's the reasoning behind the search system.
6:37
So LMS are mostly powering like all this paradigm shift towards
6:42
cognitive search, yeah. The idea of the next best action
6:50
or why are you making this search and can we leapfrog what
6:56
you're trying to do here and just get you what you're looking
6:59
for is a fascinating one, especially because I've seen
7:03
that pop up now and again with when I'm talking to a LLM or a
7:12
chat bot. And then it says would you like
7:14
me to do XYZ next? Is that?
7:17
What you're saying? Yes.
7:20
Also like previously, we used to, for example, measure the
7:24
effectiveness of a search system using the, let's say the
7:28
relevance of the query. For example, you have like these
7:33
many documents and out of these like we return these many.
7:37
So what's the accuracy of this search result?
7:40
But now we also like have started looking at not only the
7:44
accuracy, we want to look at the like happiness of the user.
7:47
For example, like is the user even happy?
7:50
Like they understood what this search result return and did
7:55
they? Does this actually satisfy what
7:58
what they're looking for? One thing is also, for example,
8:00
let's say your search system return a result and what action
8:04
did the user do next? And did they, for example, click
8:07
on some post, did they do some action which reduce the next
8:10
search queries that they're doing?
8:12
So in this case, we can determine not just by looking at
8:15
some metrics like recall and accuracy, we can see where the
8:18
user is going next and we can use that to data mine if your
8:23
search system is actually doing what it's supposed to do.
8:26
What are some high value signals that you can evaluate if the
8:33
search is doing what it's supposed to be doing?
8:35
Yeah. So I think I mentioned one of
8:37
them is let's say a user sent some query.
8:41
Now let's say if the system didn't satisfy the user's
8:44
requirements or it didn't answer the query that it's supposed to
8:48
and then user is going to type some more follow up queries and
8:52
then they will keep on asking that some follow up questions
8:55
for that. So 1 signal is if we see that
8:58
the user hasn't like typed too many follow up queries, that can
9:03
be 1 signal. Like it's not the utmost signal
9:05
like it's there are many other reasons why the user is not
9:08
asking follow up questions. But that can be one of the
9:10
reasons why like your search system is effective.
9:14
Now also, let's say you returned some results and if you have
9:16
let's say call to action, for example, out of those search
9:20
results, you want user to let's say buy something, click on
9:22
something, go visit some website or something.
9:25
Based on that you can still evaluate that OK based on just
9:29
this search. Searching for this query once
9:32
like user got what you wanted. When they make a sale for
9:34
example, they make a purchase, sorry they made a purchase for
9:37
example. Uh huh.
9:40
So now when you talk about relevance in a way that is the
9:47
new relevance. Yeah, I think for both.
9:52
I think for some for like the dense retrieval systems where
9:55
we're using, let's say, embeddings to generate all of
9:59
this. I think the relevance was still
10:01
measured in terms of accuracy or for example, there are many, I
10:06
think we call and all of that. But now I think if you want to
10:10
design for these cognitive search then this becomes the new
10:14
metric that we care about the user rather than just some
10:20
comparing what the query return. And you know, like with dense
10:27
retrieval and just like vectors and embeddings in general, you
10:33
could throw in a library, you have very battle hardened
10:38
libraries that I think everybody knows like face and that would
10:44
get you pretty far. Like is there libraries that you
10:47
can use for the cognitive search paradigm?
10:51
Yeah, I think that is, yeah, I think not as evolved right now
10:56
as we have the dense retrieval system.
10:58
So I think in, if you want to, I think build for example, agentic
11:03
search capabilities, you have some libraries to build the
11:06
agents, but like we don't have those libraries where you can
11:11
just like use them to compute the metrics.
11:13
And this is because for each system, I think these metrics
11:17
are different because for some retrieval system, it might mean
11:21
you're, let's say your user is doing, let's say, X action.
11:24
And for some others, it might be something different.
11:27
So I think yeah, because for, because now in each system we
11:31
are not not using some standard like metrics or accuracy term.
11:37
So right now, yeah, there are no, I think such, yeah, better
11:40
tested libraries which we can use to evaluate this.
11:44
Yeah. I wonder if there ever will be
11:47
like if we're going to have a consensus on what these terms
11:54
are or what we evaluate for in that regard because it is so use
12:00
case specific. Yeah, I think I agree.
12:03
So I think for some of these terms, for example, let's say
12:06
for if you want to count the number of follow up queries that
12:10
have reduced, I think we can have some libraries to do that.
12:13
We can have some I think libraries to let's say track
12:16
general actions. For example, user clicked
12:19
something or out of out of your search results or they went to
12:23
that landing page. Yeah.
12:25
But I agree that since this is, this is going to be very
12:29
different for each system. I think you're having some
12:32
libraries. Is, yeah, not trivial.
12:35
Well, and it really what we're talking about really goes into
12:40
the area of evals and it's almost like product specific
12:50
evals kind of sort of you know, if somebody is asking a bunch of
12:56
questions to get what it wants, that should be caught in the
13:01
eval sequence, right? OK, but eval sequence you mean
13:06
what? I guess sequence isn't the right
13:09
word, but it should be caught with hopefully when you're
13:12
reviewing your data and you're seeing like this isn't working
13:19
as well as we thought it was, or the chat bot is not answering
13:23
the question or it's not doing what the user is expecting it's
13:27
doing. And so when you're evaluating
13:33
the multi turn conversations that these chat bots are having,
13:37
you catch that ideally. But to do that, you got to spend
13:41
time with the data, you got to actually evaluate it.
13:44
And maybe the maybe you can train a judge if you're doing
13:50
LLM as a judge to pick up on that type of thing.
13:54
And I guess that's what you're talking about with like the
13:55
libraries of, hey, if they keep asking the same question or if
13:59
they're repeating, if the chat bot is repeating itself, that's
14:03
like a bad signal. Yeah, I think in also if you
14:06
want to, yeah, let's say evaluate, Yeah, like catch it in
14:10
dumps during eval. Yeah, this can be one of the
14:15
manual review steps which we can take to evaluate how the
14:20
retrieval system is doing. I think.
14:22
Yeah, it can be also automated. I think the number of follow up
14:25
queries, I think that's a bit simple, Yeah.
14:29
So we can do like both manual and automated well to catch this
14:33
early. How have you seen the best folks
14:37
like doing search or giving agents a search tool to
14:45
leverage? Is it that you each search tool
14:50
is its own thing, like each way to search, whether it's hybrid
14:54
search or dense vectors or whatever keyword search?
14:59
Or is it abstracting away and it's just one search tool and
15:03
then under the hood you figure out how that works?
15:06
Like, have you seen good ways versus bad ways?
15:11
Yeah, I think so production based systems like if you they
15:16
usually have this abstraction where they hide all these things
15:20
like your system is internally it can do like multiple of these
15:24
things. So usually as systems in a prod,
15:26
they, they don't tend to, let's say stick to sparse retrieval or
15:30
dense retrieval like they tend to do all of these because in
15:34
production we have to like think about or worry about all of
15:37
these things. For example, there's cost
15:39
factor, there's latency factor. Speed is important and we have
15:43
to care about accuracy. So now like for, for things like
15:48
if you want to really speed up things you want to like have
15:52
very good accuracy, then the traditional like sparse
15:55
retrieval search, that is like a very good option.
15:58
Also you don't because for if you want to go into the dense
16:01
retrieval it you have, you'll have to like factor into many
16:04
things like GPU's cost will go up.
16:06
So you like can't always run your queries in using just a
16:10
dense retrieval. So usually there is there's an
16:14
abstraction and based on like the query needs also it, it can
16:18
decide whether it has to like go to which route.
16:21
Like most, most of the use cases, it can be satisfied
16:24
reliably using the sparse retrieval, using the inverted
16:28
index. But yeah, depending on like if
16:32
we have resources to spend, if we want to like for example
16:37
trade off between accuracy and speed, then we can it it can do
16:41
the sparse retrieval also. Talking about the scale, since
16:46
you're saying, hey, think about trade-offs, think about are you
16:49
cool with latency? Are you cool with cost?
16:54
I know that you run some gigantic scale systems and
17:00
you've played around with embeddings and we're talking
17:04
like how, how big are how many embeddings Like you, you've been
17:08
on some really yeah side of things.
17:12
And you mentioned to me before that if you're not careful, that
17:16
can kind of bite you in the ass. So can you talk to us about what
17:22
we need to watch out for when you're running at that scale?
17:26
Yeah, I think so. I think when you have for
17:30
example, let's talk about the vector embedding.
17:33
So I think big companies like in production system, they have
17:38
like billions of billions of vectors embeddings that they
17:41
save. And then at that scale it it, it
17:45
like not only becomes a simple modeling problem, like how do
17:49
you need to generate the embeddings, it also becomes like
17:52
in lot of way distributed systems problem.
17:54
So for example, how are you going to effectively store the
17:57
embeddings across, let's say you have a different partitions of a
18:01
cluster and also like how are you going to search the
18:04
embedding? So this is again, and you have,
18:06
let's say multiple partitions. You have to like retrieve all
18:10
those embeddings like effectively in real time while
18:13
serving any query. So how are you going to do that?
18:16
And then third problem is, let's say you have the embedding
18:19
store. Now you have new data coming in,
18:22
so you want to update the embeddings across all the
18:25
partition that you have. So how do you update those
18:28
embeddings in real time? That's also a big problem.
18:31
And also if you are storing your embeddings, you can't store all
18:35
of them just on on disk itself. Because retrieving that in
18:39
response to a query and then comparing them across with the
18:43
query and then returning the result to the user is going to
18:46
take a lot of time if you are going to do that from from disk.
18:49
So you have to decide what like how much embeddings you want to
18:54
store in real time to make it a live index, which we call it
18:57
live index because it's being served of memory.
19:00
And what like are you going to store in the in the the base
19:04
index, which is actually stored in the disk?
19:06
And when are you going to do that conversion?
19:08
Like when are, when are you going to keep refreshing your
19:12
live indexes from the base index?
19:14
So all this I think these are not only not only relates to
19:19
like modelling or embeddings, it's general distributed systems
19:23
problem which we have to like take into account if we are
19:26
dealing with embeddings at that scale.
19:28
Do you have a formula on how you decide that or is it case by
19:32
case based? Yeah, there's no, I think
19:34
formula hope for this. It's like it's unique to every
19:38
use case. It's unique to the size of the
19:42
embedding that you are dealing with.
19:44
You can just go away by like keeping everything on just one
19:48
computer hosted let's say remotely and just serving
19:52
everything from from the disk storage.
19:54
So it depends on your what scale you are dealing with.
19:57
It's I think doing like setting all this these things up in this
20:02
way for example like distributed and replicated, it's an over
20:06
engineer like people should not do it unless they really like
20:09
hit that scale. So they should just go off from
20:12
the base. I want to ask a little bit about
20:17
all of these different search techniques that you grapple
20:22
with. I imagine you're constantly
20:24
thinking about the trade-offs on each one, like cognitive search.
20:28
In my eyes, it sounds amazing, but it's probably not actually
20:33
going to be the most effective X percent of the time.
20:37
The same with hybrid search. It may be just plain old keyword
20:41
search is your workhorse that's going to be cheap, fast, and
20:46
easy for the majority of your queries, And then you need to
20:51
scale up. Do you think about like when to
20:56
use one versus the other and how you think through those?
21:02
In terms of scale, if I'm saying let's say if I'm just starting
21:05
up, I have let's say some users which want to search for like I
21:09
have some products in my let's say database.
21:12
Then I think I just start off with setting everything in just
21:16
the like the simple sparse retrieval way.
21:19
I'll just create an inverted index based on whatever data
21:23
that I have and then I'll store it somewhere and I'll just like
21:28
run my queries with that. Like I won't even care about
21:31
using GPU's or like doing all these vector based search
21:36
because I know that my, because users they like don't only care
21:41
about that they should get the most accurate result.
21:44
They also care about they should get the result like reliably at
21:47
all the time. Like your system is always
21:48
responding with something. I know that for, I think for
21:51
small scale, I think that's fine.
21:53
You don't have to be like 100% accurate returning results all
21:56
the time. And I think when you are
22:00
scaling, so I think in, I think that's again based on metrics,
22:04
you'll have to check that in what scenarios or how many times
22:07
your searches are not returning the results.
22:09
What is supposed to in that case, we can just work with the
22:13
all the the basic metrics that we have for search like recall
22:16
accuracy and all those things. So I think that will give us a
22:19
good idea of when to use the vector based search, when to use
22:23
GPUs to let's say start encoding your queries and also your
22:27
documents into vectors and then start using semantic search.
22:30
Then yeah, most of the small use cases, I think, yeah, that's not
22:35
necessary. Kind of in that same vein, I
22:39
talked with Jeff Huber on this podcast a few months ago who is
22:46
the founder of Chroma, and he was mentioning how there was
22:52
this paper that came out probably now six months ago.
22:57
And the headline was, you know, BM 25 is basically all you need.
23:02
It is the best that's going to work always.
23:06
But he did some digging and he saw that in that paper, what
23:13
they were running these benchmarks on, all of the
23:18
queries were uniquely suited for BM25 right now.
23:23
It was like an easy layup for BM25.
23:27
And so of course it's going to succeed because it was built for
23:31
that. And I can't remember the exact
23:33
queries that were made specifically for that, but that
23:37
was kind of his argument. I wonder if you see certain
23:42
queries in that regard where you're like, all right, BM 25 or
23:48
not. Like don't believe everything
23:50
you hear because BM 25 isn't all you need and it's much more
23:54
nuanced. Yeah, I think BM25I think for, I
23:59
think even for mid size, I think mid scale companies, I think
24:02
BM25 won't suffice. For example, let's say if you
24:06
have more than 10,000 or so users, yeah, I don't think you
24:10
can rely on BM25 to YEAH for your retrieval system.
24:14
Why is that? Yeah, I think it again, BM25 is
24:19
because based on again the text based retrieval, query based,
24:24
text based, sorry. So I don't think in BM25 you
24:29
even like like operated the vector level, you're not doing
24:33
any vector search on that. So I think most of the times
24:37
people are not, they don't know what they're looking for.
24:39
They would just want to type some query and the retrieval
24:42
system is should be smart enough to figure out what the user
24:46
intent is and based on that they have to return the result.
24:49
I think so having yeah, also I mentioned earlier if I think if
24:54
depends on the cost factor also like if you want to do a trade
24:59
off for that, you want to minimize your cost also, then
25:01
yeah, sometimes we can run on BM25 using text based retrieval
25:06
and sometimes we can do the tense retrieval also using
25:09
vectors. I'm trying to I'm trying to
25:11
digest this considering I like I, I am fairly interested in
25:18
search, but I am by no means that knowledgeable in it.
25:23
And so when I get to chat with people, I'm always excited about
25:27
it because I learned a ton. Maybe we can just, I can just
25:32
ask you this one, like, what are you excited about these days?
25:38
OK. And yeah, in information
25:40
retrieval, I think we talked about it briefly.
25:44
So I think having agent T capabilities is definitely one
25:48
of the things which I'm excited about South.
25:50
For example, by agent T capability, I mean that when you
25:55
give us, let's say, query to your system, your information
25:59
retrieval system, it doesn't just like return answer to 1
26:03
query and then stop. So it's also let's say do a
26:06
series of actions based on that answer.
26:09
So for example, the first one can be getting a response to a
26:13
query and then based on that it does another action.
26:16
Maybe it queries the same search system again.
26:19
So it does a series of actions. So this is what I mean by having
26:23
identity capabilities for information retrieval also.
26:26
Now I think search, I think it's already there at many places
26:31
like it's not based on only text based search right now, because
26:36
you can also search based on images, multimedia.
26:38
So multimedia retrieval, multimedia search is getting I
26:41
think I'm very hopeful for that. Like I think everyone now we has
26:46
support for this. I think that again calls for
26:49
vector based search because to match images to match any
26:53
multimedia with each other, you have to convert them into
26:56
vectors. So that's.
26:58
But you're converting frames of a movie into a vector.
27:03
Yeah, I think for videos, yeah, for image, I think that's we can
27:07
convert that into a vector. But for, I think for a movie,
27:14
what we can do is let's say we extract some of the frames of
27:17
the movie because it usually have a lot of frames and just we
27:23
can convert those into vectors. And then again when we want to
27:26
search again, yeah, we can match those vectors which with
27:29
whatever we have in database. So that can speed up here on the
27:32
search query that we have. Yeah, that I imagine you can
27:35
also grab the subtitles and add vectorize that too, or help with
27:41
the vector in the embedding. Yeah.
27:44
Or if you can, I think also just do an exact match with a
27:48
subtitle. Sorry, not the subtitle with the
27:51
metadata of the video for example title the tags, what it
27:53
has. So yeah, before going to that
27:57
vector based search, we can just do all this to 1st confirm that
28:01
this is what the the multimedia is about.
28:04
But you're not enriching the image with metadata of like a
28:10
description of what is in the frame, are you?
28:13
Or I guess depends on how much money you want to spend.
28:17
Yeah. So do you mean like storing all
28:20
the metadata also as vectors somewhere and then trying to
28:23
match that? Yeah, exactly.
28:25
Well, so basically you're grabbing frames from a video,
28:29
and before you embed them, you also run it through something
28:36
that says that describes what's in this frame, like a picture.
28:41
Yeah. I think as you rightly pointed
28:43
out, depends on how much money that you want to use for this.
28:47
I think you can store a lot of things.
28:49
You can store the metadata, yeah, you can store the
28:52
description also for let's say some of the frames and then
28:55
before matching the exact frame you can you can match the
28:58
metadata with this. What have you seen?
29:02
Do you know many use cases? I'm sure there's a ton.
29:05
I'm just drawing a blank right now on search for videos in this
29:12
regard. And I have a friend, Amy, who is
29:15
doing, she's her company's called Cloud Glue, I think.
29:18
And they're doing this where they're helping you search for
29:22
videos and or they're helping you search within your video
29:27
collection. Yeah, I think, yeah, I've not
29:30
seen many great examples personally.
29:33
Yeah, For example, most video based search, yeah, that I've
29:38
seen for example when I type a text, Yeah, I'm not sure even if
29:42
YouTube tries to search that text with let's say some
29:46
description or something which is going inside the video.
29:49
I think that also mostly the search based on the metadata
29:53
what what it has for the video, yeah.
29:56
Yeah, I guess one that I've seen do that well is probably TikTok,
30:02
but other than that, I don't know what the use cases are.
30:04
And I'm sure there are a ton like training videos or stuff
30:09
like that and things that I'm just, you know, I don't
30:13
encounter in my day-to-day. And so I can't think of it but.
30:18
Yeah, I think in reels. I think in reels also.
30:20
For example, when I type a query, I think to some extent
30:24
it's searching for the content inside the video because it
30:27
returns pretty relevant results based on what I search.
30:31
Yeah, yeah. So that I think is doing some
30:34
form of multimedia. Search.
30:37
So the idea of being able to have multimedia search is cool.
30:41
And the other thing that is exciting you right now is the
30:44
idea of just being able to have like almost multi turn search or
30:49
enriching the search. It's not just the first thing
30:52
that you get back, it's going over it, reasoning over the what
30:56
is retrieved maybe, and then going in, conducting another
30:59
search. So you have multi hop search.
31:02
Yeah, right. So identity here is also like
31:05
that will help us to like getting a user reach to their
31:09
goal, Yeah, rather than just returning the most accurate
31:13
result, the most relevant result.
31:15
So this again, yeah, ties to yeah, that that vision of
31:19
search, the cognitive vision of search.
31:21
One thing I've heard folks talk a lot about is the difficulties
31:26
of being able to get the right amount of context to an agent at
31:34
the right time, which as I talked to you, it's like, yeah,
31:40
that's kind of a search issue, right?
31:44
That is making sure that you're not having too much crap that
31:49
you're throwing at the agent that it needs to reason over and
31:53
you're just getting it. It's that precision and recall
31:57
type of trade off that you were talking about.
32:00
Yes, so giving too much context to the user, I think in a sorry
32:05
to the agent here. I think it again, it's yeah, I'm
32:11
not sure how like the any of the generative agents or that can
32:17
help because the context that comes it's it's mostly coming
32:21
from the user, right. Maybe a user is not able to
32:24
provide context to what they're looking for.
32:28
Maybe they are able to like bombard the agent with extra
32:31
information that it really needs.
32:34
Yeah, I. Was thinking more along the
32:36
lines of hey, the agent is going and it's doing these multi hops
32:43
and it's going and it's searching for something, it gets
32:45
something, it retrieves it, boom.
32:47
But if what it retrieves is maybe not relevant or there's
32:52
only a little piece of what it retrieves that's relevant,
32:55
that's really where we're talking about.
32:57
This is a search problem. It's less of this like new
33:01
AILLMS problem. Yeah.
33:05
In that case, I think to like ground an agent based on some
33:11
specific context that you are looking for.
33:13
I think rags can definitely help there because they an agent can
33:19
talk to that another retrieval layer and because retrieval
33:23
layer that we can control. For example, let's say if the
33:26
agent wants to query or only retrieve some of the context
33:31
which is specific to let's say my company.
33:33
So I can set up that retrieval agent in that way so that it can
33:36
first talk to that RAG layer, get some specific context for
33:41
for that whatever query that user is looking for and then
33:43
pass that to LLM or generate an answer based on that context.
33:48
So RAGS can really help to ground agents to some company
33:53
like domain specific data that we have.
33:55
Yeah, there's so much good stuff, and especially when it
33:58
comes to all the different ways that you can like just encounter
34:05
search or perfect search in your systems.
34:11
And search has been around for so long and it's so fascinating
34:16
to me that it's not a solved problem by any reason.
34:21
Right, right. Yeah.
34:24
It I think solve, yeah, I think it will never be solved.
34:28
I think fully because it's, it's just been evolving with every
34:32
new piece of technology that comes in.
34:35
Yeah, I see that, yeah. It just keeps evolving.
34:37
Now user like expect them more from search.
34:40
I think some time back they just wanted that, yeah, just give me
34:44
a list of links or list of documents that that, that which
34:50
is more suitable for my query. But now based on lens, based on
34:54
generative AI, then no user have expect like are expecting more
34:57
and more from search. So now search has to do a lot
35:00
more just to make the users happy.
35:04
Yeah, You know what? I just randomly thought of
35:07
another use case for the video search, and it's when you have
35:14
one of these copilots or like Rewind, Rewind AI, which I
35:19
haven't heard about recently. I don't know what happened to
35:24
them, but they were supposedly watching everything that you're
35:27
doing and taking screenshots every once in a while and
35:31
recording what you're doing and so that you could search over
35:34
your past history much easier. And that would be a great use
35:40
case for the video search, right?
35:43
And it's just like if every 5 seconds they're just taking a
35:47
snapshot of your screen and then you're vectorizing it or they're
35:52
creating an embedding out of it, which I imagine is how they were
35:56
doing it. But something didn't work with
35:59
that because I haven't heard anything about them for they
36:02
blew up a few years ago and then I don't know what they're doing
36:06
these days. Yeah, I don't know if I want to
36:09
search my history like that. Like every action which I'm
36:13
doing, it's recording OK, and then I have to search for that.
36:17
Yeah, never, yeah. Encountered a use case like this
36:21
where I need to search what I was doing or basically what they
36:25
are like what's the the most search, I think use key that
36:29
might do what my system is I have to look for some document
36:33
and maybe like I also want to search for something like what's
36:36
because I remember what's inside that document, but not exactly
36:39
what was its name. So I just want that kind of text
36:43
and also video, video and multimedia search.
36:46
Yeah. But he's.
36:48
Yeah. As like, oh, I had a call with
36:52
Jane yesterday and she said something.
36:55
She said this number. What was that number?
36:58
Or you know, when I was talking to this person, they referenced
37:02
something and I can't remember exactly what they referenced.
37:06
There is a world now where, all right, you probably have a call
37:09
recorder and you can go to your call recorder and check that.
37:14
But if you can have that all in one spot where you just search
37:18
your history, your past, and that might be like, I had to
37:23
call yesterday, but what if I had to call 2 weeks ago?
37:26
I can vaguely remember who was with what they were talking
37:30
about, you know, that type of thing.
37:31
Yeah. Yeah, that sounds interesting.
37:34
Yeah. I never, yeah, thought about you
37:36
having this, this kind of an app in my phone.
37:39
I think we'll phone. It would be more useful, yeah.
37:42
Yeah, the phone 'cause we use it more, right, But even.
37:45
Oh I was on LinkedIn the other day and I saw a great post that
37:50
was about this, but now I can't find that post and so if I'm
37:58
getting screenshots taken every 5 seconds I imagine you could go
38:03
and take it. Yeah, I wonder.
38:06
It will take like insane amount of memory and resources to like
38:10
store all of that that they're doing if they are making a
38:13
recording of a system every 5 seconds or so.
38:17
Or more, who knows? Yeah, the whole thing was that,
38:21
oh, we figured out this special compression technique.
38:24
And so it doesn't do that. But like I said, I haven't heard
38:28
about them in a long time. And so I don't know, it might
38:33
have failed horribly or people might have been like, I don't
38:37
want some random company just spying on me.
38:41
Yeah, right. If it's a recording, then yeah,
38:45
it might be using their data for something else.
38:48
Yeah, you never know. Yeah.
38:50
Yeah, they're training a. Lot of GPU's, maybe just due to
38:54
the cost factor, yeah, it didn't survive.
38:57
That can be one of the reasons. Yeah.
38:59
Also, yeah, who knows? But now it's a little bit of a
39:02
tangent going back to what you're thinking about on a daily
39:07
basis. I'm wondering about like the
39:10
future in your eyes and where you see there's some unresolved
39:16
challenges. Maybe it's with cognitive search
39:19
or even with the dense vectors and vector search.
39:23
How do you think that we can move forward?
39:28
Yeah, I think in just in cognitive search, I think you
39:32
also pointed out, I think some of the things for example in
39:36
vector based search we have lot of better tested things already.
39:40
But in cognitive search we are like still far from it.
39:44
We don't have a good way to measure the metrics which I was
39:47
talking about. For example, let's say user is
39:49
happy or let's say they got in the end what they're looking for
39:55
like their intent of the search was satisfied.
39:58
So that's still I think, yeah, we need a lot of work there.
40:03
Again, in terms of I think dense retrieval also like we have
40:07
GPU's, like we do vector embeddings for everything, but I
40:12
think GPU's, I think getting the cost of these GPU's down again
40:16
is a lot of challenge for big companies.
40:18
Like you can't, like they're not infinite resources.
40:22
You can't run everything on dense retrieval, you can't
40:25
vectorize everything. So like getting the cost down
40:29
for the GPU, like making your whole like AI infrastructure
40:33
pipeline reliable and also cost effective.
40:36
That's, that's also a big challenge.
40:38
So I think this is also what I hope.
40:42
Like it's it. It's looks like that is not a
40:46
big problem, but dealing with cost and accuracies with limited
40:50
resources is often one of the big problems in lot of
40:53
companies. You bring up a point that has
40:57
been stated on this podcast a few times over the years.
41:01
And I wonder how you would go about tackling this problem,
41:07
which is keeping the documents and files and everything in the
41:14
vector database up to date. Because when you create
41:19
embeddings, let's say you create embeddings for a certain
41:22
document and then you throw it in the vector database or you,
41:25
you, you know, throw it wherever you store it.
41:29
And then there's a new version of that file that you need to
41:34
have be referenced as the canonical truth.
41:38
What do you do? You just go in like purge all of
41:40
the past ones. But how do you purge all the
41:42
past ones? Like how do you make sure that
41:45
now when you're doing this dense vector search, you're retrieving
41:49
the correct file that is the most up to date one that you
41:54
want to use? Yeah, I think this problem also
41:59
like persist outside of like vector based indexes.
42:04
For example, even if you have massive let's say text based
42:07
index and you want to like also first you want to update it
42:12
constantly with the live events that you're coming in.
42:15
And also then let's say after a few days your index becomes
42:18
stale, then you have to rebuild the index again.
42:22
So in terms of vector and this is also I think it's the same
42:27
approach. So I think once you generate it,
42:30
you can you can keep it for few days or let's say depends on
42:35
like and keep it for few weeks. And then you can have updated
42:39
gradually with all the live events that are coming in and
42:42
you can keep the snapshots of those live event just in memory.
42:47
But let's say there's a need to also rebuild this index.
42:51
You can't keep on like surviving with that index for long because
42:55
the size of the live index will keep on growing.
42:58
So I think a way to do this is you rebuild your entire snapshot
43:03
of the index, which is stored on disk gradually after like sorry,
43:07
incrementally after every few weeks.
43:10
And once you do that, because so each embedding is generated with
43:14
a specific model model version. So you have to make sure that
43:18
you are using the exact like same model to generate the new
43:22
embedding again. And if you are using a different
43:24
version of the model, then we have to compare those embeddings
43:28
and then we can only make the switch from using the old
43:32
embedding to the new. So in this way we are not
43:34
actually purging the old embeddings.
43:36
So we first generated a new set of embeddings.
43:39
We did an evaluation between them.
43:41
We did a compatibility check whether these embeddings are
43:44
compatible. And then we make the switch.
43:46
And the switch is only in terms of where your queries are going
43:49
to be served. So you change date that layer,
43:51
sorry. And then you say that now just
43:54
all the queries should use these embeddings to generate the
43:57
responses and you just keep doing it after every few weeks
44:01
or so. You remind me of how my buddy
44:03
told me a story about how he had the exact same system and all he
44:09
changed was the embedding model. Everything else was the same and
44:15
he was like, it's not going to be that big of a difference,
44:17
right? It was completely different.
44:21
He was like, you cannot use those at all.
44:25
Yeah, embeddings are like like very it those are kind of like
44:28
version artifacts like we have in software systems.
44:31
So you even if you change let's say param or a weight in your
44:36
model, you like your embeddings are just going to go haywire
44:40
like you're not going to believe what it does to embeddings.
44:43
Yeah. So that is like very important
44:45
to make keep that that that compatibility between the model
44:49
and embeddings. Yeah, the the other piece that I
44:52
think is is fascinating when it comes to these, the embeddings
44:58
and the vector database of the vector stores that are kind of
45:03
difficult that I heard people talking about back like 2 years
45:06
ago. I haven't heard as much of it
45:09
these days because maybe folks moved on to new problems or they
45:14
just they swept it under the rug and pretend like this doesn't
45:18
exist anymore. Is like the access control with
45:21
embeddings and vector databases. Yeah.
45:27
I think the access control, I think it, it is similar to like
45:32
the access control that we have for any version artifacts
45:36
because we I think there is like usually a lot of like layers
45:41
that goes in that access control.
45:44
So I think for embeddings also I think the yeah, I think in my
45:47
experience the the access control that we have for any
45:50
version system like for a data set for example, that is the
45:54
same as embedding. So we consider like an embedding
45:57
same as a data set which will just keep on having like more
46:01
versions incrementally. Yeah, yeah.
46:04
Going back to what you were saying about how to update these
46:08
files, whether it's embeddings or files or just, you know, data
46:14
in general, I noticed that for certain things you can
46:21
overweight. The fact that there's, there's
46:26
like certain features that you can really overweight and then
46:29
it leads to a worse search experience.
46:32
And I'll give you a prime example of this.
46:34
When I was using Notion, I saw that I was looking for a file.
46:40
I couldn't quite remember the name.
46:42
I remembered some stuff that was in it and I spent so long trying
46:48
to query this file or trying to like just find it in my gigantic
46:52
Notion sprawl of a oh fuck it's a mess dude, but that's a whole
46:57
nother story. And Notion was over indexing on
47:06
certain features, like oh this file was checked recently, or
47:13
this file has been looked at by 5 people on your team.
47:18
You know, like there's certain features that make it much
47:22
harder for me to find an obscure file.
47:26
Yeah. So I think like depends on your,
47:30
their indexing strategy here. Maybe what I think the features
47:34
that with that that they were using for indexing was not that
47:38
great. But I think there if you're
47:40
looking for the contents of the file, I think that's the like
47:44
one of the first things which I'll like put my bets on.
47:47
Like that's the first thing which I'll index properly.
47:50
So that people they usually, because they either remember the
47:53
name of the document or the they remember what, what was the
47:56
document about South and they the search by like either of
48:00
these two things. So at least it should return the
48:05
like documents properly based on these indexes at least, yeah.
48:10
And this if I throw into a search bar or something like,
48:16
you know, a file that talks about XYZ and I kind of describe
48:22
what's in the document that is not being triggered for dense
48:29
vectors. Is it like I, I feel like I've
48:32
heard of some search techniques where first you'll pass it
48:36
through an LLM to try and gather the gist of it or create a query
48:41
that will go and find it. It Did I just make that up or is
48:45
that something that you've? Seen.
48:46
I think that's right. I think in case of documents, I
48:50
think vector search is a good idea.
48:52
Like you have generated embedding software documents,
48:55
let's say which whichever you are storing in an offline way
48:58
and then you store it somewhere. Now based on each query then you
49:01
just compare that embedding with the query embedding that you
49:04
have. So yeah, I imagine like because
49:08
searching the content and that's a good way to search the
49:11
content, yeah, for document. And now the, my friend Nishi was
49:18
talking to me about how this is a really hard problem he faced
49:21
at his company for search. And it was along the lines of he
49:27
wants to be able to return what shops are open when someone
49:37
searches for pizza on the app, You know, and so there's a lot
49:42
of restaurants that offer pizza, but if someone is searching on
49:47
his food delivery app for pizza, they don't want a restaurant
49:51
that's closed to be the first thing that shows up, even if it
49:55
is the highest rated. How do you think through that?
50:00
Like what are you? Because well, I guess the other
50:03
caveat to this was he was like, it's not like just open closed
50:07
times are this magic number that we can reference or a row in the
50:12
database. That was another thing that was
50:13
a little bit hard or even you can't like filter by that for
50:18
some reason that I can't remember.
50:21
Yeah, I think this, yeah. To me, I think it's not a search
50:26
problem. I think it can definitely be
50:28
solved using search like you can, you can vectorize
50:33
everything even based on that you can see that which
50:36
restaurant is open. But I think this is yeah, more
50:40
of a distributed system problem. Like if you can't store the
50:46
yeah, I'm, I'm just thinking out loud.
50:47
If you let's say if you have store the timings of of that
50:51
restaurant, let's say if you were not able to do that in a
50:55
relational database, then in a no SQL database and for each of
51:00
those restaurants you check whether at that time it's open
51:04
or not. I think there are few smart
51:07
things we can do there. I think let's say there are few
51:10
times where people like search that that more often then we can
51:16
create some indexes based on those time intervals that in
51:20
this time interval these restaurants are open.
51:22
So it depends on the query pattern that the app is getting.
51:26
So for example, if I get a query at 5:00 PM, then I can see at
51:30
5:00 PM which are the restaurant that are going to open.
51:32
So I can pre calculate all this and I can store it in my
51:36
database already so that I can just return it at that time.
51:40
So there are some like smart strategy that I that I can do
51:43
here. But yeah, I think this I think
51:47
I, I won't want to make it a like a search problem for me.
51:51
Yeah. I like you humouring me on this
51:55
one. The other one that he said was
51:56
hard is that people will search for vegetarian options and Pizza
52:07
Margherita is a vegetarian option but that's not going to
52:12
come up normally because it's not tagged as vegetarian.
52:16
It's just tagged as pizza and even though it is vegetarian,
52:21
it's like in vector space. Vegetarian food is very far from
52:27
pizza Margherita. Yeah, right.
52:32
I think that's why we should have a layer of like, yeah, I'm
52:37
I was going to say text based search, but in the in this case,
52:40
yeah, there is no vegetarian options specified anywhere.
52:43
So you can do a text based search also.
52:46
Yeah, again, in this case, yeah, I'll just pre calculate it.
52:49
If I see that people usually look for vegetarian or non
52:53
vegetarian options, I'll just mark my like options in the menu
52:57
as vegetarian or non vegetarian. And then based on that, yeah,
53:02
I'll just, so if they, let's say they serve pizza vegetarian,
53:05
yeah, yeah, based on that, I can just return because from
53:09
vegetarian I got these X number of things from my menu and from
53:13
that now I can filter which one is pizza out of that.
53:18
So yeah, it's almost like, and I think he ended up doing this,
53:22
but basically taking all of the menu items and then running them
53:31
through an LLM and asking the LLM to tag them with certain
53:36
qualities, whether it's vegetarian or gluten free or
53:40
Asian, whatever type of the type of food the.
53:45
And then you can use that when you are, yeah, throwing it into
53:51
a vector database. You just have to make sure that
53:54
what you're tagging is relevant. Yeah, right.
53:59
Yeah. I think tagging we can figure
54:00
out based on also like what does people search for?
54:04
Like what are they looking for? Most vegetarian is one of the
54:06
things that they look for. Yeah.
54:09
So based on search, we can keep like evolving the tags, I think
54:13
on a food menu. Yeah, I'm, I'm sure there are
54:15
limited number of things which our user is searching for.
54:18
Yeah. So yeah, that should be like
54:22
easy to estimate after like gathering some data from the
54:26
users. Yeah, it's, it's looking at the
54:29
data you have and then trying to bucket that or just asking LLM,
54:33
hey, bucket these 10,000 queries that we've had from our users.
54:38
Yeah. Yeah, because he, it's, that's
54:42
a, that's a great one. Any other hard search problems
54:45
you've run into and how did you solve them?
54:50
Yeah, I think only, yeah, I can yeah, only recollect about,
54:54
Yeah, multi model IS1 intake is another, I think mostly, yeah,
55:00
it's mostly I think what I have run into for searches not only
55:06
like related to relevance, but also about like search
55:09
infrastructure in general. So making the system like more
55:14
reliable and like always dependable.
55:17
So that is our like goal always. For example, another thing is
55:23
also like make sure that the tail end latency is not that
55:27
high. For example, you're all the, for
55:31
example, a few systems are lagging and returning a result.
55:34
We have to make sure that the user is not seeing that latency.
55:37
So most of I think these problems are also like
55:41
distributed system problems and not only in terms of, yeah,
55:45
what's the best result that we have to return to the users.
55:48
Say more about that. Yeah.
55:51
So in terms of distributed system problems, I think I
55:54
already mentioned that let's say you are using vector based
55:57
search. I think then dealing with GPU's
56:02
and keeping their cost down is one of the big factors that we
56:06
like have to take into account. Also when I think in when you
56:11
have indexes of like let's say billion embeddings or you're
56:15
also in term in cases of text based search, you have very
56:18
large indexes, then a lot of like distributed problems come
56:23
into play. For example, you have to keep
56:27
deciding like one example is how many like how many charts or how
56:32
many partitions you want to divide your data in and how do
56:36
you accurately define those many partitions.
56:39
Like how do you make sure that when you decide on X number of
56:43
partitions then your queries are not running into issues for
56:46
example out of memory and you're you are able to load all the
56:51
memory index is fine. You are able to serve the
56:53
queries from the indexes on the hard disk storage find.
56:58
So you have to make sure like you have to like run experiments
57:01
for all of that and then come into some conclusion because
57:05
there's no magic number which can tell you that for let's say
57:08
your index size is X terabytes. Then you can just go run with
57:12
these many shards, these many partitions, because you also
57:15
have to take into account which is my like, what are the query
57:19
pattern that I'm going to get? For example, do you want to each
57:23
machine to be big? Like you have to scale your
57:25
system vertically rather than just scaling it horizontally
57:28
where you just keep, let's say X number of replicas for each
57:31
partition Because then so like analyzing your read and write
57:36
queries, for example, what kind of insert is usually read?
57:40
Like there's no write which a user is doing.
57:43
So you have to analyze the read queries and then design these
57:46
systems like accordingly. Yeah.
57:49
Then again, I, I think I mentioned about live update,
57:52
that's a big challenge. How do you make sure that
57:53
you're, for example, let's say you have some X items in the
57:58
menu. Now let's say someone goes and
58:01
goes make a change in the menu. How do you make sure that users
58:04
are able to search for that updated thing, whatever it was
58:08
updated? Let's say you added a new tag or
58:10
you did something. So you have to keep the
58:12
refreshing your indexes periodically like whichever is
58:16
there in memory or on the disk so that users are able to like
58:22
users can see the latest as always.
58:24
Yeah. So mostly it's, yeah, mostly
58:27
it's distributed system challenges.
58:30
Yeah, that live index or the live updates reminds me of a
58:36
conversation that I had on here with Rohit.
58:40
And he was talking about how towards like, yeah, we need the
58:43
fastest. We need like our data freshness
58:46
to be, you know, in the milliseconds.
58:49
And he was like, you don't know how much money I've saved people
58:53
by just going from milliseconds to seconds.
58:58
And there is hardly any noticeable change that the user
59:04
feels. But on the back end, you're
59:08
saving so much money. Yeah, yeah.
59:10
That's, I completely agree with this, I think.
59:13
People, yeah, when they think of a search system, yeah, they
59:15
always think that I want my results to be fasted, I want it
59:19
to be 100% accurate. But that's not the I think
59:23
that's not always what a user is looking for.
59:26
I think you rightly mentioned that going from few milliseconds
59:30
to even a second to some users, like it won't make a difference.
59:34
Like there are some systems like Google or say you can't return
59:38
the searches in seconds. So it depends on your users also
59:41
depending on what, what kind of queries that you are serving.
59:44
So just trading like having a trade off between, between those
59:50
latency and accuracy, I think that's that's completely
59:53
acceptable. You have to decide.
59:55
I think cost is definitely a big factor.
59:57
So if latency is fine, I think then it's it's fine to do that
1:00:01
trade off. Yeah.
1:00:03
And in this specific case. I also want to be clear that
1:00:07
Rohit was talking about like the date of freshness.
1:00:09
And so that's why I was thinking about what you were saying with
1:00:12
the live updates. OK, the live updates and that
1:00:16
getting incorporated back into the search system.
1:00:19
You can try and have these live updates be constantly
1:00:24
incorporated back into the search system, like every
1:00:29
millisecond. Or you could say, you know what,
1:00:31
every 5 minutes is all right, and save a whole lot of money
1:00:36
because you're not having to perform those operations
1:00:39
continuously and, and that you can save on compute.
1:00:44
Yeah, I think most users can live with.
1:00:47
With it, if we don't provide like all the live updates within
1:00:52
few milliseconds, I think it's fine.
1:00:53
They can wait if it even if it's getting reflected after 5 or 10
1:00:58
minutes, so they don't have an issue with that.
1:01:00
Yeah, which feels like an eternity, but.
1:01:03
Still like 5 or 10 minutes. I I mean, you probably don't
1:01:06
want to do that. It's probably can be a bad user
1:01:09
experience depending on what exactly is being reflected.
1:01:14
But I mean, you definitely like a few seconds is not that big of
1:01:21
a deal. And then like dovetailing off of
1:01:24
that and going into what you were saying with us
1:01:27
understanding now that users, there's this pattern that's
1:01:32
happening at least within myself when it comes to search, I'm
1:01:36
more OK with search taking a little bit longer because I've
1:01:41
been interacting with LLMS more. And so if it takes a few seconds
1:01:47
rather than it's instantaneously milliseconds that pop up, it's
1:01:53
not the end of the world for me. I would rather it be as fast as
1:01:57
possible, but I'm not going to like, I'm not going to have a
1:02:01
worse experience that is faster. I'm OK with waiting to have a
1:02:08
better experience. Yeah, I think this is what we
1:02:11
like see with. Using all the I think generative
1:02:15
agents because there is some system which is giving us a link
1:02:20
of all the documents instantaneously, but we are
1:02:23
still fine to go and wait for the LLM to generate some answer
1:02:27
and justice reason that this answer makes sense and then just
1:02:30
give me what I'm looking for. I think.
1:02:32
So in that case, even latency is acceptable.
1:02:35
So it's fine to have that trade off there.