Latest / Agentic Conversations (formally mlops.community) / Building an Agentic AI Memory Framework

Transcript

Building an Agentic AI Memory Framework · Agentic Conversations (formally mlops.community)

0:00
So imagine memory as a tool, something which is still
0:03
debatable. If the AI agents actually need a
0:07
mechanism to forget. Imagine it's not us who are
0:10
controlling these tweaking parameters, but it's an agent.
0:12
But the vision memory should be able to communicate properly
0:15
with the textual memory that it has access to from day-to-day.
0:21
You dove into the memory world for the past 5-6 months because
0:27
you built out Cortex. Can you give me a lay of the
0:30
land at the high level? So when I was starting, I think
0:34
at that time there was not anything very specific.
0:38
There was like mem GPT and there was Lang mem, which level is
0:44
like quite popular at that time. What piqued my interest was AMM
0:49
kind of came out at that time. It's like around I think 5
0:52
months ago, 5 or 6 months ago. So also like envelopes community
0:57
had a very nice reading group discussion on AMM and we just
1:01
like discussed the whole paper there, which was very helpful to
1:04
understand the things even better.
1:06
One of the key ideas I saw in AMM was let's say you're
1:12
inserting something into the memory or like database or
1:16
whatever sort of a collection you're saying which the agentic
1:19
system will have access to. So the key differentiating
1:22
factor was it sort of forms relationships.
1:26
I'm having this talk with you right now and currently we going
1:30
to have this talk like I knew that in my mind.
1:32
So let's say a few days back. So currently I'm connecting this
1:37
current state with my past memory because it's sort of
1:42
interconnected. And if you see there's a
1:45
relationship between these two memories, it's like we can also
1:49
like name this relationship something like is a result of
1:53
that. So current state is a result of
1:55
the previous state that we were discussing about previously.
1:59
So something similar like AM does is it takes different,
2:05
different memories that you want to store in inside the whole
2:09
memory collection or whatever you say, and it sort of forms
2:12
relationships between them. So if I dive a little bit
2:16
deeper, let's say you are inserting something in the
2:20
memory, you take that and if it's like the first thing you're
2:24
inserting, you just insert it directly.
2:26
But before inserting, you just analyze it a bit like sort of
2:30
come out with a summary of like what it's about South, we are
2:32
calling it like context. Then we also come out with like
2:36
keywords and tags. So like, what are the main key
2:39
things in this now, by the way, this small memory chain can be
2:43
as big as possible or as small as possible.
2:46
That's on the client side that we're giving like.
2:49
And that's like on how you actually use this memory system.
2:52
That's what Amy was trying to do and.
2:55
So yeah, in a way it's it's like a knowledge graph.
2:58
You've got this chunk, and then you have metadata around it, and
3:02
that metadata connects to other pieces of metadata.
3:05
Yep, Yep. So the metadata doesn't directly
3:08
connect, but you have these sort of metadata and they're just.
3:12
So currently, let's say you have like few memories already in
3:15
place. Then a new memory comes in, it
3:18
sort of does a semantic search between all the memories that
3:21
are in the collection and let's say it takes out like top K of
3:24
the memory. Now once it has like the top K
3:27
memories, it asks LLM with their memory representation.
3:32
So we don't use the whole memory.
3:33
We use the summary and the key words or like tags and whatever
3:39
other material we store. It can even have the timestamps.
3:42
And then we ask the LLM with the new memory that's coming in
3:47
that, hey, do you see any sort of relationship between these
3:51
existing ones, which we found to be similar to the current coming
3:55
memory? Now, if you do, then just
3:58
connect them. So what I mean by connect them
4:00
is update the existing metadata and also give a nicer metadata
4:07
to the current coming memory. And that's how the whole graph
4:11
keeps on forming. So it just sort of makes a
4:14
connections in between these two.
4:16
And at the time of using using of the memory.
4:19
So like let's say a query comes in now we have to look for like
4:24
the most relevant ones from the memory.
4:27
So what we do is we just analyze this query with an LLM.
4:29
So the one differentiating thing I think in MMM is on every
4:35
insertion and before like every retrieval we do an LLM call
4:41
because we are trying to pre process and MMM doesn't really
4:45
focus highly on the latency side but more on the quality of the
4:50
memory that you are getting out. I was going to say it sounds
4:54
very akin to a search problem. Yeah, yeah, for sure.
4:58
So currently we just take out the metadata thingies and now we
5:04
just literally we do a semantic search on using two things.
5:08
One will be the word the query was and the other thing will be
5:12
the keywords from the meta, some from the query that we
5:15
extracted. So we do 2 retrievers parallely
5:19
at the same time and then we take those 2 results and just
5:23
feed it into the context and do whatever we want with it, like
5:27
pass it to another to answer something or.
5:29
That's very much on the personalization side, right?
5:33
Yes. I mean you can literally do
5:36
anything with that. And this is like the most basic
5:39
core part of AM where it didn't really perform that well
5:45
compared to the existing methods that was there.
5:48
But what, what's striking to me was the relationships forming in
5:53
between OK and I was kind of trying to think very deeply on
5:58
like how humans think about memory and how we can actually
6:02
improve it using this relationship foundation.
6:05
Because I the only two place I saw this relationships in place
6:11
was in graph rag. So but I, I kindly feel that in
6:15
graph rec, we have all of these like different nodes and we
6:19
compress down information way too much.
6:22
So that's that's one of the issues.
6:23
And that was not there in the AM thingy.
6:26
Then you got inspiration from Amen.
6:29
You thought deeply about what and how memory works within
6:33
humans. And then you said, let's take it
6:37
another step and let's create our own memory.
6:40
And yeah, yeah. I was, I was like brainstorming
6:44
with a few of my friends and like whoever was like in this
6:49
space and was reading papers basically and saw that.
6:55
So currently I was trying to, I think I use like deep research
6:58
on like to understand basically how human also thinks
7:02
specifically from the memory perspective, like on a very high
7:06
level and also a bit low level. So they have this thing called
7:09
like long term memory and short term memory.
7:12
So I mean, we have recently seen in the AI space or the LM space,
7:18
the short term memory. It's, it's nothing.
7:20
It's just the recent window of the chat that you're having with
7:25
the AI agent or the conversation assistant or something.
7:29
That's one form of short term memory.
7:31
But I wouldn't say that's a short term, short term memory
7:33
because it has to be a bit bigger than that.
7:36
So but about long term memory is this is something crucial?
7:40
I think that's where all the magic happens in the long term
7:43
memory. So the relationship forming and
7:47
like everything storing nicely and this whole big graph that we
7:51
are storing in inside that is part of the long term memory.
7:56
So, and we don't expect to long term memory happen like as soon
8:01
as possible. So I mean, it can always keep
8:04
running on the background. It's, it's something we don't
8:06
expect to happen instantly, but it will, it will take its time.
8:10
But as you, as we are having this conversation, we are
8:12
building onto things constantly in the back of our minds.
8:16
And it's not like, I mean, I'm using certain parts of it, but
8:18
that's from a short term memory. So short term memory is sort of
8:21
like a window, it's constantly updating, updating everything.
8:24
But in long term memory we are just storing everything like
8:27
kind of dumping. And while we are dumping things
8:31
are processing parallelly in the background which is extracting
8:34
lots of merit out of it and interconnecting things in
8:37
between. Yeah, I remember taking a online
8:40
course of while back now on and it was called Learning how to
8:44
Learn. And it talked a lot about this,
8:47
how we can create memories because that helps us learn.
8:52
And some of the things that it mentioned was the more that you
8:55
access from the long term memory it the more solidified it
9:01
becomes as something that you now know.
9:04
And so the frequency. And that's why you do like flash
9:07
cards when you're trying to learn something because you're
9:10
trying to remember it and you can or you can, maybe you can
9:13
and then you flip it over and then you remember it.
9:15
OK, cool. Yeah, there it is.
9:16
And so it accesses it again, and you basically do that until you
9:22
now know it and then you can spread out the frequency.
9:28
So if you're trying to learn something and you're struggling
9:31
with it, maybe you're doing these flash cards every day or
9:34
twice a day, three times a day. But as you then don't need to
9:38
flip over the card to remember something, you can space it out
9:44
and do it once every three days, once every five days, once every
9:46
week. And so it feels like what you're
9:50
talking about it has a little bit of that inspiration.
9:53
But the other piece that I think is interesting is that you're
9:56
you're also mentioning we've got this long term memory and we're
10:00
able to throw things in it. How do we weight it and how do
10:05
we know that things are important from it?
10:08
Because for humans, we have something that is like, oh, the
10:12
more frequently that we access it, the more important or the
10:16
the more solidified it becomes in our memory.
10:18
So I have used these like Anki flash cards.
10:21
I think it's quite popular like everyone who's trying to like
10:24
learning something. I I remember like trying to
10:27
learn Japanese. So I was just like learning
10:31
hiragana characters through like Anki flash cards.
10:34
And I mean the concept of space to repetition.
10:37
I think that's that's the term. So you have people say, so
10:40
that's quite popular. And this is something very much
10:45
seen in humans specifically because we tend to forget, we
10:51
tend to forget things a lot. And this is something which is
10:55
still debatable if the AI agents actually need a mechanism to
11:00
forget or not. So there are a few people who
11:03
actually believes that. Yeah, yeah, systems need to
11:05
forget because information gets outdated.
11:08
We have to prune the things out. Yeah.
11:10
And there are like lots of strategies to remove things out.
11:13
And also some people believe that.
11:15
No, it's an AI agent. It doesn't have to exactly act
11:17
like a human, like just dump everything and just use the most
11:23
important ones and archive maybe in a some way.
11:25
Yeah, well also the other piece I remember that you were
11:28
mentioning too was how humans sleep.
11:31
And sleep is a huge factor in how we are able to bring things
11:35
into our long term memory, right?
11:37
Exactly, exactly. So yeah, I mean there are there
11:40
are papers like where they say that human keeps on processing
11:44
when they're like sleeping. It's the thing like the term is
11:49
kind of consolidation. So like human memory
11:51
consolidation happens when you kind of are resting or like when
11:55
you stress levels are like quite low.
11:57
So and your brain is like not I guess multi functioning and
12:02
doing a lot of things together. But yeah, so yeah, it just
12:06
doesn't need to sleep. It can do so many things in the
12:09
background. But when you're thinking about
12:11
how to consolidate different memories, how did you go about
12:15
that? Because, as you were just
12:17
saying, you're inserting memories in with their certain
12:21
metadata and then you're constantly updating this
12:25
knowledge graph. So Cortex is something which I
12:28
built on top of AMM at first. So I took all the inspiration
12:33
from AMM on how it was happening and I thought, hey, we can
12:37
actually improve this thing. And there's so many
12:39
possibilities in here. So what I did, the first thing
12:43
was, oh, before I mentioned this, I have to also mention
12:46
this, that have you ever used Obsidian?
12:50
Yeah, yeah, yeah, of course. So they used one method called I
12:54
think, I think I'm pronouncing it wrong, but it's like Zetel
12:57
Kaisten, something like that. So what it is, is you can
13:02
reference things in any page from anywhere.
13:06
So at the end, Obsidian gives you this whole interconnected
13:11
graph where you can check that check like different clusters on
13:15
what all things you have mentioned and which page
13:17
connects to which page. So your whole writing journey is
13:22
kind of captured in the graph and what all are the
13:24
interconnected webs in between that comes out for you to see so
13:28
we can get the whole bigger picture.
13:30
So the MMM paper like mentions a lot about cycle Kaisten and
13:35
sorry if I'm pronouncing that wrong but.
13:38
Whatever. Yeah, so even just forms like
13:41
relationships in between them. So what I thought is currently
13:46
we value things differently all the time.
13:48
So our values and everything that depends constantly on lot
13:54
of factors. It might be our own biases and
13:56
what our previous memory was. What have we seen our life, what
14:00
we have like gone through everyday I think in our life,
14:04
which is totally different for every single person.
14:07
And by person in this case I mean a memory.
14:11
OK, So yeah, currently when we are like what the key change I
14:16
think I made was I sort of named the relationships that each
14:22
memory is being connected by. So what I mean by that is I had
14:27
some key terms. So initially I was just like
14:29
experimenting with what if we just name it extends by or
14:34
something like definition of or something.
14:39
So I had some hard coded key terms in here and I tried to
14:44
extend it, extend it more. So that was there.
14:47
And also like there were key terms like reciprocal of
14:49
something like defines. So a memory defines this memory
14:56
and now, but we don't really know that how much it defines.
14:59
Is it like, so if we talk about it mathematically, zero being
15:05
the, there's like no definition between each other and one being
15:10
like it fully is the definition of another.
15:13
So one being the definition of other so it can be scored, so we
15:17
can actually score that. What is the relationship
15:20
strength between these two? Nice.
15:22
So, and this is something we don't really do by like we don't
15:27
do like manually or like any sort of instantaneous thing in
15:31
the pipeline. So while we are forming these
15:34
metadata connections and relationships between these
15:37
memories through an LLM, when we're asking the LLM process, we
15:41
ask it to also give a score and only score it if it's like very
15:46
confident that there is some sort of like highly weighted
15:51
relationship is present. So that kinds of gives you the
15:55
whole overweight, I mean whole weight of the whole thing.
15:59
And the other thing is also we ask it to like name the
16:03
relationship out of these possible naming schemes.
16:07
So that's just a big set that we have.
16:10
And it just, yeah, I learn basically chooses that how it is
16:16
connected to each other. So does it extend or is it a
16:19
definition? And there's one more other
16:22
option that is, yeah, should you be merging 2 memories?
16:26
So sometimes in our memory, we actually tend to even merge
16:31
things into each other. So, and I, I kind of feel like
16:35
that it's, it's very important. So that's some feature I also
16:39
added there and. Yeah, sometimes we can falsely
16:45
remember things because I think a lot of times we will merge 2
16:51
memories and then we'll remember.
16:53
Oh, that was maybe if I've been to a place twice, but it was a
16:58
long time ago. I mixed up which time I did
17:02
something. This is all experimental, this
17:05
is I I just wanted to see like what happens at the end.
17:08
So there were like few, so few types of merge that we had
17:13
something like an update which you just said, So an update,
17:16
what happens? We just concatenate different
17:18
memories into one and it just becomes a new memory.
17:22
Yeah. Yeah, by the way, we are
17:24
maintaining all the history in the metadata, so.
17:26
If you need to re separate them, Yep at a certain time.
17:30
If we can do that. And did you take any different
17:32
approaches from AMM on the LLM calls and the search and
17:39
retrieval style? Yes, in in from AMMI mean they
17:43
just had a simple prompt where at the time of inserting a
17:49
memory, they used to process it first and then insert.
17:52
And what I mean by insertion is like it forms relationships
17:55
between the top most similar memories that it could find in
17:59
the database. And at the retrieval, it just
18:03
used to take the query and then just take the query and do a
18:07
semantic search on the top most. And it takes the top most K
18:12
memories and also goes one level deeper because now we can see
18:16
for all of those top K memories what are the connected other
18:20
connected memories. And you just take them also into
18:23
account. Obviously they will be like less
18:25
similar to the original query that you have.
18:29
Yeah, I think these two are like the key factors.
18:32
But something which we added in Cortex was we added
18:37
bidirectional connections between these two.
18:40
So there was only a single connection between the memories
18:45
I as far as I remember. So I changed that to like
18:49
bidirectional connections because it doesn't make sense if
18:53
one is going back to only one side, because for every
18:58
connection that is represented in the graph, it shouldn't be
19:02
only that it defines this. I mean we can even have a
19:06
backward connection saying it is defined by so that at the time
19:12
of retrieval if this one comes up.
19:14
So this also has some thing to give like more context to the
19:19
model so that at the time of retrieval, we can use this and
19:23
this together. This is what we call depth in
19:26
graph flags. So we can actually customize
19:29
like how deeper to go, how many connections deep to go.
19:32
So generally in practice, like it's, it's nice to use just a
19:35
depth of one or two because three gets like way too much
19:39
sometimes and too much noise might come out, come out.
19:42
Were you vectorizing this also? Yes, it's a, it's a whole
19:48
vectorized data set only. So I mean we just use a single
19:51
vector DB. There is no graph.
19:54
Sorry, there's no graph DB that we are managing for this
19:57
specific use case because at the time of retrieval we are just
20:01
doing semantic search. So there's that.
20:04
And by the way, what I'm talking about right now and that's that
20:07
was like the intermediate part of cortex, like when it was like
20:10
just coming out and I was experimenting there.
20:12
Oh, there's AV 2. Ah yeah, there will be
20:15
improvements on top. Of it, Yeah.
20:16
All right, Let's talk about that a little bit.
20:18
Sure, sure. So currently there are people
20:21
are like coming out with these AI assistants or agents where
20:27
they have to do multi domain tasks.
20:30
So the domain is not really connected to each other.
20:34
It can be connected somehow. It might not be connected.
20:37
So let's say you have ChatGPT right now, right or like any
20:41
sort of AI assistant which you use to ask for a lot of
20:44
different things. Now it can be about your work,
20:47
it can be about your personal life, or it can be somewhere in
20:51
between because your friend might be working as a Co worker
20:57
for you. There has to be some sort of
20:59
distinctions or like collections I would say that we currently
21:05
automatically have as a human in our mind and we know how to
21:09
separate them and also like sort of form connections between
21:11
them. So I kind of visualize this and
21:15
I like did some research on the same that this is sort of like
21:19
hierarchical collections. That's that's what we are like
21:23
going towards right now. So the existing systems,
21:27
whatever we are like seeing in the AI agent space for memory,
21:32
we're only focusing on the flat memory structure.
21:36
That's what they're calling. So what happens in the flat
21:40
memory structure is there is no sort of like hierarchy.
21:44
Or so when you say flat hierarchical structure, are you
21:48
talking about something that is less like Notion where we have
21:56
all of our different files or any file structure where you've
22:01
got your file, you click into another file, you click into
22:03
another file, and it just keeps going down the line.
22:07
That's not what you have right now and that's where you think
22:09
it's going? Or is it?
22:11
Because right now, as you mentioned, it's more like
22:13
Obsidian, where everything is just on this flat connected
22:17
space and you have clusters over here and clusters over there.
22:20
But there's no hierarchical way of doing it.
22:23
Exactly. So yeah, exactly.
22:26
You explained it very nicely. So just like how a file system
22:30
works. So in file system there is
22:33
always this like hierarchical structure.
22:35
So first of all, like if you're searching for something in your
22:38
laptop or desktop, you're looking for, let's say a folder
22:42
which has a very high level overview of like what you're
22:45
looking for. Then you go inside of it and
22:47
then you look for maybe a very specific things.
22:51
So sort of this is like the concept of categories and like
22:55
subcategories. So imagine in high level you
22:58
have like lot of different topics in your mind, like work,
23:02
it can be a personal life. Now, all of these different
23:05
topics have like subtopics internally inside work.
23:09
It can be your current company that you're working at.
23:13
Maybe you have a side project that you're also working at and
23:16
it's making money for you. And money also maybe is
23:19
connected with like finance. It's like a bigger topic in
23:23
general in your life and finance even connects with money, but it
23:28
has also a sort of triangle for each of these like top to bottom
23:33
structures. We can see sort of like a
23:35
triangle forming, which has like multiple things connected to
23:37
each other. And it's like a tree, basically
23:40
just a tree. This is something that humans I
23:43
think use like deeply because when we are thinking of
23:47
something we sort of think, we we think like very fast, but we
23:53
sort of think through hierarchies first and kind of
23:57
connect things together. So we go down from top to
24:01
bottom. Also we are taking into account
24:04
a short term memory which which creates some sort of biases on
24:07
like how we are kind of traversing the whole thing.
24:10
But this is something which is missing on the current systems.
24:14
It doesn't really represent these hierarchical collections
24:17
or the topics directly. So this is something we should
24:22
represent also and take into account and sort of build like a
24:27
hybrid search, which uses both of the both of the previous way
24:31
we were doing like retrievals and storing.
24:34
And also it makes use of the whole hybrid thingy what I just
24:38
told about the collections. So in Cortex, we like we're
24:41
saying it, it's smart auto collections, so.
24:45
But what does that enable I? Is it just that it's faster
24:49
search? No, it's it's not faster search.
24:52
What it enables a higher quality search.
24:54
So it gives you much lesser noise compared to what I mean
25:00
flat search will give. Just to give an example, let's
25:04
say you search for something like fix this.
25:08
Now fix this can mean literally anything.
25:10
It can mean like fixing your car, or it can mean yeah.
25:14
What is this? Exactly so fix this Like what is
25:18
even this? So this, it means so many things
25:21
in your life. And if you use a flat hierarchy
25:24
model, it can just fetch out the, fix your car or like, I
25:28
don't know, fix your friend's brain or something like.
25:31
That anything with fix in the metadata it's going to grab.
25:34
Exactly. We are kind of providing like
25:37
the whole, we are providing an option to give context also.
25:40
This actually needs. This is ultimately A context
25:43
problem. Yeah.
25:44
So when we are kind of doing retrieval, there's an optional
25:49
context parameter. But overall, it should first of
25:54
all go through the what are the top level collections are?
25:58
What are the top level keywords are?
26:01
I will talk about it in terms of two two ways. 1 is like let's
26:05
say you're inserting some sort of data.
26:07
So memories, These memory systems in general I think kind
26:11
of has like 2 key functions. One is insertion and another is
26:17
retrieval. So imagine you just have like
26:20
something which you're constantly inserting or it's
26:22
like happening in the background and at the background you can
26:25
constantly keep on retrieving and use that however you want in
26:28
your agentic systems. Keeping these two things in
26:30
mind, if we talk about insertion for let's say this auto
26:34
collections thing which we have in cortex.
26:37
So when a new memory comes in, let's say you have some existing
26:41
memory already in place and when a new memory comes in, it kind
26:45
of categorizes based on what the user person and what are your
26:49
priorities are. This we can actually write down
26:52
initially. So based on that, it kind of
26:54
categorizes that if that memory represents just on high level
26:59
like work, or if it represents work, then kind of subcategory
27:04
can be first job. And then some more there can be
27:10
some more subcategory like Python.
27:12
Maybe it's something related to Python that you will have as
27:17
trying to like store in your memory.
27:19
Now sort of in this way, we like every single memory when it like
27:23
comes in, it sort of creates like all of these topics and
27:28
then it's sub categories only if it means something.
27:32
I mean it's optional. It can just be a very high level
27:35
thing. You also that I hate my job.
27:39
Yeah. So that's like just something
27:41
which is like connected to your work.
27:43
Yeah. And it doesn't really have any
27:46
other smaller subcategories to it.
27:49
It might by the way. So something it can have like
27:51
let's say emotional aspect of it.
27:53
So jobs and emotions, now we form all of these like small,
27:58
small categories. And after it reaches a certain
28:01
threshold, we check like what are the frequencies of all of
28:04
these different memories and what are their categories are?
28:10
So if it exceeds a certain threshold, we form a collection
28:14
out of it. So now this is the collection
28:17
and anything whenever like we, we by the way, I didn't mention
28:21
that how we are kind of forming these categories.
28:24
So at the time of getting the metadata out of a query, if you
28:29
remember SO in AM and or or in Cortex.
28:32
You're making the LLM call. Yes, in LM call, we have also
28:35
asking to give it a category specifically and with some
28:39
context, which is optional to take into account that, hey, it
28:43
already exists with these, try to keep it minimal and noise
28:47
free. And so that it doesn't really
28:51
give you any kind of category because you can name one thing
28:55
in many different ways and that that won't be good for us.
28:58
So how it started, it should kind of like keep on going on
29:03
that direction and not really diverge a lot after this.
29:07
What happens if we ask the LM for the different categories and
29:10
their connections? I mean, not connections, the
29:12
categories and the subcategories, just name them
29:14
basically. So we check the frequency, we
29:17
form all of these like small, small collections.
29:19
And when we are falling, when we are creating these collections,
29:23
what we are saying that hey, now we have this, let's say 15 or 16
29:29
memories together, we're forming this collection.
29:32
Let's give it a description like what the whole thing is about.
29:35
So now the description will be sort of like what are the key
29:41
things that we are keeping in the memory?
29:43
So this is something that happens in the background.
29:46
By the way, it's not something is actively happening when you
29:49
are like using the memory system.
29:51
So it keeps on running on the background which keeps on
29:53
checking that if it has reached a certain threshold or not.
29:57
If it did, then start a background process to get a
29:59
collection out of it where you give it a summary which we are
30:05
also calling as a description and give it a query helper.
30:09
So what query helper is? This is something that will come
30:14
at the time of retrieval. It kind of.
30:17
It is a prompt, so it's it's a meta string you can say which
30:23
will help you to create a prompt on how to query this.
30:28
And so when you're making that background call and you're
30:31
putting it into, you're inserting it into memory.
30:34
You're doing both at the same time, One for the hierarchical
30:37
structure and then also all of the stuff that you were talking
30:40
about before with is defined by and what the score is.
30:43
And so it's 11 LLM call that will give you all of that.
30:47
And then you can parse that out and say, all right, we have
30:51
everything we need now for both of these.
30:54
Structures, these two are happening parallel in the
30:57
background, so it's not really synchronous.
31:00
So don't they don't really like affect the latency together.
31:03
So one we are calling as the global search this it searches
31:08
things like globally and the other we are calling as like
31:11
auto collection search. So this is like a very
31:14
constrained and narrowed down search if you look at it.
31:17
So at the end we take some of this and take some of this and
31:22
only take the most important ones that matches the most with
31:25
the query like what actually we are like looking for and just
31:29
feed that or like return that as a memory retrieved item you can
31:34
say. So yeah, to continue about, I
31:37
think how I was talking about the retrieval of the auto
31:43
collections. So we sort of like form this
31:46
query helper, which is like how how should you even go about
31:52
querying this collections? Because now if the queries like
31:56
fix this. Now if it matches with the
32:00
collection Python, sorry collection work.
32:02
Let's say now in work the query helper would be based on the
32:10
description. Does it really help to even
32:12
query this? So there's like two options, one
32:14
is to query or even not query. So now if you actually want to
32:19
query this whole collection then how should you modify this
32:24
query? Because in RAG there is this
32:27
concept of like query expansion or rephrasing where you actually
32:31
rephrase the query based on what the context is.
32:34
But sometimes like fix this might be related to the previous
32:37
sentence that you are talking about.
32:40
Because if you don't provide it some sort of context, it can't
32:43
understand. So query helper takes into
32:46
account like the context and this new query and it checks if
32:50
it's relevant to this collection or not because it has access to
32:54
a description also. And since it has all of these
32:58
things, it will generate an answer in yes or no like let's
33:01
say if it's yes, then if it will modify this query with fix this
33:08
thing about Python, let's say. So it will kind of modify that
33:12
whole query to be retrievable or like searchable.
33:18
So, OK, so till I since I mentioned this point, so there
33:23
is one key thing that is at the time of retrieval we are taking
33:27
this and we are doing two things.
33:31
One is on the smart collection side, we are doing two things.
33:35
One is we are taking this query and sort of doing a semantic
33:40
search over the different subcategories and collections
33:44
that we have. I mean, over the whole
33:46
collections that we have, let's say you figure out four top
33:49
collections out of it, the most similar ones based on its
33:53
descriptions. Now you give it 30% importance
33:57
only. So now we form sort of a
33:59
composite score because currently we are giving it only
34:03
30% importance. Now we are giving 70% importance
34:06
to what are we querying inside all of these top K collections.
34:12
So inside all of these top K collections, like each of these
34:14
collections might have like 15 or 16 memories or like whatever
34:18
the threshold you have set. Now among them, we do a semantic
34:22
search through the modified query for each of them.
34:27
By the way, now we are giving the 70% importance.
34:30
And finally by two important scores like 30% and 70%, we are
34:36
finally coming up with a composite score for each of the
34:39
memory points. And then we sorted and then we
34:42
see like what are the most relevant ones?
34:45
Let's say you only select top J out of the final composite
34:49
scored memories that you have retrieved.
34:52
And then you take this into account from the auto
34:55
collections retrieval and you take into account the whole
35:00
global search retrieval also. And then you just select, I mean
35:04
however much you need and you just give it back as an as
35:07
something like which LLM systems can use.
35:10
Yeah. That's that's like the whole
35:13
overview of it I guess. What about like when you want
35:19
time based questions or recency and I say tomorrow I'm doing
35:25
this but then tomorrow becomes last?
35:28
Week currently. I realize this a bit late while
35:32
we were like building Cortex but it has support for the same
35:36
thing. So there are two things I can
35:37
see right now. 1 is when we want to do these recent based queries
35:42
and when we want to do a particular date range based
35:46
queries. So it might be something like,
35:51
do you remember that what happened on March 20 from
35:55
between March 2025 to April 2025 like on the 1st 15 days.
36:02
So, so that's like one one way and another can be something
36:07
like, hey, what did I talk about?
36:11
What did I talk to Dave yesterday, something like that.
36:14
So there is one step that we are we are wanting to do.
36:18
So for all of these like date range based queries where you're
36:20
expecting them have two options. One is to like provide a date
36:24
range and another is to kind of I mean, yeah, we are calling it
36:31
like temporal weight. So what temporal weight is now
36:37
the weight can be anywhere between a zero to 1 and words
36:41
zero. Like what temporal weight being
36:43
0 means is you don't give any sort of bias to decency.
36:49
So currently it just happens like normally how it was
36:54
happening, the whole hybrid search.
36:56
But if the temporal weight is like 1.0, now what you do here
37:02
is you you do the retrieval bit from just STM.
37:06
So what I mean by STM is like from just the short term memory
37:09
that you have. And since short term memory is
37:11
like a window and now you give importance to the most recent
37:15
ones only. So that's that's something.
37:19
And now if the temporal weight is like let's say 0.7 sort of.
37:25
So in 0.7, what we do is we at the time of retrieval, we are
37:32
going to a long term memory and we are taking the whole query.
37:37
And if you remember, we have always a limit at the time of
37:40
search that how how far to look back, how like what should be
37:44
the size of K. So currently we are kind of
37:47
multiplying it by some, some factor.
37:51
So based on like how what your weight is, we take, we create a
37:55
factor and we multiply that so that we get even a bigger
37:58
window. And now we basically take all of
38:03
this into consideration because at the time of auto collection
38:07
search, we have this composite score mechanism.
38:10
If you see so in composite score, it doesn't matter if so,
38:14
I mean if you if a collection gets like higher score because
38:20
it will only have like 30% importance, there might be 70%
38:24
more importance on like something which was like lower
38:29
down in the line. So overall the composite score
38:31
can give you like a better distribution I can say.
38:37
I think I'm understanding this. Is it keywords that are
38:43
triggering how much weight you're giving to that long term
38:47
versus short term? With keyword yes.
38:49
So we have like all of these set of keyword checks so.
38:52
If I say last week, that's a keyword, or if I say last year,
38:55
that's a keyword and that returns K.
38:58
Yes, exactly. So recent would have like
39:02
something 0.6 as the K So recent, I mean, yeah, we
39:07
shouldn't like keep it more higher.
39:09
So last week we can have something if we're higher.
39:13
So the the temporal weight is getting decided based on like
39:18
what the keywords are. We have like a huge set of
39:20
keywords. I mean, this we basically did
39:23
because it's efficient. But an even better, better
39:27
method will be to ask an LLM if you're OK with having more
39:32
latency on the whole system. But that's something I think we
39:35
all can improve. And like how we are connect
39:37
currently deciding the weight. But the functionality is there.
39:42
And there's also one more parameter where we can pass the
39:45
date ranges. So it will constrain the whole
39:48
search between only these two ranges.
39:50
And it will use all of these auto collection thingy and the
39:54
hybrid the global search between a certain date range only
39:58
whenever like you're taking a query you can always.
40:02
Extract some date range out of it using the whole context
40:05
before querying the memory system so that you have some
40:08
date range to look around for. So it seems like you've got a
40:14
lot of data that you're pulling from, whether it's the short
40:19
term context and you're deciding how much short term versus long
40:22
term versus the hybrid search or the file structure, for lack of
40:29
a better word. I can't remember what you were
40:30
calling it and the knowledge graph style.
40:34
Do you find that it all kind of collapses upon itself if there's
40:37
a right or wrong answer, or because I imagine it all leads
40:43
you to the same place if there's one answer, but maybe when it's
40:47
very fuzzy you can just get this bloat of a whole ton of noise.
40:53
Yeah, Yeah. First to check the fuzziness, we
40:56
are kind of doing like when similar things are coming up,
40:59
you obviously are doing like deduplications there.
41:02
So because at the core, the memories are represented as like
41:05
an ID in the database, so you can do deduplications there.
41:09
Now, how much noise to actually incorporate, and that actually
41:15
depends on your limit. And what I mean by limit is like
41:20
how many memories you actually want to retrieve.
41:23
So you can always like tone it down.
41:26
So it can be like, I just want like 2 memories and that reduces
41:30
the noise itself. Yeah, I think if, if you're like
41:33
losing out on something, if you're like losing out on
41:36
something, just increase the K probably so you can get like
41:40
more memories out of it. And you can totally like shut
41:44
down the global search. That will be super constrained
41:47
then, so you can just remove the.
41:50
You, you can do that on a case by case basis or you do that
41:53
just oh, really? Yeah.
41:55
So it's it's just a flag in the code base.
41:57
So it's like it will be totally shut down and only the auto
42:01
collections thing you will be working.
42:03
So there you can get like a much constrained and if it's like
42:08
like a case where you don't want any false positives.
42:12
So that will be the case probably.
42:13
Have you messed around with? I feel like there's potentially
42:17
some cool stuff you could do where it's cascading or first
42:21
you try with one and if it doesn't work then you can go a
42:24
little bit deeper and a little bit deeper.
42:26
Maybe you try with just the short term memory plus hybrid,
42:32
or if you can't find it, then you add in the knowledge graph
42:38
or and if you can't find that, then you just say, all right,
42:41
well let's try again with everything and see if we missed
42:43
something or let's increase K. So this this idea, this is
42:48
something that I have in the plan that is currently you see
42:52
the whole cortex as something that human is using right now,
42:55
right? And it has like a lot of these
42:57
different parameters in it. But what I want to actually
43:00
convert this into is sort of tool where it's not humans that
43:06
will be using Cortex, it will be the Asian tech systems that will
43:09
be actually using cortex. So imagine memory as a tool.
43:13
So I mean, currently the MCP is like MCP trend is like
43:16
skyrocketing. So I guess it makes sense to
43:20
call it as an MCP tool. But imagine it's not us who are
43:24
controlling these tweaking parameters, but it's an agent
43:27
because it so if it's an agent there it is automatically
43:31
processing your request and it is forming that hey, what the
43:35
date range should be while calling cortex.
43:38
So it has cortex only as a tool. So in that way it can tweak all
43:44
of these things and see like like what parameters give it the
43:48
best result. OK, let's get some coffee from
43:52
Tiero and then we can. All right, we're back.
43:58
Maybe we can talk now for a second about context engineering
44:01
and how this all fits into that. Yeah, sure.
44:04
We at Prem are thinking very deeply about how to kind of
44:09
create agentic systems. And we are as we are like
44:12
building 1 internally that we realized that giving all of
44:18
these agents these like different tools, but making sure
44:23
that the the two uses and like number of tools that these
44:29
agents actually have is like highly, highly important and
44:33
making sure that there is only one thing on the memory side.
44:37
So that's that can be something like very crucial because now.
44:42
You're you're eliminating noise in a way.
44:44
If you only have one 2O4 memory, then you don't have to go and
44:50
use up all of the this memory kind of search aspect if it's
44:57
not needed. Yeah, yeah.
44:59
That, that actually reduces take a lot of noise on the tools,
45:03
number of tools it has access to.
45:05
So there is like a paper out there that the number of tools
45:09
that you use, the accuracy of like the task completion rate of
45:14
an agent goes actually down. So the graph is kind of like
45:18
this, I would say, yeah. And I've heard people talk about
45:22
how much more expensive it is if you now have to give access to
45:25
all these different tools, your input tokens go up.
45:28
Yeah, yeah. So like there's just getting
45:31
like more confused and confused with like so much noise of like
45:34
different tool definitions and all.
45:36
So I mean there are some strategies where we can
45:40
eliminate that. So there is there has been very
45:42
nice blog about the same context engineering by Manus also.
45:46
So where they talk about a prefix caching and how to how to
45:52
just block one tool from the LM to C so that it doesn't really
45:57
use them when it's not really required or it doesn't even have
45:59
that in its context. So all of these like new
46:02
patterns that are emerging like as a whole, the whole context
46:05
engineering thing. So I mean, it was all prompt
46:08
engineering before, but it's so much bigger right now because
46:12
currently the way you use these tools and how you feed in all
46:16
the information for the whole completion of a task is like has
46:21
has become the more thing and like more and more people are
46:23
working on the same. Now giving the whole memory as a
46:26
tool to an agent makes very much sense because we don't have
46:31
these like static flows anymore where like a human will like for
46:36
only a certain cases like a human would retrieve on what
46:39
it's required because agent can do the same thing.
46:42
And we are constraining on how much freedom to give to the
46:46
agent by working on the memory tool aspect itself.
46:50
So boy, how we are like building cortex is like we are reducing
46:54
on how much freedom to give the agent who is using this as a
46:58
tool. Then we can give it like lot of
47:00
freedom with lot of different parameters or we can constrain
47:03
it down to very minimal parameters.
47:05
That's one way I think how we can reduce down entropy
47:09
overhead. Yeah.
47:11
Last thing, you're interested in vision memory also.
47:16
Yes. So this is a coming soon feature
47:21
I would say in Cortex that I'm working on right now.
47:23
So I think there are some nice implementations done by a lot of
47:29
different people. Currently people are thinking a
47:32
lot about how to represent all form of senses or like signals I
47:40
would say as a part of their part of the agentic memory.
47:44
Because currently if you see it's all text, like everything
47:47
that is people are working with are just text.
47:50
But what I'm seeing the trend right now is people diving so
47:54
much more deeply into the whole video, vision and audio also.
48:01
So like just to see the bigger picture, I think we have to
48:05
target the whole 5 senses like how human see and feel and can
48:12
experience different thing to all of their five senses.
48:15
Something similar has to emerge for the Asian tech ecosystem
48:19
also, because all of these has to merge together and have there
48:23
should be a way that it's able to intercommunicate with each
48:28
other. Like the vision memory should be
48:29
able to communicate properly with the textual memory that it
48:33
has access to from day-to-day. Or it can be just some audio.
48:36
Obviously you can compress these down to only one domain, like
48:41
compressing down a video to only a text and it just interacts
48:45
with everything is a text at the base.
48:47
Then same with. Audio.
48:48
Yeah, but you're losing context there.
48:51
So this is not a loss less compression.
48:53
You're just compressing down the video and you're losing a lot of
48:58
context, even if like you are explaining it very nicely on
49:01
what the video is about. So there's this nice thing that
49:07
got released by Memories AI. So they're working on large
49:12
vision models something. It was large language memory
49:16
vision models something. It was like that where they are
49:19
kind of processing the whole processing a whole video to make
49:23
it more searchable, more indexable.
49:26
So now this indexable thing is not like that.
49:31
It just it's just converted into text, but sort of like an
49:34
embedding. But feel free to like check out
49:38
the memories AI. But I think that's like the
49:42
start of the things that we are going to see in the coming
49:44
months, like let's say next 6 or 12 months where we are able to
49:49
properly index and store video data or like any sort of vision
49:54
data or audio it can be. And at the core, it will be just
49:58
numbers. It will be just vectors or like
50:00
however you want to represent them because it shouldn't be
50:03
text because text should be a more high level version of it.
50:07
So at the core it should be something all the same and which
50:11
is which has like most of the context without having any sort
50:16
of like lossy compression I would say.