Latest / Agentic Conversations (formally mlops.community) / Does AgenticRAG Really Work?

Transcript

Does AgenticRAG Really Work? · Agentic Conversations (formally mlops.community)

0:00
So often agent lies in how dynamically generates a prompt
0:04
which is passed to the large language model.
0:07
For instance, if I can just say hey this, this is my table
0:11
schema, generate a SQL query for this, it'll give you a SQL
0:14
query. But the problem would be it's
0:16
going to be highly hallucinated. It's been an interesting space.
0:27
Like so far, we know we started with, you know, a very basic,
0:30
let's say if somebody asks me like like what ML is, for
0:33
instance, you know, like what is this thing?
0:35
So back in, if you remember in high school, we would have, you
0:39
know, these simple equations like we're given two data
0:41
points, X1Y1X2Y2 and we were asked that, you know, OK, what
0:46
would be the value of Y-3 and X3?
0:48
And that's extrapolation. And essentially that's what
0:50
entire ML is. It's just that instead of flying
0:52
now we have so complex, you know, data points, so complex
0:57
essentially curves and so complex counter maps.
1:01
Like it's multi dimensional. It's not even 3 dimensional
1:04
anymore. We just can't like, you know,
1:05
visualize it in a human way. But there are so many dimensions
1:10
to so much of data that we have. And essentially it's all about,
1:13
you know, like being able to predict based on what we know,
1:17
kind of, you know, finding out the models, which are
1:20
mathematical functions that could simulate what we have
1:24
going on in the real world. So essentially what we are
1:26
coming down to is creating more and more of those complex
1:29
things, be it like, you know, image processing, for example,
1:33
in our, you know, live video feed for cars in the way more
1:36
that we see out there, they're able to predict, you know, right
1:38
now when to stop based on, you know, like the distance
1:42
calculations. And that's all like functions
1:44
and how we got here towards this genii, which is a great word
1:49
right now, a huge buzz around it.
1:52
Interestingly, we used to have, we started with neural networks.
1:55
We had like CNNS, convolutional neural networks, we had RNNS,
2:00
recurring neural networks, RNN were the ones which were used
2:05
extensively for NLP, the natural language processing involved at
2:10
how we are talking right now. We should be able to talk to
2:13
machines too. And we can understand like, you
2:16
know, like what context of the any, let's say website is or
2:20
any, what somebody's talking about.
2:23
But the problem there was they did not have much attention
2:28
mechanism. The attention span was so low
2:31
that after like a few words we just did not have the attention
2:34
window that could go back and relate to what was being said.
2:37
It's like a goldfish. Yeah, essentially.
2:43
And that did improve when LSTM's came into picture, which was
2:48
long short term memory neural Nets and then Grus came into
2:52
existence. But still the attention span was
2:55
so low and that was primarily because of the architecture that
2:58
we were following. What we were trying to do is
3:00
we're trying to induce gates in these neural networks which
3:03
could ascertain how much of the context for input that is coming
3:08
through we want to retain and how much we want to forget.
3:12
But those control gates, they were not efficient enough.
3:15
There were there is no way we could have achieved what we have
3:19
until and unless the 2017 revolution happened and there
3:24
was this paper you would have heard of.
3:25
Attention is all you need. The moment it came through the
3:28
transformer architecture when it was introduced, it just changed
3:31
the entire game and here we are now.
3:35
One thing that I'm thinking about with this model evolution
3:39
that you just broke down is what the next architecture is going
3:43
to be. Because there's things that
3:45
happen with Transformers that are not things we want, like
3:50
hallucinations. And so sometimes some people
3:54
will argue, well, that's like a feature, not a bug.
3:57
And others will say, well, you know, like we really want it to
4:03
be reliable. And if you're going to have
4:06
hallucinations, then it's not going to be reliable.
4:09
But at the same time, it's AI, it's machine learning.
4:13
It is probabilistic. I don't know like if
4:17
hallucinations are the feature, but yours truly said it's all
4:21
probabilistic models, right? And it essentially depends on
4:24
like how even the transformer architecture evolved.
4:26
So from LSTMS, like as we were just touching base on that, we
4:32
essentially found out that OK, what if instead of using these
4:37
gates to control the attention span, what if we were to
4:41
actually have some self attention mechanisms from where
4:44
these architectures started evolving.
4:47
We had encoder and decoder, then we had encoder only models and
4:51
decoder only models. And it started from 2017 and
4:55
eventually it was it just grew so much.
4:59
So for instance, encoded only models were aimed at just
5:03
understanding the context of what this particular text is
5:05
talking about. And we had this query key and
5:08
value vectors. We're using soft met functions
5:11
for assigning probabilities to these words.
5:14
And then in the decoder, we wanted to make sure that when
5:16
it's able to predict, we have some masking so that it's not
5:21
able to induce a dialect from already pre learned, you know,
5:25
words so that for example, when me and you, we are talking
5:29
before I register the input, I should not have any bias induced
5:34
in me. And that was the aim.
5:36
But the problem is that that bias is somehow induced because
5:39
humans are also using these systems where these
5:43
hallucinations actually come from even more.
5:46
And since it's a probabilistic model as such, the softness
5:49
functions, they produce probabilities right now.
5:52
But moving ahead in the architectures which we are using
5:54
in production right now, RAD was a big change.
5:57
Yeah, yeah, that was the one. Yeah.
5:59
So it wasn't on the model level. It was more on the system level
6:03
in that way. And so it went from like, all
6:05
right, cool, we've got this really important model, but now
6:08
how do we architect the system around it?
6:09
And so last year, RAG was all the rage.
6:12
I think probably the last two years, Rag was very important.
6:16
And that was like the next step in the evolution.
6:19
We could say. We had the ChatGPT moment and
6:21
then we started playing with it. We started using tools and
6:25
chaining together prompts, and then RAG became very popular.
6:29
And then you moved on to agent RAG or Agentic RAG, right?
6:35
But let's talk about RAG and what you were doing there and
6:38
why it wasn't enough. Yeah, I think this is a very
6:43
interesting domain, specifically because when the helicinations
6:45
started to come in, now we have something that understands, you
6:49
know, the context of language. Now we have a model that wants
6:53
to talk, but it just doesn't know what to talk about, you
6:56
know, essentially, and RAG is essentially, I always envision
7:02
it in a way that it's like a kid, you know, you're watching
7:05
him give an exam or her give an exam, and it's an open textbook
7:09
exam. So that kid is referring to the
7:13
books and getting you what you're asking it.
7:17
But at the same time, we need to check for two things.
7:20
If it's referring to the right books when the questions are
7:23
being asked as well as like when it's answering, how much of the
7:26
context that it is giving makes actually sense.
7:30
So RAG essentially came through that picture where we would ask
7:34
let's say GPD of random question and it would start giving us
7:37
irrelevant results, Semantically correct, syntactically correct,
7:42
but contextually not so. Relevant.
7:45
Yeah. And especially, I think I
7:47
remember Rag became very popular just because of the fact that
7:51
people wanted up to And anytime there's something that comes up,
8:04
we can search the vector store and get that information.
8:07
Right. Yeah.
8:08
And yeah, and and that was a really one of the, I would say
8:13
Genesis of Rag in that sense. And moving on to now like where
8:17
we are with the agent AKI or that we say RAG agents, it's
8:21
actually being used very across different context for which if
8:24
it was part of in the beginning. So as we were discussing when
8:31
RAG started, the aim was to actually ground the LLMS, you
8:35
know, let them make much relevant decisions based on the
8:38
Victor stores that we have and for example, now the ones that
8:42
we are implementing. And it could be a very generic
8:46
case study, like if I want to talk to my databases, let's say
8:49
right, and I need to generate a SQL query, LLM can generate a
8:54
SQL query, but would it be relevant at all?
8:57
No. Again, hallucinations come in.
9:00
It doesn't even have a lot of context.
9:02
And more than that, how can we even make sure?
9:04
Because now if I'm talking to a production or staging data, it's
9:08
risky because what if some user would come in and just add a
9:11
drop table statement right there?
9:13
My job is. Gone drop table.
9:14
Yeah, your job gets more complex, but so why did rag fall
9:22
over? Like where was?
9:24
Why did you switch to a Gentek rag and what are the differences
9:27
between the two? So if I were to make an
9:31
analogous comparison. So when the REST architecture
9:35
came into existence, we started developing these REST services.
9:38
I remember we started with SOAP services, then Restful APIs for
9:43
the standards, then we delve into micro services, basically
9:48
contextualizing each service very specific to the use case so
9:53
that it's easier to scale. It's much more contextually
9:57
relevant and as well as it gives us a much more relevant results
10:03
with respect to the architecture and reusabilities of these.
10:06
And here in Agentagi similarly when we are, we are grounding
10:10
the rags for like very generic use cases.
10:12
We're getting great results. But now in the multi agent
10:15
systems we are creating those agent rags for a very specific
10:19
use case and now these agents are talking to each other rather
10:23
than having one wholesome agent for.
10:26
OK I see. So the idea is trying to like
10:29
break it down into micro services and say your an agent
10:35
that has access to this vector database and another agent can
10:40
almost like use you as a tool. Yeah, yeah.
10:43
And so the tool is search and retrieval type tool but on our
10:49
data in some place. Yeah, yeah.
10:52
And that also helps us in the terms of specializing the
10:56
context of a specific agent. It's a separation of concern and
11:02
we can have as many layers for security or for enhancing or
11:08
enriching the data in between and it becomes individual bots
11:11
who are just taking care of these things.
11:14
And are you making each agent like only give only giving
11:20
access to one database? So it's like that is the
11:25
marketing database and you can call that agent and it can
11:28
retrieve everything and then enrich or summarize the answer
11:34
and then give it back to the main agent.
11:38
Yeah, in somewhat in those terms specifically.
11:41
So aim here generally is that we create agents in a way which are
11:47
like very scalable, at the same time making sure we have data
11:50
governance in place. Because let's say also we have
11:55
different dialects of data across different systems.
11:58
And one of the ways that we can implement is using a SQL clot,
12:02
which is a security layer which can transform.
12:05
But at the same time, sometimes we that's not desirable because
12:10
of different systems in place, separation of concern being
12:13
there. So what we do is essentially
12:16
design very specific bot for very specific use cases.
12:20
And it also helps in the cost optimization.
12:23
What if this was not to be used in production, but it was to be
12:26
used for when I say production, I mean for the outer world, but
12:32
more like, you know, for internal efficiency of the
12:37
workforce, let's say, or on boarding.
12:40
So do we really need those kind of resources to put into those
12:45
agents versus the ones which are gonna be consumer facing?
12:48
Yeah. So these help in making those
12:50
kind of decisions. Yeah, exactly.
12:52
There's a lot of different trade-offs that you can be OK
12:55
with. I imagine if it's just
12:56
internally facing, Yeah. And on so many different
13:02
vectors, probably quality and on speed on or or maybe you're
13:08
like, no, we have to get it really fast because it's.
13:11
But reliability I would imagine is you just have less high of a
13:17
bar if it's internal because the internal user is going to be
13:21
much more forgiving than the external user.
13:23
Oh 100% yeah. And hopefully.
13:26
Hopefully. Yeah, exactly.
13:29
And so then all right, so I'm kind of understanding it.
13:33
I think the thing that I ask myself, if I'm understanding
13:38
this correctly, you have agents that are able to query
13:41
databases. Why not just make an MCP server
13:45
for the database? We possibly could that could be
13:50
a way to go, but the main context comes through is like
13:56
what use case are we like serving?
13:58
Essentially if it's let's say we have to like talk, let business
14:04
talk to a database and they want certain reports in just certain
14:08
ways and we want to expose that. Would we want to go through the
14:12
route of MCP server and create whole another layer to it?
14:18
It's at all like optimizing how much resource and information we
14:24
should put into for achieving a specific use case at the end of
14:29
the day. For consumer facing probably
14:31
that might make sense. For me, what it sounds like is
14:34
you have different use cases and they're very verticalized.
14:38
So maybe there's a team or there's a suite of folks that
14:43
need information and you create an AI product that can do that
14:49
thing really well. So create dashboards from the
14:53
sales data or the financial data, whatever it may be.
14:57
And then you have another product and it's in a way
15:01
separated. And so you have like separation
15:04
of concerns, which is really good, but at the same time, you
15:10
have to create a whole new product around it.
15:12
Is that it? I imagine some of the pieces are
15:14
going to be reusable. And you can say, all right,
15:17
well, this is similar. We just need to change this and
15:20
tweak some prompts as if it was that easy, but and give it
15:25
access to this database instead of that database.
15:28
But is, is that how it is? It's like each individual
15:31
product and then you have to upkeep the products for the
15:34
internal teams. Yeah, it's almost in that
15:38
direction specifically because so definitely we can't like
15:43
discuss details into the internal architecture, but in a,
15:47
in a very wholesome level, that essentially what it boils down
15:51
to. If, let's say, if I'm building a
15:54
bot for like 1 of the teams in Slack, which aims at onboarding,
15:57
for instance. And similarly, and there's a bot
16:00
which works towards, you know, working with different
16:03
databases. So let's say in sales.
16:06
And these two are going to have some interchangeable components,
16:09
but their vector DBS are going to be different.
16:10
For instance, because let's say if it were just creating SQL
16:15
queries, we don't need something like Vertex, A matching engine,
16:19
or Milva's TV for that matter. We can use something very
16:23
lightweight like Fi's which is Facebook XLRT search.
16:26
Still running hard. It's created all those years ago
16:31
and it still is just amazing how well it works.
16:34
Yeah. But I, I, so I understand that
16:38
it's, you choose what you need to use also depending on the use
16:44
case, because the use case almost dictates what kind of
16:50
necessities you're going to have.
16:53
Yes, yeah. And so some of it can be, Oh
16:56
well, you're, we're going to need access to the same
17:00
databases because there's some overlap, but I imagine you're
17:05
not using the same agents for those, you're creating new
17:09
agents that because then there could be some like context
17:13
mixing and that could be bad. Oh yeah, yeah, that's all true.
17:18
So the soul of an agent lies in how dynamically generates a
17:23
prompt which is passed to the large language model.
17:27
For instance, if I can just say, hey, this, this is my table
17:31
schema, generate a SQL query for this, it'll give you a SQL
17:34
query. But the problem would be it's
17:36
going to be highly hallucinated. It would not know where to join.
17:40
It would not know how to or which particular columns to join
17:44
on, You know, and aim is to to essentially eradicate that
17:49
middle layer where we have to constantly describe these
17:52
things. Yeah, right.
17:53
So then it boils down to, OK, how do we create that?
17:58
How do you make sure that our system or our agent creates this
18:01
dynamic prompt which is passed to the LLM?
18:04
Because LLM is going to do its hallucination on its own side
18:07
for sure. We can't stop that.
18:09
Yeah, it's like plain telephone, but how are you making the SQL
18:15
queries then if you're not letting the LLM generate it?
18:19
The prompts are actually dynamically generated.
18:22
SQL queries are definitely generated by the LLMS. OK.
18:25
So the prompt which is passed to the large language model, those
18:30
basically are generated based on the documents which are
18:32
retrieved by the retriever and those documents are retrieved in
18:37
this high dimensional space through semantic similarity.
18:41
Excuse me searching. And that's where the Victory DBS
18:46
role come into so much of picture.
18:48
Like what kind of Victory DBS do you want to use?
18:50
Yeah. Do we like really want to go
18:52
with something? Let's say if I had if I want to
18:54
search across all the like 440,000,000 products of Walmart
18:59
and I want to like find out the products for let's say all the
19:03
users, it's humongous data. Yeah.
19:06
Even creating embeddings for that.
19:08
My gosh, it just boils on the system.
19:10
So expensive, I imagine. Yeah, I'm just, I can't even
19:15
fathom how much data that would be.
19:17
Yeah. And so then you have to decide
19:21
what subset of the data you want and then throw it into a vector
19:25
DB. And you're spinning up new
19:27
vector DBS for all these different use cases.
19:31
Yeah. So we make a choice on like
19:33
based on what industry standard is being used, why, how much the
19:38
cost if do we have an open source solution for it And
19:42
sometimes open source solutions are available like Milwest DB,
19:45
great Vector DB, you know, But the problem is it can have a
19:48
little bit of high latency even at the same indexes like IVF, P8
19:53
or IVFPQ for vertex AI. But the problem there is if I
19:59
were to generate, let's say recommendations.
20:01
For my customer base and if I were to generate them once a
20:07
day, I can probably use, you know, an open source.
20:12
Why would I want to spend something on something, you
20:16
know, which is really costly for me.
20:20
And it's again, very use case, you know, specific.
20:23
But if it were an online serving model, which is, oh, I'm
20:26
generating them every 1/2 an hour, the pipeline is constantly
20:30
running or every 15 minutes in that case, yeah, I would have to
20:34
initial out that cost and then that would make sense that the
20:39
use of very low latent and very highly complex indexable vertex,
20:48
sorry vector DPS that we can actually use.
20:51
If you got the chance to just start from scratch and build
20:54
something, how would you go about it?
20:58
OK, let's take an example. Give an example like.
21:01
What would we need to build? What would we need to build?
21:04
Is there something? Is there something that you feel
21:08
is is uniquely valuable in the e-commerce space?
21:15
There are a lot of things. I would say the two most
21:18
prominent examples that come to me is so sometimes when we
21:22
launch a new product for instance, or a new customer
21:27
experience, we generally go about doing multi bandits or AB
21:31
testing. But sometimes we already have so
21:36
much of good control experience and that we don't want to you
21:43
know sway away from because it can cost us potential users.
21:47
So and if we try out even in multi banded way like we use
21:52
Thompson sampling to sway 1 or 2% of the users, are we really
21:57
willing to take that risk in that case?
22:00
Because it's like you have so much, you don't want to lose
22:04
what you have. You're you're playing defense in
22:08
a way. Yeah, yeah.
22:09
It's explore exploitation, trade off basically.
22:12
You know, OK, yeah, that's fascinating to think about that.
22:15
It's not like, yeah, you can't be Willy nilly because you're
22:18
already so optimized, yeah. Yeah.
22:21
So then in in explore, exploit, trade off that we generally go
22:25
around here, we think in a way that, OK, I'm going to go ahead
22:30
and use my control group, which is how it is right now.
22:35
But at the same time and I'm going to exploit it.
22:37
But I might like explore like 1%.
22:41
It's definitely a cost to the business, but it can yield a lot
22:45
more. But it's very dynamic in map,
22:49
you know, like we'd go about and this is called Thompson sampling
22:52
basically, right. So we just sway the users like
22:55
that. But another way that we can
22:57
possibly try is let's say we launch something some
23:02
recommendation model and we want to try or direct users to it,
23:05
but we want to do that like after they have done
23:08
experiencing the current product.
23:10
You know, after they paid money, after they paid money.
23:13
The upsell can be something that is experimental, yeah.
23:17
So they're like we once I remember used something, a kind
23:22
of a bot, you know, and that was the first experience of, you
23:27
know, us trying our Rag agents essentially that specific use
23:31
case. For instance, we were actually
23:32
going through a lot of products, a lot of embeddings from the
23:36
user data and we could not have gone with something like in a
23:42
lightweight. So at that time we did explore
23:45
very specific vector DBS like, you know, Wordx AI or, you know,
23:51
Middlewest DB. Yeah, if a use case is like
23:54
that, then yes, absolutely, I would go with those ones.
23:58
But if it's a use case, something like, you know, I want
24:00
to generate seek reports for the business, I wouldn't have much
24:07
data of the schema, you know, like it's barely in kilobytes,
24:10
you know? And in that case, I would use
24:12
something very lightweight open source, which is out there.
24:15
All I need to make sure is in that lightweight vector DB, it's
24:19
able to pick up at the right time, the right context.
24:22
By doing, you know, the right similarity matching, we get
24:25
cosine similarity, Pearson correlation.
24:29
They cover it finds Pearson correlation being one of the
24:32
really interesting things you know when we think about user
24:35
reviews and all but. Now that's, so that's on the
24:40
vector DB side. What about just architecting it,
24:43
architecting the whole system? You had a blank slate and you
24:47
come into a startup and it's like, OK, sweet, we want to
24:51
build this product. How do you go about that?
25:01
That would be very much, of course, reliant on the kind of
25:04
problem that we are solving. But let's assume if you're
25:07
solving a problem which comes through the RAG agent.
25:11
So first of all, definitely it would be what kind of data we
25:15
have. Is it a textual data?
25:18
Is it an image data? Is it like what kind of problem
25:21
you're trying to solve? And let's say if it's a textual
25:24
data, for instance, do we have a different context of the data?
25:28
Like do we have different data governance rules in place?
25:33
Like do we have different geographical locations in place
25:36
like Europe has? Like, you know, huge data
25:39
previously requirements compared to, you know, yeah, here.
25:43
This is a start up. We don't have shit.
25:47
There's, I imagine that is a beautiful thing if you have the
25:52
data governance rules and you know everything on.
25:57
Also just where the data goes and how you can't do anything
26:01
with the data unless you follow the processes.
26:04
Sounds like an amazing place to be in, but I imagine that you
26:07
get there or you need a team of people to be focusing on that.
26:16
That's not something that just magically happens.
26:19
Right, Yeah, that's true, that's true.
26:21
But essentially, when I'm trying to build up a prototype, I would
26:24
want it to be scalable to point where, whenever, because those
26:28
things are going to come through down the road.
26:31
And I would want to be like prepared with my architecture to
26:33
be able to include that. Interesting.
26:35
Yeah. So in that case, I would go
26:38
about, you know, building small agents for very specific use
26:41
cases and then rerouting the incoming queries based upon what
26:46
that specific context is. So that the agents like which
26:51
are catering to very specific data sets.
26:55
It's scalability and maintained at the same time making sure
26:59
that the rerouting that is happening, it produces very
27:02
optimal that dynamic prompts which can be then used to query
27:07
any large language model, tune it or fine tune it the way that
27:10
we wanted, any temperature that we wanted.
27:12
To and that routing happens with an LLM or that's just a router
27:17
that's like a. It's part of the RAG agent
27:21
itself. So there's an agent without
27:25
basically the last part, which is the generator.
27:27
So it's in the augmentation part where essentially we have query
27:34
coming in and then we are routing based on the what kind
27:37
of context we want to route that query specifically in and then
27:42
specifically querying those specific agents in a sense to
27:45
generate further dynamic LLM prompts that we would use.
27:49
Are there use cases that you particularly like and have seen
27:58
a lot of usefulness with? Yes, absolutely.
28:05
So of course data analytics is one part.
28:09
What used to happen before was like business would get back to
28:12
engineers and they would be like, OK, we want these kind of
28:15
reports. Of course they would have their
28:16
dashboards and like power BS would be there, right.
28:20
But now they can just straight away talk to these databases
28:24
essentially. And that's a huge, huge win for
28:29
any of. For the data team that doesn't
28:31
have to service those requests anymore, that's for sure.
28:35
Yeah, I remember talking to Donae about this 'cause she
28:39
built a data analyst agent. And one thing that she said was
28:45
the hardest part in building out the agent so that it gave
28:49
correct answers and it understood the context was they
28:53
had to build out a whole glossary of terms.
28:56
Since a lot of what you say when you are speaking to another
29:01
person, you're using this like lingo.
29:04
And even if it isn't marketing lingo, it is still fuzzy in the
29:11
way that our company or our team describes that.
29:16
So an MQL in marketing terms is like a marketing qualified lead.
29:21
And at this company, we describe it, someone who has, you know,
29:25
downloaded the ebook, but in another company, it's not until
29:29
you download the ebook and you come to a live event or you
29:33
reach out to sales because you went to a webinar.
29:36
And so there's even with the same term, the same word, it's
29:41
very loaded. And that happens across the
29:43
board. And so like, how do you do?
29:47
Did you do like a gloss or anything like that?
29:50
Yeah, I and that, that is a very interesting problem to be
29:52
honest. You know, even like within the
29:55
company and over different teams have different lingos for
29:58
different kind of things sometimes.
30:00
But yeah, and that's one of the things how I like came over
30:05
board with that was essentially defining the schema docs which
30:10
were being passed to let's say the rag agent that we have been
30:15
building. So you can define the
30:16
relationships there or the mapping essentially, which is
30:20
the glossary, you know, because as such, the large language
30:25
model doesn't understand anything.
30:26
It's just understanding the language, right?
30:28
We're telling it like, OK, what this is, how do we need to do
30:32
stuff? And it's just creating a very
30:35
semantically correct, syntactically correct answers
30:38
for us. But yeah, so a mapping layer is
30:42
for very specific lingos which can be mapped to a very context
30:48
specific terms. That becomes absolutely
30:50
necessary for that. And actually the other piece, I
30:55
think that could get tricky and I would love to hear how you
30:57
deal with it is that you get natural language questions like
31:04
how did we do this quarter, which is like what do you mean
31:09
by that, right. And so maybe the the agent can
31:13
come back and say, like, are you talking about revenue?
31:16
And it's like, yeah, are you talking about revenue generated
31:20
in the whole company on just your team on there's so many
31:24
variables that when you talk it's not clear.
31:30
And so how did, how are you dealing with that, that the
31:32
agent isn't just giving you stuff because it's the hardest
31:36
thing in the world is getting the agent to say, I don't
31:38
understand, right. Like it'll just come back with
31:42
like, oh, here's how we did this quarter.
31:44
And you're kind of scratching your head like I don't know if
31:47
that's actually what I was looking for.
31:50
So true and thanks Dimitrius for that because they really invoke
31:54
2IN thoughts in me like I I would certainly.
31:58
One is, I don't know if you've heard of Doctor D Kai.
32:02
Yeah, he actually has just recently published a book.
32:05
It's called Raising AI. Very interesting.
32:08
It's about how, you know, we need to work along with the AI
32:14
and he's a professor at Stanford here.
32:16
Really interesting. But we'll definitely get into
32:18
it. But from the technical side of
32:19
it, from abstraction to coming to a very specific use skills,
32:24
that has been the biggest challenge for us at the end of
32:27
the day and the reason why a large language model or an agent
32:33
would never say that. I don't know because it works on
32:36
our confirmation bias. At the end of the day, it just
32:38
wants to answer no matter what. So far, specifically for like 1
32:45
of the ways that I was able to achieve it was contextualizing
32:50
the prompts that we are developing dynamically.
32:54
Not the ones you know like which we write statically, but the
32:59
ones which are being developed dynamically in those agents if
33:03
and that's why that becomes the soul of the agent because if
33:07
it's able to provide from that abstraction, OK, how did we do
33:10
this quarter to OK, what do we need like revenue, we need
33:13
sales, we need products sold and all this information can be put
33:18
out. So it will give us very context
33:20
specific results and how do we do that depends on couple of
33:24
things. 1 is how is our data schema and structure and when we
33:30
say this quarter it's just going to pick up that date range.
33:33
How did we do, we can always map these queries to revenue,
33:38
products sold, let's say sales, employees performance, it could
33:43
be anything but and that's where very context specific agents
33:47
come into picture. If I am let's say business and
33:51
I'm just focused on the sales of it versus if I have I'm an HR
33:54
team, I'm working on the employees performance, I would
33:56
want to have two different answers to that.
34:00
And hence, even in the reusability of those agents, we
34:03
would have to like configure them or treat them in a way.
34:05
Although we can deploy the same ones and they can constantly ask
34:08
them, they'll get the result, but we would want them to be
34:10
very specific and precise to what they are.
34:12
So this is like a little bit of a personalized agent, yes, So it
34:16
knows that I'm on the HR team. And when I ask about how we did
34:20
this quarter, it's like employee engagement essentially.
34:25
Yeah, if I'm on the sales team, it's like how much did we sell?
34:29
But maybe if I'm on, if I'm in the C-Suite, I'm looking more at
34:33
like holistic view of how all the numbers are.
34:36
Yeah. And if you imagine, let's say we
34:40
are agents, right? What context do I have?
34:43
I'm just going to go and look up, OK, what documents or what
34:47
index I have in my DB. This is the query that came in.
34:51
There is an embedding of this query like let's say .12 point
34:55
3-4. I'm going to go and see in the
34:58
vector space where are like what are the closest points to this?
35:01
I picked that up. I bring it back.
35:04
Now those specific points in the semantic search, they could mean
35:08
very different things for different agents.
35:11
But those points which are suspended in there, that depends
35:14
on of course the embedding models that we are using and
35:19
because we cannot perform, you know, semantic search then
35:23
coming from different embedding spaces specifically.
35:27
Because what happens is it's a way to represent our textual,
35:33
pictorial or any non linear data in a numerical form essentially.
35:40
And when we are picking it up from the schema docs, let's say
35:44
or from any documents that we are feeding into the RAG agent,
35:48
those schema talks define essentially what context or
35:51
which let's say tables to pick that data from.
35:54
If I'm picking up from let's say a sales table, it's gonna give
35:57
me more context around sales. If I'm picking up from let's say
36:00
employees performance table, it's gonna give me more about
36:03
that. So inherently as an agent.
36:07
Wait, sorry, I missed that it was from vector space that it
36:12
gives you that information. Yes, because those vectors are
36:15
essentially mapping into numerical terms of what these.
36:20
I see. Documents or schema?
36:22
So you're enriching the schema with the vector with basically
36:27
vector space. Yeah, yeah, essentially.
36:30
OK. So it's like going into this
36:33
multiverse and then just finding out, OK, I just need to pick
36:36
something up, but I don't know what it's gonna map to because
36:39
the mapping has been done basically based on what it was
36:42
fed in before. And that what was fed in is very
36:46
much dependent upon what kind of relationships we have defined in
36:49
before, how the index has been created.
36:52
And that index creation essentially happens as a first
36:56
phase of these identity graphs. OK, that's super cool.
37:00
I haven't heard the adding a little bit of extra context so
37:05
that it understands and it's almost like you're saying
37:07
grounding the information again. It's going back to like Rag was
37:11
all about grounding the models and now we're grounding the
37:14
agents with a little bit of extra vector space, semantic
37:18
vectors and all that stuff. Tell me more about these
37:22
dynamically created prompts. How does that?
37:26
Work. OK, let's take an example and
37:30
we'll go back to our example. How did we do this here?
37:33
Yeah, or in this quarter, right. And let's say if I have 3 tables
37:40
or let's say we have like, yeah, 3 tables, we have a sales table,
37:45
we have orders table, we have a, let's say, products table.
37:50
Now in the schema docs, which are passed to the rag agent from
37:56
where the index is being created, we're defining that,
38:00
OK, these are the sales we tied to the products and as a key.
38:04
And there's a key for of sales products.
38:07
And third one was, let's say customers, for instance, right?
38:11
So aim is like, how did, when we say like, how did we do this
38:14
year? It would be based upon, we can
38:18
tell like what customers bought, like let's say this quarter or
38:21
what products were sold and, and how much revenue wasn't rated,
38:27
right. So that part where we are
38:30
defining this essential schema of the tables in the docs and
38:34
the relationships inside the docs, when the embeddings are
38:37
being generated, those embeddings are also
38:40
encapsulating those semantics of the relationships between these
38:44
three different tables. And when I ask this, and if I
38:48
this is the only information I have, I have not told Rag to do
38:51
anything else. When I ask like this quarter, it
38:54
picks up the time range, finds out in that time range about
38:57
like what all data it can fetch from these tables.
39:00
And then we fine tune like how the generation of these dynamic
39:05
prompts are going to be in retriever methods and during our
39:10
indexer as well. And then from there we can
39:14
actually be very specific about like what we want in the reports
39:18
outside. Interesting.
39:20
So you're getting it. I'm trying to think about like
39:24
the step by step nature of this. You're getting the query, how do
39:31
we do this quarter? There is a model that receives
39:35
that it also will go and search vector space and then it outputs
39:45
a prompt to go for another agent to go and use.
39:50
And in the output, it's where you're very specific because you
39:54
have the information of all right, this is whatever this is
40:00
the relationship between these three tables.
40:03
Here's what I want you to look for a specific agent.
40:06
Here's what you need to focus on.
40:09
And then go and find that and come back and, and then after it
40:13
finds that, it comes back and it tells that master agent, here's
40:17
what I got. And then the master agent will
40:20
have all of this in its input. And then?
40:25
Summarize it and then output something.
40:27
Yeah. And it could even use that to
40:30
let's say, do more, do more. So like we have this output now
40:35
and we are of course going to have these evals in between,
40:37
right? OK, now let's say we want to
40:41
relate it to some other set of databases and then the other
40:44
agent picks it up and goes and creates A dynamic prompt based
40:48
on that. So it's multi agent system just
40:52
communicating with each other at the same time making sure
40:54
they're very contextually relevant to their own specific
40:57
set of questions. But it's all happening because
41:01
we are able to create very relevant dense embeddings in
41:05
these vector spaces. And let's say a user query comes
41:09
in that is also actually put in the vector space or projected in
41:15
the vector space typing the right word.
41:17
And then from there the pickings are off.
41:20
Like what is the relevant context that can match this
41:24
query? So we don't even have to
41:26
manually define a lot of things for RAG agents until unless
41:30
absolutely necessary that it's not picking up, which is a very,
41:34
I would say to and fro process sometimes.
41:39
And that's why prompts are not very successful in getting large
41:44
language models grounded. Whereas RAG agents, they
41:47
introduce this additional step of suspending in this generated
41:51
index, our relevant information query comes in, it's projected,
41:56
now we find the match, we come back, we can read that dynamic
42:00
prompt now with that context and that gives us the exact results
42:05
that we want. Yeah.
42:06
It gives you a much more rich field to play from.
42:10
It's so much more enriched with all of that information as
42:15
opposed to just my simple words of like how did we do this
42:18
quarter? Yeah, Yeah, essentially, yeah.
42:21
Like, I guess I'm lost on how putting the query into vector
42:28
space right and then seeing what it is semantically similar to.
42:34
How does it already have all of this stuff that it's
42:38
semantically similar to? Makes sense so.
42:42
All right, yeah, I was hoping it did, but I didn't know because I
42:46
was confusing myself for a second there, to be honest.
42:49
No, no, no, that's very valid. Actually fine, because what
42:51
happens is. So before even we started asking
42:57
our agent any questions, what we did was we built an index.
43:01
This index was built on some documents and documents is a
43:05
very abstract term for any set of data that we would use.
43:09
Now imagine a three-dimensional space where we just have like
43:13
X1, XY, one or Z1 and we have essentially put just the data of
43:21
our 2 tables. Or even for that matter this
43:24
conversation. We let's say we want to go back
43:27
and review this conversation. Somebody wants to ask questions
43:29
about this. So we have created a kind of,
43:32
you know, analog for this. Yeah.
43:34
And what we have done is we have actually created embeddings from
43:36
this, from different, let's say each question and each answer
43:41
represents 1 document. Now when we are creating
43:44
embeddings for these different documents, we are suspending
43:47
these in the vector space. Now, let's say if somebody comes
43:51
up and asks, OK, when Dmitry asked this, what was the answer
43:57
or did these two questions make sense?
44:00
This query is essentially going to go in in the same vector
44:04
space and embeddings are going to be generated using the same
44:08
model using which the index was created from our conversation
44:11
before. Then it's going to go ahead and
44:14
perform that semantic similarity search and it can, we can tweak
44:18
it to use any similarities. It could be Euclidian, Cosine,
44:22
Pearson, any similarity and then it finds the closest matching
44:29
similarities. And we can also define how many
44:31
neighbors you want to for it to find the similarity too.
44:36
And that's like a hyper parameter that we can tune.
44:38
And let's say if we include too much of A context, then it can
44:42
actually waiver off and too little of a context, it can also
44:44
waiver off. But generally, for example, for
44:47
me when I did in the ragged and five was a 5 nearest neighbors
44:51
was a really good approximate nearest number of search.
44:54
It was a small use case, yeah, essentially.
44:57
And yeah, So in that case, when the new user, a query comes in,
45:00
it's suspended the same space and then it performs that
45:03
semantic similarity search and gives us back the dynamic
45:06
prompt, which is partial the large language.
45:08
Model and you're always updating it with the new queries.
45:11
You're always adding the new queries to the vector space.
45:15
Yeah, so that's So what happens let's say if I did not get the
45:18
results that I wanted? Yeah, I'm gonna go back to my
45:21
docs. I'm gonna check like, did I
45:22
define the schema correctly? What went wrong?
45:25
Why wasn't I able to like, you know, find a very relevant, why
45:28
wasn't it able to find a relevant context?
45:30
And I'm definitely going to update those dogs.
45:34
I'm going to define another doc which defines the scheme or
45:37
relationships between these questions.
45:39
So I'm essentially defining these things prior to make sure
45:44
that that our large language model, whatever answer it gives,
45:47
it's grounded. It comes from what exactly we
45:50
wanted to do. How are you dealing with the
45:55
problem of just having too much data and messy data?
46:01
Fortunately, the use cases that we have worked with so far, the
46:06
data was wasn't too messy. But let's say we encounter, for
46:12
example, we were taking talking about the one of our
46:16
conversations. If it's a, let's say a RAG
46:20
agent, which is talking to databases, it is going to be
46:22
very straight up, you know, it's going to be scheme of the
46:24
database. There's going to be like type of
46:26
the columns that we have and what relationships between those
46:30
tables are there. And very straight up, no,
46:34
nothing, you know, which can can be inferred essentially.
46:37
But in a normal context of things, we will definitely have
46:41
to make sure that we have proper data pre processing in place
46:44
where we are cleaning up the data.
46:46
We are removing any unnecessary tags which because we wouldn't
46:50
want our LLM to focus on very unnecessary information like
46:54
any. Yeah, random, you know.
46:56
It overweights like one word, and you're like, why?
46:59
Why did you care about that word?
47:00
It's not that big of a deal, yeah.
47:02
Yeah. And this essentially also comes
47:04
down to the our previous initial discussion about how transformer
47:07
architecture essentially works. It's essentially assigning these
47:11
probabilities to all these words and doing the MLM, which is mass
47:15
language modeling, predicting the next word.
47:17
And if you would want certain words to be upgrade more, you
47:21
would have to make sure that for a very specific use case, we
47:25
eliminate the parts which are not at all like desired or
47:28
required. Yeah.
47:29
As such, well, let's say that some data becomes stale because
47:34
for some reason or another you don't have this same policy
47:40
anymore or you don't have this. You realize that oh, this data
47:45
actually was incorrect and so we need to change it out.
47:49
How are you going about swapping things?
47:51
Because I've heard that is a real pain in the butt when it
47:55
comes to keeping your vector database up to date.
47:59
Yeah, that is actually a big challenge for sure.
48:05
One of the ways would be to and it's definitely going to be a
48:09
costly process to reindex our database.
48:13
Interesting to create the index again, and as the data
48:17
increases, indexing it becomes harder.
48:19
Yeah, and harder. Yeah, if you only want to change
48:22
one. Yeah, it's like we got to
48:25
reindex. What?
48:29
Yeah. I wonder though, like in that
48:32
case, how would we essentially go about it?
48:38
One of the ways could be inducing a negative example for
48:42
something that we don't want to include in our RAG agents.
48:46
For example, let's say if certain part of the data became
48:49
stale, we can always include a relationship in the documents
48:54
which is like if asked for this specific data, this is a stale
49:00
data, we would not want it to go there.
49:02
So even though in the semantic search it might be picked up,
49:07
the good part would be when it goes to the large language
49:10
model, it is going to reject it immediately because in our
49:13
prompt we are specifying not to actually answer in that
49:15
direction. But that's more of a sanity
49:17
check or a smoke test, you know? And it's also OK to do if
49:21
there's one or two pieces of data.
49:23
But if it's twenty, no, no, you can't.
49:26
You can't do that so easy. Yeah, it would not be as
49:28
scalable, but that's a that's a really good problem.
49:31
I'm just thinking because I heard the example of you have
49:35
your HR handbook and certain policies get updated and then
49:43
what do you do with when someone's going and asking about
49:47
their HR questions, It's getting information from which handbook,
49:55
which policies are getting referenced here.
49:57
So this was back in the rag days and I just remember people
50:00
talking about how hard it was to keep their vector databases tidy
50:06
and up to date. And so I find it interesting for
50:12
you. It's it's almost like maybe
50:15
you're not having to replace as much data.
50:19
So then you don't have that problem.
50:21
You're only creating new data and then you can just use the
50:26
time or the date created filter type thing.
50:29
Yeah, yeah. For most of our use cases that
50:31
we have like worked with, we don't have like a lot of data
50:36
which needs to be either removed or replaced.
50:41
But if it were the case, I would believe that re indexing would
50:45
definitely solve the problem. But with the scale of the data,
50:50
I'm sure like we can always re index like certain parts of the
50:53
vector DB as well by for example just using like 1 document which
50:57
is the part we need to like you know update.
51:00
So those query would be again suspended or projected in that
51:03
vector space, but with different values essentially.
51:08
And any negative examples could help us to not entertain our
51:13
previous policies. But that is definitely a very
51:17
interesting problem, and I think we'd have to look more with
51:20
detail into that. You don't do anything with LLMS
51:24
and recommenders, do you? Are you are your recommender
51:26
systems all very traditional still no, we, we, we do use
51:30
really. Yeah, yeah, yeah.
51:32
How are you using LLM for recommenders?
51:34
So so. For instance, let's say if
51:36
you're, we have some recipe recommendations and I want to
51:38
like, you know, generate a recommendations like based on
51:41
that page. So, and I would want to like
51:43
extract out, you know, like, let's say cooking tools relating
51:46
to certain recipes. Just give you an example.
51:48
And that way we it's way way easier to contextualize it using
51:53
a large language model. But you're just contextualizing
51:57
it and then you're going to the traditional recommender systems
52:01
style or? It's a hybrid, Yeah, hybrid.
52:04
Basically I've heard about this a few times from people.
52:07
It's like you slap a LLM on top of a recommender system just to
52:11
make that part so much easier because you don't have to train
52:16
this model on the whole cooking utensils and everything.
52:22
Yeah. I mean, fortunately for us, you
52:24
know, our users don't scroll Walmart, you know, page as often
52:28
as Instagram. So that definitely helps us to
52:31
have a little bit of latency or like you know, specifically for
52:34
the products that I have worked with, you know, have not as much
52:39
of an online swimming model. But in that case, yeah, like
52:43
latency is like of that most importance right now.
52:46
But like for the use cases that I have worked with and not to
52:50
like delve into too much of details for the company's
52:54
confidentiality. But yeah, so we have like used
52:57
like large language models in essence trying out like, you
53:01
know, like what features can be extract out from the current
53:05
recommendations so that we can use those features to recommend
53:10
next set of things immediately. And at the same time combining
53:17
the user feedback to make sure how they are essentially liking
53:21
and what else they will like in that sense.
53:24
So because when we are essentially taking a natural
53:28
text and trying to get the feedback out of an item or
53:33
essentially a description of an item which user is following set
53:37
of instructions it there are so many products that could be
53:41
relevant. And if one product has captured
53:44
any users attention, right there we have user engagement.
53:47
The top of the funnel is already established.
53:50
Now we have user awareness and we can essentially use the top
53:54
of the funnel again, like a little bit of marketing term
53:58
that we can essentially use that to leverage what else can we
54:02
bring into awareness. As the user awareness increases
54:05
for the products which are related, essentially engagement
54:09
increases, engagement increase leads to potential sales.
54:13
Yeah, and they definitely come back.
54:15
And for example, we were working with how are YouTube banner
54:18
impressions which have led to the sales of the items and there
54:22
is no way for us to relate that. Wow.
54:25
Yeah. And yeah, sorry.
54:26
No, no, no. OK, so, so really you're
54:31
generating the feature or you're extracting the features with the
54:33
LLM? Yeah, yeah.
54:36
And the features, but the features are already in some
54:39
kind of feature store. This is very dynamic because
54:44
what we're focusing on is what user essentially looking at.
54:47
Yeah. And when we find out, let's say
54:49
for example, there's a set of recipe that user is essentially
54:52
looking at what products can be mapped to this recipe that could
54:57
be gotten from the description of the recipe.
55:00
And that is very user specific. Now we know for a fact it's
55:04
going to be a little bit slow for an online serving.
55:08
I mean, the latency is going to be high, but we do have that
55:12
user's data. We do know like what we can
55:14
essentially further diversify the recommendations when the
55:19
user logs in next time, or even at that time I see for that
55:24
matter. So you're using the LLM to
55:27
enrich the recommendations? Yes, I understand.
55:32
So the LLM took me a while but I got here.
55:38
It was, it was a random experiment somehow, you know,
55:40
when it turned out to be, like, pretty fascinating.
55:42
OK. Wow.
55:43
Like, this works, you know? Yeah.
55:45
Yeah, yeah. Oh, I really like that.
55:48
So all of the pages that I'm looking at on Walmart, you can
55:56
extract, especially if it's any type of a blog type of post,
56:00
like a recipe, you can extract everything from that page using
56:06
an LLM and then enrich my customer profile with all that
56:11
data. Yes, for certain products.
56:14
Yeah. And then later you do the
56:16
traditional recommender system, but it's, but you know that
56:19
like, Oh yeah, he did this, he liked this.
56:22
And we have some strong signals that he might need some
56:27
chopsticks because he was looking at a sushi sushi recipe.
56:31
Yeah. And you know, the interesting
56:33
part is it's not even like that latent anymore.
56:36
If you will go right now and you will see, you will have, you
56:39
know, the, let's say, cooking tools that you need to use right
56:44
there. Yeah, it's, it's because it's a
56:46
very lightweight model and we cache a lot of things, which
56:48
helps us to immediately get back and bring it into production
56:52
immediately, You know, So it's, it's definitely, you know,
56:56
fascinating. And to our advantage
57:00
recommendation systems, they work on confirmation of personal
57:04
bias. You know, it's going to tell you
57:06
more things that you want to see at the end of the day.
57:09
And that's why, you know, like sometimes I feel whenever you're
57:12
even scrolling Instagram, you know, like I like like once kind
57:15
of page and it just pushes me in that direction.
57:17
Like, Oh my gosh. OK, wow.
57:19
Dude, I know or I have the hardest time because on my iPad
57:23
I had I can't log into YouTube on my other profile that I
57:29
normally watch YouTube on. I have it on my random like
57:33
profile and the recommendations I get are not that good at all.
57:38
It's from like 3 or 4 years ago when I used to watch things and
57:42
that I that I liked. And so IA little bit of a
57:47
tangent, but it is fascinating to to think about that.
57:51
Oh, OK, so I like it man. I I really like it.
57:54
I'm just wondering what else is there that might be worth
57:58
hitting on? Evals.
58:02
Yeah. Like how do we evaluate the RAG
58:04
agents? I like that.
58:05
All right, yeah, let's let's talk about that.
58:07
All right, we're. In a very niche space of that, I
58:10
would say we're still figuring out one of the most intriguing
58:13
parts of this space is that there's so many components to a
58:17
RAG agent. Where do we, where do we even
58:19
start evaluating? I mean, I'm getting an answer,
58:23
but there are so many steps behind that answer.
58:26
You know, there are retrieval systems, there's indexing
58:29
system. Where do I, you know, start
58:32
evaluating like you know what part is going right and what
58:36
part is going wrong? Yeah, how do you debug?
58:38
Yeah. And this, you know, boils back
58:40
to the initial example that we probably, yeah, we touched base
58:44
upon was it's like, you know, a kid who's having an exam, an
58:49
open book exam. And now what we have, we have
58:52
given a kid the question that, OK, this is the question.
58:55
It goes back to its books, finds out the most relevant context,
58:58
starts giving us the results or answers.
59:01
Now, one place to look for is, is that kid or is a RAG agent
59:07
referring back to the right set of documents or not?
59:10
That's where like the first evaluation starts.
59:13
If it's not, we need to like check into our indexing.
59:16
We need to make sure like are we like defining the documents
59:19
correctly or not? Are we passing the right
59:21
context? If that's done, that's step one.
59:25
The next thing would be once the context is retrieved, which is
59:29
let's say, right, So we can use recall in that, for example, you
59:33
know, like sensitivity, which is a very, I would say cliched, you
59:37
know, evaluation strategies in machine learning.
59:40
Yeah. And then the next would be OK
59:44
once we have that they are the prompts, the dynamic problems
59:48
we've generated, are they somehow relevant or not?
59:51
And the third part would be once the prompts are being passed to
59:55
the large language model, whichever 1 we are using, are
59:58
they actually generating the right set of output or not?
1:00:02
So these three steps essentially we use for evaluations in Rack.
1:00:06
And what are you doing just printing them out and having
1:00:11
people label this? How do you go about this?
1:00:14
And I imagine it's all offline. It's not in.
1:00:17
Yeah, in the moment, OK. Yeah, yeah.
1:00:20
It's all offline, yeah. And it's not like of course not
1:00:24
manual, I mean prototypes, yes, we started to look into each and
1:00:27
everything like debugging like manual yeah.
1:00:30
But as I like mentioned like recall like is the one parameter
1:00:35
that we use at the time of you know, document picking for a
1:00:42
rack. And good recall means like OK,
1:00:45
we are actually picking up the connection relevant documents
1:00:48
over there. And then secondly, for for
1:00:52
example, like how the large language model is performing,
1:00:55
that is essentially evaluation done based upon the hyper
1:00:58
parameters of that large language model, how much of A
1:01:00
temperature we are like setting, what is the value of K that we
1:01:03
are using? Is it producing contextually
1:01:05
relevant results or not? And then we also have LLM
1:01:09
feedback. So these results are fed into a
1:01:12
large language model to get back if they are relevant or not.
1:01:16
So it's a feedback module where the content generated by 1 RAG
1:01:21
agent is being fed into another large language model, which
1:01:24
gives us the evaluation of that. It's still work in progress
1:01:28
right now. Again, we don't have a
1:01:32
state-of-the-art methodologies, we're still getting there, but
1:01:37
these three phases are ones.