Latest / Agentic Conversations (formally mlops.community) / Context Engineering, Context Rot, & Agentic Search with the CEO of Chroma, Jeff Huber

Transcript

Context Engineering, Context Rot, & Agentic Search with the CEO of Chroma, Jeff Huber · Agentic Conversations (formally mlops.community)

0:00
And I think also people like don't want to believe that
0:01
intelligence, AI, intelligence, artificial intelligence is like
0:06
the most insidious distracting information is the information
0:08
that looks relevant, but it's not for some subtle reason.
0:11
This is why sub agents, I think are like a powerful idea is like
0:14
literally it's because context rocked, I think implies the
0:17
existence of like context window encapsulation.
0:20
And that is, by the way, maximizing recall, maximizing
0:22
precision. Maybe like the largest, you
0:23
know, installed base of in Chroma of Chroma in 10 years is
0:26
going to be, you know, inside of like 5 billion robots.
0:37
What's context rot? Are we talking now?
0:40
Yeah. OK.
0:41
I know we're going to be like, hey, everybody, like, you know.
0:44
I I don't get that quiet on it. You guys put out that blog and
0:48
it caught fire. Yeah, yeah.
0:51
I think we're really passionate about helping developers build
0:55
useful stuff with AI and there's a lot of misinformation flying
1:00
around. I think that AI is really hypey.
1:03
That's both because it has a lot of promise, but also that ends
1:06
up meaning there's a lot of also like snake oil and in particular
1:09
like understanding like what's actually working for people and
1:13
what's not actually working for people at times.
1:15
There's asymmetry in that information.
1:19
So we were hearing from a lot of developers that like, you know,
1:22
these sort of million plus token context windows, whilst, you
1:27
know, kind of advertised as being perfect because look, they
1:30
they're very, they're very good on the you go on the haystack
1:31
benchmark. They're clearly very good.
1:34
It seemed like most developers actually, their intuition was
1:37
like, yeah, I don't trust Claude past 40,000 tokens or open AI or
1:41
Gemini or any of these models, right.
1:43
And he doesn't, you don't need to pick on anybody here.
1:45
And like that's interesting. That seems really important.
1:49
We should help builders understand that.
1:52
And so we sort of launched this technical report, the multi
1:55
month investigation. We tested I think 17 plus models
1:59
across a suite of a suite of benchmarks and tasks and tests
2:02
to see like when do models. Stop.
2:06
Working the way that you expect and you know, model behaviour is
2:10
in is not invariant to the length of the context window.
2:12
When you use more context, both its ability to pay attention
2:16
goes down and its ability to reason it goes down.
2:19
And you know, obviously needle on a haystack is like the
2:22
easiest task. You have to pay attention to a
2:24
needle and you have to, if you ever look at like the needle on
2:27
haystack benchmarks, like it's basically all lexical matching.
2:30
There's 18 words in common between the search query and the
2:33
needle. And so it's like a very easy
2:35
task. 0 reasoning power required.
2:38
But of course, most like real world stuff like, oh, you have
2:40
to like connect these 18 things together and then you need to
2:43
like also reason about all that. And so that's what we wanted to
2:46
figure out. And so, yeah, I, I think that
2:48
that the technical report, I think it's like now 140,000
2:52
views on YouTube or something like that.
2:53
And you know, you go to any event and people have heard
2:56
about it. And I think it also was popular
2:58
because it's just sort of like it.
2:59
It ratified things that people were already feeling.
3:02
That's what I was gonna say. It was like, we all knew that
3:05
was happening, but we didn't have any data to point to it.
3:08
Yeah. And so then when you came out
3:10
with that, it's like, yeah, exactly.
3:13
It's also funny that needle in the haystack is the only thing
3:17
we had. It feels like we shouldn't only
3:19
have that. And props to Greg for creating
3:21
that because I love the needle in the haystack.
3:24
But at the end of the day, there should be 20 of these to really
3:28
stress test the context window and figure out, yeah, what is
3:32
going on and how we can take advantage of it to the Max.
3:36
Yes, unclear to me. Like maybe the labs are.
3:38
I think the labs probably already knew this, really had an
3:40
intuition about it. Do they have their own internal
3:42
benchmarks? They're training again so that
3:45
they know about? You know, I think the reality is
3:49
that like when you're marketing something, you invariably have
3:51
to pick the things where you do the best and you kind of ignore
3:53
the things where you don't do the best.
3:54
And of course, if there's something that you don't do well
3:56
on that nobody else measures themself on either publicly,
4:00
then you're definitely not going to talk about that.
4:02
Why would you? And so like, I think also just
4:05
like the how viciously competitive, like the sort of
4:08
state-of-the-art large language model world is today, like that
4:12
creates a negative incentive to being as transparent as possible
4:18
about the models strengths and weaknesses.
4:20
And I think also people like don't want to believe that
4:22
intelligence, AI, intelligence, artificial intelligence is
4:26
spiky. You know, we want to believe
4:27
that it's AGI. We want to believe that
4:29
artificial superintelligence is coming, that these things would
4:31
be better than humans in all ways.
4:33
And like, while that may happen someday, I'm not sure there's
4:35
really strong evidence today to prove that that's the case or
4:38
not the case. Again, it may happen, but like
4:39
today, we don't know. But I think the best way to
4:42
think about these models is that they are very spiky and what
4:45
they are good at and what they're not good at.
4:46
And of course, this is like classic computers as well.
4:49
You know, classic computers are incredibly good at arithmetic,
4:52
much better than humans that arithmetic.
4:54
And of course they're, you know, way worse out of the things.
4:56
And so that that that to me is like the most intellectually
4:59
honest way to approach models and their strengths and we're
5:03
sort of and the capabilities is to be sort of honestly trying to
5:06
pursue, you know, the map of their strengths and.
5:10
Weaknesses. I also want to talk about search
5:12
and just breakdown search because it has been something
5:17
that I, I know you think about a bunch and we can kind of
5:20
contextualize how we had keyword search or we have, we have all
5:25
of these, but keyword search. Then we go semantic search and
5:28
then we start doing hybrid search.
5:29
And then we now are like thinking about search within
5:33
agents. And all of that feels like
5:39
problems that you need to have a strong grasp on if you're
5:43
building with AI. Yeah, maybe like a quick sort of
5:48
addendum to that. Like there's this trend that's
5:50
been thrown around now for a few years, like vector database.
5:53
And we've always thought that that's like a dumb term.
5:55
These are just like VCs trying to like meme a category of
5:58
enterprise software as they do, right?
6:01
But like, that's not actually a the problem to be solved.
6:04
And that's actually also not what developers like really
6:06
actually like want to buy as a result, right.
6:08
These sort of things are very I was correlated problem to be
6:11
solved. People want to buy like they
6:13
want to solve the problem of information retrieval broadly.
6:16
And and of course, dense vector search happens to be 1 useful
6:19
tool in the toolbox. We'll unpack in a second why,
6:22
but certainly not a not a, not a panacea to all of your search
6:27
problems. And we've never claimed that it
6:29
is either. You know, the way that I kind of
6:31
describe what Chroma is then is that it's not a vector database.
6:34
What we're building is modern search infrastructure for AI or
6:37
maybe modern retrieval infrastructure for AI.
6:39
There you go. And like the modern pieces in
6:41
contrast, like legacy search systems where Chroma scales much
6:45
better, is much, much, much, much easier to run.
6:48
It's much more cost effective. And we really been able to put,
6:51
you know, the last 10 years of distributed systems research,
6:55
both on the theoretical side, on the applied side, like into this
6:58
architecture to make it that much better.
7:00
And then the four AI part I think is really interesting to
7:02
unpack because it means many things like it means the AI
7:04
developer. So it's no longer just a search
7:07
engineer who is implementing these systems.
7:09
It's everybody, every single engineer on earth.
7:11
And frankly, we're even seeing like semi technical slash
7:14
non-technical users also starting to like sort of, you
7:16
know, wade into these waters for AI also means AI search and AI
7:22
workloads. AI workloads are different than
7:24
classic search workloads. Classic search workloads you
7:26
have like 1 giant index that everybody's querying and AI
7:30
workloads, you have mini indexes.
7:32
Every team, every workspace, every user will have one or many
7:35
search indexes that they want to query over.
7:38
And so that's like a share of much different shape workload at
7:41
kind of a technical level. And then I think the other
7:43
observation here is that in the past, humans were the ones doing
7:47
the querying and humans were the ones doing the last mile of
7:50
search, which is digesting the search results.
7:52
And so much of search for so long has been about 10 blue
7:55
links because that's all that a human brain really can
7:58
assimilate an absorbent process at a time.
8:01
But now with LLMS, like they are doing a, the majority of the
8:04
querying and they're not just doing one query.
8:06
They could be doing 100 queries. And for every query, they're not
8:09
just processing 10 blue links. They could be processing like
8:11
10s or, you know, hundreds of thousands of pages of documents.
8:15
And so like, that's a much different thing at the end of
8:17
the day, it's. Funny also just to interject
8:20
about how you have the shape of search so different with these
8:26
many teams that are doing different queries on different
8:29
corpuses of data and sometimes it overlaps, sometimes it
8:33
doesn't. I hadn't thought about that.
8:35
And then the next thing of like search now has transformed from
8:40
10 blue links where we as a human interact with it in a way
8:44
that we're looking for that headline.
8:46
Is this relevant to me? We click into it, we kind of
8:49
look, maybe, maybe not. No, just try to process as much
8:54
information as you can and then give it back out and see if we
8:58
can get that like squeeze the juice out of this grapefruit as
9:02
much as possible. Yeah, exactly.
9:04
And I think like the way to reason about this, like pipeline
9:07
then of like, how do you bring information to the model or how
9:09
do you give the model a tool that it can use to go get
9:11
information? I think like a two stage process
9:13
is a good way to think about it. And the first stage is you want
9:16
to maximize recall. You want to get all possible
9:19
relevant information because if you miss one piece of relevant
9:21
information, you could get it wrong, right?
9:23
And so of course, if you're maximizing recall, you're not
9:26
going to be maximizing precision sort of by by definition, right?
9:29
So first, like first stage is maximize recall.
9:31
We'll unpack that in a second. And the second stage is
9:33
maximizing precision because also context rot, you know,
9:36
context windows are not perfect. They're very sensitive to
9:39
distractors as well. And so you want to like then
9:42
basically a massive pool of relevant information maximizing
9:45
recall and then call that pool down to only the most relevant
9:48
bits before you had the model do a kind of a final like reasoning
9:51
pass over it. And so this goes to stage
9:53
pipeline, I think is a helpful way to like think about the
9:55
goals. Because do you see still that as
9:58
you mentioned, it's very prone to distraction?
10:03
Like I've noticed that sometimes you give it more information and
10:07
then that just fucks everything up.
10:09
Yeah, yeah. And we've seen for sure, like
10:11
within, we've done some experiments and kind of agent
10:13
learning and they're sort of intuitively makes sense.
10:15
Like if you give agents access to prior similar situations
10:19
where it did a thing and it was successful, kind of give it a
10:22
few examples of that to dynamically if you shot
10:24
prompting when it comes to a similar situation again, Oh,
10:27
this would be really, this would be helpful.
10:29
I, I, the human would find that helpful probably.
10:32
And what we find is the models just tend to sort of like slide
10:36
straight into that local minima and they're like, Oh my gosh,
10:38
thank you, you already gave me the answer.
10:40
I don't think about this anymore.
10:42
And like, the most insidious, distracting information is the
10:45
information that looks relevant, but it's not for some subtle
10:47
reason. And so yeah, this is definitely
10:50
a hard. Problem wow OK and that just a
10:53
bit of a tangent goes into this whole thing on memory of task
10:58
completion and how you can help the agent complete task that
11:03
it's already done before or that it has kind of done before.
11:06
And so you're like, hey, you've, you've done this before,
11:08
remember how you did it? But if it is a slightly
11:11
different and it looks the same, then you send the agent off and
11:16
it doesn't do it. And you're like, why?
11:18
You were so close? You had everything you needed.
11:21
Yeah. Yeah, it's definitely not a
11:22
solved problem. I think that, you know, kind of
11:25
the the, the data pipelines of retrieval are a part of the
11:28
solution of what people want to call memory, which I think is
11:30
like context engineering in some sense, right?
11:32
There's sort of a right path. How do you glean insights from
11:36
prior context and how do you store those?
11:38
And then there's like a read path, query path.
11:40
How do you use those prior insights in the next generation
11:43
by the continual, the same trace or for historical traces?
11:47
But it's yeah, it's still then solve a problem.
11:49
I think the data piece is certainly an important part of
11:51
it, but not also again, the panacea.
11:52
Like there probably needs to be some like learned representation
11:55
to really be the level of expressive that we want these
11:58
things to be able to be. And I think the goal really is
12:01
like to be able to communicate tacit knowledge to models and
12:03
models like themselves and lean tacit tacit knowledge as they're
12:06
doing things and kind of like build up a library of skills.
12:09
And I think the library of skills again can both exist in
12:11
documents loaded into a search index, but also probably need to
12:15
be some way encoded into like the weights of something so.
12:18
All right, so I got you off track with precision and recall.
12:21
Yeah. And you're talking about how
12:23
basically you want to maximize both of these.
12:25
First it's recall, then it's precision, and then.
12:28
Yeah. So maximizing recall, I think
12:30
it's worth unpacking that because you asked a question
12:32
about, you know, dense vector search, text search, lexical
12:35
search, hybrid. What is all this?
12:37
So many terms that's true. Text search this is.
12:40
So much misinformation on the Internet about this.
12:42
And you know, I'll, I'll make the point, you know, unless you
12:48
have data, it's just your just an opinion, you know, and
12:51
there's just so many opinions flying around and there's like,
12:53
it's just so memetic too, you know, like, Oh well, like so and
12:55
so does this, therefore I should do that.
12:57
And you're like, have you even thought for a single moment
13:00
about either why they do that? Or their right, their scale.
13:04
I don't know if you remember back in the day, we used to
13:06
always say in the community, just like when people would come
13:10
in and be like, I'm going to build this gigantic ML OPS
13:14
platform. I need these tools and this
13:17
because, and then we're going to throw it all on Kubernetes And
13:19
it goes to the meme of like, I have 12 concurrent users.
13:23
I guess I'm ready for Kubernetes.
13:26
And then we would be sharing the same article, which is you are
13:32
not Google. It's like that classic blog of
13:35
like, don't trick yourself into thinking that you need something
13:39
that Google needs because Google scale is a whole different
13:42
level. Their company, their business is
13:44
a whole different level. And so back to this point of
13:46
like, you need to really like kind of be discerning on what
13:50
you see and why they're doing it that way just because they do
13:53
it. Take inspiration, but don't
13:55
think that it's the Bible. Exactly.
13:57
And so unpacking this like goal is first stage or first step of
14:01
maximizing recall. Like how did how to think about
14:03
that? Well, you should think about it.
14:05
First of all, you know, you shouldn't just copy and paste
14:07
what somebody else has done. You shouldn't spend the 5
14:09
minutes to think about it. And you know, for some
14:11
applications going to be, you know, which information, which
14:13
files are currently open. For other applications, it's
14:17
going to be which files or information were recently
14:18
touched or recently created, right?
14:20
Recently can be important for other, other situations.
14:24
You know, it could be semantic similarity, it could be texts
14:27
sort of matching, lexical matching.
14:30
And I think a good way to reason about kind of the strengths and
14:32
weaknesses of like dense and dense and sparse, dense and
14:35
lexical search is dense is useful when you don't know the
14:42
language of the corpus. So maybe I'll say that the other
14:47
way, full text search is very good if you know the words to
14:49
search for. And actually, in most enterprise
14:51
search workloads, you'll see lexical full text search as
14:55
being the primarily useful tool, because I'm looking for a
14:58
document that I made in the past I kind of already know.
15:01
I remember where it is, I can't. Remember where it is, but I kind
15:03
of know what I named it, you know, I know what words are in
15:06
it. And like it's essentially like
15:07
not that hard to find document using lexical search.
15:12
Semantic search becomes really helpful when you need that
15:13
semantic expansion. You need that term semantic
15:17
matching. I don't remember even what the
15:19
words were, but I need to go and find it.
15:21
Yeah, it's around this idea. Exactly, Exactly.
15:24
And so these tools are again, not, I think most applications,
15:29
it's not an either or. They're complementary.
15:31
Now it might be 9010 or 1090, depending on the use case,
15:34
right? Like which of those are the most
15:35
helpful? But I think it's like almost
15:36
like never an either or. And of course that's what people
15:38
call like hybrid search, which I think it's like a dumb phrase
15:40
because like nobody knows what it means and it means 10
15:43
different things. And so we should just like not
15:45
call it that, you know, it should just be, Oh yeah, I'm
15:48
using dense vector search and lexical search.
15:49
Just say that. Don't call it hyper.
15:51
So it's a mouthful though. It is a mouthful, but at least
15:54
it's at least people don't be talking about.
15:55
Yeah, yeah. So I think that going back to
15:58
like the the dense and sparse and electrical search.
16:00
I think also there's this recent, I will this recent paper
16:04
from Google on like the limits of single dense embeddings.
16:08
Great, mathematically provable. I think every kind of already
16:11
knew it. But again, it's this like good
16:15
it put, you know, math around which we already kind of like
16:18
knew and proved it, which is great.
16:20
You know, dense embeddings have their limit.
16:21
And of course, the entire Internet, there's like this
16:23
graph that shows like BM25 search has this like perfect
16:27
flat line and then like all these dense vectors approaches
16:31
like failing and like much, much lower kind of quality.
16:36
And again, that's fine, that's not wrong.
16:39
But if you go and look at if you go and look at the actual like
16:41
data for that they used for that experiment and you look at the
16:45
queries, they are all lexical searches.
16:49
And so of course BM 25 is going to freaking crush it at that
16:53
because like BM25 is, you know, full text search terms term
16:57
frequency based. And so like, of course, you're
16:59
going to do very good at that task.
17:01
And of course, again, the the nuances is important and
17:04
helpful, but like what the mean that went to the Internet was
17:06
that like, Oh my gosh, all the 10 Spectre stuff is BS.
17:10
It's useless. You just can be in 25.
17:12
And like maybe that's true for your use case, but like maybe
17:14
it's not. And again, I'm just really now
17:17
ranting about what people aren't thinking enough, so I'll stop
17:20
there. Yeah, I I want to bring in one
17:23
other piece which I think is important with search, which got
17:26
brought up a few weeks ago on the podcast with my buddy Nishi.
17:29
And he was saying how for him, a lot of his search woes and
17:34
problems come in when someone searches for romantic dinner.
17:40
And it's like semantic search kind of can do that.
17:46
The idea is there, but it's not quite there.
17:49
And he was having the hardest time because what was happening
17:53
is you would come in and and be like, all right, well, romantic
17:57
for you means what you need a bit of personalization.
18:01
So if you're talking to the LLM and you're saying I want to plan
18:04
me a romantic dinner with my wife, all right, but romantic is
18:08
for you is one thing. For me it's another thing.
18:11
And and then he was saying you can just really have problems on
18:18
how we have UX, the new type of experience when you're trying to
18:24
bring like old stuff into the new UX that we have where people
18:28
can ask anything and say anything.
18:31
And I wonder if you've seen that where like, OK, these, these two
18:35
types of searches are, are not breaking down completely, but
18:40
you don't have them there. And another example that he
18:43
gave, which I'll think about too, is how where semantic
18:47
search didn't really work is if you are a vegetarian and you say
18:52
I want vegetarian pizza, it's not normally going to give you
18:57
cheese pizza as an option because that's not tagged as
19:00
vegetarian, you know, so and even in like in vector space,
19:05
like vegetarian pizza. Yeah, yeah, yeah.
19:08
Cheese pizza is way over here. Totally totally totally.
19:11
Or even more complicated, if they say, plan me a romantic
19:13
dinner tonight, get to know like which.
19:15
This is a vegetarian. Open which ones are close to you
19:18
and you're yeah and you're vegetarian and like, you know it
19:20
becomes a pretty complicated query.
19:21
I think it is something that like an LLM can internally
19:25
process like fairly well like it doesn't have to be all rigid
19:29
explicit system. But I think you know the rating
19:33
dinner piece like I think it's possible to get that out of
19:35
vector space because kind of how I would approach that like a
19:39
little Sam Yelp right and I want to support like romantic dinner.
19:42
I'm going to take all the reviews and then I'm going to
19:46
have an L have an LLM generate maybe like 20 to 30 like tags
19:51
like for every review. And then those 23 tags can
19:55
become like clusters. So I understand now like which
19:59
clusters of ideas are like commonly associated with this
20:01
establishment. And then I can associate like
20:05
either the centroid of those clusters or maybe all the
20:07
information, those clusters with that business.
20:09
And then now when I search romantic dinner and if somebody
20:12
else said like whatever a synonym to romantic is cozy and.
20:16
Intimate evening meal, you know, like actually you might be able
20:21
to get that from the reviews. So I don't know that like that
20:24
is impossible to get out of like vector space.
20:25
But obviously that level of forethought that you want to
20:28
support that kind of a query and then the level of then work
20:31
required to kind of make that query possible again requires
20:36
you to think it's not just all for free.
20:38
So yeah, you can't just slap semantic search on it and be
20:41
like, we're good now, Yeah, type whatever you want.
20:43
I mean you. Can, but like, your competitor
20:45
is doing more work and therefore your competitor's product is
20:47
better. Yeah, and that means they're
20:49
gonna win. Yeah, the experience is gonna be
20:50
much better. Exactly.
20:51
Exactly. Yeah, I tend to these days when
20:54
I look for coffee establishments because I like V60 and I like
20:58
the, you know, the specialty coffee.
20:59
Yeah. I don't type in Cafe because
21:03
that can give me anything from Starbucks to Dunkin' Donuts, you
21:07
know? Yep.
21:08
And what I'll do is I try to search like V60 as a term and
21:13
you have the cafes that come up and they'll they'll show.
21:17
But this is like super easy because it's like a review has
21:21
V60 in it, right? But sometimes I'll notice that
21:24
the review doesn't have V60. It's just like, oh, this is a
21:26
specialty coffee place or this is a coffee roasters.
21:30
And so there I'm like, hey, this is interesting search issue.
21:34
Totally. Totally.
21:35
And that shows how you've adapted to the strengths and
21:38
weaknesses of the system. Like, you know, I think one of
21:40
the interesting UX challenges and opportunities now with LMS,
21:44
because we all became very good at Googling and like, how do I
21:49
compose this question I have into a set of terms and minus
21:53
signs. And it was.
21:54
Friction then. Like we all learned how to
21:57
search, kind of, but within the limits of Google's ontology.
22:02
And there was professional search googlers right I?
22:05
Mean, you know, people that are way better people that are way
22:06
worse type PDF, you know, like the whole shebang.
22:10
And like now LLMS, it's just a sort of it's viewed as more like
22:14
a magical box. And that's a good thing because
22:16
users don't have to like do the mental, you know, gymnastics of
22:19
how do I change the question that I have into this particular
22:24
kind of domain syntax that Google was going to accept Like
22:27
that's actually a better user experience.
22:30
But of course, that means that users are going to ask a lot
22:32
more questions and a lot wider variance.
22:35
And, you know, in some ways I think that's the cool part about
22:39
that from like a user research perspective is like, now
22:42
applications do have a larger surface area of like, how they
22:45
can help their users. Yeah.
22:46
And of course, that's a blessing and a curse, right?
22:47
The blessing is like, you can do a lot more.
22:49
It's pretty, it's pretty cool. The curse is that like, you have
22:52
to do a lot more. And like you know, again, it's
22:53
not all for free. Yeah.
22:54
And you're, you're now supporting all these different
22:57
use cases and all these different ways that users are
22:59
doing it. And if one of them is kind of a
23:03
shitty experience, you're struggling.
23:06
I as a user, am not going to look at this new AI feature that
23:10
you created as something that is worth my time.
23:13
And actually, it kind of brings me back to context rot in a way,
23:19
because if now we're talking to these AI features so much and
23:23
we're continuously going back and forth with them, you put
23:27
yourself in the position to have that context rot, right?
23:30
But you have to almost be like a janitor of the context and
23:35
really get good at cleaning it. And one thing that I saw last
23:40
night from a guy at the event, he was like, I'm trying to
23:43
create like a Jupiter notebook style thing for the context
23:47
window. And I want to be able to like
23:49
turn off different pieces of my context because it's not needed,
23:53
because maybe there's sometimes that I'll just ask something and
23:56
then it's there in the context and it's always referencing.
24:00
And so now it, it made me think like, oh man, like context kind
24:06
of etiquette or not etiquette, but context like cleanliness is
24:10
a thing in a way. Yeah, 100%.
24:12
You both see it in like multi turn kind of user model
24:16
conversations, ChatGPT, etcetera.
24:18
You also see it inside of like a gentic loops, where like there's
24:22
an LM running in a loop and it's, you know, getting some
24:25
context generates some output and then it needs to generate,
24:28
then it needs to receive that output into itself and then
24:32
generate, you know, more information.
24:34
And I think your intuition is exactly right, which is like
24:38
this eventually breaks down. You know, you've seen stuff like
24:40
inside of Cloud Code, there's the slash compact command, which
24:43
does like a summarization and tries to like kind of clean it
24:46
up. I think I saw something from the
24:48
Cloud Code team on Twitter recently where like they don't
24:51
use compact, they actually just they just dump it.
24:54
They just clear the entirely. They clear it.
24:56
Yeah, yeah, yeah. And actually we've done some
24:58
research that has like shown that that sort of verified that
25:01
result that like actually naive summarization of the context
25:05
history is like no better than just starting from scratch.
25:09
Yeah. And of course we start from
25:10
scratch is also sort of, you know, cheaper and faster because
25:12
you're like flushing all that that baggage.
25:14
So this sort of begs the question though, like how should
25:17
we think about prior state being available to sort of the current
25:22
iteration forgetting whether multi turn user conversation or
25:26
in a gentic loop. And I think one way to think
25:29
about this is like disturbing compaction context compaction or
25:33
context under distillation or you know, some, some word there.
25:36
And it is, it is not a solved problem.
25:40
I don't think it would be a solved problem, but there's a
25:43
few interesting ideas. You know, 1 is that you use a
25:45
prompt of some kind to help the model isolate the relevant
25:49
actually like relevant to high signal pieces and then pass
25:52
those down to the next step. Because there's a lot of like
25:54
craft, especially like coding agents, you just see these like,
25:56
you know, huge dumps of logs and these huge dumps of test results
25:59
and like they can't pay attention to all that.
26:01
And so like giving you a model the ability to like kind of snip
26:04
out the pieces that are actually important for the next turn of
26:07
the model. A detailed prompt, I think is a,
26:09
is a is more helpful than a, a generic prompt.
26:11
And then the other thing that I'll say is that like another
26:14
like interesting approach to this I've seen some people take
26:16
is that they give the the next sort of turn of the loop,
26:20
whether it be the next model or the agent, the ability to search
26:25
in the history of that actual individual conversation or trace
26:28
or whatever. And you know, these things are
26:31
not also mutually exclusive. Can you can do both, obviously.
26:35
But yeah, it's not not easy. How are you seeing the search
26:37
done with that in within the context or within the model
26:41
doing that? Because that does seem like the
26:45
obvious step to take, right? Like, hey, if we just make the
26:49
search the big piece here, yeah, it's kind of like the theme of
26:53
what we've been talking about this whole time, like how
26:55
important search is and how not solved yet it is, or still like
26:59
there's still a lot to be done in the new paradigm.
27:02
Yeah, I see it the way that most people are doing this right now,
27:06
the predominant strategy we'll use you call it search, but it's
27:10
it is kind of search, but it's also maybe not search is they'll
27:13
just use like a file and the file will have like the list of
27:17
to do's, right? Like I want to put this feature
27:20
I need to do like 15 things in this order and then each like
27:25
check box gets encapsulated by one kind of agent loop agent
27:31
trace which minimizes or helps cut down on like the the baggage
27:36
of all the context from all the different.
27:38
Steps. It's like the sub agent idea.
27:40
The sub. Agent idea.
27:40
Exactly. This is why sub agents, I think
27:42
are like a powerful idea is like literally it's because context
27:45
rocked I think implies the existence of like context window
27:49
encapsulation. That's why a sub agent is like a
27:51
good idea in many, in many cases.
27:52
Yeah, of course there are challenges with sub agents,
27:54
which all of a sudden sub agent has to be able to report up to
27:57
the orchestrator agent the right compaction information as well.
28:00
It's like. I got your next paper man.
28:02
Like how many sub agents until you get sub agent rot?
28:08
We've looked at this a little bit.
28:10
We've done some experiments with a deep research agents, which
28:14
are this shape of thing a little bit different.
28:17
You're not a checklist that you're running down, but it is
28:19
the orchestrator says, OK, the user asked this question.
28:21
I wanna look in these like 85 places.
28:23
So I'm gonna have like 85 sub agents go out and do a bunch of
28:25
research. But obviously there's some
28:28
agents can't bring back all the research that they find.
28:31
They have to compact before they boil it up to the orchestrator,
28:34
right. And I, I think, and I mean, you
28:39
know, state-of-the-art today maybe is like the deep research
28:41
implementations inside of Claude, you know, Penny,
28:44
etcetera. And I don't know if you've used
28:46
those with like topics that you know about.
28:48
But you're usually like, this is very good.
28:54
Exactly. Yeah, you're like kind of I
28:57
guess, but. Yeah, I saw, I saw a great tweet
28:59
that was like, I used deep research the other day for
29:02
something that I know a lot about and it wasn't very good,
29:04
so I'll just use it for everything else instead.
29:07
Yeah. Which is like hits the point,
29:09
hits the nail on the head, right?
29:11
It's supposed to go away to footage but.
29:13
You know what I think is also something that never happens is
29:16
when you have these sub agents and they go and do their
29:18
research and then they come back, yeah, they're never going
29:21
to tell you I didn't find shit, it wasn't relevant.
29:24
Like don't use me as a sub agent.
29:26
That is not one of the out of all the possible outcomes, I can
29:30
guarantee you that's not one of them.
29:32
Broadly speaking, you know, the ability for models to know what
29:35
they don't know still seems like unsolved.
29:38
And of course, you can put into the prompt, like, if you don't
29:40
know, please reply, I don't know.
29:43
And then that can work. And this makes some models are
29:45
more cautious than others. Like Claude is like much more
29:47
cautious about like claiming to know things versus like opening
29:50
eyes, like willing to take more leaps, for example.
29:53
And again, that's not always, it depends on the context whether
29:56
that's a strength or a weakness. But I think, you know, probably
29:59
the conservativeness of Claude is a reason that developers tend
30:06
to really like clot so. Because it doesn't blow up your
30:09
database, doesn't just go delete shit.
30:12
I mean I I'm sure there's examples of it leading shit so.
30:15
So then I feel like I, I got you off track again when we were
30:18
talking about what were we just talking about with the So you're
30:20
spawning these sub agents and they're coming back with their
30:24
information. It's, I've heard it put as like
30:27
an accordion where you can look at an accordion and it goes out
30:31
and it gets all this air and it gets all of this stuff and then
30:35
it comes back in and you like condense it down to these like 5
30:38
slits. Yeah, yeah.
30:40
And that is, by the way, maximizing recall maximizing.
30:42
Precision. I like that.
30:43
That's a great visual of all right, we're just going to get
30:47
everything and then we're going to figure out what's important
30:49
exactly. So you recently put out Android
30:53
Chroma, or what is it exactly? Elastic released Chroma Swift.
30:57
Into beta and then imminently will chroma Android.
31:01
Is it somebody that uses chroma on their phone or is it that now
31:07
you have a chroma instance on the edge like give me the break
31:10
device? I think the intuition here is
31:12
that intelligence will be everywhere, certainly in the
31:17
cloud or everybody is increasingly on devices that you
31:21
own, your laptop, your phone, and inside of every device as we
31:27
know it. In the future, maybe we'll have,
31:29
you know, devices that were previously inanimate to become
31:33
animated. You know, we could talk to the
31:34
plant behind me or something. I don't know.
31:36
Or the, the cameras. I'm sure there's a world where
31:39
we don't have to hit autofocus and we just say, all right,
31:43
folks, that's up. Yeah.
31:43
That would have been great this spring.
31:46
And then within that, right, there are trade-offs.
31:52
There are pros and cons to doing search and retrieval locally on
31:57
whatever device that is versus the cloud.
32:00
And they don't have to be mutual exclusive.
32:02
You can also do both And actually in most use cases that
32:05
we see people doing stuff on the phone and that also having sort
32:08
of a cloud syncing story going on.
32:11
But of course local, it's privacy preserving.
32:14
So you can give the user those guarantees.
32:18
Local is also going to be, if so, facto faster because you're
32:24
not going over the network, and then local is also going to be.
32:31
You don't have to worry about like.
32:33
You can be in a place where there's no reception.
32:36
Right. So, yeah, offline connectivity
32:38
and then also, sorry, cheaper as well.
32:41
If you're doing the compute on the device you already own,
32:43
you've already paid the money for that compute and you can
32:45
just now use that compute as much as you want.
32:48
And so, you know, those are very good reasons to do things.
32:51
I think probably the privacy preserving reason is the biggest
32:53
use case that I, I feel like exists there.
32:56
It's just like really great to be able to like have users do
32:59
things on their own device and not have to worry about like the
33:02
data being egressed unless they explicitly want it to be.
33:05
So yeah, but I think there's a lot of really exciting stuff
33:08
there. I mean, like we're a little
33:09
early still in like the arc of this, but I think also like the
33:12
reason that Chrome is written in Rust and like, because Chrome is
33:15
written in Rust, it can run anywhere.
33:18
Like we also there's some kind of early early work in like
33:23
getting Chroma to run on like robots.
33:26
Nice. Because like they're also gonna
33:28
need their own like level of memory and like, so, you know,
33:31
maybe like the largest, you know, install base of in Chroma
33:34
of Chroma in 10 years is going to be, you know, inside of like
33:38
5 billion robots. You could think about it that
33:40
way. So, and what was it, what was
33:42
the engineering feat that had to happen?
33:43
Did you to make it super lightweight so that people could
33:48
just grab it and have it on their device without any worries
33:53
that this is going to brick my phone?
33:57
Yeah, I mean, Rust gives you a lot of safety and guarantees,
33:59
which is useful. We have now a single node
34:06
version of Chroma which you can run as a server.
34:10
We also have a fully distributed version of Chroma, which you can
34:14
run across many nodes in a cloud environment that's also
34:18
serverless. And then that single node
34:22
version can also be run as a library embedded into any given
34:27
context. So today people very frequently
34:30
use Chroma and Python with just like Python bindings to all of
34:34
the underlying Rust. And they'll use that inside of
34:37
Jupiter notebooks or inside of Python scripts or demos or all
34:40
kinds of other, you know, context.
34:42
And it's like really useful to not have to like have a server
34:45
if you're just doing something lighter weight.
34:48
But of course, when you're ready to go to a server, the API is
34:50
the same and you just change how you connect to it.
34:52
And then when you're ready to scale to like, you know,
34:54
petabytes of data in the cloud, you get, you just change how you
34:57
connect to it. And so like, I think that has
34:59
always been our goal. It's aspirational.
35:01
It's not easy to do, but we wanted to make sure that like
35:06
wherever you have a search or retrieval workload, like Chroma
35:09
is there to serve you. I like that.
35:12
And you don't have to like pick and choose, do I want this or
35:15
that? The answer is you can have both
35:17
and. It's the same API.
35:18
What made you think this was necessary?
35:22
Were you like hearing people ask you for it?
35:25
And they were like, I'm building this app, but I need something
35:28
super small. I need to have it on device.
35:31
Yeah. I.
35:31
Mean not, not everybody I guess, but the ideal version of running
35:34
a company is you do have a vision of what you're building.
35:37
Pulling the visionary card on me.
35:38
I'm not going to actually, I'm going to walk you back like
35:40
immediately, but hear me out. Shit.
35:43
Do you have a vision of what you're building?
35:45
It's 10 years in the future, but the reality is that in business
35:47
you get 0 points for being right. 10 years from.
35:50
Now, yeah, that's true. Does not matter, does not matter
35:52
at all. What you have to do is be right
35:56
six months from now and ultimately you really have to
36:01
be. This is this is cliche to be
36:03
clear, but like you have to be obsessed with like your
36:06
customers and what they are ready for what they need and
36:08
want. So there's no reason that we
36:10
couldn't done Chroma on mobile a long time ago, but nobody cared
36:16
until very recently, cause in the last few months we've
36:19
started to see more people really want that.
36:23
But why do you think is there apps that are being what apps
36:25
are being built? It's, I mean, it's a lot of AI
36:28
chat stuff. It's a lot of like media you
36:31
have in your phone that you want to like, you know, search and
36:33
retrieve over, you know, I can't, I can't claim predict,
36:38
like why the exact timing of these things happen.
36:42
You know, similarly, we're we're chatting off camera before we
36:44
got rolling here is like, you know, accuracy we've always
36:48
thought is incredibly important. You can only manage what you
36:50
measure. Like, why would you not want to
36:52
know how this thing is in the real world?
36:55
And then nobody has really seemed to care that much.
36:59
Initiative developers and engineers have always cared,
37:01
obviously, but like the masses haven't.
37:04
And so we've sort of like held off like doing much work in that
37:09
domain, particularly kind of the most of it is like not research,
37:12
most of it is sort of like product, right.
37:13
We've kind of like held off for the reason that like not we
37:16
didn't think it was important, but that like the, our, our
37:19
customer base didn't yet believe it was important.
37:22
There wasn't an asks from the community like hey we really
37:26
need to get this accuracy thing solved.
37:28
I mean, some, yeah, exactly. Sometimes people say they want a
37:31
faster horse and they actually want as a car, you know, but
37:35
they maybe that's true, true. But like when they say they want
37:38
a faster horse, you know, you, you sort of can't jump all the
37:41
way to selling them a supersonic aircraft, you know, And so like,
37:44
I think just like being very sensitive to like the timing is,
37:48
is kind of an underrated, underrated skill.
37:50
And I mean, you can use the word like taste.
37:52
I think taste or maybe another history, the Lindy version of
37:55
the word taste is just wisdom. You know, like, and it's it's
37:59
hard, right, because like on one hand again, you you have a
38:01
vision of what you want to build and you can just stay very true
38:04
to that vision and believe in it incredibly deeply.
38:07
At the same time. You kind of need to be very I
38:12
don't either reactive, but you need to hold that loosely,
38:15
especially your near term plans. And it's like, you know, no, no
38:18
plan has survived the first minute of battle.
38:20
You know, like that's like that is absolutely the case.
38:23
Yeah, that's such a great point because you get all these
38:28
inputs, I imagine everyday of people that are asking for
38:32
things and ideas that you're seeing out there.
38:35
And then you think, wow, we're going to do it like this.
38:38
And when you bring it to the masses, then it's like, actually
38:43
we might need to change it how it's done like this.
38:45
But it's through those iterations that things happen.
38:48
This is why I tell myself that, you know, the time that I spent
38:52
online, you know, is like a good use of my time.
38:55
Your Twitter time taps. Taps me into like the the the
38:58
public consciousness, if you will.
39:00
So this is very visionary on the basically chroma running
39:04
anywhere at any scale. What else are you thinking about
39:08
as far as like the vision of how things are going to change?
39:12
Is it on the precision side? Are you going to start building
39:16
now for precision because it feels like people are interested
39:19
in it? I think it's a very good
39:21
intuition. We don't have kind of the
39:24
specifics either planned out or kind of ready to share all the
39:27
specifics ahead of what we're thinking about.
39:29
But I'll share like a high level observation, which is the reason
39:32
that Chroma started is I had worked in applied machine
39:37
learning and developer tools for 10 years and building technology
39:43
with machine learning. Now we call it AI.
39:45
Thankfully, you've dropped calling it Gen.
39:46
AI. I'm very happy about that.
39:48
Oh, didn't you? I still see some shit.
39:50
Somebody the other day was like Gen.
39:52
AI OPS and I'm like, oh really? That's that's the term.
39:57
Just listen. That's not a term.
39:59
Yeah. We're just gonna say it right
40:01
now. That's not no, sorry.
40:03
I digress. No, no, yeah, yeah.
40:05
The goal of Chroma, going back to that experience, and
40:09
honestly, the pain that I felt building stuff with applied
40:13
machine learning, a lot of it is in the computer vision domain,
40:16
is this is really powerful. It could change the world, but
40:21
it feels a lot more like alchemy than it does like engineering.
40:25
And I'm sure you've seen the XKCD comic where the person's
40:28
like, you know, oh, this is your machine learning pipeline.
40:31
And he's the guy standing on a pile of garbage.
40:34
And he's like, the follow up question is like, well, what if
40:36
it doesn't work? And the guy in the garbage is
40:37
like, oh, you just like, you know, you mix it.
40:40
Yeah, mix it up and try it again, you know, And that's
40:43
like, Santa's still the case, right?
40:45
That's like, actually is still kind of the case today.
40:47
Yeah, that's true. And change some Proms.
40:49
We turn the temperature, we try and figure out the recall or
40:53
whatever. Yeah, yeah, that's so true.
40:55
And and like, I think, you know, a lot of the labs, maybe because
40:59
of their sort of fundraising requirements, have to sort of
41:01
market this idea that like we're, we're building God.
41:06
Like we are literally building a Deus Ex machina.
41:09
We're building a God will emerge from the machine and like, it
41:13
will solve all of our problems. And like we don't have to pull
41:16
that sort of party line because we don't care frankly about
41:21
that. And I think it's wrong.
41:22
And so our term has always been AI is useful and we have like
41:27
stickers, I'll give you secure latest.
41:28
It's like, you know, AI is useful.
41:30
And of course it's only useful if you give it the right
41:33
information and the right context.
41:35
And it's only really useful if it can learn from that
41:40
information too. And so I think that's the stuff
41:42
that like we are really excited about and really seems to be
41:47
where the people on the bleeding edge of agents are kind of
41:50
running into the edge of what current capabilities is.
41:55
We have these large state-of-the-art reasoners.
41:57
They're powerful in many ways. They have their own weaknesses.
41:59
Obviously we should kind of work around them.
42:01
We have also, you know, these retrieval systems, search
42:04
systems, they are very good what they do as well.
42:08
But it kind of feels like the maps not complete yet because
42:12
sort of again, by just definition, we're not getting
42:14
the capabilities that we want. And then again, the capability
42:17
that I want is like, you know, if if I pick up a cup one time
42:22
and I put it back down, like I know how to do it again, I don't
42:25
have to like, you know, think for 45 seconds for how to do
42:28
that. And that ability to kind of like
42:30
learn from experience is something that would be
42:33
incredibly useful for making useful stuff because the real
42:35
world is fractally complex and the real world has so much, so
42:42
many edge cases. And I think for people that
42:43
worked machine learning a long time, you know, the, the phrase
42:47
getting mugged by reality, you know, will resonate quite
42:50
deeply. And, and so we need a systems
42:53
that can, can, can learn to adjust edge cases.
42:55
Otherwise, you know, we're just kind of doing what we've done,
42:59
what we did in computer vision and autonomy in the first era,
43:01
or we're just like putting on like 100 band aids on top and
43:05
around this thing to try to make it work.
43:07
Well, that's why it became so hard to put any ML model into
43:12
production because it's, it's exactly that like, and there's,
43:15
if you're in the enterprise, there's just so much alignment
43:18
that needs to happen in so many meetings that you need to have
43:22
cleared because it is a lot more risky than software.
43:27
It's not running the way that you think it's going to run
43:30
every single time. And so you have to be OK with
43:33
that for your use cases, right? You probably remember this
43:36
phrase from I think it was maybe 2017, maybe it was 26/20/18.
43:41
It was like inside. I think it was either the title
43:43
of the Google Papers inside of Google Paper that was Machine
43:46
learning is the high Interest Credit Card of Technical debt.
43:49
I still quote that we had de Scully on here back in the day,
43:52
and it was like legendary. It was one of the first people
43:56
that I was interviewing. And I was like, oh, how am I on
43:59
this call right now? Yeah.
44:00
And it is that. And you know, what's funny is
44:04
there's that new way of looking at it where you remember the
44:08
diagram in that. Oh, yeah.
44:10
Where the model was a small piece and then everything around
44:12
it. Yes, it's the same thing.
44:15
It's just that now the stuff, the boxes around it have
44:19
changed. Yep.
44:20
But it's that's where the hard part is.
44:23
And it's a little bit easier the because now we can hit an API
44:26
for the model, we don't have to create the model, but still it's
44:30
the same idea. Yeah, yeah, yeah.
44:33
What's old is new again, Yeah. Did you get a professional to
44:45
design this? TJ is a professional.
44:47
So no, I mean like the the office.
44:50
Oh, the office, yes, we also had a professional do that, but not
44:53
something on our team, obviously.