Latest / Agentic Conversations (formally mlops.community) / Making Your Company LLM-native // Francisco Ingham // #266

Transcript

Making Your Company LLM-native // Francisco Ingham // #266 · Agentic Conversations (formally mlops.community)

0:00
My name is Francisco Ingham, I'm the founder of Pampa Labs and I
0:04
like my coffee with milk and with sugar and warm, not not too
0:11
hot, not too cold. Welcome back to another
0:14
Envelopes community podcast. I'm your host, Dimitri OS.
0:19
Today we talked about what it means to be LLL native.
0:25
We got into it with Fran, all about this very topic and I feel
0:29
like there are a few tangents we went on, but we kept coming back
0:33
to why it is so important. Let's get into it.
0:37
But before we do, we had a listener shout out to Sasha who
0:43
sent me a pretty cool song that we're going to play for your
0:48
recommender system out there. This song is called Feed the
0:53
Monk by Chris Grey and the Blues band Sasha wanted to make it
1:00
clear, no, we're not Feed the monkey.
1:03
Let's go. just to feed.
2:05
The monk. All right, folks, feed the
2:08
monkey. Let's get into this conversation
2:10
with Francisco. And as always, if you enjoy this
2:14
podcast, just share it with one friend.
2:17
That's all I ask. For those that are watching on
2:27
camera, I am looking pasty white because I put on zinc sunscreen
2:31
now that I am under the sun. And that means it doesn't come
2:35
off that easy as I just found when I tried to get ready for
2:37
the podcast. I'm joined here by my man
2:40
Francisco, who is a self-proclaimed literature geek.
2:45
And since you're living in Argentina, as they like to call
2:50
it, I want to ask you, have you been reading much Gabriel Garcia
2:56
Marquez? I would say I've been reading
3:00
more The Water Kiss van Galius. Yeah, he's kind of the
3:06
representative of of Argentinian literature.
3:09
But I really also enjoy like just international good, good
3:14
writers like English, Russian, German.
3:18
So yeah. But yeah.
3:20
Wait, So what got you would do that?
3:22
It's one of those things, you know, like, kind of like a soul
3:26
match where you, it just, it resonates with you.
3:31
Some, some authors just speak at a level that really talks to
3:34
your soul. And it's, it's funny because
3:37
like I, I, I, I very much enjoy language in all its aspects, But
3:40
the way you use language to write poetry and literature is
3:43
very different than when you process it in a, in a chat bot,
3:45
right? So it's very, two different,
3:49
very different worlds. And I try to keep them very
3:51
separate because when it's like, you know, sacred and the other
3:54
one is like more utilitarian. Yeah, you don't want those chat
3:57
bots to tape all of this beautiful poetry that's been
4:02
written for centuries. Yeah, I know.
4:04
I can't, I can't handle it when they like, you know, showing
4:07
Twitter like a poem written by Alan.
4:09
It's like it, doesn't it it? It doesn't get even close.
4:12
Doesn't work. It does not work like that.
4:14
So you've been talking a lot about becoming an LLM native
4:21
company. I wanted to ask you first and
4:24
foremost, what does that even mean?
4:27
Yeah, that's a, that's a great question.
4:30
It's, it's a term that I've been thinking a lot about.
4:35
And what I think, I mean, I think we as a community need to
4:39
come to terms with with what it really means for all of us
4:42
because there can be more than one definition.
4:45
My definition is a company that understands how to leverage LLMS
4:51
anywhere that they fit. Basically some people use LLM
4:59
Native to talk about products, so like products that have LLMS
5:04
integrated in a seamless way. And some other people talk about
5:08
LLM Native people that can leverage LLMS to their work
5:12
better and and be have their day to days enhanced by using LLMS
5:17
in the right way. I think companies should be
5:19
doing both. So it it really encompasses both
5:23
sides of the spectrum for me. Yeah, dude, it's funny you
5:28
mentioned that. Like try and plug in LMS
5:34
wherever they fit. Because I've been thinking a lot
5:36
about how you have certain jobs that have different pieces of
5:44
output that they need to give. Like maybe it's the marketing
5:48
department. They need to be producing
5:51
content. And some of the content can fall
5:53
under case studies. Some of the content can fall
5:56
under blogs, others can fall under podcasts.
5:59
And I'm not saying that you just go to ChatGPT and you ask it to
6:03
write you a blog about XYZ topic because that's not going to get
6:06
you very far. But if you know what your output
6:12
you need is, then you know every like you have a process for how
6:18
you create that output. Are there places along the
6:23
production line that you can plug in an LLM?
6:26
Maybe it's the research for the podcast, maybe it is uploading,
6:31
maybe it's even listening to the podcast, the transcription and
6:35
being able to give you suggest good titles.
6:38
All of that are ways that you can plug in the LLM.
6:41
But if you have the LLM do the podcast for you.
6:44
Some people are trying it. It's cool, but it's not my
6:48
favourite way to listen to podcasts, right.
6:51
So I, I really appreciate this idea that you're talking about
6:53
of find where you can plug it in and then use it in that place.
7:01
Absolutely. That's that's very important for
7:05
me. I think you can fall to both
7:09
sides. You can over implemented lamps
7:13
or you can under implemented lamps.
7:16
And I think there, there is a sweet spot and it takes some
7:19
experimentation. Sometimes you're trying to fork,
7:21
sometimes you need to force it to just like see if it's
7:23
actually actually works. And then you you say like, now
7:26
this is it's, it's more work using lamps than doing it
7:29
myself. That's when you know you need to
7:31
back off. But I think broadly speaking,
7:35
any kind of creative work and decision making is at least has
7:42
a a very great delta in quality. If a if a human being does it at
7:47
least, and I would venture to say in most cases, Alan cannot
7:50
do it well at all. So you can have a a generated
7:55
content, but it will always be low quality.
7:57
People like, the reason why we listen to podcasts and like the
8:02
good podcast and, and read books and everything is because we
8:05
want to find that those unique insights that are not obvious
8:09
and are not like, you know, a statistic representation of a
8:12
training set. We, we want unique perspectives
8:15
on things and we, and, and the way we, we choose the podcast we
8:18
want to listen is like we kind of vet the people that we think
8:22
have these unique perspectives. So I think that that must stay
8:26
and we must enhance that so that people can do more podcasting or
8:30
more decision making and more creative work and less operative
8:33
stuff. So like broadly speaking,
8:35
anything operational is where I, I, I, I think we should all have
8:42
this call back inside of our minds that says like, hey, this
8:46
is a place where LMS at least could be applied.
8:48
Let's see, this is a, a place where if I apply LMS, it
8:50
actually saves me time. It doesn't take me more time.
8:54
Have you seen different places that are surprising in your own
9:00
life where you recognised, oh I could probably figure out how to
9:05
use an LED for this. What we do at Pampa with a team
9:09
is like we purposefully force let me see some places and we
9:13
see if it sticks. So what we do now, it's what
9:17
we're doing like an office agent that can help us do random stuff
9:21
that we do every day. So just to give you an example,
9:23
we order food every day and we need to ask everyone like what
9:27
do they want? So like we're, we're bringing
9:29
this WhatsApp agent that everyone can write to.
9:31
And then the person that is going to order just asks it
9:34
gives the the list of what it needs to order and and then we
9:37
send them to a restaurant. We're also evaluating having the
9:40
other end right to a restaurant directly.
9:42
So these kinds of things or for example, when we have expenses
9:45
that people, you know, spend something for the whole team and
9:48
then we need to do that split wise thing at the end of the
9:50
month, same thing. We have this agent, we write all
9:53
the expenses and then we, we resolved that at the end of the
9:57
month with, with, with another chapter, right.
10:00
So these kinds of things is places where we, we are actively
10:04
thinking of. Could this be be done with NLM?
10:07
And then when, when we try it out, we see if it's actually
10:10
more work or less work for us. But how does sometimes it's
10:13
actually very useful? And when you say agent and also
10:19
in these use cases like, couldn't that be done with just
10:23
pure automation and heuristics? So explain to me why you need to
10:27
plug in an LLM in there? For many use cases you can do
10:33
automation if you have if if the input is structured enough.
10:39
So like if you, if you make AUI where you have like select how
10:43
much money you spent, which were the people involved, which is
10:46
kind of what the apps do, like a split wise will do that.
10:49
Definitely you can do automation.
10:51
You don't need like an interpretative layer there.
10:55
And, and the only advantage there is like being able to just
10:57
send a voice note, an unstructured message and not
10:59
having to spend time typing and, you know, selecting the right
11:03
things. But it might be in some use
11:05
cases, it might be like a, a small delta compared to the work
11:08
it takes to actually build it and and maintain it.
11:11
But in some of our use cases, for example, ordering it from a
11:14
restaurant, it's, it might be harder because sometimes the
11:20
menu changes. So you need to keep up a track
11:23
of like the menu of all the restaurants around to be able to
11:26
select. Actually, maybe, maybe you
11:29
could, you could write it down. So yeah, I, I think it's mainly
11:34
the convenience of, of being able to write a structure
11:37
information from the platform that you're already using for
11:40
everything. So the, the cost of actually
11:43
interacting is very low because you're not, you don't have like
11:46
3 apps for three different things and you don't need in in
11:50
our case, like we are, you know, tech savvy and we like
11:53
technology, but some people they don't want to be learning how to
11:55
use each app, you know, again and again with all these like
11:58
massive set of features. So it's kind of streamlines all
12:02
your interactions with utilities in a natural language input.
12:09
You can. You can do it with automation as
12:10
well. But it's, it's interesting to
12:13
think about too that you mentioned the delta there and
12:19
how you are able to quickly spin something up that uses LLMS
12:24
versus if you were to do it with automation, maybe it would even
12:27
take you longer to figure that out.
12:30
Can you talk to me about how like your development process
12:32
now throwing something to an LLM looks like?
12:38
Like, how does that whole development go?
12:41
And then how do you ultimately make the choice?
12:44
Like, is this worth it? Is this not?
12:46
I don't know. Yeah.
12:49
So the development process there's we, we there's many
12:53
levels to how we can we can discuss this.
12:57
A big, a big part of it is just like a having an agent that has
13:00
tools and those tools like our structure output and the
13:03
structure output is, is, is processed in the back end and
13:06
that does things for you. So basically, you know, write
13:09
read operations in, in this use cases.
13:11
And I mean, discussing and streamlining all of that into
13:14
one agent that has all these tools instead of having
13:16
different apps. So that's kind of like a basic
13:19
architecture of, of how an engine like this looks like.
13:21
But then in terms of how we develop, we also try to use like
13:26
LLMS to develop, right, like many developers are doing and
13:29
try to take that to a limit. So like being, you know, cursor
13:32
power users and like trying to spend a, a less time writing
13:36
code and more time thinking about what's the right
13:38
obstructions are and like, what actually do we want to build?
13:41
And then, and once you have this is an example of like a proof of
13:46
concept or MVP that we, we did for ourselves.
13:48
But once like you're actually doing something some some real
13:51
development that needs some, some assurances in production
13:56
that that many people are going to be using it and you need
14:00
quality. Then you have like this whole
14:03
data flywheel evaluation, which you probably discussed this a
14:06
lot of times, but we try to be progressively more systematic in
14:12
the way we do our evals. So depending on the maturity of
14:15
the product, we we start with a vibe, typical vibe cheques.
14:18
Then we do like a very small evaluation set that this
14:20
integral to the whole solution. And then as this grows and it
14:24
scales and we have more evidence that there's demand for
14:26
something like this, we start doing evaluations for all the
14:29
different modules within the this this the solution, which
14:34
can or may or may not include LMS. In terms of the evaluation,
14:38
the evaluators, one thing that we have as a as a concept
14:42
internally is that we try to do evaluations as much as possible
14:46
that don't include LMS like we try to have evaluations be non
14:50
stochastic as much as possible. So for example, we would
14:53
evaluate the input of a tool, right?
14:56
Which is a strict thing, except if it's it's a query like rag,
15:00
in which case you need to do either semantic similarity or an
15:04
NLM. But many tools you just need to
15:06
have the right parameters given an input and then you don't need
15:08
an NLM to test that for for the whole system you usually need an
15:12
NLM because it's it's just text in the output.
15:15
And to do audio text you kind of need an NLM.
15:17
But we can do a lot without an NLM by evaluating the different
15:22
modules in in the application. But for something that's more
15:25
mature, when we have something we're playing around with, we
15:28
don't evaluate. We we just do vibe cheques and
15:29
we try it out ourselves. We are our own users.
15:33
What is the point Like how do you know that this has gone from
15:39
vibe check to now being we need to actually like point the
15:44
evaluation guns at this. I think it it is very correlated
15:50
to optimization. So when you have some scale, I
15:55
see it as it, it is worth investing in something
15:58
proportionally to the evidence that you have that is actually
16:01
useful. And, and, and, and it's actually
16:03
settled that that's the direction you will follow.
16:05
Because sometimes when you're starting out and you're getting
16:07
to know your users, you are kind of iterating on, on what the
16:11
final architecture of your solution would look like because
16:13
you already know what they need. So that point you don't have a
16:16
lot of evidence. And so like investment in in
16:18
your evaluation suite doesn't really make sense to be so large
16:21
because it might change and if it changes, then the relations
16:24
you made for your tools might not be worth any more, might not
16:27
be useful because the tools change themselves.
16:29
So if you start getting some traffic and the incentive to to
16:35
to have an optimised solution. So for example, you're changing
16:37
your model or you're trying to reduce your prompt or refactor
16:41
your prompt into smaller subage like, oh, what when cost or
16:44
latency starts becoming a thing because you, you want to, to,
16:49
to, to do the same utility, but in, in the scaling of, of, of
16:52
the solution, you, you need to start having more strict
16:55
requirements. That's usually a point where,
16:58
where Evas start being important because every optimization
17:01
effort that you do, you need to measure.
17:03
And it's very hard to measure many, many changes with just
17:06
five cheques. So it's it's basically like the
17:11
the requirements for optimizations come hand in hand
17:14
with more strict Evas for us. And do you have a way to track
17:21
all of these optimizations that you're tweaking?
17:26
Yeah, or really. They're experiments to try it.
17:30
Yeah, right. So you're you're experimenting
17:32
with many different things. And then you want to be able to
17:35
know that OK, when all the knobs are turned at these different
17:40
degrees, we get the best output. Yeah, yeah, absolutely.
17:46
So it is basically necessary to have an evaluation set for to,
17:51
to be able to track whether an optimization effort was
17:53
successful, because you need to be able to make the claim that
17:56
regardless of whether, you know, the cost went down later, so
17:58
down the accuracy stayed mostly the same.
18:01
So you, the only way you can really say that to a, to a
18:04
stakeholder that's not part of the engineering team like a
18:06
manager is, is that that the accuracy was not affected.
18:09
And you can, you need to show that.
18:10
So you, you need an evaluation set at least an end to end
18:13
integrate that what we would call like an integration test
18:17
for the whole system where you say like these answers seem
18:21
similar to what we had before. And now we, we can do this at a
18:24
lower cost based on this architectural change or this
18:26
decreasing the prompt or whatever.
18:30
So the, the way we would track this is we, we usually use lands
18:34
with just because we, we are partners with, with a team and
18:37
we, we like product a lot. So we would have a data set with
18:42
input outputs, we would run the the different experiments, I
18:46
guess that data set within a notebook.
18:48
So it would have different notebooks where we track the
18:50
experiment and we have the metric at the end which is also
18:53
tracked in the UI. So we would keep these notebooks
18:57
as as records of experiments. And if we can keep like the
19:01
successful experiments where we can keep the accuracy steady and
19:04
we managed to decrease cost or later see significantly, yeah,
19:08
but so. There's, when you're dealing
19:10
with these agents, there's different steps along the whole
19:16
intent system, right? And you were talking about being
19:19
able to play with tools and then going out and doing things.
19:24
If you are changing the settings on like the tools or the way
19:32
that it interprets the tools and stuff like that, is that still
19:35
captured in the notebooks, all those experiments?
19:39
And yeah, absolutely. And so that depends on how
19:47
granular the evaluation sets are, right?
19:49
Because the, the first step for us, we, we go from like the
19:52
major to the minor. So first thing is like a bite
19:55
check to just like it's very low effort and and high value.
19:59
I would say because you can kind of tell at least in like the
20:02
like in in in the 80% of happy path cases, you sound very good
20:06
to touch to to test edge cases because you might have a known
20:09
unknowns and you must might miss stuff.
20:11
But like on, on, on on the street, like normal flow, you
20:15
can test if something is seriously wrong by by playing
20:17
around with it. So that's like high value, low
20:21
effort. Then having like a full end to
20:24
end evaluation test, which we run every time we, we make a
20:29
change that's also medium, maybe medium effort, but also super
20:34
high value because we're test, we start to test these edge
20:36
cases and we, we consider these in the data set.
20:38
So it raises a flag when when some edge case is broken after a
20:42
change. And then the last point is like
20:45
high effort. And I would say depending on the
20:50
maturity, it can be high value, but only if you're, you have
20:52
like a mature product. Then we start making these small
20:56
evaluation sets for each tool, right where you, where you have,
21:02
where you can test the functionality like what would be
21:05
a unit test of that specific component and that, that that
21:09
you would have in the notebook. So then you would have you you
21:12
make a change to your agent and then you would evaluate on on
21:15
all these small parts of your system plus the the whole
21:18
integrated system. So this is fascinating and that
21:22
comes last once you've recognised, I really appreciate
21:25
how you are saying don't spend all this time on things that
21:30
you're probably going to throw out anyway.
21:33
So just do the vibe check, make sure that it's running, and then
21:37
slowly peel back the onion. Yeah, absolutely.
21:42
So basically invest in proportion to the evidence that
21:45
you have that your architecture is going to stay in that way.
21:49
And then you start having more granularity in the types of
21:53
analysis that you can make on the performance of your system
21:55
because you can tell like if it failed, where it failed, right,
21:59
and go directly there. Yeah, so you did mention
22:05
something before that I want to keep like pulling the thread of
22:09
which is you try not to do LMS judge.
22:13
And it's funny because I literally was reading Eugenia's
22:17
summary of like 40 different LMS as judge papers and all the
22:22
different ways you can use LMS as a judge.
22:25
And you're saying, like, yeah, the conclusion is if you don't
22:30
have to, don't use it. Yeah.
22:33
So I think this, this kind of, I mean this, this extends, this is
22:38
a more general conclusion, which which is if you can, like if you
22:43
can do it without an LM, it's kind of paradoxical.
22:46
But if you can do it without an LM, do it without an LM.
22:49
It's it's kind of contradictory because before I said like try
22:52
to use the LMS wherever you can. But the point is the places
22:55
where you insert the LMS must be places where it will be hard to
22:59
do that without an LMS. But now with the LMS, that's possible.
23:02
And it was, it was not possible before.
23:03
So for example, structuring and structuring input, that's
23:07
something that you need an LM for and it helps you automatize
23:10
a process. And you couldn't do that before.
23:13
So that's the sweet spot where like, OK, you, you, you would,
23:16
you would get a lot of value out of inserting LM and you pay, you
23:20
know, that stochasticity cost of, of knowing that these
23:23
results can be probabilistic and not always work according to the
23:26
planet. And you can, you can obviously
23:29
make a lot of verification and, and, and take a lot of, of
23:33
measures to, to have quite a bit of assurance of what that output
23:37
would look like. But at the end of the day, even
23:39
if you use structure output, the values themselves are
23:41
stochastic. So you, you have some kind of
23:44
uncertainty of what, how that will work in, in, in each new
23:49
interaction. But but the the reason why we
23:53
try not to use them as a judge where, where possible is that it
23:59
has this stochasticity aspect which doesn't give us a
24:04
clearview of whether the system is working correctly or not
24:07
because it changes with each run.
24:09
So you don't have this stable measurement of performance where
24:12
you say like, OK, this you, you can do like a sound called Monte
24:16
Carlo analysis where you run many times.
24:19
But you even then you need to prompt the NLM correctly and you
24:22
need to evaluate your evaluator. So there's a lot of meta work to
24:26
to make anonymous attached to work appropriately, which
24:30
sometimes is necessary. And and that's where it fits for
24:33
me. Like if you can't do it without,
24:35
for example, when you have like an end to end agent that has a
24:39
natural language answer to the user, it's very hard to test
24:42
that with just rules. So there anonym has actually
24:45
would make sense, but only there like if you can't avoid in other
24:49
parts of your system to have at a lens evaluate your input as
24:52
much as possible. It gives you much more control
24:55
of over your evaluations if you have rule based evaluations.
24:59
And are you doing any specific kind of LM as a judge when you
25:03
do use it? I'm honestly not familiar with
25:06
all the kinds there are, So I'll tell you like kind of what, what
25:11
I usually try to do, which is like basically having either I,
25:17
I, I, I read the, the, the blog post.
25:19
I like the, the fact of having a binary output instead of like a
25:23
gradient of, of, of real numbers.
25:26
So I, I, I like that approach of having this is the ideal answer.
25:30
This is what answered is, is this, is this a correct match
25:34
or, or which of these two are, is a better answer?
25:37
Like that type of comparison is, is interesting to me.
25:41
And, and I like it. And and the other option is just
25:45
like having a reference answer and, and, and handing like some
25:49
things that you want to assure, like the style is kind of
25:52
similar or the answer has the same content, broadly speaking.
25:57
And and having like measurements for each of these things and
26:00
having different evaluators for each of the aspects you want to
26:03
analyse, which are not strictly derivable by rules.
26:09
So depending on the use case, we, we will do one or the other.
26:15
I'm I don't have like a strict position taken on on what the
26:20
best way of doing it is and it depends on the use case.
26:23
Nice, So what other cut of shit are you using agents for?
26:28
One thing I am, I am so this is like what I'm telling you about
26:32
trying to force agents and see if it if it if it works, if it
26:35
sticks. One thing I've been trying and
26:37
failing at is is like we do our podcast every single week in
26:41
Spanish for like the Spanish AI engineer community.
26:44
And I've been trying to use agents to create presentations.
26:47
Like I've tried every single solution out there like Ghana,
26:51
like Slides, IO, which integrates with Google Slides.
26:54
There's a few of them. They haven't been helpful for
26:57
me, but. Presentations for what?
27:00
For your podcast, like after you've created the podcast, then
27:05
you want a presentation from the information.
27:07
So it's it's. More it's, it's not so much as a
27:10
podcast, but like like a live stream where we from Pampa, we
27:15
explain a specific concept and then we open up for Q&A.
27:18
So it has a presentation part, then has a demo part, and the
27:21
presentation part is using slides.
27:24
So many times I have the content in my head and I don't want to
27:28
design. So I'm just trying to find
27:29
something that will give me good design out of like the structure
27:33
that I already know I want to present.
27:35
And I haven't been, I haven't been successful for that at at
27:39
least yet. Another thing that I I I try to
27:43
do is use ray cast. As you know Ray cast low.
27:49
What's this so ray cast? Is like a like a all around Mac
27:53
tool that has like a lot of shortcuts for Mac.
27:55
But I think if you're paying $10 a month, you can get AI
27:59
integrated. So in any app, Slack, e-mail,
28:02
whatever you can, you can have your own custom shortcuts of
28:05
like fixed grammar or like maybe more formal and you can prompt,
28:09
you can make it your own L&M calls that, that map to a
28:13
certain shortcut. So for, for like some members of
28:17
the team that, that have that, that want to improve the
28:21
English, for example, or, or something like this, or when you
28:24
just don't want to structure everything super specifically
28:28
and you just want to jot it down and then like have a certain
28:31
structure. I, I like using LLMS like it
28:34
with humans in the loop and not like just expecting an end
28:37
output because it doesn't usually do a great job at like
28:40
giving you the finite output. It can get to like 70% of the
28:43
way. So that's, I like to jot down
28:45
those things and then I have it and then structure them and then
28:47
send them, for example. And, and I, I on our topic, I am
28:50
very passionate about this, like the solutions and we, we call it
28:54
internally like low, low command tab solutions that, that don't,
28:59
you don't need a lot of command tabs to get what you want.
29:02
So not no changing screens like you got.
29:05
I am writing something slagging and jot it down.
29:07
I I do a shortcut and it changes it right there.
29:10
I don't need to go to ChatGPT and then I can just press enter
29:13
or cursor is a great example as well where you have like
29:15
everything that you don't need to move from there.
29:17
You have every single and functionality chat, how to
29:21
pilot, copilot everything there and then you can just push so
29:26
that that's another good example Raycast so.
29:29
That that was fascinating to be because.
29:32
I. Remember with ChatGPT came out
29:36
and people were saying like this is going to change the world,
29:38
right? And there was some folks that
29:42
were talking about how there's needs to be a new operating
29:47
system, an AI operating system. And I didn't really understand
29:51
what that means. And I still do not understand
29:54
what that means. But then I saw some people
29:58
talking about how chat could be your Mission Control and if you
30:03
have the agents that you just are in the chat all the time and
30:08
the agents will go and do what you need them to do.
30:11
I don't think it's reliable enough for that yet.
30:14
And I also wonder if that whole assumption is flawed, where
30:19
we're still going to be doing what we're doing in our specific
30:23
places that we like to do them, Slack or cursor.
30:26
But the AI will come or the chat will come to those places as
30:32
opposed to were in the chat telling firing off commands of
30:38
things that need to go and happen behind the scenes.
30:42
Yeah, that's, that's a good point.
30:45
I, I myself like, I really, I think we all like, we all enjoy
30:49
seeing things and like having interaction with things.
30:53
So I don't think that's going anywhere.
30:57
Just having like a, like a, like a white chat screen that feels
30:59
sad to me in some way. It doesn't feel like a good
31:01
experience. Whereas like having this like
31:04
more integrated solution where you have chat, but you have lots
31:08
of other things within the, the, the UI feels less of a
31:13
compromise and less of a trade off and more of like an
31:16
enhancement, right. And I've seen, I've also seen
31:21
that the, the OS thing, I think Carpathy tweeted it back in the
31:24
day. I, I don't understand it, but I
31:27
think it's like, you know, when Kabatist says something that you
31:30
don't understand, it's obviously you, you're, you are the
31:34
product. But yeah, I, I really like, you
31:37
know, having, having chat within whatever I'm doing, which is
31:44
like the case with cursor, right?
31:45
Like I'm, I'm coding. I have an IDEIDES are still, you
31:49
know, a great thing. But I have an LLM enhanced ID.
31:52
So that's kind of also like the LLM native concept for me is
31:56
whatever you are already, you can be an LLM in front step,
32:00
right? So for example, this applies to
32:02
traditional industries and for professionals.
32:05
So like doesn't mean like if you're, if you do marketing that
32:07
your marketing knowledge is, is obsolete that the opposite.
32:11
It's worth more than that than than than than before.
32:14
But now you can, you can power that knowledge and get farther
32:18
with that with that domain specific knowledge by using LL
32:21
apps. And the same thing with, for
32:23
example, a traditional company. If you have users and the users
32:26
love what you do, you can serve them better because you know
32:29
them better than other people and you know how to apply Adams
32:32
to serve them. So like domain spread domain
32:36
specific knowledge for me, it's like at an all time high in
32:39
value if and, and, and especially if you use an Adams
32:43
to, to, to leverage that domain specific knowledge, be it like
32:46
from the job side or from the from the product side.
32:50
And this, it goes a kind of against the narrative of like,
32:53
you know, every job or, or every product is going to be replaced
32:56
by like LLMS. I don't think that's the case.
32:59
It, it's, it's, it's all about experience.
33:01
And then the, the experiences that have LLMS integrate in the
33:05
right way will be enhanced, but not replaced.
33:08
I'm I think. Yeah, my buddy Marcel likes to
33:13
make the point that you still need to know what good looks
33:16
like or you you still need to know what great looks like.
33:19
And so it's, it's basically saying if you're an SEO expert
33:25
on the marketing team and the LLM does stuff for you and it
33:30
gives you something. If you don't know what great
33:33
looks like, then you're kind of sitting there and you're like,
33:37
well, I guess it's good, let's ship it.
33:40
But if you know what great looks like, then you're going to be
33:42
able to say, OK, we need to optimise these H1 tags and it
33:45
looks like we didn't get the metadata right on this.
33:49
All those things that the SEO folks like to talk about, then
33:53
you're going to see that and see where it's OK and where it's not
33:58
OK and then add your added value.
34:00
But like you said, you get 70% of the way there hopefully.
34:05
Absolutely, yeah. And I would add to add like a, a
34:07
engineers don't know what a what a good SEO result looks like.
34:12
We, we don't know what a good marketing result looks like.
34:15
We don't know what is a good experience for the user in the
34:18
construction industry. We don't know.
34:20
So like it's not like we can just like automate away
34:24
everything that the systems need direction and the direction
34:26
comes from knowing that your users very well, which is the
34:30
case with traditional industry. So that's where I see like this
34:33
integration happening with like a engineers, but like very, very
34:38
close to the domain experts, right?
34:39
Yeah, do you because your background is a data science or
34:44
it's. I, I studied economics in
34:47
college and then I quickly moved on to like deep learning because
34:51
I, I became fascinated with it. But I just have a hybrid
34:54
background. Yeah.
34:55
And in your eyes, what is the difference, if any, between like
35:03
a machine learning engineer and an AI engineer?
35:05
Is it just a different name for the same kind of thing?
35:09
There's been a fair amount of debate on, on this topic about
35:13
like what really are the engineers if, if there's, if
35:17
there, if they are something, right?
35:19
I, I, I do believe there's like elements change things
35:24
materially. It's not just like another
35:26
algorithm because it, it is, it is an automation opportunity for
35:32
everything that existed before. It's not like something that is
35:35
like a module. It is transversal like a
35:37
capability. So the only thing I see
35:42
different from PE engineers, which I would call the way, the
35:46
way I see it, and I don't like to get stuck to a term.
35:48
So like the way I see it is you have different job roles.
35:50
You have a data scientist, you have a machine learning
35:53
engineer, like the traditional job roles and you have a human
35:56
resources expert, you know, and, and all these jobs have like
36:01
material expertise in a certain topic.
36:03
For example, data scientist needs to do good evaluations and
36:06
needs to have like a good experimental process because
36:09
that's kind of your bread and butter, regardless of which
36:11
algorithm you're using. So that that that stands apart
36:14
from whether NMS exist or not. And then maybe like what an A
36:18
engineer would be to me, and again, this is probably a
36:21
definition thing, it's somebody that can do AI with, with in an
36:28
LM native way. So the a engineer would be able
36:30
to include LM features in a product that they're building,
36:34
right? And they would be able to
36:35
leverage LMS in their own development process.
36:39
So, but, but I, I don't see like AI engineers per SE being like a
36:45
job for me. It's more like, like an
36:48
enhancement of maybe like a data scientist, right?
36:52
Like a data scientist that knows like the traditional machine
36:56
learning topics, like evaluation metrics, right, experimental
37:00
process. And, and they enhance that with
37:02
a lemon. That's like a 2 point O.
37:05
You know how an LM native data scientist?
37:10
Yeah, because definitely that's an estimation topic.
37:14
Well, it's fascinating you say that because the data scientist
37:17
being on the hook for evaluation and experimentation, we were
37:21
just talking about earlier, right?
37:22
When you're building out your agent, one of the things that
37:26
you're looking at is how do you experiment to bring down costs
37:30
or latency, and how do you evaluate the effects of that
37:35
experimentation? So it's, yeah, it's, it's fun to
37:42
think about these things because I know this topic has come up
37:46
quite a bit. Yeah, absolutely.
37:50
And I think really in terms of evaluation, things have not
37:57
changed that much. Maybe the metrics change and
38:00
you, you know, how need to understand how to have an LLM as
38:03
as part of your evaluation, right?
38:05
Like as an LLM as a judge that that's a new thing.
38:09
But evaluation in itself has always been something you needed
38:12
to do. And that's part of, of, of the
38:17
data science process from for our.
38:20
So having good evaluations, understanding what a training
38:24
set means, what an evaluation set means, what a test set
38:26
means, what, how many examples you should have in different
38:31
evaluation sets, covering different, different aspects of
38:35
how your solution might be used to write different types of
38:38
inputs and, and trying to get the, the, the ground truth for
38:41
those inputs. That, that kind of thinking is,
38:45
is basically science, right? It's a science mindset or
38:51
scientist mindset, which doesn't really change.
38:54
But what changed is that some modules of, of those evaluations
38:59
behave differently. So you need to adopt and
39:01
understand, you know, the stochasticity behind them.
39:03
But like, I see this a lot like some people that come into, into
39:08
AI now and they, they are like prompt engineers or AI
39:11
engineers, they have no background in, in doing science
39:14
for proper active environment. So that kind of rigour is, is
39:21
important, especially in the maturity in a, in a maturity
39:24
stage of a product. And you don't need to learn it
39:27
or you need to bring that knowledge from from like
39:30
traditional machine learning experience, because it's not
39:32
enough to just prompt and like for the LM at at your outputs
39:36
and you know, wish for the best. Yeah, you sound like you're
39:41
talking from experience. You've seen the worst of them.
39:45
Yeah, I mean, it's understandable.
39:46
Like, yeah, now with that lens, we can build whatever, you know,
39:48
the bar is super low, everyone can get in.
39:50
Like everyone can build AUI, you know, in, in one day.
39:53
That's all true. And it's fantastic for PLCS.
39:56
But then like when you start making these solutions that
39:58
actually need to handle real users and, and, and like the
40:02
users will be using all these in like whole set of unexpected
40:05
ways. You need some systematic process
40:09
for for improving your solution and finding improvement
40:12
opportunities as well. Yeah, it's the difference
40:16
between you having that agent that takes your food orders just
40:23
for your Slack versus creating an actual Slack bot that you put
40:28
on to the Slack marketplace and let anyone use that.
40:32
Yeah, and like, I, I love the scrappy, scrappy attitude of
40:37
like, you know, let's hack and, you know, let's break things.
40:40
But that's like a different mindset that you have when
40:42
you're doing a, a, a, a product for like, you know, an
40:46
enterprise Oregon, like big company.
40:47
It's, it's very different. So I, I, I like having like my,
40:54
my place to play, which is like the agents we use.
40:58
I'm not like translating that mindset to everything, you know?
41:01
Well, so talk to me when a client comes to you and they're
41:04
like, all right, we need to do some with AI because I imagine
41:09
they just come to you and say we want to do AI stuff.
41:12
Do they come with that broad of a vision or do they generally
41:15
have it mapped out in their head like, hey, we've got this use
41:18
case and we think we could plug in some kind of LMS?
41:23
It it depends. Sometimes it's more of a blank
41:26
slate with a vision where like we would like to build this, we
41:30
don't know where to start in. In that case, usually what you
41:34
do is like APOCMVP, right? Throw it to some users, Start
41:39
learning from the users and start working your way towards
41:43
like a mature evaluation suite like we discussed earlier.
41:46
Sometimes it's already something that's production and you're
41:50
starting to scale. And then when you need to scale
41:53
the requirements, the technical requirements for resolution are
41:56
different. You need more expertise in the
41:57
table. So in that case, it's more about
42:01
optimization that that is 0 to one on the usually clients to
42:09
have some background in terms of they, they have people in in the
42:14
team that already have been researching into LLMS, They've
42:17
been playing around with them those sales, right?
42:19
Like they have background and they, they want to be able to
42:26
prioritise correctly and understand what is actually like
42:30
a good thing to, to add to a solution versus like hype.
42:35
A good example for this is RAG. Like RAG has an infinite
42:38
assortment of variations and stuff you can do with RAG super
42:41
fancy things. But sometimes when you have a
42:44
Rag pipeline, it's, it's more about like understanding the
42:48
nature of the data. I'm thinking from that data,
42:51
what needs to be done to reach you need to get deep into the
42:53
use case. It's, it's very hard to apply
42:58
boxed solutions. It's just like an API that you
43:01
can hit and, and harm the great RAG.
43:03
So many times it's more about exploration, about the on the
43:06
use case, understanding what exactly is the problem that
43:10
other returns from the salt and then diving into.
43:13
And that's also like way consulting makes sense because
43:15
it's not like you would just like a buy a product and there's
43:17
many start-ups doing for every every single sort of thing.
43:20
But many times you need like to get good performance, you need
43:24
custom custom software. Basically like doing things
43:29
tailored to your solution. And can you explain the
43:32
different spectrums that you see when it comes to rag on how
43:38
you've done certain things for some folks and once you've
43:42
gotten in there, you got into it with the data you recognise, OK,
43:45
we should probably do this with this type of use case or with
43:50
this type of data versus a different option or a different
43:56
data slash use case. You need to take it entirely
44:01
different approach. Yeah, it it does really depend
44:04
on the use case. Sometimes when you have like
44:06
just documents and you want to answer a question based on
44:09
documents, it there's like all these questions about what are
44:14
you embedding? Are you embedding the title?
44:17
Are you embedding the content? What part of the content are you
44:19
embedding? A summary like all these options
44:20
are on the table. What seminary, like what
44:23
embedding space do you have? Which is the model created
44:26
you're embedding? Is this more than a custom model
44:28
or is it like a pre trained model?
44:31
And all these kind of questions relate to what's going to happen
44:34
when that query gets embedded and and and and and retrieving
44:38
similar vectors like that similarity will be defined by by
44:43
the priors that your solution has in terms of how did you cut
44:46
your documents, what did you embed and what you showed asking
44:50
those type of question. If you have like a traditional
44:52
ragpipe, that is the first thing.
44:53
And then if you start putting from there, many times you end
44:57
up doing stuff that actually has a name, but you don't know, like
45:02
early in the end days, I, I was doing ragged and I, I didn't
45:04
know what rag went. And like when people spoke about
45:07
rag, I was like, what, what are you talking about?
45:08
And then I, I realised it's like, Oh yeah, it's just like
45:10
retriever. So it's funny how like all these
45:13
terms and all these like super big techniques come out.
45:16
And it's just like sometimes it's more of a, of a common
45:19
sense approach to, to a certain use case.
45:22
That's that's, that's more for like retrieving the right
45:26
documents. And then and another big use
45:29
case which has different requirements is extraction.
45:31
So like if you have a very large document and you need to find
45:34
that right paragraph where that piece of information that you're
45:37
looking for is that is it? It's different because the the
45:44
input is determined, you know what you're looking for.
45:47
It doesn't depend on what the user says it.
45:49
You can have a certain amount of options, but but it's usually
45:53
like a limited list of data points that you want to obtain.
45:58
And in that case, you can do much more science in terms of
46:00
like what are the, you can even do like rules, specific
46:02
keywords, right? Hybrid search of, of trying to
46:07
get really good at like those finite list of terms that you're
46:10
trying to find to extract the right, the right, the right
46:14
values, the right data points. So that that's like a a
46:18
different approach to a different use case.
46:22
First of all, I appreciate you breaking this down because 100%
46:25
like I was going to go down the route of, oh, I thought you just
46:28
meant what kind of embedding model you're using, What kind of
46:34
chunking strategy are you using? But that's just one side of the
46:38
spectrum where all those questions come up.
46:41
Then you go to the other side of the spectrum and basically
46:44
you're saying it's a search problem more less than a
46:47
retrieval problem. This is just like, hey, we're
46:50
doing search and you're calling it AI.
46:54
Yeah, absolutely. And I've, I've had AI have a
46:57
background in, in ranking and retrieval because I I worked in
47:03
recommendation systems in, in medical, which was my previous
47:06
company and especially. Talks.
47:10
He's been on here a few times, absolutely.
47:12
For Litos and for Lina. Lina Javier, they are the
47:18
medical team. They were instrumental in the
47:21
early foundations of the ML OPS community.
47:24
And I think that's how we sparked up our conversation when
47:27
we met, isn't it? Yeah.
47:28
Yeah, so and I, I was part of our team as well.
47:30
So like it's it's it's it's interesting how I get old close
47:35
together. Yeah, So like for me, the 1st
47:41
when I started think because it all came from embeddings, I
47:44
worked a lot in embeddings in Mancala back in the day.
47:46
So when I started understanding like embeddings were a key part
47:49
of many of the of these agentic systems.
47:51
It, it all, it automatically rung a bed of everything I knew
47:54
from from those times, right? Vector spaces, latent spaces,
47:58
custom embeddings, fine tuning when it makes sense, when it,
48:00
when it doesn't and, and how you build the, how you purposefully
48:06
define your latent space based on your problem, which, which,
48:10
which has like different variables that you can pull from
48:14
that. That was what, what really stuck
48:16
for me. Like, OK, now this is, this is
48:18
the field we're playing on like in various LLMS are are outside
48:22
and now we're talking like later space.
48:24
Let's make this work and then let's see how it interacts with,
48:27
with the LLM. The only part of the LLM is, is
48:31
involved in is like how it creates the query, which is
48:34
basically prompting. But once you understand the
48:36
types of queries that that you want, you can treat the the
48:39
right system as like a separate part, right.
48:42
And I know this traditional retrieval questions are are
48:47
exactly the same you're trying to find, you know, I need
48:51
another step. That's it dude.
48:53
That's. Fascinating to think just like
48:56
it's almost all the same foundation, but then when you go
49:00
to RAG, you've got the prompts. When you go to Recommender
49:04
systems, it's just a bit different.
49:07
I'm not going to say abstraction, but like a little
49:11
bit different spice involved when you're cooking the
49:15
different use cases. Yeah, maybe recommend their
49:19
systems. The input is keeping by an user
49:22
directly, whereas in in Netherland rack pipeline many
49:26
times the queries is an ungenerated.
49:29
So there, there is a probability probabilistic aspect between the
49:32
user and and and the retrieving system.
49:36
But, but, but you still have like an undetermined input that
49:40
needs to get that is to find the right needle in a haystack.
49:43
I think it's actually an advantage to the fact that you
49:45
have an item in the middle because you can kind of clear
49:49
the, the whatever the user did like in a very untidy way, which
49:54
is like how users myself, like how, how, how you use
49:57
terminologies like you're untidy if it works.
50:01
So you have that like adapt adaptive layer where you can
50:03
like kind of clean whatever is, is coming in the input versus in
50:07
traditional recommended systems, you just have what you have.
50:10
This, this is the input that you should tell you if, if, if they,
50:14
if they missed a keyword or something, you know, if you are,
50:18
you're stuck with that, if, if it's misspelt, whatever.
50:24
Sometimes, you know, obviously some semantic models can account
50:27
for that, but but sometimes having a cleaner input helps to
50:31
make good retrieval. And also it helps in terms of
50:33
evaluation because you kind of with the LM you can you can
50:37
control what type of input you get and then you can build a
50:39
better evaluation set with that in mind.
50:43
Well, it. It's funny too, you mentioned
50:45
that in this specific example you're saying the user is giving
50:51
you information where they're probably typing in a search bar
50:54
what they want. And let's say that it's at
50:56
Metacala Libre and I'm looking for some new high heels.
50:59
Not for myself, for my wife, of course.
51:02
And then I get a bunch of recommendations that are saying
51:08
like, maybe I want blue or maybe I want stilettos, whatever it
51:12
may be. But I find it fascinating too,
51:18
with the recommender systems that you don't really get in the
51:21
rag. Again, this is because it's a
51:22
different spice, but the engine is the same.
51:28
In the recommender system, you can use these features of oh,
51:33
hey, he hovered over that picture for a little bit.
51:36
We should maybe show a few more of these blue stiletto heels.
51:41
And oh, he also was asking about whatever turquoise dresses.
51:47
Maybe we should try and show him a turquoise heel that will go
51:52
with that. And again, not for myself.
51:55
I don't like to wear stiletto heels or turquoise dresses.
52:00
I just hang out with three women because I've got two daughters
52:04
and a wife. Yeah, that's an interesting
52:08
thing to to consider. And like, I think specifically
52:13
now like how the commercial recommendations systems will
52:19
work with LMS in, in, in, on the table.
52:22
It's going to be, it's going to be a very interesting
52:24
development in the field. I'm already seeing stuff that I
52:26
really enjoy. What have you been?
52:28
Seeing. So like, you know, the Shopify
52:33
agent that you can just like ask.
52:36
So what I, what I really like is like you can have like this
52:38
fuzzy philtres where you can say like, yeah, I, I like it a
52:42
little bit a darker red right there.
52:45
There's been papers written on this before the LMER, which I I
52:48
like because I really enjoyed the covers problem, like fashion
52:50
clip that has like, like being able to transverse the latest
52:55
space with this kind of like latent variables, like how dark
53:02
the colour is, for example, which is not something you can
53:04
philtre for obviously. So those kinds of, of, of, of
53:09
queries start to feel a lot like, you know, a, a real sales
53:14
experience versus just like an e-commerce site.
53:17
So that's, that's super interesting to me.
53:19
And, and, and that really, as you were saying, like what you
53:22
can use to find the right product.
53:25
It's like a very wide spectrum of multimodal data because you
53:29
have all this navigation information and, and the hover
53:32
information like every, everything can be part of
53:35
whatever you, you do to retrieve the right products.
53:39
So like it, there's a lot of information that you need to
53:42
prioritise how you're going to incorporate into your system in
53:46
the, in the recommendation system.
53:47
So it's it's. It's a very interesting problem.
53:50
Yeah, I've also heard folks that are using LMS to help them get
53:58
different feature ideas for their recommender systems, but I
54:02
haven't. So I, I look at that like as
54:05
during the training part of the whole recommender system, I
54:11
haven't seen anyone quite marry the LLM up front for the user
54:18
where you can just talk to it. But now that you're talking
54:21
about this Shopify agent, I do like that idea where you're
54:25
talking to it. And so if I'm understanding it
54:28
correctly, you're getting showed stuff and you can say, yeah, I
54:32
want something like this but with a darker shade or a
54:37
different colour. Yeah, yeah, exactly like you
54:42
can. You can ask, it will show you a
54:44
few options and you can continue the discussion, which is very
54:48
compelling for me. And this brings up, just for me,
54:57
the thinking around this is very similar to a presentation that
55:04
we had, I don't know, like 6-8 months ago from Linus.
55:09
And he was talking about the different ways that you have,
55:12
you could have interfaces with LLMS. And you see it a lot at
55:16
Notion where he used to work. And Notion has the ability to
55:23
click. And it gives you kind of like a
55:26
short list of things that you're going to want to ask because in
55:30
his presentation, he was saying we've already got the mouse and
55:34
the ability to click. And so not all the time do we
55:38
want to be chatting, trying to ask more of whatever it is that
55:45
we're looking at. We just want a drop down menu
55:48
with maybe five options and we click on one of those.
55:52
Or we can if we need to continue chatting.
55:58
Yeah. And so I, I find that interface
56:00
fascinating too, on how. All right, if we can get
56:06
something that is only a click away, it's much easier to click
56:10
on something as opposed to write out, even if it is writing just
56:14
one or two words. Absolute and I think this, this
56:17
is that is probably the key topic here for a company
56:21
building with elements, which is like you're pulling from the
56:25
experience, right? Whatever you do in terms of
56:28
features or your implementations, you're, you're
56:30
pulling from a, a vision of a, of what a great experience looks
56:34
like. And it needs to be that way.
56:36
Like you can to be like non LLM dramatic or LLM dramatic who
56:40
want to do everything with chat or, you know, like, or, or we
56:43
don't want to do any, any LLM features, even even in the back
56:48
end. Like it's all about like giving
56:50
a great experience. And that requires like the, the
56:53
pragmatism of like having all these assortment of tools and
56:56
using them to, and it takes a lot of experimentation as well
56:59
with the users. But this is like the path I see.
57:02
I, I like to see companies going where they're trying out things
57:06
and suddenly they, they find this, this amazing experiences.
57:09
You know, they were, they were impossible before, but they are
57:12
not like dogmatic in any kind of way.
57:15
It, it, it's just like a, a right combination for that kind
57:18
of user. And that, that's all like the,
57:20
the layer 2 of solutions, like the foundation there.
57:22
We've seen all developments like now it's get how do we get the
57:26
settlers working for 90% of the people that don't care about
57:30
technology and they don't want to know what's going on behind
57:33
the scenes. They don't want to know.
57:33
They just like help me solve my problem with a great experience.
57:37
So that that's that's what I really like to see happen.