Latest / Agentic Conversations (formally mlops.community) / Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development // Boris Selitser // #241

Transcript

Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development // Boris Selitser // #241 · Agentic Conversations (formally mlops.community)

0:00
Boris Zelizer, I am CTO slash CPO at Acario.
0:07
You know, to do as many things as I can were well, I, you know,
0:13
had a many years of self deprivation, but I finally
0:17
bought an espresso machine and and it's the one of the best
0:22
purchases I've ever made and I make a mean cortado.
0:27
It's two shots and then just super dry, just foam.
0:32
That's that's my coffee routine kettle live without it.
0:37
What is going on in Mill OPS community?
0:39
This is another good old podcast.
0:42
I'm your host Dimitrios, and we're talking to Boris today all
0:47
about evaluation, synthetic data and how it helps you evaluate AI
0:53
agents and how you can evaluate them.
0:56
What he's been seeing on the agent field and what some best
1:00
practises are. Also a few hot takes in there as
1:03
you know how we like to do it. On this podcast, we kept our
1:07
feet on the ground. We did not get too hyped.
1:10
He even called out we may be at peak hype agent.
1:14
Let's see if we are because I still think we got a little bit
1:18
more hype to go with when it comes to the agent field.
1:24
Now he has worked at Facebook and one thing that I want to
1:28
call out from this conversation that he mentioned was how his
1:33
time at Facebook helped him understand what it means and the
1:38
power of choosing the right metrics and the power of being
1:44
able to track those metrics. And then he also left a little
1:49
caveat in there to say it's not only choosing the right metrics
1:55
because you don't always get the metric correct the first time.
1:58
So you have to iterate on it and update your metrics constantly.
2:02
And this is so important when it comes to the evaluation metrics
2:06
and how you're evaluating the output and also the retrieval,
2:11
all of the fun stuff that we know when it comes to these AI
2:15
systems. O let's get into this
2:18
conversation with Boris. And I should say that his
2:21
company is hiring for machine learning engineer.
2:23
So if anybody is interested in joining, then we'll leave the
2:28
link in the description. Alright, let's get into it.
2:32
As always, if you like this episode, feel free to just share
2:37
it with one friend. That's how we can spread.
2:40
The good word. It's probably probably good to
2:51
start where we just almost were talking about it before I hit
2:56
record And that was on like this convergence of software
3:01
engineering and then data science, machine learning
3:04
engineers. And it's something that I've
3:07
been thinking about quite a bit lately too, is how much do you
3:11
need in one? How much do you need in another?
3:13
How transferable are the skills of a software engineer into
3:18
building really production ready AI features or products and what
3:26
do they need to know? And then vice versa, like a
3:29
machine learning engineer that was traditional ML.
3:33
Now going into playing with LM's, there's a lot of things
3:36
that are very familiar and then some things aren't.
3:39
So how do you see this space? Yeah, I think that is definitely
3:44
a relevant question right now. And I think the, you know, the
3:48
way you framed it is I think is accurate.
3:51
I think the there's a lot of sort of convergence that's
3:54
happening right now between the software engineering and
3:57
traditional sort of software development world and the the
4:01
world of the, you know, traditional ML and lots of the
4:05
listeners of this podcast. I think in terms of, I think,
4:10
you know, before you had six months, you were building a
4:12
model, you know, two to three, six months sometimes.
4:15
And then at the end of it, you don't know where you're going to
4:17
get, you know, you you're you're you're you're driving like a
4:21
long term experiment. And then you're, you know, you
4:24
have to be gritty. You have to be sort of, you
4:27
know, tolerant of failure. And then you have to have this
4:32
way of, hey, can I, can I, can I drive and iterate in these long
4:36
cycles, but maybe deliver a high, you know, returns at the
4:39
end. I think with the coming of, you
4:42
know, a lot of the foundational models, I think the iteration
4:46
cycles are starting to resemble more of the software development
4:49
cycles and they're getting shorter and shorter.
4:52
And you can get that same ROI out of, you know, the prosaic
4:56
prompt tuning and tweaking. And there's lots of good
4:59
research that's saying that, you know, prompt engineering as as,
5:03
as you know, as a non engineering as it might feel for
5:08
some people, is actually quite effective in producing the
5:12
outcomes and behaviours from foundational models.
5:14
And so you get the same outputs that you would get in other
5:18
places in six months much faster.
5:21
And that that drives sort of the mentality and the tools that
5:24
people use and how they approach building systems in a different
5:28
way. And I think from a, from a, I
5:31
think from a software engineer standpoint, there's a lot of
5:33
things that are being automated away, like the mechanical parts
5:37
of writing code are slowly but surely are removing and so are
5:43
being removed. And So what does the software
5:45
engineer two, right, they're picking up all these data
5:47
skills, they're picking up all these metric driven development
5:50
skills and they're thinking about how to transition into
5:54
this rapid development with a foundational models world.
6:01
Yeah, it's funny you say that too, because I was making a
6:05
TikTok that my parents found and they gave me a bunch of shit
6:08
for. But in the TikTok I was saying
6:11
how you have this aperture that's been opened.
6:16
And I like it's not only that the potential people that can
6:20
play with AI have now increased like 100 X, but also the speed
6:26
at which you can get value from AI has shortened or it's gotten
6:31
much faster to be able to see if there's some value that you can
6:35
get from something. And what I was saying is that
6:39
it's easy to go from like, yeah, zero to 8085%, and you've got a
6:46
lot of software engineers that can do that.
6:49
And then their minds blown and there's magic that's happening.
6:52
And not just software engineers, basically anybody, if you can
6:54
think of it. And then you say, well, yeah,
6:56
let's see if we can use an MLM to help us out on that.
7:01
But to get from that 85% to that 95%, that's where all of a
7:06
sudden you're like, well, I guess I might have to figure out
7:09
like data pipelines. And I might have to like go deep
7:13
down the rabbit hole on data engineering because you're
7:16
figuring out the vector database and your chunking strategies and
7:20
how to update the data and keep it clean and data access maybe,
7:25
or just how to make sure that the data doesn't leak.
7:28
And so people become they turn from software engineering into a
7:33
bit of a data engineer too. I think yeah, there's like a lot
7:38
of, you know, non sexy parts, I guess of of of, you know, data
7:46
wrangling that that go into building good quality models,
7:50
good quality applications that use models.
7:52
And so I think there's definitely, yeah, there's
7:55
definitely like, you have to go deep and not be, like, scared to
8:00
get your hands dirty in terms of actually kind of diving into the
8:04
little bits of data and figuring out like, what makes sense and
8:07
what doesn't. And this is where I think, you
8:11
know, a lot of the use cases that people are building now are
8:14
not going to be grounded on a bunch of production data, on a
8:17
bunch of historical data. I mean, they can be informed by
8:21
that. And there's some things you
8:22
could do with it, but a lot of it is new applications that are
8:25
being built that weren't possible before.
8:28
And this is where I think, you know, synthetic data plays a
8:32
huge role. This is where people can
8:35
actually leverage kind of synthetic data to kind of
8:39
generate the scenarios and the behaviours that they expect from
8:43
the model. And this is where like there's a
8:46
set of tools that they can bring into the whole kind of building
8:48
life cycle all the way from like describing what they expect to
8:53
have in the system to begin with.
8:55
And think of that description now, not in terms of like, you
8:59
know, functional specs necessarily, but more in terms
9:02
of behaviours that are data defined, right?
9:06
And so recognising those behaviours into groups, into
9:10
things that you expect the, the model to do, including sort of
9:13
like the negative cases where you expected not to do, right.
9:16
So you know, like the very basic example is a lot of
9:19
jailbreaking, etcetera, that you include in your product.
9:22
That doesn't have to be like vanilla jailbreaking that you do
9:25
in terms of a large language model, like the, the testing
9:28
that Open AI Anthropic are doing on their models themselves, but
9:31
more specific to the nature of your application, the, the
9:34
whatever you're building. And so kind of the new surface
9:37
areas that exposes, right, including that in the kind of
9:41
the description and and testing sort of flow.
9:43
So you start with defining what it is you expect and then
9:46
evaluations become a key component, right?
9:49
Like without evaluations, you don't have baselines.
9:52
And so you have to kind of run through and actually establish
9:56
where you stand. And now your life becomes like
9:58
very metric driven. And, you know, we can get into
10:02
the whole conversation of, you know, what metrics actually make
10:05
sense in this world. But as you continue to tweak,
10:09
you have your baselines, you can iterate and then you can
10:11
actually drive the synthetic loop all the way through to
10:15
improving the model, fine tuning it.
10:18
And that's where I think there's like a lot of leverage where
10:21
you, you know, you can, you can build these new use cases with
10:24
synthetic data and not just have to kind of, you know, rely on
10:29
production data out there. What have you seen as far as
10:35
well done prompt injections or looking at a specific product?
10:42
That or AI product specifically right?
10:45
And people being able to protect themselves against prompt
10:51
injections or just toxicity or ways that it could go off the
10:57
handlebars. You know, honestly, it's, it's
11:02
kind of interesting. I think there's, you know, if
11:04
you look at a lot of the jailbreaking datasets there, you
11:09
know, they, they get quite humorous, but there's things
11:12
that are, you know, like Dan or this is the yes man or so on and
11:17
so forth. And so you could actually kind
11:19
of expand those into application specific scenarios.
11:23
But I think it's, you know, it's the, the nature of the
11:27
application that where you can actually go deeper.
11:31
For example, you know, if you're trying to build like a routing
11:34
component that routes to a collection of agents that you
11:37
have, you know, the, the obvious things would be to kind of catch
11:41
all the things that are completely off topic or the
11:44
users intentionally trying to kind of get outside the, the
11:47
main purpose of that router. And so while it's not
11:51
necessarily malicious in nature, right, it might be sort of like,
11:56
you know, misguiding routing decisions and so on.
11:59
So it's kind of like, you know, there's like jailbreaking has
12:02
this extreme sort of connotation get a lot of the large model
12:06
sense where people are coming in and intentionally like trying to
12:09
hack this thing. But then there's like the more
12:11
practical, like I'm trying to build my app and make sure like
12:13
it doesn't embarrass me, the company and so on, right.
12:19
So in that context, I think jailbreaking could be, you know,
12:23
slightly less, you know, horrific, but still impactful
12:28
and, and, and very relevant. And so I think it's more about
12:31
like actually building things that are keeping the the
12:35
application in in its lanes and sort of collecting sort of a
12:38
number of scenarios that capture that behaviour.
12:42
How do you feel about online evaluation and how needed is it?
12:49
Because I feel like I've talked to a lot of people and they're
12:53
like, yeah. That's.
12:54
Great in theory, but we're not doing.
12:58
I think so here's I'll give you I'll give you what some of our
13:02
customers are doing in in in back into maybe like what I
13:06
think about it, but the think there is there's definitely
13:13
potential value. I don't think so.
13:16
So one of our customers, for example, is, is driving this,
13:19
you know, complex DevOps automation agent system.
13:24
And so they're actually, you know, it's a multi turn
13:28
generation that's executing a shell scripts, commands, you
13:32
know, driving like Python scripts, provisioning Kubernetes
13:36
clusters and sort of wrapping all that up.
13:38
But like, you know, maybe a Jira update in ticket status and sort
13:42
of like moving all those pieces very independently, right?
13:46
And so as you can imagine, in that kind of a scenario, lots of
13:49
things can go, go wrong at many different levels.
13:52
And, and so the thing you, you don't want to do is necessarily
13:57
intervene because by, by catching these things and
14:00
saying, OK, like, you know, at iteration or generation #15 you
14:04
know, if this happens, then you should do that instead.
14:07
You can't anticipate that, you can't plan that.
14:10
And So what you, you definitely don't want to intervene in an
14:14
online fashion in in fact, in many cases, you know, these,
14:17
these planning kind of functions of LLM or these agents that are
14:23
performing a planning function will recover in very interesting
14:26
and unpredictable ways, like those that weren't anticipated
14:30
by the author in the 1st place, right?
14:32
So if they encounter certain failures, they will find
14:35
workarounds that you haven't seen before and that those are
14:37
good valuable and you don't necessarily want to, you know,
14:40
intervene when these variations introduce themselves, right?
14:44
And So what we've seen with them that was effective is actually
14:48
doing kind of an offline the post processing of a lot of what
14:54
happened in production and essentially trying to map like
14:58
different. There's two layers of metrics.
15:01
One is like, was there the overall task completion?
15:04
And then you always, if you can, want to tie that to user
15:08
feedback and some sort of user signal like was the user
15:11
involved and were they actually happy with the output with the
15:14
completion of that task. So the top line is kind of like
15:17
task completion and then the set next set of metrics that we look
15:21
for four is kind of more control metrics around, you know, was
15:26
the Max, you know, number of turn generations exceeded,
15:30
right? Was, was the thing caught in an
15:33
air loop? I, I think there are, you know,
15:35
there are a number of use cases where it could be useful right
15:40
now. Like if if you're doing
15:41
something simple like product description generation and you
15:47
can do the evaluation fast enough on like with the, the
15:51
qualities within certain parameters, then you could do
15:54
it. There's obviously the question
15:56
of tolerances and latency as well.
15:58
And I think that's kind of the reality of, you know, a lot of
16:01
these evaluations are leveraging other language models under the
16:04
hood. And so, you know, in many
16:07
applications, just introducing that if there's a user involved,
16:11
introducing that into the loop, that might be prohibitive.
16:17
There are use cases like and we have customers pushing in that
16:21
direction, like code generation, for example, where your
16:24
tolerance as a user is much higher and you don't expect it
16:29
to be instant. And so there is a level of basic
16:33
kind of cheques you can do like does this thing compile it?
16:37
Does it have imports before you sort of show that to the end
16:40
user? And so you could do that and
16:44
still kind of, you know, be effective in, in, in or we could
16:48
we see people actually before you show the code to the user,
16:52
right? You can perform those cheques
16:54
and I think that would be considered value by the user
16:57
because they they don't want to get like a bunk set of code that
17:01
immediately is wrong. After the fact.
17:04
And that's very true. Well, First off, I'm very happy
17:08
to hear that there is a potential future timeline where
17:13
we can have agents working with JIRA and we don't have to be
17:16
working with it. That is exciting for me.
17:20
The second thing is that I think you you talked about two
17:24
fascinating things where when it comes to these agents actually
17:29
doing what they're supposed to be doing, you let them try and
17:35
complete the task. You never want to go and cut
17:38
them off because they a lot of times get creative with how they
17:44
actually finish the task. It may not be the way that you
17:47
thought they were going to be able to finish it, but they
17:49
figure out a way and they do a little bit of problem solving
17:52
and dare I say reasoning. And then you can get the user to
17:56
say like, has this actually valuable?
17:58
Well, that's up to them. And then goes through all of
18:01
those different metrics that you feel like are important.
18:04
So when it comes to those different metrics, we you hinted
18:09
at this earlier, but what are the ones that you feel like you
18:12
should be looking at? I think it's been already sort
18:15
of probably well documented on this podcast that, you know, a
18:19
lot of the generic benchmarks that talk about model versus
18:24
model performance are good as a index, as a guide to, you know,
18:28
what to select and try and play around with first.
18:32
But but at the end of the day, it has to be application
18:34
specific. It has to be use case specific.
18:37
And like I, you know, I, before I started a acario with my
18:43
cofounder, I spent some time at meta and they're very metric
18:47
oriented culture and I supported a number of teams.
18:50
And on a very regular basis, you know, I would be reviewing
18:54
hundreds of metrics kind of monthly, you know, both, both
18:58
from the standpoint of like, is this a good metric to have in
19:00
the 1st place? And like, how are we actually
19:02
tracking on this metric, right. And so, you know, that has sort
19:07
of drilled into me in number of ways like what are really good
19:10
practical metrics and when do they actually kind of make
19:13
sense. And I think from a, you know,
19:16
there, there is kind of a set of work that was driven, I think
19:22
academically, which kind of talks about a lot of these very,
19:26
you know, I think, you know, generic metrics that make sense
19:31
in, in a paper, but don't make sense for your specific
19:35
application. My favourite is faithfulness,
19:37
for example. Now, what are you going to do
19:40
with faithfulness for your application?
19:42
I guess we have to have a, you know, promiscuity metric as
19:45
well. To, to, to correlate.
19:48
Was going to do it. If you didn't, I was ready to
19:50
jump in. I think I think you know, is
19:53
your LM faithful to to you? So I, you know, and, and, and
19:58
it's 3.5 truthfulness. Like what do you do with that?
20:01
Like how do you actually make that actionable?
20:03
What conclusions do you draw from that?
20:06
And I think as long as I think there's ideally it's a metric
20:10
that actually connects to your end user value or to your
20:13
application value is kind of your North star.
20:17
Now, in practise, that's not easy, but there are some
20:20
scenarios where that kind of aligns pretty, pretty head on.
20:25
So I'll give you an example of like, you know, again, everybody
20:29
talks about. Customer support, it's one of
20:31
the very popular categories of building with LM like issue
20:36
resolution rate for the, you know, the, the customer that's
20:41
trying to get support, right? That's an obvious thing.
20:43
It's existed in a support kind of industry for a long time.
20:48
And if you could actually use that as your North Star,
20:51
everything else that you build can sort of ladder up to that
20:55
metric right now. It's not always straightforward
20:59
to to even measure that, but but ultimately it's less, you know,
21:05
it, it connects directly to your business case, it connects
21:08
directly to what you're trying to do.
21:09
I think that's like that that's Oplayer kind of North Star
21:13
metrics that you you want to be application specific.
21:16
Some people use LM as a judge. Now that has its own set of kind
21:20
of problems in in here around it, but we we can we can get
21:24
into that as well. I think the next tier of metrics
21:28
that is that people should be really investing a lot of time
21:33
into is more like unit style, unit test style metrics and
21:38
evaluations. And, and they're, you know, as
21:40
much as they can be like pass or fail versus like an A score
21:44
that's always going to be more effective and actionable.
21:47
And so I think, you know, I think open AI evals have sort of
21:51
started that that trend are, you know, many people, it wasn't a
21:55
huge leap of imagination. Many people have kind of jumped
21:58
to to that, but I think you know, as much as you can put
22:01
into these kind of deterministic kind of evaluations of the
22:07
output and do that up front and do that really rapidly in your
22:11
development cycle. That's going to be again, that's
22:15
a software engineering sort of principle that's tried and true
22:18
and has been indoctrinated into people's minds for for many
22:22
years now. I think it applies really well
22:24
in the LM application building space as well.
22:27
And so if you can put like for example, so I don't sound too
22:30
abstract, right? Like again, if I'm generating
22:33
based on as the output to some next phase in my application, is
22:37
it have the the right format? Does it actually have the right
22:42
kind of sections that you expect for that output?
22:44
Does it have the right sort of structure, etc.
22:47
Does it have the right components and and do all of
22:50
those cheques right before you run anything more complicated,
22:54
before you put any more involved and latency kind of heavy
22:58
metrics on top of it. So I think those are the two
23:01
main tiers. I think that if people kind of
23:03
invest a tonne into kind of unit test style metrics and kind of
23:08
embrace the fact that this is going to be a custom
23:11
environment, your metrics are going to be customer specific to
23:14
your application. Now a lot of the data scientists
23:18
are, that's nothing new. For a lot of like machine
23:22
learning engineers, that's nothing new.
23:24
I think for a lot of the software engineers that are
23:26
coming to the AI party, it is kind of a new practise that
23:32
needs to be put in place. Now.
23:33
You can't also, you know, perfect your metric over six
23:37
months period because as I mentioned earlier, things change
23:39
very rapidly. And so, you know, you want to be
23:43
able to kind of get put something in place, iterate and
23:46
adjust and keep changing your evaluation kind of metrics as
23:50
well and keep adding to them. Because as new features get
23:52
introduced, new kind of vulnerabilities get exposed in
23:55
your application and so on, you want to continue building into
23:58
that sort of metric set. But that is kind of a
24:01
replacement for how you, you know, you build your baselines
24:05
and it obviously allows you to iterate on that baseline.
24:10
It's funny you mentioned that it's not new for data scientists
24:15
and machine learning engineers because we had a panel at the
24:20
last AIM production virtual conference.
24:23
And at the towards the end of the panel, a few people were
24:27
talking about how it's, it's almost like same, same but
24:30
different when it comes to doing traditional machine learning and
24:36
what you need to do with like your golden data set.
24:41
And now you go out and you do looms and you're using AI and
24:46
all of a sudden you're like, wow.
24:47
I think our evaluation data sets are like a golden data set.
24:52
And so it's like new term old thing that we're doing at the
25:01
end of the day. And the other piece that I
25:06
wanted to call out when it comes to metrics is that we've had an
25:11
engineer on here, Sam Beam, and he talked about how for him, the
25:17
most important thing that he's doing is choosing the right
25:20
metrics to be monitoring. And I think you kind of called
25:26
that out really well is not only do you choose the right metrics,
25:32
but how are you going to be actually tracking those?
25:36
And how are you going to be able to know if what you're doing is
25:40
moving the needle on those metrics and you're making sure
25:43
that it's the right metric and you have the right visibility on
25:46
that metric? You can get complicated and I
25:49
think, you know, that's why people want like, you know,
25:52
out-of-the-box, you know, rag metrics, for example, you know,
25:56
in, in, in some cases, that's a good starting point for, for
26:01
people to, to, to begin kind of building and experimenting
26:05
around. So they actually know, you know,
26:07
what's going on in the system, right?
26:09
Like, because otherwise you're kind of flying blind and you
26:12
don't really know if you're, you're, you know, you're making
26:15
anything that's meaningful, you're building, you're making
26:17
improvements that are actually making any sense.
26:19
I think the metric selection, like it's kind of as I connected
26:25
to, it's kind of a, it takes kind of the application side of
26:29
your brain as well as the data science side of your brain and
26:35
kind of has to bring those two together in terms of like what
26:37
matters for my application and my use case or my, my business,
26:42
right? Like what is actually the most
26:44
important thing because first of all, that changes right over
26:48
time and we often find the wrong metric and we have to iterate on
26:54
it and change it overtime to to kind of calibrate and find what
26:58
actually works. So, you know, I'll try to come
27:01
up with an example, but you know, it was the customer issue
27:06
resolved was kind of an example I gave earlier.
27:08
Now you could find a bunch of like if it's a customer support
27:11
case, is that really like the the North Star and all the
27:14
cases, right? Like, well, you know, how long
27:17
did the resolution take? Was the customer like really
27:20
frustrated at the end of it? Right.
27:23
And so your issues get resolved, but at a high rate.
27:26
But you're, you know, the resolution takes two hours for
27:30
each customer, an average. I'm just, you know, making
27:32
something up here, but they're all back.
27:34
Yeah. Well, yeah, I think that
27:36
wouldn't be resolution, right? Like that would be kind of a
27:39
repeat sort of call on the same issue.
27:42
And so I think people, people need to embrace the fact that,
27:46
yes, this will be custom. Yes, the metrics will be custom
27:50
and we'll have to change overtime and that's and that's
27:54
OK. So we had Sanket on here from
27:57
Spotify and he was primarily working on the vector store and
28:04
the recommendation engine that they have and how they were
28:07
using their vector store and the embeddings to kind of power all
28:11
these different use cases. And Spotify, maybe it was the
28:14
personalised DJ or it was search or it was Recommenders or your,
28:20
I don't know if you're a Spotify user, but they have like your
28:23
daily mix and they have your weekly update type thing.
28:27
And so there's all these products that are able to go off
28:31
the back of it. And one thing that he said which
28:34
resonates with a lot with what you're saying is he spends more
28:38
time on evaluating if the retrieval is actually good than
28:46
he does actually like trying to figure out the algorithms or
28:49
doing anything else really at his job.
28:52
For him, the most important thing is like the evaluation of
28:57
the system. And I know it's a little bit off
29:01
because he's talking about a recommender system, but there
29:04
still is that retrieval if you're doing rags or it still is
29:08
like trying to figure out what metrics are important, how do I
29:11
evaluate them, and then spending the time to really get familiar
29:17
and intimate with those metrics. Yeah, yeah.
29:22
I think you know, retrieval and is is kind of convenient in the
29:29
sense that it has a set of standard kind of information
29:32
retrieval metrics that have been defined and for for.
29:37
But now you have to like what we see our customers trying to
29:42
spending the most time on in, in it sounds similar to to to the,
29:47
you know, the scenario you outline is they defining what
29:51
like, OK, I can measure, you know, the my top most relevant
29:56
results in terms of, you know, MRR, you know, and DCG, et
30:01
cetera, type metrics. But defining what is most
30:04
relevant for a particular user, right, For a particular query,
30:10
that is kind of a lot more ambiguous and a lot more
30:14
open-ended, right? Like if I am on Spotify and I
30:16
get these recommendations, like are they, is that really what I
30:19
wanna listen to? Does that really want to align
30:22
with my taste preferences? And that's, that's never going
30:26
to be ideal, right? And I think the, you know,
30:31
defining those datasets. And this is where we're, we're
30:34
using a lot of the synthetic data to, to help people kind of
30:39
define the initial set of data based on their kind of
30:43
enterprise, their company and knowledge that they can use for
30:46
retrieval later. So essentially kind of like
30:50
taking their knowledge and generating sort of the retrieval
30:53
datasets that then ladder up or line up to the standard
30:56
retrieval metrics. So you can isolate that
30:59
retrieval step within your larger kind of pipeline within
31:03
your larger application. And then you can measure it.
31:05
But you still need subject matter experts, you need people
31:10
who understand kind of that field, whether that's actually
31:13
relevant. And they still need to kind of
31:15
philtre a lot of that data in order to make sure it's, you
31:18
know, qualified for for evaluation user for for anything
31:22
else. All right, real.
31:23
Quick, let's talk for a minute about our sponsors of this
31:25
episode, Making It All Happen, Lattice Flow AI.
31:30
Are you grappling with stagnant model performance?
31:32
Gartner reveals a staggering statistic that 85% of models
31:36
never make it into production. Why?
31:38
Well, reasons can include poor data quality, labelling issues,
31:42
overfitting, underfitting, and more.
31:45
But the real challenge lies in uncovering blind spots that lurk
31:48
around until models hit production.
31:51
Even with an impressive aggregate performance of 90%,
31:54
models can plateau. Sadly, many companies optimise
31:59
for prioritising model performance for perfect
32:03
scenarios while leaving safety as an afterthought.
32:06
Introducing Lattice Flow AI, the pioneer in delivering robust and
32:11
reliable AI models at scale. They are here to help you
32:15
mitigate these risks head on during the AI development stage,
32:19
preventing any unwanted surprises in the real world.
32:22
Their platform empowers your data scientists and ML engineers
32:25
to systematically pinpoint and rectify data and model errors,
32:30
enhancing predictive performance at scale.
32:34
With Lattice Flow AI, you can accelerate time to production
32:36
with a reliable and trustworthy models at scale.
32:40
Don't let your model stall. Visit Lattice Flow Dot AI and
32:45
book a call with the folks over there right now.
32:48
Let them know you heard about it from the MO OPS Community
32:51
podcast. Let's get back into the show.
32:55
So. But let's let's move into this
32:57
idea of the custom evaluations. And I'm pretty set on the idea
33:04
that if you're not using custom, custom evaluation data metrics
33:12
or whatever, but like you're, you're kind of screwed if you're
33:17
just going off of, as you mentioned the benchmarks, then
33:20
that's pretty much flying in the dark.
33:22
I don't know if anybody actually does that.
33:25
It seems like we've learned by now, but how have you seen with
33:32
the advent of the synthetic data?
33:34
Like are there any ways that you're like OK, this is super
33:40
useful to create that custom evaluation data set and then
33:47
also make sure that it is very use case specific for my app?
33:55
Yeah, yeah. So I think, yeah, no, I I think
33:59
there's kind of using the benchmark metrics and then
34:02
there's like using your own data versus benchmark data and then,
34:08
you know, crafting your metrics for your use case, right, Like
34:11
those are two separate dimensions.
34:13
And I think the, if we look at synthetic data, I think we, it
34:21
allows you to really kind of like, you know, focus on
34:24
behaviours that you care about and describe behaviours that
34:28
like matter to your application, right?
34:30
And so curating a data set, creating a benchmark for
34:34
yourself is a lot of heavy lifting, right?
34:37
So how do you get a jump start on that?
34:40
How do you get that going? And we see people, you know, use
34:45
synthetic generators in conjunction with some seed data
34:48
that they have from like dog fooding, for example, or like
34:52
internal kind of deployments or just their dev experiments that
34:56
they're running. And they're creating templates
34:58
for like what they envision, like the ideal flow should be,
35:02
what the ideal scenario should look like.
35:04
And then essentially using generators to create
35:07
personalities, user personalities or user behaviours
35:11
and populate that template of of scenarios, right?
35:14
I think synthetic data becomes super useful to kind of give you
35:18
kind of a set of expectations and get you going and get you
35:23
kind of tuning your stack, tuning your application.
35:25
And then it gives you that really solid baseline to iterate
35:30
on prompts to iterate on the embedding model you're using,
35:33
right, to iterate on like how you doing query extraction, all
35:37
of those different aspects of, you know, for your particular
35:41
setup, your particular architecture, like they're set
35:44
up. What I think is becomes
35:46
interesting is and and evaluations are like at the core
35:49
of this, right? What becomes interesting is out
35:52
of that you have a lot of failures that still sift
35:55
through, right? Like after you've done all your
35:57
tuning, after you've fixed your scenarios with synthetic data,
36:00
how do you now? Like there's all these stubborn
36:03
really, you know, learned in somewhere in the model's
36:07
weights, like errors that you cannot get rid of, right?
36:11
And what do you do in that case, right?
36:13
And So what, or it's intermittent.
36:15
Something happens in production and it happens once in a blue
36:18
moon, but you're like, you don't know if that's exposing you to a
36:21
lot of risk or not. Like, And So what you can use
36:24
synthetic data for in that case is actually expand and protrude
36:28
and multiply those errors and again, run them through
36:31
valuations and kind of map your risk and impact in those areas,
36:36
right? Like what's going to actually
36:37
trigger those kind of failure scenarios, right?
36:40
Is that a frequent use case? Is that something my users or my
36:43
application will do often or not, Right.
36:46
And then you're essentially building a map, you're enhancing
36:49
your overall kind of model off the behaviours that you want to
36:53
evaluate and expect your, your system to, to have.
36:56
And you can leverage that further for fine tuning, right?
37:00
Like for things now that you have enough data, right to to
37:04
kind of describe your errors and success basis, you can then
37:08
generate instruction to sets in order to fine tune in order to
37:12
get rid of those stubborn hard to see or intermittent failures
37:17
that you're seeing. And so that kind of that kind of
37:20
drives that synthetic data loop that will I think is critical
37:24
and not really leveraged in a lot as as much as I think it
37:29
could be. And a lot of the LNG application
37:31
building right now. So yeah.
37:34
And are you just asking an LLM to go and create this or how is
37:40
your like? What's your favourite way of
37:42
creating synthetic data? So the way we kind of work with
37:45
our customers is actually like give them a set, a suite of
37:48
these generators and then they assemble them.
37:51
They use templates to kind of derive scenarios that are
37:55
specific to kind of their world, their application.
37:58
And then they, they, they deploy this suite of generators and you
38:03
know, and produce today they want.
38:04
So there's a number of different generators fit for purpose.
38:08
And so I can't say that there's one like default way of doing
38:11
it. I like this vision of many
38:14
different generators depending on what you're looking for.
38:17
And you have a generator that is better suited for that use case
38:22
or that whatever it is that you're looking for.
38:24
And you can fan out and create a nice little synthetic data set
38:28
from that and potentially prompt it to get more use case specific
38:34
or very pertinent to what you're trying to do.
38:37
O Speaking of which, it feels like when it comes to agents,
38:45
the having this synthetic data there to also help the agents
38:49
stay on track would be quite useful in their evaluation.
38:53
Yeah, yeah. So I staying with like the
38:58
customer support scenario because, you know, it's frequent
39:03
out there and we have some customers that are building just
39:07
that. And the, you know, the, the
39:11
thing you could do is actually think about, you know, your
39:14
historical conversations, for example, and, and take a look at
39:18
like this was a product return, right type of conversation.
39:22
This was like, you know, a shipping issue type of
39:25
conversation. And collect those scenarios from
39:28
your historical kind of customer interactions and then build
39:32
those into templates. Like this is going to be a
39:34
template of a complicated return.
39:37
This is going to be like a template about a complicated,
39:40
you know, policy question. And then use like a language
39:44
model based generator to simulate personalities and
39:48
users. And like, don't you know this,
39:52
this customer is very aggressive.
39:54
This customer is kind of a little bit more and you know,
39:58
has this sort of behaviour to one, right?
40:01
And so essentially build a collection of simulated users
40:05
that are mapping to those scenarios, right?
40:08
And then so all of a sudden you have sort of like different
40:11
aspects that you're trying to test and then you have a lot of
40:14
these user personalities that are populating these different
40:18
aspects that. You're you're you're testing and
40:20
so then you can kind of drive that against your model under
40:26
test, the model being evaluated, right, and see exactly kind of
40:31
like what happens in that case right now, this this doesn't
40:36
have to be like an agent based system on the other side.
40:40
This could be something more simplistic, but it also can be,
40:45
you know, it like the the question of how you implement it
40:47
on the other side is a bit of a separate question, But
40:50
definitely the the the nature of this setup allows you for like a
40:56
like a mini multi churn deep interactions that you kind of
41:01
probing and expanding into with the, you know, model you're
41:04
evaluating. She she I do love the idea of
41:08
giving an LM a bunch of different personalities so that
41:12
the agents ready. So there is one one piece that
41:16
I'm thinking about when it comes to this and I feel like you've
41:21
also mentioned it, but it's not exactly like I would love just a
41:26
a clear cut idea of what are emerging age and design
41:32
patterns, especially because it feels like we've got so many
41:37
clear design patterns. If you're doing traditional mill
41:40
these days, hopefully people are doing it, but you never know.
41:46
Could still be just using an Excel spreadsheet to track their
41:50
experiments. And you've also got clear design
41:56
patterns when it comes to rags. Yeah, every day I feel like
42:01
that's getting more and more advanced.
42:03
I think people are accepting that knowledge graphs are a very
42:07
useful piece of the RAG architecture, but when it comes
42:11
to agent architecture or system design, I'm a little bit
42:15
unfamiliar with that, so I'd love if you can break down what
42:19
you've been seeing. Yeah, Yeah.
42:21
I'm, I'm also curious because you talked to maybe we can come
42:25
back to this question, but I'm also curious because you talked
42:28
to a lot of people in this space to see, you know, how far do you
42:33
think people are pushing kind of the Asian architectures or
42:37
whether it's sort of little bit, you know, on, on, on the
42:41
emerging side right now. But the, and, you know, I'm
42:46
informed more by like what our customers are doing.
42:49
I'm informed by like what I'm seeing in deployment in, in
42:54
there's a lot of really, really exciting things.
42:56
Like last year, you're probably like, you know, super tired,
43:01
different hearing the term rag. But this year, like, you know,
43:03
it's all age at the age of age. And I think, you know, it's you
43:07
know, we're about midpoint. I think it's the the the, the
43:11
hype cycle is also at its peak. I would argue the, but I think
43:16
if you look at what's in reality, you know, there are
43:20
some deployments, but, but it's still early days.
43:22
And I think, you know, I like the, I think if you think about
43:27
LLM as a stack, agents become like a natural sort of paradigm
43:33
with analogies to like programming paradigms where you
43:35
breaking up your system into modules and these modules are
43:39
collaborating and they're passing messages and they're
43:41
actually then delivering the end sort of application versus you
43:46
having like this massive prompt that you're somehow templating
43:48
in from a bunch of different parts.
43:51
And, you know, you can't really control exactly what The thing
43:55
is going to look like. And it's kind of a template
43:58
based monstrosity that develops agents becomes like a nice
44:02
abstraction that people can reason about and write code
44:06
around. And again, has very, very nice
44:09
parallels to how software systems have been built for some
44:12
time. The now I think there's, you
44:17
know, a number of patterns. I think I like actually how
44:20
Android ink breaks it down in terms of, you know, typical
44:25
patterns, right? There's the reflective sort of
44:27
agent, which is the simplest 1, which is about, you know, like
44:32
having the Asian go back and look at its output and determine
44:36
if there's any errors that need to be corrected.
44:39
Then there's sort of tool use, which we're all familiar with,
44:43
you know, like leveraging whatever tools to search the web
44:46
to do again, retrieval and other things there.
44:49
There's also planning function, which is one big distinguishing
44:53
characteristics, I think of an agent versus just, you know, a
44:56
collection of France is actually kind of that control loop being
45:00
the agent yourself, right? Like typically in these
45:03
architectures, your control loop is fixed and then you kind of
45:06
have a predetermined path when you're calling an LLM to perform
45:09
a particular thing. Now you can take the control
45:11
loop and put it into an LM itself that then decides what
45:15
are the steps and how to break down complicated tasks.
45:18
And then there's multi agents that are being that are
45:20
collaborating and together to perform a task, which, you know,
45:25
people have started using, you know, like human analogies for
45:29
like, you know, there's a manager and there's a worker,
45:32
etcetera, which I find especially humorous because, you
45:36
know, everybody jumps to some sort of job replacement thing,
45:39
which I don't think is intended by, by by that at all.
45:43
But but if you look at like how people are graduating from rage
45:47
architectures, they're typically putting that control loop in the
45:51
middle or they're adding like a routing function.
45:53
Maybe that then decides where which retrieval should I invoke,
45:58
right? Like if I have a bunch of one
46:01
agent responsible for, let's say you know, retrieval of meeting
46:06
data, another one for like meeting summarization and a
46:10
third one for, I don't know, retrieving, you know, documents
46:14
that I care about that are relevant to a particular
46:16
meeting, right? Like now you have like like a
46:19
routing agent and coordinate between these and decide which
46:22
one to invoke when and to assemble that into a larger
46:25
output. And so that's kind of this is
46:28
like a rag ish architecture that evolves to an agent based one.
46:33
And it's a natural kind of way to add complexity without going
46:37
too far too deep. But then there are I see people
46:41
actually build these, you know, agents that run on a actual kind
46:46
of compute environment and perform actions on that compute
46:50
environment. And so in that, you know, in
46:53
that case, there's a lot of interesting coordination issues
46:56
that emerge both in terms of like, you know, and getting
47:00
output from this, you know, you see two node that I didn't
47:04
expect, how do I handle it? Let me try something else and so
47:07
on. So there's a lot of like this
47:09
error handling of recovery things.
47:11
And so and, and the planning function has to be really
47:16
thoughtful with a lot of like escape hatches, if you will,
47:20
because it can get like into, you know, it's often that this
47:23
thing can get into sort of an endless air loop or kind of, you
47:28
know, start looking at it, work around after work around and
47:31
then just get kind of confused with the initial intent and get,
47:34
get, get, get off track. So, you know, there's a lot of
47:38
defensive sort of, I wanted to say code, but the right word is
47:43
prompts you have to sort of build into it.
47:47
But there is there is some scaffolding that you can sort of
47:49
also inject in terms of, you know, code kind of putting in
47:53
controls as well to make sure you're, you're not, you know,
47:56
passing those thresholds of safety.
47:59
They are so fascinating men, especially because right now I
48:05
do like that you said we're at peak height.
48:08
It doesn't necessarily feel like I've seen agents working
48:13
reliably yet in a way that people are outsourcing a lot of
48:19
stuff to them. And I 100% agree with you on the
48:24
idea of having modular agents or prompts or almost like
48:30
Microsystems or microservices instead of the mono lithic
48:36
gigantic ROM and then go figure it out.
48:39
And the idea of passing information or having like just
48:43
an agent that does something very, very small and specific so
48:47
that ideally it's not getting off track because it knows what
48:51
it's going to do it it's you've broken this task down into very
48:55
small pieces. And so hopefully it can't go too
48:59
far off the rails. I was talking about this just
49:03
last week and my thinking got a little bit clarified on the idea
49:10
of like, Oh well, prompts. It feels like we're going to
49:13
have smaller prompts and these prompts are going to be very
49:17
pointed. But after talking to a few
49:20
people, they mentioned how look at the prompt is almost like
49:24
data coming in and I recognise that there are two separate
49:30
pieces to a prompt. One is, and depending on the way
49:33
you structure prompt, let's just say that there's probably many
49:37
other pieces. But if we simplify it, you can
49:39
say you've got the task that you're or the outcome that
49:42
you're trying to get from the LLM.
49:45
And then you've got the context that you pass in.
49:48
And so the context can be data. The task is what I think needs
49:52
to get smaller and smaller and be as small as possible, but the
49:56
data can be as big as possible. So you make sure that that task
50:01
gets completed correctly. Yeah, yeah.
50:05
I mean obvious example is like tool or function calling, right.
50:11
And in processing the output of those tools, right.
50:14
Like if I do, if my agent is like checking out a GitHub repo
50:19
and then trying to like ascertain what's in that GitHub
50:22
repo and then go in and modify particular configuration file.
50:25
Wait, you know, the, the nature of the output is a little bit
50:29
unbounded and, and you know, you can't foresee it, but you have
50:32
to be able to that agent to be able to process all of that And
50:37
and then, you know, essentially make decisions again, figure out
50:42
what to do with, with the output of that tool.
50:45
And that could be of arbitrary length and structure and it
50:50
could get pretty complex, but that's kind of runtime sort of,
50:53
if you will state, right, that's not how you're building your
50:57
system. Your system then is much more
50:59
controlled and predictable and it starts to more and more
51:02
resemble kind of code. And then you're, you're
51:05
essentially kind of, you know, my, I like the microservices
51:09
analogy. That's I think is spot on,
51:10
right. Like how do you going to think
51:13
about like the right micro services that you want to have
51:16
in your system and how do you kind of split those
51:18
responsibilities and then focus on evaluating those focused
51:23
responsibilities for that particular agent?
51:26
One thing that I've noticed when it comes to agents and I was
51:29
talking to someone who has has had, they're in the process of
51:35
winding down a startup that was working with agents.
51:40
But The thing is that they were in the fin tech space.
51:45
They're very bullish about agents.
51:48
They were getting real money to put these agents into
51:51
production, you know, like serious contracts of whatever.
51:54
4050K and they stopped that company because they were like,
52:04
we couldn't build a product. Every time we went into a
52:09
company, it was us trying to do something very tailor made to
52:15
that company. And it wasn't like us having a
52:19
fintech agent that could do things because everyone needed
52:23
something new and something different.
52:25
And you think like, Oh yeah, that's the beauty of agents,
52:28
right? That's like exactly why you
52:30
would just want an agent and say go and figure that out.
52:33
But they weren't able to get it working and they weren't bullish
52:37
on the tech being able to work. So they were like, yeah, let's
52:43
wind this down. But then towards the end of the
52:46
conversation, I was like, So what are you doing now?
52:47
And they were like, well, I'm gonna wind this down, and then
52:50
I'm gonna go find a cofounder and I'm gonna start something
52:53
else up in the agent space. Just like, what the hell?
52:57
Yeah. Well, I, I think, I guess that's
53:00
a typical story right in, in in the valley.
53:04
But the, I think definitely the agent framework is going to be a
53:09
very tough space if that's kind of where, you know, this startup
53:14
was playing. And I think some people are
53:16
like, you know, can you make any money on building frameworks or
53:20
is it more like a open source type of, you know, again,
53:24
obviously there are ways of making money on open source as
53:26
well, but it's a different model.
53:28
And so I didn't know exactly how this particular startup was
53:31
pursuing it. I think it's very, very early to
53:36
to make conclusions because the whole agent paradigm of how to
53:39
build with it is going to go through like crazy evolution
53:44
over the next, you know, year or so.
53:47
And so I think we're just starting to see like some
53:50
emerging architectures and patterns and we're starting to
53:53
see some successful projects. I think it's still early days,
53:57
but it is definitely for once your application becomes more
54:02
complicated and it's leveraging LLM for a lot of different
54:05
functions. Then, you know, if you're just
54:08
doing summarization of a meeting, if you're just doing
54:10
some classification, you're using language model for that,
54:13
let's say many people are doing that, then then then you don't
54:18
need a lot of complexity. It's just a, a few prompts that
54:21
you're kind of, you know, through, through together.
54:24
But if you're actually building a lot more and you're trying to
54:28
build these things that are breaking down large complex
54:31
tasks, I think the, the there's going to be a lot of
54:35
experimentation and there's going to be a lot of things that
54:37
are going to stabilise in terms of what works, what doesn't,
54:40
whether there are like, you know, a tonne of great
54:42
businesses there in terms of like framework building.
54:45
That's a that's a very interesting conversation.
54:47
I think it will tend to go kind of the open source plus plus or
54:51
out. I think you also have to kind of
54:54
choose your domain carefully, right?
54:56
Like if we're talking from that business standpoint of which
54:59
obviously know nothing about, but you know, are you building a
55:03
generic agent for anybody out there and, and you're just
55:09
trying to attack the world of possibilities?
55:11
Or as we see some people maybe building agents that are just
55:14
helping with customer onboarding four, I don't know, tech
55:18
companies. O so then your, your focus is
55:21
much more segmented, your use cases that are much more
55:23
segmented and you're kind of focused on that and value you're
55:27
delivering. And so the verses maybe so, so
55:33
in, in in that case, you're also kind of using the agent
55:36
framework internally more so than, you know, trying to build
55:39
it for company XYZ, right? You're you're actually focused
55:42
on delivering that and value and agent happens to be a framework
55:46
to get to that, to that end. So again, without knowing
55:48
anything about this particular company, you know, I can, I can
55:52
share my kind of high level thoughts, but I think as you
55:56
say, hopefully I'm predicting peak hype because I like to kind
56:00
of get more to the, you know, plateau of productivity, as they
56:03
say. But but I think, you know, it's
56:06
definitely a hot and emerging kind of area and it makes sense
56:10
like breaking things down into micro services makes sense,
56:13
whether graph based, you know, state management is going to be
56:18
a solution or that's going to be too complicated for people.
56:21
We'll see. You know, I think that's that
56:25
sounds really well from a from a like a pure purist standpoint.
56:30
If I was to like a point on that quickly.
56:33
I think whether that architecture like graph based
56:37
design will take root, it's hard to say.
56:39
I think a simpler state machine will probably be a safer start
56:42
for a lot of people. I think like a graph based
56:46
design that has lots of kind of edges and message passing can be
56:50
a little bit unwieldy to both design and fully kind of reason
56:54
about so. But again, you know, I might be
56:58
like regretting saying this in a couple of months, so we'll see.
57:03
Dude, Boris, this has been awesome.
57:04
Thank you for coming on here. Thank you, Demetrius, it has
57:08
been a pleasure. Appreciate it.