Latest / Agentic Conversations (formally mlops.community) / Operationalizing AI Agents: From Experimentation to Production // Databricks Roundtable

Transcript

Operationalizing AI Agents: From Experimentation to Production // Databricks Roundtable · Agentic Conversations (formally mlops.community)

0:02
And the reason that I wanted to zoom out for a little bit as
0:06
we're going live, by the way, if you're joining us, very good to
0:10
have you. It's that I like, I understand
0:14
that we're going to dive really deep into how to actually
0:17
productionize agents, put them in production, do it safely.
0:22
There's a lot of challenges and a lot of failure modes.
0:24
And I get that. And this is the purpose of this
0:26
conversation. And so I'm very glad to have my
0:29
panelists here today. So thank you guys for joining.
0:32
We're going to do a round of introductions in a minute.
0:33
But I do want to just plant a flag here, which is, I don't
0:38
know what you guys are feeling. And I'd like to get your sort of
0:41
sentiment here. But from my point of view, my
0:45
mind is blown every single day by these agents and by our
0:51
ability to move so far. I mean just the I feel like
0:54
software engineering is just what we've had it for decades.
0:58
And the way I program now just looks completely different from
1:03
two years ago, a year ago. I just don't understand.
1:06
It's like it blows my mind and I'm amazed by this every single
1:09
day and what it is that I'm coding is changing and how I'm
1:14
coding is changing and every single thing about this is just
1:17
blowing my mind. And we're just, we're able to
1:20
move so fast. And I just want to at least
1:23
share that sentiment because you guys might be feeling it's too.
1:25
And I understand that when you're actually doing it in
1:27
production, yes, we're going to have challenges and yes, it's it
1:30
might be embarrassing if it goes wrong or whatever.
1:32
But every single day now, I tell my partner I'm like, look at
1:36
what this agent did right now. Can you believe she said this,
1:40
right? Like I just, and at some point
1:41
she's like, I don't know, I'm done with you and your agents.
1:43
I mean, just give me a break. You're just so excited about
1:45
that. And I am very excited about
1:46
them. But for this hour, we're going
1:49
to face reality and we're going to see how to actually make sure
1:53
that they're not doing us damage despite all of the excitement.
1:57
So thank you very much for joining today.
2:00
I'm Adam Becker and I'm your host for today.
2:03
I'm going to be the moderator of this panel for the ML apps
2:06
community. So today with me we have some
2:09
Raj, Ben and Apurva. Thank you all.
2:11
How about we start? Maybe you can give just like a
2:14
sentence introduction to you, your name, your roll, your
2:19
title, and perhaps what aspect of agent deployment has been
2:24
keeping you most busy recently. Ben, do you want to start?
2:30
Yeah, sure thing. Thanks everybody for for coming
2:33
on. My name is Ben.
2:35
I am also a volunteer at the envelopes community, but I am
2:38
the Co founder and CTO of a company called Grotto.
2:41
We're a vertical AI company in the multifamily space.
2:44
So active users and buyers of all of the agentic platforms and
2:49
systems and GPU providers that you could think of constantly
2:53
evaluating them and seeing what's best for our team.
2:56
I would say the biggest impact that agents have had for the
3:00
company besides just like making our product, the thing that we
3:03
build possible, like it wouldn't be possible without them.
3:06
But besides that is the internal agents that we have at our
3:10
company that exist in our Slack channels that have given pretty
3:14
much everybody at the company from engineering to operations
3:18
to marketing access to essentially all of the data and
3:21
the context across the system. And if you think about what was
3:26
happening in like even 202524 where a data person or a product
3:31
person or a marketing person would have to go ask an engineer
3:33
to do a query against Bigquery to get you some data, like that
3:36
doesn't exist for us. It's kind of crazy.
3:38
Like all of our data, we have backups, our agents have access
3:42
to those backups and people can authenticate through our
3:44
internal Google account into those backups.
3:46
And we'll see like 9000 plus message threads with our agents
3:51
where they're just analyzing the data they're producing
3:53
spreadsheets, like giving those spreadsheets to customers.
3:56
I mean, it's unbelievable. Like we have a team of six doing
4:00
what I what feels like it used to take 40.
4:03
Like it? It's nuts. 40 and probably years
4:07
to build. Yeah, I mean, it's crazy.
4:10
Like I, I, I Slack used to give you the ability to see the
4:14
breakdown of who was tagged the most that week.
4:17
And I want, I can't find that feature anymore because Slack
4:20
changes every three days. But like, I would love to see it
4:23
because I'm pretty sure our agents are tagged probably 8 to
4:27
10X more than any individual person.
4:30
The the most prolific employee at this point.
4:34
Nice Purva, do you want to go next?
4:36
Yeah, sure. Hi, my name is Purva Mishra.
4:38
I'm a founder of a company called Centech.
4:40
We actually do a lot of consulting work with startups
4:43
and smaller companies, building their AI strategy and building
4:47
solutions for them. And we are also in the education
4:49
space. We are doing a lot of like
4:50
workshops and paintings for companies.
4:53
So currently I'm working with the companies.
4:57
Actually most of our work is focused on internal workflows
4:59
like Ben was saying, like helping in like marketing, like
5:02
it's AI is applicable everywhere in all of the different domains
5:06
that a company has. So a lot of work is the like QBR
5:10
deck generation, like automating the existing workflows which
5:13
were not efficient and we're taking a long time.
5:16
So a lot of it is like talking to these different people in
5:19
different domains, learning what they're currently doing and
5:22
figuring out where we can sprinkle the AI magic and make
5:24
it faster. Yeah.
5:28
Nice awesome samraj. Cool.
5:30
Hey, thanks for having me on. My name is Samraj Morjani.
5:34
I'm an engineer at Databricks working on the ML flow team.
5:39
And you might know ML flow for the more traditional side of ML,
5:42
but we're working more recently on providing end to end Jenny I
5:47
platform for your observability evaluation governance needs.
5:53
Yeah. And so one of the one of the big
5:56
aspects that I work on and and something that I keep on having
5:59
to rack my brain about is quality of agents, both for
6:04
helping our customers build high quality agents and deploy them
6:08
into production as well As for ourselves building our own high
6:11
quality agents to use internally.
6:15
So yeah, that's me. Nice, awesome.
6:17
So just to set the stage then, so that everybody's on the same
6:21
page, summarize you are working on the tool ML Flow.
6:24
People might know it from the classical era of machine
6:27
learning and ML apps. And so you get to see a lot of
6:30
users and what their needs are, and then you're trying to build
6:33
some tools to help them. And you guys have your own needs
6:36
and you're obviously dogfooding your own tools and are building
6:40
towards your own needs. Apurva, you're seeing in high
6:43
depth a few different companies. You're doing consulting for a
6:47
lot of companies, mostly smaller ones, and then mostly ones that
6:51
it might just be their initial, the first time that they're
6:53
actually getting started with AI.
6:56
And then then you are perhaps one of those startups and then
7:00
you're deeply involved in building a lot of agents and
7:03
then seeing how well they run. So we got the entire sort of
7:06
like the entire spectrum here from heavy users to tool
7:10
builders. So this is excellent because
7:12
we're going to get to see sort of like all of the perspectives,
7:14
or at least how all these different perspectives are, how
7:18
the needs and challenges are manifesting across all these
7:20
different stakeholders. So that's, that's awesome to
7:23
see. I want to start us out by asking
7:25
the following question. So somebody's coming to you,
7:29
either Ben that might be an internal user or a provide might
7:33
be a new client or Samaraj might be a new user and they're
7:37
saying, I want an agent and I want that agent to do X.
7:41
What is that X? Ben, you already started to give
7:43
us a little bit of a sense, but I want to make sure that it's as
7:46
vivid as possible so that we're all on the same page.
7:48
What are what are companies dreaming their agents be doing?
7:53
Whoever wants to start, then you want to go for it.
7:58
Yeah, sure. I mean for us, it is an
8:00
extension of what I just said. It's all of the things that you
8:04
could imagine typically marketing or customer success
8:09
wanted to do but didn't have access to because it required
8:12
like engineer resources for for engineer resources for bad
8:19
reasons. You know, there's good things
8:20
like we need to upgrade in a system where we need to modify
8:25
the database because a user's data, I don't know, like was was
8:31
assigned to the wrong thing. Like you don't want an agent to
8:33
be able to do that. At least we don't want an agent
8:34
to be able to do that. Some companies might and so
8:37
great, like it has to go through an engineer.
8:38
But like, you know, for us, a user was promoted.
8:42
We don't want that to be like a part of our UI necessarily, but
8:45
a user was promoted, we have to modify their e-mail.
8:48
Like, yeah, MCP server requires human approval.
8:52
Like there's no reason that a, an engineer for us is required.
8:57
We so great. We have MCP servers, we have
8:59
easy ways to deploy them with fast MCP.
9:01
We deploy those MCP servers. They are like right tools.
9:05
So they require a human Click to approve.
9:07
We tell our internal agents that they exist.
9:09
We can figure them up through like our internal Google auth.
9:12
And then they were like, OK, dude, go go agent do this thing.
9:16
It comes back. It's like you have to click this
9:17
link to approve it, clicks it and The thing is done.
9:19
And no engineer was harmed in the, in the making of that of
9:22
that server call, things like that.
9:24
So it's simple, right tools that we are comfortable giving non
9:29
engineers access to make those rights that are never DV queries
9:33
for us. They're always very contained
9:36
isolated functions that call like internal functions that
9:40
make DV calls. But like you can't, there's no
9:42
like sequel injection risk or things like that.
9:45
And then a bunch of read access that used to be very difficult
9:48
or annoying to set up. Those for us are by far the
9:51
biggest. And then we have things like for
9:54
marketing, we want to test out different versions of our
9:57
website. So our internal agent has the
9:59
ability to host our website separately on its internal
10:03
domain, like exposed through Cloudflare.
10:05
And then our marketing team can go and modify and like make
10:08
changes, test SEO, see how it comes up with different search
10:11
queries. And then they can port that over
10:12
to a pull request that actually modifies our site.
10:15
So it's giving them access to do.
10:16
Those are real, all real examples that we do today that
10:20
would have required engineering. I'm curious then, is it is it
10:23
fair to say that it that like those needs emerge from both
10:27
camps? That is the engineers themselves
10:29
are saying, why are you giving this to me?
10:31
I don't need to do this, just do it.
10:33
And then maybe the non engineers are saying why do I need to go
10:35
to this engineer? I can't take do it myself.
10:37
Are you seeing them kind of bubble up from both both sides?
10:41
I would say it's, I mean, given how small we are, it's
10:45
definitely more the latter of, I mean, it's both.
10:48
It's, it's just a cultural shift.
10:50
Like I, I pushed everybody at our company to anytime they
10:54
think they want to ping me or my team to do a thing, I say like
10:57
first, just like go see if the agent can do it.
11:01
Also, I'm sure a lot of companies have this, but like
11:03
our agent at this point, it constantly is updating its own
11:06
memory based on interactions. And like it's gotten to a point
11:08
where it just knows to ping me. Like, like it just messages me
11:12
in threads if it thinks it can't do a thing or doesn't know how
11:15
to do a thing. So it's sort of at this point
11:17
is, is self enforcing. So what somebody on our team
11:20
will go ask the agent, they'll go back and forth a bunch.
11:22
The agent will either do it great or it won't.
11:25
And it'll literally type out and send a message like at Ben, can
11:27
you jump into this thread? Like, how do we sit?
11:30
Like how do we do this? And that's sort of how it's been
11:33
going and and and self improving.
11:35
There's like a Ben MCP. It's just there's.
11:38
It's actually just a Ben. It's just a Ben handle on Slack.
11:43
Fun. OK, Apurva, what, what, what
11:45
kinds of use cases are you saying and what are kind of like
11:48
the the grandeur of the aspirations that people have
11:51
when? They come I'm I'm just wondering
11:53
get it like loan overtime that it has to tag Ben?
11:57
Yeah, it did like so one of its intern, you know, like when you
12:01
set up a one of those systems, you often won't give it like a
12:03
sole dock, like a sole dot MD and one of those sole and in
12:07
ours not unique is just like every time you have an
12:10
interaction go like think about what you could update in your
12:15
memory about it. And it just has started
12:17
naturally tagging me with questions like that that was not
12:20
like explicitly written. It just sort of picked up on
12:22
that. I'd like it to stop doing that,
12:25
but like it, it does do that. It's interesting what they pick
12:28
up on, you know. Yeah, totally.
12:31
So the kind of use cases that I'm getting are more from the
12:33
business perspective. So the kind of like people that
12:36
come to me are like business owners or founders and they have
12:39
heard about AI, they have tried exploring it in their company
12:42
and they're trying to find some export who can build something
12:44
for them. So it's mostly like a recent
12:46
company founder, they had a SAS company, but the users are
12:51
non-technical. So it's a SAS company, he's
12:53
technical, the owner, but the users are non-technical and they
12:56
were like struggling providing support to them because the
12:59
questions would be like oh, where is the drop down menu or
13:01
where is this button? Like they are not that
13:03
experienced with using a computer software.
13:05
So they want to build an assistant which can do stuff for
13:08
them, all the complex tasks that they have like job assignment or
13:12
like e-mail template creation, whatever inside the SAS
13:15
software, they want to automate all of that and assistant whom
13:18
the person can go talk to and can do all of that stuff.
13:20
So, so a lot of my work is like narrowing down the scope and not
13:25
like building what they want is like an end to end solution
13:27
which can like do everything for them.
13:29
So I have to like help them like narrow down the scope and like
13:32
build like smaller agents that we can like put together later
13:35
on based on like how they're working.
13:36
Are they working good? Can they put the put, put them
13:39
together and put like an orchestrator on top?
13:41
So yeah, so that's like the kind of work that I get.
13:46
Like a lot of it is like is more about like communication,
13:49
building the POC trying to narrow down their scope and then
13:52
actually building the solution and like then going on to like
13:55
evaluation. And stuff.
13:56
So it's kind of like they, they come in and they want, it's
14:00
almost like they're not yet sure how to scope the agent.
14:04
And so they imagine that one agent can do everything, and
14:07
then they. Start to figure out the problem
14:09
and they think I'm gonna find this export I'm gonna solve that
14:12
problem yes and those problems most of the time AI is the way
14:18
to go. Sometimes AI is not even the way
14:20
to go, but even even for the problem like what they have
14:26
heard is like agents work perfectly all the time and like
14:29
you can just start with them. It's better to like narrow it
14:32
down, reduce the scope, build something smaller to begin with,
14:37
see how it's doing, how the employees are working with the
14:41
agent, and then like go with that.
14:43
Because a lot of it is also the behaviorship that Ben was
14:46
talking about. Like the employees are getting
14:48
used to like using AI tools and stuff.
14:50
They know about hallucinations every time.
14:52
Like every time the agent was answering something wrong, they
14:55
were saying, but the issue was the documentation.
14:59
It was just not up to date. With startups, that happens a
15:01
lot, right? Like you have documents written
15:04
about your interface and the interface changes so quickly.
15:07
Like you the button is gone from there or the drop down menu is
15:09
gone. So a lot of it is like them
15:11
understanding what context management is, what context is
15:13
like how like your documentation needs to be updated for the
15:16
agent to work well. So it's always easy to start
15:20
with something smaller so that the employees can get used to it
15:23
and then like explore more. Yeah, Samaraj and Ben, have you
15:27
guys seen similar things then where it's like you have high
15:32
aspirations for what an agent can do and then you start to
15:34
say, OK, let's start a little bit more reasonably.
15:38
And if so, what is driving that pull towards the reasonableness?
15:42
Cuz you might not have an aporva there, right?
15:45
Who's telling you actually there's some other pressure from
15:49
the environment that then forces the better constraints?
15:53
Is that how you're seeing things and what does it look like there
15:56
on that? Yeah, definitely.
15:58
I think, I think the quintessential example is like
16:00
you can try and replace an entire role or a person with an
16:04
agent. And and you know, I think a lot
16:08
of customers we've seen have built out prototypes in, in
16:11
hackathons or, or for demos and they notice like, oh, OK, it
16:15
works pretty well while we're testing it.
16:18
But as soon as they kind of release it more broadly, it
16:21
flops like really badly because people use it in ways that the
16:26
developer never expected. It's obviously non
16:29
deterministic. It will hallucinate.
16:31
You run into these issues with like out of date documentation
16:35
or even issues with just prompt engineering.
16:42
So I think a lot of the the reality check really comes when
16:47
you see these issues and quality and it kind of pushes people
16:52
towards these. The kinds of things like Aprovo
16:54
was talking about where you do have to think meaningfully about
16:59
design. You also have to a lot of times
17:03
work on aligning with your stakeholders on what the agent
17:07
should actually behave like and what it should do.
17:10
A lot of times the customers we're working with, they have
17:13
separate teams developing the agents and separate teams which
17:18
have the domain experts that actually know what the agents
17:20
are supposed to do. And so getting these teams to
17:22
actually talk to each other is one of the biggest challenges.
17:26
Because if you just have a developer going and blindly
17:30
creating an agent for a medical chat bot and they're like, OK,
17:33
it looks good enough. Someone's going to come back and
17:36
say no, this agent's, you know, doing something completely
17:39
wrong. Yeah, so you're saying the first
17:42
reality check in your mind is often just the quality
17:46
degradation or so they expect it to perform a certain way?
17:50
It's subpar. I wanna just plant a flag here
17:55
cuz I'm very curious about you guys experience.
17:58
Do people actually monitor quality of agents?
18:04
Is that actually a thing like a Poorva?
18:06
Like do you demand that everybody does it?
18:08
What does that actually look like?
18:09
Like Ben, are you that just part of like your best practice?
18:12
You don't even roll out an agent unless it has quality.
18:15
What does it mean? And is that something that we
18:17
talk about or is it something that people actually are?
18:20
Is it like second nature for engineers?
18:23
Help me make sense of this at least.
18:25
Well, I think it go for it. At least when I'm building the
18:31
solution for them, we try to have an observability layer on
18:35
top just to make it because like what some Raj was saying, like
18:40
especially in my case, like it's their first AI solution.
18:44
If it doesn't work, they would not trust this technology later
18:48
on. But like any of their other like
18:49
needs. So it, that's why you need to
18:51
like narrow down the scope and make sure like it's working and
18:54
at least like either make it work as well as they needed to
18:58
or they have their expectations. Like drop down the expectations
19:01
level it to like how the technology can like work and
19:04
then like have an observability layer on top and like start
19:07
pushing them towards like evaluation and stuff.
19:10
Because it needs to be reproducible in the sense like
19:14
if they say, OK, this didn't work or the output on that day
19:17
was bad, like we should be able to trace and like figure out
19:20
like where, which agent decision or which tool call like where
19:24
was it messed up? So observability there is like
19:28
important. I think it depends on what
19:36
you're doing. Like I think I mean, we have
19:40
unbelievably strict observability for all of the LLM
19:46
calls and pipelines that go through our product.
19:50
For sure. You can trace every decision
19:53
back to a call that's flushed into a lake house.
19:56
And like that's great. But also the agents that we've
20:00
run in our in our product are way, way narrower in scope than
20:05
the agents that we have running live in our Slack channels
20:09
because the tolerance for error is so much higher like.
20:15
When I write with Claude code, if it doesn't respond for 7
20:19
minutes and then just make some stuff up like it's annoying but
20:24
it's fine. Like my life is still 10X more
20:27
productive than it used to be. Even if occasionally or even if
20:31
frequently. It makes mistakes.
20:32
It's always making mistakes. That's OK.
20:35
The agents in our Slack channel are making mistakes too, like
20:38
all the time. And we have a small, and
20:41
obviously this is different for large enterprises, but we have a
20:43
small team and everyone's highly has, has high levels of
20:47
ownership and high levels of trust in the company.
20:49
And they're responsible for the outputs of those models and for
20:53
fact checking and for validating.
20:54
And like anytime their mistake, they're like, OK, system, update
20:57
your memory, go fix the thing. But we can iterate on that.
21:00
So like, no, we have very little observability for our internal
21:04
agents because our internal agents live in our Slack channel
21:07
with read only access to stuff like they can't go and make real
21:11
consequential decisions to the business.
21:13
They can't send the e-mail. Like we still send the e-mail
21:16
And, and that's how we kind of manage that.
21:20
But our, our tolerance for error in our product is, is 0.
21:24
And so it's a totally different game.
21:26
And those systems are way more narrowly scoped, so I think it's
21:30
just super dependent on like. Yeah.
21:31
And then? But then what does it look?
21:33
Like so narrow scope and then what?
21:36
So then you, you enforce a layer of visibility and you guarantee
21:41
that a person is there to watch, to measure, to evaluate the
21:47
quality of each of these. Like, how are you actually?
21:49
And then even organizationally, how does it, how does it happen?
21:53
For the ones in the product, we go through sufficiently rigorous
21:58
testing before changing any of the LLM systems in our product
22:03
such that like we have confidence to ship it the same
22:07
way. Like we, we treat the agents
22:10
that are in our, in our, in our, in our product are treated
22:14
incredibly similarly to just traditional ML models, right?
22:17
Like every PR that touches any prompt or touches any downstream
22:21
thing of the prompt runs through our evals and those evals have a
22:24
very, very low tolerance for failure.
22:27
CI fails, you can't merge your PR and then on top like so
22:31
that's that's pre emptive. We then obviously have sentry
22:34
alerts for any time things like schema coercion is failing too
22:38
frequently or like changes in calls like like you would maybe
22:43
back in the day call this data drift, but now you would
22:45
actually call it model drift because the models literally are
22:48
changing because we're using providers who change the models
22:50
behind, you know, non static API endpoints.
22:54
So when those are changing frequently and we're getting
22:57
century alerts and then even on top of that, we have, you know,
23:01
model like our, our, our agents in our product makes a bunch of
23:04
decisions throughout the day. And we have our internal agents
23:08
at night go and read the decisions from our product
23:11
agents and send us Slack messages of like, here's what I
23:14
think your system did wrong in production yesterday.
23:18
And then we so, so all of those are pieces that happen every
23:20
single day for us. OK.
23:22
So that, so that might be a good, a good place to kind of
23:25
pivot a little bit because what what I'm hearing is we need to
23:29
have high levels of trust. We need to do a variety of
23:33
different things to establish that trust.
23:35
Some of these things are similar playbooks to what we've done in
23:38
the traditional ML apps, classical ML era, even pre Gen.
23:42
AI. Some of it might be different.
23:45
What are some principles then, if you guys have seen at all
23:49
sort of emerge in the last couple of years that can allow
23:53
us to establish that high level of trust?
23:55
We're talking about observability.
23:57
What else do we have? Yeah, I think, you know, a lot
24:02
of the the success stories we've seen are built around this
24:06
concept of eval driven development.
24:10
And actually, if you take a look at Jen AI, I'm going to make a
24:13
claim it's actually not all that different than classical
24:16
software or traditional ML. And our playbook is actually,
24:22
again, not that different, right?
24:24
If you think about maybe like take for example, vibe checking,
24:27
right, or vibe testing or whatever it's called in
24:32
classical software, that's the equivalent of clicking around
24:34
and praying that your software works, right?
24:37
And we have a playbook for software.
24:39
It's unit test integration test, production telemetry.
24:42
So it only makes sense that for Jenny I applications, you have
24:46
the same exact guarantees to ensure that you trust what
24:49
you're putting out in production.
24:51
What that looks like in practice is like unit tests are kind of
24:55
like your evaluations. You still have production
24:57
telemetry over your Jenny I application, but you also have
25:02
additional components that help you gauge the quality.
25:06
So you know, these things are are actually pretty similar at
25:10
the end of the day. And even for our internal use
25:13
cases, we're still using a lot of the traditional MO techniques
25:16
like having training test sets, AB testing, K fold validation,
25:21
like none of this stuff has really gone away.
25:23
It's just applying to a new domain with slightly different
25:26
challenges of, you know, your outputs are non deterministic.
25:30
Users will use your agents in ways that you never would have
25:33
expected and at the end of the day, quality is very subjective
25:38
and like I mentioned, requires often requires domain expertise.
25:45
OK. I think also I think it's also
25:46
broadly what Aporva said before, which which is the thing that
25:50
hasn't changed, which is just simplify the problem.
25:52
Like when you, I think a lot of the the like the demo to fail
25:58
production deployment comes from people using some agent kit that
26:03
is super abstracted and builds the whole thing in one shot.
26:08
And when I give talks about that exact topic, really the whole
26:13
thesis of the talk is like stop delaying the product work for
26:18
the end. Like all that the initial burst
26:21
of I think LLM and Agent development did was it, it sold
26:26
a false promise that you could just no longer do product
26:29
thinking. So don't do product thinking.
26:31
Go put a prompt into production. It fails.
26:34
Like why did it fail? You go back and then you end up
26:37
just now being in Figma designing user journeys.
26:40
Like had you just designed the user journeys upfront, you would
26:43
have realized that you didn't need 1 broad prompt with like 76
26:46
different conditionals. You could just break that
26:48
problem down into normal problems, most of which end up
26:51
being like some abstraction over a classification or regression
26:55
task. And then you could just use LMS
26:57
for those classification and regression tasks, and now you
27:00
once again have test sets that you can validate against.
27:03
That's obviously not every problem, but it is like a lot of
27:05
problems. LMS state machines, like they're
27:08
all just decisions to change state.
27:10
If you are just testing transitions between states
27:14
really robustly, you sort of know when things fail because
27:18
you know when the state change was wrong.
27:20
So, so like if you just do that up front, like a pro was saying
27:23
like it it, it's not everything, but it's a lot of it.
27:26
Like that'll definitely help pretty it.
27:28
It's like what works for us. Like really, really well.
27:35
Aprova, did you want to respond to that?
27:37
I I think everything is covered mostly like unit tests,
27:40
integration tests, LLMS judge. If like you cannot do equal,
27:44
equal to like assertions, LLMS judge is the way to go there.
27:49
Otherwise, like try to break down the problem such that it
27:52
leads to a binary output that you can test.
27:55
Otherwise it's if it's subjective, it has to be LLM as
27:58
a judge to evaluate, yeah. Actually, I love that that you
28:04
said that because I think one of the most powerful things that
28:06
has come out lately, you know, I noticed a lot with my own coding
28:11
and, and cloud code is having that feedback loop, having
28:15
verifiable goals that can help tell your agent or your language
28:20
model. Like have I actually
28:21
accomplished what I set out to do?
28:23
So with every task that I do these days, I will always give
28:27
it some way to verify its own work, whether it's unit tests,
28:31
whether it's, you know, using Playwright to actually click
28:36
around and and see like, is this the expected behavior in my UI?
28:40
Even for building out agents, I will have judges run at the end
28:46
of every change so that there is a feedback signal coming back to
28:51
my agent and I can continuously iterate and help climb on that
28:54
feedback signal. So it's, it's really, really
28:57
cool how much progress we're able to make with those feedback
29:00
loops. And I think it all stems back to
29:03
that that eval driven development point.
29:07
Yeah, that that's such a valid point.
29:08
Like my team has a pretty strict rule set now where any PR goes
29:15
through essentially a Ralph loop of like PR agent, sub agent code
29:21
review, modify those changes, sub agent code review like until
29:24
there are no like high, you know, clawed, assigned high
29:29
quality issues before any person is taking a look at that code.
29:32
We just Ralph loop it essentially until the sub agent
29:35
decides that it's good and it catches so many of the problems.
29:41
Coding, Yeah, it's so much easier to like evaluate in the
29:45
coding space versus like the other spaces.
29:48
So for example, there is there was this client project that I
29:52
was working on in which we had to do automated QBR deck
29:55
generation. So like every like 2 weeks what
29:59
the agent does is like it goes and checks Gong for like history
30:03
of calls with that customer. It goes looks at the last QBR
30:07
deck looks at like what does a salesperson like is talking
30:14
about on slack and like like basically like gets information
30:17
from a different sources and figures out if the QBR deck
30:20
needs to be updated. So like it would also look at
30:23
like support tickets, like if there is a new feature that went
30:26
out, I would add this to the QBR deck.
30:29
So like so it finally generates AQPR deck.
30:33
And now you have to evaluate like is this like up to the
30:35
mark? Can this person take it to the
30:37
customer or or not? So one thing you can do is you
30:42
can put like an LLM as a judge. The other thing is like it's
30:44
hard to like align this judge with what the employee wants.
30:48
You know, you need to align with the domain export.
30:51
So you can also like bake down this problem and do like
30:55
verifiable, verifiable, what should I say?
31:01
Verifiable output. Like for example, it should have
31:04
like the initial slide in which we are talking about this
31:06
company and also the client at the end, it should have like the
31:10
necessary contact information of the company.
31:13
Then in the middle it should talk or there should be a slide
31:15
about the product features. There should be a slide about
31:17
the. So these could be like just
31:19
binary checks, you know, like what's the title this, what's
31:22
the title this, what's the title that.
31:24
So there could be a template that you check against.
31:27
Then you can also check for like, were the graphs included
31:30
or not? Maybe this company needs like
31:33
some sort of like metric system like, oh, these were the support
31:36
tickets that your company provided and we resolved so many
31:39
of them. So there should be a graph
31:41
showing this. So you can always like if there
31:43
is a problem that you're working on and the LLM or the agent is
31:47
giving you an output out of that, maybe you can like you can
31:50
try to break that down into verifiable outputs.
31:53
And if not, then yeah, you have to go for like LLM.
31:57
I do think on your, on your note of code is the easiest thing to
32:03
verify. I'm going to push back because
32:07
the thing that we find, I mean it's really, it's true on like
32:11
maybe simple, like simple systems.
32:13
What we find is that even with Opus 46, like the like the
32:18
latest models, the sneakiest thing that digs into our code
32:23
and that they often fail to catch themselves are like
32:25
asynchronous race conditions and like potential memory leaks.
32:30
It works test pass, things look good, but we have seen a lot of
32:36
problems that we've caught by hand that you would never catch
32:40
in CI because you're not going to run like 150 concurrent
32:44
streams in CI or even really as a part of like the integration
32:47
test for that PR like it's just too much.
32:51
But then you see those problems in fraud.
32:53
I'd literally be right before this was working through through
32:57
a problem that in APR that clawed open that would have
33:00
caused like a very problematic race condition for us.
33:02
So that it's it. There are parts of it that are
33:05
verifiable, but they're very sneaky issues that it that that
33:09
it introduces not. I know that's like super
33:11
tangential, but it's just, it's very top of mind because I was
33:14
just, I was just working on it. That that sort of like, I would
33:17
speed in this as well. Like even if you like add in as
33:21
many evaluations and you have like checked the solution and
33:24
it's working for you and then you put it out in production and
33:26
then the users start using it, you'll find like a bunch of bugs
33:28
that you need to fix that that that is always the case.
33:33
It's. It's interesting because it's,
33:36
you guys remember in the more classical era of supervised
33:40
machine learning, people were always talking about, yeah, data
33:42
scientists are spending 80% of their time cleaning data, right?
33:46
Just doing data processing, data pre processing, just trying to
33:50
build the data set in the 1st place.
33:53
And I feel like just based on what you're saying, I've always
33:57
thought that, well, that time, maybe some of it is gone to
34:00
waste, but a lot of it is product thinking.
34:04
That is you have to just struggle with the framing of the
34:08
problem because you need to start with the data set and you
34:11
need to have a target variable that you're trying to predict.
34:14
And and so we've almost had to do the, we've almost like front
34:19
loaded the work on evaluation in order to get started with
34:23
machine learning in the first place.
34:24
But what is happening right now is we can get started and then
34:28
we realize we never had that why that we were trying to predict
34:31
in the 1st place. And that's where all the product
34:33
thinking goes into. And so now we have to do this
34:35
after the fact. Yeah, that's what I was saying
34:37
before where it's like you one prompt your product into
34:40
production and without doing the product thinking from like a
34:42
user flow perspective. It's the same exact thing with
34:44
machine learning. Like if you didn't actually go
34:47
and explore the LMS. I mean, the thing that we struggle with in
34:50
our team with LMS is the one thing we haven't been able to
34:53
crack is our data analysis. Just like we do really, really
34:56
rigorous data analysis because the patterns in the data that we
34:59
work with are extremely hard to tease out, like sufficiently
35:03
hard that the only people that work on it on our team are PhDs
35:05
in statistics. Like I can't tease out the, the,
35:09
the signal that they're finding. And the LNS are unbelievably
35:12
good at confidently giving you incorrect narratives and like,
35:17
especially with data and charts and graphs.
35:20
And so like we, our, our, our core like data unit is like a
35:25
leasing call or a leasing tour, like with a prospect.
35:29
And we just make everybody at the company when they first
35:31
join, like listen to 150 leasing calls because they're actually
35:35
just very interesting. And it's hard to get any context
35:38
as to what we do and why what we do works without listening to
35:41
those calls, because the LMS will just massively simplify it
35:46
and and we'll sort of ruin the analysis before you even get
35:48
started. Yeah, I think we do a really
35:52
similar thing as well. It's like we know it's painful,
35:55
but you need to go and sit down and and look at essentially what
36:00
your Asian is doing, like look through examples, see what it's
36:03
saying to customers. There's, we've been working with
36:09
our research team a lot on this problem that I provide you
36:12
alluded to, which is how do I actually make sure that my
36:15
evaluator, my maybe my LLM as a judge is actually aligned with
36:20
what my domain experts want, right?
36:23
My LLM judge can say something like and say this is good or
36:26
bad, but my colleague might disagree.
36:29
And I trust my colleague more than I trust my LLM judge.
36:32
So the question then becomes, how do I actually align this,
36:36
this judge with the domain experts who actually know that
36:40
the area, right? And part of this workshop that
36:44
we've been leading is literally just sit down, look at it like a
36:49
significant handful of, of examples and make sure that your
36:53
team is aligned on what the behavior should be.
36:55
You know, make sure they're aligned on, you know, the agent
36:59
is doing the right thing here or it's not.
37:01
And and come up with evaluation criteria like a rubric based on
37:06
that exercise that you can then go and use to further down the
37:11
line your evaluations as you're iterating on your agent.
37:15
But yeah, I think this this point of actually developing
37:18
that product sense, it's just something you can't replace when
37:21
you're when you're developing your agents.
37:23
Summers, are you are you using that exercise in order to give
37:28
better prompts to those LLMS judges or in order to And yeah,
37:34
do you like modify the prompts of the judges or do you just use
37:38
that as feedback after the fact and tell it?
37:42
Here's a few more examples of what successful application of
37:45
the existing prompts looks like. Yeah, we actually use it for the
37:49
judges and I think it can also be encoded back into the agent
37:54
as well. So there's there's a ton of ways
37:56
to use that feedback and loops to both improve agents and
38:00
judges. But for us, like one of the key
38:03
things is making sure you also trust your evaluations just as
38:06
much as you trust your agent. Because if you have, you know,
38:10
CI that you can't really trust, you're not going to feel safe
38:13
deploying to production, right? And the same goes for
38:16
evaluations. So the point is to make sure
38:19
that you can trust your judges and then use those judges to
38:22
continuously hill climb performance of your agents so
38:26
that you trust your agents. I think that's a quick
38:28
reiterating on this, like observe the output in production
38:32
and you might have to change the evaluations, update the LLMS
38:35
judgement, update the system, maybe there's drift, maybe their
38:38
expectations have changed over time or the data has changed so.
38:43
Yeah, Yeah, I think that's really right.
38:45
We spend so much. We spend.
38:47
I always was like a, a big TDD person, but we do it so much
38:51
more aggressively now. And we UCI a lot like unit tests
38:57
still, but our integration tests are like substantially more
39:01
robust than I think I've ever had in a company.
39:04
Because like our system is like every system is complex and unit
39:08
tests are great. But like we actually are just
39:10
fully running like actual full simulations on every pull
39:14
request. And like, yes, that is more
39:16
expensive. And like, yes, that does take
39:18
longer. But like otherwise how, like
39:20
how, how am I going to let Ellen's touch like the core
39:23
parts of our code if I'm not at least going to run like a full
39:26
system, like a full walkthrough of the product?
39:29
So we get, I think that makes a lot of sense, man.
39:32
Are you talking about integration tests of the agent,
39:36
like including the agent behavior or just the output of
39:38
the agents having helped you with coding right?
39:41
No. Like when an agent writes code
39:43
and opens up a pull request in our code base, the level of
39:47
testing that we run is like more robust and more significant than
39:52
I've ever done in the past. And even at the expense of like
39:55
a little bit longer CI than I typically like just because it,
40:00
it overall allows us to move much, much faster with a lot of
40:03
confidence. Like when RCI goes green at the,
40:06
at the, we are at a point now, I think when RCI goes green, we're
40:09
like, we're, we're really confident that things are, that
40:12
things are working well. Like everything's being cleaned
40:14
up. We're really actually
40:15
interacting with the database, like we're really doing a lot in
40:18
CI. To make sure that we're
40:21
comfortable, we run it in the exact same environment that our
40:23
actual app runs. Like we go pretty heavy on it.
40:27
OK. So those are the, I think two
40:29
things that we're talking about in terms of nice.
40:32
So those are this is how happy it is to talk about TDD.
40:36
So we're there's like 2 aspects here.
40:39
The first is the traditional software testing is now done
40:44
more robustly, and that's separate even from the
40:49
evaluation of the agentic systems.
40:52
I think it, no, I think it's a part of it.
40:53
Like I think the part of saying like, I mean it's hard to
40:56
separate those two things, at least for us.
40:58
Like the system is the agentic system, like the product, the
41:01
software is the agentic system. So like running the integration
41:04
test is running the agentic system.
41:06
Like when I say we've run like full simulations of the
41:10
application, we literally, you know, one of our core products
41:13
is like a system that guides phone calls.
41:16
So we literally mock phone calls.
41:18
Like we generate audio and we mock the phone call and we
41:22
stream audio through a mock Twilio stream, which actually
41:26
goes through our production speech to text system and
41:29
actually gets predictions and actually makes decisions.
41:31
We do that whole thing many times as a part of CI, like
41:35
every single time. And even like a purpose saying,
41:39
I think the weirdest thing now is that the providers will
41:41
change the underlying models like pretty substantially
41:45
without telling you or like changing API calls.
41:47
And this happened to us pretty substantially when thinking
41:51
models became a big deal. Like we run a lot of real time
41:55
applications and so thinking for us in those contexts is bad.
41:58
We want it off. We have no interest in it.
42:01
Sometimes we would instruct in our prompts, you know, before
42:04
thinking, everyone was still doing thinking like that wasn't
42:07
new. We would just say like, put
42:09
thinking in, like put, add a thinking tag and like put your
42:12
thoughts out and then close the thing and then give the
42:14
response. You parse that that way.
42:17
But what we found without knowing was that the new models
42:21
that we're thinking models. If you if you tell a thinking
42:24
model to like think in tags in an API call, it goes crazy.
42:29
It'll think pretty much indefinitely until it hits the
42:32
output token window like the the output token limit, which was
42:37
like not a thing that they used to do.
42:38
They used to like be able to think briefly and one of our
42:41
prompts we just had like think briefly before answering this
42:44
and we got an alert. Like the calls that used to take
42:46
like 600 milliseconds are now taking like 45 seconds because
42:49
the LM was just spitting out text in a thought token like in
42:54
an in thought tags. And we caught that like before
42:58
it went out because it was just a part of like our RCI
43:00
integration tests. Very cool.
43:02
So I just we have about 15 minutes and there's a ton of
43:06
questions already in the chat. Before we go into the questions,
43:10
I want to see so summarize, you were talking about eval driven
43:14
development, then you're talking about CDB and how all of this
43:17
fits together, the CI, and I wonder if people want to get
43:22
started thinking even more rigorously or robustly about how
43:27
to get up and running. Any recommendations, any
43:29
resources, anything you can send them to either A blog is eval
43:33
driven development. That's the framework you guys
43:36
use when designing ML Flow. Yeah, definitely.
43:40
I mean, I'm, I'm very biased, so I will plug all the MO Flow
43:43
blogs and all the MO Flow resources.
43:44
But I think Anthropic also had a really good blog on eval driven
43:49
development. I think it's called Demystifying
43:52
evals or something along those lines.
43:54
So definitely point people along that way.
43:57
But I think of, of all of the teams that we've helped and you
44:02
know, big company, small company, doesn't matter.
44:05
The ones that we've seen the most success from are the ones
44:08
that have had this, this mindset of starting out with evaluations
44:13
and really thinking through with quality at the, the top of their
44:18
mind as they're developing. Because at the end of the day,
44:21
building an agent is really easy, but building a high
44:24
quality agent is, is still a challenge that many people are
44:27
facing today. And then there are additional
44:31
challenges on top of this. There's, you know, the question
44:34
of observability, There's a question of governance of
44:37
agents. So there's a lot of a lot of
44:39
different things that are blocking people in production.
44:42
But I think the most important thing when you're starting off
44:45
is that quality aspect. OK so and if you guys have any
44:50
other ideas for more resources drop them in the chat.
44:54
I want to start taking questions because we do have a lot and we
44:57
don't have too much time left. Apoorva, the first one is for
45:02
you. What stack tech stack do you
45:03
generally recommend for building agents?
45:08
It depends on the company that I'm working with, but I I have
45:14
been using Pyrantic AIA lot and Logfire so it helps with the
45:19
observability piece. What else?
45:24
I've used Bammel as well, quite a lot.
45:26
I don't know how many of you know about it, but Bammel is
45:28
pretty good too. And it's it's really nice in the
45:31
ID itself. Like you can like see how the
45:33
prompt changes what tokens are going.
45:35
You can change the model and like test it out right there.
45:37
So yeah, play around with that. OK, cool.
45:40
So I hope that that helps. Vidya, that was a question for
45:44
you. Rishi is asking are there tools
45:46
you guys use to help increase observability and reliability
45:49
for agents? Purva, you touched on that a
45:52
little bit summarize. I have a feeling I might know.
45:59
Yeah. So I think I work on MO Flow.
46:02
So obviously again biased, but we provide a lot of the tools
46:06
for an end to end Jenny I platform including that
46:09
observability piece. And and the thing that a lot of
46:12
people think about with observability and Jenny I and I
46:17
think I might have mentioned this before is traces.
46:20
It's essentially a way to see the end to end execution of your
46:24
agent. Really pop open that hood and
46:26
see what's going on. Every tool call, every LLM call
46:30
and attributes and metadata about each.
46:34
It's not something that's specific to MO flow.
46:36
I think every Gen. AI observability platform will
46:39
have some form of tracing integrations and most of them
46:43
including ours are really, really easy.
46:45
It's just a line at the top of your at the top of your code.
46:49
If you need more specific or like more customized use cases,
46:54
I've honestly never seen a case where we haven't been able to
46:58
trace an agent. So the observability component
47:02
is super, super important. You know you can't debug, you
47:06
can't look into negative customer feedback without it,
47:09
and it's really cheap to do, so I'd highly recommend it.
47:12
MO Flow has support for I would say like 30 to 40 integrations
47:16
now and it keeps on growing. Yeah, cool.
47:20
Nice. Another question here Bhuvan is
47:23
asking what architecture are you using for deterministic pre
47:27
execution controls and agent workflows?
47:29
So not just prompt level guardrails?
47:31
And how do you prove those controls were actually enforced
47:33
for action? That's a question.
47:37
Anyway, whoever wants to take it, yeah.
47:40
So it depends on the kind of project.
47:42
Like you can have checks for stuff after the agent has run in
47:47
some cases. Like there was a project which
47:48
was about governance and one of the like we had to check where
47:52
the policies we were updating the documents based on policy
47:55
update and like different countries.
47:56
So we had a check after each LLM call.
48:00
So like each LLM like circle like that was happening.
48:04
We had a check on like what, what happened?
48:06
What were the like? We were able to attribute every
48:09
change in the policy document and why that happened.
48:12
So it depends like how many like depends on the use case, the
48:15
kind of use case, what checks, how many like what's the latency
48:18
like? Is this a real time need?
48:20
So this wasn't so we had like a lot of checks in that
48:23
application. Can you repeat the question?
48:27
Yeah for sure. It's what are you using for
48:30
deterministic pre execution controls in agent workflows.
48:34
So pre deterministic pre execution controls, not just
48:38
prompt. Yeah, what is the?
48:41
The pre execution is the part that's throwing me off on that
48:43
question. I'm not sure like what exactly
48:45
that means. Yeah, Purva, if you had a a
48:50
certain reading of it, please. I was just going for
48:53
deterministic. How do you how do you make sure
48:57
that your thing is working the way you expect it to?
49:00
Move on if you want, if you want to unpack it, and until then,
49:03
we'll move on to. To the next one, yeah, I think
49:06
there's one thing just worth calling out is like you like you
49:09
can't. Like if you're building LM only
49:11
systems, you can't guarantee anything is deterministic unless
49:15
you run that model yourself and you control batch sizes and you
49:18
control input concurrency. Like thinking machines has done
49:21
the research like we understand why you can't actually get true
49:25
determinism on on those on those hosted AP is because of the way
49:28
batches are managed. If you run that model yourself,
49:31
set a seed and set the temperature to 0, like you sure
49:34
you can get determinism, but if you can't do that, like you
49:38
can't guarantee it and you need to be building systems where
49:41
you're comfortable enough with the level of non determinism
49:45
that happens. It's quite low.
49:46
Like if you're using a really low temperature and exceed and
49:50
one of the major providers, you can get it pretty low.
49:54
But at the end of the day, like, you have to be comfortable
49:56
working with a probabilistic system.
50:00
So, so how about this? Let's there's a couple of people
50:05
that are mentioning a few other things that we hadn't yet gotten
50:08
to chat about. And there's honestly, there's so
50:10
much that we didn't get to talk about.
50:12
And so maybe we can keep those in mind as we answer the
50:14
following few questions. So they said mentioning land
50:18
graph, etcetera, but about sandboxing off the memory
50:22
metrics, things like that. So perhaps keep those in mind
50:25
when we're answering the next few questions CD is asking.
50:29
Which kind of framework do you use these days?
50:32
There are new ones everyday. How do you secure your systems?
50:34
So Oauth, for example, between agents and memory and tools for
50:39
observability. I think we touched on some of
50:41
that already, but if there's anything here that you'd like to
50:43
unpack. I, I like to keep systems pretty
50:47
low level. I think it's a privilege of one
50:49
like being an engineer first, but also working with a really
50:53
small company. Like my startup is, is quite
50:55
small. We, we only have 7 people and
50:57
it's just because we're all able to like move very fast and
51:01
leverage this stuff in a way that like larger enterprises
51:04
maybe have to take more time to introduce.
51:06
But we keep things very low level and that gives us a lot of
51:09
control. And so we don't use like lane
51:12
chain or actually lane graph. Our entire system is powered by
51:16
Bammel because Bammel for me, in my opinion, is the correct
51:20
fundamental like abstraction layer over prompting.
51:24
It turns prompts into functions. Those functions are guaranteed
51:26
type safe because Bammel has to compile and it can't compile
51:29
without it being type safe. Those types compile through your
51:32
entire code base, both TypeScript, Python, JavaScript.
51:37
Those Python models turn into our database migrations so it
51:41
keeps the entire system 100% In Sync.
51:43
Like we we literally cannot fall out of sync unless somebody can
51:47
somehow bypass our GitHub rules like it's it's not possible.
51:51
And then because it's at the level of just making
51:55
asynchronous calls to an LLM, getting guaranteed typed outputs
52:00
or quality validation errors that you know how to handle
52:04
because you know how to write software, you sort of have
52:06
control of for everything at that point.
52:08
So Oauth becomes the normal way that you do Oauth, which is just
52:12
like the way that we've been doing it forever.
52:14
And agent agent communication. We just leverage the standard
52:18
MCP protocols for, for identity aware server side calls.
52:24
Like if you if you don't use the highest level abstraction
52:28
systems and and you keep everything pretty low, you your
52:31
code base actually just looks like regular software.
52:36
You wouldn't really know that our code base has a bunch of AI
52:38
going on in it because it's all sort of in DAMO files and the
52:42
Python And TypeScript code just looks like code.
52:45
So for us, that's how we do it. And then again, observability is
52:47
just like, all right, well, all these LLM calls are functionally
52:50
API calls and you log your API calls and they're logged in a
52:53
structured way with traces and tags and metrics.
52:56
And so you can just go look at them in century like it's like
52:59
it's very kind of boring software when when you when you
53:02
get to control the lowest levels of it.
53:05
Before we move on to the next question, do you guys feel like
53:07
that's, is that true to you as well?
53:12
So Samrage Apurva, like are you, to what extent do you rely on
53:17
higher level abstractions and these types of frameworks versus
53:20
let's just get as close to the Chrome as possible and visual
53:26
and try to render our software as traditional software?
53:31
I have been using as well. Like whatever Ben said, I'm all
53:35
in. Yeah, I think I'm also, I think
53:40
Ben covered most of the points pretty well, so.
53:43
So I want to Group A few questions together here.
53:47
So Nikolai is asking about the observability layer.
53:51
Is that another agent? What does the observability
53:54
layer actually look like? And I'll pack into that as well.
53:58
So what system are you using to create and orchestrate the
54:01
judges? Yeah, the observability layer is
54:06
actually really cheap. It's it's essentially just a way
54:09
to sniff what's happening at each step in your agent.
54:14
And if you know it, it's basically cost nothing to
54:17
implement. I would recommend that any, any
54:22
use case series about going to production has it.
54:24
And I, I think Ben and Abora, you guys have probably use this
54:29
for all of your agent use cases as well as to the second part of
54:36
the question, this was about judges, right?
54:41
So let me add to that. So what system are you using to
54:43
create and orchestrate the judges?
54:46
And then as Zeal is asking, those judges are probabilistic.
54:50
How can you trust that? So like, don't you need
54:52
something more deterministic to recreate evaluation?
54:55
Yeah, Yeah, I think at at the, you know, at the simplest level,
55:00
judges are basically just prompts, right.
55:03
That or they're just classifiers.
55:06
And often we recommend that people start off with like
55:10
binary classifiers that can say, hey, is this good or not, yes or
55:14
no. It's usually a stronger quality
55:17
signal. The, the system that you
55:19
actually use to create these and orchestrate these is, you know,
55:23
MO Flow provides judges. Other frameworks like Lang Fuse,
55:27
Arise, brain trust, they will also provide judges.
55:31
So it's just a matter of picking the right ones for your use
55:34
case, whatever works for you. And, and I think the question of
55:38
how do you actually trust your judges is, is a big one that a
55:43
lot of frameworks have been thinking about.
55:46
And at the end of the day, any out-of-the-box judge you pick is
55:50
probably not going to be something that works for all of
55:52
your use cases, right? It might work 80% of the time,
55:56
but that remaining 20% is something that you have to kind
56:01
of align with your stakeholders on.
56:03
It's something you have to tune and, and work on your, you know,
56:07
you have to work on your judges to get that full trust and
56:10
confidence. One of the examples I mentioned
56:13
before was that workshop that we ran, which really help people
56:17
align those judges with their their expectations.
56:20
And I think I'm a flow and a couple other libraries have been
56:24
providing tools to help you do this alignment automatically.
56:27
Just get in user feedback and and apply that directly to your
56:31
judges. Yeah, I think the only way to
56:34
trust your judges is to at like at some level I don't, doesn't
56:37
matter how many judges you have in the stack, at some point you
56:40
need to be making a decision that is classifiable.
56:42
And so as long as you're making a decision that's classifiable
56:45
at some point, whether it's binary classification, whether
56:47
it's a multi label, like it doesn't matter.
56:49
Like you, if you're, if you're classifying something, you can
56:51
build a data set of evaluations, right?
56:53
You could build a ground truth set.
56:56
And so if you build a ground truth set at some level of
56:59
statistical significance, you can build trust if you need, you
57:03
know, it depends how much trust you need.
57:05
If you need an enormous amount of trust, you build 10,000
57:08
samples and that's your evaluation set.
57:10
If you need less trust, maybe you only need 100 samples like,
57:13
but at some point you, you need a judge to make a quantitative
57:19
decision evaluation, not a qualitative one, right?
57:22
If if you're having them always just spit out summaries of
57:24
summaries of summaries, yeah, you can't trust them.
57:27
But like you can always build a ground truth data set and so you
57:31
can always just trust them at some point.
57:33
I feel like if you're using LLMS in inside the agent, why can't
57:38
you use the LLM for judging as well?
57:40
Like you're using it for like making the decisions like doing
57:42
the tool calls, deciding like am I supposed to like send this
57:45
answer to the next agent or not? And why can't you use?
57:47
If you like align it really well, it should it should work.
57:52
Yeah, I agree. As long as you have ground
57:54
truths, that's like, I mean, for us it's easier because our
57:56
systems are a lot of real time and like we're cost conscious
58:00
And so like we don't use clawed as our agent.
58:05
We use Claude as our judge, which is nice.
58:07
It builds like some concept of like separation of concerns.
58:11
I don't know how real that is. I don't know that anybody's like
58:13
really measured that maybe they have on archive.
58:15
But like, yeah, we, we use the really expensive Max effort long
58:20
thinking models as our judges and we have a valid ground truth
58:23
data sets and we all align that they're right.
58:26
And like at some point, you know, at this point we have like
58:29
800,000 samples and, and we've spot checked at least 10,000 of
58:33
them. And that that's a lot like we,
58:35
we have a lot of confidence that that it is aligned to the thing
58:39
that we care about for that very specific, each individually
58:42
specific task. We are just about out of time.
58:47
I want to maybe pose one last question and if one of you wants
58:51
to jump in to respond to it. It's about the again, trust, but
58:57
in this case risks involved with let's say letting any one of
59:02
these agents interact with your data or interact with your
59:05
database. There's a few questions around
59:08
that. How do you handle prompt
59:10
injection risks when agents interact with external tools or
59:12
API's or user provided data in production?
59:17
Is there anything you can tell us about?
59:20
Here you go. Gotchas to avoid when giving
59:22
agents access to the DB? Or do you feel like current DBS
59:26
are good enough for agentic access?
59:29
Can you share anything about? That yeah, I definitely don't
59:32
would not give my agent freight access to my database.
59:35
Definitely not like no shot they have read only access and they
59:39
by the way, they have read only access to a backup that is not
59:42
on the same server as the production database.
59:45
So I don't care if they take it down like they can do whatever
59:47
they want, like they're in a sandboxed environment that can
59:50
can shut down and be recreated in an hour like it.
59:54
No, my agents have almost no right access.
59:57
And to the accept that they do, it's unbelievably limited.
1:00:01
I'm sure other people are on like different ends of that
1:00:03
spectrum. But like we, we, we serve like
1:00:06
really big enterprise customers. Like I'm absolutely not risking
1:00:09
anything like that. It's not worth it.
1:00:12
Thank you everybody for joining. I feel like we could keep going
1:00:15
for hours because there's so much that we've just left
1:00:18
unpacked and it's and I maybe we should do this again.
1:00:22
I think we we probably should. Sam Raj, thank you very much,
1:00:25
Apoorva. It was a pleasure.
1:00:28
Ben, thank you very much as always.
1:00:30
And if you want to linger in the chat for just another couple of
1:00:34
minutes and maybe drop some links, I know Oscar has asked
1:00:37
you, Ben, to share a link. A couple of other people have
1:00:40
asked for links for, for the ML flow stuff.
1:00:44
And Apoorva, another couple of questions about, but I think I
1:00:49
saw a few more things for you. So about Slack QBR.
1:00:54
Yeah. So I will leave you to it.
1:00:57
Thank you everybody for tuning in.
1:00:58
I hope this was fun and useful. It certainly was for me.
1:01:01
I have a bunch of things to go and research.
1:01:04
And thank you, everybody, everybody for tuning in.
1:01:08
Thank you. Thanks for having us.