Latest / Agentic Conversations (formally mlops.community) / Tool definitions are the new Prompt Engineering

Transcript

Tool definitions are the new Prompt Engineering · Agentic Conversations (formally mlops.community)

0:00
We've got Kiara to my left, Alex to my right.
0:02
CEO of Arcade. Kiara is working on ifood.
0:06
She's at process lead data scientists.
0:08
Data Scientist. You're a lead in our book.
0:15
You've been. Working on a project at ifood?
0:17
You gave a talk last night. I loved it.
0:19
I wanted to bring you on here to talk about some of your
0:21
learnings. For those who don't know why
0:23
Food is the biggest food delivery company in Brazil, it
0:27
has a huge order volume, 160 million per month.
0:31
Brazilians use it a lot. What we realise is that users
0:34
often go to the app and they don't know what to order.
0:37
They are a bit undecided and sometimes they get frustrated
0:40
because they have too many options.
0:42
It's a paradox of choice, right? So the problem we solved with
0:45
this agent is to help them decide.
0:48
We built something that knows who the users are and knows
0:52
their preferences, their habits and the price range they're
0:57
willing to pay. And we use that plus the users
1:01
questions to suggest the best options for them, even
1:05
proactively. And we, yeah, we have a flow
1:09
where the user can can search and refine and then order at the
1:12
end we prepare 2 interfaces for the agent. 1 is in the app and 1
1:18
is on WhatsApp. WhatsApp is very popular in
1:22
Brazil. I think there are 160 million
1:26
active users. So they, they use it a lot.
1:29
If you don't know, people use it to order food with just voice
1:33
messages. It's, it's really smooth.
1:35
They just send a voice note to the, to the restaurant and they
1:37
can order like that. So we will leverage this
1:40
familiarity with conversational interface.
1:45
Both had different challenges in terms of UX, how we show results
1:51
to the user. It often happens with agents,
1:54
right. So the the biggest challenges
1:55
are not AI related, but UX and adoption related.
2:00
So the agent that we built is kind of a React agent.
2:05
We didn't use like to fancy multi agent set up because we
2:08
needed things to go very fast. Users are hungry, they don't
2:12
want to wait. So we need to have the simplest
2:15
flow. We need to make sure that the
2:18
recommendations that we give to them are are good.
2:21
They work for them. And we also need to to remember
2:24
if they don't like certain foods, like the agent really
2:27
needs to be a smart companion, almost reading their mind.
2:32
One of the things we wanted to make sure is that the
2:34
conversational interface was not just textual and there were
2:38
multiple modalities for the user to talk to the agent because
2:42
when they're hungry, they don't want to spend time typing,
2:44
right. So we we made sure that tools
2:47
that we built were connected to UX and UI elements so that the
2:51
user could directly interact with them to have some short
2:57
circuits, let's say. Like swiping.
3:00
Also, yeah, we also implemented A swiping interface.
3:03
Voice also. Also voice.
3:05
Yeah, yeah. So voice is a modality, but it's
3:08
still textual in a way. What we we did was also making
3:13
sure that there were buttons to Click to do actions quickly
3:17
without having to type like I want the third item that you're
3:20
showing me and that that's not needed.
3:22
One interesting thing we noticed is that people on WhatsApp were
3:25
way more lenient towards this conversational behaviour than in
3:29
app. In app, really, people really
3:31
want to use buttons. They expect a different type of
3:33
interface, so our tools needed to work very well.
3:38
One of the challenges we encountered was that the two
3:43
definitions that we had made a lot of sense for us, but it
3:47
didn't make sense to someone external that would see them for
3:52
the first time. We realized this afterwards, of
3:55
course, with trailer and error. What we noticed was that we we
4:00
created the tools right? Things made sense.
4:03
But as soon as we were getting edge cases in production, we
4:06
were adding this to the tool. So whenever the user wants to
4:11
order something and you don't have enough information about
4:15
it, make sure to call the get information tool.
4:18
Like a lot of edge cases for this flows and it quickly
4:22
becomes code like a false statement and that you don't
4:26
want, right. So the exercises we made was to
4:30
try to to standardise this tool. Like think if I would share this
4:35
tool with another team and if they would want to use it in
4:39
their own agent, how would I write things there?
4:42
Like the idea is to make them as clear as possible.
4:45
Would the name of the tool even make sense?
4:47
Right. And after that we had a massive
4:50
improvement in latency because we could cut a lot of tokens and
4:54
our system become became much more stable.
4:57
Making sure that a tool encapsulates the right things is
5:03
the hardest part of the problem. And I think people really
5:07
struggle with that. They they think of it as an API.
5:10
Oh, we just have an API, we'll call it.
5:11
Well, that's not really gonna work.
5:13
You know, there's a lot of usability.
5:15
It's like, well, what's the user experience gonna be?
5:17
What's the intention that the user has?
5:20
What's the intention that the agent needs to have?
5:21
And then figuring out, OK, well, what's what's right,
5:24
encapsulation of that in a tool is what everybody's struggling
5:28
with. It's a new paradigm.
5:30
It's a mental model. What we, what we see as a best
5:34
practice is people try and build that is that it's, you know,
5:38
there's like a laddering of tools.
5:42
And so you might have a shared tool that is a, a workflow or
5:48
maybe even an MCP server that's integrating to another service.
5:53
But even if you have those things, the agents tend to do
5:56
better if you then build a very domain specific or agent
6:00
specific tool to capture that particular agents nuances.
6:06
And so when you get that, you get both the accuracy and the
6:09
lower latency that you're looking for because you're
6:13
pushing more off to deterministic software.
6:16
And then you're able to abstract out the common elements so that
6:19
other teams can reuse them really easily.
6:22
I love how you talk about the difference between just a basic
6:26
API and then an agent needing to consume some kind of a service
6:31
and that one keyword which is the intention, and how that
6:37
intention plays such a big part in what is trying to consume.
6:42
Yeah, I think there's a lot of confusion right now because
6:46
every engineer is familiar with APIs and then you throw in MCP
6:51
and people go, oh, it's the same thing.
6:53
Let me just wrap in API and MCP and success.
6:57
We, we have an agent and we have tools and it doesn't work for a
7:02
bunch of reasons. MCP is just a wire protocol.
7:05
But more importantly, tools are not APIs.
7:07
And so in engineering speak, an API is a service contract for
7:13
the downstream service. So we'll use Google Drive as an
7:16
example, it has an API. That API is a service contract
7:19
on how Google Drive works. And yes, there's a lot of work
7:23
that goes into building a really nice Google Drive MCP server.
7:27
Like we have to make it a bit more workflow wish to make it
7:32
more intention based to kind of remove or at least abstract away
7:37
in the structured inputs that are required.
7:39
So, and LM has no idea what a Unix timestamp is.
7:44
So your MCP tool has to like know what yesterday is as a
7:47
concept and then translate that to Unix timestamp.
7:50
But even with all of that, if you're building a sales agent,
7:54
for example, and, and a, and a Rep is gonna ask, hey, I need
8:00
the brochure for this product 'cause I'm going into this
8:03
customer account. The agent doesn't care at all
8:07
about Google Drive. It's not its intention.
8:09
Its intention. It's to find the brochure that
8:11
it needs. And so you give it even a
8:14
pristine, beautiful Google Drive MCP server.
8:20
You're asking for higher latency and you're asking for
8:22
hallucinations. Because now you have to stuff
8:25
the context window with explanations of how to find the
8:29
brochure using the MCP server. Which means it's going to have
8:32
to take multiple turns figuring out what the right folder is and
8:35
where the right files are and how to determine which ones are
8:38
brochures and which ones aren't brochures and which is the right
8:40
brochure. But if you instead give it a get
8:43
brochure tool, and that get brochure tool can call the
8:47
Google Drive MCP server, then all of a sudden all the agent
8:52
has to do is say, oh, I need a brochure.
8:54
There's a get brochure tool. I have the context that I need.
8:56
Let me submit that as arguments and then it's done.
8:58
It's one call, it's low latency. You don't have to stop the
9:01
context window. You minimize the amount of tool
9:04
definitions that you passed over.
9:06
And then likely much of the code inside of the get brochure tool
9:13
is deterministic. You know, chances are it's not
9:16
calling a model or if it is, it's calling 1 maybe once I
9:19
think very specific. So the whole system just gets
9:22
more accurate and faster. And So what that means is a tool
9:28
is actually kind of the inverse of an API.
9:31
Well, an API is a service contract of what a of what a
9:33
downstream service looks like, like Google Drive.
9:35
In my opinion, a tool is kind of like the service contract for
9:39
the agents intentions. It's it's what it expects to do.
9:45
OK, and being able to write the proper tool definition is really
9:51
the key. It's like, it's like half the
9:53
battle. Yeah, it's like half the battle.
9:54
That's why evaluations are so important as well.
9:57
Something funny happened to me six months ago.
9:59
I asked Deep Research for a report on different GPU
10:03
providers and it was absolutely shit.
10:07
I couldn't figure out what each Neo Cloud Value prop was, so
10:11
that set me off on my latest side quest of creating a
10:15
practitioner's guide to choosing GPU's.
10:18
I'm happy to announce this guide is now ready to see the light of
10:22
day and you can download it for free right now by clicking the
10:26
link in the show notes. We've already got some community
10:29
members feedback of what they wish they would have known
10:34
before signing a gigantic contract and I would love to
10:38
hear from you if this provides any value or you have some
10:42
things that the rest of the community should be thinking
10:45
about when they're on the market for GPU's.
10:48
Go ahead, download that resource right now completely free.
10:51
It's in the link in the show notes.
10:54
Well, I think also I just want to highlight such an important
10:58
thing that you said yesterday and just now again, which is let
11:02
somebody else look at your tool definitions, let them see if
11:06
they can understand it. And if they can, then you're can
11:11
try it with the agent. But if they can't, you already
11:14
know, all right? This is probably where the
11:16
problem is. Yeah, exactly.
11:19
And this exercise really forces you to to have clear tool
11:23
definitions. And I think it's also important
11:27
to try to limit the agent choices as much as possible.
11:31
I think if there is a lesson I've learned in this year's
11:33
building agents, that's that's this one.
11:38
It decreases. Hallucinations, it decreases
11:41
context bloating as well. If you have tool that's like
11:46
tools that are always called together, like in the get
11:48
brochure case, it makes so much more sense to create a workflow
11:53
and encapsulate that into a tool.
11:55
A tool is something that LLM can use to take action.
11:59
It doesn't need to know what's inside.
12:01
It can be a it can even be another agent, right?
12:05
So yeah, one way is also to use multi agent setup of course, but
12:10
for from the point of view of the main agent, it doesn't
12:12
really matter. Yeah, we think a lot about this.
12:16
So, yeah, in a perfect world, you give it, you give it one
12:20
tool, right? In a perfect world, the agent
12:23
doesn't have to think at all. And you don't even need an LLM,
12:27
right? Because it's faster, it's
12:30
cheaper, it's deterministic. But that's not the real world.
12:35
The power of an agent is that it can handle generality.
12:40
And so there's this careful balance.
12:43
You know, Sam, my Co founder talks about turning the knob on
12:46
determinism. And so you can give it fewer
12:51
tools and you and arguably today you should because you want to
12:58
minimize error rates. But the point of all of this,
13:03
that where all this is going, where all the investments are
13:07
going, where the the model companies are investing heavily,
13:10
where the orchestration systems like land graph are investing
13:13
heavily, where we're investing heavily as a world, which is the
13:16
opposite, where you can turn up the the, you know, the non
13:20
determinism, you can turn up the ability to give it as many tools
13:24
as you want and then let the agent intelligently decide what
13:28
to do. That's very hard.
13:30
That's where we're going to get to at the limit.
13:33
You know, we ourselves have achieved, you know, incredible
13:37
things in the lab that we haven't yet announced where
13:41
we're making it possible to kind of turn that that dial up on the
13:46
tool level. But for most people today,
13:49
you're right. You're better off really
13:51
thinking through in a multi agent system, you know, which
13:56
are the right tools based on the node that I'm in or the state
13:58
that I'm in and trying to be very specific.
14:03
But the benefits of giving it more tools are huge because then
14:07
you have fewer nodes, the agents more intelligent.
14:09
But it really depends on what you're trying to do.
14:12
Yeah. Well, so I think the we haven't
14:15
touched on this topic, but the way output I would the way you
14:21
construct a tool output is also important because you can put a
14:23
lot of instructions there as well.
14:26
So by limiting the amount of choices, I don't mean the agent
14:29
shouldn't have tools like we should try to make it as
14:32
deterministic as possible. We can still live freedom, but I
14:36
think we need to be smart in where we put the information
14:39
where we put the instructions. So if for instance, I always
14:44
have certain options for tools after a given tool is called, I
14:49
can put the instructions in the tool response.
14:52
I don't need to to blow the system prompt with those
14:55
instructions. I've heard that a ton that
14:56
dynamically inserting context in there, that's one way of doing
15:00
it. There's all kinds of fancy ways
15:03
that you can make sure to leave things out that need to be left
15:07
out. And then when it needs it, if it
15:10
needs it, it's like this is a need to know basis here, yeah.
15:13
Well, I think I'll, I'll go back to this concept of layering up
15:20
tools. If you look at APIs, this
15:23
pattern already exists. You've got your, your low level
15:27
system APIs, you've got your workflow APIs in the back end.
15:33
And then you have the APIs that the, that, the, that the mobile
15:36
application or the JavaScript that kind of talks to.
15:38
And there are like 3 different sets of APIs.
15:41
And when you get all the way to the top where it's a JavaScript
15:43
app talking to the back end, those APIs are very, very
15:47
specific to that application most of the time.
15:50
And similarly in what you're describing, and we talked a
15:52
little more about this last night, so I'm gonna, I'm gonna
15:55
steal from last night. You can insert UI code in the
15:59
response, right? You can present a table.
16:03
It doesn't have to be just text. You can present a React
16:07
component in the tool that gets very agent specific.
16:12
But that's kind of the point. You minimize the amount of work
16:15
being done by the rest of the system by the tool caring and
16:19
doing a lot of the heavy lifting.
16:22
And you know, I'll go back to the Google Drive example, right?
16:26
Sure, it can pull about the brochure, but it can also go
16:28
pull a bunch of context that you know the models going to need on
16:32
the next turn. Yeah.
16:35
It reminds me of a conversation I had with Zach from Sierra and
16:39
he was saying that a lot of times since they're doing voice
16:43
agents and it's real time voice agents, what happens is they'll
16:47
have almost like a supervisor agent that will recognize and
16:53
preemptively assume if this conversation is going in a
16:59
direction that I think I may need this context for, they just
17:04
go and grab it just in case it comes up And all right, we have
17:07
it now, I can give it to you. And then you don't have that
17:10
user experience where the person is waiting on the other line
17:13
because the context needs to go and you need to grab it and
17:18
bring it back. And that takes a little bit
17:19
longer. It's just like, let's have
17:21
everything kind of loaded up and then if we need it, we can serve
17:24
it to the agent that's interacting with the human.
17:27
Well, I mean, I, I think that speaks to, you know, how quickly
17:31
the history's changed. It's November of 2025 right now
17:35
and the conversation is no longer about accuracy and
17:40
consistency being the blocker to production.
17:43
Five months ago, maybe even less, that was the only
17:46
conversation we were having. Now we're talking about latency,
17:51
everyone like that is the biggest problem.
17:53
We're going to prod now, but now we're just having, you know,
17:55
mediocre experience because we're all waiting 30 seconds.
17:59
And, and now, and it's, it's amusing to me because if we were
18:03
waiting 30 seconds five months ago, we would have been totally
18:05
cool with it, but now we're like 30 seconds.
18:09
It also connects to what I said earlier about WhatsApp versus OP
18:13
experience. So on WhatsApp people are
18:15
totally fine. We're waiting because they
18:18
expected it's a it's a familiar interface.
18:20
They don't expect it to go so fast, probably because normally
18:23
you have another person speaking on the other side.
18:26
But on app you're punished if you if you don't deliver in
18:29
time. The the expectations are
18:32
completely different even for the same user.
18:35
In a perfect world where you had user feedback from everybody
18:39
who's using your acting with your agent in real time, I'd be
18:42
very curious to see a generational distribution on, on
18:45
patients for waiting for the agent.
18:47
Because one of the things that one of the things that I see,
18:50
but I'm very curious is the, the generation of people who are
18:58
let's say under 30 expect everything to be agentic.
19:03
If they see a menu, you know, they've got to click around.
19:09
They just, they bail, right? The later generations are the
19:14
opposite. They expect a snappy UI, very
19:17
snappy and responsive buttons and clicks.
19:21
But I'm very curious how the the generational distribution might
19:26
be on on patients for latency. And I also feel like when you're
19:31
in WhatsApp, you can go and you can talk to your friend, you can
19:33
look at something else and then come back.
19:36
And so you get that because I imagine that you get a
19:40
notification when the agent is done and it's sending you the
19:44
information. You don't have to sit there and
19:46
wait the 30 seconds looking at the WhatsApp chat.
19:49
When I'm in WhatsApp, I'm talking with three or four
19:51
people at the same time and I'm going back and forth between
19:54
those conversations. The data that we got was really
19:57
clear about this. That's why we spent so much time
20:00
refining UX, thinking how to present, you know, the data to
20:04
the user. There's a lot of work done.
20:06
Around the Asian that is not Asian itself.
20:08
And I think this was one of the biggest challenges.
20:11
Also users, you know, they are familiar with AI by now, but
20:17
it's still hard to to trust an AI interface to suggest your
20:21
food. People are pretty sensitive
20:23
around that. They have their preferences.
20:26
So we really needed to build customer adoption, make it like
20:32
also in terms of the persona that we developed, they needed
20:35
to be friendly, but also engaging.
20:37
We wanted people to come back, right?
20:40
It all connects to tools at the end.
20:42
The tools that how do you define it, how you use it, how smart
20:47
you want it to be. Can I call it something you said
20:50
that I think is perfectly on point today?
20:56
The, the intersection between an application and an agent is now
21:01
complete. And you mentioned earlier,
21:03
right, like so many of the things that you ran into as you
21:06
were building this weren't necessarily like the model or
21:10
the AI, it was the, the application and, and I feel
21:15
again, you know, speaking at least today and you know, we'll
21:19
see what the world looks like in three months.
21:21
That's itself a huge transition. And now we're having the debate
21:26
is now, well, how do I make this UI work properly?
21:28
And you know, how do I deliver this to the, to the, to the, to
21:32
the user properly as opposed to getting all caught up in this
21:36
super deep ML data science of the AI?
21:41
And this also connects to evaluations.
21:43
How do you evaluate such a system 'cause when we talk about
21:47
evaluations, we have this standard metrics in mind like
21:49
faithfulness, helpfulness, so you name it, but they're not
21:53
necessarily connected to the business value that your
21:56
upbrings like. How can you leave UX out of the
22:02
evaluation? Like you, you need to take this
22:04
UX elements into account. That's part of the of the user
22:07
journey and that's also part of the agent.
22:09
It's really really hard to to separate these two.
22:12
We were joking last night that every developer believes so
22:17
strongly in Test Driven development that they recommend
22:19
every other developer do Test driven Development.
22:21
They just too lazy to do themselves right.
22:24
But I feel with agents, especially around evals, you
22:29
have to start there. And so I'm curious on this
22:33
point, which I think is a really incredible point around
22:37
assessing the connection between business value and, and, and the
22:40
user UI. How did you, how did you
22:42
structure your evals? What what were the evals you you
22:45
ultimately landed on? We have different levels of
22:49
evals, right? You have those more connected to
22:53
development like making sure everything works, like
22:56
regression tests, things you can evaluate with codes that that's
23:00
like the foundation. We also have a golden data set
23:04
that we run what we realized after.
23:08
I think the, the evil communities shifting towards,
23:12
towards this as well is to that error analysis is really, really
23:17
important. Like we, we started like many
23:20
teams to think of metrics like is the agent helpful?
23:24
Did the agent satisfy the user request, which are like, they're
23:29
OK, but they're very generic. You need to write them in a way
23:32
that is very specific for your product.
23:35
And when you don't have a product yet, this is really,
23:38
really hard to define. So we were lucky enough to have
23:43
community of eye food employees. There are more than 7000 people
23:47
employed at Eye food. So the user up and they give us
23:49
feedback. There's nothing that can
23:53
substitute Looking at the data, we had to go and look at the the
23:57
traces, identify the the errors that we were seeing.
24:01
And once you have the you have a good overview of what can go
24:06
wrong in in your application. That's when you can build a
24:10
taxonomy of errors. So we desire a taxonomy you, you
24:13
can build a test set and make it, it can inform you of, you
24:19
know, things like like how, how good you're doing with your
24:22
evaluation and if the agent is performing better or not.
24:26
And this is connected to LLMS judge.
24:29
When you write an LLMS judge, having this very specific domain
24:35
knowledge helps a lot. So we couldn't separate it from
24:39
error analysis. It's something we learned later.
24:41
If I would start a new project, I would start from that straight
24:44
away. And but you're also checking for
24:46
evals on the tools and which tools were called, was it the
24:51
right tool? So there's like higher level
24:54
objectives, did it satisfy the user?
24:56
But then there's those very nuanced pieces, right?
24:58
Yeah, we we have of course evals for the tools.
25:00
Some of our tools are Smart Tools, you could call them
25:04
agents themselves, like searching in our life.
25:07
With catalog you need to take into account user preferences,
25:11
source the items, pick the ones that have most variety and like
25:14
match with the user intent and profile.
25:17
So that's just a tool, but deserves a whole set of
25:21
evaluation itself. Is that and just?
25:23
Is that how you were able to see which tools were used together
25:27
and then create workflows from the tools?
25:29
We analyzed the data like we did a lot of ad hoc analysis.
25:35
A-Team member of us is completely, completely dedicated
25:38
to this, like assessing the quality of the agent and like
25:43
understanding what goes on behind the scenes and also what
25:47
goes on in front of the of the user.
25:49
Another thing that we did, I didn't mention yet, what I
25:52
presented yesterday was to create an evil set that was
25:57
going to be picked up by an agent that would impersonate a
26:01
user. So that was something that was
26:04
defined by product team. So what what we noticed was that
26:10
was very easy to define scenarios and what the agent
26:13
should do in those scenarios. What was hard was to create an
26:17
LLM judge. It could judge any scenario, but
26:20
given a scenario you could say like the agent should do this
26:22
and that. So we started with a set of of
26:26
queries to find this way. And sometimes you need some pre
26:31
processing steps to get there. Like for instance when you have
26:35
an item in the cart, like when you start to make sure there is
26:37
already an item, things like that.
26:40
And then we we built an interface between the agent that
26:45
would would test our endpoint and the endpoint.
26:49
And this interface made sure that when we were returning UI
26:51
elements, these were also shown to the user impersonating agent.
26:56
So we like the agent could choose to click on certain UI
27:00
elements. And at the end we would judge
27:02
the outcome of all of this. So I think this allowed us to
27:07
also test the UX in a way it doesn't substitute AB testing.
27:12
Of course, that's like online testing is another story, but it
27:17
helped us to identify regressions.
27:21
On the tool side, how much work went into evaluating the tools
27:25
themselves to make sure that you design them or at least their
27:28
definitions properly and the parameters the parameter
27:31
definitions properly to see if the model was selecting them
27:35
correctly at the right time? Yeah.
27:37
So we we did evaluate if the agent was calling the right
27:42
tools. Of course depends on the tools.
27:46
I think like some tools are so clear to use, like create cart,
27:51
you know, some definitions are so clear that you don't need to
27:55
spend too much time on that. You just want to know that the
27:58
agent can pick it up correctly. But some others are whole
28:02
workflows, like searching what I was talking about before that
28:06
had a lot of evaluation being done.
28:08
And there is still a lot going on there because it's building a
28:12
recommendation system basically, right?
28:16
Yeah. So like what we have is a user
28:21
who's searching for something or maybe just exploring options.
28:23
They just say I'm hungry, surprise me, give me promotions
28:27
without any intent. What they're building is is a
28:30
home page right in the agent. So you want to show options that
28:36
are really, really good for them, even when they don't
28:39
specify anything. So that's a yeah, that's a
28:43
recommendation problem. Is there ways that you're
28:48
plugging in? Because I imagine when it comes
28:50
to the workflow, you're plugging in structured data and
28:53
unstructured data in this, like you're giving the context to the
28:56
agent of the last three, five meals that this person ordered,
29:02
when they ordered it, the timing, what they like in
29:05
general, All of these features quote, UN quote that you would
29:09
normally put in the recommendation model, but now
29:13
you're serving it up to the agent in different moments of
29:17
that workflow. Yeah, Yeah, exactly.
29:21
We have a team at Process AI that is completely dedicated to
29:25
building a model we call LCM large commerce model.
29:29
So it's a model that was fine-tuned based on user
29:34
behavior in our OPS in the ecosystem.
29:37
So whether user searches for something like something or
29:41
there's. So using this model, we build
29:45
representations of who the users are.
29:47
So it goes way beyond the last orders.
29:50
We know what type of customer they are.
29:53
We have some segmentation as well in there that's occurred
29:56
naturally while while using this model.
30:01
We know their their patterns. So that that's like the core of
30:05
who the user is. And we have similar things for
30:08
restaurants items. So it's, it's much more nuanced
30:12
than just a list of like order history.
30:17
So we use that in the main agent to to select the best way to
30:21
communicate to them, like select to select the tools in the best
30:25
way. But within the tools also we are
30:27
connecting to white food recommendation system.
30:30
So when we plug dishes, we also have models that were trained to
30:37
to show the best recommendation. So it it's in several places
30:40
that we're doing this. It's almost like there's a,
30:43
yeah, there's like a Rexis tool in a way or like I'm going to
30:48
use the recommendation tool and it's calls a machine learning
30:51
model. Yeah, yeah.
30:53
But there are several models for different use cases for that.
30:57
Plus the agent has the brain and we have LLM friendly
31:00
representation of the user. Like for instance, if you order
31:05
pizza in Brazil, you can put a lot of toppings and
31:08
customization. If you always put pepperoni on
31:11
your pizza, I know you're a meat lover.
31:14
This you don't see it from the order history, but I can extract
31:17
this nuanced information. So the next time you say I want
31:21
something healthy, I'm going to propose you something with meat
31:24
because I know you like it, right?
31:27
So the. Yeah, there is a lot of emergent
31:29
patterns that. You just can't get with
31:32
traditional machine learning models because they're so
31:35
specific. And so here you can infer it
31:38
because of the LLM being that intelligence.
31:41
Yeah, exactly. That's the the beauty of
31:44
exploring this field and we learn this as we go.
31:48
There is so much, yeah, so many learnings on this.
31:52
And we also get feedback from the user which is amazing to
31:56
see. I feel like, I feel like I, I'm
31:58
now like excited for this app. What are you guys?
32:00
When are you guys going to be in San Francisco?
32:05
Yeah, that's it. We got to go to Brazil.
32:08
That's the easier option than them coming to San Fran.
32:11
Do you have some kind of a checker agent that makes sure
32:15
what's happening is actually what should be happening, like
32:19
overseer? I don't know what that would how
32:22
you architect it, but. We have a system in place for
32:25
guardrails and making sure you know the response that we were
32:30
given as makes sense let's say, but we didn't choose a full
32:38
blown multi agent set up because we wanted to keep it simple.
32:42
Our use case is relatively simple.
32:44
I think what's, what's difficult is given the right
32:47
recommendations, having the right context, but you know that
32:53
the, the context of the agent is pretty self-contained.
32:57
We, we don't have an agent to schedule a trip, for instance.
33:00
That would be another agent. But the set of tools that our
33:03
agent has are, you know, compatible with each other.
33:08
We, we don't need another agent for that.
33:10
What we did, however, was to have a dynamic system prompt
33:13
that would change based on the state so that we would not need
33:17
to to have so much information every time.
33:20
And yeah, that of course help with latency and having less
33:23
choices to make to go back to the stratification we were
33:28
talking about earlier. The way that someone interacts
33:31
with ifood on WhatsApp is still almost like through the ifood
33:37
app, but that's just to verify their profile and then they go
33:40
back to WhatsApp. Did I understand it?
33:42
Correctly to kick off the first authorization flow.
33:46
So we need to make sure that you or you, you know, connect to
33:50
your profile. So yeah, they didn't.
33:54
If they don't have an account, if they write to us from a phone
33:56
number, they don't have an iPhone account, we need to make
33:59
sure that they create 1 like. And then you have all the
34:03
information about that user in ifood.
34:05
Yeah. And we connect, Yeah.
34:06
Including the payment options and all of that stuff.
34:09
And so the agent, I'm assuming just says, do you want to use
34:14
your regular credit card or do you want to use one of these
34:17
options? There are some systems in Brazil
34:20
for payments that we are connecting with.
34:23
Yeah, but. I remember Nishi talking to me
34:26
about how hard it was to context engineer the types of payment
34:32
systems that people would ask for.
34:35
And it goes back to what you were talking about at the
34:38
beginning of you end up just adding all these edge cases to
34:42
the prompt and before you know it your prompt is so bloated.
34:46
And if last night 6 hour stream taught me anything, it is like
34:52
the least amount of context as necessary.
34:55
If you think about what a user can ask in an app to order
34:58
foods, they could ask for specific payment method, price
35:03
range, delivery time, like distance from the restaurant or
35:09
they're vegetarian, they're one gluten free discounts.
35:12
They have a specific membership that they want to like that
35:16
allows them specific discounts. They want to use that.
35:19
Like there is so much. It's really impossible to give
35:23
all this information to the agent because then you need to
35:26
like the agent needs to convert that in the into a query, right?
35:30
If you give all this information to the main agents, it's going
35:32
to blow up the the prompt. So what we did was like having
35:38
converting the query into something more LLM friendly with
35:42
some basic knowledge from the main agent and then giving
35:45
context of like who the user, what the user wants.
35:49
And in a way this is like preparing a task for another
35:53
agent. So we would delegate and the
35:55
main agent doesn't need to worry about anything.
35:58
And that first touch is just through like a small language
36:03
model or sentiment analysis or it still is an LLM call.
36:07
No, we, we use an LLM straight away for that.
36:09
Yeah, it's it's simpler, but we implemented some classification
36:15
later in the way. So yeah, we we have some
36:18
classifier steps to simplify. Nice.
36:22
How much are you using the foundation foundational models
36:26
like Open AI or Claude versus your own?
36:31
I would say we use foundational almost everywhere.
36:36
We use smaller models, fine-tuned models for specific
36:39
steps in the workflow like creating representations for the
36:44
users for instance. It's a task where there is a
36:46
fine-tuned model. But for the conversational part
36:51
we haven't explored yet. I think foundational model are
36:56
models are so good right now that if you don't need to fine
37:00
tune, that's probably you shouldn't do it.
37:04
I think I hear a lot about fine tuning, like maybe we should
37:08
fine tune that to fix the output.
37:09
No, it's it's just a specification problem.
37:11
You're not specifying enough or you're maybe you're specifying
37:16
too much in the front. If something can be fixed there,
37:20
that's the first step. Well, I think it's a great, I
37:22
think it's a, that's a great point I'd love to dig into
37:24
because you know, you know so much more about this than most
37:27
people. Where do you draw the line on
37:32
when and where to fine tune me? You guys have an incredible
37:35
amount of data and I know that much of it is leveraged.
37:42
Where do you decide to do that work?
37:44
So I think one of the biggest reasons why you might want to
37:49
fine tune is cost and latency as well.
37:53
If you're building a model for a very specific task and the model
37:57
doesn't need to be able to converse with the user.
38:01
They just need to take user data and build a representation of
38:05
the user, for instance. So it's a very specific task and
38:08
then you you want to scale it to 60 million users and you want to
38:11
do it every day. Foundational model is not going
38:15
to going to work very well. It's going to be extremely
38:19
expensive. Yeah.
38:20
So and also performance wise, right, because at the end we're
38:27
talking about embeddings and you want, if you, if you're working
38:31
on a very specific space like food delivery, you want to, we
38:36
want the model to clearly differentiate terms that might
38:39
look the same, but are actually semantically different.
38:43
And that nuance you don't get in foundation model.
38:48
So you at the end when you're fine tuning, you're changing the
38:52
space where your model moves and you're making this the meaning
38:58
of words that might look similar, more far apart from
39:00
each other so that that that thing you want to do.
39:05
And we proved that for our application, it's better.
39:09
Actually we AB tested this and yeah, it get better results.
39:15
If you want to just kick start the projects, then yeah, first,
39:20
you know, make it work, then make it cheap, then make it fast
39:22
etcetera. Yeah.
39:23
It's interesting we see this as well with a lot of customers is
39:29
again today versus last year people used to start with all of
39:35
their own data and with fine tuning and that's that was the
39:38
beginning of the journey and now today it's the end of the
39:41
journey. You instead you throw a
39:42
foundational model at it. Most of the time it's good
39:45
enough. You do some prompt engineering,
39:47
you touch some tools and you're good to go.
39:49
But when you want that extra last inch, now that's where
39:52
everybody's using fine tuning. I'm curious within the work that
39:57
you've been doing, where's the dividing line of fine tuning an
40:03
existing large language model, whether it's a foundational
40:07
model or open source one, versus building your own model that
40:12
might not be LLM based? Yeah, it's very fine line.
40:15
Sometimes I think regular like more traditional machine
40:20
learning approaches are good to get patterns from the data like
40:23
when the data is numeric or when you have graph type of data like
40:29
collaborative filtering for instance.
40:31
We're also working on that. I think traditional ML can find
40:37
patterns that and an LLM like conversational type of LLM would
40:43
not necessarily find because it can leverage also connection
40:47
between user data points. You're embedding different type
40:51
of information. So for instance, to give an
40:54
example with collaborative filtering, the the model might
41:00
know that we are similar users and I ordered some foods that
41:04
you haven't ordered yet and then we'll suggest you this food.
41:06
I don't necessarily have this connection in an LLM that is
41:10
conversational. So yeah, that's there is a very
41:16
fine line. What what we're trying to do is
41:18
use LLM for finding patterns based on the user behavior.
41:25
Things that are not just hard data like the the example I
41:29
mentioned before where you ordered pizza with pepperoni.
41:32
You might be a meat lover. I know you're not.
41:34
But from the opposite. Yeah, but I.
41:37
Understand but you know just to give you an idea.
41:40
And then I can extract the filter that I will use in every
41:44
query to to filter the data that I show you.
41:50
Yeah, with traditional machine learning, I wouldn't get these
41:52
patterns out. Yeah, it's it's a hard question,
41:57
I think because it yeah, at the end it boils down on what what
42:01
type of data you want to show to the user.
42:05
I think for data to fit to LLMS, sometimes it makes more sense to
42:10
get like text based representation.
42:13
But when you we need to do operation like search in a
42:17
database, yeah, that you need vectors, you need to to
42:22
optimize. Yeah, based on vector
42:24
representations. Yeah, it's a, it's a very good
42:29
question. Yeah.
42:30
It's cool to hear that you're still grappling with it and
42:32
there are these pros and cons of each and obviously it's always a
42:36
trade off. I like the idea of how you can
42:40
leverage 1 and at the same time kind of plug in the machine
42:45
learning models. So it's like the majority of the
42:49
stuff is happening with the large language models and the
42:51
agents that are going and they're doing stuff and maybe
42:54
one node or one tool that it can call is a machine learning
42:56
model. I think one of the things that I
42:58
find most, most interesting about process is how far, how
43:03
far ahead and how advanced you folks are in agent building.
43:08
And you've got a target for 30,000 agents in the
43:10
organization, which you know, which is funny because you know,
43:14
we can laugh about how crazy that number is, but then how
43:16
many do you actually have right now?
43:18
It's it's more than anyone else and so.
43:21
Not that far off. Yeah, like you, you have at
43:24
least over 1000 at this point. And yeah, over 10,000.
43:27
Yeah. And so, you know, we can joke
43:29
about 30,000, but most people are struggling to get one out
43:32
right. And and we've talked and we've
43:35
talked a fair bit and I'm sure we'll talk more in in different
43:37
sessions about how what it took to get there both from a
43:40
technical perspective and from an organizational perspective.
43:43
But you had made a really interesting point about tools
43:48
and and how, you know, different teams start in one way, but then
43:54
eventually you guys are thinking you guys start layering in
43:57
governance. I'm wondering if you could share
43:58
more about that. Yeah, this is I think transition
44:03
and happens in an other organization as well where we
44:06
are approaching Gen. AI and we have very vertical
44:11
type of vision. We're trying to build a product
44:13
for specific use case, right. So what what you do if you're a
44:19
team building this is you go very old, lean and very deep in
44:23
this vertical direction. But if you do this and you
44:27
multiply it for 1020 teams in our organization, you realise
44:30
that often similar tools are being built, for instance,
44:36
answer the user questions using knowledge base, right?
44:41
So what what I think tools are really good for is also creating
44:47
layers of governance where for instance, in the case of the
44:51
knowledge base, there is a team that makes sure that that works
44:54
very well and then can be shared across different teams.
44:58
The the trick here is that the teams defining these rules needs
45:04
to be very LLM savvy as well because the interface that this
45:08
tool would use needs to be used by an agent.
45:11
What we see sometimes is that these definitions get to
45:15
verbose. There is a lot of requirements
45:18
for how the tool should be used and like the language that
45:22
should be used. Sometimes you don't need all of
45:23
that and you really need engineers to work together to to
45:28
define the interfaces of this. And I think if you nailed that,
45:32
then you can really scale up because you can then share tools
45:35
across the organization. And this also applies to agents
45:38
by the way in multi agent setups.
45:41
It's having that ownership of the tools, being able to clearly
45:44
define this is your tool. You're expected to keep it up to
45:49
date to make sure that it's working and that the agent can
45:54
consume it in a way. Yeah, exactly.
45:58
And when you create tools that can that are directly connected
46:04
to the public image of the company, like you're defining
46:08
persona that represents the company, then the type of people
46:13
who write these rules are not developers, right.
46:15
So the the person defining these rules maybe is a designer or
46:21
like product team, but you cannot give them to the agent as
46:25
they are. You need some some translation
46:28
layer in here. And I think, yeah, governance is
46:31
really it's really important and it's important to get it right
46:35
because once you do it once, then you know, it's it's more
46:39
maintainable. And if I would let ILO be used
46:43
by other agents, for instance like expose it through agent to
46:47
agent framework or I'll let other teams use it, then I also
46:52
need to make sure that that is used correctly.
46:57
Yeah, I, I think, I think what we've been seeing, which echoes
47:02
your experience is it's very easy for an agent team to get
47:09
their agent working in, you know, in a silo and they'll
47:15
build the tools they need and, and it'll work.
47:20
But there's almost always an organizational context that
47:26
agent team is focused on its agent, but its manager or the
47:30
director or the VP or the CTO or CIO is looking horizontally.
47:35
And they're the most organizations are typically
47:38
building more than one agent. And the same problems that
47:42
everybody saw in the last cycle with APIs is, is the exact same
47:48
problem now with tools, MCP or not.
47:52
Like take the wire protocol out of it.
47:54
It's OK, great. These two people are inserting
47:57
user records into the CRM. Why do they have two different
48:02
tools that are being maintained separately with different logic?
48:06
Like that should all be the same thing.
48:07
Let's elevate that and make that a shared tool.
48:10
But then when you do that, you suddenly introduce a governance
48:13
problem that's never been resolved before.
48:15
Which tools? Well, how you know, how do we do
48:17
versioning? How do we do ownership?
48:20
Who gets access to which tools? Team A, which is working on
48:25
customer facing. We talked about this last night.
48:27
Agent A is a customer facing agent and Agent B is an
48:31
internally facing agent. And those teams probably
48:35
shouldn't be seeing the same tools because the policy at an
48:38
organization, I think, which is the policy here is an internally
48:42
facing agents shouldn't have any access to anything outside of
48:45
the organization. And so like sharing is the, the
48:50
beginning of it, but one, the moment you start sharing tools,
48:53
which is a, a best practice, it looks at first like a
48:56
productivity gain, but immediately you inherit a bunch
48:59
of governance challenges and governance gains.
49:03
So for example, who has access to the tool, this agent, What's
49:10
the policy for how access is being doled out, not just to the
49:14
individual developers on different teams, but to the
49:17
nature of the agent itself, internally facing agent versus
49:20
externally facing agent? And how do you handle versioning
49:24
and if and if you're the one that wrote the user insert tool
49:27
for the CRM and my agent depends on it, who owns the tool?
49:33
Like who's in charge of bug fixing it?
49:36
Is it me? Is it you?
49:37
Yeah. And when this impacts multiple
49:40
agents, like changing a tool definition is actually changing
49:45
the the prompt that the agent has access to.
49:48
So it's crucial to evaluate because if you, if you change
49:52
that, you're going to impact all these downstream tasks.
49:55
And it's, yeah, it's really crucial to have good evaluations
49:58
in place, not just for the tool, but for how the agent will use
50:02
this tool and will interact with that.
50:04
And and I to link it back. I mean, I think this is where
50:08
laddering up tools really matters because my agent
50:13
consuming the shared, you know, user insertion into the CRM tool
50:18
is going to likely have different set of evals than your
50:22
tool because we just have different contexts.
50:24
We have different intentions. And and I'm gonna blow your tool
50:31
irreparably for all the other agents if I try and insert all
50:34
of my needs and demands onto it. But if I then if I'm just, if I
50:39
instead I just layer up my own domain specific tool or agent
50:44
specific tool that sits on top of yours, then suddenly sharing
50:48
and repeatability still works without having to sacrifice
50:54
accuracy and consistency and latency for for the agent.
50:58
So now I can share that we can share the same tool and I can
51:01
have my own little domain specific issues with my own set
51:04
of evals that are maintained in my agent separate from yours.
51:06
I see what you're saying, where it's like you normally have to
51:10
go really deep on something and really craft it so that you
51:15
understand the problem set, you understand how to build that
51:18
agent in the best way possible. But at the same time, I feel
51:21
like if you're getting that detailed with it, you're now
51:27
creating a whole lot more work for every team.
51:30
Yeah, you are, you are absolutely creating more work.
51:33
But this is this is the trade off, right?
51:34
It's all about knobs and dials. It's, if you're a single agent
51:39
team, you don't need this, right?
51:42
But the and, and, and the individual agent developers, you
51:45
know, aren't going to be thinking about this.
51:48
It's the moment that you're running an agent program, you
51:51
know, call it a center of excellence or whatever.
51:54
I mean, what, the moment you're building multiple agents across
51:57
an organization, if you're a Fortune 2000 and you're, you've
52:02
even here, you've amended for 30,000 agents.
52:05
And this becomes a requirement at scale.
52:09
If everybody's rebuilding the wheel every single time you have
52:15
you, the productivity gains of, of flipping the script become
52:21
pretty high. Because when I go start my, my
52:24
new agent, I don't have to go build all this stuff from
52:28
scratch. I can go see in the registry,
52:31
Oh, what are all the different tools that already exist?
52:34
Oh, let me pick that one. Let me pick this one.
52:36
Let me pick this one and then I can build my own tools on top,
52:39
personalize them and everybody wins.
52:42
But that's just develop a productivity piece.
52:44
If you think about senior leadership and and the
52:48
governance problems, it becomes possible to put your arms around
52:52
it. If you don't have any kind of
52:55
reuse for reusability or registry or things like that,
53:00
you know, when the CSO or the compliance team or senior
53:03
leadership worried about performance wants to go see
53:07
what's happening, they have to go look at what 300,000 tools?
53:12
That's not going to work. They're all marginally different
53:15
and that's that's a fail, right? Yeah, at the same time, there is
53:17
no substitute for really digging deep into a problem, because
53:22
starting the other way around, I think would be a disaster.
53:25
Like if you start from defining what tools can be built for the
53:29
teams before even you're solving the business problem, that is a
53:34
recipe for disaster. Like you're going to over
53:36
engineer things and like this was going to have to be changed
53:39
so many times. So yeah, what we found at works
53:43
was really to to build something, you know, you know,
53:47
like building an agent in production is way different from
53:50
thinking about an agent like you, you you find challenges you
53:55
didn't even think about. So before you have done that, I
53:58
think you shouldn't even even start thinking about how this
54:03
can be shared across teams. I, I, I, I agree with a few
54:09
nuances. I think that's, that's generally
54:11
the best practice. We, again, we talked to a lot of
54:14
customers and they're all coming from different angles, right?
54:16
You know, there's a particular agent team that needs to go
54:19
unlock a particular piece of functionality.
54:21
They want to talk to e-mail or calendar or, you know, some
54:26
custom service. And then we'll also talk to CIOs
54:30
and, you know, their VPS who are trying to architect the
54:33
organization and the enterprise for agents.
54:36
And we, we have a saying internally, which is regardless
54:42
of where we come in as a vendor and to help an organization, we
54:46
always drive the conversation to the first agent for that same
54:51
reason. It's like, well, before anybody
54:53
gets caught off, you know, in, in building like the Sistine
54:57
Chapel of, of complex agent governance and systems, let
55:01
let's first make sure that you've got your agent, your
55:05
first agent at least successful and working and you got the
55:08
right patterns. And you're absolutely right.
55:11
You should go deep 1st and then once it's working, then abstract
55:14
out and then start optimizing for sharing and reusability and
55:18
all that fun stuff. But you can't start there.
55:20
Whoever with some third party tools that are, you know, pretty
55:25
widely accepted by now, like managing calendar emails, I
55:29
think, you know, that's a, it's a good bet to, yeah.
55:33
Those are easier, which is how you guys use arts.
55:35
Good reason to incorporate those, right?
55:38
Everybody should use Arcade. And on that we get.
55:43
Dude the governance piece is wild.
55:46
Oh my God, It's become like the conversation everywhere.
55:49
It's crazy because like three months ago I never heard the
55:53
word. Well, I remember I told you
55:55
after I was in San Francisco, I was like, oh, you see?
55:57
So I was talking about. Government, when you and I
55:59
spoke, I was like governance. Now it has been every single
56:02
conversation we're in. It is just, it's wild to me how
56:05
fast everything's changing. The maturity curve for most
56:08
organizations is rapid. We notice also teams using multi
56:12
agent setups way more and sometimes agents look like they
56:17
have multi agent setup where agents look very similar to each
56:19
other. And when when you start to dig
56:22
deeper into the reasons, it's governance.
56:24
Yeah. I'll tell you what I'm seeing
56:26
though, my bet a year from now when we do this podcast again, I
56:30
think multi agent systems are going to be a lot less common.
56:32
Oh, interesting. Yeah, we're starting to see the
56:34
early signs of it. Felt like as we talked through
56:38
this, agents and tools can be almost like interchanged in a
56:43
way. Yeah, actually we actually our
56:46
sales engineer Shub, who is incredible, he he basically
56:53
proved to all of us in a really dramatic way.
56:56
I don't think he's even doing it on purpose that an agent is
56:59
really just a collection of prompts and tools.
57:02
And people say it, but nobody really believes it, Susan that
57:05
they all want to use all all the big systems and Shub just like
57:08
slammed together a little YAML based agent builder and it
57:12
fucking works. Like take that.
57:14
Yeah, it just works. You just really just prompt the
57:16
tools and you're done. And it's got land graph under
57:18
the hood. But it's wild, right.
57:21
And what we've been seeing is the scope definition of a of an
57:27
of a sub agent is is getting bigger and bigger and bigger as
57:33
context windows get bigger as it can handle more tools all of a
57:37
sudden like you're seeing bigger sub agents.
57:40
And so if you take that to the limit like you're not going to
57:43
see as complex of sub agents as you've both been saying.