Latest / Agentic Conversations (formally mlops.community) / The Agent Landscape - Lessons Learned Putting Agents Into Production

Transcript

The Agent Landscape - Lessons Learned Putting Agents Into Production · Agentic Conversations (formally mlops.community)

0:00
Welcome everyone to a conversation between myself and
0:04
my good friend Paul Vanderboer to talk all about how the
0:08
Process team is using AI agents in their companies, specifically
0:14
Process and maybe sometimes their portfolio companies.
0:18
We get into all the nitty gritty details on how they have been
0:22
innovating and what some of the challenges have been
0:26
specifically around using AI agents.
0:30
Let's get into this conversation.
0:37
We're going to be talking all about agents and more
0:39
specifically how you all in this global group of companies,
0:45
including Delivery Hero and OLX are using agents.
0:49
You've got over 1000 ML practitioners in this group and
0:53
you're bringing agents and AI use cases to the over 30,000
0:59
people that make up the global collective group and knowing
1:03
that you've had some hard earned lessons.
1:06
And that's really what I want to dive into.
1:09
Technical hard earned lessons, user adoption, UXUI, all of the
1:14
fun stuff because you've been doing stuff since 2022 when
1:20
ChatGPT first came onto the market and trying to figure out
1:24
how to make that more useful. We should probably start with
1:29
just like what we were talking about yesterday when we had a
1:32
bit of a Mexican standoff and we said, So what is an agent?
1:35
We both looked at each other and we're like AI that can do stuff,
1:39
right? You might have a better example
1:42
of that. Let's start with that.
1:44
So what is an agent, right? So an agent in my simplest
1:48
description is essentially an LLM that interacts with the
1:51
world. And I'm like, we've obviously
1:55
had, you know, LLMS or anybody in the space working with LMS
1:57
for for years now, right? And the first versions of them,
2:01
GPT 2 and so on. We've been playing around with
2:03
everyone sort of figuring out what's what's coming.
2:06
But at the end of the day, those LMS were fairly isolated tools,
2:14
right? That's a little reasoning engine
2:15
in a box. And you can give it a token and
2:18
it gives you a token back. And that was, you know, of
2:20
course, super impressive. And we saw that with the ChatGPT
2:23
moment that sort of jumped in, in sort of everybody's lives.
2:27
But one of the things that, you know, we saw as a fairly obvious
2:32
next act of generative AI is agents when these LMS would be
2:37
able to interact with the world and you know, how they interact
2:40
with the world. Obviously, sort of it's almost
2:42
like I see like a dial, you're dialing it up right to the 1st
2:45
things we we saw were well, that maybe they can access the web,
2:48
maybe they can access, you know, compute environments, maybe they
2:51
can access APIs, maybe they can start to interact with the
2:54
browser. So that's sort of the this this
2:57
idea of an agent. And to your point, you know,
3:00
we've been working on this for years because if at pros is
3:03
where, you know, probably one of the world's largest tech
3:07
investors focused heavily on e-commerce serving about 2
3:10
billion consumers through those various companies that you
3:13
mentioned. 2 billion, two billion.
3:15
That's a lot. Yeah, it's a lot.
3:16
It's a lot, right? So.
3:17
You know how big that is? Yeah.
3:19
In you know, I food in Brazil and Swiggy in India and Stack
3:22
Overflow and delivery Hero and OLX and you know many other
3:25
companies, about 100 companies in the group.
3:29
It's you know, there's a ton of different opportunities for AI
3:33
in general. And then if you go to, you know,
3:36
agents becomes incredibly interesting and exciting to see
3:40
all the different things we can, we can build.
3:41
So that's why we've been investing in this space for for
3:43
a long time with our team here in Amsterdam.
3:46
We organized this conference. Yeah, that's kind of the the
3:50
inspiration for this whole series is because we did the
3:53
conference together as a virtual conference.
3:55
And then we realized we want to create more and go deeper.
4:01
Because what I saw in that conference was that you all are
4:04
doing some very advanced stuff with agents.
4:07
And the conference was all around agents.
4:08
It was agents in production. It was a virtual conference.
4:11
We saw the most cutting edge things that are happening with
4:15
agents out there in the world. And my conclusion was we need to
4:19
have more conversations because I want to hear what you've done.
4:23
And so this episode is going to be us breaking down the agent
4:27
space, what is comprised of. And then we'll bring on Flores,
4:31
our good friend who's going to talk about some hard earned
4:35
lessons that you all have had, what agents you've tried that
4:39
died, so the whole graveyard of agents.
4:41
And then what agents you actually were able to stick with
4:45
and have been providing business value.
4:47
How are you looking at that business value?
4:49
How are you like putting metrics around if the agent is useful or
4:53
not? And so before we bring him on,
4:57
we should talk a little bit more about like what components make
5:01
up an agent? But maybe taking a step back and
5:04
think about thinking about these agents and then why?
5:06
How do we why is it so hard to make them work?
5:11
And why is the graveyard still, you know, populated, populating
5:15
itself fast is because there's a lot of unknown pieces.
5:18
So think of an analogy, right? I don't know if you ever built
5:21
things, you know, like with Raspberry Pi or with, you know,
5:25
Mars Rovers have a young son. So we're building a Mars Rover,
5:29
which is essentially a Raspberry Pi, which connects to a, you
5:33
know, a bunch of sensors, a microphone, a camera and A and a
5:36
memory chip. So you can see sort of what its
5:39
path is. And for me, if I take this to
5:41
the world of Jenny I, what we basically have today with these
5:44
powerful Jenny I models, large language models is just the
5:49
reasoning engine is basically the Raspberry Pi without
5:53
anything else, right? And that was the LLM.
5:56
And now in the agentic world, we're trying to figure out how
6:00
do you put this into a system that can interact with the
6:04
world, right? That can actually understand,
6:08
you know, what history of interactions it had.
6:11
So memory like this Mars Rover needs to know what's its path,
6:14
where does it need to go? It needs to maybe be able to
6:17
take actions and, you know, decide to go left or right in
6:20
the agentic world. It's I need to access an API to
6:22
fetch information or I need to store, you know, create a file
6:26
and store it somewhere. It needs wheels.
6:28
That's it. So, and the tools are part of it
6:32
because that gives the LM the ability to interact with data
6:39
that's up to date. Because of course, you know what
6:41
the data it's seen as a during training time, you know, has a
6:44
sort of endpoint. So you need, you know, other
6:46
data, maybe that's proprietary data or data related to, you
6:51
know, something that happened today.
6:53
It may need to actually generate its own data to not just read
6:58
but also write because it, you know, knows that you and I are
7:01
interacting and you've asked me certain types of questions over
7:03
time. So that needs to be stored in
7:05
memory. What does that mean?
7:07
When does that memory get accessed?
7:09
Then, you know, when it generates an answer, it may want
7:15
to actually think and critique that answer.
7:17
So it's not just a one shot token prediction, but it
7:20
generates a plan based on what you're asking it to do.
7:23
It can go to that plan, critique it, look at the end of, you
7:26
know, the steps it's followed, whether that plan materialized,
7:29
go back to step and revise. So that entire system is what
7:35
you need to have working not just once, but reliably,
7:41
especially if you're going to ship it to real customers in
7:43
production and so on. And that's been the journey
7:46
we've been on, right? And but it's from it, you know,
7:49
the, the journey has been moving from small unit of reasoning,
7:53
the LLM, the Raspberry Pi that now needs to be reliably
7:58
connected to all these other pieces.
8:01
So it can help with much more sophisticated tasks and in fact,
8:05
that's the promise, right? So moving from just a simple Q&A
8:09
device to something that can, you know, actually help with
8:13
much more sophisticated complex tasks, especially in the
8:15
e-commerce journey in our ecosystem at Pros, is there a
8:18
ton of opportunity to do that? The Raspberry Pi to the Mars
8:21
Rover, that's what we're doing right now.
8:23
And that's like that metaphor. And there are some specific
8:29
difficulties that arise when you start doing that.
8:32
I think two things that we wanted to call out, especially
8:35
right now are evaluation can be really difficult and you're
8:40
looking at latency requirements and then cost too.
8:45
Yeah, because as you mentioned, if you're doing all of these LLM
8:49
calls, it can add up quickly. And we're going to go over all
8:54
that fun stuff later on in different episodes and breakdown
8:57
specifically how you can do evaluation and what you can look
9:01
for. But now we should probably talk
9:04
about the different ways that you can use agents or that
9:08
agents manifest, I think we could call it, because back to
9:13
that conference, we saw lots of different ways that agents are
9:16
being used. And in a broad sense, I kind of
9:21
bucket agents into you have agents that are like the
9:24
computer use agents that came out from Anthropic and it can
9:27
use your whole computer and take over.
9:29
You have web agents, which are a little bit more of a
9:34
intermediary between they're not using your whole computer, but
9:37
they're using your browser. And then you have agents that
9:40
are interacting with the world through APIs, which I think is
9:44
probably the most common design pattern these days.
9:47
And then you also can have voice AI agents.
9:52
And so interacting with agent that you're talking to on the
9:57
phone or maybe on a Zoom call. There's probably room for us to
10:02
throw in there, like agents that you see in video games and PCs
10:06
maybe. I don't know if you want to call
10:09
that a full blown agent, but it feels like that could be a one
10:13
also. Yeah, for sure.
10:15
I mean, you're describing the spectrum of complexity, and I
10:19
think it also gives us a sense of where we're headed in the
10:23
future. Some of it was very near.
10:25
So indeed, you know, the natural first a set of tools that you
10:28
want to give these agents to our AP is because they're well
10:31
documented, well structured, you know what needs to go in, you
10:34
know what you expect back, you can test against that.
10:36
It makes evaluation a lot easier.
10:40
So that's the first thing that if you know, the agents in
10:43
production that we are working with are typically going to be,
10:45
you know, using well defined AP is that are, you know, fairly
10:50
simple compared to doing, let's say more open-ended web
10:54
browsing, for example. Of course we're testing and, you
10:56
know, we can share what we've learned and why.
10:58
That's also, you know, very hard and more expensive and takes
11:01
time. We're.
11:02
Going to have a whole episode on web agents and so there is going
11:06
to come and be our resident expert because that's the fun
11:09
thing too. That and we should mention, we
11:12
get to pull from all the folks that are working at Process who
11:15
are doing deep dives on each one of these topics and they get to
11:21
come and tell us what they've learned over the past six months
11:24
just focusing only on that. Right.
11:26
Yeah. So that's exactly what we're
11:28
going to be doing. And I think if we, if you walk
11:30
along the sort of levels of sophistication, maybe we need a
11:33
framework for this. So somebody, I don't know, maybe
11:34
somebody's come up with what are these sort of levels of
11:36
complexities for agents, But, you know, from APIs to browsing
11:40
and just in these two levels, if we call them that way, there's
11:44
so much opportunity still to make these things work.
11:46
I mean, in the world of e-commerce, online marketplaces,
11:49
platforms, you know, there's so many things that if you just can
11:53
give an agent access to the web, to the app, to the API's that
11:57
they can help you with, all of a sudden they give booking trips,
11:59
ordering food, you know, helping you pick products and so on.
12:03
That's after web browsing. The next thing is sort of just
12:05
giving them access to a computer, a desktop.
12:07
And we're working with, you know, various companies,
12:09
startups out there that do exactly that.
12:12
And you can see the progress like on benchmarks like OS World
12:15
and so on that these agents can now, you know, basically create
12:19
pivot tables for you. They can, you know, with very
12:21
brief instructions, right? Or they can download files, they
12:24
can process them and so on. And that is going to come soon,
12:29
2025 probably give us, you know, a whole bunch of new exciting
12:32
things and then products on that front.
12:35
Then if you go one step further, they can start interacting with
12:37
the real world, right? So, robotics.
12:40
That's another agent that I didn't even talk about.
12:42
That's true. So.
12:43
That's sort of the the levels of sophistication that I think we
12:47
expect to kind of see maturing over the next, you know, months,
12:52
quarters, sometimes things go faster.
12:53
Some prime real estate for some real thought leadership there
12:57
with that map of the difficulties.
13:00
Let's do it. They're volunteers.
13:03
So one thing that we didn't really talk about is why use
13:07
agents and why not just use traditional methods?
13:12
Because it seems like we add a lot of complexity.
13:16
There is that benefit of, hey, I can just tell something to go do
13:20
it and it'll do it for me. But a lot of times you end up
13:25
banging your head against the wall because it is so difficult.
13:28
I think the, the simple answer is that you, you just add so
13:32
many more possibilities that you can, or tasks that you can have
13:36
these systems do for you. Once you move to an agentic
13:39
world, you give them access to tools that it's the obvious next
13:43
thing to do. Because, you know, we've, we've
13:45
kind of gone through the question answer, you know,
13:49
world, world, and we're doing that and sure that, you know,
13:52
we're, they're all these AI systems are part of our our
13:54
lives now and they do that reasonably well.
13:57
Of course, there's a ton of room for improvement, but as they
13:59
become agentic, there's many more things they can do.
14:01
And then, you know, again, it pros as we're a large ecosystem
14:04
of companies that, you know, help our users do things easier,
14:09
better, faster as they interact with the, you know, the our
14:13
e-commerce platforms, We see that the agentic capabilities
14:17
allow us to do much more on that front.
14:19
So, and by the way, I will also say that one of the things you
14:23
notice and we'll talk about what we learn as you try to apply
14:25
this to the current systems is that the world is not ready or
14:28
it hasn't made been made for the agentic systems at the interface
14:32
is the API. Sure, they exist, but they're
14:34
not made for agents to interact with them.
14:36
Well, they break all the time too.
14:38
It's really hard to get a very trustworthy API that even just
14:42
for weather, a weather API that yeah, you can go looking and
14:47
just getting that which you would think is a solved problem
14:51
and it's simple. That's hard.
14:53
And so trying to go a few steps deeper and get more complex APIs
14:59
or each API is different and so and it's constantly changing,
15:02
right? And if you're not up to date
15:05
with those changes or making sure that you have some way to
15:08
keep the your agents up to date, then you're looking at a whole
15:12
world of hurt. That's right.
15:14
Preaching to the choir right here, there is something I want
15:18
to bring up to around how process works with the different
15:22
companies because I tend to funnel everyone that I know that
15:25
is starting a company to you because you're in such a unique
15:29
position. And this position is that you
15:31
have a ton of users, a ton of ML talent, and you know what
15:38
problems need solving. And so there's users from the
15:43
portfolio companies, but then there's also internal users
15:46
because the process group is gigantic.
15:48
Maybe we can touch on that a little bit before we jump into
15:52
it because that gives more insight on how agents are
15:57
valuable to you all and how you know like what is actually worth
16:02
doing versus not. Yeah, that's a great point.
16:06
So because the setup we have is process, we're a large global
16:09
tech investor, the largest in Europe with operations all over
16:13
the world, right? We've got food delivery in India
16:16
and in Brazil and classifieds in Eastern Europe and education
16:21
technology in the US and media in South Africa and many other
16:25
comes about 100 companies in the group, all with a tech angle
16:29
with their own tech hubs and AI teams.
16:33
We are in a very unique position to be able to work with them
16:37
closely on lots of topics. You know, our focus of course is
16:41
AI and, and increasingly now agents to figure out how can we
16:44
solve real user problems. Like I mentioned, we have about
16:47
2 billion consumers across the group and. 2 billion.
16:51
That's a big group and you know, they're all over the world.
16:54
Every time you say that, I'm going to react.
16:56
With this year it's 2 billion, probably next year's 3 billion,
16:59
yeah. Oh my God, so.
17:01
And, and we do believe that, you know, the, the agentic systems
17:07
that we're building are going to be able to solve lots of our
17:10
user problems, helping make bookings, find, you know, make
17:13
transactions easier, find the right price they're looking for,
17:17
learn things faster. And so all of these, let's say
17:23
real user problems are things we're trying to solve for.
17:25
And when we, you know, our team, the AI team, it pros is, is
17:31
based in Amsterdam and our job is to work very closely with the
17:34
AI teams in the, in the group companies to help them basically
17:38
accelerate some of the cool use cases we think are going to be
17:42
really valuable for the group in an e-commerce space in
17:46
particular. In doing that, we to identify
17:50
typically what problems are. So when you build a systems in
17:52
production, we talked about all the issues you need to make it
17:54
affordable, you need to make it safe, you need to make it
17:56
scalable. So identify problems that we go
17:59
out there to find if anybody's offering them as a product or as
18:04
a solution. So we'll typically talk to
18:06
founders, startups in this space when we like what they're doing.
18:11
We, you know, either partner with them as design partners.
18:14
You know, we also can invest in them.
18:16
You know, we've tons of examples there.
18:19
And that's cool. On the design partner front, it
18:21
is so valuable to have a company that is so advanced and
18:26
understanding what is important and then just to be able to plug
18:32
in with you all. I know that a few companies I've
18:35
introduced you to, they come back to me and they're like, Oh
18:37
my God, thank you so much. Because again, the whole reason
18:42
we're doing this is I think you all are doing some of the most
18:45
advanced stuff when it comes to agents.
18:47
When you get companies that become design partners, they get
18:50
to see how advanced you really are.
18:52
And so if they're on the cutting edge, they get to see the scale
18:57
of what you're doing and then recognize, like, if their tech
19:00
holds up to that scale. Yeah.
19:02
I think it's a valid point. I think the problems we face
19:04
today are probably challenges that many others will face soon
19:09
as well, whether that's in months or years, as they also
19:12
start to build these systems in production at scale.
19:16
One of those things was the cost, right?
19:18
That's one, yeah. That's a great example where
19:20
we've been continuously modeling what's the impact of agentic
19:24
systems in production on the cost profile.
19:26
And the numbers we were looking at at the beginning were just
19:31
simply, you know, cost per token.
19:32
But you realize that's not really representative because as
19:35
you use agents to answer questions or fulfill tasks, they
19:39
use many more tokens can do many more things.
19:41
They we measure for internal assistance token that you spoke
19:44
about how much time they save per per user, right, per
19:48
question. So these systems can do that
19:50
much better. They consume more tokens, so
19:53
they become more expensive, but the value you get for that is
19:55
higher. And so we model these things
19:57
and. I blatantly stole that, by the
19:59
way. I took that from you guys from
20:00
the agents and Production conference and made a blog post
20:04
on it in response to some other VC who was talking about.
20:08
He was saying the same narrative that you see all over the
20:10
Internet when a new model comes out or a new update from Open AI
20:15
or Anthropic comes out. And they say like the cost is
20:17
just plummeting per token. And so I took your or EUR, one
20:22
of the other speakers from the conference Insight, which was?
20:27
Price per token is going down, but price per answer is actually
20:31
going up because of these complex systems that we've got
20:33
going on and how many LLM calls you're making.
20:37
We all intuitively know the costs go down OK, but we measure
20:40
this right. So to give an idea.
20:42
So we over the, over the last summer, we looked at how many
20:46
tokens do we use to answer a given question in our internal
20:49
assistant that went up by 150%, right?
20:52
So more than double just a number of tokens per question.
20:55
At the same time, that same period as about a 3 1/2 month
20:58
period, the cost per token went down by about 50%.
21:03
But because the questions we're answering or the tasks we're
21:06
doing with token also become more useful, right, We're saving
21:10
people more time. Because they're using it more,
21:11
right? They're also using it more so
21:13
then the tokens per user go up. So ultimately the budget that we
21:19
have, you know, and token. Starts to double.
21:21
Actually goes. Up, yeah.
21:23
Right. And so and then we measure how
21:25
much time we save per question and that also goes up.
21:28
So we actually model this and we have real time insights we use,
21:32
you know, we we will benchmark various models on quality, on
21:35
cost of our own leaderboard as you know and so on.
21:37
We'll talk about that. But one thing is everybody knows
21:41
the cost of token goes down, but what's the ROI on that that you
21:46
get, right. So does the return on investment
21:49
is a dollar of the cost per unit of intelligence goes down, but
21:52
does the return you get on, on that intelligence change as you
21:56
build a genetic systems and so on?
21:57
So we're in a position to measure this across various
22:00
tools. Cost per unit of intelligence.
22:03
Yeah, that's how we, that's a way to describe a token, right?
22:06
So you've got a token is just a basically a part of a generation
22:11
and together it's, you know, system that has some
22:15
intelligence. And and so anyway, we see the
22:20
cost of per unit of intelligence trends to 0 over the long term.
22:24
0. Yeah, I mean it look at the cut.
22:27
I think they we looked at we, we spoke about this, the
22:29
marketplace, the cost of a token equivalent to GPD 3.5 two years
22:35
ago dropped by 98%. But of course we're, you know,
22:39
we've got more sophisticated models now, the reasoning models
22:42
that one and so on. But you also asked me about
22:46
which, which companies we work with, right.
22:48
So I'll give you one example. We learned that.
22:52
We saw that as you build systems that get more degrees of
22:55
freedom, because they're agentic, they can do more
22:58
things. They can you.
22:59
Know just you saying that scares me.
23:01
Yeah, but it's also well, they they they can the.
23:04
Way you're wording it just as. They're generating, yeah,
23:07
they're generating answers. They're going out to sources.
23:10
They've got my bank details. If you've given them to token,
23:14
it will be a great partner for you.
23:15
It'll be very safe. Yeah.
23:17
But we we realize we need to make sure we understand what's
23:19
the risk. And so we invested in a company
23:22
called Prompt Armor. Then their their mission is
23:25
basically to quantify through think of it like pen testing for
23:31
Jenny I systems, what the risk is.
23:33
So, you know, we work with them. Of course we invested in them.
23:37
So trying to get my agent to buy stuff for somebody else.
23:41
Well, this is more like on the security side.
23:43
So it's like a pen running a pen pen test.
23:45
So it's on the it's on the infra, it's on the entire
23:48
system. So you just have I think of it
23:50
like you have a chat bot or system.
23:51
It's Jenny I power that can give you, you know, answers.
23:54
It can go out there and try to prompt, do prompt injection
23:58
attacks, try and do data exfiltration and all of the
24:01
other new vectors. Basically, you open up a whole
24:04
new risk surface area that you need to understand.
24:08
And that's one example where of course we invest in them because
24:13
we think it's a promising product, but it actually coming
24:15
back to the ecosystem process is they can now offer their product
24:20
if we believe it's useful. We work with them to everybody
24:22
in the group, everybody that's building Jenny I systems.
24:26
And I think that comes back to how we work with founders.
24:29
It isn't just about investing in them like a traditional fund
24:33
would and hope there's, you know, fabulous returns, but it's
24:35
about how can you know this offering that they had this
24:39
group of founders join this global ecosystem that you know,
24:43
where the sum of 1 + 1 is whatever 11, right?
24:47
Because because we're now working together.
24:50
What they do isn't just, you know, in itself an interesting
24:52
proposition, but it make sure it adds and is additive to
24:54
everything else we do across the group.
24:56
And we see that a lot and we're doing that increasingly.
24:58
Of course, our focus is e-commerce, but sort of with the
25:03
intersection of AI, there's a ton of new ideas, propositions,
25:07
products, you know, techniques, tools that we're looking at very
25:12
closely to kind of bring them into the group.
25:15
All right, now we got floors with us and it is cool to have
25:18
you here to talk about all the things that you've been working
25:21
on. All right, Floors.
25:22
So what do you do? OK.
25:24
Yeah, thanks. Yeah, So I'm an AI engineer at
25:27
Process and the last year, 1 1/2 years, my main focus was on
25:34
agents, you know, building agents, testing agents,
25:39
verifying their use cases and building them out in real
25:43
products. So it was mostly like these
25:47
cycles of like, we have an idea, you know, we want to build this
25:51
POC or MVP and we want to know if it works.
25:55
And in some cases, you know, I stuck around for a bit longer
25:59
and I actually, you know, I did some work to get it in
26:01
production. But it's yeah, it's a lot of
26:05
experimenting. And I think you mentioned
26:07
already early in the post cost the applied R and DI think is a
26:12
good way of positioning. So, yeah, so very perfect to
26:16
have this position. I'm always curious what, because
26:19
we used to have a lot of talk around what an ML engineer was
26:24
and is that somebody who's modelling, is it a data
26:26
scientist who is specifically working on ML?
26:29
And now there's the new term of AI engineer.
26:32
So what is that like, what is the day in, day out?
26:35
Are you building evals? You're working with agents,
26:37
you're creating agents and. Yeah.
26:40
So I would say like an engineer is there to solve problems.
26:45
Like an engineer will say like, OK, this is what we need to
26:48
have, you know, build it. I don't care how we reach it.
26:51
Not with AI Nowadays it's almost all software.
26:55
So you're you're part software developer, but you're also part
26:59
thinking. Like, you know, how can we
27:01
position this product? How can, how can users interact
27:04
with it? You know, it's a, it's a bit
27:05
more, you make a bit more decisions than a normal software
27:10
engineer because normal software engineers are like working on a
27:13
task to task basis. But we're our AI engineer are
27:19
more like I'm solving this task using AI and how we would how we
27:24
would fill that in. That's most, most of the time
27:28
it's a blank piece of paper or a Myro board, but and we should
27:33
just start, start building. So I think that is kind of the
27:36
the nowadays AI engineer. Over the last two years you've
27:40
been trying to play with agents. What are you looking at?
27:42
Yeah. Well, it it will be a larger
27:45
number than I think many will expect.
27:48
And I think using the word playing with agent is quite
27:50
right. You know, we've been exploring.
27:53
You know, it doesn't always needed to be a good idea.
27:55
We're just trying to work the muscle here.
27:59
But yeah, I think over 20 projects that were related to
28:04
building an agent that was solving a specific use case that
28:08
at some point we thought it was a really good idea.
28:12
And we'll get back to that because I want to ask a lot of
28:15
questions around why did you ever think this was a good idea?
28:18
But the other thing that is worth noting is how many now are
28:23
actually still being used or are real projects.
28:28
I guess they made it past that filter.
28:31
Yeah. So it's so there are two that
28:36
actually made it. But there there is like a
28:39
caveat, there is like a few of them were merged into one.
28:44
So because these were exploratory projects, we saw a
28:49
value in it. Yeah.
28:51
But on a standalone feature, it was like, OK, it's not adding
28:55
any value. But if we if we bundle this all
28:59
or we add it to it like our our tokon then it adds value again.
29:03
So, Tokon, for those who don't know, what exactly is it?
29:06
Yeah. So took on as our general
29:08
assistant, the idea started kind of like having this extra Co
29:13
worker. So it started those on Slack and
29:16
now it's also on the web. It's been evolving a lot, still
29:19
is evolving, but it started this.
29:22
You just send a Slack message to this agent and it will do part
29:27
of your work. And of course, it started with
29:30
just the simple summarizations and, and, and now we're building
29:34
it out into more complex systems where it can do full analysis
29:38
and, you know, you can save stuff.
29:44
And yeah, kind of build this this project on top of Tokan
29:48
having this interaction of back and forth like you would have
29:51
with a real colleague. And the other one that still
29:54
exists to this day is the Sequel Analyst.
29:57
Yeah, the the token analyst, yeah, it's mostly used for, it's
30:02
used for sequel. And yeah, that 1 is really
30:06
successful because we really saw it was was adding yeah, value
30:11
and we were saving people's time and money, yeah.
30:15
Nice. And we're going to do a whole
30:16
episode on that, like a deep dive case study.
30:19
Now what I want to talk about here, like you've seen over 20
30:24
use cases. What are some green flags and
30:26
red flags of an agent that is going to work versus fall flat
30:31
on his face? Yeah.
30:33
So to come back to my earlier comment of like, you know, we
30:36
bundled a few I think when we were really trying like, OK,
30:40
let's try many ideas. One of our experiments was what
30:45
if we did an agent that could do, you know, less, you know, so
30:50
more specific. So we call this fertilized
30:52
agents. Will the accuracy be much
30:55
better, consistency be much better so people trust it more
31:00
and use it more. So we the, the one, the test we,
31:02
we, we, we ran was we had an analyst agent which was making
31:07
plots Python. It was reading Excel sheets,
31:11
doing statistical analysis, anything.
31:14
But we saw that sometimes with cleaning data, I'd like make
31:17
mistakes. So they're like, OK, let's make
31:19
a cleaning agents separately. So you first go to the cleaning
31:24
agents and it will clean and then you can come back to the
31:28
analyst and it will do the analysis.
31:30
So you have the separation. But actually people were not
31:35
using the cleaning agent more because they sound like, yeah,
31:37
but you know, I'd rather have it.
31:39
That 80% of the time we'll just finish the task in one go is so
31:43
much more easier than me having to switch.
31:46
An extra step. Yeah, so the extra step was not
31:48
worth it. That makes sense, no?
31:51
What are other red flags? Yeah, I think every agent that
31:56
was really hard to test saying like, you know, it's it's right
32:01
or wrong because then, you know, I had a colleague that always
32:06
was saying like, you know, we're measuring vibes.
32:08
I think that was a really good measurement of things we you
32:11
should avoid when building agents if.
32:13
It's not binary if it's not like the code runs.
32:16
And I've heard that a lot with coding agents and assistants
32:20
that one of the reasons that they are such a strong use case
32:24
is that it's like the code runs or it doesn't, it compiles or it
32:28
doesn't. And you know if the AI generated
32:31
code or the agent that assisted you worked or it didn't.
32:36
Yeah, exactly. And you see it also in, in, in
32:38
O1 now, you know, it's, it's, it's really funny that you, you
32:41
see that same things that we see in agents that you see that in
32:44
O1 and O3, because open AI itself, you know, said like,
32:49
hey, guys, if you want to do creative stuff, still just use
32:52
four O because four O is still preferred by humans as being
32:57
better at creative writing. And that is exactly due to the
33:01
same issue where O1 is being trained on, you know, being
33:06
right or wrong. And the moment there's no right
33:08
or wrong, you know, it's kind of improve itself.
33:11
So O1 is, is, is amazing at all these like analytical and more,
33:16
yeah, bad tasks. But the moment you're getting
33:19
into the creative stuff, it's anything subjective.
33:22
Yeah. And that's something we saw in
33:24
the in the in the agents as well.
33:26
It's like, you know, when you're more in like this creative side,
33:30
it's like, you know, how do you know it's right?
33:33
Yeah, yeah. So, so making sure that whatever
33:36
the task is, there's a clear way to evaluate if the tax was
33:41
executed or it wasn't, that's another green flag.
33:45
I would say red flags for any other that you have that come to
33:51
mind. Yeah, so it's actually quite
33:56
funny because now it seems that that the tables are turning.
33:59
But like a year ago we had these web search agents and that is
34:05
one of those agents that also got like, you know, it, it
34:08
merged with with Tokon. But at the beginning, we, it was
34:11
a separate agent. And I can still remember the
34:15
feedback of like, yeah, this can never work because, you know,
34:18
the latency is way too big. You know, it was doing deep
34:22
research. And maybe you recognize his name
34:26
sounds familiar. It's, it's something that, that,
34:28
that Gemini or Google is now doing and they, they released
34:31
like this deep research and now people are fine with like, you
34:35
know, I have 7 minutes if I get a cool report with a lot of
34:39
sources that works. But you know, we, we were doing
34:43
something similar like a year ago, but people are saying like
34:46
it takes too long. You know, I don't know where in
34:48
zone sometimes took like 10 minutes.
34:51
And yeah, I think the also the biggest change what happened is,
34:56
you know, we were in Slack, so we couldn't provide this multi
35:00
page documentary. In theory we could, but we at
35:03
that time there was not the way how we were thinking.
35:05
We're just want to have like a concise message on Slack.
35:09
So that's where we're saying like, yeah, for this, just this
35:11
message, it's not worth the waiting time.
35:14
Yeah. So yeah, we didn't kill the
35:16
project, but we didn't kept it as an agent.
35:20
You know, we just distilled it a bit and moved it to the more
35:24
general one. That's funny because I find that
35:27
my own workflow. I tend to ask AIA question and
35:32
then go do something else. And so I'm in the camp of I'm
35:36
totally cool with just waiting, seeing what happens, coming back
35:40
to it, when I get to it. Sometimes I forget and then come
35:43
back a day later and it's like, Oh yeah.
35:45
But we were in an era, you know, where, where jet GDP was the
35:48
norm, you know, and that responded immediately.
35:51
You know, you had the streaming. Yeah, the streaming it was.
35:54
So within 300 milliseconds, you know, the first word started and
35:57
you started reading. So if you then introduce a
36:00
system that you need to wait 5 minutes for, people are like,
36:02
no, that's we can't too much. So it's also like the, the, the
36:06
public adapting to this view of these agents doing stuff.
36:10
So the more people know that there is work being done, the
36:14
more they appreciate that waiting time.
36:17
And they actually are like, Oh yeah, but it's normal.
36:20
And like you're saying you're doing this asynchronous work.
36:23
You know, I even have some like 3 tabs open and ask you 3
36:26
questions at the same time and you're really multiplying.
36:30
Multitasking. Yeah, yeah, yeah, yeah.
36:32
Multitasking has a new definition now.
36:34
It's yeah. It's quite funny.
36:36
What are the ones that died? So we had this hackathon, you
36:41
know, since, you know, we're really like, you know, there are
36:44
no bad ideas, just just develop. So the idea was like, let's get
36:48
the whole AI team, you know, 24 hours or yeah, it was a bit less
36:53
and make Asians and one of these ideas where I thought like,
36:59
yeah, this this is going to be the agents from the hackathon.
37:02
Because it was kind of the idea like hackathon home run the
37:05
best. Yeah, exactly.
37:06
I would have I would have invested, but it was it was the
37:10
JIRA agent. Yeah.
37:11
So it was of course was doing the JIRA taking.
37:14
Nobody likes JIRA. That's so true.
37:16
And the thing was, you know, we saw it working.
37:19
So in this in this in this test setup, you know, where they
37:22
built the JIRA board with the agents and I started adding
37:25
tasks and changing tasks and asking summaries about tasks.
37:29
It was working really nice. You know, again in Slack, it was
37:33
super useful And but the fun thing was the moment we
37:38
connected it to our gyro saying like, OK, we're going to be the
37:41
first beta testers. Yeah, you know, it completely
37:44
broke down. You know what already you know
37:46
the reason why why what happened?
37:48
It was, it was the, the human text, you know, all like these
37:54
acronyms and all these like really short sentences, like
37:58
minimal information that was needed for a human to
38:01
understand. It was, it was messing up the
38:05
agent. It was saying like, you know,
38:07
this is no description. You know, like, like half of it
38:09
didn't even have a description of the task.
38:12
But all the humans in the team are like, yeah, of course we're
38:14
building this project. So this must be that, you know,
38:17
but all that context that was not in that GYRA agent.
38:20
So that's why it completely did not work.
38:23
And that's why we were like, OK, you know, let's not continue
38:27
with this because we need to. Change too much.
38:29
If today you would take a different stab at it and you add
38:33
in some kind of a knowledge graph with Slack messages and
38:36
with emails or with other context, do you think it would
38:41
have been more successful? Yeah, you, I think today I
38:45
would, I would make like an interview agent as well and
38:49
first interview the team saying like, you know, provide me all
38:53
the current information, then convert that into documents and
38:56
then you know substitute to JARA agent with that.
38:59
Like you know, if things are missing, you know, look at this
39:02
interview I had with your colleague, maybe that clarifies,
39:05
you know, that is that is more more stable approach.
39:08
But another approach would be, and that is I think something
39:11
that we'll be seeing much more of is if you go AI first,
39:16
because what is the, what is the the reason why that port was
39:19
messy was because we needed to type every word ourselves.
39:23
And then you're, you're doing like basically the SMS language,
39:28
you know, you're trying to do as many as the least amount of
39:32
keystrokes, but like. T9 back in the day where people
39:36
who are texting on flip phones. Yeah, like instead of oh sorry,
39:40
write this out or just add an emoji.
39:43
No, but it was. So if you go AI first, you know,
39:47
and you said like I built this board with AI and then I
39:50
maintain it with AI, you know, then there's this chance.
39:54
That's what that's what what that was the reason why the test
39:59
was the test was really successful because the test was
40:03
built with AI and then questions with AI and then it understood
40:08
its own language. So there's also a route that you
40:11
can say like, OK, you just need to force people to remake the,
40:15
the the entire board. But it would be a board, it
40:19
would be a whole separate tool. It wouldn't be using JIRA.
40:23
Or design decision, you know it's it's can still use Gyra,
40:27
but then just a fresh board or or you know you can make your
40:31
own UI I. Think that's a maybe something
40:35
that we've we've learned over and over again as it, as you
40:40
bring these systems into existing workflows or you know,
40:46
ways of working in particular when we go into the e-commerce
40:49
world like people have expected expectations on patterns of use
40:54
and that's of course it's not surprising, but it's so
40:59
important to get that right. And, you know, in our world, you
41:03
know, we, we give Tokan access to, you know, our, our GitHub
41:06
and it would sort of comment on code and you know, comment and
41:10
so on. And we switched that thing off
41:12
in no time because it was so noisy.
41:13
And then we tried other products like code Rabbit and it was very
41:16
similar because at the end of the day, it's very easy.
41:19
It's cheap to generate content and comments, but there's still
41:22
cognitive load to go through it. And that's you want to spend
41:25
that on, you know, high value information.
41:28
And so these things there's not we try, you know, one of our
41:31
missions is to become sort of the, the best AI first team,
41:34
right? So we have AI assistants
41:36
everywhere. We've got our own AI
41:38
statistician, we got all these little AI layers.
41:42
So we test everything, but very frequently on some of these
41:44
workflows, we, we let go of the tool because it doesn't make
41:48
sense yet, right? It doesn't work.
41:50
And I think and, and, and part of it's our expectations and how
41:53
we interact with this with, you know, with each other and the
41:57
tools, but also the tools like Jira in, in Flores's example, it
42:01
wasn't, you know, made for, you know, interacting with these
42:04
agents as it is today. Maybe it will in the future, but
42:08
it's not there yet. Linking that back to what you
42:10
were talking about earlier on the cost per intelligent unit or
42:14
what did you call it? The cost per unit of
42:17
intelligence. Yeah.
42:19
And you think about how that is not a unit of intelligence.
42:23
It's outputting something, but it's actually a unit of
42:27
distraction. Yeah, in this case, it's a it.
42:30
Well, it's costing our us cognitive load and you want to
42:33
do the opposite, right? And.
42:36
Every right question saves time, but every wrong question
42:40
expensive. Yeah.
42:41
Yeah. I thought one thing to add here
42:43
is that this come back to this theme of cognitive load that we
42:46
add sometimes, you know, without thinking about the current state
42:49
as we've got a big platform called OLX classifieds platform
42:53
millions of listings being uploaded every day and a natural
42:57
place for us good example. The kind of work that we do is
42:59
we try to see how agentic systems can help, you know
43:04
people transact goods and I think it's one of the strange
43:10
consequences of ChatGPT is that we've tried everybody's tried to
43:14
basically make a ChatGPT for X yeah for everything right.
43:18
So we also naturally, when we started that journey said, Hey,
43:21
we need a ChatGPT for OLX, right, for people that try and,
43:25
and buy and sell stuff on unclassifieds.
43:28
And we, we realize that, you know, people today, I mean,
43:31
hindsight is obvious, but they, you know, they go to the
43:33
website, they see ton of images already, they have a search bar
43:37
and that's how they discover. And then we said, you know what,
43:40
we're going to, you know, introduce a conversational
43:42
agent. But what's the cognitive load,
43:45
you know, placing on this user, right?
43:47
This user needs to come in and say I'm looking for a piece of
43:51
furniture for my home, which is such and such style and it needs
43:55
to be under and people just wouldn't use it.
43:58
And so there was a huge, let's say drop off because of that
44:04
additional friction. We'd introduce the cognitive
44:06
load for people to input things. And even when they did use it,
44:10
they put in blue couch. So it's the same as search.
44:14
So it becomes it's just a search bar and then the agent would
44:18
come with, you know, 500 tokens. Of, of of.
44:22
Questions and content and then the and then the user would say.
44:27
Cheaper. There is another thread that I
44:31
wanted to pull on where you're talking about different layers
44:35
of if an agent project makes it into production, and one is you
44:43
design it a certain way and you take as the creator of this
44:48
agent certain design decisions. And then the other is later it's
44:52
out in production. And maybe it's increasing the
44:56
cognitive load, but it could be that it's increasing the
44:59
cognitive load on the user because they don't know how to
45:02
properly use it, or it could be because it's a shitty project.
45:07
Yeah. So you have to decide later
45:10
which one of them is it? Do we need to educate the users
45:13
more or do we just need to kill the project or take a different
45:15
design decision? Yeah, I think that blue couch
45:17
example of Paul is amazing. Like, you know, 100% all the
45:20
developers working on the project were nicely filling in
45:25
the full prompts, you know, like typing, like, hey, I want this
45:28
couch that looks like this. And the moment they indeed gave
45:32
it to real users, they were like blue couch.
45:35
So which? Seems so clear in hindsight.
45:39
Of course I don't want to have to fill out a form.
45:43
I don't want to have to put more than I need to to get what I
45:46
want I want. If I can do it in one click,
45:50
that's better than typing out. Words.
45:52
Now there's another side to it. Right now, I think to the kind
45:54
of things that we learned as we build these agents is that the
45:58
the systems have a much better ability to understand complex
46:04
queries. So as soon as you put in
46:07
something like modern couch, most search engines today fail.
46:11
But it actually like, you know, Jenny, I based system can
46:15
actually understand what modern is and what that made look or
46:17
feel like. And so we've sort of leveraged
46:20
that and said, OK, well, actually we need to represent
46:23
our catalogs in ways these agents can link more complex
46:27
queries. Whether you know, a modern couch
46:30
is, is an example in the classified space, in the food
46:33
space, you say something light and healthy.
46:37
Like no search engine today in any food ordering platform knows
46:41
how to handle that, right? But we actually can, like these
46:44
LMS can suggest something light and healthy.
46:45
Give me 5 suggestions or what the look and feel like.
46:48
But then you need to match that to the underlying catalogue.
46:52
So then you need to have a system that does this sort of
46:54
what we've called magic or smart search to retrieve that.
46:59
And then you have another layer, which is like, well, if we can
47:02
actually understand that and we want to overcome that friction,
47:06
there are places in the world where we work, like Brazil and
47:08
India, people work with voice. And so if people can actually
47:13
say through voice, hey, I'm looking for a quick meal tonight
47:17
in my house for two people, that's fairly frictionless,
47:21
right? If you can send it through a
47:22
voice message and actually an agent can decipher that and say,
47:25
oh, house is there, this is where they live.
47:29
This is what the, I mean with these are meals that would
47:32
satisfy two people. And so you can actually take a
47:34
much more unstructured different modality of input from a user,
47:38
give that to an agent, they can process that and translate that
47:41
to a set of items that then can be presented to the user.
47:45
So they are other opportunities that open themselves up because
47:48
you've got stronger reasoning capabilities, multi modality and
47:51
so on. Yeah, yeah, I do like that, how
47:53
you don't have to think in the traditional way.
47:56
And that's what's becoming clear is that if you're trying to fit
48:00
the agent into old workflows, it almost feels like it's a square
48:05
peg in a round hole type situation.
48:07
But when you start thinking out-of-the-box and you think,
48:10
OK, well, since the agent can do just about anything that we
48:14
throw at it, what can we try to make that as a new workflow that
48:19
doesn't, that the user isn't already trained on how they're
48:23
used to using the app or how they're used to interacting with
48:26
this or that. And so you're going to get
48:30
inevitable dead ends on that path, which I think you saw
48:34
where the glorified search bar. But then like, yeah, the voice
48:39
note sounds incredible. If I could just send a voice
48:41
note to an agent that would give me suggestions all the time.
48:44
That is a really awesome use case.
48:48
What are some other use cases that died though?
48:50
This is where I want. To get back to the graveyard.
48:53
Yeah, I want to hear more than Halloween episodes.
48:57
What? Because it's almost like that's
48:59
where the best learnings are, right?
49:00
You you always see people writing blog posts and
49:03
especially companies of your size, they're writing blog posts
49:07
about the successes, but you don't really hear companies
49:11
talking about the graveyards and what they had to do to get to
49:15
that success. I think I've I've won one more
49:19
that is quite interesting because normally we were always
49:23
like so heavily focused of like and it's skill.
49:25
But here is, is we had one example where we kind of forgot
49:29
that part and it also ended up the graveyard, but it was, we
49:33
called it the, the UX researcher.
49:36
So we had real people coming to us with a, with a issue.
49:41
They're saying like, Hey, I have like all these open form
49:44
questions that we get like as a, as a review or, or comments on
49:49
our products, but there are like too many for me to process.
49:54
Can we build a system? Can we build an agent that goes
49:57
to those comments and kind of summarize them like, hey, name
50:02
me the the top three features people dislike about our site.
50:06
You know, these are questions that that we foresee were like,
50:10
OK, this is indeed something agents can solve.
50:14
And when we started with this, we built this whole tool that
50:19
was analyzing that was going row after row doing like MapReduce.
50:25
So it was first checking you know what are the subcategories,
50:29
then dividing into subcategories and then for each subcategory
50:32
you know, finding given the users objective like what is
50:37
then the answer. So it was combining all these
50:41
techniques. It was super fancy.
50:43
Wait, can I stop you right there real fast?
50:45
Because if it was, was it a clearly defined workflow each
50:50
time, or was it that you asked the agent and the agent would
50:54
figure out the workflow on its own?
50:58
So it's it could manipulate which workflow, but it was quite
51:04
repetitive. OK, so it almost choose its own
51:07
workflow. Yeah, and that was where the
51:08
agentic. Part came in and you know, we
51:11
tested it with, you know, excels of like 100 rows, 1000 rows and
51:16
we're, we're testing this. I was working fine and then we
51:19
went back and they were asking stuff for excels of 100,000
51:27
rows. It could not do that.
51:31
You know, it was, it was like we, we, we knew it was going to
51:35
be large, but you know, we thought like 100 or 1000, you
51:39
know, because it, it could do like 10 with without any any
51:43
sophisticated. So we already like multiplied by
51:45
100 and but it, it, it, it was devastating.
51:49
And, and the worst part was also that we designed this in a row
51:52
by row and they had their answers in like a fertile eyes.
51:57
They they just transpose their, their whole table.
52:00
And it also broke the entire thing.
52:02
And it was it was just a mess because we thought we were so
52:05
enthusiastic. We're like, this is a great use
52:07
case. And we saw all these ways how we
52:09
would solve it, but we completely forgot how how how
52:13
they would solve it and and asked.
52:15
We didn't ask enough the. Basic question, yeah.
52:17
How big of a file are we talking?
52:19
Yeah, yeah, yeah. So.
52:20
But there wasn't a period, you know, where we were like, you
52:22
know, there are no stupid ideas, you know, we just need to make
52:25
agents, agents, agents to see what sticks.
52:29
So. You did mention one thing about
52:30
why you have that mentality, which I thought was pretty cool.
52:34
And, and you all look at it like it's going to the gym and you're
52:37
building the muscle of creating agents and you're trying to
52:41
figure out how you can create these new work flows, these new
52:46
products that are agentic first. Yeah, because if, if, if at some
52:52
point, if you made enough agents, you know, it's like
52:54
learning physics. You know, if you learn something
52:57
new in physics, you walk around the real world and you see that
53:01
that formula taking form in real life.
53:04
And the same happens with agents.
53:07
You know, if you build a few, you'll be instead of if you
53:10
enter a site, you know, instead of seeing the UI, you know, you
53:13
start to see tools. Yeah.
53:15
And you're like, hey, this, this can be a tool and this can be a
53:18
tool. And then I have a chat window
53:20
and then I can just remove this entire UI, You know, that's,
53:23
that's, that's how you start. Till a user asks for a blue
53:25
couch. Yeah.
53:28
But it's, it's like this, this whole new way of thinking and
53:31
looking at things. It's something that you need to
53:33
practice because the first time I saw the agent, you know, I, I
53:37
remember I was, I was, I, I worked a week at process and I
53:41
was set in this room and the only thing we knew is like,
53:45
we're going to test a new agent. You know, Ahmed built this one.
53:48
It was the, the analyst. It was doing like all these
53:51
pattern analysis and they just gave like the chat window, like
53:56
good luck, you know, go test it. We need to load test it, you
53:59
know, does it scale, blah, blah, blah.
54:01
And I was amazing, you know, it was, it was like I, I didn't
54:04
know, I didn't know how it was doing it, but it was doing it.
54:09
And, but it was also like, you know, where are the limits?
54:12
You know, it was really, really hard to, to find those, you
54:15
know, in the beginning because you didn't, you didn't know what
54:17
the system was. And the more you were, and at
54:20
that time, you know, it, you just did some things.
54:23
But you really saw that after three months of working with it
54:27
and developing it, you know where you were better, much
54:30
better at testing it, you know, finding those X cases because
54:32
you know, OK, this is how it works.
54:34
So this is how I can annoy it or this is how I can make sure it
54:38
works. And there was there was quite
54:41
interesting to to see this, this muscle.
54:44
Grow. Let me add to that why because
54:46
yes, why do we do that right and it's exactly like first we need
54:50
to understand what makes these things work and and why is it
54:54
important for us. This process is because we
54:57
fundamentally believe that these agents are going to be able to
55:00
help us build better products for our users.
55:03
And we've made predictions around this, right.
55:05
You were at the marketplace. So one of the predictions we
55:07
made was in a year time, 10% of the actions done on our
55:15
platforms will be done by agents on behalf of our users.
55:20
That's a pretty bold prediction. And, and whether it's in 12
55:23
months time or 36, it will happen.
55:25
We're fairly confident about that because these systems, you
55:28
know, will, why won't you send out your agent to help you, you
55:32
know, get whatever you need, whether that's food or other
55:34
things, if it can do that reliably for you.
55:38
But we can only build those things if we fully command the
55:41
technology and have a very good intuition.
55:43
And you can see how floors, you know, by basically trying a ton
55:47
of things with, you know, with the rest of the team has
55:49
developed that intuition. Yeah, right.
55:51
We said that that we're not ready for that.
55:53
But this sure that tool we can build and it'll get us to 80%
55:57
accuracy. We measure these things, we test
55:59
the tools and so on. So that's the larger picture of
56:02
why. It's also because it's awesome,
56:04
you know, it's I just, I just want to.
56:06
Yeah, it says that the wow effect, you know, I think that
56:10
agents won't give me that wow effect.
56:12
I think they will will still be a while before that will be
56:17
removed. And so maybe you can give us
56:19
some just tactical things when you're putting agents into
56:22
production and you want to make sure that you've covered and
56:28
you've checked all of the boxes. What are some things that you've
56:33
learned or you've done that have helped you to make that jump?
56:39
On the B to C side, I think there are people who know much
56:41
more than I do, but we did work with a few agents that were
56:48
around data so that the data analyst and there we really saw
56:55
that that improved in prompting and security is when we would
57:00
repeat, you know, after we gave the answer is like, we gave this
57:04
answer under these assumptions. And so because we, we really
57:11
tried to, to to ask as many questions to make sure it was
57:14
not ambiguous question, but that's just hard.
57:17
And there were still, you know, a few questions that came
57:20
through that like that defense mechanism.
57:23
But at the end we're like, OK, then let's just recap and saying
57:28
like, OK, you gave this question, you know, I did this.
57:31
So that means that I made these assumptions and listing those at
57:35
the bottom is also a way, a security way of saying like, you
57:41
know, maybe I made a mistake because you don't you want to
57:45
minimize mistakes, especially when you're doing data analysis
57:48
because we want to position these tools like, you know, you
57:51
want to make decision based on this, you know, we want to make,
57:54
we want to make everyone be able to make decisions on data.
57:59
Now the better those decisions are good.
58:01
So one of these mechanisms is like the assumptions and I think
58:04
we haven't seen that in other tools.
58:07
So you're just asking as a final step in the prompt, like, tell
58:11
us what you did, tell us what the prompt was.
58:14
Yeah, it's a, it's a separate mechanism.
58:15
So it's not the agent itself. You know, we really want to kind
58:18
of let the agent do it's thing, but there's like a second Adalam
58:23
call or agent call that basically refused the steps and
58:26
saying like OK user started with this question.
58:30
But I've seen you also added this filter in the SQL query.
58:35
I see you changed the date format, you know, maybe that
58:38
changes things, you know, so. It's like a proofreader
58:41
basically, right? Like a layer of checking before
58:44
and validation before it gets sent back to the user.
58:47
Yeah, it's critiquing everything.
58:48
But it's not saying like it's wrong or right, you know, but it
58:51
ends to the user like it's, is it right that I made these
58:54
assumptions? Because mostly those assumptions
58:56
are made because there was no other way of calculating it or
59:00
it was because it's some rule and some document that we added
59:03
to the agent, you know, OK. Other thing that I want to
59:07
finish on is your view of the evolution of prompting.
59:11
It's it's come a long way, you know, so like the first time I
59:16
was using the large language models in some kind of like
59:19
assistant way, it was like the Neo X.
59:22
It was, it was an open source model, 20 billion parameters.
59:26
And I remember, you know, prompting it like as if I was
59:30
writing a paper and then stopping at some point and then
59:34
it would like finish some complex question because it
59:37
would do like it would write the paper that would answer that
59:41
question, which was insanely trickery.
59:45
You know, it was it was it was basically we're tricking that a
59:47
lamps, you know, and then three-point the Da Vinci game.
59:51
And still, you know, we needed these tricks.
59:53
We needed examples. We needed to kind of massage,
59:57
massage it into this pattern and.
1:00:02
Then the era came of instruct models, you know, and that's the
1:00:05
beginning of Chet GP where you could just ask a question and it
1:00:09
would understand that that is an instruction.
1:00:12
But this development kept continuing.
1:00:15
So people thought like, OK, the moment you can ask a question,
1:00:18
it works. But we've seen it in, in these
1:00:20
system prompts, as you call them, you know that in the
1:00:23
beginning we needed to tell them every single thing.
1:00:26
Like this is how Python works, you know, this is how you use,
1:00:29
this is how you be friendly. This is how you use emoticons on
1:00:33
your message. You know, it's like we were
1:00:35
writing like the tiniest bits of Corrections that we want to see
1:00:38
consistently we needed to write down.
1:00:40
So you had system prompts of like 3000 tokens and, and maybe
1:00:46
even more for, for some agents. But over time, you know, we, we
1:00:51
had, we had struggle converting these, these prompts from model
1:00:55
over model. But what we actually saw is like
1:00:57
if you just removed everything, you know, and started again,
1:01:01
like, you know, with empty prompts and then adding the
1:01:04
parts that indeed were failing, you ended up with a shorter
1:01:08
list. So what actually was happening,
1:01:11
you know, open AI was training these models better and better
1:01:13
in doing many of these things, people were forcing it into to
1:01:18
be part of the native behaviour of agents or of of models.
1:01:23
And that is, that is, that is a trend that I really see.
1:01:25
Like if I now build an agent, you know, I literally start with
1:01:30
three lines of system prompt. Wow, there was there was
1:01:32
unimaginable in the time of 3.5, you know it was not possible.
1:01:37
I think it's a that's a really great outline of prompting,
1:01:40
right, going from these base models that were just dumb token
1:01:42
next token predictors, the auto the core auto aggressive
1:01:46
function to like actually instruction fine tuning.
1:01:49
Well, actually you at first at the few shot, yeah, then you had
1:01:52
an instruction fine tune, then you had alignment.
1:01:54
Now you have, you know, now you have the O ones, which basically
1:01:58
they do like, like, you know, chain of thought suggesting
1:02:00
before they actually start executing.
1:02:02
But because we're talking about agents, we also see that if you
1:02:06
look at the prompt that we use in our genetic systems, they're
1:02:11
essentially, you know, it's like it's a piece of code where you
1:02:14
start inserting all sorts of parameters, right?
1:02:18
So it's basically like dynamic prompt building or or you know,
1:02:21
composite prompt building where you've got placeholders for all
1:02:25
sorts of things that come from the system.
1:02:29
And it can be, you know, information about the session or
1:02:33
the context or the user or whatever.
1:02:36
But also, of course, you've got the tools and the function
1:02:39
calling that you need to describe where, you know, the
1:02:42
way you describe it floors. I think it's absolutely right.
1:02:44
Like now we're kind of, you know, you can't put in 2000, you
1:02:48
know, function functions and describe them.
1:02:51
It doesn't work yet You can do a couple.
1:02:53
We know kind of where the sweet spot is depending on the model.
1:02:55
Where have you found? Is it like 10?
1:02:57
It depends on the model, but no, it's.
1:02:58
More you can do. It depends how complex the
1:03:00
functions are and so on. And if you need to change the
1:03:03
functions if they look alike, you know if they're super far
1:03:06
apart, you can add as many as you want.
1:03:08
It's when they look alike, right?
1:03:09
And then it gets confused and it's like, yeah, that's the
1:03:11
same, isn't it? Yeah.
1:03:12
So we build evals on one can, can it actually pick the right
1:03:16
function at the right moment? But then the next step is you've
1:03:20
picked the right function. Can it actually provide the
1:03:23
right parameters for that function to be executed?
1:03:26
Typically, if you do code execution, you need parameters.
1:03:28
Or if you go to the web, then you need parameters, right?
1:03:30
Search queries and so on. And that's a second evaluation,
1:03:34
right? Can you actually ensure that
1:03:37
when you've identified the right function, you pass the right
1:03:39
information to come back? Now all of that stuff comes out
1:03:42
of that prompt, right? So in fact, your question of we
1:03:46
know how this prompting changed is super relevant for folks
1:03:50
building agents. Because the way you think about
1:03:52
a prompt and the, the orchestration around it, what
1:03:56
information you pull in, what information you get back.
1:03:59
If you do sequential chaining of, of, of, of tools in the, in
1:04:03
the agentic workflow, all that stuff needs to somehow, it's
1:04:08
very stateful, right? Needs to be stored somewhere.
1:04:11
It needs to be managed. So anyway, we've, we've ended up
1:04:14
in, you know, all sorts of worms, cans of worms.
1:04:17
Because if we try to make these things work, as you change the
1:04:21
model, add a tool. Breaks everything well.
1:04:25
Yeah, you need to make sure you understand what breaks.
1:04:27
Yeah, when when it breaks, you learn something.
1:04:29
Yeah. I think that is that is really
1:04:30
important. Yeah.
1:04:32
Yeah. I think that's almost the first
1:04:33
question I asked people that made some agent or an agentic
1:04:37
system is like, what can't it do?
1:04:40
You know, because that's really, really important.
1:04:43
You know, something that we saw on a, on a project that was, you
1:04:46
know, we, we constantly knew what it couldn't do.
1:04:50
So we knew that it was our next target.
1:04:53
And then, you know, we could we're able to do that.
1:04:55
And then we're, you know, spend a few minutes and like, OK,
1:04:58
where does it now break? And then like, OK, that's our
1:05:01
next target. And then you move on, moving
1:05:04
target to target until you're like, OK, these tasks are edge
1:05:09
cases, but we still know it doesn't work there.
1:05:12
But that's off limit. Well, that goes back to that
1:05:15
binary execution, right? Because you know, did it
1:05:19
complete the task or not? Yeah, exactly like binary.
1:05:22
You know a lot of people saying like right or wrong, but indeed
1:05:25
a task completion. You know, if I, if I have 8
1:05:28
steps to finish a task, you know, I, I don't, I don't really
1:05:32
care how it achieves the task. You know, I just wanted to
1:05:36
achieve the task at the sort of consistency.
1:05:39
So that is also a binary thing. It doesn't have to be like.
1:05:45
There's that, but then there's also the almost higher view of.
1:05:50
Are these the right tasks? If you ask an agent to do
1:05:53
something, it may complete all the tasks no problem.
1:05:56
But. I think I think it's a good one
1:06:00
where you're also, you know, if you take time back into it, like
1:06:03
if you waste more time trying to get that task to work, but it is
1:06:09
then automated, you know, that's also kind of like, is it then is
1:06:13
it then worth it? I think that the the cognitive
1:06:15
load on checking it and and and making sure it works, that needs
1:06:19
to be in check in the value it delivers.
1:06:22
And I think what you mentioned early in the podcast, you know,
1:06:25
the computer use, the web use, you know, I think there we are
1:06:28
really in a stage where there will be a period where we're
1:06:32
saying like, OK, it can do it, but you know, maybe I will just
1:06:36
make that pivot table myself because typing it will probably
1:06:40
take longer. Yeah, than I want to use my
1:06:42
computer. Yeah, yeah.
1:06:44
And I don't want to sit behind it.
1:06:46
Yeah, that's also well, like, does it save you time if you're
1:06:48
not able to operate the computer at the same time?
1:06:51
Or you need a second computer just for your.
1:06:53
I mean, for us, we generally think about make it work first,
1:06:58
then make it fast because users don't like to wait and then make
1:07:01
it cheap and we're typically always pushing the frontier
1:07:04
doesn't work. And so, you know, it's, it's
1:07:08
perfectly fine to pin up the spin up 10, you know, basically
1:07:12
agents that will try and solve your task and whichever 1 gets
1:07:14
to it first, because you know, that's more of a having a right
1:07:18
answer is more valuable than having, you know, 9 or 10 of
1:07:21
these things doing this in parallel with the cost of that.
1:07:24
So we're always trying to push for the boundaries of can we
1:07:27
make it work? It's also interesting for
1:07:29
process itself to know which tasks are solvable by AI because
1:07:33
then we know there's a time factor that within X amount of
1:07:38
months or years this will be viable from a from a cost
1:07:42
perspective. So we just need to know it could
1:07:45
be solved, but if it's the right time isn't another question.
1:07:54
A huge shout out to the process team for their transparency
1:07:57
because it is rare that you get companies talking about their
1:08:02
failures, especially companies that are this big in the AI
1:08:06
sector and really helping the rest of us learn what they had
1:08:10
to go through so painfully. Sometimes a mention that they
1:08:16
are hiring. So if you want to do cool stuff
1:08:18
with the team that we just talked to and even more, hit
1:08:23
them up. We'll leave a link in the show
1:08:24
notes. And if you're a founder looking
1:08:26
for a great design partner on your journey, then I highly
1:08:30
encourage you to get in touch. We'll leave all the links for
1:08:33
all that good stuff in the show notes.