Latest / Agentic Conversations (formally mlops.community) / Getting to Grips with Web Agents

Transcript

Getting to Grips with Web Agents · Agentic Conversations (formally mlops.community)

0:00
Today we're talking web agents. This episode is going to be all
0:05
about how the Process team has been leveraging web agents and
0:10
why you would even want to go down the path of trying to get
0:15
web agents to work. Because that was my biggest
0:18
question. Isn't it just adding complexity
0:20
and you can use these API agents?
0:23
Let's get into this episode. This is the ML OPS Community and
0:26
Process collab. We are doing this limited series
0:29
all on agents in production. All right, man, Web agents, here
0:38
we go. We're getting into what they
0:40
are, what they aren't. And you told me something
0:44
yesterday or two days ago about how computer use is full use of
0:52
the computer like it, the name States.
0:54
And then Web agents is almost a hybrid of AP is and computer
1:00
use. What exactly is a web agent?
1:02
Why are they useful? Let's get into it.
1:04
Yeah. So we talked about the fact
1:07
that, you know, what are agents and that my simple definition
1:09
going back to that is agents are LMS that interact with the
1:12
world. And one way to interact with the
1:15
world, certainly one way that we as humans interact with the
1:19
world, is through. The Internet.
1:21
The web. Right.
1:23
And browsers more specifically. And because at pros, you know,
1:27
we're a large tech company, we interface with our consumers,
1:29
about 2 billion of them all over the world through our companies
1:32
and their products, which are almost all entirely on the web
1:36
or through apps. As we think about what's the
1:39
next stage of agents, of course, we're, you know, we're, we're
1:42
also exploring how the web can be used by agents to help users
1:49
navigate, find, discover, transact, buy and so on.
1:53
And and that's why we have an active, let's say team working
1:58
on on web use for agents. Why only web agents and why not
2:02
just go full all in with computer use?
2:05
Yeah, I think the way we're starting to to, to, to give
2:09
these agents ability to interact with the world is gradual.
2:13
And, and that graduality, sort of incrementality sits in the
2:16
fact that you kind of increase complexity.
2:19
And think about the 1st access that we gave LMS to the rest of
2:23
the world were through simple APIs and function calling.
2:27
And now we're saying, well, maybe they can actually interact
2:30
with the world through a browser.
2:32
And then of course, after that it will be very likely the
2:35
computer as we're already seeing and then maybe after that the
2:38
physical world. And so web browser sort of the
2:42
next step, the natural step that we wanted to take also because a
2:47
lot of as I said, are, are are interfacing with our consumers,
2:51
customers happen happens through the web.
2:54
I, I know that there is a ton of e-commerce stuff that you all
2:58
are doing like OLX for example is a huge one.
3:01
And if you have web browsing with agents, why would you use
3:07
that instead of AP is why would you?
3:10
How would you build AE commerce site that's optimized for web
3:14
browsing for agents? Like how do you think about
3:17
that? Yeah, let let's start with a
3:19
question. Why would why would we want to
3:20
do that? Well, as I mentioned, I think
3:23
the IT, it isn't a very important way that we interact
3:27
with the world today. The way we discover things, we
3:31
learn, we may buy and shop and, you know, exchange goods in
3:36
general. And so that's why we're trying
3:41
to solve that process. All of our companies have a web
3:44
interface to, to the world. And then the question is, OK,
3:51
well, how do you start going down that road?
3:53
I mean, why not just through API as well?
3:55
The the truth is that actually, you know, we're most of the
3:58
things that we want to give agents access to aren't made for
4:01
agents to access. They aren't API FIDE yet.
4:05
And so. What do you mean by that?
4:06
Like a photo on a website or what is not API ified.
4:13
Well, if I, let's say I, I want, so one of the predictions we've
4:16
made at the marketplace at 10% of the e-commerce transactions
4:20
will be done by agents on behalf of consumers in the next year,
4:24
right. Whether it's 10% or not or not,
4:27
I think we, we, it's safe to assume that agents will start to
4:31
take actions on behalf of us on the web in some near future.
4:36
But then the question is, OK, which types of action?
4:38
Let's say you want to order a flight or you want to order, you
4:42
know, some a pizza to be delivered to your house today.
4:46
You obviously go to the browser, you have a mouse, you click
4:49
around. There's no API for all those
4:52
actions yet, right? You go to URL and the agent
4:56
needs to then figure out, OK, what, where does it click?
4:59
For example, does it do that based on vision on the, the,
5:02
the, the site map? And so that's not a, a regular
5:06
API. Obviously there's a lot of AP is
5:08
behind the website that you're interacting with, but there's no
5:13
obvious place to kind of plug the agent into in a very
5:17
structured way that the agent, for example, today interacts
5:20
with all sorts of other tools that we're giving them access
5:23
to. Yeah, and I want to call out to
5:27
that. I've heard the pros and cons for
5:29
both API and web browsing or computer use agents and we're
5:36
going to be having a full debate on this particular topic to hear
5:42
different thoughts and viewpoints from engineers on why
5:46
to use one and why to use the other and why it's the future.
5:50
So when I think about web agents, one of the clear value
5:54
props is that you can build it once and it can go out and do
5:59
what it needs to do and you don't rely on AP is which is a
6:02
huge selling point because AP is are very finicky.
6:07
And so I've said this before that you don't have to think
6:12
about building a bunch of different API calls.
6:15
And then the agent has to choose which tool it's going to use,
6:20
which API it's going to go out and find in.
6:22
And you don't have to worry about, oh, this API now changed
6:25
it the way that it works. And so we have to update.
6:28
And then it wrecks the agent. You just build the web agent, it
6:32
goes out, it explores, it interacts with the web in the
6:35
way that we think about interacting with the web.
6:39
And so it is more human like. Now let's take the second part
6:43
of this, which is how do you think about building websites,
6:49
particularly in e-commerce that are agent friendly, web browser
6:55
agent friendly, I mean. That's as agents are developing.
7:00
It's even unclear what agent friendly means.
7:03
But you know, to go back to your question, you know, why just not
7:06
use an API like AP is today, as you will understand, listen to
7:09
this podcast is you've got some kind of parameter and input, you
7:13
send it somewhere else and you expect a standard response.
7:16
It could be, you know, even in the Jenny I world, it could be
7:18
like, hey, I've got a prompt. It goes to an API and that
7:22
prompt is used to take text to image.
7:24
Head to a text image model and returns an image in a certain
7:27
format and you know what's expected in and you know what
7:31
you can expect back. There's no API, for example for
7:36
Amazon or. Even LinkedIn doesn't have an
7:40
API. You can't say them all.
7:41
Let's say Amazon, you know, a gift for my niece.
7:44
There's not something you can say.
7:46
That's the query. And then, you know, add
7:48
something to my shopping bag basket back.
7:50
There's actually a set of steps that we do as humans to get
7:52
there. You obviously look at that.
7:54
You, you search, use the query, you search, you look at certain
7:56
items, you click around, you maybe you read and reviews.
7:59
So a lot of things that go into the shopping experience.
8:03
And the same is true, by the way, for the food ordering
8:06
experience, which we do a lot of as we talked about, or finding
8:10
secondhand goods in the classifieds space where you need
8:13
to talk to sellers. And so all those steps are
8:16
things that we now use the web and the browser for that we're
8:20
exploring how well agents could do any of these steps in the
8:24
value chain. And you can't do that with AP is
8:27
that's a very clear, yeah, reason why you would want to do
8:30
it with a web agent, yeah. Well, if it's already designed
8:32
for us, I mean, that's how you know, billions of us interact
8:35
with the web today is the browser.
8:38
If you can do that reliably, then that means you can, you can
8:43
very quickly start to give these agents a whole bunch of other
8:46
capabilities because you can say, hey, go check online and
8:50
book me a restaurant for two, right?
8:52
And then tonight at 7:00 PM and it will be able to look at
8:55
what's available and so on because you can just browse the
8:57
web like you or I would do that. I will say since we've been
9:02
talking about agents so much in the last couple days that this
9:04
whole week I've had like agent Boot camp coming to the process
9:09
offices and talking to everybody on the AI team about what
9:12
they're working on, how they're dealing with agents.
9:14
What are some of the challenges when I open my calendar to find
9:21
the location of this place that we're recording this podcast at
9:26
and then I have to copy and paste the location into Uber or
9:30
into Google Maps. I'm sitting there and I'm like
9:33
this is so backwards because agents will be taking over this
9:39
or how does my phone? At least it doesn't need to be
9:42
an agent. How does the intelligence of my
9:45
phone not know that I have a calendar invite for this time?
9:50
Maybe I should just already have an Uber that is being ordered so
9:54
that I can get there on time. Yeah, I think you're already
9:58
thinking a couple of steps ahead in terms of navigating or first
10:01
across an operating system, multiple apps.
10:05
You know, we're looking at frameworks that are made to do
10:07
that. So they really that's sort of
10:09
computer use across apps, you know, certain benchmarks like OS
10:13
world. And so those are measured
10:14
against these kinds of tasks that, you know, require opening
10:18
a directory, loading some files, reading them and putting them
10:22
into Excel and opening a pivot table, right.
10:24
Those are sort. Of multi multiple steps.
10:27
Yeah, multiple. Actions.
10:28
Maybe you could copying the address like you mentioned and
10:30
ordering an Uber through the their app.
10:33
That's I think that's definitely coming.
10:38
But even the step beforehand where you within one website,
10:44
right, Let's say within an e-commerce website like oh,
10:47
Alexis, website glove or ifood or emag or take a lot.
10:50
All these websites we have along the group.
10:53
If you want to go and buy something, even that is
10:55
difficult because there are pop ups coming in.
10:59
There are, you know, if you're trying to order some food, it'll
11:01
say what, you know, what side dishes do you want, what
11:04
toppings do you want? Obviously there will be, you
11:08
know, a whole bunch of other filters.
11:10
Sometimes we've designed things. Am I human or are, you know, are
11:14
you a robot like captchas and so on.
11:16
Those are all things that make it it's a hard to solve the
11:22
entire task of or getting food or ordering, whatever.
11:25
Hard for humans, even. Well.
11:28
Hard for some of us, for sure, and even harder for agents at
11:31
this moment. Yeah, it just reminds me of the
11:34
whole conversation we had a few episodes ago about the cognitive
11:38
load and how we want certain things that we do want to be the
11:45
least amount of cognitive load possible so that we can use all
11:49
that precious brain juice for something that actually is going
11:53
to take us the whole cognitive load.
11:55
But let's now skip into the part where we get to talk with Kiara,
12:00
who has been working on web agents for the last six months.
12:04
She spent a half year diving into it and getting real contact
12:09
with what's been working, what hasn't been working, playing
12:12
around with web frameworks and learn from her and get her
12:16
insights. Great.
12:19
Chiara, you're here. Thank you for joining us.
12:21
And I would love to start out with a brief overview of the
12:27
project that you've been working on so that people can know the
12:30
web agent journey that you've taken.
12:35
So the first project that I worked on when I joined the team
12:39
was to build an agent that will help people order food.
12:43
And it's, it sounds like a simple task, but it's really
12:48
complex. So you need to understand the
12:50
user, what, what would they like, what are the dietary
12:53
restriction context? Also what time of the day it is,
12:57
where are they located? Are there events in the area?
13:01
And this agent should be able to order food.
13:05
So go to food platforms, maybe several of them, advise the user
13:10
on what's available where, the promotions, and ultimately being
13:15
able to order food. And as Paul said before, not
13:20
everything is available through an API.
13:22
So we decided to delegate this task to a web agent.
13:28
And web agents, I mean, there's a reason why they're popular
13:32
right now. You've talked about it already,
13:36
and they're really powerful. And we saw that there were a lot
13:41
available at the moment, few, few of them were coming every
13:45
month. So it was really a challenge to
13:47
understand how to navigate this landscape of agents.
13:51
So I immediately started to try out all these tools.
13:55
So there are a lot like multi own is probably the most famous
14:02
at the moment, but there are a lot of open source projects as
14:05
well from companies and research groups.
14:08
And these tools are all really nice, but we discovered pretty
14:14
soon that they have a lot of limitations.
14:17
So one thing that is really a big problem for agents is that
14:23
websites are built for humans. So there is a lot of information
14:27
that is not agent friendly at all.
14:30
You have dynamic content and loads the the Dom of the page
14:34
could be huge, could change. There are of course the standard
14:40
things like captchas. It sounded like something that
14:45
would be really easy for us, so we will just delegate execution
14:49
of these tasks to an agent, but it was not at all.
14:52
I mean, at this point, we already been building the data
14:54
analysts and other things which are, you know, from our magic
14:58
for it seemed like much more complex to get right.
15:00
Because you have to get the database, you have to run
15:02
queries, you have the code executor, you have to validate
15:04
like, well, what we're going to do now is just we're going to
15:06
send this agent to to a web and basically, you know, navigate to
15:10
beautiful soup style, just go and cycle.
15:12
Yeah, yeah. And actually it, you know, it
15:15
took, it took forever. It didn't complete any of the
15:17
tasks even multi on at that time.
15:19
We looked at all the other frameworks so.
15:24
Yeah, we saw one good example is Web Voyager is one of the first
15:28
web agents. They also published a benchmark.
15:33
It's public. Many agents benchmark against
15:35
this and what this is that the success rate can change a lot
15:39
from one website to the to the other and from 1 task to the
15:42
other. For instance, website where you
15:44
need to perform a lot of actions like booking.com or Google
15:49
Flights, they're really hard to navigate for an agent,
15:52
especially when an action can trigger something else that the
15:57
agent doesn't expect. Like when you book a flight, you
15:59
select an airport and the destination airport changes
16:03
depend on depending on the starting airport, right?
16:08
So this is all really complicated for an agent.
16:11
So yeah, we've started trying out all these tools.
16:16
We built a lot of internal knowledge about them and all the
16:19
techniques that they use, and at the end we decided to build our
16:24
own basically. Framework like your own.
16:27
Build our own web agent. Yeah, and how does the logic
16:32
work? Or what is the backbone of a web
16:35
agent do? Because I know the API agents
16:40
use tools and you have the potential to do function calling
16:45
or whatever, but what is with web agents?
16:47
What does it look like? So it boils down to something
16:51
really similar, actually. An agent is something that has
16:55
access to information and can decide what to do next, right?
16:59
So the information in this case could be the screenshot of the
17:02
page or the Dom HTML. Lots of things that you can get
17:06
from a browser and what it can do are the actions that the
17:10
human can also do, like clicking, entering text,
17:14
scrolling and so on. Those are like the tools it has.
17:17
Yeah, those are nothing but tools that the agent can call.
17:22
So the way it goes that the usually, usually there is a
17:26
planner that decides once it gets the task, decides how to
17:31
perform it. It could be several steps for
17:34
instance, to go involving several websites.
17:36
And then there is an agent that chooses which tool to use, like
17:44
if there is a cookie banner and it's to click accept for
17:46
instance. It's something that could be
17:48
unexpected. So that's why you need an agent.
17:51
You need to be able to react to an open world.
17:55
And by the way, this is a great example because there's, you
17:58
know, we were benchmarking on, you know, and looking at others
18:00
benchmarking on web Arena, which turns out it doesn't translate
18:03
at all to the tests we were doing, right?
18:05
It is one that the, the actual average results didn't, didn't
18:09
compare, but also they were super unpredictable, right?
18:13
So one time they worked, one time they didn't.
18:15
So we'd have to devise simulations where we go to look
18:18
at how many times out of 20 would this thing succeed on on a
18:22
task that we care about, right? Wow.
18:24
Yeah, and most of the times we saw agents getting stuck in
18:27
loops and yeah, just not knowing what to do next.
18:34
You ended up stopped because maybe the task was not clear or
18:37
the task that the planner gave was not clear so.
18:40
Or the action space is not clear because we also saw that, you
18:43
know, you're looking at a website and for us it's very
18:45
obvious, right? You look at a website and
18:47
there's all these, you know by now, like patterns that we're
18:50
all familiar with. Their search bar scrolling is
18:52
one that was super hard for these agents to do.
18:55
But you basically have, you've determined this is the, the
18:58
website you, you know, you've got some reasoning through some
19:00
LLM that tells you what you want to do next.
19:02
But where on this website do you Click to do that?
19:05
Just understanding what is the coordinate that corresponds to
19:10
the action I wanna take, right? And that's not something these
19:15
multimodal image models were good at.
19:17
Like just taking an image, understanding what are the kinds
19:21
of actions are there is fine, but then saying okay then you
19:23
need to click on such and such coordinate to execute that
19:25
action or Scroll down because it's probably lower on the page.
19:29
Because I don't see it. Or you don't see it because
19:31
there's a privacy or a cookie banner in the way.
19:33
Yeah. Yeah, one of the first things we
19:36
worked on was scrolling actually.
19:38
So we looked at the open source frameworks that were around and
19:42
we tried to use similar strategies, but we chose only
19:48
specific strategies that applied to a use case.
19:51
And I think this is really important because the way to
19:54
make an agent succeed is to limit the amount of choices it
19:57
has to do as much as possible. This all these tools were
20:02
optimised for specific goal which is being able to surf the
20:06
web. But our goal was different.
20:09
So in our case, we couldn't use a tool like that.
20:14
It wouldn't work for us. So we needed to build a web
20:17
agent that could interact with platforms to order food and
20:21
that's a different task. And since the the scope of this
20:25
this task is smaller than we could optimize for that.
20:29
So we built an agent that could get more informations about the
20:33
page depending on on the platform that we were working
20:39
with, We could prompt the agent to behave in a certain way, like
20:42
first you search the search bar, maybe you need to enter your
20:46
postal address and so on. There are certain things that
20:50
always put together. And yeah, this is also another
20:55
thing we did. We took certain tools and we
20:59
merged them. Oh, nice.
21:00
So if there are actions that you always do at the same time, why
21:05
would you use two tools for that?
21:07
For instance, when you search for something, you type in and
21:11
then you press enter. Yeah, so these are two separate
21:15
actions, but you can combine them into one tool because we
21:19
needed that. You basically never type without
21:23
pressing Enter. Yeah.
21:27
Another thing was improving the scrolling.
21:29
When you have long menus, lists of restaurants, you need to be
21:33
able to fetch all the information.
21:35
So we adopted the strategies to work with our use case and we
21:40
got a good success rate for that.
21:42
So I think the lesson here is that if you want to build a web
21:46
agent for a specific task, keep in mind the task that you have
21:50
to do and be smart about it. If if there are things that you
21:56
don't need to, don't, don't add them to the agent.
22:00
And so did you go and map out the trajectories and the user
22:04
flow on these food ordering apps?
22:07
And maybe it was like you would go to I food and say, hey, I
22:12
want to order pizza. And then go through that flow
22:15
yourself so that you could use it as a golden data set for the
22:19
evals of the web agent. So one thing that we did was to
22:24
do all this flows manually, like I think you cannot build
22:28
something until you try to do it yourself and understand what are
22:31
the pain points. So this was first thing that we
22:34
did and then we tried to prompt the agent to interact with the
22:41
web page in a certain way and this instruction were loaded
22:44
dynamically based on the page it was on.
22:47
This was one thing. So all this these methods don't
22:54
really change the speed at which the agent works.
22:59
But what what we did was also storing all the trajectories
23:03
that the agent had done. We defined 3 modes that the
23:08
agent could operate. So one was the traditional mode
23:12
where it would grab a screenshot of the page, load the content
23:16
and then decide what what action to take next.
23:20
The other was faster mode that didn't involve a screenshot.
23:25
And we would do that on pages that we knew.
23:27
So if we would search for a certain food and the food would
23:32
be different, but the task would be similar, we would not load
23:35
the screenshot of the page because that was not needed.
23:38
So the agent would know exactly where to click because it had
23:42
seen that task before. And there there was a third
23:47
mode, which we called reflex mode, in which we would automate
23:52
the web actions directly, like sort of a macro, let's say.
23:57
Some parts of this can be automated.
23:59
So why would you have an agent do it right?
24:02
So yeah, we combine all these things and the agent would try
24:06
to do things in a fast way and then if it would not succeed,
24:09
would do it in a slower way. So it was almost like the slow.
24:13
It was the Plan B. In case it couldn't do it fast,
24:16
it would do it slow and you would give it a little bit more
24:19
reasoning or you would give it the screenshots and it was more
24:22
thorough. I think that's an interesting if
24:26
you try to understand like what what do we think these things go
24:29
is? You know, we start with a set of
24:31
tools and frameworks that aren't really made to interface with
24:34
the web necessarily. But through those 3 modes that
24:37
can I just explained, we were able to actually have it, you
24:42
know, because of these trajectories that it knew were
24:45
successful on specific sites for specific tasks access that, you
24:50
know, that learned, let's say, learned action space for
24:56
websites. It was then sort of quote UN
24:58
quote familiar with. And I think that's something
25:00
that if we think about our world like you would want to have,
25:04
like when we go to websites, we know we're familiar, we can
25:06
navigate, right? You evolve into booking.com, you
25:08
go there, right? You know exactly what to do if
25:10
you order your food many times and you can see and you, you
25:13
don't need to kind of rediscover that page.
25:15
And these agents essentially we're starting to see as we as
25:18
we are able to create that persistent or you know, learned
25:23
intuition about a website. They become experts, well first
25:26
familiar than experts and can get you to the your desired
25:29
output much much faster. And I know that I'm still trying
25:33
to just separate like what's different and what's newer with
25:37
the web agents And with that learned experience, something
25:41
that it's seen, How are you saving it and how are you making
25:44
sure that the agent has access to it?
25:46
Are you thrown it in a database? Are you caching it?
25:48
What does that look like? I think the storage itself
25:52
doesn't matter that much as long as this is something that is
25:58
privacy compliant and doesn't lead to leaking user
26:02
information. But you're storing the path that
26:05
it took or you're storing the action.
26:08
What are you? Because you're not storing the
26:10
screenshot and then loading that up again, right?
26:12
No, we store all the path and the state that the page was on
26:17
so the agent knows how the Dom looks like, what are the
26:20
elements it can click on. OK, another thing I didn't
26:23
mention before is that we did some work to understand how to
26:28
clean this Dom because there is a lot of information there, but
26:31
the agent doesn't need all of it.
26:33
It should get as least information as possible, so we
26:38
only took the elements that were clickable, for instance, and we
26:41
combined this with a screenshot in fast mode.
26:45
The agent could be a bit more blind, let's say, and knowing
26:49
where to click because the task was really similar.
26:53
Nice. Now the other thing that I think
26:56
we wanted to talk about was this, the differences between
27:01
planning and execution and the models that you use for each of
27:05
these. Because we know that there's the
27:07
reasoning that models that you probably are using for the
27:11
planning. But then do you offload that
27:13
onto a model that is smaller and just executes or is it
27:17
fine-tuned? What does it look like?
27:20
So for this specific task, we use foundation models.
27:25
We did experiments with all the major foundation models and we
27:29
saw of course some differences. For the planner, of course, it
27:33
helps to have a model that is good at planning, like a one for
27:38
instance. And for execution itself, you
27:42
don't really need the model that is is good at planning, let's
27:47
say, because as long as it knows what to do, this is a very
27:52
limited action space, right? It's an important sort of
27:57
pattern that's emerging, the separation of the planning and
27:59
the execution as you start to interface with the world, right.
28:03
We're talking about now, of course, web is, is one of those
28:06
interfaces because planning itself requires a lot of
28:09
reasoning of understanding the intent of the user.
28:12
If you do the execution, that basically means I actually need
28:15
to know the action space really well and to be able to translate
28:18
that plan into the action space of my, you know, world, which
28:22
could be at the broadest sense could be the web or could be a
28:25
domain like I food or OLX, right?
28:27
Or pay you or any of those you know, basically websites that we
28:31
know and understand well that the second execution agent then
28:36
needs to navigate to help to get to that outcome successfully.
28:40
Is that where the simulations were coming in?
28:42
And tell me more about what the simulations were and how those
28:46
helped. In terms of the simulations, I
28:49
think what it's also really nice with the web, which is different
28:53
from other places we've been applying LMS. You you can just
28:57
send these agents out to go and explore the websites like a web.
29:00
Crawler. Right.
29:01
Essentially, you can basically say go and find me a blue couch
29:06
in Warsaw and OLX and it can then go and explore.
29:10
And as long as we've defined what success looks like, like
29:14
for example, found the couch or added it to cart or whatever it
29:19
is, then we can do this 100 times.
29:22
And it learns what the trajectories are that are most
29:24
likely to get it to that state. And that's where it's sort of
29:28
it's, it's, you know, it's simulate, it's more exploration
29:32
to learn what these websites can and cannot do that allow you to
29:36
get to a system that actually is really good at executing within
29:40
your, your catalogue or your e-commerce environment.
29:43
It's basically like you're mapping out the space, correct?
29:48
And then once you have the map you can traverse it easier.
29:52
Well, no, I I think we're seeing it in computer use similarly
29:55
that people are starting to map out the applications, right.
29:58
So if you were to know every button on Excel or on Word or,
30:02
you know, the commonly used apps, and if I go to you and
30:06
say, hey, you know, please make me a presentation in dark mode
30:10
with certain such fun and background and you've done
30:13
powerful many times, you know where to go.
30:14
Yeah. You know, and I think that kind
30:16
of mapped action space is something you can simulate
30:21
essentially because you just go and have your Agents Explorer
30:24
apps. Are you using hotkeys for as
30:27
tools? You could.
30:28
Yeah, yeah. Top, for instance, is a very
30:31
useful hotkey. You can do most of the things,
30:35
you can do most of the things on a web page through the top.
30:38
And yeah, this is also. This is also one of the reasons
30:44
why it's good to separate planner and execution, because
30:47
execution only has limited tools and does only it needs to
30:52
understand whether it has finished or not.
30:55
So we would have a planner telling the the executor a very
31:01
specific task. For instance, if I would search
31:06
for AT shirt on OLX, first thing would be open the OLX page,
31:09
search in the bar and so on and the planner would take all the
31:14
sub tasks and execute them and stop when it was finished and
31:18
give the response to the main agent that then would process it
31:21
and decide what to do next. Oh.
31:24
Nice. So yeah, it's you can map the
31:27
space and give a better information to the execution
31:32
agent. So, but from both sides
31:34
basically both from the planner and execution side.
31:38
So I think there's a lot of room for improvement once we will get
31:43
more data all. Right.
31:45
So now let's talk about some of the frameworks that you used.
31:48
You did mention Web Voyager, you also mentioned multi on.
31:51
I imagine there were things that you really liked from some of
31:54
these and is there anything that just stands out at you from one
32:00
of these web frameworks that was particularly, maybe it's a novel
32:05
way of doing things or a good way that you feel like was
32:09
something that you brought back into the own your own framework
32:12
that you created. So something I really liked was
32:18
Web Voyager. It was the simplest.
32:22
All the other frameworks, many of them they built on top of
32:26
that, but it was really clear the separation between planner,
32:30
executor, executor didn't have many tools available, just basic
32:37
web interaction through an SDK. Like in that case it was using
32:42
Selenium, which is a testing tool.
32:46
We, we decided to choose another one.
32:48
But the the strength there, it's it's simplicity.
32:54
So yeah, I think that's really powerful.
32:57
And I also liked the visual approach because the the Dom
33:04
does does not always bring you in the right direction because
33:09
you could have misleading information there.
33:12
But what the user sees is what what is important at the end of
33:17
the day. So yeah, that that approach I
33:20
think is really is really useful.
33:23
Other frameworks built on top of that and they added more
33:26
complexity in terms of planner. For instance, we saw agent E
33:30
that added the more hierarchical type of planning, which
33:34
increases the success rate it does, but it also makes the task
33:37
more difficult and slower to execute.
33:41
We saw other approaches like Monte Carlo 3 search for
33:44
planning. This is open source project from
33:49
multi on as well. At the end, we decided to choose
33:52
the simplest, simplest possibility because our task was
33:58
clear, we knew what we had to do and we ended up using this agent
34:02
sort of an API. We we created a code that was
34:05
very modular so we could delegate things to a web agent
34:09
that we wouldn't know what it was doing.
34:12
It was kind of a black box within our application and would
34:16
give us the response and with that we could take action and
34:20
interact with the user because at the end that's what's
34:22
important. Basically, we're six months in
34:26
the future of your journey. Knowing what you know now, what
34:30
would you tell yourself if you could six months ago about this
34:34
whole journey? A lot of things, so let's start
34:41
with. I think the most important one
34:43
is to really understand what is the problem you're trying to
34:47
solve. Dive in, try to do the things
34:50
yourself because web agents are automating tasks, so try to do
34:53
it yourself. Try to see what are the pain
34:55
points. I would explore all
34:59
possibilities that are around, but reminding myself that these
35:05
tools are not necessarily what I need.
35:07
The other thing that helps is to approach this as a software
35:12
engineering problem rather than data science.
35:14
And I say that because I come from data science background, so
35:17
this was really, really big for me.
35:21
What does that mean? What's?
35:22
The difference, Yeah, I'll come to that.
35:24
So this is not the data science project, but software
35:29
engineering project where there are some, some LLM steps in.
35:36
And this means that you can adopt all the good practices of
35:39
software engineering like keeping things modular,
35:41
separating responsibilities, and keeping things as simple as
35:47
possible, like trying to have more control.
35:51
I think you discussed this about the SQL agent.
35:54
This is even even more important there.
35:57
But yeah, it's it's important to understand where you need the
36:04
agent and where you don't and try to limit the amount of LLM
36:07
calls as much as possible. Why I say this?
36:11
Because when you approach this agent projects, it's really
36:15
tempting to use all this high level frameworks with high level
36:18
of of abstraction, do everything through an agent.
36:22
But this is not the right way to do that.
36:24
I mean, it's nice to do proof of concept play around, but if you
36:29
need to build something that works, you need to have control.
36:34
So you it really helps to think of different modules.
36:39
So I have a planner that needs to think very well, but the
36:42
execution part doesn't need to be done necessarily by an agent.
36:50
Like if there are things that can go through a deterministic
36:55
approach, it's much better. An example, in this tool, we had
37:00
to pull user information because we had to understand user
37:04
background, whether they had data restriction, for instance,
37:08
things like that. We didn't always need it, but
37:11
most of the times we needed it. And I very naively built an
37:15
agent that could interact with the database, retrieve the
37:18
information, but actually we didn't need to do that.
37:21
Why? Why would you use an agent if
37:23
you can just pull the data and add it to the context, like to
37:27
the prompt? So that was a big revelation for
37:31
me because it made things much more simple and give us control.
37:38
So make use of the frameworks where you need them, but keep in
37:43
mind that you can go low level and have more control over the
37:47
parts that are important and test things.
37:51
Try try to find edge cases, try to find what doesn't work and
37:56
have fun. That's also.
37:58
That's what you would have told yourself, but you didn't have
38:00
fun. Yeah, I had a lot of fun, yeah.
38:05
The the idea of where to use an agent and when to use it is
38:12
really a fascinating point because like you're saying, you
38:17
can build an agent to do this thing, but if you can do it
38:19
without an agent, it's going to be more predictable.
38:22
I've heard the other side of the argument be I can prototype an
38:27
agent or I can create an agent so fast it's almost faster if I
38:31
do it through an agent versus if I do it through traditional
38:34
software development. Have you seen or do you have
38:38
thoughts on that? I mean, I think in general for
38:40
prototyping, that's definitely true.
38:42
You can prototype a lot of stuff very quickly, but at the end of
38:45
the day, you know, we our, our main job is to take something
38:48
that we can see work and scale it.
38:53
Yeah. And so our next step is always
38:55
because of the the size of our platforms is to scale it to 10s,
38:58
hundreds of millions of users and their deterministic
39:02
workflows are much more preferred if you can or at least
39:06
narrow. Down so many reasons, right?
39:08
So I think we tend to, yes, prototype, you know, in any way
39:12
we can quickly. But then after that to Kiata's
39:14
point, we can start. We need to distill the the
39:17
actual essence of, you know, the system to one that can be put
39:22
into production and scale. And often that may so include
39:25
function calling and agentic components, but it's not as sort
39:29
of free reign or as as you know, maybe we do and when we just
39:35
start exploring it. Yeah, yeah, yeah.
39:37
Did you create any benchmarks or or particular evals for this
39:42
project? Yeah, we had, we chose some
39:46
tasks that were representative of typical e-commerce
39:51
interactions and we tried to optimize for those and then we
39:56
added variations of those and yeah, try and different
40:00
possibilities and until we were happy with it.
40:04
Did you have a certain accuracy score that you needed to get
40:07
above? So our target was 80%, yeah.
40:11
OK. But it of course it heavily
40:13
depends on the task and on the website and on the user itself.
40:17
Because it really it depends what the user wants and like how
40:24
specific the request is as well. So of course, there is a whole
40:29
planning step where the agent talks with the user and tries to
40:33
understand whether it has all the information.
40:36
And then the other part, the execution is to have all, all
40:41
the right information to be able to perform the task.
40:44
So this was very important. For instance, you cannot order
40:48
food if you don't have an address.
40:51
Like the user has to be willing to to provide the address.
40:55
Yeah. So yeah, this was was a
40:59
challenge and to be back to go back to the deterministic
41:03
approach. There is some deterministic
41:06
component in here as well, because if I need to perform a
41:10
certain task, I need to inform the the web agent, I need to
41:14
give give it the right information to be able to do
41:16
that. And that's deterministic.
41:19
Like the planner needs to know that it needs to provide an
41:22
address list of dishes and so on.
41:26
And yeah, that's, that's really important and it increases
41:30
accuracy a lot. We we tried both ways and that's
41:34
definitely better. I think one way to kind of see
41:39
what, what we concluded from doing all this work is that, you
41:42
know, on the one hand, I think if we just take an agent and
41:44
want to go to, you know, one of our existing websites or
41:49
platforms, we can get pretty far.
41:52
But I think there's a sweet spot where we can also, because
41:54
there's a lot of things we don't display on websites that may be
41:57
useful to help a user, you know, going through an ecommerce
42:00
journey. To give you an example, like
42:02
when we're in OLX, you actually, you know, as OLX, we know it's,
42:06
it's a classified space, you know, buyers and sellers of
42:08
secondhand goods. We know what the reputations
42:11
are, certain sellers, we know, you know, location of people, we
42:15
know what kind of things they've searched in the past.
42:17
We know what the supply and demand are.
42:19
We know what reasonable prices are for categories.
42:21
Those are not things that necessarily an agent has access
42:24
to if they just act, you know, just go to the website of the
42:27
marketplace. But if we're building the agent,
42:30
you know, as the marketplace and we've got access to all that
42:33
rich marketplace dynamics and information and customer
42:36
reviews, there's certainly relevant at the moment of going
42:40
through a transaction. That's where I think we can, you
42:43
know, create really useful agents.
42:45
And that's certainly a conclusion we we drew away from
42:48
because of course here the experimenting we're doing is
42:50
just purely from the outside in. But if we combine that and and
42:53
sort of build a agent that has, you know, is integrated with the
42:57
with the platform and is available to the user at the
43:00
moment they want to, you know, find or exchange things, that's
43:04
going to create a completely new AI first.
43:08
Yeah. You know e-commerce experience
43:09
and you know, we'll hopefully be able to talk about some of that
43:12
in well it goes. Back to building your website
43:15
for humans or for agents, because you can also expose that
43:19
data for other agents to use, or you can choose to not expose
43:25
that. And I know that's a, a debate
43:27
topic that we're going to have too, because it's like, well, if
43:30
this is useful for me, then it might be useful for other
43:33
agents. And if we expose it in a way
43:37
that a human's not going to see it, but if an agent is using the
43:41
website for some reason, they will be able to see it.
43:44
I don't know how that would look, because if it's not
43:47
exposed in the GUI and you're using a web agent, the web agent
43:51
has access to the GUI, right? Or it also has access to the
43:53
Dom, so maybe you put it there and then that exposes it.
43:57
We saw some website that started to add the markdown, for
44:01
instance with the description of the page.
44:03
That's already really helpful. So I see some progress in this
44:08
direction. This helps especially with
44:11
e-commerce because you might have a lot of items in a page,
44:14
so it's just much faster if the agent can load them.
44:18
The lessons we learned is that we had to be very specific with
44:22
the, with the instructions, break them down as much as
44:25
possible. So limit the amount of of.
44:30
Sorry, limit the limit the amount of thinking that the
44:34
executor has to do. Delegate that to the planning
44:36
agent. So have the instructions as
44:40
detailed as possible. Break them down steps.
44:43
Try to make use of all the tools you have available, but select
44:50
them in a smart way. Like if you if you only need to
44:54
do a certain interaction in the page, just make only those tools
44:57
available for the agent doesn't need to have all the space.
45:02
I've learned so much in the six months.
45:04
I don't know what like if I look back six months ago, I'm, I'm,
45:08
I'm a totally different person now, so.
45:11
Well, we're always looking for more, more smart people.
45:15
So interns, others, if you want to come check out what we're
45:19
doing, reach out to us. Nice, yeah.
45:22
Really smart people, except for some of them that sit at this
45:25
table. That's awesome.
45:30
You too, to me too. That's how we work together.
45:32
Exactly. Yeah, there we go.
45:40
The Process AI team is hiring, and you can find all the links
45:44
to everything you need to know in the show notes below.