Latest / Agentic Conversations (formally mlops.community) / The Hidden Bottlenecks Slowing Down AI Agents

Transcript

The Hidden Bottlenecks Slowing Down AI Agents · Agentic Conversations (formally mlops.community)

0:00
The eval is 1. Typically the bottleneck is
0:02
actually not the tool itself. It's about having the ability to
0:06
generate those eval sets and having a product feedback loop
0:10
which you can then take data from to measure against, you
0:13
know, new models and improve and sort of have a real flywheel,
0:17
let's say. We approach the builders by
0:23
question when it comes to building agents and what kind of
0:27
vendors you want to onboard in the enterprise setting.
0:31
Bruce and Paul talk us through this.
0:33
Paul is the VP of AI at Process, and Bruce is an AI engineer.
0:37
I myself am Demetrios, the founder of the ML OPS community.
0:41
Let's jump into this conversation.
0:45
You're saying, hey, we have this culture of we should buy it, we
0:48
shouldn't build it. We don't want to think about
0:49
building it Despite that, as we'll hear from Bruce later.
0:54
It's hard, Yeah. It's so immature at this moment
0:57
in time, the tools that you have.
0:59
And I kind of went step by step over the main areas where you
1:03
have these categories forming. Well, maybe what I can emphasize
1:08
is, you know, sometimes the tool is not the hard part.
1:13
Yeah. So let's say if you want to have
1:17
an eval solution, you want to know whether your models are
1:20
good, that's the problem you have.
1:21
And you want to know whether they get worse and how they
1:23
perform and whether they do worse and whether they do
1:25
better. The hard part is not the tool.
1:28
The hard part is you need an eval set that you curate.
1:31
You need to have real users that give you, you know, new
1:34
conversations that you can measure against.
1:36
You need to have that capability if you have, if you don't have
1:40
that, that doesn't matter what whether you build the tool
1:43
yourself or not. And so there in that thing, the
1:46
tool itself, at least in our setup, right in our setup, is
1:50
not the thing that unlocks our ability to do evals better.
1:56
And human time is expensive, yes.
1:58
What I think about though is there's an argument for if you
2:02
have a tool that makes it much easier for humans to get that
2:06
eval set, then you're saving time.
2:09
Right, but I think that that you're absolutely right.
2:13
But the you know, what we do in the team to give an idea is we
2:16
have these labeling parties, right?
2:19
So we buy pizza, we invite folks, everybody's welcome,
2:22
right, Not just folks from the IT or engineers, like anybody in
2:25
the company is welcome. And we'll, for example, like
2:27
show them, you know, these are certain set of answers for.
2:30
So we'll do like, I don't know, images for food as one example,
2:33
or we do code. Obviously, if you do code
2:35
evaluation, you need the folks who can, can, can rank or, you
2:38
know, score or evaluate code snippets.
2:42
We will do, let's say, customer conversations.
2:45
And then you need people in different language, if it's
2:47
Polish or or Portuguese, Brazilian or whatever.
2:51
So we'll do these labeling parties to create data sets,
2:54
right? And these are the eval sets that
2:56
we can then score. You can see that I'm pro LLM,
2:58
which Luku will talk about in another episode, I think.
3:01
So then the hard part is getting those eval sets.
3:06
And it's not that if we then all of a sudden had a tool and we've
3:10
tried them, all, right? Whether it's spellbook or human
3:13
loop, human loop or orc AI and so on, some of them are really
3:17
great. And by the way, they offer 15
3:19
things. One of them is evals.
3:21
The evals one, typically the bottleneck is actually not the
3:25
tool itself. It's about having the ability to
3:28
generate those eval sets and having a product feedback loop
3:32
which you can then take data from to measure against, you
3:35
know, new models and improve and sort of have a real flywheel,
3:38
let's say. Well, one thing that I think is
3:41
wild is how you're actively encouraging the team to go out
3:44
there and buy tools. You would prefer that they do
3:48
that because as Bezos said back in the day, focus on what makes
3:53
the beer taste better, and creating your own orchestration
3:57
system is not what makes the beer taste better.
3:59
It's not going to make Tokan a better tool.
4:01
If there's something off the shelf, go and grab that.
4:04
And also you have the angle of if you find out about a tool,
4:09
then you might want to become an investor in it.
4:11
Despite that you have had a lot of trouble per SE.
4:19
I guess that might not be the best word, but you've had a lot
4:21
of difficulty finding tools that you can actually buy.
4:24
Yeah, less adoption of outside tools for our agents in
4:28
production systems then I'd like or then we'd like.
4:32
But yeah, listen, we're all like we're time and, and and, and
4:36
people are scarce resources, right.
4:37
So, you know, I always say to the team, if if there's a tool
4:41
that we built that we could have bought outside, there's no, you
4:45
know, there's no brownie points for that nobody like there's
4:48
just no value in that. We could have done, you know,
4:50
other things moved faster or whatever.
4:52
So we're very, very actively pushed the team like, hey, check
4:56
out this, this new whatever open source solution, check out this
5:00
new, this tool out there for doing evals, for doing
5:03
observability, for doing whatever.
5:05
And I, I do think our team lives a little bit in the future in
5:09
some ways because we're so like, our job is to test this the
5:12
latest and greatest and then figure out what makes sense.
5:15
We have real problems that probably other folks will have,
5:18
you know, 1218 months down the road.
5:21
And so it's not just, you know, I want my team to like have all
5:25
the time that they can to spend on the things that make a
5:29
difference. And so I encourage them to buy
5:31
and use tools wherever that makes sense.
5:33
It's also that if we understand, hey, this tool out there, these
5:37
guys build something for a problem we know we have and
5:40
others will have down the road, we should partner with those
5:43
guys. Like, let's make sure everybody
5:45
in this group, we've got, like I said, you know, thousands of
5:47
engineers all over the world that that will have those
5:50
problems and has already have that.
5:52
We should then expose, you know, this tool to like, hey, hey, did
5:55
you know that such and such actually can now help you with
5:57
authentication in the agentic flow, or they can help you with
6:00
whatever observability of such and such part of the stack And,
6:04
and that happens. And, and it also means, you
6:07
know, because pros has big investment arm, we can also
6:11
help, you know, partner with those founders if they're
6:13
raising money to say, hey, we've won one, we're already using
6:18
you, we want to use you. 2, you may want to have other users in
6:21
the group because there's, we're a big global company.
6:24
And 3rd, if you need help, you know, investments and so on, we
6:26
can provide that because we use tools in my team to try and
6:31
solve real problems that we think others will have soon.
6:34
I tend to trust your opinions about what's real and what's not
6:39
a lot. I heavily weigh them because you
6:43
do live in the future a little bit and you're able to tell me
6:47
what is hype and what is real. And so when I see you saying,
6:53
oh, this is great, but it's not for, it's not for us.
6:58
That kind of is a signal for me to say, uh, there's, there's
7:01
something wrong with that piece. Why is that?
7:04
What is it? What and the fact that you
7:06
haven't onboarded a lot of these common tools that you hear about
7:11
everyday is a very big signal for me.
7:15
Whether that is an orchestration thing like a Lang chain lane
7:20
graph, whatever, or an observability tool like a Lang
7:23
fuse insert, your favorite Lang and an eval tool.
7:29
That for me is a huge signal. It makes it all the more
7:34
important when you do onboard a tool that I stop and I listen to
7:41
why you chose that tool. Yeah, I think sometimes like I
7:45
said, like we, I, I do think in some ways we live in the future
7:48
because we're, we're earlier than, than, than than many and
7:51
trying to build things at scale and production agents or other
7:55
other let's say AI powered products.
8:00
So I, I think one thing that we do understand was what are the
8:04
problems that others will have down the road.
8:07
We're also a bit of a strange team.
8:09
And I've come to realize that, as you know, sort of look back
8:13
is like we are because we're much closer to, you know, we
8:15
train our own models. We work with the biggest labs
8:19
out there. We have our own evals.
8:21
We are not representative of many users that are coming down
8:25
the line. So if you have other teams that
8:27
already, you know, they have ML models or maybe they don't.
8:30
They're smaller teams and smaller companies, they will
8:33
have different needs and they, you know, even if you Fast
8:38
forward them 18 months, they will still not be the same as as
8:41
a sort of full AI team, right? So some needs and the ways to
8:45
solve those needs that says, hey, the problems are common,
8:48
but the ways to solve them for a team that is less AI
8:52
experienced, they will maybe favor a more drag and drop
8:56
solution, for example, as one, right?
8:59
So I'll give you on like, there's a lot of observability
9:00
tools out there and we looked at all of them and many of them had
9:05
this sort of drag and drop solution, right?
9:07
And it turns out that for some people, that's great because
9:12
they, they don't want to get into the intricacies of the
9:14
coding and so on. But you know, my team would say,
9:18
well, listen, I need an SDK or I need this to be headless or
9:21
basically be able to kind of interact directly with the
9:24
underlying things. We can debug it.
9:26
I need to see the logs and so on.
9:29
So those things that they will be, it's like if you look at,
9:33
let's say the observability tools in the, the ones that
9:36
create these chains of, of agents, you have Zapier, right,
9:41
which is just anybody can use it and can have has access to a
9:44
keyboard and a mouse. And then you have all the tools
9:46
like, I don't know, the things that are native in, in in GCP or
9:51
AWS, right, that kind of help you with observability and chain
9:53
these things together. Those are very different end
9:58
users. The people that access the AWS
10:00
environments and the GCP environments and the Azure
10:02
environments are your Sr. user engineers.
10:05
The Zapier users are, you know, anybody.
10:08
And so you need to solve for those different needs as well.
10:13
Are you using coding agents? Yes, we are.
10:17
We started already using them early on.
10:20
And by the way, coding agents, I assume you mean things like
10:23
Devon and Manus and and and many others.
10:27
That's something that you haven't built yourself.
10:29
No, we have not no. So that's a clear area where we
10:34
see, you know, amazing team shout out to the cursor team,
10:37
for example, who have really focused on, you know, the
10:41
persona being the software engineer and how to really
10:46
change the user experience for them to be able to use these
10:50
things right. So we we don't have that's not,
10:52
that's not we, we build products that are consumer facing, right?
10:55
So we would never build a coding agent, but we use them all the
10:58
time. We test them all the time.
10:59
Very excited by all the things coming out.
11:02
We're investing in some of them as well.
11:06
But if you think about, let's say, the first generation of
11:11
tools in this space, the first one, we're get up copilot like
11:16
they were the first ones to come in this space.
11:18
If of course partner with Open AI and we could already see the
11:22
early days that this is actually pretty good.
11:24
And then soon thereafter you, you, you saw, let's say
11:28
newcomers like of course, cursor.
11:32
But source craft codeium. Yeah, replate and so on that
11:36
that came and started to do this slightly better.
11:41
And then they evolved. And actually in my view, they
11:43
they they overtook let's say the let's say the first movers in
11:47
this case copilot to build what are now like software native
11:51
sort of agentic flows. And this sort of conversation
11:55
and the interface that you had that that cursor build where
11:58
they train their own models to actually execute this as was
12:01
very quickly like it was good. That was better.
12:03
And in fact, you know, my whole team moved to cursor and now we
12:07
are sort of at the next level. Think of it like the autonomy
12:10
levels that they have a celebrant care level 1234 like
12:13
we're we're now at the next level where let's say Devon like
12:16
agents who can don't they just don't don't just do auto
12:19
complete. They don't just edit code.
12:21
They can create entire environments, right and and go
12:24
execute a task end to end. We were playing with them, I
12:28
think started about a year ago when the Devon announcement came
12:31
out. And yeah, it was a little bit
12:34
underwhelming, very promising, but underwhelming.
12:38
Now Fast forward, you know, almost a year, we see that it's
12:44
much, much more useful. You still need to look at this
12:49
different, let's say, use cases. So if I'm in my hobby project at
12:53
home and I need to create a little, I don't know, a little
12:55
app or a little dashboard to track whatever household chores
12:59
to say something right by myself, I can use Devin perfect
13:03
now today and it will work basically like first time right?
13:07
Before I had to try it five times.
13:08
Now it'll work for something simple.
13:10
Great. But how much of that?
13:11
It really happens at work, right?
13:13
So. Yeah, a lot of legacy.
13:15
Yeah, I need to commit stuff. I have maybe smaller projects,
13:19
some of them are more demo like there you can also start to use
13:23
Devin. But what happens there is if you
13:26
do what my experience at least today is if you let it create
13:29
the entire code base from scratch for thing that we need
13:33
to actually work on together and maintain like you create it in
13:37
ways you didn't like. The architecture isn't exactly
13:39
as you prefer and so it's not to your taste and it's not easy to
13:43
maintain. That's where I don't see it
13:46
working yet. What we are and now finding it
13:49
works really nicely. Isn't that other type of
13:52
projects where you actually already have a code base that
13:55
isn't, you know, a code base that 3000 engineers are involved
13:58
in, but it's a code base that maybe dozens of engineers, maybe
14:02
hundreds are involved in. But it's, it's, you know,
14:05
clearly documented repositories where you already have a CICD
14:09
pipeline. Things are, you know, already
14:11
documented better because you got more people than you can,
14:13
you know, individually see, so that agent actually comes into
14:18
something and you can ask it to do these tasks that you know now
14:24
are, you know, a card on a JIRA board, right?
14:27
A ticket like, I don't know the make sure this becomes XYZ
14:31
compatible or insert this feature, have Devin give it a
14:35
shot. And that's where these things,
14:37
in my experience, are now becoming great because it's an
14:39
existing code base. We already thought about how the
14:41
architecture should look, but you're making tweaks and
14:44
modifications that are somewhat incremental.
14:46
And the agent, it's not that large that the agent can't
14:50
handle the context. And of course, all these
14:54
features that we're seeing now is, you know, like Devon
14:57
creating its own wiki, right? And indexing the code base,
15:00
various repos together. You can give it instructions
15:04
that then it knows for anybody else working with Devon on that
15:06
code base, that's cool. And we're tracking sort of how
15:09
Devon Dow, like how many PRS is Devon actually making on the
15:14
team? And that's a.
15:16
KKPI or. Is that something like, well,
15:20
it's, it's definitely like we, we want people to do we have
15:24
this duty now we're basically saying Devon duty, right?
15:27
So somebody's on duty this week on, on Devon and this week is
15:30
Devon. But next week it'll be Manus or
15:32
cursor or whatever or Replit. But the goal is that during that
15:36
Devon duty, any task you pick up from the JIRA board, you should
15:40
first think. So let me run it with Devin.
15:42
Wow, cool. To see, right, because then
15:44
otherwise, how do we know if this thing got better and where
15:47
do we apply it? And so that's the KPI that
15:50
everyone needs to try that be on duty to sort of learn and
15:52
discover. And then eventually we may set
15:55
like hey, 10% of the PRS, you know should be Devin, right, if
15:59
we believe that's feasible or 20% or 25% or?
16:02
Whatever, how do you ensure the cognitive load doesn't just go
16:04
through the roof because people are submitting a bunch of shitty
16:07
PRS? Yeah, well, we had this, right.
16:09
So like we actually built our own like PR reviewer, right?
16:16
That thing was so bloody verbose, right?
16:18
It would people turn it off after a couple of hours because
16:21
it was just answering a ton of comments and then the cognitive
16:25
load of you having to read those comments compared to the
16:28
submitted, you know, PR like it's easier just to look at it
16:31
yourself. And you can, you can spare the,
16:33
the, the, the commenting from this AI verbose AI thing.
16:37
So that's the question, right. So where, what's the sweet spot
16:41
where the work done offsets, let's say the cognitive load.
16:46
You still need to additional cognitive load you need to put
16:48
in to understand that AI is work.
16:51
And that's what we're trying to figure out.
16:53
And then then and so is it, you know, how many PRS are worth it
16:56
to sort of give to a, to an AI software agent.
17:01
We're going to play with an SRE agent soon.
17:03
So there's, we just try to understand like where is it
17:06
useful and where not? And some to some extent, there's
17:08
also a team thing, an individual thing, more senior engineers or
17:11
some more juniors. Are you familiar with the code
17:13
base or not? And so on.
17:14
For onboarding for example, is great.
17:17
Tell me more. Like, I mean, so for me, like I
17:19
don't spend a lot of time coding, unfortunately, let's say
17:22
committing like production level code.
17:25
But I obviously want to understand what's happening in
17:27
the codes, which I can just inter like interrogate a code
17:31
base using cursor, right? Or any of the other tools.
17:35
New hires are the same. They come in and they're like,
17:37
hey, I need to work on this new repository, remove people from
17:40
projects to projects. Then they can come in and say,
17:42
hey, tell me, describe to me, you know how this code base
17:45
works. What are the key, you know,
17:47
endpoints or the key services, what are the standard agreement?
17:50
What are the utils? And it'll just describe that to
17:53
you. And so it's a good way to get
17:54
people up to speed or people like me that aren't sort of
17:57
day-to-day in the code base. I think we should bring on Bruce
18:02
now to talk about what tools you all have tested and where you
18:07
decided to build your own and why.
18:12
Great. You're here now.
18:15
I am. Who are you and what are you
18:17
doing here, Bruce? So I'm Bruce.
18:20
I work at process in the AI team, been out for two years
18:23
there now and I work as an AI engineer in the team.
18:26
What I spent most of my time on this building tocom, which is
18:30
our own agent or agent platform and we distribute it to the
18:36
portfolio companies. So I work as a like an AI
18:39
engineer to software engineering a mix of both of those.
18:43
And you spent a good amount of time trying to incorporate
18:47
different vendor tools that will help you build Tokon faster.
18:54
I know that you've been working on Tokon for years.
19:00
And so there is this maturity factor that goes into the tools
19:06
that are out there that you can use and buy even if you wanted
19:09
to pay money for something, maybe it's just doesn't exist or
19:13
it's not at a solid maturity level.
19:16
I typically see folks using tools in a few different places
19:21
and you can tell me why or why not you ended up buying a tool
19:25
in these places. And we can talk about which
19:28
areas there's actually value and there which areas you
19:32
potentially see value in the future if it matures or there's
19:36
no value. 1 is prompting tools or prompt tracking.
19:46
Yeah. So for, I think for a lot of
19:47
products out there and also for what we are doing, prompts are
19:52
very essential to your system and storing that somewhere else
19:57
I guess is fine. Of course, we also stored in a
20:01
database, but we're not storing it in an external service that's
20:06
doing the evaluations for us because.
20:09
It feels so essential to have a very good prompt that I think
20:13
it's something you want to spend time yourself on.
20:16
And then the evaluation piece. So we've built our own
20:20
evaluation flow for that two reasons.
20:24
One big one is that within the AI team, we also have a team
20:27
that do evaluations that have a leaderboard.
20:29
So we have the knowledge to do those things.
20:32
And a second big reason is if you're going to use an
20:34
evaluations tool, you're going to need to send all the
20:38
conversation data to an external party, which usually is not.
20:44
It's hard to convince our legal department that we're going to
20:47
do that. So you bring up a great point
20:49
there on the fact that maybe you wanted to use this tool, but to
20:55
actually get it through and get the OK from legal is a whole
20:59
nother beast. Yeah, yeah, I guess so.
21:03
So it's not only like getting the OK from legal, I guess you,
21:08
I also want to be confident that something like that goes well.
21:11
And if there's like a very new startup companies out there and
21:14
say, oh, you can just send all the whole conversation, it'll
21:17
keep everything in track. It does feel a bit strange to,
21:22
to send it all out there because it's core to your product that
21:25
you do well on when you change the prompt.
21:28
For example. And let's say I'm talking to a
21:33
colleague at another company and they ask me this question.
21:36
I would think like, I'm a bit hesitant to tell you how these
21:40
do this because they are probably not going to like if we
21:42
send out all the, the PII to, to these eval sets.
21:46
I think it really helps then if you have these companies that
21:49
provide that that if they have a self hosted version so the data
21:52
is still at your place. Yeah, that's a common design
21:57
pattern or vendor offering where it's like bring your own cloud
22:02
or VPN and so then you don't have to send that data anywhere.
22:05
No, that's true. But even still, I can imagine
22:08
legal is 1 vector that you're thinking about when you're
22:10
thinking about that build versus buy.
22:13
And so you said we're going to do the prompting tools and the
22:18
evaluation tools in house. Yeah, those are two big ones in
22:22
my mind that people will pay for.
22:25
The other one is orchestration. And so these are frameworks like
22:30
the Llama indexes and the lane graphs, lane chains, that type
22:34
of thing. So I have an example in our code
22:36
base where we built our own orchestrator and we just added
22:40
Vertex AI and we noticed that sometimes we are throwing time
22:44
out so that the request was taking too long because we set
22:48
the time out, it was 5 minutes, so that's very long.
22:50
So because we're doing the request ourselves, where did the
22:55
API implementation ourselves? We weren't using the SDK.
22:58
We were pretty close to what the code does.
23:01
So we easily added metrics there to see, oh, what time, how long
23:06
does it take to get the first chunk from Fertix AI?
23:09
So it was super fast. So we still didn't know what's
23:11
happening here. And then we added an extra
23:15
metric which did the time between chunks.
23:18
And then we noticed all of a sudden it's sometimes the time
23:20
between chunks is super high, like minutes, which is probably
23:24
just a yeah, which is probably just a malfunctioning or Fertix
23:28
AI, which is fine. But if you then using something
23:32
like a library to do that, you're probably missing a way to
23:36
fix that. So now because we did do that,
23:38
it was very easy for us to 1st add a special time out saying,
23:43
OK, if the time between chunks is really lost, we just try
23:46
again with so the user can get a reply quicker.
23:50
And because we're making the whole API notation ourselves, we
23:54
saw that in the header. So it was a request ID that
23:56
Vertex AI send us. We could just easily add that to
23:59
that metric again and it logs that we have, we exported those,
24:02
send those to Google and they easily can then see what
24:06
happens. And I think you need to be quite
24:10
lucky if you accept precisely want to do that, because it's
24:13
quite nice, but it is a fix. But I feel like using an
24:17
orchestrator there probably wouldn't have that or he would
24:20
be very lucky and would pass it all the way up top.
24:22
But yeah. It's that transparency piece.
24:25
You have so much more control of what you are able to see and
24:28
what you're able to do. And you do say that this is a
24:31
very niche and specific situation, but it can be
24:34
generalized to. If it's not this specific
24:37
situation, I'm sure you encountered 5 more like it.
24:41
Yeah, yeah. And then it's nice that because
24:44
CNK, one of our main advantages of using US instead of using a
24:49
different orchestrator or agent framework is that we take care
24:54
of it. We're very reliable.
24:56
On the reliability, do you think that the orchestration
25:00
frameworks, although they give you a lot on the abstraction and
25:04
they're able to make it much easier, they're able to make the
25:07
prototype experience nice, The reliability experience drops.
25:13
And so it's like you have to balance those.
25:16
That trade off is very clear. You have speed to prototype is
25:21
very high, but then reliability is very low.
25:25
Yeah, definitely. Reliability on those frameworks
25:28
can be super good as well, and if they just have what you need
25:32
then it's perfect. We'll explain why they didn't
25:34
have what you need, what you needed.
25:36
I know that you had mentioned the different SDKS and
25:40
specifically like a lot of these frameworks are in Python.
25:43
Yeah, yeah. So we use Go and there, there,
25:48
there are SDKS of course out there that do this kind of
25:51
stuff. But even Google and Anthropic
25:56
say, OK, this is our beta version, which is not funny
25:58
because Anthropic says in their documentation use our SDKS if
26:02
you're streaming because the API, the direct API
26:06
implementation might not work as nicely.
26:10
But. So it's not only with the
26:12
orchestration or the evals or these other tools, it's with the
26:16
foundational models too. Yeah, definitely.
26:19
I think a lot of these for they like these AI products out there
26:24
are really focused on Python doing very, they're probably
26:27
very good at building the SDK. But yeah, if you're not using
26:30
Python, yeah, it's pretty hard to.
26:33
Yeah. And what are some downfalls or
26:38
what are some pitfalls that you've had because of that?
26:41
Not using Python? Yeah, we were using Python two
26:44
years ago or even a year ago still, I think.
26:50
Yeah. So one thing we're missing is
26:52
the SD case because we're doing sometimes like these small
26:57
tweaks that I just talked about. You could do that in Python with
27:00
monkey patching, which is also not possible, and Go.
27:03
But using Go was such a nice transition for us because it's
27:09
compiled and there's already, almost already so much
27:13
uncertainty with what comes out of a model.
27:15
If then the code is, you know, OK, this is fine and.
27:18
And you didn't look into using something like pydantic?
27:21
We did. Why didn't that work?
27:25
That's just for like the models that come in like the objects.
27:29
But then in the codes we I think it just requires a lot more
27:35
effort to make Python fully, fully fail safe where it would
27:40
go. It's way easier to to do
27:42
something like that. It's almost like it comes
27:43
out-of-the-box. Yeah, you can go and in Python
27:46
you have to do this extra work. And super if when I switch from
27:50
Python to go then it was it felt super restricted.
27:54
But in the end now it feels way faster because I just know OK, I
27:57
can't do this, I should do it this way, something new comes in
28:01
and it's just easier to switch. It's funny that it's restricted
28:06
in a good way. Yeah, I guess so, yeah.
28:09
Now you do, so on one hand with Python you are or without
28:17
Python, you're able to have that reliability and you get all of
28:22
these things out-of-the-box. But then on the other hand, when
28:25
you're working with the different foundational models or
28:29
orchestration or any kind of tool, vendor tool, you have to
28:34
do extra work. Do you see it that way or do you
28:36
see it a little bit different? Yeah, it's a little extra work.
28:42
Yeah, I would think that's the case.
28:44
So of course there there's frameworks out there and
28:49
orchestrators that are super hard to replicate because they
28:51
do something very good. Then that's also the reason why,
28:55
for example, we use LLMS that are that we don't make ourselves
29:00
for this, for this agent, because that would be too much
29:04
work. And then for agentic tools as
29:08
well that those are too much work to build the whole OCR
29:11
pipeline. We can just use something that's
29:12
out there. But for this, for these core
29:16
functionalities like an LLM orchestrator in your agent, I
29:19
guess, yeah, it's a little more work maybe, but maybe that's in
29:25
the beginning. But then in the end, if you're
29:26
trying to find that time between chunks, it's way easier if you
29:30
have the code and it's you don't have to fork anything and change
29:33
the library, or if you're implied to monkey patch
29:35
something to to get that that one thing you need.
29:39
Yeah. So I, I see it a little bit as
29:42
you're investing more time upfront, but then you save it on
29:46
the back end. And when you have these edge
29:49
cases, that's where you really see things shine because you can
29:53
go and debug much easier. And we try to figure out what do
29:56
we need. And then it's way more easy to,
30:00
to build something yourself. And then maybe in six months we
30:02
feel like, oh, this, this needs to be expanded because it needs
30:07
to. Let's say we do memory, memory
30:11
on the user level. And 1st we just build it on what
30:15
the user and assistance is saying.
30:18
But then after a couple of months say, OK, it would also be
30:20
nice if it remembers how did tool call.
30:22
So because we're now integrating a lot of tools and maybe ID if I
30:27
say, OK, I want to send something in a Slack channel and
30:30
I'll always call the channel operations.
30:32
But then actual ideas like token operations, it would, it would
30:38
be super nice to that it remembers then that.
30:41
But then if you didn't build that whole memory thing yourself
30:43
to begin with and they don't provide that extra feature, then
30:47
you're going to need to rewrite the whole thing again instead of
30:49
if you have built it yourself, you know what you want, you just
30:52
add it. And of course you can do a
30:54
feature request. But yeah, I just like that if
30:57
something is so core to your product and you want to expand
30:59
the feature, that's easy to do instead of having to hack around
31:03
or change the library or for the last for the newest feature,
31:08
right? The next piece that I think a
31:12
lot of people end up paying for or that there is a lot of
31:16
attention around is the observability piece.
31:20
And so tools like Lang Fuse or I think Lang Smith also does this.
31:27
Obviously insert your favorite word after Lang and there's
31:33
probably a tool that is called that.
31:36
Are you all using a specific observability tool for the
31:41
agents? Not specific for the for the
31:43
LLM. So I think most of those laying
31:47
something are all for agent specific observability.
31:53
But we use Datadog for our observability.
31:56
So I guess those laying products, let's say link tools,
32:03
I guess those link tools do a little more things
32:08
out-of-the-box for, for the for those agents, for the
32:11
observability on the agents. But yeah, we started using
32:15
Datadog and it's also easy just to add the metrics there and the
32:22
logs there that you would probably also get at other
32:25
observability tools. And it also depends a bit
32:29
probably on your back end as well.
32:30
So most of the time with those Lang products, you can also
32:35
rerun if you make a change and to your prompt.
32:39
That's something we have in the back end ourselves.
32:42
That's how it built with event streaming.
32:45
So yeah, I guess, I guess you could use also traditional
32:50
tools, but if you don't have that in place, then it's
32:54
probably nice to do something like that as well.
32:56
And you don't feel like you're losing out by using a Datadog
33:00
which is not specific for it. No, Yeah, maybe a little, I
33:08
guess a little. Of course, when there's a new
33:10
model coming out, it takes a little more effort for us to to
33:13
change it to do to roll out that model and check whether it's
33:18
still performing as well because we don't have those tools.
33:23
But then we save that we don't need to send all the data to
33:26
that tool as well. And then we have a data for the
33:28
overstory. We have a vendor tool that does
33:33
the LLM logs and traceability and then we have data doc, which
33:38
more more does the software engineering side.
33:40
Now it's all just combined, which is also nice to work with
33:44
and. So it does go back to this idea
33:49
of to bring on a new tool is a bit of a journey.
33:58
You need to get it through all of these different things,
34:02
whether it's contract and pricing and legal etcetera,
34:06
etcetera. That when you look at what's out
34:09
there and their capabilities versus what you're using in
34:14
house with a data dog, you say, I think if we squint, we could
34:19
probably make Datadog work up to 9095% of what you get from all
34:24
these other tools and we don't have to go through any of those
34:27
hoops. No, the yeah, yeah, I think so.
34:32
I do, I get, I always catch myself if I knew look at any of
34:37
these new products, I'm super excited about them saying, Oh,
34:39
they can do this and they can do that and it's better than
34:42
Datadog because it can do this and this and I convince everyone
34:45
in the team we should do it. And I make the PR and I we
34:48
tested on staging and then, oh, this.
34:51
And then, and then we find out or either something is missing
34:54
or we could have just done this in data like it's not that
34:57
different or like a more traditional software engineering
35:01
tool, not that it's traditional, but in a sense it's not AI.
35:05
It seems like that's a little bit of a trust issue also,
35:08
right? There's advertisement and what
35:10
they say they can do and then what they actually do, and you
35:14
don't find that out until you test it out yourself.
35:17
Yeah. So have another example for
35:20
that. We were looking into adding a
35:23
provider like a vendor tool, sorry, we were looking into
35:27
adding a vendor tool that could help us roll out tools more
35:31
easily. So we have a bunch of like more
35:33
general system tools, we call them.
35:34
So OCR, it's image generation, all that kind of stuff that you
35:37
expect from an agent. But then we also wanted to do
35:41
Oauth tool. So going to Google on the on the
35:44
with the users Oauth credentials and maybe creating a doc or
35:47
going to get up and list PRS, which is a lot of work because
35:52
you need to go into the documentation of Google, go
35:54
through the API and be creative and think if I have these scopes
35:57
and I have this endpoint, I could probably make a tool
35:59
that's called to create a blank document, which is nice to.
36:05
Fight. Why?
36:06
Why can't this just be done with Oauth?
36:08
Oh, it can be done. We want to do it with O outs,
36:10
but like finding out the writing, the tool descriptions,
36:13
coming up with what a tool should does, it should do.
36:17
It's not like in the Google documentation say O and these
36:20
endpoints are super nice to create an agentic tool for.
36:24
Kind of looks like I'm talking about MCP now.
36:28
Yeah, what a step back. So we were looking to a company
36:32
that would provide all these tools for us that would do the O
36:35
out flow and we could just provide a user ID saying OK,
36:38
this user ID, these arguments and this tool.
36:42
So we found 1A composer and they had a huge list of tools
36:46
available. So I was super confiscated.
36:48
This is it. This is super nice.
36:51
I tested a couple of them, they worked.
36:53
So I made the PR, went staging, we all started testing it.
36:57
And then we noticed that a lot of those weren't working and
37:01
we're just minor, minor things each time that needed to change,
37:04
like, oh, you need to request an extra scope or yeah, there was
37:09
something you needed to add in the dashboard, but you didn't do
37:12
that. But it was also not really well
37:15
documented because they don't. Companies like new new startups
37:19
don't focus that much on documentation.
37:21
Sadly, I wish more did, Yeah. So yeah, that's definitely an
37:25
example of where I got really excited and I was like, oh,
37:27
this, this was a little too, too soon to communicate to the team
37:31
we should do this. Because then the teams looking
37:34
at you like dude, come on. It's more so I'm saying, OK,
37:40
this task is done at the end of the week because and then we
37:43
have 20 tools and then we have 30 tools and then we and I
37:45
tested five of them and then you test the under 15 and then you
37:49
needed extra scopes and getting certain scopes at Google is so
37:52
much work. And yeah, just things you didn't
37:58
expect because it wasn't in the documentation.
38:00
What I do like about the startup so is that they're always very
38:03
much listening to you. So if you're if you will tell
38:05
them something like this, they would probably help you out.
38:09
And they also love the feedback, I guess because they this feels
38:14
like a feature request because yeah, they're just new anyways.
38:16
And the the road map is not that defined yet, but yeah.
38:21
So then you scrapped that one and you decided to just go with
38:25
MCP? No, he didn't.
38:28
So a lot of our users are not working used to working with AI
38:31
or are very technical people in HR are using.
38:35
So this we're we're distributing our agents across the portfolio
38:39
company. So some from someone from HR is
38:43
not going to go online download some MCP hope that it the the
38:49
the, the docker instantly. I don't think even like a
38:52
regular person easily can set that up.
38:55
So we considered it and in the future it might be nice if for
39:00
example, Google themselves are going to offer MCPS and it just
39:03
maybe you go to your Google account and you say you can
39:05
create an MCP link. That would be nice, I guess.
39:08
But if the users need to run the MCPS locally themselves, it's
39:12
going to be too hard for them. So basically you check that one
39:16
off the list. Yeah.
39:18
And then another thing with the startups, they we really tend to
39:24
also look into the legal part and the PII and the privacy
39:29
because we need to be compliant and that kind of things.
39:31
So I'm used to, if I go into Datadog, I won't find PII.
39:34
If there's PII, we get an alert saying, OK, there's, there's
39:37
probably PII edits. We never log PII.
39:42
Sentry, the same thing. If I want to look at user data,
39:45
they need to consent. That's per conversation.
39:49
But then when you start using some of these Fender tools, so
39:51
when I started using composio and some users edit their
39:55
accounts so we could test it out, there were like buttons
39:59
where you could pick one of the tools, for example, search
40:01
documents and could pick one of the user IDs and just execute
40:05
the tool. So I thought so.
40:07
You had guard mode. Yeah, basically.
40:09
So I thought, OK, maybe this is a one off, it's not that bad.
40:12
Maybe there's they're going to add it at my feature later or a
40:15
PRI feature. But then I looked recently into
40:18
Memo, which is like a long term memory service and did a bun
40:25
added that to our system just locally to test it went to their
40:29
dashboard again. And then I saw my whole
40:32
conversation being there like on the front page of the dashboard
40:35
saying like, Oh, this is your recent traffic.
40:37
And then I went to the memory section.
40:39
I saw all the memories, which is.
40:41
It is basically all PII because it's about OK this user likes
40:46
this and this user prefers it if you do that.
40:49
Wow. Yeah.
40:49
So those tools are not really made for like made in a way how
40:55
they respect BII in respect to data.
40:57
There's probably feet, probably buttons and settings out there,
41:00
but it's there was something that I needed to get used to as
41:03
well. It's a little eye opening.
41:04
You weren't. It's like you weren't trying to
41:07
look, but you had to. And.
41:09
Then all of a sudden I could see you looking at your screen and
41:11
then trying to put your head down.
41:14
Just locally me but. I didn't mean to say that.
41:17
I promise it does beg this question of you want this
41:25
capability inherent in the memory capability is you need to
41:29
know things about people. But the way that the company is
41:34
setting it up, they now have to think, oh, if we want to go to
41:41
users with various users, we want to make sure that we're not
41:49
inadvertently making their lives hard with doing things like
41:53
keeping PII or making it front and center.
41:56
So. Prototyping.
41:57
You need to be able to see that quickly, right?
42:00
But yeah, out-of-the-box seems just so different than like the
42:05
more traditional software engineering tools where you
42:07
definitely don't want PII in places like Datadog.
42:10
Yeah, and it's like you would have to add this extra layer of
42:15
getting rid of all this PII. So adding that extra work makes
42:19
you then come back to the question of should we just try
42:22
this ourselves? Yeah, yeah, and the self hosted
42:26
version has done a really good option as well.
42:28
Yeah, it, it feels like you're taking 2 steps forward and then
42:31
one step back or two steps back and one step forward.
42:35
Sometimes with the different tools, because they give you
42:38
certain capabilities, it makes your life easier on one vector.
42:41
But then when you are looking at other vectors, you're saying, Oh
42:45
my God, to actually get this into production and to get rid
42:48
of all this PII and to comply with these norms that we have
42:52
set up, we would have to do so much extra work.
42:56
It's not worth it for us. No.
42:57
And then if, if it's easy enough to do, then you're probably
43:03
going to save some time in the long run if you start from the
43:05
beginning just experimenting and building something like that
43:08
yourself. And do you see this becoming
43:13
more common or is this just a maturity thing?
43:17
No, I don't. I still think these Fender tools
43:20
out there are probably going to stay for a long time because
43:25
there's always people building prototypes and you don't always
43:28
need to be super compliant, you don't always need to be able to
43:32
scale it indefinitely. So I do think these Fender tools
43:37
out there are perfectly fine, just not if you're having agent
43:41
in production and you need to be it to be compliant.
43:44
You need to do scalable ETC. Let's change gears for a second
43:49
and talk about how you're leveraging different coding
43:53
tools. Are you using Windsurf Cursor,
43:56
Devin? A Little Mix of all of them took
43:58
on even. Yeah, we're doing a mix right
44:00
now. So of course we're using our own
44:02
tool. But now since the IDE assistance
44:07
got so good, we also use that quite a quite a lot.
44:10
So I use cursor, we use Devin and we started using get a
44:16
copilot now to do PR reviews. Yeah, for example, the PR
44:20
reviews, it really took away that first step where you may be
44:24
made in spelling agor which didn't get caught by a linter or
44:27
you copy paste it some mongo adopter and you need it another
44:31
mongo adopter. You just copy paste the whole
44:33
thing and then you forgot to change one little name and the
44:37
Co pilots, the gits of Co pilots PR reviewer is pretty good at
44:40
catching that We did try other ones out there as well, but we
44:45
stopped using those because they were super chatty and very
44:49
confident. And now with the GitHub copilot,
44:52
you can it collapses comments that it thinks it's not
44:56
confident that that's the case. So yeah, it really helps with
45:01
like if I made an error, send it off to my colleague and I see oh
45:04
GitHub cope. I already said, OK, this, this,
45:06
I don't think this is right that that that saves a lot of time
45:10
because it saves the time that I asked the colleague.
45:12
The colleague instantly goes back, OK, just the comment
45:14
there. I need to fix it then I need to
45:16
wait again till, till he has time.
45:18
So that's super nice to to have. So these coding tools add to
45:25
this piece that I find fascinating, which is you've
45:30
been building a lot of tools for building agents like the eval
45:37
tool or the orchestration tool. You and do not go out there and
45:41
buy something, but it's not because there's a lack of push
45:46
from the team. I was talking to Paul and he was
45:49
saying that he's really trying to encourage you guys to buy
45:53
stuff. Despite that, you come back and
45:56
you say it's just not there yet and there's this vector.
46:02
Maybe that is worth exploring in your eyes on.
46:06
Before when you thought about building it yourself, it was
46:10
with one scope in mind. But now that you are using these
46:14
coding tools, maybe you're a bit more ambitious on being able to
46:18
build it yourself. Do you look at it in that way
46:21
where you say, yeah, I can probably build this myself with
46:25
the help of cursor and windsurf or whatever in a weekend?
46:30
Definitely, definitely if. If something comes by and I
46:34
think I get, I get how it works. It's so much easier to have
46:41
something like cursor indeed explaining what I think it
46:44
should do and then also me and Cursor trying to find out how it
46:48
works. If cursor wasn't there, it would
46:50
have been so much work to try to to copy something you think is
46:54
worth copying instead of instead of buying like copying the the
46:58
functionality. Yeah, Yeah, I really, yeah.
47:03
I really think that's tools like Cursor enable you to quickly
47:10
build these these versions of existing tools that you think,
47:14
OK, they're liking something like this, we can probably do it
47:17
better. But if, if Cursor wasn't there,
47:19
it's super hard. It's it's it takes way longer to
47:22
experiment for for a very small feature.
47:24
Definitely, Yeah. That's all we've got for today,
47:28
but the good news is there are 10 other episodes in this series
47:33
that I'm doing with process deep diving into how they are
47:38
approaching building AI products.
47:41
You can check it out in the show notes.
47:44
I leave a link and as always see you on the next one.