Latest / Agentic Conversations (formally mlops.community) / Building Coding Agents: Design Decisions, Prompting Tricks, GUI Anti-patterns

Transcript

Building Coding Agents: Design Decisions, Prompting Tricks, GUI Anti-patterns · Agentic Conversations (formally mlops.community)

0:00
So our, our primary objective is to save human time.
0:05
You saw a lot of like interesting demo videos, but I
0:07
would say like nothing that really worked day-to-day.
0:09
The more we played around with tool use models, the more we
0:14
realized, hey, the the assumptions here have changed
0:18
around what the model is capable of.
0:19
You know, users of AMP, which is our our new coding agent
0:23
generating like 80 ninety 95% of of their code.
0:27
We we do spend a lot of time thinking about how to nudge the
0:30
user toward the the right way of doing things.
0:33
So one specific example of this is.
0:39
When we talked last, what right when you released a coding
0:43
agent, I think. That might have been shortly
0:46
before or shortly after we released Cody, which I think was
0:49
yeah. Yeah, it was right around when
0:50
Cody got released. So I think that was maybe like a
0:53
little bit after the initial release of ChatGPT, yeah.
0:58
And so that was at least like one AI era ago.
1:02
That's a good way of putting it. And So what has changed since
1:05
then? Like I would say like at that
1:07
point the the dominant modality of using LMS for coding was
1:11
still this kind of like copilot auto complete mode where you
1:14
just like type a couple of characters of code yourself and
1:17
then you get a line completion. And at that point what we were
1:21
really excited about was the whole like rag model.
1:25
So we figured out pretty early like hey, if you combine search
1:28
and information retrieval and use that to fetch relevant code
1:33
snippets in conjunction with the really high quality chat based
1:36
LLM, it was a really powerful way of of doing code generation
1:40
and technical question. Answers we're uniquely like,
1:43
well suited for that, if I remember correctly, because.
1:45
Yeah. So source graph of course like
1:47
our our first product to market was a code search engine.
1:50
And so it's like, this makes a lot of sense, yeah.
1:52
So like where our bread and butter is helping developers
1:55
read, understand search code. And turns out that that's,
1:58
that's very, very useful still in, in the AI era.
2:02
But the, the big shift since then has been the, the
2:07
emergence, shall we say, of coding agents.
2:11
And really it's, it's the tool use and reasoning models that
2:16
have driven that or enabled that.
2:18
Because a lot of people tried building agents in in the era of
2:22
chat LLMS and you saw a lot of like interesting demo videos,
2:25
but I would say like nothing that really work day-to-day.
2:28
But now we have really good agentic tool use models that are
2:32
enabling A essentially a new application paradigm that we
2:35
call agents that are kind of taking the level of automated
2:41
code generation to the next level.
2:43
It's going from, you know, 30 to 50%, which is what we saw in the
2:47
chat base LLM era to now, you know, users of AMP, which is our
2:53
new coding agent generating like 80, ninety, 95% of of their
2:58
code. Your theory is the IDE
3:01
environment is already dead. Yeah.
3:03
So I think this is like a a general theme, which is the
3:07
application architecture that was made possible by Chat LM.
3:11
So think LMS in the era of GPT 3.5 or 4, this sort of like
3:17
ChatGPT era, there was a very specific type of rag application
3:22
architecture that was ideal for that kind of model.
3:26
And our, our, our coding assistant Cody of all that
3:30
model. So did many of the other tools
3:32
of, of that era. So the AIIDE, the VS Code fork
3:36
was kind of the, the pinnacle of that era, I would say.
3:40
But what has basically happened with this new generation of LLM
3:46
is that they have unlock this new capability tool use plus
3:51
reasoning. So those two together provide
3:55
this kind of like agent A capability and that in turn
3:58
unlocks a new set of interactions at the application
4:01
layer such that a lot of the UX that application builders built
4:06
for the old era of chat LM is now outdated and in fact I would
4:11
say in direct tension with the ideal UX of coding agents.
4:17
All right, this is where it gets spicy.
4:18
So then, what does the new world look like?
4:21
So the new world is much less manual context management and a
4:27
lot less kind of like GUI Chrome and like different toggles.
4:32
So I, I think what you've, what we've seen with the, the chat
4:36
base LLM application architecture, the Ragbot
4:40
application architecture is there's so many toggles now,
4:43
like you got to like manually specify through different like
4:47
rules files and yeah, different ways of like tagging in relevant
4:52
code snippets. There's a lot of UX Chrome
4:55
around managing what goes into the context window so that the
4:59
LM can do a single shot with. Engines with that, just sorry to
5:03
interrupt. The funny thing there is if you
5:07
know certain tricks, it performs better.
5:11
And so there's some people that are doing it really well because
5:14
they've played around with it and they've been able to tune
5:17
the knobs. Yeah.
5:18
And then there's others who are like, yeah, like kind of helps,
5:22
I guess, but it also kind of messes up.
5:24
Yeah, exactly. So there's this kind of like
5:26
strategy for getting the most out of like a a chat based LLM
5:30
tool. And now a lot of that strategy
5:32
is essentially gone out the window because with agents,
5:36
agents have the ability to use tools themselves, fetch context
5:39
themselves. And so it's much less the human
5:42
manually managing what goes into the context window and more you
5:46
describing the higher level of what you're trying to do and
5:48
letting the agent go figure that out.
5:50
The analogy I like to draw is it's roughly similar to the
5:54
transition from like the Yahoo era of the Internet to the
5:57
Google era of the Internet, where like in the Yahoo era,
6:01
it's like what what did, what did what did Yahoo look like?
6:04
It was like 1,000,000 hyperlinks and the, the, the UX was like
6:10
I'm clicking through a nested series of links to find what I'm
6:13
looking for. So there's a lot of pointing and
6:15
clicking and kind of like manual following of links.
6:18
There's a lot of like knowledge that I have built in for, you
6:21
know, what pages are good. When Google came along, they
6:24
essentially took that whole UI and they're like, you don't need
6:27
that anymore. Just type what you're looking
6:29
for and we'll get you to the right thing.
6:31
And I think agents, coding agents specifically are, are
6:34
very similar in, in, in that regard.
6:37
Whereas we're in like a lot of the strategies that people
6:41
develop for getting the most out of chat alms.
6:43
That's like the point in cooking in the Aqua area.
6:45
You no longer need to do that. There is a new skill set that
6:48
you need to learn, which is, but not as trivially as using
6:52
Google. It's it's more like how you
6:53
prompt agents to do their work most effectively without active
6:57
human intervention. But that's sort of a a very
7:00
different skill set, often in direct tension with the skill
7:03
set that people learned in the chat LLM.
7:05
Area with the GUI? Yes, yes.
7:07
Interesting. Because with the GUI leads you
7:10
to do is it really leads you to kind of like micromanage the
7:12
LLM, which was necessary in the old world, but now with agents,
7:16
you kind of want to just give it the appropriate context, the
7:19
appropriate feedback loop and let it run.
7:22
Yeah. And you want to let it do what
7:25
it does and almost give it that freedom that yeah, if you're
7:30
putting too many micromanagement on it, yes, in a way, it's not
7:36
able to get what it wants done because it's feeling restricted.
7:40
Yeah. And the other, the other point
7:43
or the other thing that happens when you micromanage an agent is
7:45
that you as a human also get frustrated.
7:48
So like a common failure mode that we see is people who have
7:51
over indexed on sort of like the the cursor way of doing things.
7:55
They're like, I want to be in there and like I want to direct
7:59
it at every turn because that's what that UI trains you to do.
8:02
It's like at every turn I want to review the change before I
8:05
apply it. But with agents, what you want
8:08
to do is you want to give it enough context to figure out
8:11
itself what the right things to search for, what the right
8:15
feedback loop is. It's similar to like another
8:19
analogy I like to draw is like the the previous generation, it
8:21
was like the AI was, was like a coding student.
8:26
Like you had to be there to review every single little thing
8:30
they did. And they're like, OK, you did
8:32
this right now let's go apply what you did.
8:34
Now it's gotten to the point where it's more like a, a, a
8:38
professional engineer, maybe like junior or mid tier
8:41
engineer, maybe maybe even senior in in some domains, but
8:45
with with an actual professional engineer.
8:47
What you don't want to do is you don't want to babysit them.
8:50
You don't want to be instructing them at every turn.
8:52
Like, OK, now read this file. Now go do this change.
8:56
It's more like, hey, here's the overall context.
8:59
I think you should, you know, use this command to run the
9:02
tests in this case, or use Playwright to take a screenshot
9:06
because you're iterating against UI code.
9:08
Here's the general shape of the feedback loop that you want to
9:10
construct. Now go off and figure it out
9:12
yourself. And it is very much like a
9:16
declarative way of doing things. Yeah, yeah.
9:19
I would say it's less like it's less in the weeds.
9:22
It's more like, let me articulate at a high level the
9:26
key points and I'm going to let you figure out the low level.
9:28
And how have you seen the best folks being effective in getting
9:35
that context that it needs? I would say there's, there's a
9:40
wide spectrum of how far people of able have been able to
9:45
stretch the coding agent so that the top 1%.
9:50
We love those users because in some sense they're kind of like
9:54
discovering the future along with us.
9:56
So when we look at the AMP user base, a lot of the
9:59
forward-looking things that we build are directly targeted at
10:03
emergent behaviors that we observe in the kind of like top
10:06
1% of users. These are people whose token
10:09
consumption is like 10X in some cases, like 100 X what the
10:14
median user is. And there we, we observe a, a
10:18
couple different things. One is there's an emergent set
10:22
of strategies or tips for instructing the agent to get as
10:26
far as possible. This is like, you know, what
10:28
sort of details do I do? I put in the upfront prompt to
10:31
enable it to construct the right feedback loops to, to search for
10:34
contacts in the right places. And then the other thing we, we
10:37
notice is, is more and more parallelization.
10:40
So AMP is available both as an extension inside VS Code as well
10:46
as ACLI. And so there's a lot of people
10:49
who use the editor extension for more complex tasks, tasks, tasks
10:55
that involve more like complex chains where you want to kind of
10:59
like remain in the driver's seat, so to speak.
11:02
And then they'll use the CLI for paralyzing a bunch of shallower
11:05
tasks. So like you'll have a like a
11:07
team UX window where they have, you know, 3 or 4 AMCLI instances
11:11
going on different, you know, shallower issues or bug bug
11:15
fixes. That is so cool dude that is so
11:18
wild to see. It's crazy.
11:20
And how are you? So I guess you're just doing
11:23
product feedback with all of these power users?
11:28
So we've we've hired a good number of them.
11:30
I like to say like smart man, Yeah.
11:32
So it's a great source of, you know, forward thinking devs,
11:38
some of some of whom we've welcomed to the amp port team.
11:41
And we also talked to them a lot.
11:43
You know, we interact over social media channels, we have a
11:46
discord, we hop on phone calls. But it's, it's great to, to talk
11:51
to that set of users because in some sense, like we're so early
11:55
right now, like people talk about a is if it were one
11:59
monolithic block, you know, that has, you know, the wave has, is
12:04
but one giant wave since Chacha BT, but it's actually a
12:08
succession of, of multiple waves, I would say.
12:11
And we are so early on the agentic model era that a lot of
12:17
our product development process is really kind of like
12:21
partnering or sitting down with our our power users and
12:24
discovering alongside of them you know what the possibilities
12:27
are. Is that how AMP came to be?
12:30
Because you saw that folks were clicking around too much and you
12:33
realized maybe this isn't the best UI and UX that we can have.
12:39
That was a big part of what motivated us to to build
12:42
something from from the ground up.
12:44
So, you know, we had Cody, which is an assistant that was really
12:47
good in the chat LLM era. But the more we played around
12:52
with tool use models like, you know, Sonnet 37 and now Claude
12:57
4, the more we realized, hey, the, the assumptions here have
13:03
changed around what the models capable of.
13:05
And if you're, if you're holding it properly, you can actually
13:09
get a lot more out of it than you could in, in the previous
13:13
era. The problem is that a lot of the
13:15
old UI paradigms are kind of like actively working against
13:19
you getting the most out of of coding agents.
13:22
So this is this is something that we kind of realized in
13:26
using it ourselves heavily and also talking with a lot of our
13:28
power users. In fact, one of the one of the
13:30
folks that we hired this guy by the name of Jeff Huntley, he was
13:33
a Canva at the time. He actually wrote a blog post
13:36
about how he thought most people were using AI coding tools
13:40
incorrectly because they were still kind of using it like, you
13:45
know, Google Search Plus plus or in a very like chat based
13:48
paradigm. And we brought him onto the team
13:52
because we're like, this is a guy that gets things.
13:54
Different and. That really understands, hey,
13:57
you should be instructing these things.
13:59
You should be. It's almost like you're
14:01
programming them through natural language, if that makes sense.
14:05
So like you're, you're articulating a very set of,
14:08
you're articulating a set of a very precise instructions in
14:14
much the same way that you would kind of like articulate those
14:16
instructions to a smart, but still, you know, junior
14:20
engineer. So that you're giving a lot of
14:21
context upfront and you're allowing them to get much
14:26
further on their own. And how about the idea of just
14:29
validating when code is working or not?
14:33
The beauty of agents is that they have this kind of like
14:36
built in ability to construct these feedback loops.
14:40
And so when you're using AMP, for instance, when it's
14:44
generating code for you as part of that code generation process,
14:50
it will seek out an appropriate feedback loop.
14:53
So if you're doing front end code, it can use a tool like
14:57
Playwright, for instance, to screenshot the front end of the
15:01
application as it's working. So if you say like, hey, go make
15:03
this background red or green or blue can actually take a
15:07
screenshot and verify whether a change it made to the code had
15:11
the intended effect. Similarly for back end code, it
15:15
might be like a unit test suite or some other command line
15:18
invocation that it it can use to validate.
15:20
Whether it did was whether something it did was correct in
15:25
much the same way that you as a human developer would would seek
15:28
out these feedback loops, right? It's like re evaluate, print
15:30
that sort of like core loop. That's what agents are good at
15:35
figuring out. Some cases they need a little
15:37
nudging, just as you know humans need need a little nudging or or
15:42
some pointers in in some cases. But by and large, if you can get
15:46
the agent to figure out that feedback loop with with very
15:50
high confidence it it will iterate to something that is is
15:56
is mostly correct. Why do you feel like coding
15:59
agents had this breakout success and were uniquely positioned for
16:05
such a lift with LMS and the whole AI revolution?
16:11
I think the, the, the answer to that question comes down to the,
16:16
the immediately preceding question, which is, you know,
16:20
how do you validate something is, is correct?
16:22
And coding is one of those domains where you have a very
16:25
strong validator in the form of a compiler or unit test runner.
16:31
And because you have that validation point, you
16:35
essentially have a way, a very reliable way to generate high
16:40
quality synthetic data. So a model evolution is, is
16:43
ultimately a data game. And there's two ways to acquire
16:46
data. You can either collect it from
16:48
the wild or you can create a synthetic learning environment
16:54
in which you place your kind of like robot or agent in there and
16:58
allowed to do stuff with feedback about what's good and
17:01
what's bad, sort of like reinforcement learning
17:03
environment. And I think at this point we've
17:06
exhausted the amount of, of publicly available large Corp
17:12
corpi of, of, of data. So like you know that that those
17:17
sources of data are are largely played through, but coding is
17:21
one of those domains where it's like you can create a simulation
17:24
environment with unit tests and and like.
17:25
Are you guys doing that? To, to, to a certain extent.
17:29
So for, for certain special use cases, we, we don't do
17:33
foundation model training as of yet.
17:35
But for for certain targeted use cases, we we do that sort of
17:42
kind of like validation and and. Training I've just been hearing
17:45
about how more and more people are doing simulations more it's
17:49
more common to do that just to figure out where you have strong
17:55
capabilities and where you're you maybe are are failing
17:58
silently sometimes even. Yeah, it's, it's essentially
18:01
what you're doing is you're designing a game that
18:04
approximates the what you want in real life.
18:07
So in, in all the domains where AI has gotten really good, you
18:11
know, think about like, you know, playing chess or playing
18:14
some other form of game. It's because you have this
18:16
feedback mechanism that tells you, hey, you're winning or
18:19
you're losing. As long as you have that
18:21
feedback mechanism, you can turn that into a reliable source of
18:24
training data. Because essentially what you,
18:26
what you do is you take your model out of given snapshot and
18:30
then you just run it. You, you, you say go play the
18:32
game, you simulate the game and based on the moves that the
18:36
model takes, you say like, OK, you know, plus points or, or
18:40
minus points. And that's essentially what
18:41
you're doing in these coding reinforcement learning
18:43
environments when you say like O compiler error or O unit test
18:46
failure. I like that way of looking at
18:49
it. You you just around all day and
18:51
you're thinking of a knowledge, thinking of analogies, Huh.
18:56
Yeah. The the thing that I'm also
18:58
wondering is it feels like we had a big jump from these rag
19:03
chat bots and the way that we were copilot writing code.
19:08
Yeah, to then all right, we're in the ID and we are doing this
19:14
almost like click OPS type of stuff and very micromanaging.
19:18
Yeah. Now you're saying we've got a
19:19
whole new era that's being born with AMP and how you're giving
19:24
it this context, as much context as possible and then letting it
19:27
do its thing. Is that the last era or do you
19:31
feel like there's another one that you want to get to?
19:34
It's just not yet possible or it's in the works.
19:38
I don't think we're in the the final era.
19:40
I think things will continue to evolve.
19:43
So you know, one of the things that we're doing now is we're
19:46
thinking about how to combine multiple models effectively in
19:51
this new agentic paradigm. So in the old world, the the
19:56
name of the game was was simple Rag.
19:58
So like every. AI coding assistant had like a
20:01
model selector where you're just like, you know, whatever model
20:03
you want to use, you can use that and then we'll just fetch
20:06
the relevant snippets, put in the context window and generate
20:09
the response. I think in the agentic area, you
20:12
have to be much more thoughtful about the models that you use.
20:14
So you know, we use one model for the core kind of like tool
20:17
use and agentic driving of amp. And we just shipped a feature
20:21
that allows you to use another model, O3 actually for for in
20:25
depth reasoning. Because turns out there's
20:27
certain types of like nuanced problems that you might want to
20:31
tackle that were where these like reasoning heavy models can
20:35
do a lot better than the the models that were kind of like
20:40
trained primarily for agentic tool use.
20:42
So that that's one way in which the the paradigm continues to
20:46
evolve. It's like now moving beyond just
20:48
simple agents to maybe like reasoning agents or agents that
20:52
can use different types of models to do more things in
20:56
depth. And you want to abstract that
20:58
away from the user? Or you want to have it that
21:00
every time I go and I give a task to an agent, I can say
21:05
here's your 3 or 4 models you can choose from.
21:08
You figure it out. I would say if you we want to
21:13
enable kind of like a, a spectrum of use.
21:16
So for the the first time user, you know, you don't have to know
21:20
that we have this. So the tool is called Oracle
21:23
because O3 is, is such a powerful reasoning model.
21:26
It's, it's like talking to an Oracle of sorts.
21:29
And you know, we, we don't want that to be a prerequisite to
21:34
being able to use AMP. So if you don't know about what
21:36
tools AMP has access to, you don't need to.
21:39
It will just select what it's what it thinks is the best tool
21:42
for the job. But at the same time,
21:45
instructing coding agents in our view is a pretty high ceiling
21:49
skill set. So you can get good at coding
21:52
agents in the same way that you get good at your editor of
21:55
choice or you get good at your programming language of choice.
21:58
And for our power users, we do see prompting or, or query
22:02
patterns where they're saying like, OK, I want, I want you to
22:05
use the Oracle in this case because this is a little bit of
22:08
a hairy problem. It's it's more nuanced.
22:10
I want some more in depth thinking.
22:13
So there is some exposure, but it's not at the point where it's
22:16
like, OK, decide what LLM you want to use for this case.
22:20
That is now an implementation detail.
22:21
And I think it's it's almost, you know, it was a best practice
22:24
to expose that to the the end user in the chat LLM era.
22:28
But now I think it's it's a more.
22:30
Not even needed. Yeah, yeah, it's funny.
22:32
Are there any other anti patterns that you're starting to
22:35
see and then maybe surprised you?
22:38
The number one anti pattern is people trying to use coding
22:42
agents in just the same way that they use the the chat based
22:47
coding assistance. And I would say those anti
22:50
patterns largely fall under this umbrella of in the chat based
22:54
world, you wanted like the human had to be in the inner loop of
23:00
like back and forth between you and the model out of necessity,
23:03
right? Because each, you know, model
23:07
invocation was like a roll of the dice.
23:09
And in, in the chat based world, you know, the, the probability
23:13
of it just working was probably lower than 50%.
23:17
Like more likely than not, it would make some subtle bug and
23:20
it had no way of correcting itself because it couldn't
23:22
iterate against feedback, it couldn't use tools and so it
23:25
couldn't fix its own mistakes. And so as a consequence, you
23:29
wanted to be in the loop, so to speak, as a as a human to
23:34
constantly course correct it with agents.
23:38
It now has the ability to gather that feedback on its own.
23:42
And so if you, if you instruct it properly, in many cases, the
23:47
fidelity you get from a single model invocation, like a, a
23:50
single file edit or single, you know, bash command that it runs
23:55
is closer to like ninety, 9599%. So like you can actually, like
24:01
you can get, get out of the way much more if you use it properly
24:05
and, and it can do more for you. But that almost requires almost
24:11
like an active rejection of a lot of the best practices that
24:17
people learned in, in the chat based LM era.
24:20
So it it's almost ironic like some of the people who are
24:23
struggling the most to use coding agents effectively were
24:26
the ones who early adopted chat based coding tools.
24:31
This little microcosm of the macrocosm or that it's a yeah,
24:36
just let go. Yeah.
24:38
Trust the process, man. It's going to work out.
24:40
Yeah. Oh, that's hilarious.
24:43
So I had, I had a question about that.
24:48
I can't remember now. It's not coming to me because I
24:50
was thinking about trusting the process.
24:52
I'm not. Hold on.
24:53
This is the power of post production.
24:54
We can call it the process. Yeah, trust the.
24:56
Let me trust the process of my question asking capabilities.
25:01
So you mentioned the different power users and how they're
25:04
paralyzing different things. And that seems to be 1 very
25:08
advanced way of doing it. I wonder if you've found nice
25:14
tricks other than that, that maybe aren't the power user
25:17
tricks, but it's just in the way that you're prompting or you're
25:21
asking the agent to do things. Like my mind instantly goes back
25:25
to the early days of ChatGPT and we started asking it like, think
25:29
step by step on this. And everybody was like, whoa,
25:32
it's so much better when you do that.
25:34
And have you found any of those almost like prompt tricks?
25:39
Or maybe there's other tricks that aren't even in the prompt
25:43
or in the way that you're asking.
25:45
Yeah. So by and large, and, and the
25:49
best way to to discover these things is really through
25:51
experience. So I'll do my best to like tell
25:54
them to you in the moment. But it's, it's no substitute for
25:57
actually like using it and kind of like building the intuition.
26:01
But in my experience is this kind of like 3 buckets of of
26:05
like prompting tricks. There is what I would call like
26:10
context hints #2 would be feedback loops and #3 is kind of
26:17
like structured approaches slash planning.
26:21
So the, the, the first bucket is really about helping the agent
26:28
figure out what tools it needs to invoke or, or essentially
26:32
where to look for the relevant context.
26:35
So especially in, in large code bases, often times it can be a
26:40
little bit tricky even as a human to find the exact spot
26:44
that's relevant to a particular task.
26:47
And so agents, agents, agentic LMS, the context windows these
26:53
days are much larger than they used to be.
26:56
You know, I think, you know, sonnet 4 has 200,000 tokens
27:01
total. Gemini now has a million.
27:06
And so the, the context windows are larger, but they're still
27:08
finite. And what that means is the, the,
27:14
the more information you give to the model about where to look,
27:17
the fewer tokens it has to expend, finding the sort of like
27:24
general vicinity of, of what's going to be relevant.
27:28
And so the, the more hints you can say like, oh, like look in
27:31
this part of the, the code base, or I think it's under these
27:33
directories, or maybe like use, use this tool to fetch the
27:36
context. That can help a lot.
27:39
The second thing is feedback loop.
27:40
So, you know, I was telling you before about feedback loops and
27:42
how they're critically important.
27:45
These are essentially like, hey, you know, use this tool or use
27:48
this command to validate your approach.
27:53
Often times it will will infer the appropriate tool to use on
27:57
its own. But in in cases where it's, it's
27:59
not a trivial to figure that out.
28:01
Again, you can save on on the main context window by by kind
28:07
of like nudging it in that direction.
28:09
And then the the third approach is just adding structure to the
28:13
overall approach it uses to to solve the problem.
28:17
So in the simplest form, this is just like, hey, before you go do
28:22
this large and complex task, I'm about to give you first right
28:25
out of a plan of steps and, and maybe even let me as a human
28:29
review that set of steps so I can ensure that they're correct.
28:33
And then, and then more and more we've we've sort of like built
28:38
additional features into the product where knowledge of those
28:40
features can help. So one of the things that we've
28:42
shipped, well, I guess not so recently now it was like a month
28:47
and a half ago. So it's like, you know, few
28:49
months ago in in AI land, we shipped sub agents.
28:54
So sub agents are essentially, I mean, as the name suggests,
28:57
they're agents within an agent. So like the the main agent can
29:00
invoke a sub agent to go do a subtask like, you know,
29:04
searching the code base or going and implementing a feature in
29:09
one part of the the code base. And so having knowledge of what
29:14
sub agents are good at and kind of like nudging the main agent
29:18
to use them where appropriate can help you conserve the
29:22
context window. Because the beauty of sub agents
29:24
is once they complete their subtask, they don't essentially
29:30
like the the tokens they use, the context window gets garbage
29:32
collected. They don't use up the context
29:34
window of the main agent. So that allows you to get
29:37
further in complex tasks because you're you're essentially
29:40
chaining together the sub agent calls that don't eat into your
29:44
overall token budget in the main agent.
29:46
That makes sense. Yeah, I've heard that described
29:50
as agents, as tools. A lot of people are.
29:53
That's like the hot buzzword these days.
29:55
It's like, oh, agents as tools, It's coming.
29:57
And. Yeah, everything is a tool, and
29:59
tools are just function calls at at the end of the day.
30:02
Yeah, I was laughing with a friend because I was saying, you
30:04
know, even humans are tools. At the end of the day, when
30:06
you're asking for the human to give you the feedback, it's like
30:11
invoke the human tool. Yes, yes, it's like in is is the
30:15
agent the tool or am I the tool for the agent to get his job
30:18
done? Sometimes the line.
30:19
Blurs a little bit, yeah. Yeah.
30:21
The part that you just broke down really well on how to
30:26
conserve that context window in a way so that you're not using
30:31
it all. A, because it's finite and maybe
30:34
you don't have the ability to throw everything at it, but B,
30:39
it's really good for the cost and keeping the cost lower.
30:43
I can imagine when you're looking at agents and folks that
30:48
are using agents, I think there's probably 2 lenses that
30:53
you look from. You look from like, OK, the
30:55
consumer is trying to keep their costs low, but they're
30:59
interfacing with Source Graph in a way, an amp.
31:01
And so you also have to be weary of costs and passing on the
31:06
right cost to the users and the pricing and all of that fun
31:09
stuff. How are you looking at all of
31:12
these different costs and how do you feel like we've all heard
31:16
this idea that Oh well, LLM calls are just basically going
31:20
to 0, right? And so it's just getting cheaper
31:24
and cheaper. But now if you're talking about
31:26
agents using sub agents that are doing super complex tasks, it
31:30
still could be like 50% or $0.50 for a task to get done and or
31:37
maybe even 5 bucks who knows. So our, our primary objective is
31:42
to save human time because that is still the most valuable
31:47
resource by a huge margin. And so one of the, the core
31:53
principles we, we've adopted is essentially to, to not, to, not
31:59
to not worry too much about keeping the cost of the agent
32:04
super, super cheap. So, you know, agent of coding
32:07
tools, they look expensive relative to chat based coding
32:11
tools like your, your average token spend is, is growing from
32:16
on average 10 to $20 per user per month to, you know, in the
32:21
hundreds in some case, in some cases in the thousands of
32:24
dollars per user per month. Like when, when we talk about
32:27
the top 1% of users who are really like redlining, the what
32:32
the model can do and what, what the coding agent can do, often
32:35
times they're, they're pushing into thousands of dollars per
32:38
month territory. And so that looks expensive to
32:41
people. But if you look at the amount of
32:44
human labor that's being saved, given how productive people are
32:48
with these tools, it's like a no brainer trade off.
32:52
And I, I think a lot of other tools, I, I think they narrowly
32:58
focus on that. They over index on like, Hey,
33:01
how can we keep the cost low as compared to the chat based LLM
33:07
era? And I think that's that's that's
33:09
a very poor trade off to make because it it's kind of like
33:14
saying like, Hey, I have this magic tool that can save you the
33:17
human hours of hours per day. Your time is super valuable.
33:19
Like a human developer time is, is still by far and away like
33:23
one of the most precious resources within your
33:26
engineering org. And even, you know, if you're
33:29
spending like $1000 per month, that comes out to what like, you
33:33
know, 30, forty, $50 per day, Like how, how, how much, how
33:37
many, how many minutes of human developer time does that
33:40
translate to, to saving? And so I think what you're going
33:44
to see is more and more people start to have this realization
33:50
over time that I shouldn't be thinking about the baseline cost
33:54
of this because really the upside is far greater.
33:57
The amount of additional productivity and the amount of
34:00
additional feature velocity that I can unlock in my engineering
34:03
team with coding agents is far greater than the cost that I
34:08
will be paying for them. So like right now you're, you
34:11
still see, you know, CF OS and people in accounting and the
34:14
finance department being like, you know, oh, it's, it's
34:16
difficult to forecast, but I think it's just going to play
34:18
out over time where the companies that trust the
34:21
opinions of their, of their developers and encourage people
34:24
to make the most out of coding agents, They're just going to
34:29
move much quicker. And over time the the market
34:32
will reward the people that prioritize developer, developer
34:37
productivity essentially. Yeah.
34:39
Do you feel like we are not going to be doing much of the OR
34:49
actually? So I play around, I've played
34:53
around with a lot of the different tools and I feel like
34:56
there is something that's happening right now where I no
35:00
longer want to click around and get things done.
35:06
I also don't even want to like code to get things done.
35:09
I just want to say it and then it goes and does it for me.
35:14
That is very easy to do with almost coding tool type things
35:19
like if I say hey, connect my website to this database, great,
35:25
that should be possible. Yeah.
35:28
I don't know it if there is a possibility for the world to
35:34
look like this in the future, but I would love it if I could
35:37
do that with any application. I no longer want to have to
35:43
write out the OR click around to get things done.
35:48
And I almost feel like we are getting spoiled in a way with
35:53
the software being able to do it when you're coding or when
35:57
you're creating your your application with whatever.
36:02
But like, are we going to be able to do that in JIRA and
36:06
Confluence at some point? Are we going to be able to do
36:09
that with just like voice next right, Like I you got to think
36:16
it's going there. So I saw a tweet the other day.
36:19
It was something to the effect of the GUI.
36:23
The graphical user interface was a blip in between command lines
36:28
and agents, which I think, you know, rang very true in, in the
36:35
sense that I, I, I think it's exactly what you said.
36:40
I, I think the, what we think of as a software application today
36:45
is, is going to look much different in a few short years
36:50
from now. So it's not that like, I don't
36:54
think like visual interfaces are going to go away entirely.
36:58
I would, I would say like precisely what I think is going
37:01
to happen is that the primary input modality to computers or
37:07
to to software applications is going to shift from graphical
37:11
modes of input like pointing and clicking to more textual forms
37:14
of input like typing or speaking.
37:16
Now the output, what you get back from the computer might
37:19
still be visual. It's like, you know, I type in
37:21
what I want and then show me the results.
37:23
Maybe like you know. Airbnb listings with photos.
37:27
Yeah, yeah, there's certain things that chat cannot
37:29
describe. Yeah, exactly.
37:31
It it like I don't want to read all that text, just show me like
37:34
a picture. A picture is worth 1000 words.
37:36
But in terms of like how like articulating what I want out of
37:39
the application, I, I, I just think of it in terms of like bit
37:44
rate, right? Like what, what are, what's the
37:46
bit rate of these input modalities?
37:47
Pointing and clicking is very low bit rate.
37:49
It's like couple bits per seconds at most because you have
37:52
to like, you know, takes time for you to drag your mouse
37:55
cursor to the right button. And then you got to, it's like,
37:58
it's a very primitive form of, of communication.
38:01
It's like, you know, monkeys and apes do that, right?
38:04
Humans, we have the, the we have this innovation called language.
38:09
And language is beautiful because it's, it's a relatively
38:13
speaking, a high bit rate form of, of communicating our intent.
38:16
And now computers can actually understand language and also
38:19
translate that language to a series of actions that actually
38:22
perform what you wanted to do. And so to your point, I think a
38:27
lot of the application input experience is going to shift
38:32
toward textual or or voice driven in forms of articulating
38:37
what you as a user want. So what are some gnarly things
38:41
that you encountered while building out amp?
38:43
Maybe on the infrastructure side, maybe not necessarily the
38:48
the users and the evals and all of that stuff, but stuff that
38:52
you as you're building it out, you're like, oh shit, man, Like
38:55
I didn't think it was going to be this hard.
38:58
Yeah, there there's a lot of like trickiness and and nuance
39:01
to designing the, the user experience And I, I think it
39:09
really requires thinking from first principles what you're
39:11
after. So like we've given a lot of
39:13
thought to how to bake in the appropriate feedback loops, help
39:18
it get to the right feedback loops, help it to get out of
39:21
like common failure modes where it kind of like loops and tries
39:25
the same thing over and over again.
39:26
That's a, that's a common issue with, with a lot of agents, how
39:30
to conserve the context window. We, we've given a lot of thought
39:33
to that, where to use the appropriate models, like what
39:37
models are best for, for which tasks.
39:40
But these are very different questions than the questions
39:43
that we traditionally asked around UX design, because
39:46
traditional UX design is very visual.
39:48
It's like, you know, how do I lay out the set of, how do I lay
39:51
out the button panels, so to speak?
39:52
What? Color should the button be?
39:54
Yeah. Exactly.
39:55
Where's people? I remember reading A blog post
39:57
on how people look at web pages. And there's like the F shape of
40:02
the where their eyes go and the attention goes.
40:05
Yep. And so, you know, like AMP
40:08
doesn't have any of that really. Like the the input interface is
40:12
very simple. It's just like text box and
40:14
like, you know, write, write what you want or, or some cases,
40:17
in some cases people like to do the voice mode.
40:19
So they'll use like the like Mac OS like voice input to, to, to
40:24
speak to the, the agent and, and that's the, the primary way of,
40:29
of getting what they want. But it's a very different set of
40:31
questions. So you kind of, you can't rely
40:34
on the rules of thumb that people developed in, in the kind
40:39
of like point and click GUI world.
40:41
You really have to think from the ground up, like, OK, I as a
40:45
user, what do I actually want? How do we want to guide the user
40:49
to this behavior? That's kind of like new in this,
40:51
this new kind of like paradigm that unlocks the capability, but
40:55
still feels familiar enough. Fortunately, developers are are
40:59
accustomed to using command driven interfaces a bit more
41:02
than the average computer user. So so that helps.
41:07
But it we, we do spend a lot of time thinking about how to nudge
41:11
the user toward toward the the right way of doing things.
41:16
So one specific example of this is AMP is maybe one of the only
41:23
I I think we're one of the first agents where typing enter in the
41:30
input box does not submit the query.
41:32
So typing enter just introduces a new line.
41:35
You have to hit command enter to actually submit your request to
41:39
the agent. And the reason we did that was
41:41
it was a subtle nudge to encourage users to create longer
41:45
prompts, because the more information you give the agent,
41:48
the more reliable it becomes, the more it can do for you.
41:52
And so that was like a subtle nudge to users to say like,
41:55
look. Don't stop here.
41:56
Don't stop here. This is not Google search.
41:59
Don't type, you know, 4/4 keywords and then expect it to
42:04
read your mind in, in many cases, getting to what you want,
42:10
especially if what you're trying to do is, is more sort of like
42:12
out of band or or unique, right? You actually have to give it the
42:16
information because it's not a mind reader like the, the
42:18
information has to come from somewhere.
42:19
Either it's baked into the priors it learned during
42:22
training, or it's it's it's going to be embedded in the the
42:26
words and tokens that you give it right now.
42:29
Did you think about adding certain shortcuts or hot keys?
42:35
And I've seen this done where you have the prompt box, but you
42:39
also have little boxes underneath that you can click on
42:43
where it's like, here's some common work flows or some common
42:46
questions, that type of thing. So I think this is one of the
42:50
the things that we did very differently.
42:51
It was sort of like a contrarian take.
42:54
We, we actively didn't want to add additional toggles or
42:58
modalities at the bottom because number one, that's like mental
43:03
overhead and it, it makes it so you have to point and click
43:06
again, which is I, I think, you know, we're now living.
43:08
Trying to get away from yeah. We're trying to get away from
43:10
that like it's the age of agents.
43:12
You should just describe what you want and and be able to get
43:14
what you're looking for. The second thing is that with a
43:17
lot of other applications that do this, the the more toggles
43:23
and switches you add, they essentially exponentially
43:28
complexify the the interface surface area of your
43:32
application. To the point where like if you
43:34
if you introduce like a single binary toggle that you know like
43:37
other other coding tools for for instance, they have like an ask
43:41
mode or an agent mode or like a. Planning mode, yeah.
43:45
Like if you introduce a toggle that has like 3 different modes,
43:48
now you essentially have three different like mini product
43:52
experiences that are all very different from one another.
43:54
If you introduce another toggle again with three modes, now it's
43:57
three by three because you have this like combinatorial.
44:00
Like OK, what if I choose option one from the first toggle,
44:03
option three from the the, the second?
44:05
Now there's like 9 possibilities for what the product experience
44:08
looks like. And it's very hard to maintain a
44:11
high quality product experience if all your users are using
44:15
essentially different products, right?
44:16
Different product experiences because they all have some like
44:19
different configuration that that they're using.
44:22
And how about the times where you want to zoom in on specific
44:26
things? Because I know like if you're,
44:28
if you mess around with lovable, they have the the bullseye and
44:33
you can click on that. And it's this new mode that, all
44:35
right, it's got the specific parts of the web page that I can
44:39
click into and then prompt it so that it changes that.
44:43
Yeah, maybe you thought, well, we don't need this.
44:47
We're just going to have the user put that into the chat box.
44:50
The the way we like to do it. So there's some like capability
44:54
or some behavior that is more specific to a particular use
44:59
case. The way we like to do it is put
45:02
that into a form of a tool that the agent can use and and then
45:08
enable the the agent to invoke that tool in the the right
45:12
situations or enable the the user to nudge the agent to use
45:16
that particular tool in the right situation.
45:18
Like, hey, you know, go use Playwright for iterating on this
45:21
UX. And the reason why we think that
45:23
is better than modalities is that tools, the complexity of
45:28
tools, it's, it's more like linear growth and complexity
45:32
rather than combinatorial growth and complexity.
45:34
So like each additional tool, it's not like, it's not like
45:37
modes where it's like, bless you.
45:41
Each, each additional tool that gets added, it's, it's just
45:45
another tool that that can be used.
45:47
It's like you know O of N in terms of complexity, whereas if
45:50
you added a new modality now it's like you know end by end by
45:55
you know K it's you get this kind of like exponential blow up
45:57
in in terms of. The surface area and so you have
45:59
tools that are standard off the shelf for the users to use, and
46:03
there's a description and the documentation about it.
46:05
Yep. But then users can also bring
46:07
their own tools, I imagine. Yeah.
46:09
So we have three kinds of tools. There's the built in tools that
46:14
are just, you know, as the name suggests, built in, you know,
46:17
basic things like reading and writing files and executing
46:20
batch commands, as well as more advanced built in tools like
46:22
like the Oracle that use different models for, for
46:26
advanced reasoning and, and other use cases.
46:29
There are what we call connections, which are tools
46:33
that call out to 3rd party API. So think about like, you know,
46:36
bringing in your issue tracker or bringing in your
46:39
observability tool, pulling in additional contacts from those
46:43
sources. And then the third type of tool
46:45
is MCP server. So MCP is, you know, by now,
46:49
like everywhere, right? And so the, the beauty of that
46:52
is that you have all these different other tool builders
46:55
out there that have built MCP servers that front their
46:59
application or their service that can then be pulled in to to
47:04
to AMP. How are you thinking about the
47:07
tools that you support and are natively giving to users versus
47:13
the other two? Yeah.
47:16
So I would say all the the tools that are necessary for the
47:20
day-to-day, like core inner loop of software development where
47:25
it's like, you know, you're in the code you're reading and
47:27
understanding the code you're writing code.
47:29
That's our bread and butter. So we like to bake those in as
47:32
as really good first class tooling experiences.
47:37
And then there's also like a second wave of third party tools
47:43
that are just like so common that we also want to make sure
47:45
that those integrate well. And that's why we have these
47:49
like connections, these kind of like first party connections to
47:52
3rd party services. So like bringing in things like
47:55
Linear or GitHub or Sentry, the, the way we bring these tools in
48:02
the, the description of the tool and the set of arguments and,
48:05
and how to invoke them, It's very difficult to abstract
48:09
fully. It's not like you can have the
48:11
same set of tools that work really well in a coding agent as
48:14
works well in, I don't know, like a generic, you know,
48:19
enterprise knowledge retrieval agent or, or whatnot.
48:23
And so we, we try to refine the, the tool definitions for those.
48:27
But then we also recognize that there's going to be a long tail
48:30
of things that people want to integrate, especially because,
48:32
you know, we, we serve a lot of the Fortune 500 and, and there's
48:36
a, you know, each, each Fortune 500 code base is, it's like its
48:41
own special environment with all, it's like different
48:46
internal tools and unique combinations of external tools.
48:51
And so we, we also want to enable our users and customers
48:55
to build tool providers, MCP servers that connect out to the
49:01
unique tooling environment within, within their own
49:04
company, bring that into hand as well.