Latest / Agentic Conversations (formally mlops.community) / Getting Humans Out of the Way: How to Work with Teams of Agents

Transcript

Getting Humans Out of the Way: How to Work with Teams of Agents · Agentic Conversations (formally mlops.community)

0:00
If you didn't, pixel diffs have a thing that finds out the
0:02
actual error and zooms in on that.
0:04
Less pixels for the agents to have to look at and the all like
0:08
tools that create a report automatically that summarize the
0:11
most important thing as going what to pay attention to.
0:13
Yeah, ask the agent to tell you what was hard.
0:15
He'll tell you. Just talking to Erica and she
0:25
was mentioning she downloaded the skills that we created from
0:29
the coding Agents conference transcripts of the live stream
0:33
that I fed in the cloud. And I said go and create skills
0:37
from this, right? And Erica told me.
0:40
So I downloaded those and I have them and they're great.
0:44
I think, I don't know if they're doing anything, but they're not
0:48
making it worse. So that's our glowing
0:52
recommendation from users. That's a pretty low bar.
0:58
Oh, exactly, man. But you had some really cool
1:01
skills that you presented in the Hot take session that we did.
1:05
And I wanted to go over them with you because I'm not sure
1:09
that I'm fully grasping the power of the visual regression
1:16
idea and the screenshots, like it's just screenshots.
1:20
And so I created the skill. I've got Claude running the
1:24
skill. You know when to use it, when
1:26
not to use it, but give me the gist of what it is.
1:31
So I think this one kind of fits into a broader philosophy I've
1:34
got, but you should teach your agents to manage up.
1:37
So if you've got a human who works for you, your important
1:41
skill of them reporting to you is they know what information to
1:44
give you, what questions to give you and how to present their
1:47
work to you. As if you're managing agents
1:51
particularly, there's lots of them and your attention is
1:54
scarce. You don't want to having to
1:56
manually QA the stuff they built.
1:58
Let's say you're doing the old school approach of an agent.
2:02
They write some feature for your app and then you, as he would
2:05
have to like labour through it, clicking through the app, seeing
2:09
if it works. And that's not a good use of
2:12
your time. And you can just ask the agent
2:15
to do that for you. So I've got a skill I like to
2:18
use. Well, I just say that as part of
2:21
your validation process. Do you have to do as part of any
2:24
block of work? You have to create what I call a
2:27
feature walkthrough doc where it takes a sequence of screenshots
2:32
illustrating the new functionality that has been
2:34
built, where each one is cropped to focus on the the interesting
2:38
part. And each one has text describing
2:41
what is going on and why. And then before it presents that
2:44
to you, the a different sub agent itself should also go
2:48
through this walkthrough dock and confirm that all the
2:51
screenshots actually show what they're supposed to show.
2:54
And this does multiple things for you.
2:57
So one thing, if there's another error catching thing that if the
3:01
code didn't actually do the right thing, hopefully that
3:03
second sub agent was caught in and confirmed, no, this didn't
3:06
actually work. The second useful thing is if
3:10
I'm going to look at the what my agent did, I can now very
3:14
quickly scan through this feature doc, see if everything
3:16
looks right, and say yeah, approve, let's merge this for
3:19
me. Oh, because you're looking at
3:20
the screenshots. Yeah, I don't look at the code.
3:24
So when I merge code, most of the time I do not actually try
3:27
the app. I've looked at the walkthrough
3:29
doc and say OK was all the functionality implemented and he
3:33
could maybe have some like malicious misaligned AI which
3:36
fakes the screenshots to pretend to have built the thing goes to
3:39
the. Nana Banana Avi and creates some
3:43
random UI. In theory, that could happen.
3:47
It hasn't happened yet, and so maybe it has happened.
3:50
Maybe there's all sorts of ephemeral features.
3:52
I think my agents are built when they haven't, but I think we
3:55
haven't got that level of deception right now.
3:58
But yeah, I can just scan through this doc quickly as it
4:00
works in. Yay.
4:01
And the third cool thing this does is regression.
4:05
So let's say I'm going to ship a release.
4:06
I want to make sure absolutely nothing is broken.
4:09
I just have it run all of those feature doc scripts again,
4:12
because each of them has a a playwright spec it can run to
4:15
generate it and they confirm all of them.
4:18
Either the screenshots pixel match or the agent can look at
4:22
that what changed and infer that this is indeed a change was
4:26
supposed to happen. And then it can write another
4:28
report for me saying here's all the diffs that you as a human
4:32
should pay attention to because maybe they're interesting.
4:35
Oh, that's why you were calling it the regression test.
4:39
That's, that's the part that I didn't fully wrap my head around
4:43
when you first told me about it. And I appreciate that Like last
4:47
little part and the third cool thing on the first part.
4:50
And I can imagine this happens quite a bit when it doesn't get
4:57
through and it doesn't do what it wants to do, that first agent
5:01
will come up with some kind of error message.
5:05
Does it go and automatically try and update and so it doesn't hit
5:11
the error message? Or do you need that second agent
5:13
to say, hey, here's an error, let's go try to fix this?
5:17
I just tell it keep working on this feature walkthrough till it
5:20
works and it can run the sub agent to verify and come back
5:24
with issues and say build builder.
5:26
Let's say builder screenshot based feature walkthrough
5:30
showing this works. If it doesn't work, keep it
5:32
racing till it does. Yeah, there's so many times I
5:37
like how you, you kind of called me out.
5:40
I feel a little hurt on the old way that we used to test
5:44
features was I would walk through and go and try and do
5:48
it. I've spent so much time QA ING
5:50
things especially like some random little toy app that I
5:54
have and this now is becoming my go to with OK, I don't have to
6:05
be the one who's QA ING it all the time.
6:08
I do think like for example, let me give you my latest toy that
6:12
I'm playing with. I'm trying to create like this
6:15
shorts for podcast creator that uses remotion and it will create
6:21
different animations on top of the shorts with respect to what
6:25
the transcript is and a lot of times I'm hitting a wall because
6:33
there is the wrong dimensions in the short that is created.
6:39
I want it vertical so I can post it on Tiktok or YouTube shorts.
6:43
But I click through and I don't find that out until I've
6:48
uploaded the video that I then go through and it creates the,
6:53
IT grabs the transcript and it will create the animation.
6:59
And then it's not till that last step that I see, oh this isn't
7:03
the right dimensions. You think it could fix that?
7:07
Because I haven't even tried. But this feels like something
7:10
totally fixable by that. Definitely.
7:12
I think one of my broader philosophies right now is that
7:15
the main role of a human when working with a team of agents is
7:20
coming up with the validation processes.
7:22
If you can define what good means and you can work out ways
7:26
of sculpting and defining that, ideally with like agents and
7:29
scripts and constraints and stuff, then a lot of the time
7:33
the agents can then deal with that.
7:36
Another thing I want to throw out, we talked about like the
7:38
old way of working and the weird thing about AI is these models
7:42
are changing so fast that the old way can literally mean a few
7:45
weeks ago. Like the latest anthropic model
7:50
of Oprah's 4.6 whatever only came out in February.
7:54
That really wasn't very long ago.
7:56
And there's new things which are practical to do now that weren't
7:59
before. And so like we're this weird
8:02
world, but there's really no such thing as best practices
8:05
because things haven't really stabilized long enough for
8:09
everyone to work out what the best thing to do is.
8:11
I'm sure there's however good you are at using these agents,
8:15
there's going to be some like believing the obvious better
8:18
thing you can be doing that you haven't realized because it only
8:21
became practical to do it a few weeks ago.
8:24
And it takes longer than that for information to like pervade
8:27
through the network of people. Yeah.
8:29
Well, the the information may be out there, but there's it's
8:34
surrounded by a bunch of noise. And so you're busy trying to
8:38
test the other million things that others are talking about.
8:42
And this was one of the pieces that I was hitting on when we
8:46
were at the coding engines conferences, how we're all now
8:49
our little mini R&D departments. And because of that, you get
8:55
folks like yourself or folks like me and all the other
8:58
hundreds of thousands if not millions of developers out there
9:01
that are testing things. And then some of them are
9:04
talking about what's working. But like you said, what's
9:08
working changes every week. And so then if I'm talking about
9:12
what's working now or this is all of a sudden able to work
9:17
because of the new model drop that needs time to reach me or
9:25
you or and go around in the circles and get passed off two
9:30
different folks. Yeah, the one thing is
9:32
predictable dive is each level where the models get better, you
9:37
can give them moral autonomy and step back more and Mike can
9:42
manage them less. If you think about the
9:43
progression, like at the beginning you had like the
9:46
really early models where all they can redo is autocomplete
9:49
and it was like you could stir a few words or lines of code.
9:53
You press tab because that's as much as they could do without
9:56
needing some guidance. And then you had things where
9:59
you could write in a sidebar in your IDE where they could run
10:03
for a bit, but like half of what they did was wrong with crossing
10:06
guidance was you kind of had to peer program with them.
10:08
So smalls to complete the peer programming.
10:11
And then we got to the stage where they can run for long
10:14
enough chunk of time makes sense to run in a terminal.
10:17
And then you get the point where we're at now where they can run
10:20
for long enough time by themselves.
10:22
You can manage multiple of these at once.
10:25
I think it's only fairly recently made sense to do that.
10:28
And you got the point where they can do their own QA, they can do
10:32
their own documentation, they can do their own validation.
10:34
And you really can each level step up your level at which
10:38
you're managing these from like it's all to complete to peer
10:41
program at a line manager to like manager of a whole team.
10:45
And it's that each time requires like a different philosophy, but
10:48
also each philosophical changes a bit like how you work with a
10:52
human, but like how you work with a human has a greater level
10:55
of organizational abstraction. Yeah, and you don't have to deal
10:59
with the human drama, which is always, it throws a little bit
11:05
of a wrench in things. Yeah, I think it's actually one
11:09
of the main ways that managing agents is different from
11:11
managing humans. So you talk of managing agents
11:14
as being like managing humans, but with humans.
11:17
So these are crew, so you have to be nice to them.
11:20
It's like if you hire a human and they're not actually useful,
11:24
it's really awkward because firing someone is really, really
11:26
mean. Stopping using an agent isn't.
11:29
Or like if you want to have four agents try and do the same thing
11:34
and throw the work for all the except the one did best, that's
11:37
OK with an agent, OK to waste their time with humans, that's
11:41
really, really mean. Or another example, like with
11:44
agents, you can make them do really laborious stuff that
11:47
humans wouldn't like having to do.
11:49
Like they have to document absolutely every file and every
11:52
and in lots of different ways after for extremely strict
11:55
rules, have to have unit test coverage for every line of code,
12:00
have to like, really, really carefully test everything in a
12:02
way that really annoy real humans because it's so much like
12:06
bureaucratic work. But maybe in some future time
12:10
we'll decide. You've got to like, have empathy
12:12
and rights for agents and stuff. But right now you can work them
12:16
pretty hard and make them do things with annoy humans.
12:20
And that distinction between managing humans does make a
12:23
difference in how you make them work.
12:25
Yeah, you can also use all caps lock and no repercussions when
12:30
you talk to the agents, whereas if you use all caps lock with a
12:34
human it may be a little bit sketchy.
12:37
But I have heard, and it don't act totally, that if you were
12:40
too mean to agents, they behave less well, maybe partly because
12:44
again, that's some racing humans.
12:47
I did hear that, yeah. If you, there's like a few hacks
12:52
where if you tell them they're going to get fired, they'll
12:55
perform better because again, it's like you're, you're
13:01
imitating that asshole manager in a way.
13:04
But dude, there is one piece that I, I want to go back to
13:08
because I think you have more to say about it.
13:12
And I don't want to let it slide by without squeezing all the
13:15
juice out of it. And that is on setting up the
13:19
systems to verify. And so you have the QA that
13:24
you've done, but you're thinking about how to verify the code
13:30
that the agent writes at a whole different level.
13:33
What are some other ways that you're doing that?
13:35
Are you writing tests? Are you having the agents write
13:38
the tests? And then you're looking at the
13:39
tests and, like you said, lending every line.
13:42
Like, what are some ways that you're verifying in your systems
13:45
that you're using? Are there a broad philosophy
13:48
here is to get the human out of the loop.
13:51
So any time you find yourself having the micro management
13:54
agent or look at every line of code or you spot something out
13:58
doing wrong, you should think OK, how can I should have a
14:01
verification process that captures all of this?
14:06
Like every time you notice it like doing let's say I have one
14:09
simple thing. I noticed that some of my agents
14:11
were producing like functions were really long and unreadable.
14:15
So I added the lint rule that just like required maximum 50
14:18
lines of a function and that helps a fair amount.
14:21
Not another thing is like I had a thing that was writing
14:25
components. I realised there was things to
14:26
do a lot of like custom micromanagement of styles of
14:31
CNFCFS and stuff in a way it wasn't very readable.
14:33
I had an agent right lint rule for me that enforced that
14:36
starting had to be done with a fairly compact, well defined set
14:39
of components. Actually, an important thing to
14:42
note here is that if there's a human writing customer rules is
14:45
a real pain. But agents are really good at
14:48
writing customer rules. Any time you see a agents
14:51
writing code in the way that you don't like, whether
14:53
stylistically or like too much complexity or whatever, stick in
14:58
a customer rule and you never have to look for that ever
15:00
again. But let's say you have your own
15:03
strict opinions about how you react Coach using user effect.
15:05
I happen to have those kind of opinions.
15:08
Define what those opinions are. Have a lint rule.
15:10
You never have to look for that problem.
15:12
Therefore, again, you can deal with huge numbers of lint rules,
15:15
but you can deal with that very tightly defined coding style.
15:19
Similarly, it's good to make sure that you really are very
15:21
exhausted about unit test coverage.
15:23
And again, agents are really, really good at writing unit
15:26
tests. You can have crazy high levels
15:28
of unit test coverage. You can have like really good
15:32
simulation of the things outside your code that you want to be
15:35
simulated. You can basically arbitrarily
15:39
good about does the code work? Arbitrarily good about is this
15:43
code clean. Other thing I'll touch on in
15:45
that space Rosha also talked about in the coding conference
15:50
is documentation like the way the agents navigate the code
15:55
base is, broadly speaking, semantic search.
15:57
They graph around, they read files, they find related stuff.
16:01
They will do a better job of that if the code is itself well
16:05
documented. So I have a lint rule saying
16:07
that every file has to have a comment at the top saying what
16:12
is this file for? What does it do broadly
16:16
speaking, how does it do it? What are the key design
16:18
decisions it makes in doing that and how is it well here to other
16:22
key files. And I I require a read me for
16:25
every folder which says what is this folder about?
16:29
Philosophically, what are the files in here doing?
16:32
What design decisions are made in the stuff in this folder?
16:35
And then a one line summary of the purpose of every file in
16:37
that folder. And that makes the code way more
16:41
navigable, both for humans and for agents.
16:44
So this is quickly becoming a common design pattern.
16:48
Have you been digging around in any of the companies that are
16:52
recreating the file systems for agents?
16:55
Do you honest? No, I don't have it.
16:57
I'm sure there's important stuff that I'm not on track of.
17:00
Yeah, it's, but it sounds a lot like what you're doing.
17:03
It sounds like, hey, let's rethink and make sure that we're
17:09
not creating our repos for humans anymore, or not only for
17:13
humans, and really think through what's going to give us or give
17:18
the agents the most information and context in the easiest ways
17:25
when they need it. And so I really like this idea
17:27
of of folder for every file and then or.
17:32
Sorry. Read me for every folder.
17:35
A read me for every folder and then at the top of each file
17:40
just some quick context on what this file is.
17:43
Yeah, and critically, this helps humans as well.
17:46
I think that we talk about building stuff for agents.
17:49
A lot of what we can do for agents helps humans too, and is
17:53
often helpful when it helps humans to both because it makes
17:56
it easier for us to know if actually doing a good job, and
17:59
two, because some of you might want humans to work in the code
18:02
base as well. There are some ways it makes
18:04
sense to have a work for agents but not people, particularly in
18:07
terms like the tools you write. You can talk about that as well
18:09
about the importance of writing tools for your agents.
18:12
But I think there's another thing here.
18:16
I think there's kind of like this like made in Japan analogy
18:18
here, where but if you write code badly with agents or with
18:22
previous generations of agents, the code can be kind of shitty.
18:26
So people got used to the idea that agent written code is
18:28
shitty code. And I think that agent written
18:30
codes did well is often better than human written code.
18:35
And the reason is that you can force them to do the effort of
18:37
being super strict about documentation, about following
18:40
guidelines, about testing, about verifying as the code can just
18:44
be like really, really forced to be structured.
18:47
Similarly, along those lines, when humans write code, things
18:51
get hairy over time, can make start announcing really, really
18:54
elegant and extensible. But The thing is, when you make
18:57
a code base, it's really, really easy to extend.
19:00
People do. And when they extend it often
19:02
then becomes kind of disgusting. And we know the solution with
19:06
humans. The solution with humans is once
19:08
your code becomes disgusting, you refactor it.
19:11
Refactoring is a lot of work and humans don't like doing work and
19:15
work is expensive. So in practice, you usually
19:17
don't refactor it and you have a code base that once a while a
19:21
few years ago was beautiful and now it's horrendous because it's
19:25
too much affetry factor. But the cool thing about agents
19:28
is you can waste their time. And one of the ways you can
19:31
waste their time is by constantly refactoring stuff.
19:33
And so you can just like, oh, that was the wrong design.
19:35
That's refactored to a completely different way.
19:38
And you make sure it still works because you've got the
19:40
verification flows. And that means, I think, that if
19:44
you manage things well, you can potentially have code that stays
19:47
nice a lot longer. This was a good segue into
19:52
paralyzing agents and talking about how if you're constantly
19:57
refactoring, or if you are giving different agents the same
20:02
goal, but many different agents get to fire off different ways
20:08
of achieving that goal. And you're using your
20:12
verification process to see what the best route to achieve that
20:17
goal is. And then maybe you're also
20:20
iterating over that one that you choose, like.
20:23
It feels to me like you've got some secrets in how to do that
20:30
then I want to know about. I've heard stories but I have
20:34
not been able to fully realize this.
20:39
Partly because I'm probably not using Brumi which you've created
20:43
specifically for something like this, right?
20:45
But where you can fire off 5 agents to complete the same task
20:51
and those agents will all do it different ways and then
20:55
whichever one pass the most tests or is verified as the best
21:00
option is the one that you go with.
21:03
So for the multiple variant thing, what I do is fairly low
21:06
tech. I'll just say guess again.
21:09
So Brew makes it really quick to spell, spell out agents.
21:11
I can talk more about that separately.
21:13
So you can very, very quickly, like a few keystrokes you spun
21:16
up on agents with its own work tree doing its own thing.
21:19
I'll just like describe 5 different angles in the task and
21:22
say run with it. And at the end I look at their
21:24
verification results and what they've done and pick one.
21:29
Well, they if some of them get stuck, someone seems to do
21:31
something good when when they describe their design decision
21:35
at the end, some just seem nicer.
21:37
So that that one's not particularly hard.
21:39
I think the more self there, I think people often get stuck
21:42
with working power agents is the the merged conflict stuff.
21:45
Because if you've got a lot of agents running at once,
21:47
particularly if some are doing refactorings, you end up with a
21:50
shit ton of conflicts. And I think when the humans deal
21:54
with merge conflicts, this is considered like a painful,
21:56
annoying thing because humans having to like thought how to
21:59
resolve stuff is like a huge load of work.
22:03
But I kind of feel with agents that merging is mostly a solved
22:06
problem because even this writer skill saying, Hey, agents put in
22:12
the latest from Maine, look at every PR that you're merging
22:17
with, understand what they're philosophically doing, and then
22:20
make a decision about how to resolve every conflict.
22:23
Both the syntactic ones where get detected there was a
22:27
conflict and the semantic ones where this is changing the way
22:30
to think about this thing and the way that effects how we
22:32
think about what you've done. And broadly speaking, that works
22:35
really well. And again, if you're going to
22:38
get human to do that, there'll be a ton of work.
22:40
You've got to read every PR you're merging with, but agents
22:44
are great at doing a ton of work, so I feel like merging is
22:48
just a solve problem now that really changes the extent to
22:50
which you can go in parallel. But this is when you are sending
22:57
the agent like you're telling the agent from the get go.
23:01
Once you have something, go ahead and merge it to main or
23:05
that's an extra step that you as the human have to drop in and be
23:08
like Oh yeah and by the way, if you hit any merge conflicts,
23:12
make sure to do XYZ. So what I do in brewery is
23:16
you've got this idea of customizable commands.
23:19
So on the left hand, see you've got the left far left hand side,
23:22
you've got your the set of sessions that are running.
23:25
These are agent sessions, each of its own what tree.
23:29
You've got your agent panel where you can see your agents
23:31
doing some stuff. You've got your file panel where
23:33
you can see the code. It's actually the same editor as
23:37
Monaco, same editor as PS codes. It's very familiar if you use to
23:40
that. And you've got your like panel
23:42
in the middle, which is your source code control stuff, which
23:45
could also be do like code reviews and stuff.
23:48
And in these panels here, there's a set of customizable
23:52
buttons you can just Click to make it do a standard thing.
23:55
So you can do. So you can do like, OK, write a
23:58
report of what you've done or it looks like you're stuck.
24:01
Explain what's going on here or. Oh my God, that's.
24:06
Put in the latest put in the latest from Maine merge it make
24:10
all decisions Is anything anything not obvious?
24:13
Write me up a quick report, say what the options I'd ask and ask
24:16
me another one, which is create PR.
24:19
But what it does is it like commits everything pulls an Acer
24:22
from Maine makes good decisions on how to merge does does the
24:25
screenshot walkthrough writes who will up creates APR send it
24:28
to me to review. And that's the whole other
24:30
stuff, all the one go and I these are all customizable.
24:34
It's just dot brew me slash commands dot Jason.
24:36
So you can totally customize what these are.
24:39
And in a sense, this feels like a minor feature because you
24:41
could literally just type these same things in it's like slash
24:44
this skill, whatever. But the fact it's a single click
24:48
does make it a lot quicker. Huge.
24:51
Especially if you're going to do that various times.
24:53
You know that every time you're going to pull from main you want
24:56
to do XYZ and it's a common workflow.
24:59
It's so much easier to have it as a click as opposed to even
25:03
like if it's a slash command like, I would much rather click
25:06
once than have to start typing something.
25:09
Yeah, particularly managing a lot of these, a lot of these
25:11
sessions, like if you've got like 10 or 20 of these running
25:14
at once, you really want to drop in and click move back.
25:18
So, OK, that guy's been spinning it for a while.
25:19
Let's ask him to tell him what's going on.
25:21
That that one is wrapped up. Yeah, that report seems good.
25:24
OK, merge that one. I've got another button like.
25:27
If you've assumed the PR is good, you could go through
25:30
GitHub and say OK, squash and merge.
25:33
What that doesn't do is check that all the actual mergers are
25:36
safe. So I instead like to my own
25:38
merge action where it itself pulls pulls the mergers in.
25:41
Make sure there's no semantic conflicts revental validation,
25:44
which is safer than just doing a squash and merge from GitHub.
25:48
So how are you actually running 10 or 15 agents in parallel?
25:55
I have a hard time even running like 3.
25:58
So the party for having a very, very beefy laptop?
26:02
Yeah. And partly step one have a very,
26:05
very beefy laptop. Step 2 is by being thoughtful
26:10
about when different things are run like you.
26:13
If you have very expensive end to end tests and where you don't
26:16
run them too often, you can blow up your laptop really quick.
26:19
So it's worth being thoughtful about how to tell agents what to
26:23
run when. Also partly like if you've it's
26:28
going to depend a lot on the code base and how expensive
26:30
their tests and things are to run.
26:33
And so I plan to add support in Broomie in somewhat near term to
26:36
allow run your containers in the cloud model is locally.
26:40
So right now Broomie has support for you.
26:43
Either run totally locally or a dev container.
26:46
But of course the dev containers add a bit of extra of heads.
26:48
Actually I don't usually use dev containers because I find if I
26:51
got too many sessions going at once it gets too overwhelming.
26:54
I think pro the answer long term is a combination of in the cloud
26:58
and being thoughtful about how to run stuff cheaply.
27:02
And what does this look like? Can you walk me through the last
27:05
feature or features that you created or your last coding
27:08
session, right? Like what did you do?
27:12
Well, I can talk about how I broadly work when I'm working on
27:14
breweries. When I'm working on brewing,
27:15
what I'll usually do is I'll work in my day job, use brewery
27:19
for my work. I'll have a dock where I record
27:22
everything that annoyed me or any feature I'd like it to have.
27:25
At the end of the day, I just like open a ton of parallel
27:29
sessions, one for each of those things.
27:30
And just like drop in what the idea was.
27:33
I say go build this for me. Then they all go and build their
27:36
stuff and all right, have a feature walk through a
27:38
screenshot, show what they've done.
27:40
It's all individual features. It's not like, and that's, I
27:44
think with the, the key is you have to have a sprawl of feature
27:49
requests right there. It's not like you're saying,
27:52
hey, one feature that is really deep, it needs a lot of work,
27:59
and you're having paralyzed agents working on that one
28:02
feature in different ways. So, so there's there's two
28:06
things here. So sometimes when I give, if
28:08
it's a reasonably large feature, the agent itself might spell out
28:12
some agents like Claude Code is pretty good, like doing some
28:14
degree of orchestration, spending out sub agents.
28:17
And so that kind of like break a big thing into smaller things.
28:21
Right now I mostly just leave it to like Claude Code and that
28:24
kind of stuff. Actually there is importance
28:27
segue subways can go to a quick sidetrack.
28:30
I think there's kind of two orthogonal topics in play here.
28:34
1 is orchestration, which is like, if you want to do some big
28:38
complex thing, how do you break it out into different agents and
28:42
different things either running in parallel or in sequence or
28:47
different. Like sub agents have their own
28:49
contexts. They're not like biased by each
28:51
other's assumptions and that kind of stuff.
28:53
And the second thing is observability and
28:55
controllability, which is if I've got multiple agents
28:58
running, how to look at what they're doing.
29:01
And for the orchestration, orchestration can be done by
29:06
agents themselves. It can be done by structured
29:09
code things. It can be done by humans like it
29:12
can be like I just ask called codes do a thing and it spins
29:15
out some sub agents. It can be semi structure.
29:18
I've got a skill that Claude code uses and it says how you
29:22
should spit out things in the sub agents.
29:24
It can be a purely structured thing.
29:25
Some people have these workflows where a very structured way to
29:29
break into sub agents in some structured way.
29:31
Or if it's a human, I see this thing.
29:33
I realize, OK, I as a human think you've got to build this
29:36
system here, got to do this refactoring here, the Buddhist
29:39
UI thing here. And broadly Speaking of the more
29:43
big and gnarly a thing, the more like it as a human has to do
29:46
orchestration and the more predictable way of breaking down
29:50
is the more like it is you can do it with a skill or structured
29:53
thing, but that's kind of orthogonal in some ways to how a
29:57
human observes it. Like Claude code has this new
30:00
feature of agent teams where it spins out these agents running
30:04
at the same time communicating, which are the in parallel to
30:07
message boxes. And right now Rumi can't see
30:11
those agents in the agent teams, but I'd like it to when I've got
30:15
that, they'll have a situation where you've got parallel agents
30:18
that a human can see. In Brumi, the were actually spun
30:21
off by an agent. I can imagine the longer term
30:25
you can have this mix of parallel sessions kicked off by
30:29
humans versus kicked off by agents, with both of those
30:34
things being a things where a human can jump in and see what
30:36
they're doing, spot things going sideways, ask for a status
30:39
report in a kind of managing by walking around kind of way.
30:44
And these are somewhat orthogonal dimensions.
30:47
Yeah, yeah. I like this spectrum of how
30:52
parallel agents or sub agents are kicked off and if it's a
30:56
human doing it or if it's a skill that's doing it, or if
30:59
it's just like Claude itself knows to do it, that's one
31:02
thing. And it doesn't really matter how
31:05
you get there as long as you get there and you get and accomplish
31:08
what you need. What you're kind of talking
31:11
about back to when you're coding with Brumi is all right, I've
31:15
got 5 features that I want to get done.
31:19
And so I kick off all these 5 features and now I've got 5
31:22
different agents that are going about coding these features and
31:26
they're looping back and forth. And then you go grab coffee and
31:31
you come back and you're looking at, did any of them get stuck or
31:34
do I need to merge domain? All of those quick access
31:37
buttons that you were mentioning before you kind of have there at
31:41
your fingertips. Yes.
31:43
I'm just kind of like the consumer saying, OK, yeah, that
31:45
looks good. Based on that report, you can
31:48
merge that one. Then the other ones, they
31:51
typically there'll be some conflicts because usually the
31:52
features on throwing up aren't entirely separate.
31:56
And then they would just like automatically like see as a
31:59
change, pull that in, refactor stuff to work with it.
32:02
It's actually a pretty fair amount of the work my agents are
32:05
doing is resolving conflicts from the other stuff I asked
32:07
agents to do. Don't you think it would just be
32:11
easier if it wasn't parallel? Would there be less work if you
32:17
did it synchronously? There will be less total work,
32:20
but there will also be a longer time for start to finish.
32:23
Like a lot of the probably most of the work is separate.
32:26
Most of the features are separate, but there's going to
32:30
be some stuff which overlaps. And so like, if you aim is to
32:34
get stuff shipped as quickly as possible and make best use of
32:38
human attention, do stuff in parallel.
32:40
If your aim is to absolutely minimize agent work and human
32:44
attention is less scarce and time is less scarce, then do
32:47
stuff in series. But I think usually the
32:49
situation we're in is that human attention is scarce time you
32:53
want to get stuff done quickly and the agents attention is,
32:58
although I wouldn't say super cheap, like Claude isn't the
33:00
world's cheapest thing. Cheap enough is not a thing to
33:03
prioritize. Yeah.
33:05
And how are you recognizing, if at all, when an agent is going
33:12
off track? Because I'm sure that or you did
33:17
not properly describe what you wanted.
33:20
So then the agent is not able to execute on your vision.
33:25
Again, a lot of that comes back to the idea of teaching your
33:28
agents to manage up. Like teach them to, once they've
33:31
thought through their things, be clear what their key design
33:36
decisions are, ask you the right key question.
33:38
But one thing I do is I actually don't generally use Claude's
33:43
built in plan mode. I have it create a planned MD
33:45
file and ask me the core top level questions.
33:49
And if it does a good job of asking me the top initial
33:53
questions, it usually won't have gone sideways.
33:56
And also often I won't actually read the plan because the plans
34:00
are too long and my attention is scarce.
34:02
But I will read the questions it asks me and I'll either say,
34:06
yeah, your assumption is correct or all these or Steve this do
34:09
you that's do you that and typically ask like 3 to 5
34:12
questions. And if you do a good job of
34:15
teaching how to ask you the right questions and a good deal
34:18
with asking those questions, you probably don't have to read the
34:21
plan. It's still good to have the plan
34:23
written up because one, it helps the system and its own context
34:27
because that plan in the thing gets paid out of context.
34:31
It can always just go back to the plan and look at it.
34:33
Another little detail thing is that sometimes with parallel
34:37
agents, it's good to have them look at each other's plans
34:39
because if you've got 2 agents running in parallel and this is
34:42
going to do things going to affect that one.
34:44
Sometimes it's good to know what's going to land from the
34:46
other one. And if they're not in a
34:47
container, they can actually look in the and the subtrees
34:51
like, and those work trees just like power folders or siblings.
34:56
Like you can literally tell us look at look at this sibling
34:59
work trees plan and see what it's up to.
35:01
And not only is possible when you have a planned document.
35:06
Yeah, I personally think it's good to have the planned
35:09
documents in the repo. Yeah, yeah.
35:13
I like that design pattern. I remember you mentioning that
35:15
and I totally have not been using it, but you told me at the
35:20
conference and now I'm like, oh shoot, why haven't I been doing
35:25
that? That is a very good way of doing
35:29
it. And so there's a few other
35:32
angles that I want to go here before we hit on tool creating
35:36
tools and tool use and all that fun stuff.
35:39
And it is the idea of containerizing and also the
35:44
cheap, first expensive and how you're identifying what's cheap,
35:48
what's expensive. Like how do you know that this
35:54
is going to be a bit more time consuming or expensive if you're
35:59
not knee deep in the code per SE?
36:03
Seclude itself does most of its work in the cloud on their
36:07
servers. The main thing it tends to bog
36:11
down your machine is when it's running unit tests, when it's
36:15
spinning up local servers, when it's running to end to end
36:20
tests, and that kind of stuff. And depending on how heavyweight
36:24
the software you're testing is, it might be that spinning up 10
36:27
of your local server at once is going to really slow things
36:29
down. And if you really slow down all
36:31
your parallel agents, you lose some of the advantages of hang
36:34
power agents in the 1st place because you want to like the
36:38
only machines that get bogged down so slow.
36:41
So ideally you want to like wasting time on cloud servers,
36:43
not wasting time on yours. Or at least The thing is running
36:46
if some multi threaded thing or running it four times at once is
36:49
fine. And so part of that is just
36:51
about writing the skills. So it doesn't like running E to
36:54
E tests excessively. Like don't run the E2E tests
36:58
until you've already passed your unit test, you've already passed
37:02
your lints and your other checks.
37:05
Another really simple thing is called off as this really bad
37:09
habit of like running your E2E tests multiple times just to
37:12
grab them for one line, which obviously did not want to do.
37:16
So the simple things like tell it not to do that and for like a
37:20
broader principle just like spotting ways in which your
37:23
agents are wasting resources on your machine and tell them not
37:27
to do that, but you're spotting. That how is it through the
37:32
observation when the parallel agents are running and you're
37:34
able to see in Broomie like, oh, it's running another unit test,
37:39
maybe we don't need that. A lot of this, just the general
37:42
practice and management were walking around so that you
37:44
shouldn't be like micromanaging your agents is good to jump
37:48
around through them and seeing what broad patterns of things
37:51
they're doing. And then rather than micro
37:53
managing, just that agent update, update your guidance and
37:57
your tools and your linting rules to stop them doing things
38:00
they shouldn't be doing, whether that's running excessive
38:03
heavyweight things or writing code in bad ways.
38:07
And that also occurs to some extent like going down bad paths
38:10
and wasting tokens. Like often an agent will do
38:13
something manually where it could instead write a tool to do
38:17
easier. One example I've got that is I
38:20
was having agents write a lot of UI components, the mats and
38:25
Figma designs. And if you leave the agent
38:29
through entirely by its own devices, it's going to do a lot
38:32
of like heavy work looking in detail at the PNGS from the
38:36
Figma and the and the MCP rendered versions of its own
38:40
components. Really try to understand why
38:41
they're different. It takes ages.
38:43
So instead I had my agents write a tool I call Diagnose where it
38:47
looks at the the component it's got and it looks at the the
38:52
Figma stuff and it says exactly instructed stuff.
38:56
Here's what's different, here's what to fix.
38:59
And then that allows the agents to do much quicker work.
39:01
It's kind of like when you work when you've got humans, humans
39:04
will do much better work if they've got the right tools.
39:07
Agents will do much better work if they've got the right tools,
39:09
but you can have agents write those tools.
39:12
So part of the art of having a good code base is also about
39:15
having the right tools that make all the work really easy.
39:20
OK. And by tools, you're talking
39:22
beyond skills, you're talking any kind of tool.
39:27
I'm talking a variety of things and a lot of this is just
39:31
scripts. Like there's common things that
39:34
your agents need to do and it's not the most efficient way of
39:38
doing it manually. Have a script with it all app it
39:41
can run that does that bit of work and have a skill telling
39:44
how to use it. Just like you give a human a
39:45
calculator, or you give a human a compiler or a type checker,
39:49
work out the common things agents need to do and say, is
39:53
there a way you could put this in a structured script which
39:56
just does a lot of the boring work in one go, and then tell
40:01
all the agents that can use that thing?
40:05
And you're finding that out again by just that whole walking
40:08
around premise. Yeah, manage it.
40:11
Or another thing I do is I ask the agents to tell me what was
40:13
hard. So I've got this standard
40:16
action. I get things to do like, OK,
40:17
you've been doing working for a while.
40:20
What are you spending your time doing?
40:21
What is hard? And the agent will tell you what
40:23
is hard. If I tell you something is hard,
40:26
think as a human, is there a way this can be made not hard?
40:30
And often the answer is what? A tool that automates a lot of
40:33
the hard thing. Because just like with humans,
40:36
there's some things that are better accomplished by thinking
40:40
of a human like agent. Lightweight agents really are
40:42
like humans in the way I think. And there's some things like
40:45
just give them a tool by the script, give a command, and
40:49
it'll do most of the simple things like if you didn't pixel
40:52
diffs have a thing that finds out the actual error and zooms
40:54
in on that less pixel for the agent to have to look at.
40:59
And the all like tools that create reports automatically
41:01
that summarize the most important things going on to pay
41:03
attention to. Yeah, ask the agent to tell you
41:06
what was hard. It'll tell you.
41:09
Wow, I hadn't even thought about asking it or what have you been
41:14
spending your time on and then being able to recognize, OK,
41:23
it's been spending its time on this, we should probably go in
41:26
there and fine tune that. That's like the new technical
41:31
debt is what have you been spending your time on?
41:34
That's how you're you're getting rid of tech debt by finding what
41:37
is hard for the agent. Yeah, and some of it, sometimes
41:40
it's a script, sometimes it's a skill.
41:42
Some of us need to refactoring. Sometimes your code base is just
41:44
hard to work on in this way. And let your agent does not mean
41:48
you've got humans. Let your agents let humans tell
41:50
you what's hard of you. You do a one-on-one meeting with
41:54
one of your reports, ask them what's what they're enjoying or
41:57
not enjoying, what's easy, what's hard, and then use that
42:01
to work out how to make things better.
42:03
I feel a lot of the role of a human now with agents is this
42:06
kind of work. It's the validation, it's the
42:08
tools, it's the the the refactoring that you need and
42:12
knowing how to make life easy if you're agents.
42:15
Yeah, that is so wild. That is so fascinating to think
42:24
about you. You kind of blew my mind right
42:27
there. I had never thought to ask that
42:29
question, and I appreciate this conversation so much because of
42:33
it. Oh, man.
42:35
Well, what else? What else do you got for me?
42:37
Is there? I feel like.
42:40
I don't even know what question to ask you now because shit,
42:43
man, that fuck. Like, how do we even get there?
42:47
That's so cool. Let's talk more about this idea
42:50
of what we should be doing as we're managing agents.
42:55
And you broke down validation. You broke down like helping
43:00
optimize the agent workflows. What other things are you
43:05
looking at and thinking? My job is completely changing
43:10
now because I'm going from writing the code to now being
43:15
this orchestrator or debugging, helping the agents debug
43:18
themselves or giving them better tools to be able to fully take
43:23
advantage of these sessions. Like what else does it look like
43:27
now? I think this is one of these
43:30
things where the world is changing so fast, it's hard to
43:32
predict what the role of humans will be in the future.
43:35
I think the role of humans is to do the bits that the agents
43:38
don't know how to do. And that's not stable.
43:43
Like every few weeks a new molar comes out and that thing that a
43:46
human had to do, now they don't. So I think the right now what
43:50
humans have to do is kind of sunsets the taste makers of say
43:55
what it is we're trying to build, what problem we're trying
43:57
to solve. If to the extent the humans
44:01
still need to look at code and right now they still somewhat
44:03
do, what does it mean to have a clean code base?
44:06
What does it mean for this to be human intelligible to me?
44:10
And to the extent and right now humans still need to like
44:13
unblock agents and work out ways that tool needs to get better or
44:17
refactors are needed, etcetera. I don't know whether that's a
44:20
long term problem with the orchestration.
44:22
I can imagine the future of the agents themselves as they get to
44:25
high levels of orchestration. You have your master
44:28
orchestration agent realize cases where the lower level
44:31
agents need help or guidance etcetera and that will get I
44:35
think agents do as well. I think this is broader thing
44:38
that goes beyond code of what is going to be the purpose of
44:42
humans in society. As agents get smarter and
44:46
smarter and smarter and get better at humans, a larger
44:50
number of things. And it's like this thing where
44:54
we keep defining real intelligence as being the things
44:57
that computers can't yet do, and that gets smaller and smaller.
45:01
But I remember the day when people thought, oh, chess is a
45:04
marker of real intelligence. Computers will never get good at
45:07
that. Or if they can, that's beyond
45:09
what we can imagine. Then they got really good at
45:11
that. Or people thought like, great
45:13
art was really the mark of real, real brain.
45:15
Now, great art, it was the better of that.
45:18
And then everyone said the way to defend your career long term
45:22
has learned to code because that's really, really hard.
45:24
And it's a matter of that humans shouldn't code anymore.
45:27
So I'm really kind of loathe to give strong pronouncements about
45:30
what humans will do, because by the time people watch our
45:33
podcast, that might not be true anymore.
45:37
So you took the diplomatic route.
45:38
You covered your ass. I appreciate that, but there.
45:42
Is say what we do today if I don't know, like it's hard to
45:45
know what it means for a human to be defensible because it's
45:48
hard, so hard to predict. Yeah, and maybe not
45:52
forward-looking, but just like you as your day today, what you
45:58
notice your spell self spending more time on you mentioned a few
46:01
of these things, right? You're now just observing a lot
46:05
of agents and trying to help them debug and you're thinking
46:11
of what you want next. I, I had kind of pondered this
46:16
too, whereas the best skill that I can have, which sadly I don't
46:22
know how good I'm doing on this, but it is something that I am
46:26
going to work towards is being able to articulate what I want.
46:30
That is, like, for me, a very, very powerful trait right now,
46:38
yes. Yeah, I think the main for the
46:41
most invaluable thing for humans now is to know what problems
46:44
need solving. Was actually remarkably
46:47
difficult scale. Like the world is inherently
46:51
imperfect. There's always things that could
46:54
be better. And the knack is to spot two
46:57
things. 1 is what are their ways in which the world isn't
47:01
perfect, and two is which of those things actually solvable.
47:05
And what is solvable change as frequently as new technologies
47:09
come out, as new things change as assumptions change.
47:12
And so the people who have the big impact are often the people
47:16
who are aware of both ends. They're very sensitive to ways
47:19
what isn't quite right, but also very sensitive to ways in which
47:22
the landscape, what is possible is shifting.
47:25
Because there's the whole analogy that no one leaves $100
47:28
bank note on the floor in this. It's only just appeared there.
47:31
And so if a problem was being solvable for a long time, it's
47:34
probably already got solved. Where things are interesting is
47:37
for the problem only just became solvable.
47:40
And so knowing the shifts in what has become a solvable and
47:43
wasn't, and that the shifts than the landscape of what what you
47:48
might do is really like potent superpowers, you've got it.
47:54
Dude, I'm fired up. Like you just got me ready.
47:57
I'm about to go get some Broomie Parallel Agent sessions going
48:01
on. For anybody out there listening,
48:04
I highly recommend you give Broomie a test drive.
48:08
It's super cool that you're building it.
48:11
It is open source, right? So anybody?
48:14
In this source works at all CLI agents.
48:17
You're not locked in, just one of them.
48:19
Yeah, which is another angle that are you using potpourri of
48:25
different agents or are you sticking to like is Claude your
48:28
daily driver? Claude's my daily driver.
48:30
I use codecs where I run out of Claude credits.
48:34
I try Gemeli every now and then to see has it got good yet and
48:38
then say no it hasn't got good yet but that's the other two.
48:40
And then you remember why you use Claude.
48:43
I remember why I don't use Gemini, it just doesn't really
48:45
work. There there was a great Hugging
48:47
Face article that I just saw about and they benchmarked
48:52
Claude versus codex and it was like yeah for kind of like the
48:58
the cheaper work that you want done, the simpler stuff use
49:01
Codex and then meteor stuff it recommended to use Claude but.
49:07
Yeah, Codex isn't bad. It's like it's a constant
49:10
leapfrog race. I can imagine there'll be times
49:12
in the near future when Codex was ahead.
49:14
So like my impression as of right now is that Claude is
49:19
usually better, but like I wouldn't want to lock myself
49:22
into a work photo or Quabi tone use cloud.
49:25
And so it's good to be using a platform and setup which isn't
49:28
like built one of the major providers.
49:30
Like even if say like clouds on app was amazing, I'd still be
49:35
reluctant like totally locked myself into it because I want to
49:37
switch to codecs if I now that and Gemini might get really
49:39
good. Like Google's bought a lot of
49:41
efforts this stuff and like on the benchmarks, Gemini looks
49:44
amazing. You know, somehow something
49:46
about the harness makes him not quite live up to that when I use
49:49
it. Or maybe, just maybe, it's
49:49
actually amazing us haven't worked out the right practices
49:53
of using it well. Yeah.
49:55
You haven't put the time in and that's potentially 1 aspect or
49:59
it is like you're saying like the harness is not quite there
50:03
and it may get there in the next week as we've been talking
50:07
about, you know, like so it's good to have that optionality.
50:11
Yeah, I'd, I'd never like write off Google.
50:13
I've worked at Google. They're a very well run company
50:16
and they've got some very smart people.