Latest / Agentic Conversations (formally mlops.community) / Sandboxing, Agent Harnesses, and Agent Teamwork

Transcript

Sandboxing, Agent Harnesses, and Agent Teamwork · Agentic Conversations (formally mlops.community)

• 0:00
  We had this like wild discussion in the office where I was
• 0:03
  talking to one engineer saying, what are you using?
• 0:06
  And he's like, oh, I used caffeinate.
• 0:08
  And then I think William walked in and he's like, oh, I'm on
• 0:10
  amphetamine. Like anybody else listening to
• 0:12
  this would be like, the fuck are these guys talking about?
• 0:15
  But these are like legit apps that are there to keep their.
• 0:19
  It's not a bunch of junkies that are.
• 0:21
  Talking about maybe? Where should we start?
• 0:28
  Which one should we do first? Why don't we do something with
• 0:32
  you? Like I, I think you're pretty
• 0:34
  into the harness stuff, right? Let's let's go.
• 0:37
  I I think we can go from like the like tech, like the harness,
• 0:42
  like how agents are done to the human.
• 0:44
  So we'll end on the human. Harness to human.
• 0:47
  Honest to human, that's correct. How much harness is too much
• 0:52
  harness, and what even do you feel like is needed on top of
• 1:01
  the harnesses that you get out-of-the-box?
• 1:04
  I guess like how would you define harness?
• 1:06
  Like how do you see it because it's so open?
• 1:09
  Yeah, I think about harness as all the stuff that you expose to
• 1:16
  the agent to make it perform better.
• 1:19
  And so this is I kind of recycled the blog post from that
• 1:25
  dude Viv from link chain. And it's like all of the
• 1:31
  prompts, all of the skills, all the if you have like logs, if
• 1:36
  you have the files, file systems, the model itself is
• 1:42
  just one thing or the agent itself is one piece of it.
• 1:45
  And then you have everything that you add on to the LLM that
• 1:50
  helps it perform better. You know, Claude code Codex Eli,
• 1:55
  these are harnesses in themselves, right?
• 1:57
  And then you'll have, you know, like I think Viv and Harrison,
• 2:00
  like they've got their own harness.
• 2:01
  I think open claw is pie, if I'm not wrong.
• 2:05
  There's a whole bunch of them in there and they're I guess like
• 2:07
  different levels, levels of abstractions.
• 2:10
  And I'd say like the Claude code codex Eli is probably the the
• 2:13
  most serious level of abstraction, right?
• 2:15
  Like it's the highest. Why you you, it's just, you
• 2:18
  know, clearly it's kind of just giving in instructions, but then
• 2:21
  you can build even more on top of that right before you
• 2:25
  actually reach the user. So that's why I'm always like
• 2:27
  curious like where that like definition is.
• 2:29
  So if we're saying the harness is basically everything, are we
• 2:34
  saying that? Are we saying it's between like
• 2:36
  the user and like the model, like everything in between?
• 2:39
  And then what's So in that case, I don't know how much it's too
• 2:45
  much like the way I I think about it, it's like the job of
• 2:48
  the harness in this case is really to do to do things right.
• 2:51
  Like agents really like fast feedback loops and agents have,
• 3:00
  I guess what you couldn't say is like this tendency to do really
• 3:03
  dumb things as well. So you almost want to protect it
• 3:06
  from itself and you wanted to have this fast feedback loop so
• 3:11
  when it does do something dumb, it can react quickly and adjust,
• 3:15
  right? I know.
• 3:16
  Like I know what it needs to do to adjust.
• 3:18
  And when you say protect it from itself, you mean protect it from
• 3:22
  an agent that is doing the wrong thing or cause I've also, I was
• 3:26
  literally just having a conversation at lunch with my
• 3:29
  friend Mark about protecting agents, like sub agents from
• 3:33
  stepping all over each other. And so there's many ways you can
• 3:37
  protect it from itself, whether it's isolation or it's just like
• 3:41
  that's true, giving it the right tool at the right time.
• 3:43
  That's true. That's true.
• 3:45
  So like, so it's like a really, really simple one.
• 3:50
  I remember in the early days of Cleric, we were extremely
• 3:57
  careful about how we wrote tools and so we didn't give it access.
• 4:01
  So you imagine Kubernetes like we didn't actually give it
• 4:04
  access to the Cube Cuttle CLI. We wrote custom tools that were
• 4:08
  extremely rigid so the agent could just do the thing that we
• 4:12
  wanted to do and nothing else. But then, you know, models get
• 4:16
  better and then very soon, like, you know, six months later, that
• 4:20
  turns out to be a really dumb idea.
• 4:22
  But at the at the time, it makes a lot of sense, right?
• 4:24
  Because like you're actually preventing and doing more dumb
• 4:27
  things. But now, like once you know,
• 4:30
  today, these things are just so good at composing multiple tools
• 4:34
  to get to the answer at once. And then you get to a point
• 4:37
  where another harness is like the thing that's restricting you
• 4:40
  from getting to like, you know, better.
• 4:42
  That's why the big question that I have is how much harness is
• 4:45
  too much harness? And like also how much is, if
• 4:50
  you're trying to really map out what you want step by step and
• 4:56
  forcing it into one path, there's going to be times that
• 5:01
  you want to do that, but then there's other times where you
• 5:04
  don't need to. And so by doing it, you're kind
• 5:06
  of wasting your time. Right.
• 5:08
  Which is why I think it's an evolution, right?
• 5:10
  Like, you know, eventually at the current state of the model
• 5:15
  and the current state of like whatever harness you're using,
• 5:18
  you, whatever you're doing, if you're building an agent, I, I
• 5:22
  hope that you're monitoring these things and you're having
• 5:25
  these traces to see like why it's screwing up and why it's
• 5:28
  not right. So, you know, like we have, we
• 5:31
  document failure modes, like every time it does an
• 5:33
  investigation, we do a review and we see what did it do Well,
• 5:37
  but mostly like we look at, so we, we, we, we take the trace
• 5:42
  and we put it through another element we have documented like
• 5:45
  labels, right? And then we try to figure out
• 5:48
  like, you know, did you spend too much time, you know, like,
• 5:51
  for instance, like did we run this log query that just flooded
• 5:54
  the context window because we, we went too wide, right?
• 5:57
  Or we tried a query that, you know, took six attempts when it
• 6:01
  should have just taken 1 because we were just experimenting right
• 6:06
  now, the thing that you don't want is to be too restrictive.
• 6:10
  But then when you notice that it's constantly doing these
• 6:13
  similar mistakes, then that becomes more of a design
• 6:16
  question. Like is that something that you
• 6:17
  build into the harness so you almost protect it from itself,
• 6:21
  like I said, or is this like it's OK, right?
• 6:25
  Like it's, it's not, it's not so fundamental that it's going to
• 6:28
  screw up because if you try to overfit and you try to fit fix
• 6:32
  that, you're going to screw up something you know elsewhere.
• 6:35
  And like you said, if it gets that fast feedback loop and it's
• 6:38
  making the mistake but it understands it, can it fix it
• 6:41
  and correct course correct in flight?
• 6:44
  Exactly. So to me, it's almost like a
• 6:46
  game that you're playing where the model is just getting better
• 6:48
  and better at reasoning and then it's getting a lot more capable
• 6:52
  at handling more subjective kind of tasks like this.
• 6:55
  Because obviously like what you don't want is to be very
• 6:57
  deterministic where it's like 1 size fits all.
• 6:59
  Anything beyond like the most basic use cases, like I sorry,
• 7:02
  is probably one of the worst ones because it's so dynamic,
• 7:04
  it's so complex. You actually want that to be
• 7:07
  some play right for the agent. Like mistakes are OK, like 100%
• 7:13
  perfect. Like investigation is very
• 7:16
  suspicious. Like it means that either you
• 7:18
  like, you really overfit to that like one thing, which is like
• 7:22
  weird, or it was just so simple that you probably shouldn't, you
• 7:25
  know, pay attention. But now if it's like, you know,
• 7:27
  like 20%, it's just not getting anywhere.
• 7:30
  It's just constantly getting stuck.
• 7:31
  Now you have a problem. And then you could then I think
• 7:34
  that makes sense for it to be sort of a harness level, you
• 7:37
  know, evaluation. And see, maybe the way I would
• 7:39
  frame it is that not everything has to be an agent.
• 7:42
  Yeah, right. Like maybe an example that I can
• 7:46
  think of is, you know, in our in our CICD pipeline, we go, we we
• 7:54
  have actually like we try to do a like a security review and a
• 7:59
  like a code conventions review, things like that.
• 8:02
  That does not need like 100 different tool calls.
• 8:05
  It doesn't need to be like very, very complex thing.
• 8:08
  It's pretty simple. Effectively you're trying to
• 8:10
  see, OK, like what's the DIF that's trying to be shipped?
• 8:13
  And then what is some objective metric of what good looks like
• 8:17
  that you're passing to this LLM flow?
• 8:19
  And then you just want to know is it pass or fail?
• 8:22
  And if it's fail, why? That sounds pretty deterministic
• 8:25
  to me. It's got like some LLM involved.
• 8:27
  And so maybe that's like, you know, so that's where maybe a
• 8:30
  good example of where you don't have to overthink things and
• 8:33
  have this really fancy harness and really fancy agent probably
• 8:36
  like get something off the shelf and like it'll work pretty well.
• 8:39
  Whereas like, you know, if you're trying to do, you know,
• 8:42
  like a coding agent or like an SRE agent, yeah, you're not
• 8:45
  going to get away with something as, you know, simple as that.
• 8:48
  Talk to me more about the evolutions that you had to make
• 8:53
  as the models got better within Clare.
• 8:57
  Like you were saying, you know we we used to do this but now a
• 9:01
  a few months later even we realized we don't need to do
• 9:05
  that anymore. Yeah, these are always fun when
• 9:08
  you talk about like, all the ways you screwed up.
• 9:10
  Yeah, right. Instead of talking about like,
• 9:12
  oh, look at, look at, look at. I'll say it's done.
• 9:17
  I, I think we're like, you know, when we started, I remember like
• 9:21
  we had this like big debate. This is like, you know, late
• 9:24
  2023 and we went really fancy. I remember talking about an
• 9:30
  architecture of like an Ant colony where, you know, you have
• 9:34
  a queen that's talking to its workers and its workers are like
• 9:39
  very distinct jobs. So, you know, in this case, like
• 9:41
  the queen would be the incident commander almost.
• 9:44
  And like it would just fire off all these ants to go get data.
• 9:50
  And I mean, I think we just spent a little bit too much time
• 9:53
  on that. Like it's.
• 9:54
  Trying to create that architecture.
• 9:56
  Yeah, because like there was clearly a time when, you know,
• 10:00
  just just for context, this was GPT 3.54 just came out.
• 10:04
  I don't think Anthropic was really seriously on the scene in
• 10:09
  the scene at that point. Maybe they were, but like
• 10:11
  definitely everybody was like really around the open AI models
• 10:15
  and they just weren't very good. I remember I just for fun, I
• 10:20
  looked at our git history and I looked at the like the prompts,
• 10:24
  like we were screaming, you know, it was like.
• 10:27
  All caps. Yeah, like don't do anything but
• 10:30
  Jason, like I just please return Jason like do this, do that.
• 10:34
  It's like all caps exclamation marks.
• 10:36
  I don't know if we like never expletives, but I wouldn't be
• 10:38
  surprised. And then, you know, your
• 10:40
  coupling that kind of kind of, you know, basic usage of the
• 10:46
  model and you have such like, you know, sort of expensive
• 10:48
  architecture. Something has to give, right?
• 10:50
  So then we rebuilt everything Like it took us like, I don't
• 10:55
  know, like took us at least like 3 months to guys realize, OK,
• 10:58
  like this is not going to get us anywhere.
• 11:02
  And then we made it a, we, we made it really simple.
• 11:04
  And I think that was good. You know, we call it a query
• 11:08
  agent where it was just like, OK, like can we just have
• 11:11
  something very reliable where I give you a question and you
• 11:15
  figure out where to go and just answer the question.
• 11:19
  So it's not like 10 workers, no supervisor, blah, blah, blah.
• 11:23
  It's like a single agent. And surprising like I think
• 11:26
  that's that's actually like cloud code was amazing because
• 11:30
  it was also actually very simple.
• 11:33
  And that was, I'm not saying like we got simple first, but
• 11:37
  like we went really Uber complicated and then we went
• 11:40
  simple. And I thought that was like
• 11:41
  really effective. The, the other piece was like,
• 11:44
  obviously like the tools thing that I talked about, right?
• 11:46
  Like we, it was almost like a thing of if you know what you,
• 11:54
  the problems that you want to solve, if you fix the tools, we
• 11:59
  felt that that was a good way of getting this thing to be as
• 12:02
  deterministic as possible where we wanted it to be
• 12:04
  deterministic. And so then let the, the model
• 12:08
  actually figure out like how to, you know, compose its way to the
• 12:12
  answer. But I think, you know,
• 12:15
  eventually like what, you know, Clark goes to the world did was
• 12:20
  say, well, actually the models have gotten good enough that you
• 12:24
  don't have to fix, you know, the the tools anymore.
• 12:29
  You can just write bash, you can write Python And run that.
• 12:33
  And So what we had was basically relaxing that constraint and
• 12:39
  then moving right. And this again is like what I'm
• 12:43
  hearing is you're saying, oh, we just were giving like the
• 12:48
  harness was less deterministic of it.
• 12:51
  We weren't so on top of it. Exactly.
• 12:55
  But then The thing is, like, you always pay a cost, right?
• 12:57
  Because when you have tools that are that deterministic, security
• 13:01
  is not a problem because you know off the bat exactly what
• 13:06
  the agent can and cannot do. Yeah, the moment you give it
• 13:09
  bash in Python now things get scary, right?
• 13:11
  And then so we couldn't really make that switch until we had to
• 13:15
  just sandbox the shit out of this agent, right?
• 13:20
  So the way at least like my mental model has shifted over
• 13:23
  the years is just you want to have as much non determinism
• 13:28
  inside the box, but you want to make a really strong box.
• 13:32
  Uh huh. So I can't like as much as you
• 13:34
  can. I can't you you want.
• 13:36
  To you guys were some of the first folks that I heard talking
• 13:41
  about sandboxing and you were creating environments too, which
• 13:46
  is another one that became very popular because you had to
• 13:49
  simulate cloud environments to see if the SRE agent was running
• 13:55
  right. Yeah, yeah.
• 13:57
  Like it's, I think probably the most complex thing that at least
• 14:01
  in our domain, is that every environment we would go to is so
• 14:05
  different, right. Like, I mean, I use myself as an
• 14:09
  analogy. Yeah, I used to work at Gojek
• 14:13
  running the ML platform and I'm working on problems like, you
• 14:16
  know, how do you make model inferencing for pricing, driver
• 14:19
  allocation, uh, you know, as as efficient as possible.
• 14:22
  It's all in Kubernetes. But then everything I just said
• 14:26
  is going to sound very familiar to someone at Lyft, at Uber,
• 14:30
  any, any, any kind of ride hailing food company.
• 14:33
  But it's not like I can join them today and I'm going to be
• 14:37
  productive. Yeah, because they've got their
• 14:39
  own way of doing something like, yeah, the high level problem
• 14:41
  statement is the true is, is, is the same, but the way they do it
• 14:45
  is so different. And so, you know, if you assume
• 14:47
  that I am the agent and I've gotten really good at these
• 14:50
  tools like Kubernetes and all the stuff, I'm still going to
• 14:54
  suck when I move into Uber, right?
• 14:57
  And so we, the way we try to solve for that, I know, like I,
• 15:01
  I think you had a really good job with William and Shreya
• 15:03
  about this, right? A few What was it like a year
• 15:06
  ago? That was probably a little more
• 15:08
  than a year ago now. Right.
• 15:10
  Yeah. And so this is the primary
• 15:12
  problem, like how do you test very quickly across diverse
• 15:18
  environments and know that your agents actually up to the task?
• 15:23
  And so then like in like it's, it's been like a really
• 15:27
  interesting phase. You know, you're talking about
• 15:30
  like the evolution of cleric. The investigation agent has just
• 15:33
  become the least important part of the stack.
• 15:38
  It's almost like the easiest thing to build now, because that
• 15:40
  is like not what's actually durable.
• 15:43
  It's it's like these environments, like how can you
• 15:46
  test and how do you know it's working?
• 15:47
  Like that's durable? Like the learning stuff, you
• 15:50
  know, we can talk about, it's knowing I was like, monitoring
• 15:55
  it and verifying whether it's correct.
• 15:57
  This becomes like far more harder and more durable
• 16:00
  challenges. Whereas I think, you know, you
• 16:03
  can get your cloud code connected up to a bunch of MCPS
• 16:07
  and say, hey, go figure out what's wrong.
• 16:09
  You'll probably do fine. Yeah.
• 16:11
  Because the search problem is much less complex than that.
• 16:17
  Is this working like the verification problem right?
• 16:20
  Right, right. Like once you go from beyond a
• 16:23
  simple kind of task where it's like, you know, if something's
• 16:27
  wrong, it's ironic. Like the more, the more serious
• 16:30
  the problem. It's almost like there are some
• 16:33
  cases of that, but it's actually very simple because it's
• 16:36
  serious, because something is down.
• 16:38
  So it's quite binary, right? So it's, it's pretty obvious
• 16:42
  this is the problem, right? But like the most subtle it
• 16:45
  gets, now things get hard, right?
• 16:48
  Because now it's like very reducing credit.
• 16:49
  So that's where it's like really hard to know.
• 16:53
  That's where the verification comes in.
• 16:55
  That's where the learning comes in.
• 16:56
  Because like you need to actually like look at the
• 16:59
  problem through the lens of that company, like their conventions,
• 17:03
  the way they do things. Because if you just come in with
• 17:06
  just world knowledge, it's going to be really complicated.
• 17:09
  Because like the kind that you know, out of maybe 100 different
• 17:12
  reasons this could have failed, you have no way of knowing which
• 17:16
  ones you should be paying more attention to because they're
• 17:19
  more likely to be the cause. Which an engineer who spent ten
• 17:21
  years in the company would immediately do, right?
• 17:25
  Which at the end of the day is kind of still the search space,
• 17:29
  but you have to verify the search in a much different way
• 17:34
  than. And I like what you're saying
• 17:36
  there, like the it's a huge problem, you know, oh, the
• 17:40
  database blew up or, you know, it's normally pretty clear if
• 17:45
  it's a very, very if it's Seb one or whatever incident, you're
• 17:49
  like we understand. But when it's just something
• 17:54
  that you're like, why is latency spiking by 20% here?
• 17:58
  What is going on? Those are the worst.
• 18:00
  Yeah. And so then you really have to
• 18:02
  dig in and you have to verify through that whole the search
• 18:07
  space again and have the agent know if it's on the right path.
• 18:11
  And so that's been where you're focusing the attention.
• 18:13
  Now you've and figuring out the sandboxes, figuring out the
• 18:16
  environments, figuring out how to verify.
• 18:20
  Yeah, like how do you just relentlessly and ruthlessly keep
• 18:25
  getting the accuracy up, and then as you're doing that, you
• 18:29
  keep falling into more and more of this subtle territory?
• 18:32
  Then you need to build more stuff to make sure that you're
• 18:36
  on the right track, right? Because you once you've moved
• 18:38
  away from simple, that's where like all this other stuff that
• 18:41
  we talked about matters. I think it's almost like that's
• 18:46
  why I've, I've started thinking a lot now about just sort of the
• 18:50
  human agent interactions now, right?
• 18:54
  Because you know, in a world where the agents, you know,
• 18:59
  something like 30 to 40% correct.
• 19:04
  I actually, I mean, there's a little bit of a detour, but
• 19:06
  like, I was, I was talking to somebody like an engineer who
• 19:09
  said, I almost like it when the agent's always wrong because I'm
• 19:13
  suspicious and I'll check right. Whereas like if it's like mostly
• 19:19
  right, I start trusting it. And then at one time it's like,
• 19:22
  so, so your Spidey sense is now. And I'm, I'm, I've just seen the
• 19:27
  Spidey sense just continuing to get challenged because, you
• 19:33
  know, I can, you know, all the stuff I was talking about with
• 19:35
  Tarek in the early days, you know, when we're doing these
• 19:37
  tools and things like that, you know, what's not that it's like
• 19:41
  what just being explicit about is that, you know, we weren't
• 19:44
  very effective outside pretty simple kind of problems, right?
• 19:47
  And you didn't trust it at all. I imagine exactly like, unless
• 19:50
  it was like very obviously the problem and we found it, you,
• 19:56
  you, you, you know, you didn't see that trust barrier being.
• 19:59
  So that was like our focus for like, you know, it's been like 2
• 20:02
  1/2 years of just like trying to get to that point where like,
• 20:06
  like, you know, you're just implicitly trusting this thing
• 20:09
  and that's been happening. But now it's creating new
• 20:11
  problems, right? Because when that happens,
• 20:17
  people are now trying to throw you into more and more complex
• 20:21
  areas, like they want to like, give you more work.
• 20:24
  And now it's getting a lot more subjective on like what's what's
• 20:28
  correct and what's not. And then people are worried
• 20:31
  about, well, you know, like, how am I going to keep learning if
• 20:36
  this thing's getting so much better, right.
• 20:39
  And so that's that's that's. Where you're saying the human
• 20:42
  aspect comes in, you have folks that are starting to be like
• 20:46
  suspicious of it, but also they're recognizing that, well,
• 20:51
  if I'm always thrown it at all of my problems, then am I
• 20:56
  outsourcing the thinking? That, but also, I think it's
• 21:01
  like more of a, this shift that people have gone through, which
• 21:04
  is, you know, say, say 12 months ago, people weren't, I don't
• 21:10
  know if people were actually that like concerned, like
• 21:13
  outside people who are like really deep into AII don't know
• 21:17
  if engineers thought, but these things are really good.
• 21:20
  And I'm not sure. I, I, I, I heard that more and
• 21:23
  more. No, but we're clearly at a point
• 21:26
  like, I think people just went, went into Christmas break,
• 21:29
  played around with like all this stuff.
• 21:31
  And then we're like, holy shit, like things have really changed.
• 21:35
  And now you can just see this trajectory that, you know,
• 21:39
  whatever it's not good at now it's going to get there.
• 21:41
  Like I know it is. And so now all of us are kind of
• 21:44
  kind of thinking like, OK, so what's next?
• 21:46
  Like what does what, what happens now where I'm not needed
• 21:50
  anymore in this one task, right? So imagine like, you know, six
• 21:55
  months ago you're talking to cleric, maybe cleric gets 60%
• 21:59
  there and now you have to like have two more follow-ups and get
• 22:02
  to their answer and then you're done.
• 22:05
  But now he's going 65, now it's 707580.
• 22:08
  It's just keeping on going. And then you're like, OK, So
• 22:11
  what does this mean, right? Yeah, where do I add value?
• 22:15
  Yes. And that's what I'm really
• 22:17
  interested in. And so, so part of it is what
• 22:19
  you said, which is like the learning loop.
• 22:22
  But then also I think it's more, you know, like a more
• 22:25
  philosophical question, right? Like where sort of am I So in,
• 22:29
  in my mental model? It's like, I don't know, I'm
• 22:33
  still working on like what the right analogy would be.
• 22:35
  But it's almost like all of us are going to become managers.
• 22:40
  I don't think we're going to like, you know, if you look at
• 22:46
  sort of the, the kind of the online chatter and like this is
• 22:49
  even happening in our in our team, people are token maxing.
• 22:54
  People are like worrying about how do I make sure my agents
• 22:58
  have what they need to get work done?
• 23:00
  Yeah. How does it continue through the
• 23:02
  night while I sleep? It still is working.
• 23:05
  Right. Like I had this, like we had
• 23:06
  this like wild discussion in the office where I was talking to
• 23:10
  one engineer saying, what are you using?
• 23:12
  And he's like, oh, I use caffeinate.
• 23:14
  And then I think William walked in and he's like, oh, I'm an
• 23:16
  amphetamine. Like anybody else listening to
• 23:19
  this would be like, the fuck are these guys talking about?
• 23:21
  But these are like legit apps that are there to keep your.
• 23:25
  It's not a bunch of junkies that are talking.
• 23:28
  About but but yeah, like so so then what?
• 23:34
  You know, if you just like double click a little bit beyond
• 23:36
  like the sweatshop vibes, it's basically it's like, hey, I have
• 23:41
  these workers and I kind of care about making sure that they're
• 23:46
  OK. I want to make sure that they're
• 23:48
  not going to be interrupted. They've got what they need to
• 23:50
  get stuff done. It's pretty kind of managerial
• 23:53
  problem. Yeah, right.
• 23:55
  It's just the managers don't tend to give humans
• 23:59
  amphetamines. No.
• 24:00
  And that's. Well, not yeah, at least not all
• 24:03
  of them. Like the ones we you, you should
• 24:05
  watch out for. Those are the ones.
• 24:07
  That exactly. Yeah.
• 24:09
  So, but the apart from that, I agree with you wholeheartedly
• 24:15
  because when we had the coding agent conference a month back, I
• 24:20
  remember one of my friend's bosses who is there like VPN
• 24:26
  Jet, a fairly big company. Like he was so excited because
• 24:32
  he went to the workshop and he was like hands on shipping code,
• 24:38
  like merging PRS super excited. And then when I wrote about that
• 24:42
  on LinkedIn, people were like, yeah, well, why wouldn't they
• 24:44
  be? Managers have been doing this.
• 24:47
  Like what do you think a manager is?
• 24:49
  It's almost like a vibe coder. You're managing agents or you're
• 24:53
  managing people. It's just people are actually
• 24:56
  kind of harder to manage a. 100%.
• 24:58
  Exactly. Humans are very messy agents,
• 25:01
  less so. You can really grind and give
• 25:04
  them the amphetamines and it's all right.
• 25:08
  Yeah. Or you can just be.
• 25:10
  Yeah, there's there's no need to be too nice to your.
• 25:13
  Agents. Yeah, polite.
• 25:14
  They don't ask for raises every couple months or more, shares
• 25:19
  all of that stuff. Or they don't like complain
• 25:22
  about sitting next to somebody that smells bad.
• 25:25
  All of that stuff that humans have to deal with, right?
• 25:29
  You don't deal with that and. But it's funny to me because
• 25:34
  it's, it's like I, I think people have gotten really caught
• 25:38
  up with how addictive it is, right?
• 25:39
  And I think that's coming from this place of, you know, like, I
• 25:44
  think if you just went through and got to, you know,
• 25:49
  eventually, like there instead, it's a very stressful job being
• 25:53
  a manager, right? Like you're constantly like,
• 25:56
  hey, I've got, you know, 10/20/30 people depending on me
• 26:01
  and what I say we're going to do.
• 26:04
  I've got 30 times like the bandwidth of that week assigned
• 26:09
  to this problem that I've said we have to do.
• 26:11
  And now you know, like, and this problem goes from CEO to
• 26:15
  engineering manager. It's pretty freaking worrying,
• 26:18
  right? Like you.
• 26:19
  And so like what I'm seeing with the amphetamine and like all
• 26:22
  these conversations is like the the Colonel is that same worry
• 26:26
  like I I've got all this like bandwidth of tokens.
• 26:31
  Am I spending it correctly? Like am I doing the right thing?
• 26:34
  Some of us aren't, but just like shipping all kinds of shit, but
• 26:37
  like generally like it's it's just going to end up there,
• 26:40
  right. And and then you're back to
• 26:42
  being this like person who's like, OK, like here's I've got X
• 26:46
  resources. I've got a problem to sort out.
• 26:49
  How do I, you know, corral my army of people or like agents to
• 26:54
  actually go, you know, solve this problem?
• 26:56
  Yeah, make sure we're all rowing in the right direction and
• 26:59
  taking advantage of every moment because any moment that the
• 27:02
  agents aren't right, looping well.
• 27:05
  Hopefully we don't get too stressed out about that because
• 27:07
  that, that is a like a dark hole.
• 27:09
  Yeah, I think. But yeah, exactly.
• 27:12
  You don't have 8 agents constantly running and you feel
• 27:16
  like you're what are you even doing right?
• 27:18
  I know. I mean, I feel that like I, I,
• 27:21
  I've, I don't know the last time why I haven't.
• 27:24
  So, so we use, I'm sure like people are using their own
• 27:27
  terms, but like internally we use the term like the hopper.
• 27:30
  It's like you're cooking like, well, I'm just going to put
• 27:32
  another one on the hopper. And so I, I, you know, I, I
• 27:35
  need, I feel like I have this need to put something on the
• 27:39
  hopper before I go to bed. And I have just infinite
• 27:43
  satisfaction waking up to see it's like almost done.
• 27:46
  And they're pretty cool with, with the harness discussion and
• 27:48
  all that stuff as well, because like you're what you're
• 27:51
  implicitly doing is every time it screws up, you're kind of
• 27:54
  thinking, oh, what should I have given it?
• 27:56
  Like, what could I have given this?
• 27:57
  Like, you know, and it's like really what managers do, right?
• 28:00
  Like, you know, I was, I was reading a post from, I can't
• 28:03
  remember who, but she was like the first thing she asked on her
• 28:06
  one-on-one was like, well, how, what's your work set up like?
• 28:10
  And can I fix something like, are you comfortable?
• 28:13
  I mean, are we literally doing that with agents now?
• 28:16
  That's the harness. That is the harness exactly.
• 28:20
  What I've been seeing, and I imagine this is going to be
• 28:22
  something that maybe it's fleeting, maybe it becomes more
• 28:26
  prevalent, is that you have to do a lot of the upfront work in
• 28:31
  order to be able to put something on the hopper and let
• 28:35
  it run. Yes, yes, but by a print work I
• 28:39
  assume you mean like, you know, speaking it out properly.
• 28:42
  The scaffolding around it, you can't just kind of like Willy
• 28:45
  nilly throw in one or two sentences and be like go and
• 28:48
  have fun. Well, you can, but then you're
• 28:51
  just going to have a look at pretty shitty.
• 28:53
  Shitty outcome. Exactly.
• 28:54
  I mean, that's where I guess, like cost and things like that
• 28:57
  come in. But yeah, So what I'm thinking
• 29:02
  is like, OK, so right now we're kind of like, you know, some of
• 29:05
  us more than others are using this more and more like
• 29:08
  effectively. And then, you know, what are
• 29:11
  these people doing that's more effective and how are we
• 29:14
  learning from them? I'm like, where does this all
• 29:15
  go? Like to your point, you know,
• 29:18
  this whole spectrum and development and the Ralph Wigan
• 29:20
  loop and all the stuff like it's coming up because like, you
• 29:23
  know, these are like decks was amazing at the coding engine as
• 29:27
  you always like. It's all like these lessons
• 29:29
  learned from people who are just like doing it all the time.
• 29:31
  And we're all trying to learn from it.
• 29:34
  But I guess like my, my, my thought here is that I think
• 29:39
  it's kind of like you want to be a mix of a manager and a
• 29:43
  Craftsman, right? Because, you know, in a world
• 29:47
  where you can build anything you want effectively, what is left
• 29:53
  is your taste and your judgement, right?
• 29:56
  And that's why I think there's a very dark way of taking this,
• 30:00
  which is that you lose all agency.
• 30:02
  You stop learning like you like this thing is just gonna and I,
• 30:06
  I don't know, I guess like if you take any anything from this
• 30:08
  conversation, I hope that we can have a more positive view of it,
• 30:11
  which is that like actually the really fun stuff about building
• 30:14
  product is putting your personality and like your
• 30:17
  thoughts and like your view of like what you want to bring to
• 30:19
  the world out there. Now you can get there much
• 30:21
  faster. And so like, how do we build for
• 30:25
  a world like that where that becomes like all the why you
• 30:31
  spend all your time on, whereas like the sort of the toil of
• 30:34
  getting there is something with the agent is doing.
• 30:38
  It goes back to those feedback loops you were saying before.
• 30:41
  That's what I was thinking. Like if you know, if you,
• 30:44
  because the thing with I think the managerial word, which I
• 30:46
  don't like as much is that it almost suggests that you don't
• 30:52
  know much about the craft. You're just focusing on the sort
• 30:56
  of public mastering aspect of it, which I mean, I, that's why
• 30:59
  I don't like the word. I like the whole craftsmanship
• 31:02
  thing because like it, it, it's about mastery, right?
• 31:05
  So you're almost like in charge of a workshop and you're trying
• 31:07
  to get something done. And what you don't want is to
• 31:10
  judge based on what's produced. You actually want to know what's
• 31:14
  going on, right? You, you don't want to be
• 31:18
  abstracted away too much. So I think like, even with the
• 31:21
  agents, I think the mistake would be to say this thing's so
• 31:25
  good. So we're just going to wait for
• 31:27
  it to finish all of its code and then I'm going to review all of
• 31:29
  its code. I'm not sure that that's
• 31:31
  effective. Well, you didn't want to say
• 31:33
  something about and before we hit record about how you want to
• 31:38
  be able to see as it's making its decisions, what it's making
• 31:42
  and then steer it. And that's steerability I think
• 31:46
  is a little bit lacking sometimes.
• 31:50
  But also, if you're going to try and scale to running 8 parallel
• 31:55
  agents, you just can't expect to steer 8 agents in parallel.
• 32:01
  That's true. But that's why I think the point
• 32:03
  of abstraction is like useful, right?
• 32:05
  Because you know, if you've got 8 agents and you're writing
• 32:09
  some, you know, big project, it's going to produce just reams
• 32:13
  and reams of code. You can't read all that code.
• 32:17
  But an abstraction above that is what are the decisions that it
• 32:20
  took to get to that code, right? Like a really dumb example is,
• 32:26
  you know, an agent is probably gonna be really good at writing
• 32:29
  a Mac OS desktop app. It's gonna get, like, really
• 32:32
  good. It's gonna look beautiful.
• 32:34
  But maybe the judgement is that we shouldn't write a desktop
• 32:39
  app. It should be a web app, right?
• 32:41
  Because I want more customers. And like, I, I think there's a
• 32:45
  bunch of people in Linux who's gonna like this stuff as well.
• 32:48
  I mean, that's a judgement call. And so then there's a deletion
• 32:51
  trace somewhere where your agent or your product manager agent is
• 32:56
  like, we should build a Mac App. And so then, you know, at that
• 32:59
  point you could probably go, well, no, we shouldn't.
• 33:03
  I think we should build X instead, so that is, I think,
• 33:07
  infinitely easier to review than all the code that was written,
• 33:12
  right? But isn't that just making you
• 33:15
  the bottleneck? Well, that's the thing, like I
• 33:18
  think it if you become the bottleneck, then it should be
• 33:20
  for a good reason. You are actually like putting,
• 33:23
  inserting yourself in there. I think there's the kind of two
• 33:25
  constraints like 1 is your test. So hopefully, like, you know,
• 33:29
  you are operating at that level where you're injecting a test
• 33:32
  and that ideally is like something you're exercising and
• 33:34
  you're getting better at. And the second one is like, I
• 33:37
  think the real world also has constraints, right?
• 33:39
  Like in a world where everything gets amazing and like you can
• 33:43
  produce infinite code, your customers can't take that much
• 33:47
  change anyway, right? Like you can't have, you know,
• 33:52
  like a lot of these companies and the services that we are
• 33:54
  used to took decades to evolve into what they are.
• 33:57
  Like, you know, if I, if you just speed rang someone who uses
• 34:01
  Uber today to everything that's they've offered, you know, over
• 34:05
  the last 10-15 years, I mean, that's not possible that people
• 34:08
  can't take it. So I think, yeah, it is a
• 34:11
  bottleneck. But like it's they only make
• 34:13
  sense if it's adding value. And I think there's those are
• 34:15
  two ways of adding value. Well, that's a fascinating topic
• 34:18
  to think about. Is where you need the human to
• 34:24
  plug in. Because if it is, then going
• 34:28
  back to your example of the desktop app versus a web app,
• 34:36
  that's one question that potentially you plug in a human
• 34:40
  for or you don't, and it's because maybe it just doesn't
• 34:44
  matter and you can build it. But like, what decisions need to
• 34:50
  actually be outsourced to the human?
• 34:54
  You know, where do you plug the human in?
• 34:56
  And how does the model know that this is a really big decision?
• 35:01
  Before I make this decision and go down this potential path, I
• 35:05
  should probably ask someone. Yeah, I, I mean, I think it's,
• 35:12
  it's useful thinking about like what the ends did.
• 35:14
  And I mean, there's no true end state, but at least like I think
• 35:17
  in the next 12 months, like what where I think we should be.
• 35:20
  It feels to me like you're kind of approving exceptions, right?
• 35:24
  Like when you know, like say you have an employee and they're
• 35:27
  supposed to like, you know, do something like, for instance,
• 35:30
  like book this podcast studio, right?
• 35:33
  It's, it's, it's likely that you're giving them some
• 35:35
  guidelines, right? Like here's your budget.
• 35:37
  Here's kind of the area that I want you to look at and you
• 35:41
  know, you go figure it out. But if you're going to violate
• 35:44
  any of these bounds, and there's a good reason, then I need to
• 35:47
  know and I I think it's kind of similar here.
• 35:50
  But again, then you have to really upfront define those
• 35:54
  bounds clearly and put that scaffolding in place and then
• 35:58
  let it run. But but I think that's the work
• 36:01
  which is valuable. Yeah, right.
• 36:03
  Because I think those are the decisions which today, like, you
• 36:06
  know, live in people's heads is all implicit decision making
• 36:09
  that's happening, right? Which is not explicit, but it,
• 36:12
  it was OK. Or at least it's OK in a world
• 36:15
  of humans. Because when we are, when we're
• 36:18
  not sure, you call a meeting, you send a slack or things like
• 36:21
  that, but it just does not work with agents because agents work
• 36:25
  on agent time, right? Like you, it needs an answer
• 36:27
  right now. Like from an SRE standpoint,
• 36:30
  like, well, you know, that's a domain, you know, I think about
• 36:32
  a lot. It's like how can you reliably
• 36:36
  get an agent to fix something at 2:00 AM if it doesn't have clear
• 36:39
  bounds of what it can do and what it can't?
• 36:41
  It just can't. It can't call a meeting at 2:00
• 36:43
  AM like we call meetings at 2:00 AM because it's an incident and
• 36:47
  like you call everybody and was like, guys, what do we do?
• 36:49
  Like should we restart this? Should we stop it?
• 36:50
  Should we do this? Should we do that?
• 36:53
  Unless those decisions become, you know, preserved or like, you
• 36:57
  know, queryable, you're not going to get anywhere.
• 37:00
  So I think the phase where in now it's all about actually, you
• 37:05
  know, I think, you know, Jaya from Foundation Capital did an
• 37:08
  amazing piece called Context graphs and which really just
• 37:11
  blew up. But I think it's like all around
• 37:13
  kind of a similar idea here where that's the valuable data
• 37:16
  now. Like what are the decisions that
• 37:18
  you know, are OK? Like what are the bounds?
• 37:20
  And then how do you actually track the decisions that the
• 37:22
  agents are making so that you're actually having this compounding
• 37:26
  loop of preserving? And it's these instructions in a
• 37:30
  way that humans can read them and, and like, go, yeah, yeah,
• 37:33
  that looks fine. No, no, don't do this.
• 37:35
  Or at least you're defining an exception.
• 37:37
  Like, as long as you're over here, do whatever you want.
• 37:40
  But if you're coming out, talk to me because I need to tell you
• 37:43
  what to do. Are you guys doing that?
• 37:45
  Is that one of those more complex things that you started
• 37:48
  to add where you have decisions being made in Slack and you need
• 37:53
  to then incorporate that into the agent so that if it
• 37:57
  encounters this problem again in the future, it kind of has a
• 37:59
  road map? We're doing that.
• 38:02
  So that to me is like part of learning where you I don't think
• 38:06
  it's as advanced as I'd like, but that's the path which I want
• 38:10
  to be on. So you know, I've been railing
• 38:13
  on, you know, on social media about like this for deployed
• 38:16
  engineer kind of model. The reason I'm not bullish on
• 38:19
  that is that you're actually not building the thing I'm talking
• 38:23
  about, right? Like what I want when you deploy
• 38:26
  cleric is I'm giving it every opportunity to learn these
• 38:30
  rules. And if you don't give it that
• 38:33
  opportunity and you're just going to throw engineers at the
• 38:35
  problem and try to like tune it, then you're not solving that
• 38:39
  real valuable problem, right? So to your point on Slack,
• 38:44
  there's a wealth of information, like when we get added to
• 38:46
  another channel, I mean, there's a ton of stuff like, you know,
• 38:50
  when this thing goes, goes down, usually it's because of this
• 38:54
  other thing. And here's what, you know, over
• 38:57
  the last six months, our engineers have tried and like
• 38:59
  this is what generally works. This is what doesn't you can
• 39:02
  build a pretty effective map just from this data.
• 39:04
  It's, you know, it's like unstructured data.
• 39:07
  It's like it's, it's, it's very valuable, right?
• 39:09
  And then that can be replayed back to the human to say, here's
• 39:13
  what I learned. This is look right, right.
• 39:16
  And then now to a human that's actually like a very compressed
• 39:20
  form of knowledge, like these are sort of what I've learned,
• 39:24
  like what I should do, how your environment works.
• 39:27
  Does it look right? And to them it's like very easy
• 39:29
  to just edit it. Yeah, I do like the distinction
• 39:32
  too of how you're saying the real value is in that type of
• 39:36
  thing and being able to create these paper trails in a way or
• 39:43
  create that like decision making process the Sops and then doing
• 39:50
  the upfront work of creating those so that the agent can do
• 39:55
  the hard work of like actually implementing it, Yes.
• 39:59
  Exactly. And you, you want to design your
• 40:02
  systems to make that hard work like valued, right?
• 40:08
  Like, like it, it has to happen like we've tried a lot of times
• 40:11
  like to do like in the SRO world.
• 40:14
  You'd insist on teams doing postmortems, documenting them,
• 40:17
  things like that. It's really hard because you're
• 40:21
  asking people to do something with no immediate tangible
• 40:24
  value. Like unless you've really got a
• 40:26
  good process down. And this happens in every
• 40:28
  industry, right? Like just document your shit.
• 40:31
  Like, just make sure you're writing stuff down.
• 40:32
  Everybody knows that it's a good idea, but it's not necessarily.
• 40:37
  But I think agents are super interesting because now you do
• 40:40
  get that feedback very quickly, right?
• 40:42
  Like if I write something down, oh, I tell the agent, hey,
• 40:46
  remember this. And then the next time something
• 40:49
  happens, it's like, oh, yeah, thanks, Dee, you told me about
• 40:52
  this. So I, you know, I did it this
• 40:53
  way instead. Now you feel, oh, OK, that's
• 40:56
  that's cool. Right, like so in a way, what
• 41:01
  you're constantly doing is updating documentation and
• 41:05
  updating. Pretty much like just for the
• 41:07
  agent. Yeah, just for the agent and,
• 41:10
  and I remember, I think William told me, yeah, normally you
• 41:13
  don't necessarily need to ingest the whole Slack thread because
• 41:18
  by the end of a Slack thread, there's a decision that's made.
• 41:22
  And did you do it with the is it the user experience that like at
• 41:27
  the end of the Slack thread, once you know what needs to
• 41:29
  happen, then you just like at Cleric and you know, like update
• 41:34
  this or here's the information. Or is Cleric just there reading
• 41:38
  everything? Because I'm wondering how you
• 41:40
  don't have a lot of noise. Well, that's a great question.
• 41:45
  Noise is inevitable. You will learn completely
• 41:49
  useless things. This will happen, but I think to
• 41:55
  us like it's more about curation, right?
• 41:57
  Like how do you like, we call it rot?
• 41:59
  I think it's pretty standard term at this point that you're
• 42:04
  just going to have this sort of memories that you've learned
• 42:07
  which aren't very helpful. I mean, there's a couple of
• 42:12
  ways. Like 1 is we give people an
• 42:16
  explicit way to create them and forget them.
• 42:18
  So if it's like we totally screwed it up, we learned
• 42:21
  something we shouldn't have, or like we're somehow not learning
• 42:25
  something we should, then you can just take over the wheel and
• 42:27
  be like, yo, you need to know this.
• 42:30
  Like remember so. It's like, yeah, commit this to
• 42:32
  memory, yeah. Or forget this exactly
• 42:35
  eternally. Exactly.
• 42:36
  Yeah, I don't do not mention this to me again.
• 42:41
  But like that's usually for extremes, right, Because people
• 42:45
  are lazy. Like you got a design for that,
• 42:47
  Like you, you don't want to make people say remember this all the
• 42:50
  time. So we do implicitly learn as
• 42:53
  much as we can like from people's conversations.
• 42:56
  We've also tried to make it really cheap for people to, you
• 42:59
  know, say goodbye cleric and bad boy like with ratings and things
• 43:02
  like that. Because then that also gives a
• 43:04
  signal that all the decisions that we made so far, if it's a 5
• 43:08
  out of five, it's pretty good chance that we've did the right
• 43:11
  thing, right? So then we can look through it.
• 43:14
  Whereas if it's like a one out of five, even though we're like,
• 43:16
  oh, we did a really good job, then you know, OK, something
• 43:19
  went wrong. So either that's something we
• 43:20
  missed or like something we, you know, over indexed on or
• 43:24
  something like that. So that's that also helps a lot
• 43:26
  from the learning. Back, continuing with the noise
• 43:31
  theme, I imagine you can and never be satisfied on all the
• 43:36
  data that you're collecting like you, whether it comes to all
• 43:42
  right, you've got your data dog, you've got your code, you've got
• 43:46
  the Slack conversations, you've got the post mortem transcripts,
• 43:50
  you've got everything and anything that you can get your
• 43:53
  hands on. Are there, is there a little bit
• 43:57
  of like a 8020 principle where you say, you know, what if we
• 44:00
  just get the Slack conversation and the commit history, that
• 44:05
  gets us pretty far. Yeah, I'd say Slack's amazing,
• 44:11
  code's amazing, and logs are amazing.
• 44:15
  These three get you very far. Slack mostly for just historical
• 44:21
  stuff. And this is again, like with the
• 44:23
  assumption that you haven't been very disciplined and storing
• 44:26
  everything. Like, you know, there are some
• 44:28
  folks we work with who've that's done a really good job.
• 44:31
  So, you know, if you go to that confluence page, it's like
• 44:33
  really well laid out, like all their lessons learned are in a
• 44:37
  separate database. So then that's like obviously
• 44:39
  much better. But generally like, yeah, like
• 44:42
  this kind of text based knowledge is helpful.
• 44:47
  The code is interesting because you're, you're looking for kind
• 44:53
  of like a source of truth kind of knowledge, because no matter
• 44:56
  what you see on Slack or documents or things like that,
• 45:02
  you never really know how up to date it is, right?
• 45:06
  Like, you know, I, I could say something today and a commit
• 45:11
  tonight could completely invalidate that, right?
• 45:15
  Like completely change everything because we've done a
• 45:17
  big refactor. And so you still need to have
• 45:21
  that source of truth. So it's like, do me to us like
• 45:24
  code and Kubernetes, like infrastate is really helpful
• 45:27
  because the agent can just be like, OK, hang on.
• 45:29
  Like, like what is true today? And then like, can I work
• 45:33
  backwards to, OK, what do I know historically?
• 45:35
  And then like, how do I you, you can't go the other way around.
• 45:38
  Like you're trying to piece together a truth from history.
• 45:41
  It's hard, yeah. Now, the thing that William
• 45:46
  always tells me when I talk to him, it's like it's pretty easy
• 45:51
  to get something up and running in terms of an SRE agent that
• 45:57
  you can demo. Yeah, like all 100% any any
• 46:01
  agents basically. But I get it overnight and then,
• 46:04
  yeah, look, look pretty convincing.
• 46:06
  And then he's like, but dude, when you get into real
• 46:09
  production systems, it is so much harder to get an SRE agent
• 46:16
  going. Can you explain like why is
• 46:19
  that? What's the difference there?
• 46:21
  That is the production gap that you see.
• 46:26
  I'd say it's two things. One is just a human element.
• 46:31
  The expectations are so much higher, right?
• 46:34
  Because I mean, the way I like to think about it is if it's a
• 46:37
  coding agent and I got 70% of the code right as a user, you
• 46:45
  know, you just save me 70% of the keystrokes I would have to
• 46:48
  make. I'm, I'm pretty happy, right?
• 46:50
  Like I can, you know, look through it, even if it's not
• 46:54
  right, it's, it's kind of cheap for me to know that it's not
• 46:57
  correct. I can write some tests.
• 46:59
  I can, I can get it over the line.
• 47:01
  I'm not, I'm not pissed off with you for being 70%, but we're on
• 47:06
  the other extreme, right? Like, you know, if I'm 70%
• 47:09
  correct and I LED you down this path, which is not exactly
• 47:13
  right, and you've spent 20-30 minutes in a pretty stressful
• 47:16
  situation, sometimes you're going to hate me, right?
• 47:21
  Like like forget about tech. It's just from a user point of
• 47:24
  view like that's. Yeah, right.
• 47:27
  Like why did you lie to me? Right.
• 47:30
  And you know, and. That's actually, yeah.
• 47:33
  I like to think about it. Yeah.
• 47:35
  Yeah. You know, like the, like the
• 47:36
  personality of people who are in the, you know, the, the, the
• 47:42
  like, the, you know, it's like the, the beast of the belly,
• 47:44
  right? Like you're just the belly of
• 47:46
  the beast belly. You, you, you get a certain kind
• 47:52
  of personality, right? Like you're not going to
• 47:54
  tolerate, like you're very risk averse.
• 47:56
  Yeah. Adrenaline's pumping.
• 47:59
  Right. Because you know, like basically
• 48:01
  the kind of companies we work with are larger companies
• 48:03
  because like if you're a startup, you don't really have a
• 48:06
  production problem, like you don't have that many customers.
• 48:08
  But once you have a lot of customers, you're going to
• 48:11
  allocate a certain set of people to look after them.
• 48:14
  And these people are like, obviously for very good reason,
• 48:17
  like very, very careful. But on the then when you bring
• 48:21
  into the technical side, that's also where the complexity is
• 48:24
  because you know, for I'll just take code again because I think
• 48:31
  most folks probably listen to this, are using coding agents.
• 48:34
  Most of the truth that you need is in your code, right?
• 48:40
  Like like for for a coding agent, you're actually going
• 48:44
  deep. You give it this one thing,
• 48:46
  you're coding repository, maybe it's couple and you're saying go
• 48:49
  build me this new feature. The complexity of the coding
• 48:51
  agent is the kind of inputs of the tasks you give the coding
• 48:54
  agent is potentially infinite. Like you can ask it to build
• 48:57
  anything you want, but the base it's doing it on is finite,
• 49:01
  right? It's just with SRE, it's the
• 49:04
  opposite, right? The the answer could be anywhere
• 49:10
  so that there's no base like it's a, it's a, it's a
• 49:12
  horizontal kind of breadth first search.
• 49:14
  But the kind of problems are kind of finite, like it's a
• 49:16
  latency spike. It's a, you know, something's
• 49:19
  down, the user's upset there's a bug.
• 49:22
  It's not like, so the complexity really is like from a technical
• 49:26
  standpoint that like the, it's a needle in the air stack, like it
• 49:29
  could be anywhere. And so when you couple that with
• 49:31
  the user problem, hopefully that gives you a sense of why this
• 49:34
  is. Yeah, flip that right on its
• 49:36
  head, dude. Oh, man.
• 49:39
  One of the big questions that I am sure you think about a lot is
• 49:45
  like, are we just going to end up becoming a skill?
• 49:51
  Is this whole thing going to get so good that now we're a
• 49:55
  markdown file that's part of a coding agent?
• 49:59
  You don't like my default mode is I'm just a very optimistic
• 50:06
  guy. So that's probably like a good
• 50:08
  thing to just put it out there. Like that's why I think Willem
• 50:11
  and I work really well because he's definitely a lot more like
• 50:14
  on the like he'll see all the risks, like you know him well,
• 50:17
  right? So I see all the optimism and
• 50:19
  then we're trying to like meet in the middle.
• 50:23
  I think that's part of why we should be having this this
• 50:25
  discussion and like people like I should be talking about this
• 50:27
  more because like the whole point is to avoid that future,
• 50:30
  right? Like that's why I think, you
• 50:33
  know, redefining how you see yourself as this Craftsman, as
• 50:37
  this manager and like, where are you actually getting value?
• 50:39
  And just knowing where the value is and just going for that.
• 50:43
  That's what you got to do, right?
• 50:46
  Because you know, you know, you just think about like there's 2
• 50:51
  problems that obviously there's many, but I'm trying to reduce
• 50:55
  it to 2 problems that the AI world is working on.
• 50:57
  The first one is like, can you produce output that's, you know,
• 51:02
  like problem free, like no bugs. It's you know, what I'm saying
• 51:05
  is generally correct or rather like it, it works.
• 51:09
  And the other one is like, is the work you're doing correct in
• 51:13
  that? Like is it directionally what,
• 51:15
  what you want? Like is it a, should I build a
• 51:17
  Mac OS app or should I build a web app?
• 51:20
  Like that takes subjectivity and taste.
• 51:22
  Whereas the first one is more like, you know, if I asked you
• 51:25
  to do a thing, did you do it right?
• 51:28
  If your whole world is around the first problem, I mean, I've,
• 51:33
  there's, I don't have good news for you.
• 51:34
  Like, like it is you have to assume that AI will just get so
• 51:39
  good at solving that first problem that you need to think
• 51:43
  about like, OK, what happens with the second problem, which
• 51:45
  is like, you know, that's where the subjectivity and taste piece
• 51:48
  comes in and the real world constraints that you can't just
• 51:51
  build everything overnight. People can't take it.
• 51:54
  Then I think you want to be like, how do I be the purveyor
• 51:57
  of taste and how do I be the one who's really good at, like
• 51:59
  making these decisions? Because I have this thing that
• 52:03
  can actually produce working artifacts, but how do I make
• 52:08
  sure it's better correct working ones?
• 52:10
  And continuing along those lines, do you ever feel like
• 52:17
  Cleric could just be abstracted into a skill?
• 52:21
  Could be. I mean I, I think.
• 52:24
  Like. I think our job as founders is
• 52:27
  like to constantly think about what is a durable problem and
• 52:32
  what is a problem that you should not be spending an iota
• 52:36
  of time on, right? Like, like when we started my,
• 52:40
  my big rule to everybody was like, we are not fine tuning
• 52:43
  models. Like this is like ludicrous.
• 52:48
  People will figure this out. They're going to get way better
• 52:51
  at like building way better models.
• 52:53
  Just assume the models will get better.
• 52:54
  I know you hate screaming at the model with JS, like please give
• 52:57
  me JS on, but this will be fixed.
• 52:59
  Don't worry. Like spend time on the agent.
• 53:01
  And then our whole thing like last year was like the agent
• 53:03
  layers commoditized. Like it's a waste of time.
• 53:05
  Like I think there's going to be way better harnesses than we can
• 53:08
  ever build. There's going to be like better
• 53:10
  agents and all that stuff. So then, you know, what we're
• 53:12
  thinking about is like very much on, you know, the verifiability
• 53:14
  and learning and things like that.
• 53:17
  I think the same time next year I'm going to be saying something
• 53:19
  different. Like that's the point, right?
• 53:20
  Like you don't build something that everybody else is going to
• 53:23
  do better than you. Like build a thing that's hard
• 53:26
  in your domain that's going to last.
• 53:27
  And like, I think the really tricky thing with AI is that
• 53:30
  that thing is changing every 12 months, but then you got to be
• 53:33
  primed to keep making that jump every time.
• 53:36
  You will talk to me about what 2027 looks like it you kind of
• 53:40
  mapped that out in this blog post that you wrote.
• 53:42
  Right. I was trying to think about like
• 53:44
  what this world looks like. And to me it's almost like, you
• 53:50
  know, you're, you're waking up and you've, you've had all these
• 53:54
  agents that have been working, you know, for you and then
• 53:57
  they're sort of surfacing exceptions.
• 53:59
  So maybe your coding agent shipped a change like to a
• 54:04
  config and something ended up breaking at 2:00 AM and your SRE
• 54:10
  agent notices this reverses the change and watches what happened
• 54:17
  until the metrics seem to be fine.
• 54:20
  And then you when you're waking up, you're just getting a report
• 54:22
  that you know, this is what happened.
• 54:24
  Here's the decisions I took. Like, you know, I noticed that
• 54:28
  the coding agent did X, so I decided that this was wrong, so
• 54:32
  I reversed it. And so, you know, hope that's
• 54:35
  OK, Don't worry like everything's fine.
• 54:37
  Or it could be like it's about to make a mistake because it
• 54:42
  doesn't know. So it's like surfacing kind of
• 54:44
  like a question to you, like should I roll back?
• 54:47
  Because I don't know, this is uncharted territory.
• 54:50
  But that's almost like to me, the interface that we will be
• 54:53
  operating with these agents. So to me, there's kind of two
• 54:56
  interesting parts there. One is I call it like the
• 55:00
  one-on-one problem, which is like as a human, like at what
• 55:04
  level am I doing my one on ones with my agents?
• 55:08
  And that's like the policies that we're setting, right?
• 55:10
  So you know, like it's OK to roll back if X happens.
• 55:14
  This is the way we build things, things like that.
• 55:17
  And then the other one, I call it like the stand up problem,
• 55:19
  which is I think it's going to get up maybe a little bit more
• 55:22
  interesting where once you have teams of agents, I'm not sure
• 55:26
  we've done a great job yet as an industry to have these agents
• 55:28
  working together, right? Like today, you know, Cleric's
• 55:32
  an SRE agent. And I've just talked about how
• 55:34
  it's totally different to the coding agent, but it's so
• 55:37
  obvious that we have to work really well together.
• 55:39
  Yeah, and how do you share context?
• 55:41
  How do you share context things like?
• 55:42
  That how do you work off of the same planning, right And you're
• 55:47
  able to cross one thing off or yeah, that's that is a really
• 55:52
  interesting one with with the proper isolation, right.
• 55:57
  So it's a little bit tricky in that regard because you want it
• 56:00
  to be isolated, but you want it to also share and share
• 56:03
  resources when needed. And so it gets, yeah, messy.
• 56:09
  I mean, I, I think, I think what's been really interesting
• 56:12
  about just thinking about agents is that there's so much overlap
• 56:15
  with like how we've solved the same problem for humans.
• 56:19
  And like, you can borrow a lot of that stuff.
• 56:21
  Like, I don't think we need to reinvent the wheel in this case.
• 56:24
  You know, you forget about agents.
• 56:26
  Like I've got a product engineer.
• 56:28
  A product engineer's main motive is to increase the value of the
• 56:32
  product to their customer. They're not.
• 56:35
  I mean, hopefully they are, but they generally don't think about
• 56:38
  security and reliability. They're just like, ship this as
• 56:40
  fast as I can get my promotion. And it's always like, whoa, what
• 56:45
  are you doing there? Slow down like this.
• 56:47
  This could break something and that's going to be bad.
• 56:50
  Security will have the same thing, right?
• 56:52
  But then like, how do they work together?
• 56:54
  So, you know, there'll be something like if it does a big
• 56:57
  change, maybe the product engineer will be like, hey, I'm
• 56:59
  shipping APR. Can one of you guys have a look
• 57:01
  and let me know what you think? Pretty sure it's OK, but you
• 57:04
  know, let me know. There'll probably be some
• 57:07
  conventions that the SRE has written, like here's how you do
• 57:10
  incidents, here's how you investigate.
• 57:11
  Yeah, I was going to say that that feels very synchronous.
• 57:15
  And there's probably a world where it's happening in parallel
• 57:19
  or it's they're happening together or yeah, like
• 57:23
  conventions might be a very rudimentary way of looking at
• 57:26
  that. Or they're just one is writing
• 57:30
  the code and the next thing you know, there's not even like APR
• 57:32
  that needs to happen because one writes the code, the other one
• 57:35
  comes and refractors it as it's happening, right?
• 57:40
  But I think like, at least on the human side, it's more the
• 57:43
  recognition that you have these two entities who are experts in
• 57:46
  their own domain. And we've found very slow ways
• 57:50
  for them to talk to each other. But like we, we agree that these
• 57:53
  are two separate entities because they're very separate
• 57:55
  objectives. So they they do belong in the
• 57:57
  org, but you know, they've got to talk.
• 58:02
  So I don't, I just think that, you know, by 2027 we'd
• 58:06
  definitely, whether I figure this out, like I know, like
• 58:08
  we're thinking about this deeply, like how do we integrate
• 58:10
  even better with, you know, the cloud codes and the codexors of
• 58:13
  the world? And how do we actually bring
• 58:16
  back all the knowledge that we've gained from production so
• 58:20
  that, you know, you're giving it to the coding agent.
• 58:23
  So. The coding agent doesn't make
• 58:25
  the same mistake exactly. Yeah, that to me is value,
• 58:28
  right. Like that's, you know, like one
• 58:31
  of the things which is like my big bet beeve is, you know, the
• 58:36
  whole point of SRE is you're trying to protect the
• 58:40
  reliability for your customers. It's not to reduce incidence.
• 58:44
  Like that's a byproduct, right? Like the whole point is like
• 58:46
  that's your main goal. So if you can prevent the
• 58:49
  incident, that's what you should be doing, right?
• 58:51
  So seeing like any model where it's like, oh, I improve MPTR
• 58:56
  and I've like I'm on jumping on every incident my F.
• 58:59
  One score, it's just like the back in the day with the machine
• 59:03
  learning models and being like, wow, my F1 score's amazing.
• 59:06
  And then you're like, actually nobody uses that model, so.
• 59:09
  Where you missed the prime. Yeah, exactly.
• 59:11
  Exactly. Oh man, I did want to ask you
• 59:16
  about how you're bringing the learnings back from production
• 59:21
  into the coding agents. So I, I mean, I, I think it's
• 59:25
  rudimentary right now. So today you can have the coding
• 59:29
  agent speak to Cleric through MCP and, you know, you kind of
• 59:34
  like get Cleric to weigh in on what it's on, what he thinks.
• 59:40
  We also have, you know, like these conventions that you can
• 59:44
  have in the repository, which we know that they're coding agents
• 59:46
  also use things like that. But I don't think that that's
• 59:52
  where it's going to end. I think we, you know, like
• 59:54
  there's already kind of attempts.
• 59:58
  I know that Google's working on A to a I'm not sure how much
• 1:00:01
  like the adoption has been, but like you, we're trying to create
• 1:00:04
  these open stands for these agents to talk to each other,
• 1:00:07
  right? We haven't adopted any of these
• 1:00:09
  yet, just to be upfront, but I think they will become more and
• 1:00:12
  more of a thing because to your point, you don't want it to be
• 1:00:15
  synchronous, you want it to be like info.
• 1:00:18
  Yeah. And how do you just keep it so
• 1:00:20
  that it's not blowing up the context when you do have
• 1:00:27
  whatever it is that you want to give it as much information
• 1:00:31
  about what you've learned from production, but you don't
• 1:00:34
  necessarily want to give it everything because then you're
• 1:00:37
  just blown up the context every time.
• 1:00:39
  And so is it where you need to even add it to the context or
• 1:00:44
  you just have that sub agent that's coming in behind every
• 1:00:47
  coding agent that is your sub agent, SRE agent that says, Oh,
• 1:00:52
  yeah, I know about these things. And I know that this happened
• 1:00:56
  recently and it has all of the necessary information.
• 1:01:01
  And like, that also doesn't feel like the right way to do it, you
• 1:01:06
  know, so have some clean up agent that is constantly going
• 1:01:11
  behind the coding agent. And it's just like, but how do
• 1:01:16
  you make sure that you do get that 360 degree?
• 1:01:21
  You have the security, you have the, the stability, you have all
• 1:01:25
  of these things that you're checking for.
• 1:01:29
  I, I've heard when we had Sid do the coding agent conference, he
• 1:01:37
  said that one thing that they'll have is an adversarial agent
• 1:01:41
  that will try and just, you know, find problems with
• 1:01:45
  whatever it is. And so that's one way that
• 1:01:47
  they'll do it, but really relaying the information from
• 1:01:54
  production to recognize that, hey, this happened.
• 1:01:58
  It's the second time that this has happened.
• 1:02:00
  Let's not have a third time, right, guys?
• 1:02:03
  Come on, make sure. I just, I want to see what the
• 1:02:10
  dynamics are of the agents that you wake up to.
• 1:02:14
  And the SRE agent is just Debbie Downer all the time.
• 1:02:17
  Like you know this PM the agent is constantly shipping crap and
• 1:02:24
  I have to clean up after. What is this?
• 1:02:29
  Wait, you don't like that's happening now And it's already
• 1:02:31
  just, it's, it's hilarious. Like we, we have a sub agent
• 1:02:36
  that we call the investigation auditor and the investigation
• 1:02:41
  auditor's job is to be Debbie Downer.
• 1:02:44
  It's like you have not done a good enough job.
• 1:02:46
  Keep going. And he is like, why he's screwed
• 1:02:48
  up and things like that. And then, you know, sometimes
• 1:02:51
  the main agent will complain saying like this is like too
• 1:02:54
  many Times Now I, I think I've done the best I've heard.
• 1:02:58
  So I'm just going to return back to the user.
• 1:03:01
  Yeah, it's, it's, it's really funny just reading through these
• 1:03:03
  traces. But it's not even mentioned that
• 1:03:06
  because I think, you know, like stuff I talked about at the
• 1:03:09
  beginning, like, you know, the Ant colony and things like that.
• 1:03:11
  Like, I think we've got to the point where we're now doing a
• 1:03:14
  lot of experimentation on what are the right abstractions,
• 1:03:17
  right? Because I think, you know, for a
• 1:03:19
  long time it's been sort of the main agent, only then it was
• 1:03:23
  main agent, sub agent. Like I think and topics are
• 1:03:26
  releasing. I think it's still in beta where
• 1:03:29
  you actually have like Asian teams, which are some totally
• 1:03:32
  different agents and they don't share any context and they're
• 1:03:35
  just like talking to each other through, you know, some form of
• 1:03:38
  protocol. So I think we'll just have more
• 1:03:40
  and more of these things. I think, I think to me like the
• 1:03:44
  coding agent, SRE agent, the team of agents paradigm makes
• 1:03:49
  much more sense because, you know, coding agent doesn't need
• 1:03:53
  to know the internals of the SRE agent.
• 1:03:55
  The coding agent just need to know that, hey, that's his
• 1:03:57
  expert that I need to talk to you about.
• 1:04:00
  Yeah. And you know, but you know, I'm
• 1:04:03
  not, I'm not going to totally listen to this agent, but I just
• 1:04:07
  want to get some, you know, like, isn't that like a kind of
• 1:04:10
  a consulting thing where it's like the racing matrix, like
• 1:04:13
  responsible, accountable, consulted, informed.
• 1:04:17
  I think you'll have that for agents, right?
• 1:04:19
  Like, you know, you must listen to the SRV agent or no, you may
• 1:04:23
  consult the SRV agent and things like that.
• 1:04:25
  So I think these kind of like governance structures that we
• 1:04:28
  come up with are going to be very, very valuable.
• 1:04:30
  Yeah, a lot of the software that we use and love is going local
• 1:04:38
  1st. So as opposed to me building
• 1:04:41
  something and exposing it to the world, I build something that I
• 1:04:45
  exposed to myself. And so because of that, I don't
• 1:04:48
  necessarily need to have all the security and all of the sure, it
• 1:04:52
  doesn't need to be battle hardened for production if it's
• 1:04:54
  just me as the main consumer because it's very bespoke
• 1:04:58
  software. And now, even more so, I'm
• 1:05:03
  looking at it like, wow, a lot of things that I used to build,
• 1:05:08
  I don't even need to build anymore because I just create
• 1:05:10
  them as a skill. Sure.
• 1:05:13
  And so it doesn't require a full-fledged app, it just
• 1:05:18
  requires a very clean skill, right?
• 1:05:22
  And so the, the act of moving from production grade software
• 1:05:27
  to skill, which is very, very thin.
• 1:05:31
  And it's like the lightest form of a application possible,
• 1:05:37
  right? It's just markdown, right?
• 1:05:39
  So that's, that's kind of how I look at it.
• 1:05:42
  Not to say that the software is not, but like, I still feel like
• 1:05:49
  there's a lot of very valuable software out there.
• 1:05:53
  And so I'm not fully bought into the software's cooked thing.
• 1:05:57
  There's just a class of software that you just never going to
• 1:06:00
  write code in my view like. Payments processing, Stripe,
• 1:06:06
  you're not gonna. That that's a good one.
• 1:06:08
  But like, you know, some, some like like Big Query, for
• 1:06:11
  example, right? Like we dump a lot of our data
• 1:06:16
  in there just for analysis, things like that.
• 1:06:19
  I'm not gonna wipe code that like, I mean, it's just doesn't
• 1:06:23
  make any sense. Like the this is a product that
• 1:06:25
  probably took years and years and years to get right.
• 1:06:29
  And there's almost like a level of accuracy that you expect.
• 1:06:32
  Like when I write a query, I get it exactly like what I want back
• 1:06:36
  and it's performant and all that stuff.
• 1:06:39
  So, you know, physics still applies, right?
• 1:06:41
  Like there's cost of having an engineer thinking about writing
• 1:06:45
  the software, making sure you're maintaining it because it's,
• 1:06:49
  it's always been true with software that the easy part is
• 1:06:51
  building it. The hard part is maintaining it.
• 1:06:53
  That's the reason we exist. You build a whole bunch of shit,
• 1:06:56
  it's in production now, you got to maintain it.
• 1:06:57
  And now most of the engineers are spending time maintaining it
• 1:07:00
  and not actually getting to create.
• 1:07:01
  So our whole value property is, you know, you can go create now
• 1:07:05
  because we can take some of that work, but the cost is real,
• 1:07:08
  right? But yeah, you're right.
• 1:07:10
  Like there's going to be, you know, but some software where, I
• 1:07:14
  don't know, there's like 20 features, but you only need one.
• 1:07:18
  Maybe that's just the skill. Maybe that's like a little app
• 1:07:20
  that you build. I think where things get more
• 1:07:23
  interesting to me is like in that, in that like kind of
• 1:07:26
  middle ground where I feel, and I'm pushing for this internally
• 1:07:32
  a lot where we want to identify everything we do.
• 1:07:37
  So I mean, I, I remember like like a year ago, William and I
• 1:07:42
  were just really frustrated by how long it took to get a
• 1:07:46
  website update done. It just, it was like, you have
• 1:07:50
  so many ideas, but I was like, you need a, there's a meeting,
• 1:07:53
  you know, talk to this person, that person, blah, blah, blah,
• 1:07:55
  blah, blah. And I think like a few months
• 1:07:58
  ago, like, all right, that's it. We're moving to Astro.
• 1:08:01
  Everything's going to be code. A whole marketing site is
• 1:08:04
  basically an Astro site now. It's no longer on any kind of no
• 1:08:07
  code platform. And man, it's amazing, like.
• 1:08:10
  You're just so much faster. So much faster.
• 1:08:13
  That's wild. It's and so so that's going to
• 1:08:16
  happen, right. Like it's like if your platform
• 1:08:21
  is not agent native in that agents like talking to you and
• 1:08:26
  working with you, then that's that's trouble.
• 1:08:30
  Yeah, right. The part that is worth
• 1:08:35
  considering is again, what's durable?
• 1:08:38
  What are the things that you're not going to vibe code?
• 1:08:41
  Sure, versus what are the things that you could probably get away
• 1:08:44
  with vibe coding? Or what are the things that you
• 1:08:48
  definitely need to be a gentic first or you want to be able to
• 1:08:53
  use with coding agents, You want to be able to have that seamless
• 1:08:56
  experience with coding agents. And so like looking at those
• 1:09:00
  that Spectrum is another good one.
• 1:09:03
  And I've, I feel like the system of records thing that's probably
• 1:09:09
  not going anywhere because that's pretty hard to do
• 1:09:13
  Databases, like you're saying, Big query is another hard one.
• 1:09:17
  Like what you really are going to go vibe code to database.
• 1:09:20
  There's very performing databases and they've been
• 1:09:24
  performing. Tested through like years and
• 1:09:28
  years and years, yeah, I think it's almost like a dimension
• 1:09:30
  thing, right? Which is like, how hard is it to
• 1:09:33
  actually build? And what is the friction it
• 1:09:37
  creates for you to get an agent to work with it?
• 1:09:40
  And then if that friction is high enough and the amount of
• 1:09:44
  time you need to build it. Is.
• 1:09:46
  Oh, that's a nice little diagram, yeah.
• 1:09:48
  Now that's going to be white coated.
• 1:09:49
  I'm sorry, but it's it's gone. You're feeling that frustration
• 1:09:54
  of like, I really want my agent to be able to plug into this.
• 1:09:58
  Yeah. And could I build it?
• 1:10:01
  Let's try. All right, Because I want to be
• 1:10:03
  able to have something that my agent can plug into easily.
• 1:10:06
  If my agent builds it itself then it's definitely going to be
• 1:10:09
  able to plug into it. 100% like, I mean, I at this point, I mean
• 1:10:13
  I've, I've, you know, I have an agent that I've connected to
• 1:10:17
  like all my stuff like my Gmail, my, you know, hotspot and
• 1:10:22
  everything. And I'm constantly looking for
• 1:10:25
  like OK how do I make this thing faster and better?
• 1:10:29
  You know what the biggest one for me was?
• 1:10:32
  My buddy JQ said, hey, we got rid of linear.
• 1:10:35
  We just went to all markdown files.
• 1:10:37
  Our whole thing is all markdown files and agents are able to do
• 1:10:41
  whatever with the markdown and they update it and they keep
• 1:10:45
  state on how the files are going.
• 1:10:48
  And so he's like, we realize like, what are we paying all his
• 1:10:52
  money to linear for? We don't even need it because I
• 1:10:55
  don't he he was saying, I realized I wasn't going into
• 1:10:58
  linear and looking at it and all I was doing is asking the agent,
• 1:11:03
  Hey, what's the status on this linear issue?
• 1:11:05
  And so there was no need to actually have linear.
• 1:11:09
  But then the funniest part and it's full circle moment, he
• 1:11:13
  said, yeah, so now we realize there are people that like to
• 1:11:17
  see the status. So we're now vibe coding a front
• 1:11:21
  end. So I'm like, wait a minute.
• 1:11:24
  So I mean the irony would be if you, I guess that's what you're
• 1:11:27
  implying with going full circle, you're kind of building linear.
• 1:11:30
  Yeah, in a way that is much more lightweight and he doesn't
• 1:11:34
  necessarily need like linear. I thought it was durable, but it
• 1:11:39
  at the end of the day it's not really durable, right.
• 1:11:41
  Like all of your, your status can be marked down and so if
• 1:11:46
  everything is marked down and then it can grab context from
• 1:11:50
  other markdown files. So if you're you really go all
• 1:11:53
  in on a file system, then why do you need linear to be the middle
• 1:12:01
  man? I would disagree that, but like
• 1:12:04
  I I think for everybody it's different.
• 1:12:06
  Like to me, in that same dimensions, linear is very low
• 1:12:11
  friction for an agent to use. I think the we use it
• 1:12:16
  internally. So although I do wish to release
• 1:12:19
  an official CLI, I'm not a big fan of the MCP.
• 1:12:23
  But generally, there's so much training data on linear that
• 1:12:29
  agents are pretty good at using them.
• 1:12:31
  And yeah, the, the effort to wipe coat it is probably not
• 1:12:34
  that high. But since the friction is so
• 1:12:35
  low, you're just like, it's not really worth my time to be
• 1:12:39
  thinking about this thing. I'd rather go focus on something
• 1:12:42
  else, right? So like to me, it's like the
• 1:12:47
  friction has to be really high that it's like actively impeding
• 1:12:51
  me for me to think it's worth going ahead because and to the
• 1:12:55
  point now, am I building a linear alternative or am I
• 1:12:59
  building my business? That's what I'm thinking.
• 1:13:01
  I wonder if it is for JQ. The friction was just in him
• 1:13:08
  seeing the bill. Could be, yeah, it could be
• 1:13:10
  that. Was what he was like, man, what
• 1:13:12
  is this like it it just eats at him.
• 1:13:15
  And so maybe it was just like a a little bit of a side project
• 1:13:18
  that became a a bigger project and he realized wow, all right,
• 1:13:22
  let's yeah, let's just see. And if you look at the bill, I
• 1:13:25
  don't think it's that expensive, right.
• 1:13:28
  So he really probably was like, let's just try.
• 1:13:31
  But he also is trying to get agents doing everything.
• 1:13:37
  Like he has five different names of agents in his slack that
• 1:13:41
  he'll call on to do things. You know, because they're the
• 1:13:45
  different agents that oh, this is the whatever linear knock off
• 1:13:49
  agent, right, right. Linney Linney Yeah, like, but
• 1:13:58
  but actually it's ironic that we're talking about linear
• 1:14:01
  because I think this CEO of linear, I don't I can't remember
• 1:14:04
  his name, Carrie. I think he's probably got one of
• 1:14:07
  like this court that I, I really liked.
• 1:14:11
  He talked about how like a whole point of building a company is
• 1:14:17
  like you got to be focused on your main quest side quest will
• 1:14:21
  keep coming, but you got to focus on your main quest.
• 1:14:23
  Like, that is like your number one thing as a as a founder.
• 1:14:27
  And the thing with the agents is that the side quests have become
• 1:14:30
  really tempting. Yeah.
• 1:14:32
  'Cause you can, you know, you look at like what you're, you're
• 1:14:36
  like, I'm just like reflecting as you know.
• 1:14:38
  Yeah, what JQ did, he was like again.
• 1:14:41
  I don't know, like it could be like a so I can't I can't speak
• 1:14:44
  to like whether there was a side quest or not.
• 1:14:45
  Like it seemed yeah, like I don't want to roast him on like
• 1:14:48
  something I don't know. But like one of the things which
• 1:14:51
  I, I've I've seen is that being an engineer founder is actually
• 1:14:56
  one of the hardest things because like coding feels so
• 1:14:59
  good. You feel like you're being
• 1:15:01
  productive, you're getting shit done.
• 1:15:04
  And when you see stuff like this, you can be like, Oh, I can
• 1:15:07
  help my team. I'll just, you know, build this
• 1:15:09
  thing. But then like to me, it's like,
• 1:15:11
  well, that's not what's going to make Derrick successful.
• 1:15:15
  I I don't think, you know, like my team needs me to do like a so
• 1:15:19
  the only thing like like we've really like wide coated
• 1:15:24
  seriously outside like cleric is like this, like this one off
• 1:15:31
  kind of MCPS or CL is that makes it easier for us to maybe like
• 1:15:35
  respond back to a customer or something like that.
• 1:15:37
  But all the tools that we're building is really either to
• 1:15:40
  improve the CICD or something like, but if it's does a thing,
• 1:15:45
  it's working and it's not crazy expensive science living.
• 1:15:49
  You know, it's the exact opposite of what my talk was in
• 1:15:54
  the coding agent conference, cuz I, my whole talk was called
• 1:15:58
  enjoy the side quest. Right, right, right.
• 1:16:00
  And so it was saying how I gave the example of basically me
• 1:16:06
  being like, oh, so I wanted to build an app and I wanted to
• 1:16:12
  schedule it. So then I Start learning about
• 1:16:14
  like GitHub workflows and boom. So then that's cool.
• 1:16:18
  But then I wanted to make sure that it was using a local model
• 1:16:27
  because it started to get too expensive when I was sure doing
• 1:16:31
  this. So then I grabbed and I, I
• 1:16:34
  started learning about like quantizing models and run it on
• 1:16:37
  my own machine. Then I was like, ah, well,
• 1:16:39
  actually, you know what, I need to get a GPU.
• 1:16:43
  So then I Start learning about hosting GPU's and this and that.
• 1:16:46
  And so then there's this whole side quest that I go on.
• 1:16:49
  And afterwards I started out just wanting to build like an
• 1:16:53
  RSS reader. I ended up, you know, an expert
• 1:16:56
  in Kubernetes type thing and. So that's awesome, but that
• 1:17:01
  that's almost like I think that it's almost like, you know, if
• 1:17:05
  you're doing your main thing, then if you recognize the side
• 1:17:09
  quests of what it is, that's very different to deluding
• 1:17:13
  yourself to saying that side quests actually helping a main
• 1:17:15
  quest, right. Like I wipe good stuff on the
• 1:17:19
  weekend. Like I've, you know, like the
• 1:17:21
  Asian. Like I because a big part of it,
• 1:17:24
  I mean, I tell myself is because I want to be in touch with agent
• 1:17:27
  and stuff, but like, who am I kidding?
• 1:17:28
  Like I enjoy it. Like I, I like it, but you know,
• 1:17:31
  I want to, but I I just don't want to tell myself that Oh
• 1:17:35
  yeah, I did all this stuff and like, that's definitely pushing
• 1:17:38
  Turkey. Like it's clearly not.
• 1:17:40
  Right. It's it's just a hobby.
• 1:17:42
  It's another hobby. And obviously awesome.
• 1:17:45
  Like you learn so much, like right, like I you and you never
• 1:17:49
  know what you don't know you. You're trying new stuff and then
• 1:17:51
  you learn something. I'll tell you what man, there is
• 1:17:54
  no better feeling, no bigger dopamine hit.
• 1:17:58
  Yeah, when people talk about social media giving you dopamine
• 1:18:02
  vibe coding. Oh, I saw that that was.
• 1:18:04
  A good post yes and and getting an open source and basically
• 1:18:08
  vibe coding a bug fix and then getting your PR merged to main.
• 1:18:14
  That's some dopamine right there.
• 1:18:16
  That is just I don't miss that. That's our new slot machine.
• 1:18:20
  That is what we've got. I think that's part of like why
• 1:18:24
  it's so addictive, right, Because it feels so productive
• 1:18:28
  and there's nothing like I I've always like this is like total
• 1:18:32
  tangent, but like some of the more popular YouTube videos are,
• 1:18:37
  you know, people who go and like mow the lawn, you know, like
• 1:18:42
  people have like you. Just have this need to just see
• 1:18:45
  something satisfying. Like something gets schemed up
• 1:18:48
  dude I. Actually got into really deep
• 1:18:52
  cleaning of my car because of too many damn TikTok videos and
• 1:18:57
  so I would break out a toothbrush and get in the all of
• 1:19:01
  the cracks. And crevices?
• 1:19:02
  And did you take a before and after?
• 1:19:03
  No, I wouldn't film myself, but I just the oddly satisfying part
• 1:19:09
  of watching someone do it. I just got, I got influenced,
• 1:19:13
  you know, to do it myself. And so now my hobby has become
• 1:19:19
  deep clean in the car, which we make a day out of it.
• 1:19:24
  I bring my kids and have them work on it too, because they're
• 1:19:28
  the ones who make it dirty. They might as well help clean it
• 1:19:30
  up. Right, but it also sounds like
• 1:19:32
  you're deeply in this open source repository too with the
• 1:19:35
  PR, right? Like that's my point.
• 1:19:38
  You know. Like it's like this thing of
• 1:19:39
  like, oh, I fixed it. I made it a little bit better.
• 1:19:41
  Yeah. And so you see that and it's
• 1:19:43
  oddly satisfying. And then you get that dopamine
• 1:19:45
  hit and you go, I helped, I did my part.