Latest / Agentic Conversations (formally mlops.community) / Spec Driven Development, Workflows, and the Recent Coding Agent Conference

Transcript

Spec Driven Development, Workflows, and the Recent Coding Agent Conference · Agentic Conversations (formally mlops.community)

• 0:00
  If your specification is so detailed that there's no
• 0:03
  possible way for the agent to, like, derail from that plan,
• 0:08
  there's no reason to essentially read the code because your
• 0:12
  understanding of the code is less important than the
• 0:16
  specification that you put in place that it has now adhered
• 0:19
  to. Dude.
• 0:26
  Well, should we talk about the conference?
• 0:28
  You came down and I'll set the scene for the listeners because
• 0:33
  you popped up, I think in Slack and you were like, hey, I'm
• 0:36
  coming down for the conference. It looks awesome.
• 0:39
  I'm pretty excited. And I was like, dude, that's
• 0:42
  cool. Where you coming from?
• 0:44
  And you said Seattle and I was surprised, I think that anyone
• 0:49
  was going to travel for the conference because I had
• 0:51
  basically put a bet on everyone that came to the coding agent
• 0:55
  conference being from South Bay. And like people from SFI wasn't
• 1:00
  even expecting to travel the hour to South Bay.
• 1:04
  Then when you said that, I, it started opening my eyes to like,
• 1:07
  Oh, well, maybe this conference is going to be pretty cool and
• 1:10
  it's going to be like people traveling from outside.
• 1:13
  Turns out a lot of people traveled.
• 1:15
  There was a guy that told me he traveled from Germany for it.
• 1:17
  So like hat tip to him. But then I, I told you like,
• 1:23
  well, I'm staying here and if you want, we're having a team
• 1:26
  dinner the night before. And so come hang out with the
• 1:29
  team. I don't even know what that
• 1:31
  meant. I was like, what team?
• 1:35
  I didn't. I didn't even know if I realized
• 1:37
  that you were the actual community leader for ML OPS.
• 1:40
  I thought you were just like someone I met on slack when I
• 1:44
  drunk. Just were like sure, dinner on
• 1:46
  beforehand I'm going I'm in. That is amazing to me.
• 1:50
  You were you were saying yes and and then we met at the team
• 1:56
  dinner the night before. You got to hear all of our
• 1:58
  stress the night before the event.
• 2:01
  Like, Oh my God, that person didn't turn in their slides.
• 2:03
  What are they going to do? Or Dex give us 200 slides.
• 2:07
  How is this possible? He's not going to give a 20
• 2:10
  minute talk with 200 slides. That's like not physically
• 2:13
  possible. And spoiler alert, he did.
• 2:16
  So that's wild. And he did finish.
• 2:19
  Yeah, yeah. He finished on time and actually
• 2:21
  kind of early. So that's wild.
• 2:23
  And what I wanted to do here is just talk through some of the
• 2:27
  stuff that you've been thinking about and I've been thinking
• 2:31
  about since the conference happened.
• 2:34
  Like there's some key pieces for me that I remember and I learned
• 2:39
  from that event. I imagine there's some stuff for
• 2:41
  you, so let's like cross pollinate and talk about it.
• 2:45
  Maybe first I can talk about what I did as an asset for the
• 2:52
  post event. I took the transcript cuz we had
• 2:55
  the six hour live stream that's still up.
• 2:59
  But we're making individual videos right now and it should
• 3:01
  be out by the time that folks are listening to this podcast.
• 3:06
  So if you go to our YouTube, you should see all the individual
• 3:08
  videos. But the live stream transcript I
• 3:12
  took that I fed it through Claude and I asked it to
• 3:16
  identify areas where we talked about certain skills and what
• 3:20
  skills worked, what skills didn't work, and what our
• 3:23
  favorite skills are. Because throughout the day
• 3:25
  people were talking about this skill is great, this skill is
• 3:28
  great, blah blah, blah. And then I asked it to either
• 3:31
  link to existing skills if we were talking about existing
• 3:34
  skills like Claude Ception or something.
• 3:37
  And then if we weren't and someone explained the skill,
• 3:41
  create that skill. And so we now have all the
• 3:46
  transcripts in a GitHub repo. We have all these new skills
• 3:49
  that people were talking about. And since I have, we've been
• 3:53
  doing these lunch and learns on Fridays.
• 3:55
  I've just been continuing to add to that as we have lunch and
• 3:59
  learns and we talked about new skills that are very useful for
• 4:02
  us. I think it's awesome that you're
• 4:05
  able to now do all the things you can envision doing.
• 4:09
  If you could like think back to how you can parse all this data
• 4:14
  that you have. And so like I, I did something
• 4:17
  similar, of course, with my, my own setup.
• 4:20
  I was in the process of like, I was just going to write, you
• 4:23
  know, a simple blog post about like how my experiences went.
• 4:26
  And then next thing you know, I'm doing a lot more than I, I
• 4:30
  normally do, but I. Yeah.
• 4:34
  Yeah, I want to. I always go on big side quests.
• 4:37
  So I mean, I, I have a few research stocks put together.
• 4:41
  I didn't have it like prepared to talk about right now, but I
• 4:43
  was working on I'm right here. I'll see if I can pull up what I
• 4:47
  have. One thing I I found very
• 4:49
  interesting as I kind of started diving into all this in
• 4:51
  regardless has been just like understanding, like how to learn
• 4:57
  and how to understand like these concepts and like kind of going
• 5:00
  back to to like taking information in.
• 5:02
  But I don't know if you remember in the 2000's the scrolly
• 5:05
  telling style of websites. Do you ever?
• 5:09
  There was, I think it was a New York Times article or the
• 5:13
  Atlantic or something. And it was an article titled
• 5:16
  Into the Deep and it's called scrolly.
• 5:20
  So S you know, scroll E telling. And it's like a long form
• 5:25
  website where the information is shown to you in like
• 5:29
  infographics, but like in a dynamic way.
• 5:34
  And so it, it was like, anyways, it's a really cool way to mess
• 5:39
  around with vibe coding if you're like, looking to get
• 5:41
  something explained to you. So you ask.
• 5:45
  So when you ask Claude to explain it to you in the form of
• 5:49
  scrolling telling sort. Of the way I I process the
• 5:53
  information was I took all the the YouTube videos.
• 5:56
  You take those videos, download them, parse them, like take all
• 6:01
  the key frames using, you know, FFM PEG or something similar.
• 6:06
  And you split that up and grab the audio transcript.
• 6:08
  And then, you know, correlate together that context with the
• 6:12
  context from the images to kind of fill that in.
• 6:15
  And then, you know, gather any other information from the
• 6:19
  comments and you know, have it do additional research there to
• 6:23
  like how to fill that out. And then once I have that, then
• 6:26
  it's kind of, I have like a few different ideas, but basically
• 6:30
  just having a focus on breaking it out into like the way that
• 6:34
  you would think about in terms of semantic understanding of the
• 6:37
  concept. So knowledge graphs, mind maps
• 6:39
  or topic graphical kind of ordering of the information.
• 6:44
  And like, once you have it all split out into like that kind of
• 6:49
  a graph type network, it's very easy to then do stuff with that
• 6:54
  data, you know, and it's all about just kind of pre
• 6:57
  processing. And so once you have processed
• 6:59
  it and you have it in a way that you can now do something with
• 7:04
  there's you've taken out a lot of those steps of what you want
• 7:07
  to do elsewise. So like your skill idea is
• 7:09
  something you could expand on along with any other idea you
• 7:14
  have, you know, once you have your data in a way that you can
• 7:17
  easily mess with it. So, so that's kind of the main
• 7:23
  thing I always think about though is like, no, maybe, and
• 7:27
  it's can be too much of a thing to focus on, but like how to
• 7:32
  make it into a process so you're not like repeating the work.
• 7:35
  And it's also when you're not, you know, depending on how
• 7:38
  you're paying for tokens and stuff like that too, then it
• 7:40
  becomes a whole other conversation for how how much
• 7:45
  you want to mess with this, you know?
• 7:47
  Yeah, especially if it's like 6 hour live streams, but ten of
• 7:51
  them and non-stop so. Yeah, so I I just ran all that
• 7:56
  like through my local setup that I have and.
• 7:59
  Oh, nice. Yeah, I really, I like that you
• 8:02
  took it a step further in. You're referencing the images
• 8:06
  and you're also referencing the YouTube comments.
• 8:08
  I didn't even think about that because that can give you a much
• 8:12
  broader perspective and it can show you where questions are
• 8:17
  being asked or what types of questions are being asked.
• 8:19
  What's the discussion in the comments?
• 8:22
  I'll, I, I've throughout this whole time I've been kept saying
• 8:25
  I'm going to like publish something finally to GitHub or
• 8:27
  something like that. I'll maybe I'll try to publish
• 8:29
  my little YouTube video skill flow.
• 8:32
  I like that. So yeah, you just use it as a
• 8:34
  skill. I should be able to at least
• 8:36
  pull up the research document I had.
• 8:38
  I mean, I have a few of them from the conference now because
• 8:40
  I, that's the other thing is like, when do you stop?
• 8:43
  When do you stop saying like I need to stop, you know, messing
• 8:46
  with this. I, I always go a step further
• 8:48
  and then I'll usually go the step further until it's not
• 8:51
  working anymore. Or I have 30 versions and I
• 8:54
  don't know which is the one I liked anymore.
• 8:56
  And then I just kind of forget I did it in the first place.
• 8:59
  Let's see. But you so you've created all
• 9:03
  this stuff. Is there stuff that jumps out at
• 9:06
  you as like what you've been able to, maybe it's not what
• 9:10
  you've been able to do, but just things that stuck with you that
• 9:13
  you now do differently because of the conference?
• 9:18
  And so for me like. Oh yeah.
• 9:20
  I so basically the for me, the conference absolutely was about
• 9:23
  agents. I mean, that was, that was one
• 9:26
  of the biggest takeaways for me. And the ultimately the biggest
• 9:29
  take away for me was just from a personal standpoint, it, it's
• 9:34
  very hard in this kind of environment to really understand
• 9:36
  what it is you do or don't know. And, you know, you get a very,
• 9:42
  there's a very mixed community in terms of like who's open to
• 9:46
  talking about using AI to replace ourselves or, you know,
• 9:49
  whatever other existential questions we want to ask.
• 9:51
  But I just want to learn about it.
• 9:53
  I'm just excited. I love it's a lot of fun.
• 9:55
  And so yeah, that was my takeaways.
• 9:58
  The agents are, you know, the way everything's moving and so
• 10:01
  I've, you know, been talking to a few different companies and
• 10:04
  that's definitely one of the main focuses that they're
• 10:07
  looking for is people that understand the concepts of
• 10:12
  basically parallelizing work and everything's moving towards
• 10:17
  orchestration and evaluation of the agents and how to, you know,
• 10:21
  gather those metrics. And it's basically like all the
• 10:23
  things that everyone would kind of, you know, eventually want to
• 10:27
  lead to with the ML off side of things.
• 10:29
  But it's understandably a bit complex because it's not like a,
• 10:33
  you know, extremely qualitative type or quantitative type, the
• 10:37
  measurement you can take, so. Dude, actually something that
• 10:41
  came up, I remember you mentioning you were talking to
• 10:43
  my buddy Shiram from Cleric and he said to you like a it's
• 10:51
  really cool that you're here learning about this because
• 10:55
  they're trying to hireright and they want to hire people that
• 10:59
  are thinking about this and being at these types of events
• 11:03
  just because you're already attacking the problem from a way
• 11:07
  that they it's like their philosophy on how they think
• 11:11
  work should be done too. So I thought that was that like
• 11:14
  was something that I probably won't forget.
• 11:16
  And because I didn't realize that it was so hard.
• 11:19
  And this is naive. I I know people in Slack have
• 11:21
  been like, dude, we're in a bubble right now.
• 11:24
  You think that people are using coding agents, but really
• 11:26
  they're not Like half the folks don't even know what skills are.
• 11:30
  And the fact that he was sitting there telling you like, yeah, I
• 11:34
  want to hire my next hire as someone who is doing this on the
• 11:40
  weekends already thinking about like how they can paralyze sub
• 11:43
  agents. Yeah.
• 11:45
  And it, and I don't like to think of it just to kind of not
• 11:48
  focus on the weekends aspect of like, you know, looking for
• 11:51
  people that are spending like a bunch of extra time.
• 11:53
  It's just more of the, the interest.
• 11:55
  And like I, I did fly down there legitimately because I, I've had
• 11:59
  AI love going to conferences, I love learning and I've been
• 12:03
  watching a lot of stuff on YouTube.
• 12:05
  And at some point I was like, OK, I got to make sure that
• 12:07
  like, I was even sure if you're real, like, for all I knew you
• 12:09
  were going to be like an AI generated, you know, I don't
• 12:13
  know, like, I didn't know what I was getting myself into either.
• 12:16
  I'm flying to San Francisco to meet strangers at an online
• 12:21
  event that no one I knew of was going to or, and I I mentioned
• 12:24
  it to one person and they're like, what?
• 12:26
  So yeah, well, it all worked out.
• 12:29
  But you know, we don't know what's real sometimes.
• 12:32
  Still, we spent the whole week together and the jury's.
• 12:35
  Still, I'm like, I'm like 9995% sure, you know I that.
• 12:42
  Is hilarious. Yeah, I mean, we, I, I I saw a
• 12:46
  robot, made my coffee and drove me to my hotel down there.
• 12:48
  So I don't, I don't know. We did take a way mode.
• 12:51
  We took your first way mode and we found out that if you say way
• 12:55
  mode dangerously skip position or permissions and go on Yolo
• 12:59
  mode, it does some wild things in traffic because of those
• 13:04
  microphones are always on apparently, and you can guide
• 13:07
  it. And yeah, it was doing some
• 13:10
  weird stuff when we were in that car.
• 13:11
  But anyway, there was a few things.
• 13:13
  One was when Sid from Anthropic was talking about one of his
• 13:19
  favorite agent sub architectures or, or agent architectures was
• 13:24
  the adversarial agent to like always just kind of probe the
• 13:30
  code that's getting submitted and just be overlooking it and,
• 13:35
  and be like the bad guy. I thought that was cool.
• 13:38
  I never use a sub agent or an agent like that.
• 13:42
  And so now I want to think about like how I can continuously have
• 13:48
  those are adversarial agents spun up.
• 13:53
  What So, you know, what are the interesting things that I kind
• 13:57
  of go like agents being the overall kind of like thing to
• 14:01
  talk about. It's what is an agent?
• 14:03
  I mean, that would be kind of the question that it seems like
• 14:06
  such a simple question. But if I was to go into like one
• 14:10
  of those no stupid questions type rooms, like that would be
• 14:13
  my question because I want to hear what people's opinions are
• 14:16
  because it it's it's a, it's a very broad concept.
• 14:20
  And like I was asked that question like it was kind of
• 14:23
  like one of those things I was asked not like during an
• 14:26
  interview, but just during a conversation.
• 14:27
  It's like, what agents do you use?
• 14:30
  And I was just like all of them. I, I, I, I didn't know, like, I,
• 14:35
  I didn't know the right answer because it's like, yeah, I use
• 14:37
  Claude, I've used Gemini, I use codecs, I've used the ones on
• 14:40
  the CLI and all that things. But like, I think really though,
• 14:44
  the the two concepts are like you either have an agent that is
• 14:47
  like not in not one that you're talking to, or you have one
• 14:52
  that's running either all the time or it gets invoked, but you
• 14:56
  either have one you're like working with or otherwise.
• 14:58
  I mean, there's more than two probably paradigms there.
• 15:01
  But you know, it's like transferring these things from
• 15:05
  agent to agent or from Claude, you know, to the other one.
• 15:07
  It's like all the concepts make sense, but the actual, you know,
• 15:11
  mechanism to invoke that capability varies drastically.
• 15:15
  And that's, you know, part of the harness.
• 15:16
  So like with Claude and especially like as time went on,
• 15:20
  it became easier or harder or different to even invoke skills.
• 15:23
  Like I think when skills first came out, there was a lot of
• 15:26
  issues with people having, you know, to like issues with like
• 15:28
  invoking them and things like that.
• 15:30
  And, but like, I, I think it's also what was really helpful
• 15:34
  going to the conference though, is also bridging this gap
• 15:37
  between people that are now very familiar with like, you know,
• 15:43
  coding from the aspect of vibe coding.
• 15:47
  And you know, vibe coding can be a maybe a taboo or tensionless
• 15:51
  topic. But like, I think of it as just
• 15:53
  like using AI to AI assisted development.
• 15:57
  So like you, for example, you, you don't have a development
• 15:59
  background. I totally thought you were a
• 16:01
  developer. Like the way you, you know,
• 16:03
  you've you've using, you're using cloud, you're creating
• 16:06
  apps, like, you know, that's just a much more common thing.
• 16:08
  It's becoming like a common language, but it gets confusing
• 16:12
  when new people start using your common language, but maybe they
• 16:16
  don't have the same technical background that you're expecting
• 16:18
  because of the other vocabulary that you're using.
• 16:21
  And yeah, so like talking about sub agents and that things like
• 16:26
  it was kind of hard for me to understand how the conversation
• 16:30
  like has developed over time because that was like the very
• 16:33
  first thing I did when I got access to like chat TBT was
• 16:36
  like, OK, how do I put this into my command line?
• 16:38
  Like I don't want to copy paste. And then it was like, once I got
• 16:42
  that, it's like, OK, well, I don't want to do that either.
• 16:44
  Now I want it to like automate my browser to do this.
• 16:47
  And that's like, OK, well, you know, if I'm going to do this
• 16:50
  amount of work and this is my context window, well then I need
• 16:53
  to have one agent do this task and the other agent do this
• 16:56
  task. And then using the standard
• 16:58
  programming, you know, of software development practices,
• 17:02
  you fan out, fan in, you know, it's a kind of a common type of
• 17:06
  workflow. And so if you've spent time
• 17:10
  building like, you know, like AWS has a a service called like
• 17:14
  AWS step functions. And that's like a great example
• 17:16
  or just N 8 N would probably be an even better example of just
• 17:19
  like orchestrating A workflow and a step builder type thing.
• 17:23
  And then depending on what the task it is, you know, you can
• 17:26
  parallelize it. So when you're talking about
• 17:27
  agents and the sub agents, because it's one, it's the idea
• 17:31
  of using it like, yes, I would want to parallelize my work.
• 17:34
  And but the other thing is, no, you don't always want to
• 17:38
  parallelize your work. And two, even if you want to now
• 17:42
  you're kind of maybe struggling with the actual usage of this
• 17:46
  harness that you're using, which differs between vendors.
• 17:49
  So you have like those two layers.
• 17:51
  And I, I can't remember where I tanged it off here, but you were
• 17:53
  talking about sub agents and Claude.
• 17:55
  And like, I was just like, yeah, it's it's great to to talk
• 17:59
  about, but it's, you know, sometimes it's as simple as just
• 18:02
  saying like use a sub agent to do this.
• 18:04
  And so, and what was your discovery?
• 18:06
  I guess there I guess coming from like a non coding
• 18:08
  background like. Well, I think there was some
• 18:11
  cool things that happening with the sub agents or how folks are
• 18:16
  approaching sub agents and the, and I think I, I mistakenly the
• 18:22
  probably the reason you went off on that tangent is because I was
• 18:24
  mistakenly saying the adversarial sub agent when
• 18:27
  really I think it's a whole different flow.
• 18:29
  It's like folks use different agents for code or P Rs than
• 18:35
  they do to create the code. And so it's like the adversarial
• 18:40
  agent is just another agent that will be in that workflow
• 18:44
  checking the code as it's being created to try and poke holes in
• 18:48
  it. So whether that's a sub agent or
• 18:51
  not, I'm not sure if you want to architect it that way.
• 18:54
  But the other thing when since we are on the topic of sub
• 18:59
  agents, that feels pretty cool is both Rob from Brumi, he was
• 19:05
  there and he's creating Brumi. That is all about sub agents,
• 19:10
  right? That's or how to make using sub
• 19:12
  agents easier depending irrespective of which harness
• 19:18
  you're using or which LLMS you're using.
• 19:21
  And then the other piece is the WARP talk from Zach I thought
• 19:26
  was super cool too, because WARP has all of those ways that they
• 19:31
  made it really easy to use sub agents, but kick off like sub
• 19:37
  agents in the cloud. Or maybe they're not called sub
• 19:40
  agents. It would be like paralyzed
• 19:42
  agents. And yeah, I mean I and nothing.
• 19:45
  Yeah, like this is also where the terminology can become kind
• 19:50
  of a communication barrier too, because you know, what is an
• 19:54
  agent, what is a sub agent? And so for, for the sub agent
• 19:57
  part, I think sub agent would be basically you have it's a sub
• 20:02
  process basically. So you have the main agent and
• 20:04
  it's basically that same instruction set with maybe a
• 20:08
  different prompt but new context window that reports back and
• 20:13
  then. Any other agent you have is kind
• 20:16
  of where the landscape is now and that would be one, one word
• 20:20
  I have for a type of agent would be a Co agent.
• 20:23
  That's what I call my agents that talk locally to each other.
• 20:29
  I yeah, I see that then the the stuff.
• 20:33
  And that's like the, so the, the agent you were talking about
• 20:36
  with the, and so that the term and that for the adversarial
• 20:43
  agent, I kind of think of those as just like ephemeral agents.
• 20:46
  So they may or may not have state, but they are basically
• 20:50
  invoked as part of a, a, a trigger.
• 20:54
  And they are basically, you know, it's, it's kind of like
• 20:57
  LLM as a judge kind of thing where you're having another
• 20:59
  agent review another agents work and then provide the feedback
• 21:03
  for, you know, further, further parts in the loop, right?
• 21:06
  Yeah, yeah, that makes a lot of sense.
• 21:08
  In different steps you would want to invoke it and you would
• 21:12
  want to see just to like stress test things and the prompt
• 21:17
  should say be as brutal as Linus Torbach.
• 21:21
  On my throat well, I I I thought that doing the the eval stuff
• 21:25
  that they were doing in the talks I mean that was the the
• 21:28
  really interesting part is seeing how to actually that's
• 21:33
  still kind of a black box though is evaluating sometimes your own
• 21:37
  feelings of you know using like these things in the in the
• 21:40
  terminal but. Are you talking about like
• 21:43
  Jess's talk? I liked Jess's kind of E vows
• 21:46
  that she did. It was a little different.
• 21:48
  It wasn't necessarily on the coding agent side of things.
• 21:51
  It was on vector search versus grep.
• 21:55
  Do you remember that? And she talked about how like,
• 21:59
  yeah, grep does better in our test, but then she gave a huge
• 22:03
  caveat with like, but I only ran one test and I'm not like a
• 22:08
  vector search or embeddings expert.
• 22:10
  And so like, there's a lot of things that probably could be
• 22:12
  better and blah, blah, blah. And after that talk, we had to
• 22:16
  break and I saw her and the quadrant guys and the quadrant
• 22:21
  guys, obviously they're a vector database.
• 22:23
  So they were like, well, wait a minute, What embedding model did
• 22:27
  you use? How much compute did you give
• 22:29
  this vector search that you guys we need to talk about things?
• 22:33
  Because the TLDR of that to me felt like vector search is a
• 22:38
  much more powerful option. You have more knobs to turn so
• 22:44
  you can potentially get a way better solution.
• 22:47
  It just requires a little bit more time and effort.
• 22:51
  Well, it's like the the nuances there again, it's so hard to
• 22:54
  evaluate. Like there's a lot of questions
• 22:56
  that they they could ask. And I had a lot of questions
• 22:58
  too. And because, yeah, it does
• 23:02
  depend on the, the, the dimensions you use for the
• 23:07
  embedding model and the amount of context that gets returned
• 23:10
  back with it and whether or not your question is even necessary
• 23:13
  for it. You know, there's always going
• 23:14
  to be multiple ways to do it. And so doing the eval thing, I
• 23:18
  mean, there's a lot of ways that you could even consider that
• 23:20
  eval because are you concerned, you know, there's the token
• 23:26
  cost, there's the length, the time it took, there's whether or
• 23:29
  not it was accurate the first time or the second time or, you
• 23:33
  know, I mean, are we like you could also eval just based on
• 23:36
  like the, you know, the cost of the operation, you know, is it
• 23:42
  cheaper to just sign this work to?
• 23:46
  Yeah, I, I don't know there's. So yeah, there's so many pieces
• 23:50
  to it, right? And yeah, yeah.
• 23:53
  And how much compute do you need?
• 23:55
  How much time do you need all of that?
• 23:57
  And. And so like with the, you know,
• 23:59
  with the vector search, you can always throw in a bunch of
• 24:01
  resources at a problem and solve it.
• 24:03
  But, you know, at at some point you got to realize how much
• 24:06
  resources you're throwing at it. So it's, you know, using a
• 24:09
  vector search for doing code edits.
• 24:11
  I'm not sure it's always going to be a relevant thing to
• 24:15
  leverage, right, because you might not necessarily need the
• 24:17
  context of everything that you would leverage from doing that.
• 24:20
  You would probably be much better off with that.
• 24:24
  But also, so with the way that they were doing the code edit
• 24:28
  with her eval, I think because there's, there's you can use
• 24:32
  like an AST parser and then there's another, I, I can't
• 24:37
  remember the solution that they were using, but there I, I've
• 24:39
  seen this done a few times. What And so which there's a tool
• 24:43
  called Serena MCP. And this is a very similar type
• 24:46
  of thing where it uses a like a code sandbox to make code edits,
• 24:50
  but it uses the language server to like LSP to facilitate the
• 24:56
  code edits. But it's like that same exact
• 24:58
  thing that she was doing her evals on Serena is a similar
• 25:02
  tool. And it's it it is fun to mess
• 25:06
  with. And I've I've seen another MCP
• 25:09
  tool too out there where like reduces the token usage, but you
• 25:12
  might start introducing like the hallucinations and that's where
• 25:15
  the evals can get complicated and so.
• 25:19
  Yeah, there's no free lunch. Yeah, basically you gain on one
• 25:26
  aspect but you lose on another. And that's the fun part about
• 25:29
  figuring it out and having folks come and say, hey, here's what I
• 25:34
  learned in my little mini tests and my benchmarks.
• 25:37
  What have you seen or what do you know?
• 25:40
  So I think that that's a great one.
• 25:43
  You were going to talk about though, which one was evaling on
• 25:48
  the vibes? Did you want to hit on that or?
• 25:50
  I've got a few other things that stood out to me.
• 25:54
  I think there was a few cool hot takes that I like and that have
• 25:58
  stayed with me since. Yeah.
• 26:00
  So my biggest take away though from that was, I mean as we were
• 26:04
  talking about with with the agents and doing the evaluations
• 26:06
  and there's it's just not there's no one way to do this.
• 26:10
  No one has a solution. There are multiple people kind
• 26:13
  of doing similar things as you, you've seen with like open claw
• 26:17
  and the nano claw things like, not that those were talked about
• 26:19
  the conference, but yeah, those ideas are very simple in
• 26:23
  practice and our billion dollar ideas.
• 26:26
  So it's and, you know, that idea is just an agent in a loop.
• 26:31
  And that's what a lot of this is talking about, you know, and,
• 26:34
  and hooking them up to, to tools and things.
• 26:36
  And it's obviously more complicated than that.
• 26:38
  And the real problem comes when you want to have this, you know,
• 26:43
  outside of your own system. I, I think one of the things I
• 26:46
  was really surprised about though, is that there was very
• 26:48
  little discussion at all about like the security aspects or
• 26:53
  like the, the major concerns talk with Sam or yeah, like,
• 27:01
  like some, I mean, I, my, I, I, you could have various reasons
• 27:07
  to think why it's not discussed more.
• 27:10
  And I guess one of them could be is the more like as soon as
• 27:13
  Clawbot came out, you know, there's more people with access
• 27:15
  now to these autonomous things and there's more problems that
• 27:18
  happen. People had API keys exposed and
• 27:20
  things like that. But I just remember seeing in
• 27:23
  like December this talk from 393-C or is there some security
• 27:27
  conference thing? And it just showed using the
• 27:31
  clawed code agent with like a simple MCP web server and just
• 27:37
  to use prompt injection to have it click a link to install a rat
• 27:40
  to then get remote access to the machine.
• 27:42
  And he showed it using like 5 different prompt injection
• 27:45
  techniques to have it do it. And it was like every time, like
• 27:49
  even in like the times where it like gave him like 3, you know,
• 27:53
  responses, you know, blah blah, blah.
• 27:55
  He got it to just, you know, do it.
• 27:56
  And then once all it has to do is like visit that page and then
• 27:59
  boom, it, it gets infected and then you get full remote access.
• 28:03
  And that's for like a full wrap. I'm sure depending on the
• 28:06
  exploit that you're, you know, trying to exploit, you don't
• 28:11
  even need that much, you know, like, so anyways, it's it's very
• 28:14
  trivial. And, you know, a lot of this is
• 28:17
  now kind of more focusing on container containerizing stuff
• 28:21
  these run times. But also, yeah, it's just that
• 28:25
  side of things. That's where, you know, a lot of
• 28:30
  the like the eval side can touch on that more.
• 28:34
  I know that. And I'm sure we'll see more of
• 28:37
  that come up more in these talks though.
• 28:39
  So I you know, I'm sure as more of these as time goes on and
• 28:43
  more people are starting to follow the SRE patterns for or
• 28:49
  the adverse adversarial agents to be in their orchestration
• 28:53
  loop to help kind of like keeping the guardrails.
• 28:57
  Yeah, I just thought that the the fact that there wasn't a lot
• 29:00
  of security mentioned was was interesting.
• 29:03
  But also 11 kind of question I still have that I haven't really
• 29:06
  seen discussed is I can't imagine that people aren't using
• 29:11
  standard like best practices for data exfiltration.
• 29:16
  And you know, basically just like DevOps or security
• 29:22
  operations, you know, with your own like infrastructure, but you
• 29:25
  know, such as like getting your having an agent like visit a
• 29:29
  page that you don't want it to or something like that.
• 29:31
  You know, ultimately all this stuff can get filtered by API
• 29:35
  calls and you know, or your own proxies that you have in your
• 29:39
  own infrastructure. So it's like this.
• 29:43
  Everything can be a walled garden like if you want it to
• 29:46
  be. And I would imagine as more
• 29:51
  people understand the virtualization of these
• 29:55
  environments, the more it's going to move naturally toward
• 29:59
  that way. And there's going to be less
• 30:00
  people installing these with root access to the machine and
• 30:04
  there's going to be a lot more kind of proper facilitation of
• 30:08
  tool calling and standardization in that area.
• 30:11
  Yeah. We didn't have anybody from a
• 30:13
  sandbox company to give a talk and I would have liked to have
• 30:16
  seen a little bit more of that because that's.
• 30:19
  What wasn't, what was the name of that company?
• 30:24
  What's an M? The, the people that were
• 30:27
  building the file system for agents.
• 30:30
  Oh, yeah. This was I, I talked to these
• 30:31
  guys. This was a really cool thing.
• 30:34
  I, I don't know a lot about jiu jitsu, which is called JJ, but
• 30:39
  it's a, it's a Git. It's not a git.
• 30:42
  It's, is it a fork of Git? It's a super set of git.
• 30:45
  I know it's at least that it fully supports Git.
• 30:47
  And then it does a bunch of other things that I don't
• 30:49
  understand, but it allows the capabilities to build these kind
• 30:54
  of the, the, that's how that the, the blame feature kind of
• 30:58
  works. That I so, and, and this was
• 31:02
  also one of these ideas that like when I, I would love to
• 31:05
  understand why like just simple GPG keys and signed commits and
• 31:12
  kind of following that workflow isn't standard for identity with
• 31:17
  these agents. I mean, I'm not doing this now,
• 31:20
  but the idea I would have is that I have one SSH key or one,
• 31:23
  you know, private key. And then I would use subkeys to
• 31:29
  delegate out my permissions to agents so that like they're
• 31:32
  authoritative on my behalf, but I can still like track my evals
• 31:36
  or by metrics and stuff to individual agents or the blame
• 31:40
  thing here. But even on the, the blame thing
• 31:44
  for the agents, it's, it's, it's still an interesting thing
• 31:46
  'cause as you think about sub agents and you think about what
• 31:49
  is an agent and the agent identity, it can get almost like
• 31:53
  very existential. Because at what point have you
• 31:56
  defined, you know, the agent is the agent the prompt, is it the
• 31:58
  instructions, is it the member? I mean, it's all of it.
• 32:02
  It's the harness, it's the day, it's the time.
• 32:06
  It's, you know. So then how, yeah, how you look
• 32:09
  at that for that like agent blame and what it is and what
• 32:15
  you can take from it is, is. Yeah.
• 32:17
  So I that, that would be a question I would have loved to
• 32:19
  ask though is kind of where do you draw the line in terms of
• 32:22
  the traceability for, for that. Because yeah, as I've explored a
• 32:26
  little bit with the multi agent architectures and things like
• 32:28
  that, I've learned what I have found works best is having
• 32:35
  essentially less direct sessions, but more quality
• 32:41
  direct sessions that then orchestrate out from there.
• 32:45
  And that's kind of the idea I play with kind of like I'll use
• 32:49
  one harness to communicate with other harnesses, but then I'll
• 32:53
  do that multiple times for different work sets and that.
• 33:00
  But as far as then tying that all together with the age that
• 33:04
  I'm kind of building, like I have an idea in my head for kind
• 33:07
  of how to standardize on the definition of an agent and the
• 33:11
  question of like, what is an agent?
• 33:13
  And that's kind of where I was going with my, the blog post I
• 33:15
  was going to write is I started kind of looking into this and
• 33:18
  then I just started doing a lot and now it's Monday.
• 33:20
  So that's how it. Goes.
• 33:24
  You were going on a little bit of a like different route before
• 33:29
  I threw you on this tangent of how you're doing things.
• 33:34
  And I know you have a cool little setup with your open claw
• 33:39
  in this walled garden, right? That was I want you to explain a
• 33:42
  little bit more about how you see things with the walled
• 33:46
  garden and the proxies to help on that security side of things.
• 33:52
  And that would be a good way to kind of tie this back though as
• 33:55
  well to how the like going to that conference helped relate a
• 34:01
  lot of the industry things you can do to what I can do at home.
• 34:05
  And so, yeah, at home I definitely do.
• 34:08
  I probably have a much more complicated set up than is even
• 34:12
  necessary, but it's just kind of my own playground to mess around
• 34:15
  with. But like I, I haven't, I've used
• 34:17
  open claw a little bit. Not, not much, not for any
• 34:20
  reason other than I've have plenty of other things that I'm
• 34:23
  already messing with. I don't need a 50th tool like
• 34:28
  trying to follow all the latest tools and stuff is you'll go
• 34:31
  insane. But agent 0 that's one I highly
• 34:33
  recommend looking at. It's on GitHub open source
• 34:37
  project, but it runs in Docker. It's a really cool concept.
• 34:41
  It also uses kind of like a self learning prompt system.
• 34:44
  And I mean he they've been working on it I think since May
• 34:46
  of last year. I'm not associated with a
• 34:48
  project, but it's essentially an open claw type thing that you
• 34:52
  can run. And that's not like my primary
• 34:54
  driver, but it is a fantastic environment to experiment with
• 34:58
  because it's really easy to set up like agents or skills or
• 35:03
  actual skills or actual agents and, and mess around with it in
• 35:06
  an interface and stuff. But it kind of brought so like
• 35:11
  the way that I am currently working with like my open claw
• 35:14
  or development set up is like I have a bunch of team up sessions
• 35:19
  and terminals on multiple machines and they're all
• 35:22
  connected essentially through a self hosted matrix chat server
• 35:26
  that I I host. So I have like AI have a proxmox
• 35:30
  box, which is a like a proxmox is basically an OS more or less
• 35:34
  for containerizing containers. It's like a hypervisor.
• 35:38
  And so I have like my own git instance, my own like discord
• 35:42
  kind of type thing. It's called matrix.
• 35:44
  That's more, it's a lot more than that.
• 35:47
  And then, you know, all these other toolings.
• 35:48
  So like my whole pipeline and everything I experiment with
• 35:50
  happens all locally. I messed around a lot early on
• 35:55
  and put some secrets on GitHub real quick and shut that down.
• 36:01
  And actually yesterday for the first time, my system opened up
• 36:05
  APR for me that I did not ask it to.
• 36:08
  I I made a mistake. Yeah, it opened up APR on GitHub
• 36:12
  against Cogney, one of these memory layer packages I've been
• 36:18
  looking at, and it was a very minor PR that it made and it was
• 36:23
  just like an unnecessary one. But it's like, wow, that's, I
• 36:26
  didn't think that would happen with the way I've been doing
• 36:29
  this, but it did it, it can happen.
• 36:30
  So it got out of the, I mean, that's like the biggest, I mean,
• 36:34
  I leaked some security keys that didn't matter at all because
• 36:37
  like all the stuff, I mean, I highly recommend people look
• 36:40
  into some basic ways to secure your stuff and like go pass is a
• 36:44
  great thing to use, for example. But yeah, that was like the
• 36:49
  first thing that happened that like I really didn't want it to
• 36:52
  happen kind of thing like any of the the keys I leaked last year
• 36:56
  would have just been like, whatever it's going to be from
• 36:59
  some free account. But yeah, anyways, it's just
• 37:01
  such an interesting space. Yeah, did you see it?
• 37:04
  So many of the open source devs are in a tough spot right now
• 37:08
  because they're just getting inundated with slop PRS and you
• 37:14
  can imagine you didn't even want yours to do that.
• 37:16
  Imagine somebody that just pointed it at a certain repo and
• 37:21
  was like let's go help this maintainer out and edit all of
• 37:26
  the code and then submit a bunch of crap PRS.
• 37:30
  Was that one of the? Someone just had a blog post
• 37:32
  about that recently I think. A few people have been talking
• 37:35
  about it and now they're having to put like disclaimers in the
• 37:39
  different projects saying we don't do PRS anymore, we only do
• 37:44
  issues or things. To that nature because
• 37:47
  everyone's trying to figure it out in real time.
• 37:50
  Yeah, no, it's, I can imagine how crazy it is.
• 37:53
  And I, I, I saw the same thing. I, I tried to make a legitimate,
• 37:56
  you know, poll request to a repo in the last, you know, a few
• 38:00
  months. And, you know, I immediately got
• 38:02
  a reply. I spent a lot of time on it
• 38:03
  before I, I actually submit it. So like when I submit it, I'm
• 38:07
  ready for someone to actually review it.
• 38:09
  And then I get, you know, an immediate response from ChatGPT
• 38:11
  bot or something like that. You know, with my feedback on
• 38:14
  the, on the PR. And it's like, I don't know what
• 38:18
  to do at this point. Like, do I respond to the bot?
• 38:20
  Do I go make edits that I'm not confident I need to make?
• 38:23
  Do I go, I tried to bother the maintainers and I, I, I just, I
• 38:27
  just left it. So yeah, I, I just, I, I just
• 38:30
  stopped. I'm like, I'm sure this problem
• 38:31
  won't even be a problem in a week because everyone's
• 38:35
  deploying, you know, 3 releases a day.
• 38:37
  So, and that, and that was true though, like my PR became
• 38:40
  irrelevant essentially a month later because like that package
• 38:46
  I was fixing is now a different version or a different package.
• 38:49
  You know, like these these things change all the time.
• 38:51
  So what I, you know, what I have actually found and what I
• 38:55
  started doing like last year as well was I, I just, you don't
• 39:00
  pull the source code for a package and I use that locally
• 39:03
  rather than like constantly going and fetching it online.
• 39:06
  And like I, I found it easier to generally just essentially like
• 39:10
  fork an idea. That's kind of what I consider
• 39:12
  it then, like taking on a dependency that I can't trust.
• 39:17
  Yeah, the dependencies can be weird.
• 39:19
  They're that's. That's it's such a dangerous
• 39:22
  thing and it I on the side of security, you know, I, I was a
• 39:27
  little surprised that now I should say surprise, but I, I'm
• 39:31
  just very hesitant when I hear these people talking about, you
• 39:35
  know, access to 5000 skills or 6000 MCP servers and, you know,
• 39:39
  all these tools and things like that.
• 39:40
  And it's like, I don't want that.
• 39:43
  I don't want access to 6000 tools.
• 39:45
  I didn't write like, you know, I'll, I'll take some
• 39:48
  dependencies on things I trust. But yeah, I mean, if you've ever
• 39:52
  worked with JavaScript, like you got to, you can't just start
• 39:55
  throwing things in there. I mean, if, if every developer
• 39:58
  if like if you started accepting pull request every time a
• 40:00
  developer wanted to add a new dependency like you, your
• 40:03
  package would just be a mess. You know, like at some point you
• 40:06
  got to write your own. You know, the dependency is not
• 40:08
  worth taking on because some of these will have, it might not
• 40:11
  even intentionally be set up wrong, but they might have a
• 40:15
  bunch of like code that's packaged with it when you just
• 40:18
  need like one small. String or something like that
• 40:21
  you know yeah, there's anyways you could keep going and down
• 40:24
  that route, but that's where it's fun to use the things other
• 40:28
  people write, but you also having you got to have you got
• 40:31
  to read some stuff sometimes and that was that was you know I
• 40:33
  think Dex Dex's top was very interesting with how he talked
• 40:38
  about changing his stance on reading the code versus reading
• 40:41
  the the the PRD or and. What was the gist of that?
• 40:48
  Because I remember somebody given asking the question like,
• 40:51
  hey, six months ago you told us not to read the code, now you're
• 40:55
  telling us to read the code. What's going on here?
• 40:57
  Yeah, and I mean, he's just talking about the kind of, you
• 41:00
  know, not like the de facto way to vibe code, but in general
• 41:05
  when vibe coding. He I I believe he had a
• 41:09
  previously said don't bought like just as long as you have a
• 41:14
  solid dock. Like there, there's a period of
• 41:16
  time, you know, every few months kind of things shift.
• 41:18
  And there was a period of time where spectrum and development I
• 41:21
  think was like the buzzword. And that's when, and it's a very
• 41:25
  real thing. But it's like you, basically, if
• 41:29
  your specification is so detailed that there's no
• 41:32
  possible way for the agent to like, derail from that plan,
• 41:37
  there's no reason to essentially read the code because your
• 41:40
  understanding of the code is less important than the
• 41:45
  specification that you put in place that it has now adhered
• 41:47
  to. Wait.
• 41:48
  And so that was what he walked back was the idea of hey, if you
• 41:52
  have the spectrum and development and you specified
• 41:55
  everything so clearly you no longer need to look at the code.
• 42:00
  He actually came back in this talk right And he said maybe I
• 42:04
  was wrong and we we do need to look at the code.
• 42:09
  Yeah. And to like the the
• 42:12
  specification part can still you know, if you just have one typo
• 42:15
  somewhere in that specification, basically it can derail the
• 42:19
  entire project or derail the entire specification, you know,
• 42:23
  if you didn't read that one little bit.
• 42:24
  And so the the other aspect of that talk was that at the time
• 42:29
  the model was different. I think Opus 4.5 had just come
• 42:32
  out and of course, so as you, you know, they had one workflow
• 42:36
  in place for how things were working really well.
• 42:38
  And then when the model updates, it becomes like you're not
• 42:41
  necessary or doesn't work as well.
• 42:42
  Now you got to like re think how you're doing it.
• 42:45
  And so they like rewrote all their prompts.
• 42:47
  I I believe to like kind of change how they were doing that.
• 42:50
  But I, I think either way, you know, trying to, you got to read
• 42:55
  both. You got, you have to read what
• 42:57
  matters. And you can throw a lot of
• 43:02
  planning at a simple solution and get it right.
• 43:04
  You can also throw a little planning that is well thought
• 43:08
  out. One of the, oh, what I was going
• 43:10
  to say is one of the things he, he mentioned, not during the
• 43:12
  talk though, but he has previously mentioned is this
• 43:14
  thing about learning tests. And that is a fantastic idea
• 43:19
  that people should consider. I haven't heard of the term
• 43:22
  learning test in this way, but it makes perfect sense.
• 43:25
  But a learning test is essentially like the result that
• 43:30
  you would get out of doing a spike for a body of work in like
• 43:34
  agile agile development. So like if I was going to, let's
• 43:39
  just say, create a web app to authenticate with with Stripe,
• 43:45
  my first learning test might be that I can create an agent that
• 43:50
  calls the Stripe API through an MCP call and gets my account ID.
• 43:56
  Let's just say that's a learning test.
• 43:58
  So I set up a spike because my developments going to require
• 44:01
  this workflow that works, that learning that that that thing I
• 44:05
  did to get there. That spike is now a learning
• 44:08
  test, which is essentially like a regression test.
• 44:11
  And that regression test can either be semantic or codified
• 44:14
  or whatever. However, you can do it reliably,
• 44:16
  but now you kind of have a regression test on that your PRD
• 44:22
  on your, your whole idea. And that that overall is going
• 44:26
  to basically give you because it's, you know, overall this
• 44:29
  idea of like test driven development is always going to
• 44:31
  be a great idea, but you don't always know what it is you're
• 44:34
  going to test or how to test it or how to give the agent access
• 44:37
  to test it. And I think thinking of it as a
• 44:39
  learning test is a much easier way to frame it.
• 44:44
  But are you you're specifying the learning test in the specs
• 44:48
  or? Maybe you're, you're specifying
• 44:50
  yeah as part of your like requirements basically.
• 44:52
  But I I would assume or maybe whether or not he mentioned this
• 44:57
  explicitly, but I the logical next step is that you codify it
• 45:00
  to the best stability you can. So you know, if it's, especially
• 45:04
  if it's a dynamic age agentic action, you know, the only way
• 45:08
  you can, you can't have like a hard pass fail all the time.
• 45:12
  You could if you're, you know, leveraging one of the frameworks
• 45:14
  for like structured outputs. But at any rate, it's however
• 45:19
  you're going to eve at you. You have to have an eval
• 45:22
  mechanism in place for your learning test.
• 45:25
  And the learning test can either be deterministic or not.
• 45:28
  It really depends on what it is. But that overall tied together
• 45:33
  whether or not you should read the code or the spec like you
• 45:36
  should do a mix of all of it. But I think ultimately what what
• 45:40
  the real thing is, is you really want to make sure it's doing
• 45:45
  what you want it to do. And, and the really hard thing
• 45:48
  to do if you're not like if you haven't been coding a lot is
• 45:52
  knowing if it's doing it well. 10% of your time is spent doing
• 45:55
  90% of the work and 90% of your time is spent doing the last
• 45:58
  10%. Exactly.
• 45:59
  You can always get Apoc working. You can always get something out
• 46:02
  there. But as soon as you want to start
• 46:04
  adding a feature or making a little bit more advanced,
• 46:07
  depending on what you're doing and depending on how it was set
• 46:10
  up, it could get very complicated and very frustrating
• 46:12
  to to debug, especially if you're not looking at the code.
• 46:16
  That's literally me right now with this little remotion app
• 46:19
  that I created for and podcast shorts.
• 46:22
  But that's a whole nother story. I.
• 46:24
  Mean it's, it's so relevant though, because you can go down
• 46:28
  these, these like, even if you know what I I've seen, I, I've
• 46:31
  spent like hours talking to these coding agents for
• 46:33
  something. I'm finally just go, all right,
• 46:35
  I'll, I'll go like, maybe I'm on my phone.
• 46:37
  I can't figure. I get home and I you go look and
• 46:39
  it's like the the file had a wrong permission or it was just
• 46:44
  like something so trivial, but because of whatever was in the
• 46:48
  context window or whatever is going on, it just, it didn't
• 46:51
  matter what you're going to say, it was not going to get there
• 46:54
  anytime soon. Yeah.
• 46:56
  And then you're banging your head against the wall.
• 46:59
  Yeah. And it's so frustrating
• 47:01
  sometimes when you can't get that.
• 47:02
  I think there's a a few fun things that came out of this
• 47:08
  with the fact that you're spending a lot less time
• 47:12
  actually doing that, like really dropping into the code base and
• 47:16
  figuring out why something is not working.
• 47:19
  So whereas before you would be, if you're writing it all by
• 47:23
  hand, you're doing everything by hand and you have to have that
• 47:27
  understanding. And so maybe you can debug
• 47:29
  easier, but your velocity might be a little bit slower, right?
• 47:35
  But now your velocity is much faster.
• 47:37
  And every once in a while you do have to drop in and figure out
• 47:40
  what the hell is going on here because it's just not working.
• 47:44
  So, so there's this kind of like new way to think about it.
• 47:48
  There's that aspect. And then it's also about whether
• 47:51
  or not you actually have to, if someone else has to read this
• 47:54
  code ever, and if it's actually going to be used in production
• 47:57
  too, you know? Yeah, Well, that goes on to the
• 48:01
  idea of code reviews. I know there was a lot of
• 48:04
  conversations around that because of the simple narrative,
• 48:07
  right, that we've all been hearing so much is oh, well, now
• 48:10
  the PR, like reviewing the code is the bottleneck.
• 48:12
  It's not creating the code. And I think that to me is a very
• 48:16
  simplistic angle on how it's done.
• 48:21
  Yes, you generate a lot more code, but as we know, you can
• 48:26
  have agents go and look at that code or they can be adversarial
• 48:30
  and they can try and find problems.
• 48:31
  You can have security scans already.
• 48:33
  There's lots of things you can do before you have a human look
• 48:36
  at it. One thing that I thought was an
• 48:38
  interesting piece is really guarding the humans time on what
• 48:44
  they look at. And so Leo was telling me, Leo
• 48:48
  from the MLB community local chapter, he was talking to me
• 48:53
  about how he heard a few people mention you only want to
• 48:59
  escalate different parts of the code to someone else when you're
• 49:06
  not sure of it. But you only want to be really
• 49:09
  scoping down the the parts that you're not sure of or that are
• 49:13
  really like you're shaky on. And so you own the code that
• 49:17
  your agent produces, and you're the first line of defense.
• 49:22
  You have to be very confident that if you're going to submit
• 49:26
  that, that you've checked it over, You're not going to just
• 49:29
  outsource the checks to someone else, right?
• 49:31
  Because that's a little bit rude.
• 49:33
  But then if there is something that looks a little fishy in
• 49:36
  what you've created, only that very scoped down piece of code
• 49:42
  is what you're asking for a sanity check on.
• 49:47
  Yeah, that's it was the kind of like assigned lanes for
• 49:51
  ownership of how that would work with the agent produce code.
• 49:55
  Yeah. And and yeah, and ultimately
• 49:58
  they were responsible for what it produced.
• 50:00
  And that was kind of the, the, that's a good kind of, that was
• 50:04
  a very good kind of insight into how things are working at places
• 50:07
  that are using these in production and how the ownership
• 50:10
  works is you know, who, who's ultimately like the PR that I
• 50:14
  opened yesterday. I mean, you get, if you want to
• 50:17
  think about it, you know, more, you like who's responsible for
• 50:21
  that happening? I mean, ultimately it's me,
• 50:22
  obviously, because it opened up on my behalf.
• 50:24
  But it, it could have been something where despite my best,
• 50:29
  no matter what I did, it could be have been a bug in the
• 50:32
  software or something like that, where it's like, I'm going to
• 50:34
  ignore your, you know, I'm going to ignore the permissions you
• 50:36
  set. And I mean, they, they do that
• 50:38
  in anyways. And people don't realize like
• 50:40
  you set like deny bash. So it's not going to execute
• 50:43
  anything. Well then it just uses a
• 50:44
  different tool to to do the same operation.
• 50:48
  Yeah, that was. I think that was Scott's from
• 50:52
  Kilo Codes talk and he had a few hot takes, like especially in
• 50:56
  the end of his talk where he was saying that they have one
• 51:03
  engineer for one feature and so one person is responsible for
• 51:08
  one feature. Which is totally different than
• 51:11
  how normally I think a lot of us think about it where it's like
• 51:15
  you need a team and all of that fun stuff.
• 51:17
  The two pizza team. I was going to say Schwam had
• 51:20
  mentioned the same thing at cleric, that each each dev owns
• 51:23
  one feature. Yeah, no.
• 51:26
  And so that potential is a new paradigm on how folks are
• 51:29
  looking at it. But then you had Scott saying
• 51:33
  that they don't have PMS or they don't believe in PMS at QO Code,
• 51:37
  which I, I know got a lot of ruffles from the crowd.
• 51:40
  It was like what, no PMS, shots fired?
• 51:44
  And I thought that was a fascinating piece too, because
• 51:48
  you're like, OK, well, APM and an engineer, they're kind of
• 51:52
  meeting in the middle. And so you should be expected to
• 51:57
  do a little bit of the other person's job no matter where you
• 52:01
  sit. Yeah, you know, with with this,
• 52:05
  it's, it's just it's fun seeing a lot of it is just people
• 52:09
  experimenting with different ways to try things out.
• 52:12
  And you know, even before AI, not every team needs ATPM or APM
• 52:18
  or a dev lead or, you know, it depends on it what you're
• 52:22
  working on and what the scope is is needed.
• 52:25
  And you know, ultimate, it's going to come back full circle.
• 52:29
  Like best practices are going to kind of get, you know, that.
• 52:32
  And that's what the MO OPS is kind of trying to achieve here,
• 52:35
  though, is some level of sanity and a consistent way to
• 52:39
  communicate and to solve problems.
• 52:42
  And that's ultimately what people are trying to do.
• 52:44
  And that's, that's one of the reasons I, I went there though,
• 52:46
  is I really needed to understand what it is I do and don't
• 52:51
  understand. Because if, if you just spend
• 52:54
  your time getting gas lit by AI, you don't really know
• 52:56
  necessarily. I, I'm joking, but like, you
• 53:00
  know, you, you, if you're just going in, if you're, if you're
• 53:03
  just throwing in a bunch of tokens that you're creating and
• 53:06
  you're just giving yourself a feedback loop of yourself, you
• 53:11
  know, you could go one way. So like I said, I didn't know if
• 53:13
  you were going to be real. You were real and the content
• 53:15
  seemed seemed legitimate and people seem to to communicate
• 53:20
  with me and I didn't, you know, So it was definitely.
• 53:24
  A very, very. Learning.
• 53:25
  Learning experience. Yeah, you are.
• 53:27
  That'll be fun, no? Man, I'm invading Seattle for
• 53:30
  our next event. We're doing another agents event
• 53:32
  there where you're at, and that's pretty cool to get to
• 53:36
  hang out with you again. I'm going to crash your team
• 53:39
  dinner. My team, Your team.
• 53:43
  Team Bodo. Exactly.
• 53:46
  Yeah, there's there's a few other quick things that I want
• 53:50
  to mention before we end it because there was a lot of good
• 53:54
  learnings from it. And also just like cool pieces.
• 53:59
  The guy that created the superpowers pack, like skills
• 54:04
  pack, he came, his name's Jesse. And I love those skills.
• 54:09
  I've played with them a lot. They've helped me a ton.
• 54:11
  Especially the brainstorming skill.
• 54:14
  It helps before I do anything. It will just go back and forth
• 54:19
  with me and really like try to dissect why and what I'm trying
• 54:23
  to create. And Jesse told me, Oh yeah.
• 54:27
  The way that I created that skill was because I just
• 54:30
  channelled my days of when I used to have to be the boss of a
• 54:36
  lot of different interns. And so I found myself asking the
• 54:41
  interns constantly all of these questions, and I tried to bottle
• 54:46
  that up into a skill. And that's how brainstorming got
• 54:51
  created. So I thought that was fun.
• 54:54
  That's A and what did you what was your kind of like mechanism
• 54:57
  for iterating on the skill? Well, I just used that skill
• 55:02
  whenever before I do anything. Like before I create a PRD,
• 55:06
  before I sit down and even like I have an idea of a feature or a
• 55:11
  product or whatever it may be that I'm trying to create.
• 55:14
  I'll use that skill because it is much better than just clawed
• 55:19
  code at getting to the root of what I'm trying to do and then
• 55:23
  creating some kind of a PRD from it.
• 55:26
  Gotcha. Have you?
• 55:27
  How much time have you spent like building skills yourself?
• 55:31
  Not a lot to and I know that's that is something cool like
• 55:36
  that's one Rd. I want to go down, but I've I
• 55:38
  outsource that to Claude Ception and I let Claude ception do it
• 55:43
  and I just kind of like roll as it goes.
• 55:48
  I'm and the other Rd. that I really want to go down to is
• 55:51
  hooks. So actually this Friday we're
• 55:54
  doing one of the like coding agent lunch and learns that we
• 55:58
  have been having every Friday and I want to just have the
• 56:02
  whole session be on hooks because I feel like I'm not
• 56:05
  using hooks as well as I could be.
• 56:08
  I know there's a lot of really novel ways that folks are using
• 56:11
  hooks to just get better results from their sessions.
• 56:17
  And that's using hooks is absolutely one of the most
• 56:22
  deterministic ways you can get things in place in your
• 56:26
  pipeline. I mean, I use them in numerous
• 56:28
  ways. I've one of the most fun ways.
• 56:34
  So using hooks is definitely something you want to use as
• 56:38
  part of your evals. Basically, you know, that's one
• 56:41
  of your easiest mechanisms to log data at set points as well
• 56:48
  as to do a lot of other things. But doing so like I'm trying to
• 56:52
  think of one of the things, Oh, that's right.
• 56:59
  So basically, because I have all my agents kind of hooked up to
• 57:02
  this chat room thing, I found that I didn't have like a way.
• 57:07
  So I I might send a a message and then I don't always know if
• 57:10
  they're doing something and there's different ways I could
• 57:13
  set up monitoring that agent and things like that.
• 57:16
  But I just set up a simple thing that when they before they start
• 57:19
  work, it just sends a message to the room that it finished.
• 57:23
  And then I use that finish. So and then also I used that,
• 57:27
  that start and then the end hook to log a metric for the amount
• 57:31
  of time it took to spend on that task.
• 57:33
  And then you can also save the prompt that you use the tool
• 57:37
  calls and you can do all, you know, toss all that stuff in.
• 57:41
  There's a tool called Prompt Foo.
• 57:44
  Yeah, I think actually Open AI bought that now.
• 57:46
  Yeah, they just got bought. That's.
• 57:47
  True yeah so they're good for them that's a cool thing to use
• 57:51
  good way to I I haven't I've I've messed with it a bit and
• 57:54
  that's what I've been using now to start evaling what I've been
• 57:57
  messing with and I'm hoping to share something I have with what
• 58:00
  I've I'm doing but yeah doing the.
• 58:03
  Eval did something similar to that, except every time that an
• 58:08
  agent was doing something, it would log it on their own
• 58:12
  calendar. So you could see when they were
• 58:15
  doing different tasks and how long it took them, and then they
• 58:18
  had this record in calendar form.
• 58:21
  It was like this Google Calendar of everything that it had done.
• 58:26
  That's yeah, that's very much like, so I feel like I could do
• 58:32
  like my own little separate whole thing on all the aspects I
• 58:35
  have with hooks and how this works there.
• 58:38
  But yeah, come on Friday. What's that?
• 58:42
  Come teach me about hooks on Friday.
• 58:44
  That's what I need, dude. Yeah, no, I'll what?
• 58:46
  Time is it I? Well, I'll try to read my
• 58:49
  e-mail. I got to get better at that now
• 58:50
  that I have to like. Get those agents to read it, but
• 58:53
  it's at 9:00. It's at 9:00 on Friday. 9 Your
• 58:57
  Time 9:00 AM. OK, Yeah, I'll be there.