Latest / Agentic Conversations (formally mlops.community) / Building Claude Code: Origin, Story, Product Iterations, & What's Next // Siddharth Bidasaria // #342

Transcript

Building Claude Code: Origin, Story, Product Iterations, & What's Next // Siddharth Bidasaria // #342 · Agentic Conversations (formally mlops.community)

• 0:00
  I've seen people with like 100 MCP servers installed or
• 0:04
  something like that. And I was like, what do you even
• 0:06
  do with 100 MCP servers? But hey, it works for them.
• 0:09
  You know, like they, they're happy with it and you know, it
• 0:12
  works for them. So yeah, there's always like
• 0:14
  these like surprising things where all the composable parts
• 0:17
  of cloud code like they're being used in such creative ways in,
• 0:22
  in ways that we didn't quite expect them kind of to be to be
• 0:25
  used. Dude tell me about Claude code.
• 0:36
  What what's the genesis of it because I I never heard the
• 0:39
  origin story. Yeah.
• 0:43
  So the origin story is we Entropic had this team called
• 0:48
  the Labs team and this the charter of this team was just a
• 0:54
  prototype new products to kind of play around with things and
• 0:57
  see what hits more of an experimentation team really.
• 1:02
  And that's kind of the team team that I joined when I when I
• 1:04
  joined in Throbic. And then one of my colleagues,
• 1:08
  Boris, he had he had this like nifty little like prototype when
• 1:14
  he joined in Throbic. He like he already like he built
• 1:16
  it in his first week and it was basically axis and clawed in a
• 1:20
  terminal. Nice.
• 1:23
  And it couldn't it couldn't like write code or it didn't have any
• 1:25
  access to any tools. It was it was literally just
• 1:27
  like, you know, calling the the entropic APIs from the terminal
• 1:32
  and you could like type into it. And I remember the first demo
• 1:36
  that he showed us he was just like he spun it up and then the
• 1:39
  first thing he wrote into it was like 2 + 2.
• 1:41
  And I was like, bro, that's like the worst thing You want to ask
• 1:43
  another them, you know, like ask it something, something nice,
• 1:46
  But it's funny. I think like, so that was he's
• 1:49
  kind of like, you know, had that as a prototype and then we were
• 1:51
  experimenting with other coding surfaces and other coding
• 1:54
  products. But at some point he ended up
• 1:57
  adding two tools to to this like little terminal thing he had.
• 2:03
  The first one was the ability to control Spotify.
• 2:07
  Yeah, nice. It was like, I want this thing
• 2:10
  to control my music, so he's like, all right, like play me
• 2:12
  this. So he added that from the
• 2:14
  terminal. From the terminal.
• 2:17
  Because who doesn't want to operate Spotify from their
• 2:20
  terminal? Exactly, exactly.
• 2:23
  You know, are you really a hacker if you, if you can't
• 2:26
  operate your Spotify from your terminal?
• 2:29
  But the second thing he added was just like, like file dual,
• 2:33
  file dual. So like file reading, file
• 2:35
  writing, stuff like that. And the moment he added that, he
• 2:38
  like, you know, shared it with us.
• 2:40
  And I was like, holy shit. Like it was just like a
• 2:43
  instantly I was like, there's something here.
• 2:45
  It's, it feels really ergonomic. It feels very different from,
• 2:49
  you know, like web-based apps and, and things like that.
• 2:53
  So I, I, I think I was still working on kind of some other
• 2:56
  surfaces then, but I was like, you know, I, I think I just
• 2:58
  wanted to join you and hack on this.
• 3:01
  And he was, he was also kind of getting pulled into other
• 3:05
  projects, but then at some point kind of made a decision to be
• 3:08
  like, OK, let's just like double down and work on this a little
• 3:11
  bit more. So we spent a couple weeks
• 3:13
  hacking and then we kind of released it internally at
• 3:18
  Anthropic and, and like almost immediately just like got on
• 3:21
  like wildfire. Like people were just using it
• 3:23
  or like everyone was super excited and we were like, whoa,
• 3:26
  this is like within two weeks we had like some 300 active daily
• 3:29
  users. And this was when Anthropic was
• 3:30
  like 600 people in the whole company.
• 3:33
  Even the CFO was using it. Yeah, the terminal.
• 3:37
  You're controlling their Spotify, you know.
• 3:42
  Do you remember what model it was?
• 3:45
  This was 3/5. This was Sonic, this was 3/5, I
• 3:49
  think, yeah, Around there it's like October, October, November
• 3:53
  of last year. Yeah.
• 3:57
  And I think, like, that's kind of how it all started.
• 3:59
  And then once that, once we got some momentum internally, there
• 4:02
  was talk about launching externally.
• 4:05
  But honestly at the time, like we weren't really sure if this
• 4:08
  is going to be a thing. We were like, you know, this is
• 4:10
  still in the terminal kind of still feels a little jank.
• 4:13
  You know, it doesn't quite feel as polished as like cursor is
• 4:16
  like cursor feels like, oh, like, you know, they have this
• 4:19
  cool interface and they have like the tab auto complete and
• 4:22
  they're like forking an IDE like this is literally just a
• 4:25
  terminal. Like it's just like a terminal
• 4:27
  app. So we, we had no idea, you know,
• 4:30
  if this was what would, what would happen and like how this
• 4:34
  would be received. And that was like some fun and
• 4:36
  there was like, you know, it's kind of all kinds of discussions
• 4:38
  going around. But yeah, ultimately we decided
• 4:42
  to do like a small like EAP with a few customers, like seeing
• 4:45
  what the response would be like. And, and people who just really
• 4:50
  liked it, they like grew attached to it very quickly.
• 4:52
  And and that's when we kind of decided to launch more
• 4:55
  externally. And then once we launched it, it
• 4:57
  just like after a couple of weeks, it kind of just blew up
• 5:01
  on, on social media and things like that and just had a life of
• 5:06
  its own. But but yeah, it was it was not,
• 5:09
  it was not a pattern at all. Like it, I feel like it was a
• 5:11
  lot of like happy accidents that happened along the way that like
• 5:14
  kind of got caught us to that point.
• 5:17
  There's so many things that you said that are so fascinating
• 5:20
  right now. 1 is that there's just like an R&D department to
• 5:24
  go and try and create cool new stuff and you are on that team,
• 5:29
  which I love. The second is the one tool that
• 5:34
  you added that changed everything.
• 5:36
  And what was it about that tool You were like, it's ergonomic,
• 5:39
  there's something different here.
• 5:41
  Why did you have that intuition there?
• 5:46
  Yeah, I think the the file edit file rate tools, it was I think
• 5:51
  the biggest thing was that these files were local and you could
• 5:53
  spin up cloud anywhere with the most with kind of most products
• 5:58
  before that, you'd have to have some sort of like
• 6:00
  synchronization or like set up step where you need to like copy
• 6:04
  over your files or you'd need to you know, like create a Docker
• 6:09
  image with your repo or give them like access to your good
• 6:12
  repository or like some form of like sharing your files.
• 6:15
  And then it could start working on those things.
• 6:17
  And then it was it wasn't like super clear how to how to be
• 6:21
  collaborative in, in, in a product like that.
• 6:24
  Like if, if you're, if everything is being exported
• 6:27
  over to AVM where like your agent is running, how do I then
• 6:32
  very easily take what the agent's done and then continue
• 6:36
  that and then like improve that myself?
• 6:38
  Like, you know, like the models are, are pretty good and they're
• 6:40
  getting better, but they're not good enough to like zero shot or
• 6:43
  one shot everything. So I, I need to go in and make
• 6:47
  some changes, adjust things like, you know, do things, make
• 6:50
  it so that it, it looks like actual code And that was very
• 6:55
  hard to do with these other products or like with kind of
• 6:58
  this other world view. And I think in the terminal,
• 7:02
  when you can just like spin it up, no matter what, what files
• 7:05
  you have, like what repo, it doesn't, it doesn't matter.
• 7:07
  It's just what matters is that you have the files locally.
• 7:11
  You can just spin up Claude, ask it to read a bunch of files, ask
• 7:14
  it to explore kind of the, you know, how a human would.
• 7:18
  It just felt really magical. It just felt really like low
• 7:22
  friction, if there's a very low friction way of getting started.
• 7:26
  And I think like that's kind of what felt really ergonomic about
• 7:29
  it. What's been your favorite
• 7:31
  feature that you added? I'll tell you mine.
• 7:33
  First, the clear context window. Yeah, slash clear.
• 7:39
  Yeah. Yeah, yeah, yeah, slash clear is
• 7:41
  cool, yeah. How about you?
• 7:45
  I think my favorite feature that I added would probably be
• 7:49
  honestly like the To Do List the the To Do List was.
• 7:54
  I think that that remains my favorite feature.
• 7:55
  Just at least something about like tasks getting checked off
• 7:58
  and just like. You know you aren't the one that
• 8:01
  is doing them. Yeah, yeah, it feels satisfying.
• 8:04
  You know, like. Best feeling ever, yeah.
• 8:06
  Yeah, like it's just doing all this stuff for me.
• 8:08
  I'm just watching like it's like it's.
• 8:10
  Like watching those TikTok videos where it's deep cleaning
• 8:13
  of cars or houses and you're like, this is oddly satisfying
• 8:16
  because I'm not the one that has to be the one cleaning right
• 8:19
  now. Yeah, exactly, exactly.
• 8:21
  And I think like adding the To Do List also kind of just made
• 8:25
  the model stay on track longer for like longer horizon tasks.
• 8:30
  And that was cool to see too. Like you'd see a lot of times
• 8:32
  like you'd be like, hey, like I have these hundred files.
• 8:34
  Can you go and like edit their names or, you know, do something
• 8:38
  to them And it would do like 30 of them and be like, I'm tired
• 8:42
  now. You know, you can do the rest by
• 8:43
  yourself or something like that. And then like you give it A to
• 8:46
  do list tool and like it uses it to 1st create A to do list for
• 8:50
  itself and like batches the 100 files and like, you know,
• 8:53
  batches of 10. Everyone keeps checking them off
• 8:56
  one at a time and it goes through the whole thing is like,
• 8:58
  oh, this is this is quite satisfying.
• 8:59
  Like now, this is becoming a a little bit more deterministic.
• 9:02
  Not super deterministic still, but it's a little bit more
• 9:05
  deterministic than it was before.
• 9:08
  Did you notice? Because I remember we had my
• 9:11
  friend David Hershey on here who created Pokémon.
• 9:15
  Claude plays Pokémon. And he was talking about how for
• 9:19
  him, he saw such a unlock that it was almost like this latent
• 9:25
  unlock where people didn't realize when a new model dropped
• 9:29
  the potential it had until they were playing around with it for
• 9:32
  a little while. And then you saw, whoa, this is
• 9:35
  really good at coding. And there's something here.
• 9:40
  Did you notice like big jumps from different models?
• 9:44
  Oh, 100 percent, 100% I think. I think 3, like 3/5 was pretty
• 9:52
  good. And, and, but 37 is kind of like
• 9:55
  where things really started to come together.
• 9:59
  I think you could see a step change in the complexity of
• 10:04
  tasks that the model was now able to handle.
• 10:08
  You could see a step change in the way that the model kind of
• 10:11
  navigated through complex structures.
• 10:15
  You know, it's like it it almost feels like there there is like,
• 10:20
  you know, if I was to tackle a task, what would I do?
• 10:23
  It would like do a lot of those same things to kind of get to
• 10:27
  its conclusion. And that was really fascinating.
• 10:31
  And I think that I saw that step jump between 35 and 3 seven
• 10:35
  where like immediately like 3-7 felt like it was just way more
• 10:38
  capable. And I think because like I, I'm
• 10:42
  using the models all day every day, right.
• 10:44
  I think a lot of people are in Tropic are and just like even
• 10:47
  outside, like, so you, you get quite attuned to model behavior.
• 10:53
  And then once you see a model behaving slightly differently,
• 10:56
  it jumps out. Like you can tell it's like it
• 10:58
  has a, it has a slightly different, like smell to it.
• 11:01
  You know, it's like, I can, I can tell this is something
• 11:03
  different. Yeah.
• 11:06
  It's like, wait a minute, you just broke my frame of
• 11:09
  reference. Yeah, shouldn't do that.
• 11:12
  Yeah, yeah. And it's also kind of tricky,
• 11:15
  you know, because like once you especially if you use coding
• 11:18
  agente coding products a lot like you, you start to form an
• 11:21
  intuition about the kinds of things that it might be able to
• 11:24
  do versus the kinds of things that you might not, it might not
• 11:27
  be so good at. And you you'd probably have to
• 11:29
  play a more hands on role in getting that tastic completion,
• 11:33
  But then, you know, a new model will come along that's actually
• 11:36
  much better and a step change. And then you have to rewire your
• 11:39
  brain and they're like, OK, no, now that I this can, it can
• 11:42
  actually do this now. And that can be tricky for
• 11:46
  people sometimes like it can. And I think that might might
• 11:50
  also just be a blocker and like adoption of AI tools kind of
• 11:53
  going forward because people kind of form a point of view
• 11:56
  about what it can do, but then that that point of view needs to
• 12:00
  be adjusted, you know, very, very frequently or more
• 12:04
  frequently than people realize. It's funny you mention that
• 12:07
  because we had my buddy JQ on here and he was talking about
• 12:10
  how if for some reason the model like leans towards doing
• 12:16
  something what you would consider the wrong way.
• 12:20
  Then try to change your frame of reference into.
• 12:23
  Can I make it fit into what the models doing?
• 12:27
  Like if the model wants to always do this, can I see if I
• 12:31
  can now change me to let the model do that and see if it can
• 12:37
  then complete the task you know? Yeah, yeah, that's, that's a
• 12:40
  fascinating idea. Yeah.
• 12:43
  Like something like keeping the model on distribution.
• 12:45
  Like there's something, there's something to that, right.
• 12:47
  Like if you if you keep it on distribution, it's kind of doing
• 12:51
  what it really innately wants to.
• 12:55
  You're not trying to like, you know, force it to go a certain
• 12:58
  way or like steer it a certain way.
• 13:01
  Yeah, I, I think that's an interesting field of research
• 13:03
  and study too. And just like seeing kind of how
• 13:06
  keeping the model on distribution might actually help
• 13:09
  for or or or or even if it's unintuitive to to to region 1.
• 13:14
  There's something I want to talk about that's close to your
• 13:16
  heart, which is this idea of forward compatibility and making
• 13:22
  sure that when you're creating today, you're not spending a
• 13:28
  bunch of time on something that is going to change in the next
• 13:32
  model drop or with some kind of unlock with maybe it's like, oh
• 13:36
  great, now we can just use MCP servers for that.
• 13:40
  And so I wonder if you have examples of things that you have
• 13:46
  done in the past that ended up being like, you almost shot
• 13:50
  yourself in the foot or it was just a quote UN quote waste of
• 13:53
  time. And I'll, I'll give you an
• 13:55
  example of one that my buddy Flores was talking about on the
• 13:58
  podcast a few months ago. And he said, when ChatGPT came
• 14:03
  out, I spent so much time trying to extend the context window.
• 14:08
  I did everything I could to make it so that it was so janky, it
• 14:13
  was so hacky, and all I needed to do was be patient and wait.
• 14:18
  And the context windows for all these different models just got
• 14:21
  bigger by default. And so I wonder if like you have
• 14:25
  felt that in different areas where you're sitting there now
• 14:30
  looking back, you realize I spent way too much time on this
• 14:33
  thing that I could have just been patient on.
• 14:35
  Yeah, there's a lot. I, I think like one of the core
• 14:39
  like philosophies of, of, of our team is we, we absolutely love
• 14:43
  deleting code and like taking the and just deleting features.
• 14:48
  And, and that I, I think that's like by design somewhat because,
• 14:53
  because of this exact issue, because you're like the, you,
• 14:57
  there's like a phrase that it's like unhobbling the model,
• 15:01
  right? Like it's like, basically it
• 15:04
  translates to let the model cook, right?
• 15:07
  Like that, that is the, the centerpiece and the showpiece
• 15:10
  here. And the harness is really just
• 15:13
  about providing the model with the tools that allows it to cook
• 15:16
  as opposed to like trying to steer it in a way that's like
• 15:20
  unnatural or like steered in a way that, you know, might be
• 15:24
  redundant, you know, in, in a few model releases.
• 15:28
  So an example of that is like when, when clock code first came
• 15:30
  out, we, we had, we had like a bunch of tools.
• 15:33
  We had like the LS tool we had that was literally just for
• 15:38
  listing files. We had a few other tools, but
• 15:42
  that were just like very file system specific.
• 15:44
  And then we realized that we don't really need any of these
• 15:47
  tools. Like we can just like literally
• 15:48
  just give it the bash tool and it like, we'll know it can just
• 15:51
  do all of these things just through the abstraction of, of a
• 15:56
  bash command. So we stripped them all out and
• 15:58
  we like, OK, we, we, we don't want, we don't really want this
• 16:01
  in here and, and just give it the bash tool And it still is
• 16:04
  able to do the same exact things, maybe even better than
• 16:07
  before, just because like it understands the concept of bash
• 16:10
  so much better than a, a tool that we invented and gave it
• 16:15
  gave it gave to it. So there's, there's like other
• 16:19
  kind of fascinating examples of like, you know, how much harness
• 16:23
  is too much harness, right? I think like it's, that's a
• 16:27
  question that kind of, I think everyone who is building AI apps
• 16:32
  is kind of asking and there there isn't a clear answer.
• 16:36
  I think it just kind of there's a balance.
• 16:38
  It's more of an art really. It's like you got to balance it
• 16:40
  out because you want to be useful now to your users, but at
• 16:46
  the same time you, you want to be able to adapt and you want to
• 16:50
  be able, you want to be really agile so that if the model
• 16:53
  changes tomorrow, you can just like RIP out entire pieces of
• 16:56
  your code base and just like do something else and just let it
• 16:58
  cook. So that's always like like
• 17:01
  hooks. I don't know if you've kind of
• 17:03
  used cloud code hooks yet, or, but they're, they're like,
• 17:10
  they're like a feature that allows you to kind of hook into
• 17:12
  the life cycle of cloud code. And what I mean by that is,
• 17:16
  let's say you want to a simple example is like logging, right?
• 17:21
  Like you want to log every single tool call that cloud code
• 17:25
  makes this part of it's like operations.
• 17:27
  So you can, you can create a hook, you can register a pre
• 17:30
  tool call hook, or I forget what the actual name is.
• 17:33
  But like that's effectively what it does is like it's a piece of
• 17:36
  code that the harness runs before each tool call and that
• 17:41
  piece of code is provided by you, it's provided by the user.
• 17:44
  So you can have a batch script or you can kind of have any kind
• 17:47
  of code that takes in some parameters for like what tool is
• 17:51
  being called, what's the context before that and a few other
• 17:53
  piece of information. And it can return some
• 17:56
  information back into cloud. Like it can like disallow that
• 17:58
  tool or it can like steer it in a certain way.
• 18:01
  So it's like it's a very flexible system that kind of
• 18:04
  makes you that makes cloud code a little bit more deterministic
• 18:08
  and like more composable according to and how you want,
• 18:12
  want it to be. I remember already distinctly
• 18:13
  like when my colleagues like Dixon kind of came up with this
• 18:17
  and I remember thinking I was like, OK, is this too much
• 18:21
  harness? You know, like, do we do we
• 18:22
  really want this? But then I think like his point
• 18:26
  was like, quite, I mean, clearly a lot of people are using it.
• 18:30
  A lot of people like, like hooks, but his point was that,
• 18:33
  you know, like we, we need to kind of be somewhere in the
• 18:35
  middle. Like we need to we, we, this is
• 18:37
  a problem that we're facing right now with the model, you
• 18:39
  know, like does something that people don't want it to do and
• 18:41
  they want to steer it. So let's kind of create an
• 18:44
  abstraction that that works here.
• 18:47
  Like we're not trying to like do something very heavy-handed, but
• 18:50
  we're just going to make like life cycle hooks, which are a
• 18:51
  very common concept in, in many kind of like death tools, right?
• 18:55
  Like you have you have these life cycle hooks.
• 18:59
  Yeah. So it's like it's, it's always a
• 19:00
  struggle kind of like trying to figure out like what is how much
• 19:04
  harness is too much harness, but kind of we lean towards this
• 19:09
  being leaner rather than being more harness heavy.
• 19:13
  It's so funny that you mentioned that specific thing because last
• 19:17
  week I gave a talk all about how there are a lot of downfalls
• 19:21
  with the chat interface. And one of the downfalls that I
• 19:25
  see is that we don't have this granular steerability if we want
• 19:32
  it because we're using words and words are inherently fuzzy.
• 19:36
  They're like symbols of things that we want to happen, right?
• 19:38
  And especially with language itself, you don't get like a
• 19:43
  knob that you can turn. You have a word that you use or
• 19:48
  you use a stronger word or you use a weaker word.
• 19:52
  And so you saying that makes me rethink, well, do we even need
• 19:59
  steerability? Right?
• 20:01
  Like, because your whole thesis is in the way of just let the
• 20:05
  model cook. You don't need to steer it.
• 20:08
  You just need to get out of its way.
• 20:10
  And that was also kind of like J QS thing is when it's doing what
• 20:14
  you don't expect it to do, then figure out how to make you fit
• 20:19
  into its patterns as opposed to it fitting into your patterns.
• 20:23
  And so now I'm. Like going to have to go back
• 20:27
  and rethink this whole idea of this steerability.
• 20:31
  Yeah, it's, it's a balance. It's a balance, right?
• 20:34
  I think like I don't think like needing or wanting steerability
• 20:39
  is inherently bad. I think it makes, it makes
• 20:41
  sense, right? Like people are using it today
• 20:43
  and they want it to be useful for them right now.
• 20:45
  And if what they want right now is steerability and the model is
• 20:48
  not able to give it to them, then really the only option you
• 20:51
  have as an application developer is to create some harness that
• 20:55
  would allow the user to steer the model in one way or the
• 20:58
  other. But there's kind of two things
• 21:00
  to it. There's like 1 is the point that
• 21:02
  you were making earlier is like, oh, should I be changing my
• 21:04
  frame of reference and would that help?
• 21:07
  And then the second is what would happen if, let's say two
• 21:10
  months from now there's a new model that comes out and like it
• 21:13
  just does it for you. Would you just spend a whole
• 21:16
  bunch of resources on it for nothing?
• 21:19
  So, so I think it's like being agile, like having a tech stack
• 21:22
  that allows you to be really agile, a tech stack that kind of
• 21:25
  allows you to experiment a bunch and you don't feel bad really
• 21:28
  throwing away a lot of your code like that.
• 21:30
  That becomes really key, right? Like it's, it's, it's because
• 21:33
  you, you should and you should experiment with harnesses and
• 21:35
  you should experiment with letting it steer or, or like
• 21:38
  more heavy-handed steering mechanisms.
• 21:41
  But then at the same time, when the time comes, it should be
• 21:44
  easier about it's not like you've now like built five more
• 21:47
  things on top of that steering abstraction.
• 21:49
  And now it's like impossible to take out from your code and now
• 21:52
  you're fucked, right. Like it's like that's like you,
• 21:54
  you gotta you, yeah. So it's it's a balance.
• 21:57
  It's like I can't delete that it, it holds up the rest of this
• 22:02
  mountain of code that I have. And so you, you really like, got
• 22:06
  yourself into a pickle. Right.
• 22:07
  Yeah. What features are on the
• 22:12
  chopping block in your mind like you would be happy to delete?
• 22:16
  Oh, I would be really happy to delete hooks.
• 22:20
  Like I think, I think like if, if the model was just working
• 22:25
  perfectly in a perfect world, you wouldn't need hooks, right?
• 22:29
  Like you, you, you can just like tell the model what you want to
• 22:31
  do and you can have like different ways of ingesting some
• 22:34
  sort of context into the model and the model just adheres to it
• 22:37
  perfectly every single time. Let's say you have like, let's
• 22:41
  take like, like probably the most common use case of hook is
• 22:44
  hooks is logging. So you can literally just tell
• 22:48
  the model, hey, like every time you use a tool like log it using
• 22:52
  this endpoint. And if it adheres to your, your
• 22:56
  contacts or like your request perfectly, you don't need hooks
• 23:00
  anymore because hooks are their way of like adding determinism
• 23:04
  to, to, to do like an inherently like abstract, inherently like
• 23:09
  non deterministic process. So that would be really cool.
• 23:14
  Let's see what other tools I think we, we've like kept.
• 23:18
  We've, I think we've done a pretty good job of keeping like
• 23:20
  our tool list pretty lean. Like we've been leading a bunch
• 23:23
  of code, a bunch of tools like overtime.
• 23:27
  That's like the the big one that kind of comes to mind is maybe
• 23:31
  maybe like some memory features, right?
• 23:34
  Like if the model had perfect memory and perfect recollection
• 23:40
  of things that you've set it in the past, Like let's say you
• 23:42
  have infinite contacts going forward, you just kind of like
• 23:46
  point it to the transcripts that it had before and it just knows
• 23:51
  what to what to look for. You don't really need any kind
• 23:54
  of special scaffolding for memory or use of references.
• 23:57
  It just kind of knows, but these are these are these kind of
• 24:01
  sound like pipe dreams, but not really.
• 24:03
  I think like there is like I can see a world where it it kind of
• 24:07
  gets to that. But yeah, I think that those
• 24:10
  would be two features that I might be delete, might delete.
• 24:12
  Well, it's kind of like certain things are almost Moore's Law
• 24:20
  ish in how much they're developing and how fast they're
• 24:23
  developing. And one of them is the context
• 24:26
  windows are getting bigger. There's debate there around like
• 24:31
  how effective it is to keep everything in context, right,
• 24:34
  and how useful these large context windows are.
• 24:37
  But we do see like the push for bigger context windows.
• 24:41
  And then another one is like the price per LM call is dropping
• 24:47
  and that continues to drop in like a Moore's Law type of way
• 24:51
  where you're like, it's a better model and it's cheaper than
• 24:53
  before. And so, yeah, I potentially
• 24:59
  could see it. Maybe it's not like in the next
• 25:01
  month, but if we're talking a 10 year horizon, there's a world
• 25:07
  where that makes a lot of sense. Yeah, yeah, I agree.
• 25:13
  Yeah, I think like the timeline is always really tricky with
• 25:15
  these things. You don't really see it coming
• 25:18
  until you used to see it. I mean, like, you know, it's
• 25:20
  just like you're you're oblivious, you're oblivious,
• 25:23
  you're oblivious and boom, it just backs you in the face.
• 25:25
  And I go, shit, I can do this now.
• 25:27
  Or at least that's been my experience with that at least.
• 25:30
  So yeah, timing is kind of a, a bit of a, you know, oddball.
• 25:35
  But I I do think that there is a world where where something that
• 25:39
  exists. Speaking of timing and how
• 25:42
  things have evolved, I know that we had your colleague Eric on
• 25:47
  the ML OB Agent Builder Summit panel that we did in San
• 25:52
  Francisco in May. It's now we're in September, and
• 25:57
  one thing that Eric was talking about back in May was how hard
• 26:01
  verification is and how that's something that keeps him up at
• 26:04
  night. You had mentioned like there's
• 26:07
  been some advances, but it's still a very hard problem.
• 26:10
  Can you go into that a little? Yeah, yeah, for sure.
• 26:13
  So, so for contacts like verification is the ability of
• 26:18
  the model to verify that the work that it's doing is correct
• 26:23
  or like some form of like course correction mechanism where like
• 26:26
  if it's going down a bad path, it like recognizes that and can
• 26:29
  course correct. I think there's like 2.
• 26:32
  I would kind of break this down into two different sub problems
• 26:36
  really. The first is a model behavior.
• 26:39
  Like does the model know that to check its work regularly and the
• 26:45
  can it elicit that behavior? And the second is, do you have
• 26:50
  the tools or have you given the models the tools to effectively
• 26:55
  check its own work? And I think both of those things
• 27:00
  are kind of advancing at their own pace.
• 27:03
  I think the first one where like the models are now they know
• 27:08
  about verification and they know that like it is, it is a path to
• 27:13
  success. It is a good path to success.
• 27:15
  And it's more on distribution for them to, for them to be
• 27:19
  doing this. And I think that that's chugging
• 27:21
  along. And you know, I think like we,
• 27:23
  my, my hunch is that we'll see that kind of keep getting better
• 27:27
  pretty rapidly over the next few months and years.
• 27:31
  But the second piece of it is, is the tools that you're
• 27:34
  providing to the model. So in, in the coding context,
• 27:39
  really we, we have, you know, there's, there's a really wide
• 27:44
  variety of tasks that you can be coding up.
• 27:47
  And I think we have figured out verification for a small subset
• 27:51
  of these. Like for example, if you're, you
• 27:55
  know, coding up a website and you're mostly dealing with like
• 27:59
  JavaScript components and CSS and HTML and just like the
• 28:03
  visual hierarchy of elements, that is a little bit easier to
• 28:08
  to verify now that you have some MCP servers, like there's a
• 28:12
  Puppeteer MCP server that can open up a web browser for you,
• 28:15
  take a screenshot, send it back to the model and the model can
• 28:20
  iterate based on that screenshot.
• 28:21
  So if it's like changing the color of a button or if it's
• 28:23
  changing the position of a button, it like has instant
• 28:26
  feedback of like what, what it's doing.
• 28:29
  But then there are like more complex things here too.
• 28:31
  Like let's say it's like an animation, right?
• 28:33
  Like, let's say you're like coding up an animation and it's
• 28:35
  like, or some sort of user interaction that requires some
• 28:39
  sort, sort of animated frames. Those are still harder to, to
• 28:42
  kind of verify like you, it's, it's hard to kind of provide the
• 28:46
  model with a video stream of what's happening, at least in,
• 28:49
  in how today's models are conducted.
• 28:52
  It's still possible you can like slow down the animation frame
• 28:54
  rate and take like screenshots at certain times.
• 28:57
  But you know, it's, it's fuzzy and it's unclear how, how
• 29:01
  effective those things are. And then you've got things like,
• 29:04
  you know, like you could be doing some sort of like
• 29:05
  performance work, like you're like, you're working on some
• 29:07
  sort of like inference thing. And it's like super low level
• 29:10
  performance work. Again, like what is verification
• 29:13
  even look like here? Like, you know, how do you
• 29:15
  measure that? How does the model know to
• 29:16
  measure that? So the easiest thing to do, and
• 29:19
  this is a bit of a cop out, I realize, but it's like unit
• 29:22
  tests, right? Like I I think that unit tests
• 29:25
  and like smaller units of tests that humans pay extreme
• 29:29
  attention to and make the framework for making unit tests
• 29:34
  and maybe even end to end tests much easier.
• 29:38
  That will be probably the shortest path to verification
• 29:42
  success. And so if you're, you know, if
• 29:47
  I'm creating a new product tomorrow, the one thing that I,
• 29:50
  I will for sure do is make sure that I have a unit testing
• 29:54
  framework that is able to test as large of a surface area as
• 29:58
  possible of my code. Because if I do that, then I can
• 30:01
  leverage AI tools to write more of my code and trust the output
• 30:05
  of those tools. So I think, I mean, just test
• 30:09
  through and development has been around for ages, right?
• 30:11
  Like it's not a new concept and beep, beep, but I think it is a
• 30:14
  concept that is being rediscovered to some degree.
• 30:18
  Like I think everyone talks about test driven development,
• 30:20
  But if I look back at my career, like how many times have I
• 30:23
  really written a unit test before I write the actual code?
• 30:25
  And like, not that many, I'm going to be honest, right?
• 30:27
  Like it's like, but, but like now with, with the models
• 30:31
  capable of writing the code like to the unit test code too, that
• 30:35
  becomes it, it, it becomes kind of easier to digest, you know,
• 30:39
  it becomes a little bit lower friction to kind of write code
• 30:42
  that way. So that's, that's kind of the
• 30:45
  other piece of verification which is unsolved.
• 30:47
  And I think it's it's going to be tricky to to get to a point
• 30:52
  where we can make completely solve that.
• 30:54
  This unit test thing sounds like it's going to take forever.
• 31:00
  Like you got to run the test too though.
• 31:03
  Yeah, yeah, you do have to run the tests, but hopefully the
• 31:06
  model just figures out which tests it needs to run and just
• 31:08
  like runs those tests and optimizes that for you.
• 31:11
  So yeah, there, there. There's, there's degrees of
• 31:15
  being AGI pilled and you know, like depending, depending on
• 31:18
  kind of where you fall in that spectrum, you might see it
• 31:21
  differently. Yeah, the model is just going to
• 31:24
  learn from the worst of us and be like fuck it, I'm testing in
• 31:27
  prod. Like a troop row?
• 31:31
  Yeah, exactly. Yeah, yeah, I've seen enough of
• 31:35
  these models do that kind of stuff, like on GitHub issues or
• 31:38
  something where it'll comment it's like this was a human
• 31:41
  behind it, wasn't it? This this cannot be really.
• 31:45
  Yeah. Have you seen those?
• 31:47
  Like I think there was a trend on I think levels IO on, on
• 31:51
  Twitter, like kind of tweeted about this and then people
• 31:54
  started doing it was like, I think he called it like vibe
• 31:57
  DevOps or like vibe infra or something like that where he
• 32:01
  didn't have any like deployment strategies or like deployment
• 32:06
  code. It was there was no
• 32:08
  infrastructure as code. He would literally just like SSH
• 32:11
  into his production box, open up cloud code and just like tell it
• 32:16
  to deploy stuff and like like do like, you know, do database
• 32:18
  maintenance or just like, you know, upgrade, upgrade the
• 32:21
  schema or something like that and then.
• 32:23
  Oh I miss this. People, people just kept doing
• 32:26
  that over, over all their projects.
• 32:28
  And I was like, yes, this is maybe this is the future.
• 32:31
  I don't know. Yeah, oh man, that makes me so
• 32:35
  nervous. But I am glad that somebody's
• 32:38
  trying it and pushing the limits.
• 32:40
  Yeah, for sure. Speaking of pushing the limits,
• 32:43
  I want to hear about the power users and how they're using
• 32:48
  cloud code. Like how have you seen people
• 32:51
  just surprise you with what they're doing?
• 32:54
  Yeah, there's, there's a lot of power users who have surprised
• 32:59
  me. I think the, the, the number one
• 33:03
  thing that surprised me was like this one person who was using a
• 33:09
  fleet of like 10 to 12 clods for like 1 problem.
• 33:14
  And he was using the file system as a mechanism for all of these
• 33:21
  different instances to talk to each other.
• 33:24
  And he like gave all of these different instances a, a
• 33:29
  persona. So he he had like a, you know,
• 33:33
  like a a back end engineer as a front end engineer, like the,
• 33:37
  the, yeah, exactly like the and they all operated on like
• 33:39
  different folders in his code base.
• 33:43
  And I was extremely impressed by that set up, so much so that I
• 33:47
  think the sub agents kind of feature in cloud code today is
• 33:51
  like inspired heavily by that approach.
• 33:55
  That makes sense. But it was, yeah, it was, it was
• 33:57
  extremely cool to kind of see that.
• 34:00
  And there's, there's I think people who have, I've seen
• 34:04
  people with like 100 MCP servers installed or something like
• 34:07
  that. And I was like, what do you even
• 34:09
  do with 100 MCP servers? But hey, it works for them, you
• 34:12
  know, like they, they're happy with it and you know, it works
• 34:15
  for them. So yeah, there's always like
• 34:17
  these like surprising things where all the composable parts
• 34:20
  of cloud code like they're being used in such creative ways in,
• 34:25
  in ways that we didn't quite expect them kind of to be to be
• 34:28
  used. Plot twist, the 100 MCP servers
• 34:32
  are just playing Spotify from the command line.
• 34:37
  Yes, love it, love it. Yeah, I actually want to create
• 34:40
  this like little web app that literally just like controls
• 34:44
  your spot Spotify and that's all it does that you just like off
• 34:48
  to it and it has like clot sitting on top and just like all
• 34:51
  it does is like connect your Spotify doesn't want to play.
• 34:54
  I would love to be able to just sync with that on finding new
• 35:00
  music, Yeah, like, you know, I want and just give it all a
• 35:04
  brain dump on I want a playlist with this kind of music and try
• 35:09
  and explain everything that I'm feeling in that moment and then
• 35:12
  it goes and creates it. Love it.
• 35:14
  Oh, what if he what if he gave it a webcam and a mic?
• 35:19
  So what you could do is that you can like set that up at a party
• 35:24
  in the corner and it like looks at the vibe of the party and
• 35:27
  plays music according to it. It's the jukebox.
• 35:31
  It's the 2025 jukebox. Exactly, exactly.
• 35:35
  And then people can go and whisper requests into the
• 35:38
  microphone. Oh hell yeah, I think.
• 35:40
  I think we just created the next billion dollar app dude.
• 35:44
  Yes, I'm going to save this episode and not release it until
• 35:48
  I get 2025 jukebox.com or dot AI website.
• 35:54
  Yeah. And so I've heard sub agents
• 35:58
  called agents as tools. I'm also just been seeing how
• 36:04
  incredible it is to have that like sub agent kind of design
• 36:09
  pattern for lack of a better word.
• 36:12
  How do you see or foresee sub agents evolving?
• 36:18
  Yeah, I, I think sub agents are the, the possibilities are just
• 36:24
  too many. There's a, there's, there's,
• 36:26
  there's, there's a lot of different ways that this going
• 36:28
  to shape up. I think when we first like
• 36:32
  launched sub agents, there was like no good way for kind of
• 36:37
  like everyday users of an agent coding product to create their
• 36:41
  own custom sub agents. And the idea really was let's do
• 36:45
  the thing, the simple thing that works, right?
• 36:47
  It's like we want to do like the absolute simplest thing that we
• 36:51
  know will work and like people can play around with and
• 36:53
  experiment with, but there's so many other things you can do.
• 36:58
  Like right now, the way at least it's implemented in cloud code
• 37:02
  is, is you, it's it's sub agent as a tool.
• 37:04
  It's an agent as a tool, right? So it's like Mama Claude or
• 37:07
  like, you know, the, the, the main thread, like we'll call
• 37:10
  into these tools and then wait for those tools to return back
• 37:13
  to them at at some point and then continue with the
• 37:15
  conversation. But then what if you just had
• 37:18
  like, you know, a master master model where you had like
• 37:21
  multiple sub agents kind of all doing their own thing.
• 37:24
  There is no like parent. They're all communicating with
• 37:26
  each other. The, the, the way that they
• 37:30
  communicate with each other becomes an interesting kind of
• 37:34
  decision as well as like, is it, is it message buses?
• 37:36
  Is it like point to point communication?
• 37:39
  And when you, when they are communicating with each other,
• 37:42
  like when do you actually inject those messages in?
• 37:45
  Like do you inject, did you preemptively inject them or do
• 37:47
  you wait for them to finish? Kind of how important is that,
• 37:51
  you know, where are these sub agents really running?
• 37:53
  Like are they going to, is the code that they're writing going
• 37:56
  to conflict with each other? Or is it, is it going to be read
• 37:59
  only or like do you designate some of them as read only and or
• 38:02
  you designate some of them as like read, write like there's a
• 38:04
  lot, it opens up a whole can of worms, but it's also really
• 38:08
  exciting. I think there's a lot of
• 38:10
  research that's going to being done about this, a lot of kind
• 38:14
  of open source repos also that are kind of exploring this using
• 38:17
  cloud code as the base. So I'm extremely excited about
• 38:21
  kind of this direction and kind of what we find.
• 38:23
  To be honest, it is unclear to me right now whether some of
• 38:27
  these like more complexed like a more complex agent or sub Asian
• 38:31
  topologies lead to better results.
• 38:35
  And I think unless we have some form of evidence of like these,
• 38:42
  this kind of Asian topology like works really well, I'm, I'm
• 38:47
  hesitant in kind of adding that complexity to the product
• 38:51
  because it, it not, it's not just like the complexity of
• 38:55
  adding it like the code that you're going to add, but it's
• 38:58
  also the complexity for users to understand how it works.
• 39:02
  And, and, you know, I feel like a lot of lot of coding agentic
• 39:07
  coding tools are kind of geared more towards power users to
• 39:10
  begin with. And what, what's more exciting
• 39:14
  to me right now? I mean, power users are great.
• 39:16
  And I, I, I think I'm a power user, so I build for myself all
• 39:18
  the time. But what's more exciting is like
• 39:20
  there's this whole other spectrum of users who, who are
• 39:23
  just like, just about, you know, just getting their feet wet with
• 39:27
  this, like they, they're skeptics or they don't quite
• 39:31
  believe in AI and they like, you know, oh, like, I can write this
• 39:34
  better. It's wasting my time.
• 39:35
  Like there's like all of that stuff.
• 39:37
  But they're, they're not wrong. I mean, I think they're right.
• 39:39
  If, when, when you for certain things.
• 39:43
  But I, I'd love to kind of have sub agents and the future of sub
• 39:47
  agents kind of cater more towards them and helping them
• 39:50
  get a better outcome from, from, from the product.
• 39:54
  There's so much that you said there.
• 39:56
  And one thing that stands out to me in all of that, when you have
• 40:03
  agents going about and you add that extra complexity of just
• 40:08
  like a swarm of agents, right? And or a hive of agents that are
• 40:14
  flying around and doing their own thing and then communicating
• 40:16
  with each other and figuring out how and when and where you want
• 40:20
  them to communicate. Or you want to add extra
• 40:23
  context. Is complex in itself.
• 40:27
  I instantly think about there's certain things in the context
• 40:34
  windows and maybe it's a little less so for coding agents.
• 40:37
  But if we were to extrapolate this out and say for agents
• 40:43
  across the web, the whole thing that I think about as you're
• 40:47
  talking about these complex agent intertwinings and passing
• 40:52
  around a lot of context from one agent to the other and not
• 40:57
  really having a superior agent. But even just if, if you did
• 41:01
  have a superior agent that is overlooking it, all of it, you
• 41:04
  know, like the I've Sauron agent, you still are going to
• 41:08
  have a really hard time being able to identify what is
• 41:14
  sensitive in all of this information that's being passed
• 41:18
  around. Like there's no protocol right
• 41:20
  now for this is sensitive. And it's not.
• 41:24
  I'm not even saying like PII. I'm saying like stuff that I
• 41:28
  wouldn't potentially want going around the Internet, you know,
• 41:32
  and it's not like my Social Security number.
• 41:35
  It might just be like my YouTube preferences.
• 41:39
  Yeah, No, I get it. I I think that that is, I think
• 41:43
  you kind of nailed that one. It's like, I think that's 2
• 41:46
  problems, right? Observability is, is almost, it
• 41:50
  is really hard for, for complex sub agent, apologies.
• 41:56
  And because of that, as a result of that, like how you do
• 41:59
  permission management and how you do like that kind of becomes
• 42:03
  complex too. So like for example, like if in
• 42:07
  in card code, if you, let's say you spin up like parallel sub
• 42:10
  agents like that you just spin up like 3 parallel sub agents
• 42:13
  that, that are like going off and doing their own thing for
• 42:15
  you. Anytime any of them encounter
• 42:19
  some form of like tool that it doesn't have permission to run
• 42:22
  like bumps up and like tells you, asks you say, hey, can I,
• 42:26
  can I use this tool? And then you like go and say yes
• 42:29
  or whatever. And then it like it goes off and
• 42:30
  does it's thing. This is not scalable, right?
• 42:35
  Like it's not. It works because you know, the
• 42:38
  sub agent implementation we have is quite simple and so it works.
• 42:43
  But if you have more like complex agent topologies, it
• 42:46
  doesn't scale. What is the answer?
• 42:50
  I don't know. I'm not sure.
• 42:51
  It's like it's, I think it's something that we're going to
• 42:55
  have to come up with a better way of dealing with this.
• 42:58
  And then permissions is one thing I think like once you get
• 43:02
  permissions to do something, what do you do with that and who
• 43:06
  do you pass it on to? And like how do you know that
• 43:08
  that person or that agent will not will do the right thing with
• 43:13
  with it? So.
• 43:15
  And I think he's dynamic, like you can't give permission
• 43:18
  forever. It's permission right now.
• 43:21
  And then maybe for 5 minutes or maybe for an hour, but I'm going
• 43:25
  to revoke that permission because I don't want you to be
• 43:28
  able to always do that. And so that's a little bit weird
• 43:31
  too. I, I, I find permissions
• 43:34
  fascinating too, especially again, like extrapolating it out
• 43:38
  for just regular agents that sometimes I do want to tell the
• 43:44
  agent to go buy something and I want to give it permission to
• 43:48
  buy it. But then sometimes I don't.
• 43:51
  And I definitely don't want it to buy the same thing like 5
• 43:55
  times. So if I give it permission once,
• 43:57
  it potentially has that permission to just keep buying.
• 44:02
  Yeah, I mean, oh, I found a cheaper deal on this Ferrari.
• 44:06
  You wanted to buy $500,000? Boom.
• 44:10
  Yeah, that would be bad. Yeah, especially if it's the
• 44:14
  fourth one it just bought. Yeah, one for each day of the
• 44:18
  week. I thought that's what you
• 44:19
  wanted. I'm just trying to help you out.
• 44:21
  Exactly. Oh, exactly man.
• 44:24
  Yeah, you can get into a lot of trouble there.
• 44:26
  I'll use like quad code amp lovable and what I noticed going
• 44:31
  back to that permissions part is whenever it comes up and it will
• 44:35
  ask me, hey, can I do this? I joke like I just will
• 44:41
  constantly hit yes so that I don't have to review it and be
• 44:44
  like, I guess I'll know later when it breaks and it's kind of
• 44:49
  like the new looks good to me on like a merge pull request.
• 44:54
  That's that's really how I feel whenever I press yes without
• 44:58
  looking at anything. I'm just like, send it.
• 45:01
  We'll see what happens, yeah. We we, we've like joked about
• 45:04
  just having this like little red button that we like sell to
• 45:07
  everyone or just like even just give out to our customers that
• 45:10
  just says like you are absolutely right and you just
• 45:12
  like press that and that just like auto access permissions for
• 45:15
  you every single. Time everything just like take
• 45:17
  it all. I don't want to be bothered
• 45:19
  again. Yeah, it's like super Yolo mode.
• 45:22
  Yeah, that's Peter level style that is.
• 45:26
  But also you were talking about something fascinating with at
• 45:30
  scale. I'm not going to be able to save
• 45:35
  any time if I have 12 sub agents that are constantly asking me
• 45:39
  for permission. It's a full time job.
• 45:41
  Like trying to hit yes even if I'm not even looking at what
• 45:45
  it's doing. God forbid I have to like look
• 45:48
  through the code it generates right?
• 45:50
  But if I am not even looking, I'm still going to have to just
• 45:53
  constantly be pounding the yes button.
• 45:56
  Yeah, I, I mean, I think my take on this, at least right now, is
• 46:02
  that as the models get smarter, we'll be able to rely more on
• 46:06
  them to do kind of these like, you know, the Eye of Sauron,
• 46:12
  like Asian that you have like is, is, is going going to get
• 46:15
  really, really good at understanding user intent and,
• 46:20
  and like providing dynamic permissions.
• 46:23
  That's the only way that I see the skilling, to be honest.
• 46:26
  I don't see how like you can create a really complex web of
• 46:30
  like dynamic settings and dynamic config and auth and just
• 46:33
  like, you know, but it's going to have a breaking point.
• 46:38
  And, and the more I'm, I'm kind of working with LLMS, I'm
• 46:44
  realizing that if you're, if you're scaffolding is just like
• 46:47
  really, really complex, you will not survive, you know, and then
• 46:52
  so maybe it solves the need right now and maybe like it is
• 46:55
  useful for a few months, but looking to the future, I think
• 47:00
  it is going to be offloaded to some sort of like model that's,
• 47:03
  that's highly tuned to something like this.
• 47:07
  How often are you going to the researchers and just saying
• 47:11
  like, watch me work real fast and I'll tell you what's
• 47:15
  painful, and then maybe you guys can figure out how to fix that
• 47:19
  on the model side. Yeah.
• 47:21
  I mean, I think we like our team and like, we're like quite in
• 47:25
  touch with the research in, in my ways.
• 47:27
  There's, I think that's like one of the advantages of working at
• 47:33
  A, at a lab is, is that you kind of have that and the, what you
• 47:38
  can come up with then is, is just like blessed and backed by
• 47:43
  research. And, and also it's like a
• 47:45
  flywheel. So it's not just like one way
• 47:47
  information coming in, but it's also like what you learn from
• 47:49
  user behaviour and what you learn from product usage kind of
• 47:52
  like feeds back into research and like what the research
• 47:56
  priorities are going forward. So it that's like a flywheel
• 48:01
  that, you know, we're definitely tapping into and then we, yeah,
• 48:05
  we make it a point to kind of tap into that.
• 48:08
  As much as I imagine you also have the boon of being able to
• 48:11
  see what's coming down the pipe and then see like, OK, is this
• 48:17
  new model, whatever characteristic, can we leverage
• 48:24
  it in some way, shape or form with cloud code?
• 48:27
  Yeah, I think that yeah, that's, that's, that's definitely a, a
• 48:31
  big advantage. We, we kind of like have access
• 48:35
  to, you know, some of the preview models and things like
• 48:37
  that and that that helps set product strategy kind of a
• 48:42
  little bit earlier than we would if we didn't have that.
• 48:46
  Do you and this might be above your pay grade.
• 48:49
  So, and we'll have PR listen to all this, so they may tell us to
• 48:53
  cut this question completely, but do you foresee a world or is
• 48:59
  it already happening where there's just like so many
• 49:06
  resources that are going towards Claude code and just coding
• 49:08
  models in general that that is now like a gigantic model
• 49:16
  machine in a way? I mean, I think Entropic has
• 49:19
  leaned into coding for for a few model releases.
• 49:22
  Now, that's not to say that there's not other stuff
• 49:25
  happening, but there's definitely is, you know, a kind
• 49:28
  of a focus on coding just because our models are really
• 49:31
  good at coding. And I would argue the best, I
• 49:35
  think it's kind of unanimously known that people try, but
• 49:40
  they're like, it's not as call it code as like what you
• 49:43
  benchmark everything else against.
• 49:46
  Right. Yeah.
• 49:46
  I think that we've been kind of fortunate.
• 49:48
  And of course, like props to everyone who kind of built these
• 49:51
  amazing models. But yeah, I think that there's
• 49:56
  definitely a focus on it. I can't say so much about like,
• 50:00
  what if there's like trade-offs or like, you know what things
• 50:04
  like that, I honestly don't even know.
• 50:06
  But I, I, I, I know that that that is an area of focus and,
• 50:11
  and you know, there is a significant number of people who
• 50:14
  are kind of looking into it.