Latest / Agentic Conversations (formally mlops.community) / Why Agents are Driving Software Development to the Cloud

Transcript

Why Agents are Driving Software Development to the Cloud · Agentic Conversations (formally mlops.community)

• 0:00
  However, if you're reviewing a code that your agent wrote,
• 0:04
  that's a ridiculous flow. It's like why am I leaving my
• 0:07
  workbench to go up to GitHub? You shouldn't.
• 0:11
  You should just. Do it right as the agent is
• 0:13
  writing it. And So what you're seeing is
• 0:16
  some of the things that like we're like the hero features of
• 0:19
  GitHub, like I think collaborative code review are
• 0:22
  going to be moving out of GitHub.
• 0:30
  I want to I want to start with the fact that you've been
• 0:33
  creating Warp before that you did Google Docs.
• 0:35
  You are all about the collaboration.
• 0:38
  And so Warp is a logical next step in collaboration, making
• 0:44
  the idea of software engineers collaborating in the cloud be as
• 0:48
  easy as the knowledge workers collaborating on Google Docs.
• 0:52
  But there was something funny that I read when I was doing
• 0:57
  research to start this podcast with, and I think you have a bit
• 1:03
  of a hot take when it comes to sandboxes.
• 1:07
  You're not super stoked on sandboxes?
• 1:11
  I'm cool. I'm cool with.
• 1:12
  I'm cool with sandboxes. Well, what is it?
• 1:15
  Tell me why you don't like the idea of the paradigm of
• 1:20
  computers. OK, so no, I, I'm cool with
• 1:24
  sandboxes. It's a funny hot, it's a funny
• 1:27
  way to start with a hot take. My my take is just, there's
• 1:30
  this, I've seen a bunch of tweets around like the right way
• 1:34
  of moving agents into the cloud. It's like set up like a cloud
• 1:38
  computer and let your agents kind of loosen it.
• 1:42
  And that's fine. That's not exactly how I think
• 1:47
  it ought to work. I think that the primitive that
• 1:50
  makes more sense to me is actually like imagining your
• 1:53
  agents is sort of like teammates who run in the cloud.
• 1:57
  And as teammates they have like sets of permissions and are
• 2:01
  doing stuff as themselves. And the really simple paradigm
• 2:04
  of like will move your laptop to a cloud dev box someplace and
• 2:09
  use that as a sandbox is kind of wrong because what you actually
• 2:13
  want is different agents doing different kind of things.
• 2:16
  And each one of those agents has its own set of like access
• 2:20
  controls. And just to make that more
• 2:21
  concrete, it's like on our team, when we're using agents, we have
• 2:25
  like we have agents that are automating all sorts of stuff,
• 2:28
  but some of those agents are automating things that like
• 2:30
  touch our CRM. And we don't want those agents
• 2:34
  like touching our code base necessarily.
• 2:36
  Some of those agents are, you know, checking for fraud and
• 2:39
  need to look at our production database.
• 2:41
  And so I would rather view the agents less as like that you
• 2:45
  plop them into a cloud computer and more as like they are sort
• 2:48
  of teammates, each with a set of permissions and roles in their
• 2:52
  own little workspace or they just have from like a least
• 2:55
  privilege perspective exactly what they need to do their job.
• 2:59
  I think that the the like cloud computer thing is like an
• 3:03
  analogy. It just is not the right, not
• 3:06
  quite right. It's almost like a difference
• 3:08
  between like a long running server and a like Lambda or
• 3:12
  cloud function. I think we want our agents
• 3:14
  running honestly more like lambdas or cloud functions.
• 3:18
  But I don't know. It's not like I don't, I'm not
• 3:21
  like upset about this view. Like I, I'm totally fine if
• 3:24
  people want to watch cloud computers.
• 3:26
  I just think it's, it's not, it's not exactly the way that I
• 3:28
  would try to set it up technically.
• 3:31
  So what you're saying is there are other beliefs and views that
• 3:35
  you hold more dearly and you are upset?
• 3:37
  About this is not this is not a thing that I spent like my
• 3:39
  nights. Thinking about you're going to
• 3:41
  die on. No, no, not at all.
• 3:43
  This is like a a kind of, oh, I see everyone moving this
• 3:46
  direction. I think maybe there's like a
• 3:48
  slightly better direction to to move in besides that, that's
• 3:52
  all. What are some hills you want to
• 3:54
  die on? I mean, a big thing that I'm
• 3:57
  like focused on right now is I do think we want to get agents
• 4:00
  off of people's laptops. And so I, I do think that the
• 4:05
  move from interactive agents, which is what last year was, was
• 4:09
  all about. And I'm talking about the coding
• 4:10
  space here where, you know, last year, I think really caught on
• 4:14
  things like Cloud Code, Codex Warp to some extent, where it's
• 4:18
  like you as a developer are sitting there talking to your
• 4:22
  laptop, asking it to build features, asking agents to pick
• 4:25
  bugs. Yeah.
• 4:26
  We're at a point now where agents need to transition from
• 4:32
  being like a solo sport to a team sport.
• 4:35
  And this is what I'm hearing from companies, too.
• 4:37
  Like, we talked to a lot of business leaders, like, they
• 4:39
  want a way of building automations with agents.
• 4:42
  They want a way of seeing what these agents are doing.
• 4:44
  They want agents, You know, right now if you do things
• 4:47
  locally, it's like every time you start a new agent, you
• 4:50
  basically you're starting from scratch in a sense, because the
• 4:53
  knowledge of the agent isn't really being built up over time.
• 4:57
  And so this is the year I, I think you got to get agents off
• 5:01
  laptops. You got to get them into some
• 5:02
  centralized system that's running in your cloud where you
• 5:06
  are tracking what they're doing, where there's cumulative memory,
• 5:09
  where they can run when your laptop is closed and so on.
• 5:12
  And so that's like, I don't know if it's a hill I want to die on,
• 5:15
  but it's like very much what I think is that's just like what I
• 5:18
  think needs to happen this year. And I think is is starting to
• 5:21
  happen for sure. Well, how are you seeing the
• 5:24
  best teams collaborate? Yeah.
• 5:28
  So it's pretty cool. I I mean, I can speak to to our
• 5:32
  team, which I think is a pretty good team.
• 5:35
  It's like I can pull up now in ounces.
• 5:39
  Oz is our like cloud agent platform and I can see what
• 5:42
  everyone on our team is doing with agents, which is pretty
• 5:44
  cool. And so if like I want to go like
• 5:48
  review APR or something, I can actually see how the agent, how
• 5:52
  the agent got there. I can see what the plan was.
• 5:54
  I can see the conversation, I can see what verification steps
• 5:57
  happened. I can also like set up basically
• 6:04
  environments for agents to run in that other people on the team
• 6:06
  can use. I can set up skills so things
• 6:10
  that are like repeated knowledge where you know, if I or a good
• 6:15
  examples like our person who runs our data team has set up a
• 6:18
  skill where anyone on our engineering team can now use an
• 6:22
  agent to access our data warehouse and the DBT tables and
• 6:26
  the agent understands the schema and all of that.
• 6:28
  And so we can all like now self-serve into doing data
• 6:31
  analysis when we launch new features, we want to explore
• 6:33
  user behavior or even people who aren't on the engineering team,
• 6:36
  like who are on the go to market team or on the growth team can
• 6:40
  use that same skill. And so I think a really high
• 6:43
  functioning team is building with agents right now is
• 6:45
  thinking about like, how do you accumulate knowledge?
• 6:49
  How do you make sure that everyone on the team is like
• 6:52
  working, you know, as effectively as possible.
• 6:55
  And that involves like a lot of transparency around.
• 6:57
  However, everyone is doing their stuff with agents.
• 7:00
  All right, folks, real quick. Hyperbolics GPU Cloud delivers
• 7:04
  NVIDIA H1 hundreds at $1.69 per hour and H2 hundreds at $1.99
• 7:12
  per hour with no sales calls, long term commitments or hidden
• 7:17
  fees. Spin up one GPU or scale to
• 7:20
  thousands in minutes with VMS and bare metal clusters, high
• 7:25
  speed networking and attachable storage.
• 7:27
  You only pay for what you use. Save up to 75% less than legacy
• 7:34
  providers. You need steady production grade
• 7:38
  inference. Well then choose dedicated model
• 7:40
  hosting with single tenant GPU's and predictable performance
• 7:46
  without running your own hardware.
• 7:48
  Try now at app dot hyperbolic dot AI.
• 7:54
  Let's get back into this show. Yeah, there is a lot of the
• 8:00
  boring word would be governance stuff that you.
• 8:03
  Should. Figure out here, and one of
• 8:05
  those that I've been experimenting with or just
• 8:08
  realizing that companies are butting their heads against is
• 8:11
  when do we make a skill for the whole team?
• 8:14
  When do we make a skill for the whole company?
• 8:16
  How do you track thousands of skills across the company in
• 8:19
  this skill repository or database?
• 8:22
  And then that's just for skills. What about for special prompts
• 8:25
  or special rules or all the ways that you want to now kind of
• 8:30
  solidify these practices in these operating procedures?
• 8:35
  Yeah, so it's it's exactly what companies need as they scale is
• 8:41
  you can't just make everything available to everyone and every
• 8:44
  agent or you're going to get agent chaos and you're going to
• 8:48
  get agents doing things that you don't want them doing or, you
• 8:52
  know, more sort of prosaically, like you're not.
• 8:55
  It's going to overwhelm the agent.
• 8:57
  Like agents have limited contacts, like they're going to
• 8:59
  use the wrong thing at the wrong time.
• 9:01
  And so, you know, just in the same way that we've spent as an
• 9:06
  industry building, I don't know, 20 years of SAS, 25 years of SAS
• 9:12
  for humans to to do all of these internal tasks.
• 9:17
  What we need is, is a SAS equivalent essentially for the
• 9:22
  agent knowledge worker or the agent coder to like find the
• 9:27
  right tasks to work on, to find the right data stores.
• 9:30
  And I don't think that SAS is going to look like human SAS,
• 9:34
  like human SAS has. You know, there's been a lot of
• 9:38
  work put into what are the dials and knobs and UI that a human
• 9:44
  needs to use in a tool like Salesforce or linear Notion or
• 9:47
  whatever. And that stuff tends to be like
• 9:50
  irrelevant for agents. What matters to agents is like,
• 9:54
  it's like what data is available?
• 9:55
  Can they understand the data schema?
• 9:57
  What actions can they take? How do they report their
• 10:00
  artifacts? And so it's actually the much
• 10:02
  lower level, like machine oriented set of primitives.
• 10:05
  But I do think like, what's the SAS platform that needs to exist
• 10:08
  for agents to do knowledge worker tasks?
• 10:10
  Is is is a reasonable way to think about it?
• 10:13
  So I've heard you talk about this meta App Store type of
• 10:18
  thing before, and now I'm starting to wrap my head around
• 10:21
  it because it feels like there's going to be an App Store, but
• 10:25
  it's not going to be the traditional SAS that we think
• 10:27
  of. And it's probably going to be
• 10:28
  very bespoke to each individual organization.
• 10:33
  Yeah. So, so the way I see SAS sort of
• 10:37
  developing is that increasingly the front door to any like
• 10:42
  knowledge work task is going to be what I call a like a meta
• 10:46
  app, which I think the easiest way to think about it is
• 10:51
  something like Warp. It's something like cloud Code
• 10:53
  or Cloud cowork or GBT codecs where you as a knowledge worker,
• 11:00
  the mindset that you're going to be in is like, not like I want
• 11:04
  to go use a tool to do a task. Like there's only going to be 1
• 11:07
  tool. And that tool is where you ask
• 11:09
  an agent to do something and the agent can actually do the thing
• 11:13
  for you and do like draw it. Make this more concrete.
• 11:16
  For most of my career, if I wanted to do some kind of data
• 11:19
  analysis, I would probably import the data into a
• 11:23
  spreadsheet. Like I'm spreadsheets person.
• 11:25
  I helped build Google Sheets. I would go in there.
• 11:28
  I would probably like add a bunch of formulas.
• 11:31
  I would maybe create a pivot table.
• 11:32
  I would format it. I would try to like generate
• 11:34
  some charts and then I would try to like, but like, why am I
• 11:38
  doing that? It's because I'm trying to
• 11:39
  understand something or get some insight.
• 11:42
  And today the spreadsheet for me is being disintermediated
• 11:46
  because I will just ask Warp to like do the analysis for me.
• 11:52
  And what's really cool is it's like Warp can literally build me
• 11:58
  the spreadsheet on demand if I wanted to.
• 11:59
  I, it can build me a table, it can build me formulas, it can
• 12:02
  build me charts. I just don't need to as a user,
• 12:05
  learn another interface. And it's, it's like a pain to
• 12:08
  learn how to use a spreadsheet. Like, you know, it's like, it's
• 12:12
  what people finance spend, spend years, spend years doing.
• 12:15
  It's like you, you don't actually need to shape as a
• 12:20
  human your, your skill set to the app rat like like in the
• 12:26
  past, you would need to like go like read the manual.
• 12:28
  Essentially, there's only going to be 1 manual to read, which is
• 12:32
  again, it's a tool. It's like or, or cloud code or
• 12:35
  Codex or one of these tools that can basically go from your
• 12:39
  intent to solving your problem. And the types of problems that
• 12:44
  these apps can solve is like, it's incredibly wide-ranging.
• 12:47
  Like for another example is like, I don't really make
• 12:51
  presentations anymore in a presentation tool.
• 12:55
  I just ask an agent to make me a website.
• 13:00
  And like, I asked it to read our brand manual.
• 13:03
  So the website is styled the way that I want.
• 13:06
  And it's just so much faster to have it do it via code than it
• 13:10
  is for me to like go into Google Slides or Figma Slides and start
• 13:14
  like figuring out how to like, you know, you know, draw cursors
• 13:17
  and shapes. I just would rather go right
• 13:20
  from my intent to the outcome. And so if you think of it like
• 13:24
  that, you should start thinking about like, well, what value do
• 13:28
  these SAS interfaces really provide?
• 13:33
  And I think at the interface level, it's not that much.
• 13:36
  I think there is a lot of value that they provide in terms of
• 13:38
  repeatable complex business logic.
• 13:41
  So for instance, like an app that I think is probably fine is
• 13:44
  something like Stripe where it's like it's like years and years
• 13:49
  of encoding, like regulations and integrating the payment
• 13:53
  systems and it's. Not a thing that you're.
• 13:54
  Going to do on the fly, but like there are apps, you know, a lot
• 13:58
  of apps that are just sort of a simple UI on top of a database
• 14:02
  schema that I, I, I don't want to use like our, our, I'll give
• 14:06
  you one more example. I know I'm talking a lot like
• 14:08
  our talent team built our own interface on top of our AT our
• 14:15
  applicant tracking system, our ATS.
• 14:17
  And we like it more than the than the interface that this ATS
• 14:21
  has been building for the last 10 years or whatever.
• 14:24
  And this is built by our talent team who don't know, they don't
• 14:27
  code. And so it's just like bespoke
• 14:31
  apps for these internal applications or even no apps and
• 14:35
  just go straight to the answer. Just go like from a prompt, use
• 14:39
  the database, get what you want is the future.
• 14:41
  Yeah, I've said this a few times before where it's like the agent
• 14:45
  is now the equivalent to the browser in a way, because it's
• 14:49
  your gateway to the world. That's a great analogy.
• 14:55
  That's a great analogy. It's.
• 14:56
  Yeah. We don't want 50 million
• 14:59
  browsers, we just want one browser that can get us to the
• 15:01
  web page. Yes, I think that's maybe that's
• 15:04
  even like the better word for it.
• 15:06
  Like I'm, I'm calling it a meta app, but it's like it is a
• 15:10
  browser that does things essentially.
• 15:13
  Is maybe a good way of putting it, but yeah, it's the only app
• 15:17
  that you really are going to need.
• 15:19
  Is the way of interfacing with the agent for a lot of tasks.
• 15:22
  Not for everything, but for a lot.
• 15:24
  Yeah, it feels like then your app is just a skill in a way.
• 15:30
  Yeah, it's, it's kind of true. It's like some the data needs to
• 15:34
  live someplace, right. So I think data still matters a
• 15:37
  lot. You need to guide the agent in
• 15:41
  terms of what the data means and how to use the data.
• 15:45
  And that could be a skill. There's some place for code
• 15:51
  still in terms of, you know, one of the problems with agents is
• 15:54
  that they are there's, they're non deterministic.
• 15:57
  So even if you give an agent a really good skill, if you have
• 15:59
  like a business process that needs to be done exactly right,
• 16:05
  that needs to be in code, in my opinion, like again, just going
• 16:09
  back to like a financial transaction system, you don't
• 16:12
  want an agent like inventing how to do that on the fly.
• 16:15
  But you know, the, there's just a lot of things where it's
• 16:20
  totally fine to have the agent do it on the fly.
• 16:22
  And so it's, it's going to be a really different world of
• 16:25
  software. Software is going to still
• 16:27
  matter a ton, but the, you know, the, the way you interact with
• 16:32
  it is going to be very different.
• 16:34
  And I think, honestly, I think that's one of the things I
• 16:36
  haven't heard people talk about quite enough is like, there's a
• 16:39
  lot of talk about the way that you build apps is very different
• 16:44
  now, which is true, right? There's it's very easy to build
• 16:47
  an app in war for cloud code or even like a thing like lovable.
• 16:50
  But what? But what actually is going to be
• 16:52
  the biggest change for, for knowledge workers is the way
• 16:56
  that you use apps, right? Not the way that you build apps.
• 17:01
  And so the it's the way that you like learn how to prompt and
• 17:04
  talk and work with an agent is going to be the biggest change
• 17:06
  that that's coming. So one thing that happened to me
• 17:10
  and I have an exact experience of this is I created a little
• 17:15
  electron app and vibe coded it up and I was super stoked
• 17:19
  because it did all these different things to create short
• 17:23
  clips for my podcast, right? And later on I thought, wait a
• 17:30
  minute, why do I even need an app for this?
• 17:33
  Why did I go through the process of this app when now I just can
• 17:38
  create it inside of the agent and I can tell it all right, go
• 17:43
  ahead and use this skill to extract the transcript, then use
• 17:48
  remotion. And it's like a process that it
• 17:51
  has now in a skill form. And I don't actually need an
• 17:54
  app. I don't need to drag and drop
• 17:56
  the clip. I can just say here's my video,
• 18:00
  go and do your thing. Exactly.
• 18:03
  That's how it's going to work. Like just encode the knowledge
• 18:05
  and a skill. And then this one interface of
• 18:08
  like interacting with the agent can basically get you the
• 18:11
  output. And honestly, if you needed an
• 18:14
  app, it can do what I call like build you a just in time app.
• 18:18
  It can build you a little interface in the course of
• 18:20
  interacting with the agent. But I don't know, like the idea
• 18:23
  of like these long, long lived apps where you have to learn how
• 18:28
  to use them. I think is, is going to, is
• 18:32
  going to change. I think that's actually pretty
• 18:34
  cool. I think it's to be pretty fun as
• 18:35
  someone who is it's, it's a better way of interacting with
• 18:39
  computers. The way I, I called it in my
• 18:41
  article, it's like we're all like digital gods now.
• 18:43
  We can all just like have computers do what we want when
• 18:48
  we want, you know, at least until the singularity or
• 18:52
  whatever, until they they, they stop listening.
• 18:54
  But like at this moment, we've never had, it's never been
• 18:57
  cooler in my opinion, is just be able to tell a computer what you
• 19:00
  want it to do. And you don't have to shape your
• 19:03
  behavior to the computer. You can just like it will do
• 19:06
  what you want. It's pretty neat.
• 19:07
  Well, that goes to this idea that you have on, like how
• 19:12
  important it is to be able to express your intent.
• 19:16
  And I've also said this where articulating what's inside of
• 19:21
  you that you want to get out is one of the strongest skills that
• 19:26
  we can have these days. You know, whatever's inside of
• 19:30
  you, you've got all these ideas or you've got this stuff that's
• 19:33
  doing, and how do you articulate that in a way that can then be
• 19:36
  taken by the agent and created? Yeah, this this is one of the
• 19:42
  first things that I wrote about when agents started to become a
• 19:44
  thing, which was like the limiting factor here is actually
• 19:49
  humans ability to express what they want, right.
• 19:52
  So if you're, and that's true for if you have a, an agent that
• 19:58
  can basically do what you tell it to do, it will do what you
• 20:03
  tell it to do, but it can't like yet read your mind and so.
• 20:09
  And the other thing that's interesting here is like often
• 20:12
  you don't actually know what you want.
• 20:14
  I believe. That until you start doing it
• 20:18
  and iterate. And I believe creativity and
• 20:20
  like building stuff is fundamentally an iterative
• 20:22
  process is how I work. At least where OK, I think I
• 20:26
  kind of want something that has this shape and the agent will
• 20:29
  give me something I'll be like, no, no, no, I don't want it.
• 20:31
  That's that's the wrong shape. Let's go again.
• 20:34
  And so that's going to be where what sets, I think great
• 20:39
  knowledge workers apart, knowledge work like great keep
• 20:43
  builders, you know, creator, whatever is the ability to
• 20:48
  iterate and express the intent that you want to an agent to
• 20:53
  actually get to something that's cool and useful and and like
• 20:57
  worthwhile. And so it's really interesting
• 21:00
  from the standpoint of like, OK, if you're someone who is, you
• 21:03
  know, entering the workforce or like, you know, you're going to
• 21:06
  try to figure out what do you, what should you learn how to do?
• 21:09
  You should actually learn how to communicate really well.
• 21:13
  You should learn how to like write well.
• 21:15
  You should learn how to like, think very clearly and express
• 21:18
  intent, which is, you know, it's not the same exact skill set as
• 21:25
  like learning how to program. It's, it's like a little bit
• 21:29
  different. It's related.
• 21:32
  And I wanted to get into a little more of something you
• 21:36
  glossed over earlier where working in Warp you have the
• 21:40
  whole team that can see the whole history of how APR was
• 21:46
  created. I know that you have been pretty
• 21:49
  bullish on file systems from the get go.
• 21:52
  Can you explain how that looks when I submit APR?
• 21:58
  How I can see that provenance? How everything is working.
• 22:03
  Yeah. So this is one of the big things
• 22:05
  that we're we're building. So we built this platform called
• 22:09
  Oz. And the idea behind Oz is to
• 22:11
  make it really easy to, you know, basically move all these
• 22:15
  cloud agent conversations and prompts and execution off of
• 22:20
  laptops into the cloud and get to something that's like, you
• 22:24
  know, like Google Docs type system where it's like we're all
• 22:28
  collaborating together as a team and moving from the world of
• 22:31
  like Microsoft Word on everyone's individual laptop to
• 22:35
  a world where these agents are sort of team visible and team
• 22:37
  owned. So the way that it, it actually
• 22:40
  works is like when you, when you run a cloud agent through Oz, it
• 22:48
  basically everything is cloud backed.
• 22:50
  And so the cloud backing can be stored in our database or we're
• 22:56
  making it so honestly. So I think companies want the
• 22:58
  ability to own this data, right? They want to own the
• 23:02
  conversation traces, they want to own the prompts.
• 23:06
  And so we're making everything pluggable.
• 23:09
  So if you want as a company to be like, OK, we will provide a
• 23:13
  data store into which Oz cloud agents can record all of this
• 23:17
  stuff so that we own it. Oz is basically the
• 23:21
  orchestration layer and the coordinator.
• 23:23
  And the thing that says, OK, when Zach launches an agent,
• 23:26
  we're going to, we're going to run this agent in a particular
• 23:29
  environment. We're going to run it on a
• 23:31
  particular cloud machine. We're going to write all the
• 23:33
  data to a particular cloud location and it's all API based.
• 23:39
  So, you know, that's all flexible.
• 23:41
  So it could run on our hosting or could run on another
• 23:45
  company's hosting, which by the way, is the thing that the
• 23:47
  companies really want is the ability to have these agents not
• 23:52
  lift their code out of their own infrastructure someplace else.
• 23:56
  It can write the data into our back end or it can write the
• 23:59
  data into their back end. And so our our product approach
• 24:03
  is really around flexibility for companies that want to like
• 24:08
  deploy something like this, not trying to force everyone into a
• 24:12
  particular way of doing it. Where we do where we do sort of
• 24:17
  force consistency is at the API layer.
• 24:19
  So there's a bunch of contracts where the agents like they need
• 24:23
  some place to write a conversation that conforms to a
• 24:25
  particular model. Does that make sense?
• 24:28
  Yeah. I guess where I'm questioning
• 24:31
  now is do you have a spectrum of how like locked down it is so if
• 24:36
  somebody doesn't want anything going outside, they can't use
• 24:40
  the research lab models? Like certain they're not Privy
• 24:43
  to codecs or quad code. So we support a variety of
• 24:49
  things here. What I find that companies often
• 24:51
  want is to provide their own LLM endpoints.
• 24:54
  So companies will often want to serve these models through like
• 24:59
  things like Amazon Bedrock or to cloud like Vertex or Azure.
• 25:05
  And so we, we try to support that as well.
• 25:10
  Again, there's like a real spectrum.
• 25:11
  Some companies are just like host the whole thing for me,
• 25:16
  like they don't care, they want ease.
• 25:18
  Some companies are like, we can't have inference leaving our
• 25:22
  cloud. We can't have code leaving our
• 25:25
  cloud. We, you know, and so there's
• 25:29
  just a variety of things and we're trying to build something
• 25:32
  that is, you know, we don't do like a full on Prem yet.
• 25:37
  Like we, we, we could, it's like that that starts to be hard for
• 25:41
  us to manage. But within everything but that
• 25:44
  we are trying to make it so that it's very, very flexible in
• 25:48
  terms of, you know, data ingress and egress.
• 25:51
  Because, you know, companies, this is their their IP that the
• 25:55
  agents are working on. Sometimes it's the customer
• 25:57
  data. And so you need to be really
• 25:58
  careful. Yeah, I.
• 26:02
  Can imagine why companies would want all of this provenance on
• 26:05
  how folks are interacting with the agents.
• 26:07
  But what are the big use cases there?
• 26:10
  Is it audibility? Exactly.
• 26:13
  So OK, so the big use cases are auditability.
• 26:16
  So especially if you know something goes wrong, you want
• 26:21
  to understand why. And that could be for, you know,
• 26:26
  it could be for like a security, it could be for just like, yeah,
• 26:32
  like something caused the crash. You want to figure out why.
• 26:35
  So it's like could be debug ability you want.
• 26:39
  And, and sometimes honestly for like compliance reasons, like
• 26:42
  companies, it's just like not safe to have agents doing stuff
• 26:45
  that people you'd be tracking. You want cumulative knowledge.
• 26:52
  So like, I would call it like, like like like a context
• 26:57
  management so that over time the agents improve.
• 27:01
  And so one of the foundational building blocks for that is like
• 27:06
  some way of seeing what agents have done, how they've been
• 27:08
  prompted, probably evaling like, is this especially for
• 27:13
  automations? Like is this, is this the right
• 27:16
  setup? Is this the right way of
• 27:17
  configuring the agent? So you want, if you want to do
• 27:20
  that, you need to see record of the agent runs and how it, how
• 27:23
  it worked. And then a third thing would be,
• 27:27
  you know, hand off and steering. So if you have everything in the
• 27:33
  cloud with a conversation trace, you can do things like you can
• 27:38
  have an agent start a task in the cloud and then you can have
• 27:41
  a developer pick it up and or you could have multiple
• 27:44
  developers pick it up and like, and so that that sort of
• 27:48
  functionality again, is unlocked by being in the cloud.
• 27:52
  So there's a whole, a whole host of reasons why I think it's it's
• 27:55
  a better architecture. Yeah, there.
• 27:59
  Was something that I think you said in your talk at the Coding
• 28:02
  Agents conference. We did and it felt like it was
• 28:05
  almost like these 10 commandments that you've thought
• 28:09
  about when it comes to what you would want with a tool like
• 28:15
  this. Can you go over those real?
• 28:16
  Fast. Yeah, sure.
• 28:18
  So I think you want some of them I just mentioned.
• 28:21
  I think you want an audit trace for the reasons I just said.
• 28:24
  I think you want hand off because these agents are not
• 28:29
  always able to complete a task. I think you want access control.
• 28:37
  So you want the ability to grant agents granular permissions to
• 28:43
  systems and understand like and be able to limit what they what
• 28:47
  they can access. You want memory, so which is
• 28:52
  I've been calling it like cumulative context.
• 28:54
  Memory is probably a better word for it.
• 28:56
  You want overtime the system to work better.
• 29:02
  You want evals. So you want the ability like to
• 29:05
  measure how these agents are performing over time.
• 29:11
  And then from a like deployment perspective, you want flexible
• 29:17
  deployment. So meaning like, you know, you,
• 29:20
  I think it's really important to companies that the agent can run
• 29:23
  in existing environments and you don't have to like necessarily
• 29:26
  forklift code over. And then the final thing I would
• 29:29
  say is you want programmability. And so that means anything you
• 29:35
  can do with an agent through the UII think should be honest,
• 29:40
  everything should be built as a programming stack.
• 29:42
  So there's an SDK for launching agents, there's an API for
• 29:45
  launching agents. If I want it programmatically
• 29:48
  get the artifacts in an agent produced, I should be able to do
• 29:53
  that. And so, yeah, I think that's the
• 29:55
  shape of a, a good agent system. I see a lot of companies trying
• 30:02
  to build this. Some companies will be
• 30:04
  successful building it. I think a lot of companies are
• 30:06
  going to waste a bunch of time trying to build something cuz
• 30:09
  it's very easy to like get started with the website where
• 30:11
  you launch a cloud code. But that's like not, that's like
• 30:15
  not going to get you to a real scalable solution for for big
• 30:20
  companies, it will like, if you're like Stripe and they have
• 30:22
  a product around this, you can do it.
• 30:24
  But I would think for most companies, you should be looking
• 30:27
  at what are the components that you should be using from
• 30:31
  something like Oz where you can sort of like, you know,
• 30:34
  bootstrap your agent system and not waste time building it to
• 30:38
  make sure it works well. So one thing that you didn't hit
• 30:42
  on, or maybe I missed it, was observability of the agents
• 30:46
  working. I missed this one, sorry.
• 30:49
  Yeah, this is the other one. It's like, yeah, you should be
• 30:52
  able to go in at any given time, get a link to an agent and just
• 30:56
  see what it's doing and steer it honestly.
• 30:59
  So if you want to hop in and change its direction, you should
• 31:03
  be able to do that. You should be able to do it from
• 31:07
  anywhere. Is another thing like agent
• 31:08
  access anywhere. So you could do it from, you
• 31:11
  know. AAA from.
• 31:13
  Your phone, you should be able to do it from, you know, like
• 31:17
  wherever. And so these agents are things
• 31:19
  that you're just like, you know, if you have a teammate on Slack,
• 31:23
  you should be able to like access your, your agent teammate
• 31:26
  wherever you are. Yeah.
• 31:28
  OK. So, and that is going back to
• 31:33
  what we were just saying earlier, how we interact with
• 31:37
  the agents and how we interact with apps in general is a much
• 31:42
  different experience because we're now interacting in a way
• 31:48
  that we want to see all these agents in the UI.
• 31:51
  But that might be the only UI that we care about.
• 31:55
  Yeah. I mean, I think, OK, I think
• 31:58
  your job as a builder and I use, I'm going to use builder even
• 32:03
  more broadly. It's not just like software
• 32:05
  engineer anymore. It's like even as a knowledge
• 32:09
  worker in general is going to be like, how do you effectively
• 32:13
  prompt manage and like, you know, kind of like steer a bunch
• 32:19
  of these agents to do useful stuff?
• 32:23
  And so, you know, you should imagine you're going to have a
• 32:27
  lot of them. They're going to be running on
• 32:28
  different time horizons. They're going to have different
• 32:30
  skills, they're going to have different outputs.
• 32:33
  It's going to be really like one thing that is really kind of
• 32:35
  annoying is it requires a lot of multi threading in your mind.
• 32:40
  I don't know if you feel this is your like I, I constantly feel
• 32:43
  like I'm like losing track of like, yeah, exhausted.
• 32:47
  It's exhausting. It's it's kind of a just from
• 32:52
  like a psychological perspective.
• 32:53
  This is one kind of like unfortunate outcome is like I'm
• 32:57
  getting more done, but I feel like I'm also working harder.
• 33:00
  Yeah, it's like more mentally taxing on me.
• 33:02
  We're working. Harder, but less at the same
• 33:05
  time. Yeah, I'm like, just like
• 33:07
  juggling a lot of stuff, you know, like typically when I do a
• 33:12
  podcast like this, I'm like, I have agents working on stuff in
• 33:14
  the background and I'm like, OK, I'm going to have to go back and
• 33:17
  check that PR when this is done. And like, it's, it's not really
• 33:23
  making my life easier. It's making me more productive.
• 33:26
  But it's not. I don't think that demands on
• 33:29
  people are going to be less, unfortunately.
• 33:32
  No. Well, yeah, I mean, fortunately,
• 33:35
  unfortunately, it definitely is a lot more like brain juice that
• 33:41
  you got to put towards something and.
• 33:44
  It's not, it's not the fun. It's like the, it's like the
• 33:48
  contact switching brain juice, which is what I hate.
• 33:50
  Like it's like it takes you out of flow.
• 33:52
  I would much rather be in flow coding something.
• 33:56
  Deep. But yeah, that's like more fun,
• 33:59
  but I just can't really justify doing that when I could be
• 34:02
  building 5 things at a time, you know?
• 34:05
  It's which yeah, it's it's weird situation.
• 34:08
  It is. Well, and I have to ask, I know
• 34:11
  you've answered it a few times before, but I imagine listeners
• 34:15
  are probably wondering this question of like, how do you
• 34:21
  feel like you can compete against the labs that are just
• 34:26
  pushing out subsidized tokens? Yeah, it's an awesome question.
• 34:31
  It's like it's the it's the most challenging thing for our
• 34:34
  business for sure. And so just to expand on the
• 34:36
  question, it's like when we offer AI and Warp, we
• 34:42
  essentially are like paying API rates.
• 34:45
  It's somewhat better than API rates because there are scale.
• 34:47
  We were able to like negotiate discounts with the model
• 34:52
  providers that you know, that if you're like an individual using
• 34:56
  the API, you can't really do. But the, the short answer is
• 35:01
  like, I think we are trying to be complementary to the other
• 35:07
  tools on the market. We are trying to build something
• 35:10
  that is actually a great place to use Cloud Code if you want to
• 35:13
  use Cloud Code or if you want to use Code X.
• 35:16
  We're building features that make it possible to use like our
• 35:19
  coder view and our file tree and voice input and like actually
• 35:24
  make a really nice workbench for any coding harness with a bunch
• 35:28
  of launches that are coming out around this Oz orchestration
• 35:33
  platform is also we're making it work multi harness.
• 35:37
  And the idea is just like, I think it's better for us to
• 35:42
  compliment rather than compete when we're in a period of cost
• 35:49
  being a major driver of what people choose from a pure
• 35:52
  product standpoint, I'm very biased, but I think our product
• 35:56
  is like better than a lot of these like other TUI, like I
• 36:00
  think it's better than than a lot of the other stuff out on
• 36:03
  the market. But we're not in a moment of
• 36:07
  pure product competition because the labs have a cost advantage,
• 36:13
  which is not like unfair. It's like it's their model.
• 36:16
  It's like it's fine. I do think though, for the
• 36:19
  record, it's temporary because I think open weight models, you
• 36:25
  know, NVIDIA just invested $25 billion in this.
• 36:28
  There's like reflection AI, there's a whole bunch of open
• 36:30
  weight models that are going to become good enough for the
• 36:34
  coding use case and that's going to drastically change the
• 36:37
  economics of selling tokens. But for now, I like, I don't
• 36:42
  think we should try to compete too much on that.
• 36:44
  That said, it's actually we're still having like pretty wild
• 36:48
  growth even as like a, a business of selling API, sorry
• 36:53
  of like, yeah, selling these API tokens.
• 36:55
  Like we're still growing really fast.
• 36:57
  Like we're, we're growing at like over 1,000,000 new revenue
• 37:01
  every week recently. It's all like, and it's all we
• 37:06
  don't lose money on it, right. And so it's pretty cool from the
• 37:10
  perspective of like, even though we aren't the cheapest option,
• 37:14
  it's such a good product and it's so useful to people that
• 37:18
  they're totally willing to pay a slight premium to get to use AI
• 37:24
  through us. And so, you know, it's and it
• 37:27
  also just speaks to just like the value, like the market that
• 37:30
  we're in being such an incredible growing market that I
• 37:33
  still think we can build a really big business even if
• 37:36
  we're not like the cheapest product around.
• 37:39
  That's all. That's great answer and it's
• 37:44
  interesting that you talk about swapping out harnesses.
• 37:48
  I would have considered warp to be a harness.
• 37:55
  You don't. Look, we have our own owners.
• 37:57
  Yeah, so. So we have.
• 38:01
  Well, here's how I think about it.
• 38:02
  So in case people are not totally familiar.
• 38:06
  So warp itself is the is the terminal or we, we now call it
• 38:11
  like an Ade. It's so it's a agentic
• 38:13
  development environment. It's a term that we we coined
• 38:15
  actually that's. Becoming it's like a hybrid
• 38:17
  terminal, right? Yeah, it's like a, it's like a
• 38:19
  hybrid terminal that has some coding features in it, but only
• 38:23
  a coding features that are really useful for agentic
• 38:25
  development. Within WARP, we have our own
• 38:29
  agent, which is, is think of it as like our version of Cloud
• 38:35
  code, where the advantage of using our agent is that it's
• 38:39
  really tightly integrated with the UI.
• 38:41
  And so like, it just is like you get better user experience than
• 38:47
  if you use a pure CLI. So like we give you editable
• 38:50
  diffs, we give you code review, we have LSP that runs over the
• 38:54
  agent stuff. And so there's all these nice
• 38:55
  features around using our agent and it's a really, really good
• 38:58
  agent harness. So like historically we've been
• 39:02
  many times at the top of the public evals.
• 39:05
  So one of them is terminal bench.
• 39:07
  We've been in the top three on suite bench.
• 39:09
  So it's a really great harness. But we're about choice at this
• 39:14
  point. And we, we recognize that some
• 39:18
  users honestly for cost reasons would prefer to use cloud code
• 39:22
  or codec to what user subscription or whatever.
• 39:25
  And so you can use those things in warp and have a really,
• 39:28
  really great experience that way as well.
• 39:30
  And so we, we think it's like a, you know, it's, it is, we do
• 39:34
  have our own harness, but we don't sort of force it is, is
• 39:38
  our take on it right now. It's bring your own harness or
• 39:44
  use off the shelf in a way. Yes.
• 39:46
  And like developers are like curious, right.
• 39:48
  So sometimes developers want to use Gemini.
• 39:53
  It's like I don't, it's Gemini's fine like sometimes, but like
• 39:57
  there's a totally good reason why if you're a developer, you
• 39:59
  might want to like switch between art warps, harness the
• 40:03
  cloud code harness, codex harness.
• 40:06
  There you you might want to run all three honestly, for
• 40:09
  different types of tasks. You might want to compare them
• 40:13
  with warp. You can like, you know, I think
• 40:16
  one really cool thing that comes out of work destroyed multiple
• 40:18
  harnesses is like you can have you can fire up free agents and
• 40:22
  have. Gemini, try to do a task Codex
• 40:25
  try to do a task clock code. You can have clock code review
• 40:28
  Codex's work. It's, it's just like it's a,
• 40:31
  it's a cool thing to sit one layer above from a like
• 40:36
  orchestration standpoint. And so there's real power in
• 40:39
  that. And for companies, we can
• 40:41
  provide metrics in terms of like, well, which one of these
• 40:44
  harnesses is actually performing best?
• 40:47
  And so there's there's like a nice thing just being one layer
• 40:51
  above. Yeah, that optionality, but also
• 40:54
  again going back to the observability and being able to
• 40:57
  tell, first of all, what are people preferring?
• 41:00
  And then second of all, actually, should they be
• 41:03
  preferring something else? Because like what's exactly for
• 41:06
  and on like a, a tokens per, like you can measure the
• 41:10
  efficiency of the harnesses, you can measure like the PR
• 41:13
  throughput from observability standpoint, it's really great.
• 41:16
  And then the other thing that companies are concerned about is
• 41:19
  lock in, right? So if you are, they should be
• 41:23
  honestly, like you should be concerned if you're locking in
• 41:26
  your entire stack to like clock code or codecs or something.
• 41:29
  It's like, I don't know, there's this crazy stuff where the
• 41:31
  government is designating Anthropica supply chain risk and
• 41:34
  like even beyond stuff like that, It's it's you should be
• 41:39
  setting yourself up as a company for a future where some of your
• 41:42
  models are open weight. I really strongly feel that you
• 41:45
  should not be imagining that you're going to have to pay a
• 41:48
  ton for intelligent tokens forever.
• 41:52
  And so I would be very wary as a company of like just betting on
• 41:56
  a single Frontier Lab to provide all your tokens because you're
• 41:59
  going to end up overpaying and giving it a bunch of power.
• 42:03
  Yeah, I make this joke a bunch, but I saw one of my favorite
• 42:07
  tweets was instead of having codecs review quad codes work
• 42:16
  and say hey, like review this work and tell me where there was
• 42:19
  problems. You tell one of them that the
• 42:24
  other one just went through the code base and introduced a bunch
• 42:27
  of bugs and it needs to go find them.
• 42:31
  That's smart. And so it's much more rigorous
• 42:35
  and much more like, oh man, there's bugs everywhere.
• 42:38
  We got to go find it. That's awesome.
• 42:40
  That's hilarious. So yeah, we're, I mean, we're
• 42:44
  investing also in an actual orchestrator.
• 42:49
  And So what I mean by that is a so with with Oz, we have all
• 42:55
  these orchestration primitives, you know the five things I
• 42:57
  mentioned earlier. But what you really what's cool
• 43:00
  is once you have those primitives, you can start to
• 43:02
  have like different patterns of orchestration.
• 43:05
  And so you can have an orchestrator where you have like
• 43:07
  a team leader that says, I want to, if you give it a complex
• 43:11
  task, I want it to spin up five agents to do this.
• 43:14
  We're going to have two agents that are verifying, We're going
• 43:16
  to have one that's designing. And there's, there's like a ton
• 43:20
  of different patterns you can do for this, just like there's
• 43:22
  different patterns for how humans work together.
• 43:25
  And so this is going to be native functionality that we
• 43:28
  have for making it possible for agents to do harder, longer time
• 43:32
  horizon tasks. And so it's it's going to be 1
• 43:36
  of like the most, it's just going to be like a crazy, a
• 43:39
  crazy type of sci-fi experience that you see agents coordinating
• 43:44
  each other. But yeah, that's definitely
• 43:46
  that's coming. It's pretty cool.
• 43:48
  Yeah, you have the QA agent and I imagine you have different
• 43:52
  compositions of teams that you could have again where I can
• 43:56
  bring my own and create my own composition or I can grab some
• 43:59
  off the shelf for more. Exactly.
• 44:01
  Yeah. And so my attitude on this is
• 44:03
  like, I don't know the right pattern.
• 44:05
  If it's like a hierarchy, if it's like they all work on a
• 44:08
  team, if everyone has roles. And I don't think anyone knows
• 44:12
  the right pattern. I don't actually even think that
• 44:13
  there is a right pattern. So what we're trying to do is
• 44:16
  make it very flexible so that, you know, people who want to try
• 44:21
  different stuff and can build different patterns of
• 44:24
  orchestration on what we're doing and what we'll provide.
• 44:27
  Like you said, some out-of-the-box so that you can
• 44:29
  sort of like measure and see how they do.
• 44:32
  Yeah, it's pretty cool to see agents having conversations with
• 44:35
  each other. It's like.
• 44:36
  Kind of tricky. It goes back to we're all our
• 44:38
  own little R&D departments and you never know, the pattern
• 44:43
  might be, did you ever do that in grade school where you would
• 44:46
  write like a paragraph and then it would be time and you would
• 44:49
  give the paper to the person sitting to the right of you and
• 44:52
  you would get their paper and then you would write the next
• 44:55
  paragraph. And by the end you had all these
• 44:57
  different stories that were all completed by everyone.
• 45:00
  That's so fun. That's cool.
• 45:02
  That might be the best way that agents work too.
• 45:05
  You never know until you try it. I don't know.
• 45:08
  I have a feeling that they'll work in ways that are similar to
• 45:12
  how humans organize because they're just trained on so much
• 45:15
  human, human intelligence. But fundamentally they're like
• 45:20
  alien brains. And so they they might work in
• 45:23
  ways that we don't work well together.
• 45:26
  Have you thought through the limitations of GitHub and what
• 45:33
  you want to do about that? Such an interesting question.
• 45:36
  I know there's a lot. Yeah.
• 45:39
  So, so First off, if I thought it through a little, we're
• 45:44
  feeling it because GitHub has all sorts of reliability
• 45:47
  problems right now. It's it's a huge bottleneck to
• 45:50
  productivity. You know, the other, the other
• 45:55
  place that I thought about it is like, there's no real good
• 46:00
  reason to have this like inner loop and outer loop type
• 46:05
  workflow when you're working with agents.
• 46:07
  And so we do one of the cool features that we have in Warp is
• 46:12
  you do code review within Warp, which is the same place the
• 46:14
  agent does its work. And so I think it's a stupid
• 46:17
  wasteful flow to build something with an agent, push it up to
• 46:21
  GitHub, look at their code review UI, have like a code
• 46:24
  review bot that does its thing there, pull it back in.
• 46:28
  Like, why are you doing this dance of like up to GitHub, down
• 46:31
  to your local, up to GitHub. And so I think that that part
• 46:36
  we're already like building stuff to just make it so there's
• 46:39
  no, you don't have to do that. The other thing that is
• 46:42
  interesting, it's just like. But is that through just like
• 46:45
  auto save and then you can roll back if you need to?
• 46:50
  So it's still using git. Just to be clear, it's still
• 46:52
  using Git, but the piece of it that we are moving into warp or
• 46:58
  have moved, really it's just like iterative code review.
• 47:01
  There's you could just bring it all into your app.
• 47:04
  And so it's again, if you're, if you're code reviewing with a
• 47:08
  collaborator on your team and you're reviewing their code, I
• 47:12
  think it's fine to go to GitHub. However, if you're reviewing a
• 47:16
  code that your agent wrote, that's a ridiculous flow.
• 47:20
  It's like why am I leaving my workbench to go up to GitHub?
• 47:25
  You shouldn't. You should just do it.
• 47:26
  Right as the agent is writing it.
• 47:28
  And So what you're seeing is some of the things that like
• 47:32
  we're like the hero features of GitHub, like I think
• 47:35
  collaborative code review are going to be moving out of
• 47:38
  GitHub. I think, I think there's like a
• 47:42
  more interesting question of like, is git good?
• 47:47
  First of all, I hate git. I don't know how, how you feel
• 47:49
  about Git. Like, I don't like Git.
• 47:50
  I don't, I don't understand. And part of the reason like I,
• 47:55
  I'm, I spent a long time at Google.
• 47:57
  We didn't use Git at Google. And like the thing that we use
• 48:00
  at Google was, is basically based on the Perforce, which is
• 48:04
  like another source control system.
• 48:05
  And it made so much more sense to me intuitively.
• 48:09
  And so I, I hate Git. So I'm not a big fan of Git.
• 48:11
  I would love to see the world not be working on top of Git.
• 48:15
  And I don't know if Git is great for tracking agent metadata, but
• 48:19
  the problem is there's so much tooling and scripting around Git
• 48:25
  that I think I think it's more likely that Git will stick
• 48:29
  around and you'll find ways to stick to basically build
• 48:33
  metadata around what agents are doing in to get then to invent
• 48:37
  some new thing. But I don't know.
• 48:40
  Yeah, I think I think GitHub is at risk of disruption, which is
• 48:43
  a crazy thing to say because it's it is the developer tool
• 48:48
  that has the strongest network effect of any developer tool.
• 48:52
  But I think it's AI think everything is kind of at risk
• 48:56
  right now, to be honest. And so if I'm them, I'm thinking
• 48:58
  about like, how do I rebuild this so that agents are the
• 49:02
  primary customer or not? Not necessarily.
• 49:04
  Humans all right. So this episode is brought to
• 49:09
  you by the good folks at ML Flow, the open source platform
• 49:13
  for developers who want to build production ready AI
• 49:16
  applications. Enhance your AI applications
• 49:19
  with end to end AI observability, all in a single
• 49:22
  integrated platform with ML Flows Gen.
• 49:25
  AI capabilities. You can evaluate AI applications
• 49:28
  using a suite of built in or custom judges, visualize trace
• 49:33
  executions and agentic analytics, and continuously
• 49:37
  monitor evaluations, all while tracking every run in one place.
• 49:44
  Ship better agents, faster. You know that's the name of the
• 49:46
  game. Get started at mlflow.org.
• 49:50
  There was something that you were about to say before I cut
• 49:53
  you off with the how you're doing the auto save or get like
• 49:59
  features. Was it that idea of is get the
• 50:03
  right abstraction or is get the right tool for?
• 50:05
  The job, so we, we do everything on top of Git right now.
• 50:09
  So like for instance, if we want the we're launching or it should
• 50:14
  be launched is like a checkpointing feature, right?
• 50:17
  Where if the agent does some work and you want to go back to
• 50:20
  it, we we build that on top of Git.
• 50:24
  We build our code review features on top of Git primarily
• 50:27
  because like things we want to reinvent now and there's things
• 50:31
  we want to maybe reinvent in like a year.
• 50:33
  And I don't think like we're, we, we're so stretched on the
• 50:36
  things that we want to like reinvent right now.
• 50:38
  I don't think that we, we're not prioritizing like a whole new
• 50:41
  sort of source control system. But yes, right now our stuff is
• 50:46
  built on top of Git, but it is moving some of the stuff from
• 50:50
  GitHub out of GitHub into our app directly.