Latest / Agentic Conversations (formally mlops.community) / Future of Software, Agents in the Enterprise, and Inception Stage Company Building // Eliot Durbin // #293

Transcript

Future of Software, Agents in the Enterprise, and Inception Stage Company Building // Eliot Durbin // #293 · Agentic Conversations (formally mlops.community)

• 0:00
  Elliot Durbin, I'm with Bold Start Ventures.
• 0:03
  I started it with my partner Ed SIM 2010 and we are we call it
• 0:08
  inception stage investing because seed is what we called
• 0:12
  it 10 years ago and we backed companies that eventually sell
• 0:18
  the enterprise with first check and how do I take my coffee
• 0:23
  black. I'm a New Yorker through and
• 0:25
  through, so it usually just meant it was easiest and fastest
• 0:30
  at the counter. We are back for another ML OPS
• 0:34
  community podcast. I am your host Dimitri OSS and
• 0:40
  today talking with my friend Elliot all about his last 10
• 0:45
  years, 15 now in the venture space, what he looks out for,
• 0:51
  what he's excited about. I've been enjoying talking to
• 0:55
  the VCs recently because they get to see some of the newest
• 0:59
  stuff. They get to talk to the folks
• 1:01
  that are pitching them on what the future is.
• 1:04
  And so I want a little bit of that.
• 1:06
  I want to know what they feel the future is going to become.
• 1:11
  But Elliott was also a bit philosophical and he talked
• 1:17
  about founder mindset and what it takes to really build one of
• 1:25
  those valuable companies and what he looks for when he is
• 1:30
  that first check in. So let's get into the
• 1:33
  conversation with Elliot. But before we do, if you are
• 1:36
  listening on Spotify or any of the podcast players, I am going
• 1:44
  to go ahead and give you a treat for your recommender system and
• 1:48
  play you one of my new favorite bands, Emil Rendquist Trio.
• 3:33
  Tell me what you just said, but again.
• 3:36
  We I'm relearning the same lesson over 20 years of doing
• 3:41
  first check investing, which is if I have a notion in the back
• 3:45
  of my brain that I just should invest in someone despite them
• 3:48
  maybe not having the sharpest product focus or me not
• 3:53
  believing in something, I regret it and I would be retired by now
• 3:58
  if I if I just listen to the little Jiminy Cricket going this
• 4:01
  person special, you should back them.
• 4:03
  Oh man, so many. Times.
• 4:06
  And that's the practice, you know.
• 4:07
  Yeah, yeah. Have you, do you put together a
• 4:11
  anti portfolio? Are you that?
• 4:14
  I do, I do. And I find that find that the
• 4:18
  reasons that I pass out something that ends up becoming
• 4:21
  successful are usually something that the founder figures out.
• 4:29
  And I'm it's a feedback loop. Everybody needs feedback loops.
• 4:33
  The ones in Venture are long, so you've got to survive long
• 4:35
  enough to to get them. But it always comes down to,
• 4:39
  hey, I had a special feeling or a special notion about that
• 4:42
  person's ability to privatize something brand new and to have
• 4:47
  empathy with a user and they figure out something new that I
• 4:52
  could have never imagined. And that is that.
• 4:57
  And that's what gets me up in the morning is like, can I get
• 4:59
  better at that thing? And, and there's plenty of times
• 5:02
  where I have that notion and then maybe something doesn't
• 5:05
  happen. But no one ever hears about
• 5:07
  those. So yeah, really.
• 5:09
  You don't write blog posts on those.
• 5:11
  Maybe one day, maybe one day, it'll be next chapter someday.
• 5:16
  You did say something when we first chatted that I really
• 5:19
  liked and it's been staying with me.
• 5:20
  It's like, how can I find a founder that I can just
• 5:23
  continuously invest in that It doesn't matter if this is their
• 5:28
  first company or it's their fifth company.
• 5:31
  I just want to go along for the ride.
• 5:35
  Something I didn't create. I learned it from a lot of the
• 5:38
  folks that I learned this business from and, and what I
• 5:41
  love most about when I got into investing in venture.
• 5:45
  I call it invite because I don't want to identify really as a
• 5:48
  venture capitalist. I, I just kind of spot product
• 5:52
  intuition and I can help you with things that are involved in
• 5:56
  building a company that may not be of interest to you.
• 5:58
  But knowing when a founder is on a mission, sometimes that
• 6:03
  mission takes the form of multiple companies.
• 6:07
  And I'll give you the the quick and dirty on this if you want.
• 6:11
  Best founders I found to be irrational.
• 6:15
  They could get jobs doing logical things that make a lot
• 6:21
  of money, but yet they decide this thing in their brain has to
• 6:25
  exist. Superhuman.
• 6:26
  Superhuman is a great example, right?
• 6:28
  Raul's first company was reportive.
• 6:31
  He hacked Gmail and put the sidebar in there because he
• 6:36
  wanted to make people brilliant there still does make people
• 6:39
  brilliant at what they do and e-mail was the best vehicle to
• 6:42
  do that and make folks super productive because we spent so
• 6:45
  much time there. So instead on his next company
• 6:49
  when that we couldn't monetize inside of a free platform, we
• 6:52
  didn't know you learn these things.
• 6:55
  He comes back and he says I want another shot, but this time and
• 7:00
  build the whole client or make it a Chrome extension.
• 7:03
  It's going to be Electron and it's going to be 100
• 7:06
  milliseconds or less on every single action because that's
• 7:09
  when the brain registers as instant.
• 7:11
  And if amongst the billions of Gmail users, a couple 100,000 of
• 7:16
  them want an upgrade at a premium price.
• 7:19
  I'm a venture scale company that literally that was the and and
• 7:23
  his whole pitch deck was an X through Gmail.
• 7:26
  He's like, all this, all this is hideous and it's slow and we
• 7:30
  need to, you know, it's kind of like welcome to Walmart.
• 7:32
  You've got it's built for billions of people, but
• 7:34
  superhuman was built for a narrower set of folks.
• 7:38
  So I think the best founders are on missions.
• 7:42
  But you tell me what, where can I double click on those things?
• 7:45
  There's so many things to talk about.
• 7:47
  So much, so much good stuff and thanks for.
• 7:50
  Having me, I'm a huge fan of your community, really kind of
• 7:53
  tuned into it and I've heard our founders mention it, so I'm
• 7:56
  really excited. That's awesome.
• 7:58
  That is, that is very cool to hear.
• 8:01
  The places where we can double click on that in my eyes are
• 8:05
  like, what are the missions that you've seen besides Superhuman
• 8:13
  that you're also like, very stoked on right now?
• 8:18
  Right now it's top of my what? Let's get into the the meat of
• 8:21
  it. We talk about agents and AI.
• 8:23
  Yeah, yeah. That's the hot topic this year.
• 8:26
  Well, it's exciting. I remember.
• 8:29
  So I remember I had a buddy who was at Open AI and he invited me
• 8:34
  in beginning of 2022. I just remember walking in there
• 8:39
  and I've been around a little while and the energy in that
• 8:43
  office in the beginning was, does anyone know where my desk
• 8:50
  is? Because they were growing so
• 8:51
  fast and they're like, hey, does anyone have any GPU's?
• 8:54
  Like I ran out and spent. The energy was like almost
• 8:58
  Facebook like hyper scaling, figuring things out on the fly.
• 9:02
  Everybody's kind of zipping around and I just remember
• 9:05
  thinking to myself, well, this is great news.
• 9:07
  I can keep my job for a little bit while because there's
• 9:10
  something brand new that creates magic.
• 9:14
  And it was they built that first hyper scaling consumer app.
• 9:18
  And if we look at the history of Internet even before that, I
• 9:21
  mean, it's been the hyper scaling consumer apps that have
• 9:24
  built the infrastructure that allows everyone to, I mean, a
• 9:29
  guy who sold books, enabled Uber and Netflix and things like
• 9:34
  that, they feel like build on each other.
• 9:35
  So we're at this great stage where we get to build again on
• 9:39
  an order of magnitude clip and everything's faster,
• 9:44
  everything's faster and it always surprises you how much
• 9:47
  faster as a rate. I'll give you an example.
• 9:51
  The you know, you asked me in 2015 sixteen right around the
• 9:56
  time RP as going on robotic process automation, if you
• 10:00
  remember that. Which is explain that real fast.
• 10:05
  Further, so it started out even earlier than that.
• 10:07
  It was like screen scraping and automation on your desktop.
• 10:10
  So you get automate repetitive manual tasks.
• 10:14
  Big companies in the space are Pega systems back in the day,
• 10:19
  but more recently automation anywhere, UI path, things like
• 10:23
  that. They would go to companies and
• 10:24
  say, hey, we can automate all those tasks that your labor
• 10:30
  force does in ways, and we can keep a human in the loop so that
• 10:36
  you can control it. But it's going to be very cost
• 10:38
  efficient for you going to save a whole bunch of money.
• 10:41
  And I feel like it accomplished a lot, but it, it had its
• 10:45
  limitations because it was based on microservices, lots of APIs,
• 10:48
  things broke. Biggest problems were like
• 10:51
  extraction. Hey, I've got this invoice with
• 10:55
  handwriting on it with a bunch of VAT codes and things like
• 10:59
  that. And I need to get this piece of
• 11:02
  paper that comes in via fax. I, I, some of the Fortune 500
• 11:06
  companies will remain nameless. I walked in, I said, how do your
• 11:10
  purchase orders come in and they go, oh, we have this room of fax
• 11:13
  machines. And I started though sitting
• 11:17
  there trying to figure out how to with OCR, how to put that
• 11:23
  into an SAP database. It was hard.
• 11:27
  It was and it broke. And if everything wasn't just
• 11:29
  right, it it wouldn't process correctly and wouldn't execute
• 11:34
  LLMS make quick work of that. So is the first thing I noticed.
• 11:38
  I said this can easily digest handwriting and probably even do
• 11:43
  a better for you. OK.
• 11:45
  I mean, there's a lot here. I think the first problem they
• 11:48
  ran into, especially with agents was they're not predictable.
• 11:54
  And that was a lot of the last 18 months of of time was they
• 11:59
  hallucinate. You know, my super duper
• 12:02
  engineering friends were like, it's not even close to there.
• 12:06
  I'm not letting this near production things like that.
• 12:09
  But then I met this guy who shows up and it looks like he's
• 12:14
  plugged into a a wall outlet. He's like agents lots going on.
• 12:18
  Hi, I figured something out and I open sources think like, OK,
• 12:22
  soda, what did you do? This is well, I I allowed agents
• 12:26
  to be separated into crews and put in in roles with discrete
• 12:33
  tasks. And when you do that, you
• 12:35
  increase the accuracy and you're able to run automations much
• 12:41
  more accurately at scale in more of a production way.
• 12:45
  And that was crew And he he was at clear bit before that
• 12:50
  building a lot of their ML infrastructure and got access to
• 12:52
  LMS even before opening. I started scaling and it was
• 12:56
  really hard back then. So I think when they got
• 12:59
  acquired by HubSpot, he hates when I tell the story, except I
• 13:01
  think he's he's his wife is giving him a hard time.
• 13:03
  But his wife's like you should go blog about AI because you
• 13:07
  know a lot about it and you're kind of I think driving me a
• 13:10
  little crazy. I don't know.
• 13:10
  That was the inference. I don't know for sure but.
• 13:13
  Like I'm sick of hearing about. It, well, you know, I don't
• 13:15
  know, I wasn't there. But what's cool about it was you
• 13:19
  have these folks that have this experience, like Joao Ed crew
• 13:26
  who open sourced crew AI as a way to kind of share with the
• 13:30
  community. And it grew so fast that you
• 13:32
  kind of almost was forced into thinking about how this could be
• 13:36
  a company because big companies would find it and call them.
• 13:41
  I haven't seen that stuff since like CICD back in the day or
• 13:45
  maybe even like GitHub or I think Sneak experienced a lot of
• 13:50
  it. It was built for developers and
• 13:52
  then all of a sudden by its use because it showed where the
• 13:55
  dependencies were and what libraries were being used and
• 13:59
  how the vulnerabilities in those libraries could affect your
• 14:02
  application. Yeah, it became a security tool.
• 14:07
  So in a lot of ways, crew AI, I feel like was out there helping
• 14:12
  people organize these apps and it was so easy.
• 14:15
  My, my partner, Alan's an engineer.
• 14:17
  She was on the phone with me and she set it up while we were
• 14:19
  talking to him for the first time.
• 14:22
  And she goes, this is delightful, interesting.
• 14:26
  And we didn't, I didn't know what an agent was at that point.
• 14:28
  This was February last year. And what's happened since is I
• 14:36
  think a lot of the same echoes of the past with automation,
• 14:39
  because that's what enterprises want to do.
• 14:41
  They want to automate things and they want to save money and be
• 14:44
  more efficient. So now we're, we're in the
• 14:48
  process of deploying into these enterprises and a lot of the
• 14:54
  same enterprise stack is being built.
• 14:56
  See of the, the application layer, the monitoring layer, the
• 15:00
  infrastructure layer, you know, and then you go all the way down
• 15:03
  to compute and the questions become billing pricing.
• 15:08
  How do you, how do you minimize variability in your, in your
• 15:12
  pricing so that an enterprise can consume it, but also not get
• 15:16
  stuck with a bunch of the costs because these things aren't
• 15:19
  exactly, there's a lot of innovation going on.
• 15:21
  I won't touch that one, but the cost will go down, but not yet.
• 15:25
  And it's still variable. So it's really putting companies
• 15:29
  together very fast again because of the existing demand.
• 15:35
  And when I 2015 to full circle back, I would have told the
• 15:39
  company, hey get to a million in ARR over 18 months and you're
• 15:44
  killing it. Now get to a million in ARR in
• 15:51
  couple months and go from 1:00 to 10:00 in 12 months.
• 15:57
  So everything is is growing faster.
• 15:59
  So the speed of execution you really have to look for for
• 16:04
  founders. And if you're going to start a
• 16:05
  company, just be ready to be really narrow.
• 16:08
  But when it runs, ship right into it, lean into it and run
• 16:13
  fast. Deliver what the customers want.
• 16:17
  Because you have to kind of if you're going to ride the wave,
• 16:20
  ride the wave. Don't if you're a surfer, don't
• 16:22
  pop too late. You know, especially on a big
• 16:25
  wave 'cause you're going to get tumbled.
• 16:28
  But I don't know, a lot of the, it's cool because I see a lot of
• 16:31
  the founders that have been maybe not excited.
• 16:34
  Maybe they found a company in the past, maybe they've got
• 16:36
  other things going on. They see this and they're like,
• 16:38
  I'm sorry, I have to get on that wave looks super fun.
• 16:43
  So it's an exciting time. But what am I super excited
• 16:46
  about? It's new companies that are
• 16:48
  building Greenfield agentic applications, but also are
• 16:53
  existing companies maybe a little more mature, Clay being a
• 16:56
  good example. clay.com and and Clay's been around for seven
• 17:01
  years and they're a really powerful tool for enriching your
• 17:04
  data. And then eventually Rev OPS
• 17:06
  people found it and it's become this best practice.
• 17:09
  If you're building an outbound model for sales, you pull in all
• 17:15
  your leads via, you know, all these different data sources,
• 17:18
  kind of like Clearbet offered the minute they added these
• 17:23
  things called clay agents. They have great branding.
• 17:25
  It's all Claymation and all this sort of stuff.
• 17:28
  But these agents would go out and and scour the web from even
• 17:33
  more information to add to your data set, which made their
• 17:39
  application better for the users and more people start using it.
• 17:45
  So you get this agents can be great as the foundations of
• 17:49
  software and they can be great to enhance existing software.
• 17:57
  So it's an interesting delineation between the
• 18:01
  Greenfield and the existing. All I know is that it the
• 18:03
  companies that didn't leverage that you're not getting any of
• 18:06
  the benefit of of the demand that's out there from these
• 18:10
  companies. And I think that's the best
• 18:14
  thought exercises when you have an existing product and you're
• 18:21
  trying to push the boundaries of, OK, can we find a way to
• 18:26
  make this compatible or better with AI in some way, shape or
• 18:34
  form? It might just be an LLM call.
• 18:36
  It might be a full on agent that we can add to a feature set.
• 18:41
  But the other side of that coin is you just have product
• 18:47
  managers trying to stuff AI in any nook and cranny.
• 18:53
  And so there's going to be a lot of those failures where you
• 18:56
  think that, oh, we should add AI here.
• 18:59
  And then what you find is that no AI was needed.
• 19:03
  In fact, the less intelligent this product is, the better.
• 19:10
  Yeah, in some cases, that would be frustrating.
• 19:12
  To me, that's exciting because I don't know, we're back at a
• 19:17
  stage where it's like cool to fail.
• 19:19
  It's like what? You tried it and it didn't
• 19:22
  really work. I only fault that whole
• 19:25
  scenario. If you then try and continue to
• 19:28
  make it work. You guys just recognize when
• 19:31
  something's not working and adjust best.
• 19:33
  Founders fail so fast. I tell our folks, you don't even
• 19:37
  see it. And so if you can get into a
• 19:41
  loop where you're like, OK, baby AI here.
• 19:45
  Nope, next no one will remember. But like I remember when Raul at
• 19:51
  Superhuman really taught me a lot about it because he, it was
• 19:53
  very early to this curve, adding and embedding these features
• 19:57
  that it can summarize, it can, it can draft and compose in an
• 20:02
  iterative way. And now we're starting to see
• 20:06
  that. Move forward into it can
• 20:09
  automate things for you and what I'm what I haven't seen yet, but
• 20:16
  I'm excited to it on the personal level.
• 20:17
  Like granola is one of my favorite things because it sits
• 20:20
  on the desktop and it doesn't get in your way.
• 20:22
  Is AI finding more ways to get out of the user's way and you
• 20:28
  may even start using it without even knowing it.
• 20:31
  I don't know, like it's as close to consumer investing as I can
• 20:34
  get. Yeah, the granola one is
• 20:38
  fascinating. I remember talking to the
• 20:40
  founder when he was starting it and he was like, we're going to
• 20:45
  do like a Zoom recordings, but for, you know, for like in
• 20:51
  person meetings. And they've since pivoted and
• 20:55
  people love it. It's I've seen them everywhere.
• 20:59
  I wish I'm not invested in it, but it's just such a great
• 21:02
  product. And the other thing I love about
• 21:04
  it is it's always updating. I was just double check when I
• 21:07
  when I log in, I'm like, is there a new version 'cause if
• 21:09
  there is, there could be something really cool in there.
• 21:12
  And it's a good lesson for the founders out there.
• 21:14
  The best way to to inject confidence in your users is to
• 21:19
  rapidly release things. It's like this mini dope mean
• 21:23
  hit that I get every time I update the software something
• 21:27
  new. But it's an.
• 21:30
  Example of great product, Superhuman's great product.
• 21:33
  And I think on the agent side that the challenge of me what's
• 21:37
  the interface? How does it represent itself and
• 21:43
  how do you express what you want it to do?
• 21:46
  And the biggest thing that I see happening now is companies are
• 21:50
  proud of their evals. Did the agent do what I wanted
• 21:58
  it to do? And how do you start to just
• 22:01
  monitor that at scale? So you saw Google came out with
• 22:05
  that service where you can evaluate if you're line chain or
• 22:08
  crew or or whatever. You can have a service that
• 22:10
  basically checks your e-mail structure.
• 22:14
  So it yeah, in cloud people are like, we don't need to test it.
• 22:18
  I think Netflix started testing is it was so high up time,
• 22:21
  right? We didn't see that stage for
• 22:24
  years when cloud came out. Everybody's just throwing stuff
• 22:28
  against the wall. But to have it already, the
• 22:32
  conversation already starting to tilt towards testing.
• 22:35
  That's pretty wild to me. But I think it's because
• 22:39
  businesses are back to the demand that I have never seen
• 22:43
  demand like this. You have two people, one person.
• 22:48
  I have a one person company that just sold their product to
• 22:52
  Microsoft because Microsoft found it and started using it.
• 22:56
  Wow, looks at me and he's like, I, I think we should get some
• 23:02
  more people. I said I think so.
• 23:05
  But then again, he's like, well, maybe not because there's almost
• 23:07
  this culture of I can do more with less.
• 23:11
  So let's think carefully about it.
• 23:13
  I think that that we're going to start to see more and more
• 23:15
  efficiency on the business building side as well.
• 23:19
  So I it's pretty broad. So I can I can double click
• 23:23
  anywhere in there. What do you think?
• 23:28
  A first of all, just like it is it insanely competitive piece of
• 23:35
  the stack. And so that's one part.
• 23:39
  The other part is, are you not worried that it is built on top
• 23:46
  of other open source tools like I think it uses and leverages a
• 23:52
  lot of lane chain, right and. Then it did.
• 23:54
  Now it's deep. So it's moved away from that
• 23:57
  since. It's moved towards a more
• 23:59
  agnostic OK framework because I, yeah, I think that that
• 24:05
  everybody's standing on shoulders whether they know it
• 24:07
  or not. And if you're not, you're
• 24:08
  probably spend a lot of money and you're probably a pioneer
• 24:10
  and you're early and OK, cool, right.
• 24:15
  But I think that what crude did initially in my brain and you,
• 24:18
  you know, you'd have to ask as well what, what was in his head,
• 24:22
  but he made it super easy and it was the easiest I'd ever seen.
• 24:28
  And I think that that for new stuff, if you can make it really
• 24:32
  easy and intuitive for, for folks that express themselves in
• 24:37
  code, you're going to up the chances that someone create
• 24:45
  something very big using your framework.
• 24:49
  And that's exactly what happened.
• 24:50
  It's like big company coming in automating things.
• 24:55
  I mean, one of the big accounting firms was like, yeah,
• 24:58
  we're looking at kind of automating our, our audit
• 25:01
  function. And I'm like, OK, I mean, I'm
• 25:08
  sure there are things wrong with that, but you got, you got 2
• 25:10
  choices. You could be kind of irrational.
• 25:13
  They're like, cool, let's figure out what breaks and and fix it
• 25:16
  and just keep shipping. And you know, you see the
• 25:18
  boulders and this happens with any good startup.
• 25:20
  You see the boulders in front of you and you just try and smash
• 25:24
  them as quick as possible by shipping stuff.
• 25:27
  And that's what he's doing. So I think the value of crew is
• 25:30
  that it's super easy, but also it gives you a lot of control
• 25:34
  and I think there's a lot of ways to do that.
• 25:35
  The way that crew is doing that is I think native natively to
• 25:41
  the next generation of apps versus the prior.
• 25:45
  And I think other frameworks may be going whether it's a graph
• 25:49
  infrastructure or or some sort of kind of design that will
• 25:54
  allow a, it's kind of like saying analog to digital Crew's
• 25:57
  going to allow you to control your agents and, and the legacy
• 26:04
  that it creates. I asked him, I was, I told him I
• 26:06
  was coming on the the podcast today and I said, if you got one
• 26:10
  message out there about crew, what would it be?
• 26:14
  And he says the decisions you make now with your tests will
• 26:18
  dictate how much maintenance and core infrastructure you have to
• 26:24
  build to maintain your applications for the next
• 26:27
  decade. So think carefully about how you
• 26:30
  design your genetic systems, whether you build them yourself,
• 26:33
  You use a third party, you're using multiple frameworks,
• 26:37
  multiple models. You just have to create a way
• 26:40
  that it can leverage the, the, the innovation of agents
• 26:44
  themselves to self heal, to, to monitor each other, right?
• 26:51
  Cruise that monitor cruise is one of my favorite things.
• 26:54
  You know, I always, I, I told them we should probably get like
• 26:58
  emojis or something like the clipboard crew.
• 27:00
  Yeah, check, check, check. But is it built on other things?
• 27:06
  Absolutely. But like LMS, we couldn't have
• 27:08
  them unless we had Internet. We wouldn't have Internet unless
• 27:11
  we had like Google, right? So everything I feel like is a
• 27:15
  dependency on something else. But that's, I don't worry about
• 27:18
  these things. I try and get ahead of them.
• 27:21
  And I think that the tide's rising.
• 27:22
  So the real question is who can take those opinions from the
• 27:25
  market and create the best product?
• 27:27
  Yeah. Now thinking about the way that
• 27:31
  they manifest, I really liked how you put it where the chat
• 27:37
  and how we interact with these agents is still yet to be
• 27:42
  determined. And maybe that's not the best
• 27:44
  way forward. And there's a lot of people that
• 27:46
  are trying to find new ways. They're actively trying to
• 27:50
  figure out like, what can we do instead of the chat?
• 27:53
  And then the other thing is like making sure they do what they
• 27:56
  do. And I you, you kind of broke
• 27:58
  down three different areas that I would love to talk a little
• 28:03
  bit more about because there's how we interact with them and
• 28:06
  then how we know that they're doing what they're doing right
• 28:09
  and. Let me zoom out one level higher
• 28:11
  because I think that's the right, yes, that's all happening
• 28:15
  now. But I think for context anyway,
• 28:16
  that's listening to this, it's going to be like, OK, that's a
• 28:19
  little too in the week. You've got this journey
• 28:23
  happening and I had a head of AI engineering from one of the top
• 28:27
  five banks. I won't mention which one so
• 28:30
  they don't have the media approval, but.
• 28:34
  Don't want to get sued? I we're on this journey where
• 28:40
  you have, it starts with Co pilots on one side of the
• 28:43
  beginning of the journey, you have agents in the middle and
• 28:47
  then you have autopilot. So Co pilots to autopilots,
• 28:51
  agents of the bridge. So if you think about things
• 28:56
  from that kind of framework, you're going to need to trust
• 29:00
  that system, create a copilot so the humans get in there and are
• 29:03
  empowered with in the loop to do the things they do repetitively,
• 29:09
  faster, better, more accurately. We're seeing that in things like
• 29:12
  QA. You can replace your whole QA
• 29:16
  team. Now for those companies that
• 29:18
  have QA, right, Best engineers are like, I write my own hand to
• 29:21
  hand test. I'm like, well, that could be
• 29:22
  automated in cursor too. They're already automating
• 29:27
  things. Best engineers.
• 29:28
  But the QA teams for the more mature companies can go away and
• 29:31
  those agents can be more accurate with playwright than
• 29:36
  any human could ever be. That's the latest thing that I'm
• 29:40
  seeing. So you literally can walk into a
• 29:43
  company and be like, do you do QA?
• 29:44
  Yeah, we don't like it. None of our engineers like it.
• 29:47
  It always causes reproducibility into all these things.
• 29:51
  That's gone. It is now a no brainer to do
• 29:56
  that with agents. How do you interact with that?
• 30:01
  Well, you're going to have to figure out little UX components
• 30:04
  and ways to kind of notify the user that it's been done and
• 30:08
  that it's been done correctly and allow them just a nice
• 30:11
  mental check. Yeah.
• 30:15
  That's that's a a design experiment in a lot of ways.
• 30:20
  So I think the first interface will be at the application
• 30:24
  level, interacting with the user on a copilot basis that says hey
• 30:29
  we made this better and you can trust us.
• 30:32
  Cool. Then you get into agents and
• 30:34
  agents that start automating things underneath in an even
• 30:38
  more discreet way. Say in a multi agent system that
• 30:44
  I used the original crew example research as a blog post, writes
• 30:49
  A blog post, allows you to edit multiple versions of that blog
• 30:53
  post, then post it to LinkedIn or something.
• 30:57
  That's another kind of interface because then you're designing
• 30:59
  more discreet agents that can do things and you kind of eval each
• 31:03
  one. Did they do the right thing?
• 31:05
  You know, I don't want to automate this fully because
• 31:07
  that's kind of like my brand out there and I'm not sure it's
• 31:11
  going to start spewing a whole bunch of stuff that I don't want
• 31:13
  to associate with my brand. But you can do that with Cruise
• 31:16
  Innovation, which is discreetly organized multiple agents into a
• 31:20
  system. What comes next is the wild
• 31:24
  stuff, which is autopilot. Once you trust those systems and
• 31:28
  then what is that going to allow us to do?
• 31:32
  What's that going to free up a whole bunch of time to do?
• 31:36
  If you no longer have to test your software, that gives you
• 31:40
  time back to do even more. So I think that that that is
• 31:45
  coming. I haven't seen anything that's
• 31:47
  that defined on the on the future end of the spectrum.
• 31:53
  So we're just kind of in that first phase moving from Co
• 31:56
  pilots to agents to eventually I think autopilot.
• 32:02
  So going back to this UX idea and almost looking at it as
• 32:08
  different knobs that we can turn and different ways that we can
• 32:11
  try and experiment with the human to copilot relationship
• 32:17
  that we have now in the first innings of this journey, I've
• 32:23
  been thinking a lot about how 1 design pattern that's coming up
• 32:28
  seems to be firing off agents from Slack or from like a chat.
• 32:35
  That is OPS. Yeah, it's it's very much chat
• 32:39
  OPS. I like that term.
• 32:40
  I haven't heard it before but and I got a whole.
• 32:42
  Book of those, if you like those, yeah.
• 32:46
  It's, it's funny because there is almost this world that I
• 32:51
  could see where you are directing your agents and
• 32:58
  orchestrating your agents in the place where you are talking and
• 33:04
  having the most communication with your team and so.
• 33:08
  And it's unstructured and that's quick work for these things.
• 33:12
  Well, and you're maybe you're making decisions in a thread and
• 33:15
  then you just tag in the agent and say, OK, we made the
• 33:17
  decision, now go do it. Yeah, we're seeing that in
• 33:20
  support our company customer with AK.
• 33:24
  You know, there's some of the best engineers that I know.
• 33:26
  And when we back them originally it was because the graph
• 33:28
  database existed. They're like, hey, we've built
• 33:31
  like 5 customer support companies and now there's a
• 33:34
  graph database so we can make a timeline and get rid of tickets.
• 33:37
  Now what they're, they're saying they're going, OK, now we have
• 33:40
  vector databases. So we can literally have virtual
• 33:44
  agents that can do everything a human can do.
• 33:47
  But we'll start with automating this kind of and you lower the
• 33:50
  exception queue of stuff that the agent can't handle.
• 33:54
  But to your point, that's, that's kind of 1.0 interfaces
• 33:58
  moving to 2.0 in this space from a UX component.
• 34:04
  It's all like in that user trust, I feel like to automate
• 34:08
  things and once the user starts automating things or or requests
• 34:13
  or demands that, that's when you're ready to take the next
• 34:17
  step. But I don't know, I sit on the
• 34:18
  VC side of the table. So I'm, I'm not very good at
• 34:21
  making those decisions. I just get to point when it
• 34:24
  works and be like, Oh yeah, that's how you do it and you
• 34:26
  need. To take credit for it and say I
• 34:28
  saw. I don't I it's one thing I don't
• 34:31
  do. I was, I grew up around really
• 34:32
  senior founders who were like, no, no, no VCs are, you know,
• 34:37
  maybe seen but not heard. But that's changed a lot, you
• 34:40
  know, as you gotta be no ego job because it's not about us.
• 34:48
  It's about supporting the founders who kind of figure
• 34:50
  something out that's big that everybody wants and helping them
• 34:54
  figure out how to deliver that with quality at scale really
• 34:58
  fast. That's really the job.
• 35:00
  It's how do I scale a company to meet the demand that's out
• 35:03
  there. And with these agents, I mean,
• 35:06
  companies are buying them. If you're selling an AI agent to
• 35:09
  an enterprise, I mean across the board you are seeing more demand
• 35:14
  than I've seen in the history of doing my job.
• 35:17
  Amazing. Going back to that, and it's
• 35:19
  this is one way that I imagine you are having lots of
• 35:23
  conversations with founders about the agent space is the
• 35:27
  pricing discussion that you were talking about earlier and how
• 35:30
  much of A thorn in the side that can be.
• 35:32
  Because you don't want someone to have to think twice about
• 35:37
  using your agent because they're thinking about the price tag in
• 35:41
  the but you also don't want for some random reason they fired
• 35:47
  off a few more agents this month and now the bill went up by 40K
• 35:52
  and the engineering managers coming in and saying what the
• 35:55
  hell. Happened.
• 35:56
  It's like a fire alarm goes off and your customers like the
• 35:58
  budget change what you know, or they say, no, no, I only want
• 36:03
  open AI, not Anthropic. And you built your thing with
• 36:08
  open and you're like, right. So there's a lot of a lot of
• 36:12
  figuring that stuff out. I mean, on the extreme level,
• 36:19
  about 6-12 months ago, I started to see the labor component of
• 36:22
  this. So it's replacing human labor
• 36:26
  and historically speaking, the sales cycles for a a software
• 36:29
  license and SAS to a big enterprise.
• 36:32
  You have to prove your ROI, right?
• 36:34
  And that takes 6 to 9 months. You know, in the best case,
• 36:37
  maybe three months. And I think they went from being
• 36:40
  called PO CS to PO VS. I don't know when that switched.
• 36:43
  What is the V? For I, I I've.
• 36:45
  Heard that too. Proof of value concept.
• 36:49
  I don't know. There's a lot of acronyms.
• 36:51
  I, I try to avoid them because I get confused, but you know, they
• 36:55
  test it out to see if it's value more valuable and it doesn't pay
• 36:58
  for itself, especially in, in this climate.
• 37:01
  I think that's a lot of the fast growing companies are replacing
• 37:04
  three things and then creating a ton of value.
• 37:06
  So they get the budget. But with pricing, I see a lot of
• 37:11
  them going and saying, you know, for my first couple of
• 37:13
  customers, let's sell it as a service that works.
• 37:19
  But that only works for so long. I think what we're seeing,
• 37:23
  Palantir is my favorite example. My partner Ed and I go back and
• 37:26
  forth on this all the time because a lot of people doubted
• 37:30
  Palantir's model in the beginning with the four deployed
• 37:32
  engineers. They're like, I don't want to
• 37:33
  send a bunch of engineers to an oil rig and know the ocean.
• 37:37
  So like figure out data processing at the edge, like,
• 37:41
  but turns out their sales cycle super efficient because they,
• 37:47
  they almost get in there with customer and they just set it
• 37:52
  up, but they set it up with the customers.
• 37:53
  They're training on it. And you have the Palantir AIP,
• 37:57
  which is it's a great product. It was very early, but like the
• 38:00
  adoption of that thing is wild. So I'm actually seeing a lot of
• 38:02
  our startups adopt that model. So you got to go in there with a
• 38:07
  forward deployed engineer. Oh, open AI just put job specs
• 38:10
  out for four deployed engineers. Check it out.
• 38:13
  I I quickly sent it to all my companies.
• 38:15
  I'm like, hey, this is a great 1 great artist steal.
• 38:18
  So just take the elements that you want and put J DS out there.
• 38:22
  You know, I think it was a really great rack.
• 38:24
  So kudos to now, but I think a lot like early anything you're
• 38:29
  going to have to deploy experts in to train your customers, but
• 38:33
  you don't want to call it training like workshop.
• 38:36
  You can charge for a workshop and I think a lot of the
• 38:42
  companies that I've seen if there's a best practice involved
• 38:47
  or a skill set involved on how to leverage the software
• 38:49
  appropriately, the best way to train your customers is to send
• 38:53
  an expert in that they can they can utilize to set it up.
• 38:58
  And that's fundamentally what we're seeing in a lot, whether
• 39:02
  it's a security company, whether it's, you know, even crew, I
• 39:05
  mean, it's everyone needs a little bit of expertise with
• 39:08
  their software so that they can practice the best way.
• 39:12
  But they're not going in as a consultant and then installing.
• 39:18
  That's true. That is the core fundamental
• 39:23
  muscle you got to build this product muscle.
• 39:25
  And the example I give a lot of our our founders that have not
• 39:28
  done that before is your answer to things can't be bespoke.
• 39:34
  They can just make sure you add 2 zeros to the cost of that.
• 39:41
  But you got to productize things.
• 39:44
  You know, it's like, hey, how do we productize our onboarding
• 39:47
  process so that an engineer that's forward to play can go
• 39:52
  into a customer, but they are teaching them how to use the
• 39:54
  product and then onboard other teams.
• 39:57
  So you're either a sales driven company, which I think a lot of
• 40:00
  consulting firms are. Or you're a product driven
• 40:03
  company or you're a hybrid. And I think it's really great to
• 40:06
  be a hybrid because then in the beginning you can go in and
• 40:08
  figure out what's most important, privatize that and
• 40:12
  then see if it translates to the next customer.
• 40:16
  And if it does, that's a win. Celebrate it.
• 40:19
  Startups are hard enough, yes, celebrate these things.
• 40:22
  But if you can get the same customer, you know one customer
• 40:24
  up and then the next one, you can reuse some of those
• 40:27
  components even with human assist.
• 40:32
  I think that's the new definition of repeatability.
• 40:35
  Yeah, that you were talking about evals and how testing
• 40:40
  didn't come till much later in the cloud life cycle.
• 40:43
  And now evals are here very early, which is a form of
• 40:47
  testing in a way. And there's a lot of pieces in
• 40:50
  the cloud that are very mature. And I wonder if you feel like
• 40:55
  they're going to also translate to the agent world?
• 40:59
  Well, let's, let's take what you just said for a minute and just
• 41:02
  kind of really hear it. The cloud is mature.
• 41:06
  I remember the days where, you know, enterprises were like, I'm
• 41:10
  never, ever going to the cloud with my Tier 1 workloads.
• 41:15
  That's, that's crazy. Well, then the CIA did it.
• 41:18
  Everybody's like. If they can do.
• 41:21
  It we, we can reduce our costs. And I mean, I just remember the
• 41:24
  last 10 years we're, I'd say 15 watching those global Msas, the
• 41:31
  master support agreements and sales agreements to go out and
• 41:35
  literally take department by department of the Fortune 500
• 41:38
  and put them in the cloud. I think that that set up the
• 41:43
  infrastructure for what we're seeing now.
• 41:45
  So there's CICD, right, there's source control, you know, and I
• 41:51
  think developers are relatively empowered, but now there's this
• 41:56
  huge gain. So testing it relative to the
• 41:59
  existing software isn't with the whole company.
• 42:03
  Tessel who's championing the AI native development wave.
• 42:10
  But more importantly, I think how enterprises will consume
• 42:13
  that and what's important to them.
• 42:16
  And if we're at a a stage where you have these agents that will
• 42:20
  write code, that's a lot of code.
• 42:24
  That is a lot of code, that's more codes going to get written
• 42:27
  this year than I think probably last 10 years because it's now
• 42:32
  like, I remember talking to engineers back 2010 and they'd
• 42:35
  be like code generators. I'm like, never work.
• 42:38
  Never. No one will ever trust that
• 42:40
  code. Now it's like, Oh yeah, I just
• 42:43
  scripted up my whole integration with my database.
• 42:45
  And you're like, what? So I think that that testing is
• 42:49
  going to be more important because there's a lot more code
• 42:52
  that's being written. And I think that it doesn't
• 42:56
  change the uptime that you need. It doesn't change the quality
• 42:59
  that you need to ship your software with.
• 43:02
  So it's, it's makes sense to me that testing is happening
• 43:05
  faster, but it's also the fact that sometimes these things
• 43:08
  hallucinate. And you know, less than a year
• 43:11
  ago we were worried about the accuracy of agents completing
• 43:15
  the task as we expect. So I think new ways of testing
• 43:19
  will be created. I'm spitballing here, but I'm
• 43:24
  thinking about looking at how you would analyze a company.
• 43:29
  And let's say that I come to you and I say I've got this amazing
• 43:34
  idea for a company that it does alerting for agents when they
• 43:41
  spend too much money. It's basically you're alerting
• 43:45
  software. Are you going to tell me that's
• 43:48
  a feature that AWS is going to build in natively?
• 43:52
  Is it going to be something that you are just like alerting is
• 43:57
  too small of a pie? And and then maybe I say, OK,
• 44:01
  well. I'd ask you why because I just
• 44:05
  to be clear, so bold start, we started in 2010 with $1,000,000
• 44:09
  fund. We're now investing out of our
• 44:10
  6th fund Ed and I have been backing people, specifically
• 44:15
  engineers that have a problem space that they want to solve
• 44:19
  and they have an opinion on how it needs to be solved.
• 44:23
  That's the earlier point, almost irrationally so.
• 44:26
  Guy from Sneak, his last company is called Blaze IO and Blaze was
• 44:30
  solving mobile website caching, got acquired by Akamai.
• 44:35
  So he sat at Akamai and he basically saw that everyone was
• 44:38
  shipping faster and that was going to create a big security
• 44:41
  hole. So it comes in, I think.
• 44:44
  I think the the vulnerabilities inside of these dependencies
• 44:47
  that developers are using without even knowing it are
• 44:51
  going to be a huge problem. And then Apache Struts and
• 44:56
  Equifax, if they were using Snake, it wouldn't have been a
• 45:00
  problem because it would have patched that vulnerability.
• 45:05
  So all of a sudden things were, Oh my God, how do we, how do we
• 45:07
  fix this? And turns out security doesn't
• 45:09
  really didn't at the time really know a lot about how developers
• 45:12
  worked. So he built this tool for
• 45:13
  developers. So he made the big bet that I
• 45:17
  might build a developer tool that will eventually enhance
• 45:20
  security and become a security product.
• 45:24
  Same thing applies to alerting. Why?
• 45:27
  Why is alerting such a big thing and why will it become even more
• 45:30
  important? Is it because that creates the
• 45:32
  triggers to automate other things?
• 45:35
  Interesting. But if you're if you're looking
• 45:38
  to start a company, stop thinking about a company.
• 45:42
  Companies are built around really successful products and
• 45:47
  successful products are created by people, people with an
• 45:51
  opinion that this is the way that is done best.
• 45:54
  And the more credible the opinion, the easier my job is.
• 45:58
  So some days I'm like, Oh, yeah, yeah, yeah.
• 46:02
  You're coming out of coming out of some great company that
• 46:05
  scaled really fast and this is the way you did it makes sense
• 46:09
  that other people are going to do it that way.
• 46:11
  And they'll probably listen to you because you're fully
• 46:14
  credible and you'll be able to recruit people and you'll be
• 46:16
  able to get your first customers.
• 46:18
  In other cases, they're like, but there's demand out there
• 46:22
  already. And I, I built a thing that I
• 46:25
  just kind of put out there like crew.
• 46:26
  And all of a sudden it's growing.
• 46:28
  I mean, crews growing faster than anything I've ever seen
• 46:31
  ever and been involved with. So handling that demand, I feel
• 46:36
  like is is the big challenge. So the why in your case on
• 46:41
  alerting, we'll go back to why is that important?
• 46:46
  Why will that be important? What will your users realize
• 46:50
  that that can do? What do you want to prove is
• 46:52
  possible? Brilliant in a static
• 46:56
  environment? Any ideas, good or bad?
• 47:00
  Yeah, but I think it's the the what does it prove if you get
• 47:06
  that first revolution on the bicycle successfully done?
• 47:10
  Can we? My favorite example is can we
• 47:14
  rent out the couch in our spare bedroom to someone we don't know
• 47:18
  from our phone? I mean, in the beginning,
• 47:22
  literally everyone passed on that because, like, what are you
• 47:25
  talking about? Why do I do that?
• 47:28
  Kind of intuitive now? And so it proved that it was
• 47:32
  possible to create, you know, Airbnb and whatever it was, it
• 47:36
  was. I'm not invested in that, but I
• 47:37
  saw it happen. And I'm just like, wow, cool.
• 47:43
  And there is something that you said there that I really like,
• 47:46
  which is just how, what does that enable?
• 47:50
  So you come with an idea and what does that get you?
• 47:55
  Is there more to it? And and then also the founders
• 48:04
  usually have strong opinions about how the best way to do
• 48:08
  something is. Oh we just backed a founder who
• 48:10
  had to deal with vault all day every day.
• 48:14
  High score vault. Amazingly successful product.
• 48:19
  And with agents, though, I'm not so sure it's best way anymore.
• 48:27
  And if agents can go in and read all the identities and have
• 48:29
  their own identity and secure all the applications that are
• 48:33
  kind of communicating with each other and keep your secrets and
• 48:36
  make sure it's available at the right place, right time to the
• 48:38
  right user. I mean, there's fire behind
• 48:44
  their eyes because they're like, OK, we can replace that now.
• 48:48
  Oh, all right, let's go. And I mean, that's, that's
• 48:53
  really what I look for. It's like, do you have this is
• 48:56
  burning need to like just put a dent in or make something better
• 49:03
  or get rid of something that is just terrible.
• 49:06
  But there was no better alternative.
• 49:08
  So on the tool builder side, great.
• 49:10
  We're investing a lot of the application level because I
• 49:12
  think that's the stage we're at right now.
• 49:14
  So whatever you build, it has to be for users.
• 49:18
  It doesn't have to be infrastructure for
• 49:19
  infrastructure's sake. I don't, and I don't do a lot of
• 49:21
  the, I don't invest in validational models.
• 49:24
  I'm not, you know, looking for the not sitting at the big poker
• 49:27
  table as they say. I'm much more comfortable at the
• 49:30
  application level or the kind of tooling for applications at the
• 49:36
  moment and security and, and I think a lot of the, the
• 49:40
  workflows that can be automated are exciting, but these things
• 49:44
  are integrating databases faster than I've ever seen.
• 49:48
  I think we'll see new Erps coming out.
• 49:50
  I think we'll see new PLMS coming out like stuff acronyms
• 49:54
  from, you know, Jurassic Park back in the day.
• 49:57
  You're like, I could redo this now.
• 49:59
  And I I'm just looking for the right founders who, whether it's
• 50:03
  through experience or maybe having the pain themselves, just
• 50:06
  have a very strong opinion about how it should be done so they
• 50:10
  could share that. I just saw Paige Bailey who
• 50:16
  works as a dev advocate at Google I think on the Gemini
• 50:20
  team post about how there was this Hacker news posting where
• 50:26
  someone said we had this whole OCR software and it has a human
• 50:33
  in the loop and we were using it.
• 50:37
  But since testing with Gemini 2 point O, we now just got rid of
• 50:44
  the OCR software because Gemini does it better and faster and it
• 50:49
  is a much easier experience. And so I wonder about when
• 50:54
  you're, when you're talking about applications, there's
• 50:57
  going to be those subset that are applications, but then
• 51:00
  there's probably going to be a lot that just gets done.
• 51:03
  Oh, yeah. But that was the case back in
• 51:05
  the cloud too. We used to have meetings,
• 51:07
  Amazon's wised up to it there used to have meetings for us
• 51:11
  that reinvent for all the VCs. Which companies are we going to
• 51:14
  kill that? I mean, it's part of it.
• 51:18
  But they, you know, they gave us a heads up, which was nice, you
• 51:21
  know, and so I get, I can give them, listen, if you're going to
• 51:24
  go into the jungle, get ready. There are lions and tigers and
• 51:27
  bears in there and you probably won't make it.
• 51:31
  And I think a lot of as, as, as an investor, I think that is
• 51:35
  like the most important skill set.
• 51:37
  Be patient, don't get in the founders way because you can't
• 51:39
  fix things. But more importantly, you got to
• 51:44
  be comfortable that it might not work because if you're not
• 51:48
  comfortable with that founder is going to have a tough time being
• 51:50
  comfortable with it. And the only way it works is if
• 51:52
  they're real comfortable to spot that failure.
• 51:54
  And just, I had companies that literally wouldn't have made it
• 51:57
  unless Opening eye came out. And then they started leveraging
• 52:01
  LLMS and stuff started to grow really fast.
• 52:03
  So it goes both ways. And the last thing I'll say
• 52:06
  about that, yeah, Amazon, Google, they could kill your
• 52:09
  business. It's happened over and over,
• 52:12
  over. But you can move a lot faster
• 52:15
  than them and they're going to build a generalized product.
• 52:20
  And if you pick the right narrow set of users and right narrow
• 52:24
  kind of market to go after that, they may not kind of custom fit
• 52:30
  their solution to Sage maker be a good one.
• 52:35
  You know, the fault injection service being, you know, on and
• 52:38
  on and on. You could create the best
• 52:41
  practice that they will eventually have to write back
• 52:46
  to. We saw that snake.
• 52:47
  I mean, all the big companies were like, well, I'm building a
• 52:50
  vulnerability database and guy was like, well, you can write
• 52:53
  back to mine if you'd like. I was like, wow, well, that's
• 52:58
  best matters are like defiant, you know, and, and the best day
• 53:02
  was when, you know, they they said, OK, we'll write back to
• 53:04
  it. We'll integrate you in
• 53:05
  lighthouse or whatever it is. But if you really believe that
• 53:08
  you have the best way and you can endure that game, you know,
• 53:14
  kind of chicken almost, or you're like a Firebase and OK,
• 53:22
  well, maybe I'll sell the Google and then I'll enable all the
• 53:25
  mobile developers with my best practice because this is the way
• 53:29
  to do it. I want to make, I want to make
• 53:30
  everyone better. And that that's to me, the the
• 53:34
  bullseye. I don't always get right.
• 53:36
  But if I can find a founder like that, they're not going to give
• 53:40
  up. But more importantly, I think
• 53:41
  that they're going to see the mission is continuing on
• 53:45
  whatever happens to their company because it's it's what
• 53:49
  they're about in a lot of ways. So yeah, the risks are there.
• 53:53
  It's scary. They're releasing all sorts of
• 53:55
  things. Probably do it as a service.
• 53:57
  Amazon will probably release it before they even have a product
• 53:59
  for it. But if you've got the best way,
• 54:03
  you've got the best way. Yeah, and Amazon will steal some
• 54:08
  open source version of it. It's very high level like you
• 54:10
  know, I'm trying to too, too abstract.