Latest / Agentic Conversations (formally mlops.community) / What Business Stakeholders Want to See from the ML Teams // Peter Guagenti // #222

Transcript

What Business Stakeholders Want to See from the ML Teams // Peter Guagenti // #222 · Agentic Conversations (formally mlops.community)

• 0:00
  On Peter Borgetti, I'm president and chief marketing officer of
• 0:03
  Tab 9. Tab 9 is the AI coding assistant
• 0:06
  that you control. I am a coffee addict, and I am
• 0:09
  usually if you find me at a coffee shop, it'll be a dry
• 0:11
  cappuccino. At home, it's just black coffee
• 0:14
  and pot. Welcome back to another
• 0:18
  Envelopes Community podcast. I am your host, Dimitrios.
• 0:22
  Today we're talking with Peter G Wow, I loved it.
• 0:26
  OK, I'm going to be straight with you all.
• 0:28
  I usually say no to people that reach out to me and if they are
• 0:33
  not engineers or they do not have a technical background, I
• 0:36
  usually will say no, not right now.
• 0:40
  Maybe later and later never comes.
• 0:43
  Let's be honest, this is the first time that I had someone
• 0:48
  from a marketing department on the podcast.
• 0:52
  But I do not look at Peter as a marketer.
• 0:55
  He talked about his whole life and how he started in tech and
• 1:02
  how he went through so many different companies, building
• 1:05
  companies, selling companies, joining companies.
• 1:08
  And the reason I wanted to have him on here was because of his
• 1:13
  diverse range of experience right now as a marketer.
• 1:16
  But he's been in product. He's LED teams that dealt with
• 1:21
  ML and AI. He's also been the CEO of a
• 1:26
  company that scales from 20 to 300 people.
• 1:29
  So I felt like he had a lot to offer.
• 1:32
  This was not sponsored in any way.
• 1:34
  I'm going to come right out and say it.
• 1:36
  I was not held with a gun to my head to have him on.
• 1:40
  It was my choice. I really wanted to see what we
• 1:44
  could get into in this conversation and I am very glad
• 1:48
  that we did because he gave me some antidotes that I was
• 1:52
  needing to hear. I really appreciated the
• 1:54
  conversation. I would also love to hear what
• 1:57
  you all thought about it. I know there's been a few people
• 2:00
  that have reached out to me that have asked about bridging the
• 2:03
  barriers between the technical teams and the business teams and
• 2:07
  how to make sure that all stakeholders are happy when you
• 2:11
  are dealing with data, data, products, machine learning
• 2:13
  products. This feels like it is a
• 2:17
  conversation for all of those people that have reached out to
• 2:20
  me asking for that. If you just want to hear the
• 2:23
  nitty gritty on the tech and on what the newest, latest and
• 2:28
  greatest tools and methods, architectures, system designs
• 2:34
  are, then we've got a whole lot of other material for you.
• 2:37
  But if you're interested in getting some insights around how
• 2:43
  to build a more effective team when it comes to technical and
• 2:49
  business stakeholders, this is what we got into today.
• 2:53
  There were so many Nuggets in here, I can't even begin to list
• 2:57
  them off because it will take me at least 20 minutes and I think
• 3:00
  it's better that we just get into the conversation.
• 3:03
  I will drop this bomb on you. Peter G is hiring.
• 3:08
  The company is hiring for engineers, machine learning
• 3:12
  engineers like crazy right now. They're growing.
• 3:14
  They're doing so well and you may be using them, you may have
• 3:20
  heard of them. They are a coding solution.
• 3:23
  It seems like the coding space, the AI generated code space, is
• 3:28
  one that is blowing up right now just on this podcast.
• 3:32
  In the last eight months, we've had four people that are doing
• 3:38
  something in the code generation space, and none of those people
• 3:43
  worked at Copilot, the big name in this space.
• 3:47
  What I love about this space is that all of these different
• 3:52
  companies are having to differentiate themselves in one
• 3:55
  way or another. I know there's a lot of people
• 3:58
  out there that are using coding tools and coding assistance to
• 4:02
  help them get more done. So if you're that person and
• 4:07
  you've been using it, what I really would like to hear from
• 4:10
  you is how much of your code stays in the code base four
• 4:16
  weeks after you merge that PR. That's what I really want to
• 4:20
  know because we got into a bit of a survey that just came out
• 4:24
  recently today actually on how much of the code that is
• 4:29
  accepted from GitHub. Copilot ends U having to go and
• 4:34
  be completely rewritten four weeks after it is.
• 4:41
  Accept it O if you're using some copilot or some coding assistant
• 4:46
  let us know. I want to know is it worth the
• 4:49
  hassle? Does it really power up all of
• 4:52
  your abilities or is it something that you have to go
• 4:57
  back and refactor once or twice a week?
• 5:01
  Let's get into this conversation with Peter.
• 5:02
  Hope you all enjoy and as always, it would mean the world
• 5:06
  to me if you share this with just one friend, just one
• 5:10
  friend, let him know. MLA's community, we out here
• 5:15
  doing it. Let's get into this combo, to
• 5:25
  start man with the exact same way that I started the call with
• 5:29
  you a week ago. Tell me about that skateboard,
• 5:31
  because it is the centrepiece. I can't stop staring at it.
• 5:35
  Biologic so I I I was a I was a derelict as a teenager.
• 5:40
  I was a sponsored skateboarder as a teenager grew up in New
• 5:43
  York City and skateboarding was my was my outlet actually taught
• 5:47
  me creativity actually that community taught me DIY and
• 5:51
  entrepreneurialism and so it's always been my roots and and the
• 5:55
  board specifically behind me is it was it's a recreation of a
• 5:59
  very famous skater this and Mike Vallely out of out of the East
• 6:02
  Coast. It was one of the first sort of
• 6:05
  double ended boards. There was sort of this maybe
• 6:07
  that became the primary shape all skaters used, and so when he
• 6:10
  reissued it, is it a limited run, autographed and numbered by
• 6:13
  him. I had to do it.
• 6:16
  I had to do it. I I had the original of that
• 6:18
  deck. I used the hell out of it and
• 6:20
  ended up in a bin, I'm sure at some point.
• 6:22
  So it was great to to get the the recreation of it.
• 6:26
  Were you skating pools or parks or hand rails?
• 6:31
  What are you talking here? Yeah.
• 6:32
  I was a street skater. So I grew up in New York City.
• 6:35
  So I was a skater in the late 80s early 90s sort of at the the
• 6:38
  heyday of of St skating. It was streetscaping sort of was
• 6:41
  born. This is pretty X Games.
• 6:43
  This is like they did the first X Games just as I was getting
• 6:47
  out of the sport. So this is when skateboarding
• 6:50
  was not cool. It was it was it was sort of the
• 6:52
  dead zone in between the era and the and the X Games era.
• 6:56
  But a lot of folks, you know, like a lot of the kids I grew up
• 6:59
  with, you know, these are the founders now of Supreme.
• 7:01
  These are the guys who ended up running skateboarding at the big
• 7:04
  companies like Levis and Adidas and and others.
• 7:09
  So it was. It was a great honour to come up
• 7:10
  and it really was. So now you said it taught you
• 7:15
  entrepreneurship. You had and have had and are
• 7:19
  continuing to have a bit of a run when it comes to starting
• 7:22
  companies. Maybe let's just start there.
• 7:25
  Like when did you first get the bug?
• 7:27
  What was the first company break it down for us?
• 7:30
  Yeah, I I, I, I like to joke that it was born in all fair.
• 7:34
  Even when I was a skater, it was have a little side hustle.
• 7:36
  I did. I did a little clothing brand
• 7:38
  and a skateboard deck brand for a little bit that was just low
• 7:41
  production, sold in the local shops and in the tri-state area
• 7:44
  in New York. But when I was in school, so I
• 7:47
  was in university in the mid 90s and I was when I was in school
• 7:51
  was the birth of the web right? It was the start of all this
• 7:53
  stuff I had been doing. I've been paying more through
• 7:56
  school, working as a photojournalist and and was
• 7:59
  doing was doing content creation and other things and had an NEA
• 8:03
  grants for publication that I was doing.
• 8:05
  And someone came up to me in the computer lab in 94 and said,
• 8:09
  hey, you should put this on the Internet.
• 8:11
  You've reached such a bigger audience.
• 8:12
  And I was like, what the hell is the Internet, right?
• 8:15
  It was 1994, right? So by summer of 95 I was doing a
• 8:20
  bunch of web development and I got an opportunity to do some
• 8:23
  contract work. So I did a build the first on
• 8:25
  site for People magazine, did a bunch of work for some of the
• 8:28
  record labels or other media companies in New York, So 96.
• 8:32
  I ended up starting a digital agency.
• 8:33
  So two worlds simultaneously building, building websites,
• 8:38
  building web applications, doing some of the earliest, earliest
• 8:41
  eras of digital transformation, and then doing digital marketing
• 8:44
  with time. And that was a very new thing.
• 8:47
  I built and sold that company at a year and a half and then went
• 8:50
  and did it again. So sold that company, were
• 8:52
  holding company and then move to the West Coast, joined our group
• 8:56
  of guys who were who were trying to do the same thing and took
• 8:59
  that company for 20 to almost 300 and a few years and sold
• 9:03
  that to one of the big companies and wow that was it that that
• 9:05
  could sort of gained a bug. So I've been in, I've been in
• 9:09
  tech now for, you know, almost almost 30 years.
• 9:12
  So just just a little shy of 30 years.
• 9:16
  Always in digital transformation, always in
• 9:18
  transformative technology and loved it.
• 9:21
  You know, I've always loved it. So Fast forward to today.
• 9:24
  You know I was in services businesses for years you know
• 9:27
  and then switching to software companies.
• 9:29
  I've been in some companies that folks would know.
• 9:31
  I was early employee at Acquia behind the Drupal open source
• 9:35
  project that was a huge huge waiting for us.
• 9:38
  I was at Engine X and built built that company up from just
• 9:42
  a Russian open source project to being the powerhouse that they
• 9:46
  became. I've been in data companies, AI
• 9:49
  companies you know and and now I'm now I'm at an AI for
• 9:52
  software developer company and then I'm going to I'm an
• 9:54
  investor and advisor and a number of companies and mullahs
• 9:58
  and AI and in in other core parts of the data infrastructure
• 10:03
  which I think is really interesting right now.
• 10:06
  It always amazes me when I hear these stories about people back
• 10:11
  in the day building websites and how easily it was that these
• 10:17
  gigantic companies were just letting kids build their
• 10:20
  websites. Because it was almost like the
• 10:22
  Internet was so low stakes that you were like, yeah, all right,
• 10:26
  we kind of need something there. You could go.
• 10:28
  And I can only imagine what the conversations you had with
• 10:31
  People magazine were like. Hey, we should probably get you
• 10:34
  a website, right? You want that?
• 10:36
  And actually, there wasn't as low stakes as you might think.
• 10:39
  I remember when when when I first put people.com lies, I I
• 10:44
  convinced them in in 95. I convince them not just to put
• 10:47
  the regular publication out but to do like custom content, which
• 10:51
  was radical at the time and they're big thing at the time
• 10:54
  was like Sexiest man Alive, Sexiest woman Alive.
• 10:57
  There's always like great things and I and I I convinced them
• 11:00
  that we should put up like the extended galleries of these
• 11:02
  people because they have all these great photos that they
• 11:05
  took. So I put up extended galleries
• 11:07
  of these people, got not just what was in the publication,
• 11:09
  which was just wonder photo of each and we got over 1,000,000
• 11:13
  uniques in one week on on that on the property which is. 95.
• 11:19
  There were like, not that many people in the freaking Internet,
• 11:21
  and there's not that many people would actually like fast enough
• 11:24
  access to be looking at all of this stuff.
• 11:27
  So it was. It was now those low stakes as
• 11:29
  you would think it is, Although it is really funny.
• 11:31
  I mean, I was 19 years old when I did that, right?
• 11:34
  I think about it now like there's no way in hell, you
• 11:37
  know, a top 20 media brand would let some 19 year old get access
• 11:41
  to their, their, their platform nowadays.
• 11:44
  But at the time you nailed it. There were very few of us who
• 11:47
  understood it. There were very few of us who
• 11:48
  were doing it. I definitely was more mature
• 11:53
  than my years at the time, but a lot of it was just, I was in a
• 11:56
  space that was really interesting and exciting.
• 11:59
  And you know, I was one of a limited number of people who was
• 12:03
  and who cared about it. I'll tell you it's it's easy to
• 12:06
  look back now and it'll be your yourself and maybe many of your
• 12:09
  your your viewers or folks who aren't even they weren't even
• 12:13
  born when the browser started, right?
• 12:15
  But I remember in the mid 90s, the 90s having conversations
• 12:19
  with CEOs and saying this is the future, like this is not just a
• 12:23
  another channel. Like we are going to do
• 12:25
  everything through the Internet because why wouldn't we, right.
• 12:29
  And it was impossible for them to understand.
• 12:31
  And then you Fast forward just 10 years and you think about the
• 12:35
  whole world flipped on its ear, particularly when the iPhone
• 12:37
  came out because the iPhone was out of phone.
• 12:39
  The iPhone was the first truly personal computer, right?
• 12:41
  It was, it was the fulfilment of of Steve Jobs's vision, you
• 12:45
  know, 30 years earlier of putting a putting a computer in
• 12:48
  everybody's hands, right. It just happened to be literally
• 12:51
  be in your hand, right. And so, you know I remember
• 12:55
  going through those those errors and I feel like we're going
• 12:57
  through the same thing with AI right now.
• 12:58
  I've been at a I companies now for eight years and I remember
• 13:02
  having these conversations even four or five years ago.
• 13:03
  Like you don't understand, AI is going to change everything.
• 13:07
  Every app will change, every business will change, all of our
• 13:10
  processes will change and people will tell you, crazy, right?
• 13:13
  There's no light. And I think now we're finally
• 13:15
  starting to see it. I can only imagine what the
• 13:18
  hosting costs forthepeople.com extended galleries were, they've
• 13:24
  learned. I so I agree with everything
• 13:27
  else you're saying. So forgive me for continuing to
• 13:29
  harp on this in the 90s, but that is probably ridiculous.
• 13:35
  Oh yeah, no. They have their own data centre.
• 13:37
  They have their own data. It was it was pretty remarkable.
• 13:40
  I'll tell you just how you think about like DevOps and how far
• 13:43
  DevOps has come and even scripting like I used to
• 13:46
  actually have to be in the office at nine 10:00 at night to
• 13:52
  push the site live every week. Then move to California.
• 13:57
  Got involved in the same kind of thing.
• 14:00
  Built another company, sold it. You thought, hey, it worked
• 14:03
  once. Why not test my luck?
• 14:05
  See if it works twice. Built the company.
• 14:07
  Gigantic Company and up to 300 people.
• 14:10
  Impressive stuff. I know you mentioned having a
• 14:14
  few tips and tricks when it comes to selling companies and
• 14:18
  teaching. Almost like helping us learn
• 14:22
  from your experience, yeah, of selling companies, building
• 14:26
  selling companies, particularly selling companies is quite
• 14:29
  interesting I I can imagine. And especially in this
• 14:32
  environment, maybe there are companies out there that have
• 14:36
  raised too much or that are just ready to get out of the game.
• 14:41
  Yeah. What are things that you learned
• 14:43
  when you sold your various companies?
• 14:46
  Yeah, well, you know the the craziest thing about this is
• 14:48
  you're never go into a company. You expect them to sell it,
• 14:51
  right? That's never the thing, you
• 14:54
  know, the first, the first agency I sold, I knew the the
• 14:58
  company I sold to was inside of one of the big advertising
• 15:02
  holding companies. When they started to roll all
• 15:04
  these these companies together and they were friends, they knew
• 15:07
  me, they knew my work, they knew my team.
• 15:09
  I had a team of 30 people doing really interesting work, which
• 15:12
  is actually how I was able to sell it and not stick around
• 15:14
  because they were, you know, they knew what they were getting
• 15:16
  because they knew me really well and they knew my people.
• 15:18
  And so and the same thing with the with the second agency sale,
• 15:23
  the I was actually pitching services to a big Fortune 50
• 15:29
  brand into actually and it was in their offices and it was
• 15:31
  myself and my CEO and a couple of others.
• 15:34
  And you know we won the business, but in the room was a
• 15:39
  senior executive from their traditional ad agency who
• 15:42
  actually was responsible for M&A.
• 15:45
  And so you know she called us a week later and said I was really
• 15:49
  impressed with you guys, you're amazing.
• 15:52
  Would you be open to us acquiring a piece of the
• 15:55
  business we want, We want exposure to more people.
• 15:57
  We want to get you into other accounts.
• 15:58
  We want to, we want to help you scale and we think we can do
• 16:01
  Better Together. And so you know these things
• 16:04
  happen when you least expect them and it's software was
• 16:06
  different. You know Aqua was a sale of
• 16:09
  private equity and it was a good, it was a good sale, right.
• 16:12
  It was Vista and acquiring the company, the company had built a
• 16:15
  really strong independent business.
• 16:16
  It was cash flow positive but was you know was was missing
• 16:21
  that next level of scale that it wanted right.
• 16:23
  And and you know Visa came in and it was it was a great thing
• 16:27
  or Engine X was a similar story of the company had been scaling
• 16:29
  it was growing at a healthy rate but there were all these big
• 16:32
  install base companies that were were in controlling that
• 16:37
  category in the Fortune 500 and could walk you know Ajax into 50
• 16:41
  times the number of customers that they could do on their own.
• 16:45
  And so the best way to sell a company is to not try to sell a
• 16:48
  company right. The best way to sell a company
• 16:49
  is actually to go and build a healthy independent business
• 16:52
  that's differentiated in space and has customers who love them,
• 16:57
  right. And this is the secret of every
• 16:59
  business. The purpose of a business is not
• 17:01
  to create a product. The purpose of a business is not
• 17:03
  to create revenue. The purpose of a business is to
• 17:05
  create and keep customer relationships.
• 17:07
  And if you have customers who love you and you have deep
• 17:10
  relationships with them, you will generate profit, you will
• 17:13
  generate healthy revenue and you will have an attractive business
• 17:15
  that somebody will want to be part of.
• 17:17
  So I think that's the focus Now when you get to a level of
• 17:20
  duress where you need to sell well, which I think is happening
• 17:23
  right now, I think a lot of people raise money to try to
• 17:26
  build businesses. It was so easy to raise capital
• 17:29
  even if the idea wasn't good, right?
• 17:31
  Or it's so easy to raise capital even if you maybe you didn't
• 17:34
  have the skills to do it like and that's that's a fair thing
• 17:36
  to admit, right? It's an entrepreneur.
• 17:38
  I have my strengths of things I'm terrible at, right?
• 17:41
  And some people don't make because they can't get the right
• 17:43
  staff, maybe they don't actually get to breakthrough.
• 17:46
  That's a different point. Then my best advice to people,
• 17:49
  whether in that situation where they have to sell, is if you
• 17:52
  still love your baby, you still want it to thrive, then reach
• 17:56
  out directly to the companies that you think that your team
• 18:00
  and your product could thrive in, right.
• 18:02
  And I've seen that happen. I've had some some companies
• 18:05
  that I have advised that are invested in over the years that
• 18:08
  had to do those sales, had an AI company that did that, but
• 18:13
  they're they're thing was there. They're super happy with where
• 18:15
  they ended up because the realities are surrounded by
• 18:17
  people who get their vision, they're surrounded by people who
• 18:20
  really understand them and understand what they're trying
• 18:21
  to do. And they were able to keep
• 18:23
  chasing their dream inside of a large company instead of trying
• 18:27
  to do it on their own inside of a small a small software
• 18:31
  company. And can you speak a little bit
• 18:33
  more to people not having the skill sets to execute on their
• 18:39
  vision? You mentioned not being able to
• 18:42
  recruit the talent as being one. I've been through my fair share
• 18:47
  of startups and I feel like each time I encounter a new problem
• 18:53
  as to why. But I also am kind of back seat
• 18:55
  driving, right? Like I don't have this stress or
• 18:58
  the pressure of millions of dollars on my back.
• 19:01
  So it's easy for me to be like, well, this guy doesn't know what
• 19:05
  the fuck he's doing or this person is way out of line with
• 19:10
  how they're running this business.
• 19:11
  Because I I just can't know what I would do in that situation.
• 19:17
  And so I feel like you've seen seen it first hand and maybe
• 19:21
  there are things that you have felt as you've developed as an
• 19:27
  entrepreneur that you recognise now 20 years ago you did not
• 19:32
  have that skill set. Yeah, I did.
• 19:34
  I'll give you, I'll give you my sort of foundational theory of
• 19:38
  successful startups, right. If you're going to be a tech
• 19:42
  company, then there's there's two core functions that every
• 19:48
  every person in a software startup needs to fall into, and
• 19:52
  you need both equally. They both need equal weight and
• 19:55
  they're hackers and hustlers. And what do we mean by that?
• 19:58
  So it's technology people and go to market people, right?
• 20:03
  But more importantly, it's the right mindset of technology
• 20:06
  people and good market people. Hackers say yes, right?
• 20:10
  You need people in a technology role in startups that aren't
• 20:15
  gonna tell you all the reasons why something can't be done.
• 20:18
  They're going to sit there and hustle and figure out like, what
• 20:20
  is the way I do make it happen? What?
• 20:22
  How do I make that real? Because the act of creating a
• 20:24
  business is the act of generating something from
• 20:28
  nothing. And if it was an easy thing to
• 20:33
  generate, that a business would already exist, Somebody would
• 20:36
  already be doing right. And so you have to embrace this
• 20:38
  mindset of I'm going to make it happen no matter what.
• 20:42
  And then that sort of hacker mindset, not the, not the
• 20:45
  negative, you know, breaking systems hacker definition, but
• 20:49
  the like, I'm gonna, I'm gonna just DIY until it's real, right.
• 20:53
  That mindset, I'll tell you, I've worked, I've been able to
• 20:57
  work with some really really remarkable remarkable computer
• 21:00
  scientists of course in my career.
• 21:03
  Some of the top, top 1%. And they all showed that.
• 21:06
  They all showed that mindset of I can see the way forward.
• 21:10
  I believe in it and I will bring it to life, right.
• 21:13
  I will reach hurdles along the way.
• 21:15
  But I will come over those hurdles and I will and I will
• 21:17
  make it happen where I feel like software has gone wrong.
• 21:21
  You know, for the last. I mean, gosh, it's been at least
• 21:25
  1015 years now was the diminishing of the gold market
• 21:29
  side of the house, right? Because there's this whole thing
• 21:31
  of you and Mark and Drayson was famous for pushing technical
• 21:35
  founder CEOs. But if you come from a technical
• 21:39
  background, you don't know how to sell, you don't know how to
• 21:42
  market. In fact, I would always argue
• 21:45
  with some of my, some of my peers at some of the tech
• 21:48
  companies I've worked at as a software engineer, your world is
• 21:51
  math and physics, right? Your world is formulas that
• 21:54
  balance out, right? They come down to a simple thing
• 21:58
  but sales and marketing is psychology and sociology so the
• 22:03
  formulas don't balance out and in fact you you don't really
• 22:07
  know 100% for certain why anything goes right.
• 22:11
  You know, and example is always use of my technical founders is
• 22:14
  you know that higher education rates in a society reduce crime
• 22:19
  rates, right? OK.
• 22:21
  No one has been ever able to actually show you a direct
• 22:24
  causation in that no one has ever been able to actually trace
• 22:27
  that out, right. But it's just two data points.
• 22:30
  That's what it is. The marketing and sales is the
• 22:32
  same way. Why did it?
• 22:34
  If I spend money on awareness advertising, the business grows.
• 22:39
  Can I show a direct correlation? No.
• 22:41
  Until I can put a chip in someone's brain and keep track
• 22:44
  of everything they've seen and out, I'll never real for sure,
• 22:46
  right? But you have to have both sides
• 22:48
  of those coins. And the Hustler line set says
• 22:51
  I'm going to build a brand, I'm going to build a business, I'm
• 22:53
  gonna make money no matter what. And the irony of all of this is
• 22:57
  the most successful businesses and technology over the last 50
• 22:59
  years, usually we're not the best product, they were the best
• 23:03
  go to market, right. Very rarely have we seen those
• 23:07
  just products that one very rarely right.
• 23:11
  The ice on being another great example is like best product and
• 23:15
  best go to market combined, right.
• 23:17
  But I guarantee if you look back over the history of even
• 23:20
  products you've used like, oh, I love that thing and they didn't
• 23:23
  make it right, this other company ended up taking the
• 23:26
  category. 100% There's The graveyards are littered with
• 23:33
  incredible products and I find it funny too.
• 23:37
  Like just reminiscing on a lot of conversations that I've had
• 23:39
  with startup founders and how they're very much like, alright
• 23:45
  we're just gonna product, product, product and then once
• 23:48
  we have the product right then we'll turn on the go to market.
• 23:50
  Like, it's just a light switch that you can flip on and off and
• 23:54
  it's like you need to be kind of thinking about that at the same
• 23:57
  time. You know, it's not something
• 23:58
  that you can just turn on and and then all of a sudden the
• 24:03
  leads just come funnelling in. Yeah, so it is.
• 24:07
  It is interesting to think about that and on the other side, what
• 24:10
  you're talking about with the hacker mindset, especially in
• 24:14
  the startups. I have seen the and I think
• 24:17
  those people, when you encounter them in startups, it's a joy to
• 24:20
  work with them because a lot of times what you'll get are the
• 24:25
  other type that's like going to tell you all the reasons why
• 24:28
  we're resource constrained and you can't do that thing.
• 24:31
  That crazy idea that you came up with at 6:00 on a Thursday night
• 24:35
  when everybody was having dinner together, right.
• 24:37
  It's like, no, that's not possible.
• 24:39
  But then you find that person at the on the team that's like,
• 24:43
  ohh, yeah, like hold on, I'll see if I can throw something
• 24:47
  together. I think we could do it.
• 24:48
  Maybe like this. Yeah.
• 24:50
  Next thing you know, it's happening.
• 24:51
  And like you said, you're literally creating something
• 24:55
  from absolutely nothing. It was an idea on a Thursday
• 24:58
  while you were having some doll and Indian food, you know, like.
• 25:03
  And then Friday it was kind of in motion.
• 25:07
  And then next Tuesday it's actually a thing that you're
• 25:10
  talking about and you're touching and maybe you're
• 25:12
  demoing. So I.
• 25:14
  Ran. I wrote a bunch of different
• 25:16
  functions in my career. I've read of product for years.
• 25:19
  I'd companies and there's two types of product people hidden
• 25:24
  in companies. There are those who come bottoms
• 25:26
  up they're they're usually software engineers or engineers
• 25:29
  by degree and they and they think requirements and features
• 25:33
  and they sort of build products from the foundation up and then
• 25:36
  they'll go to market. Product people right there are
• 25:39
  the people who their mindset is there is white space in the
• 25:43
  market. There is a need in the market.
• 25:44
  I understand enough of how the technology works that I can see
• 25:48
  a path forward. I was always that guy now I was
• 25:51
  I broke COVID early in my career.
• 25:53
  I'm very technical. I probably would have been a
• 25:55
  software engineer, had a had a different upbringing, and knew
• 25:58
  that was an option, right? I probably would have done
• 26:00
  something else. Instead, I became Ostler instead
• 26:01
  of a hacker. But what's really interesting is
• 26:03
  I think there's a certain amount of joy in knowing enough about
• 26:07
  the technology to know that it is possible, but not so much
• 26:12
  that you stop yourself and say, well, but it's really hard or
• 26:16
  oh, but this other thing is in the way.
• 26:17
  There's a certain amount of value and ignorance in that.
• 26:20
  And you know, that is entrepreneurialism.
• 26:23
  Animalism in its core is just looking and saying I I'm, I'm
• 26:28
  gonna make it happen. Like, I believe in it enough.
• 26:31
  You have to have this sort of suspension of disbelief where
• 26:35
  you're not afraid of failure, you're not afraid of getting it
• 26:37
  wrong, you're not afraid of hitting a wall, You just say, I
• 26:40
  want it to be so I'm gonna do it.
• 26:43
  And what's really interesting is my earliest forays into machine
• 26:46
  learning and AI products came from that because I understood I
• 26:51
  was trying to accomplish this thing.
• 26:52
  But if I could get an algorithm to do it instead of a simple
• 26:55
  logic of tree in an application would be this much better,
• 26:58
  right? Knowing it was hard but being
• 27:00
  able to look and say but it's so much better.
• 27:03
  It's worth trying right It's absolutely worth trying and then
• 27:07
  just making it happen. And I'll tell you what I was at
• 27:09
  Aqua we created this product called Aqua Lift which was a
• 27:13
  personalization and targeting engine for media companies and
• 27:16
  it used machine learning. It was multi armed bandit
• 27:18
  algorithm to do it at 1st. And when I propose this product
• 27:23
  I meant nothing but resistance internally nothing but CEO shut
• 27:26
  it down said like that will see the value I don't get it
• 27:29
  couldn't see it. He also ended the engineers were
• 27:31
  like well it's really hard and and how would you use how would
• 27:34
  collect all that data to feed the algorithm and what we do
• 27:37
  with that. So there were a handful of us
• 27:39
  who really believed in it like really really believed in it.
• 27:42
  It was my one of my engineering managers myself one of the guy
• 27:45
  who ran my sales engineering team and we saw this called
• 27:49
  gardening days. So every three sprints we would
• 27:51
  have a a dead Sprint in between. So the 4th Sprint was you could
• 27:55
  do whatever you wanted as be a big open source company.
• 27:57
  Most most of us contributed back to open source.
• 27:59
  When that happened we used the 4th sprints to build the product
• 28:04
  but we just believed it 2 sprints and we had a functioning
• 28:08
  MVP fully functional and came to the company and said, I know you
• 28:13
  said no, but we believe it enough.
• 28:14
  We built it, but here it is, and it works right.
• 28:17
  And it became like two years later, it was 1/3 of that level
• 28:21
  of the company. And it's a great example of like
• 28:23
  if you really believe in it, then then Partridge, but put
• 28:27
  your ass on the line and Justice do it like just go and make it
• 28:30
  happen, right. And we get to Polish rough edges
• 28:33
  and you know there were some things we changed about it, but
• 28:35
  that product ended up, you know, evolving and continuing to grow
• 28:38
  and is now a core part of that company's business.
• 28:40
  Exactly. And you get to see how putting
• 28:44
  something in your production so quickly gives you that feedback
• 28:48
  that you were looking for. And now people can't talk about
• 28:51
  why they can't do it because it's this thing that's in the
• 28:56
  air and it's not scoped properly or it is something that is
• 29:00
  intangible, right? When you get something that's
• 29:02
  there, you throw down. It's tangible.
• 29:04
  Look, here we're getting feedback from it.
• 29:06
  It's a whole different conversation.
• 29:08
  Exactly. I could take it to your, like
• 29:10
  for us it was our biggest customers.
• 29:12
  I can go to the sort of saying, I have an idea everything, you
• 29:15
  know, NBC was in our biggest customers at the time.
• 29:17
  But yeah, customer there who was really great at giving the
• 29:20
  feedback and really was like a good partner.
• 29:22
  He wanted things to work so and he would pull no punches on his
• 29:26
  feedback at all. And so he was the first person
• 29:29
  we took it to As soon as it was functioning, we're like, oh, by
• 29:31
  the way, we mocked it up on your site.
• 29:32
  We're using your data results of this stuff.
• 29:34
  And he's like, this is amazing. These are the 10 things that I
• 29:37
  don't like. It's like and we just iterated
• 29:40
  it and it's you know, you can, you know ideas are ideas are
• 29:44
  sober, cold codes, gold like just get it in production and
• 29:48
  assume it'll change and you'd be amazed at what you what you can
• 29:52
  accomplish. Yeah.
• 29:55
  And that that iteration speed is what makes startups so dangerous
• 30:00
  because you can't do that type of thing.
• 30:04
  There's there's a million reasons why you can't do that
• 30:06
  when you're at a big enterprise, right?
• 30:08
  Absolutely. And all that resistance that you
• 30:10
  faced. Maybe when you're at a startup,
• 30:12
  you face the resistance from a few people on the team, but just
• 30:16
  the fact that there are not that many people at the company, you
• 30:19
  can't face that much resistance. Whereas if you're with at an
• 30:22
  enterprise, you're having to jump through so many hoops and
• 30:27
  it jumped through hoops of people's jobs.
• 30:29
  It's only their job to, like, kill your dreams.
• 30:32
  Yeah, it's true. It's true and it's politics
• 30:36
  cause like of its its politics. That's the problem, right,
• 30:40
  Because you see it in big companies.
• 30:42
  Yes, I I worked my biggest customer when I was an agency
• 30:45
  exec in the 2000s was Microsoft and this is the bomber era.
• 30:50
  This is the Kevin Turner era. This is when Satya was just
• 30:53
  running the online services group which owned Bing and a
• 30:56
  couple of things. And it was remarkable at how
• 30:59
  good they were at saying no to everything and telling you why
• 31:03
  everything was bad. And it was politics.
• 31:05
  It was literally just like, I don't want you to win because
• 31:07
  then it may diminish. I win and it almost killed that
• 31:10
  company. Like that attitude almost killed
• 31:12
  that company. And you, you know, you look at
• 31:15
  what happened. It required them actually having
• 31:17
  into your death experience and having Google overtake them in a
• 31:20
  bunch of categories and having Amazon overtake them and and and
• 31:24
  you know, software infrastructure and IT
• 31:26
  infrastructure before they finally woke up and said we we
• 31:28
  can't do the same anymore and all the politics got pushed
• 31:31
  aside, right. And but I've seen that even
• 31:33
  happen in small companies, right?
• 31:35
  It's very easy for for people to give feedback or to make
• 31:41
  decisions that are not actually about the best interest of the
• 31:44
  company and the best interest of what you're trying to achieve.
• 31:46
  And I think that's the most important thing you can achieve
• 31:48
  as a leader in one of these is don't let that stuff creep in.
• 31:52
  Right, real fast. I want to tell you about the
• 31:54
  sponsor of today's episode, AWS Trinium and Influencia.
• 31:57
  Are you stuck in the performance cost trade off when training and
• 32:01
  deploying your generative AI applications?
• 32:03
  Well, good news is you're not alone.
• 32:07
  Look no further than Aws's Tranium and Influencia, the
• 32:12
  optimal infrastructure for generative AI.
• 32:15
  AWS Tranium and Influencia provide high performance compute
• 32:21
  infrastructure for large scale training and inference with
• 32:25
  LLM's and diffusion models. And here's the kicker, you can
• 32:30
  save up to 50% on training costs and up to 40% on inference
• 32:35
  costs. That is 50 with A5 and A0.
• 32:39
  Whoo. That's kind of a big number.
• 32:42
  Get started today by using your existing code and frameworks
• 32:46
  such as Pytorch and Tensorflow. That is AWS Tranium and
• 32:50
  Influencia. Check it out, and now let's get
• 32:53
  back into the show. When when people start saying no
• 32:56
  to things, ask why? Why are they saying no to it?
• 33:00
  Is it actually because they are protecting the business or is it
• 33:02
  because of protecting themselves?
• 33:04
  Right. And that's something I've seen
• 33:06
  that happen in, you know, 50,000 person companies, but I've also
• 33:09
  seen it happen in 500 person companies.
• 33:11
  And so, you know, that will become a bigger hindrance to
• 33:14
  your success than anything else. So it was joke.
• 33:16
  Politics will kill the company faster than a bullet.
• 33:19
  That's so good. So there is another piece that I
• 33:21
  wanted to get into which was all around dealing with data teams
• 33:28
  from the business side of the house.
• 33:31
  Yeah. So there as you mentioned
• 33:33
  there's like there's the software side of the house and
• 33:35
  then you've got the go to market side of the house.
• 33:37
  And I know that you've gone through your career and you've
• 33:41
  been the stakeholder. That's like what we would say on
• 33:46
  your like the non-technical stakeholder, although you are
• 33:49
  technical in your own right. But in these cases, you're the
• 33:53
  person who needs things from the business side of the house, and
• 33:58
  you're working with machine learning and you're working with
• 34:02
  the different software teams. Yeah.
• 34:05
  And I want to hear from this side because I feel like in the
• 34:11
  205 podcasts that we've done, you're the first person that we
• 34:15
  talked to. And it makes me feel like a
• 34:17
  little bit ashamed that it took us this long to talk to somebody
• 34:22
  in your position because it is very valuable.
• 34:26
  And we hear people beating the drum about it all the time.
• 34:29
  How important it is to take care of your stakeholders.
• 34:31
  Your stakeholders are your customers at the end of the day,
• 34:33
  especially if you're an ML engineer, you've got to look
• 34:37
  after your data scientists. You've got to look after your so
• 34:41
  many different pieces like your analysts, your marketing team,
• 34:46
  whatever it may be, whoever's got their finger in the pie on
• 34:49
  wanting to see the results of what is happening with the data
• 34:55
  products that are going out that is you like you.
• 34:59
  You've seen the other side of it, and so now what can you tell
• 35:03
  us about? That, yeah, that's a great
• 35:05
  question. And and I feel like I've, I've
• 35:07
  lived on both sides of the house, you know, both in
• 35:10
  building the products for a business, you know, both in my
• 35:14
  agency days as well as in product development, then being
• 35:17
  on the other side as a consumer of these tools.
• 35:19
  Right. And as somebody who who has set
• 35:22
  the requirements and said, listen, this is what I'm trying
• 35:24
  to achieve. And I feel like the biggest
• 35:25
  mistake that gets made by most teams is they get too deep too
• 35:31
  fast and they start getting into a level of specificity around
• 35:37
  how something is going to be constructed without really
• 35:41
  understanding why something is being constructed.
• 35:44
  And you think about the data world, you think about
• 35:47
  operational analytics and machine learning and these
• 35:49
  applications of machine learning and AI, great.
• 35:53
  It's very easy to work bottoms up like I have an algorithm, I
• 35:57
  have a certain way of reporting on things or I have a certain
• 36:00
  data set or data structure and that starts to drive what it is.
• 36:04
  And I see this all the time, the marketing for example, because
• 36:07
  marketing is like just overwhelmed by tracking data
• 36:12
  like the amount of like things that touches we track and the
• 36:16
  interaction we track and the the user data we track and all these
• 36:19
  other things. It is a soup of just tonnes and
• 36:24
  tonnes of data. Data is not insight.
• 36:27
  Insight is not action, right? And what you're actually trying
• 36:30
  to get to is an action, right? I don't care about the raw data
• 36:36
  under all of that interaction. I'm trying to understand how I
• 36:38
  get more people to buy my product.
• 36:41
  I'm trying to understand how to how to get, you know, focus so I
• 36:46
  can dominate a category as opposed to just spread like
• 36:49
  peanut butter across a larger universe.
• 36:52
  And I feel like what happens is a failure on both ends.
• 36:56
  I feel like the people asking for these things don't ask the
• 36:59
  right questions, right? And the people building the
• 37:02
  systems don't actually understand what it is these
• 37:05
  people trying to accomplish. So my biggest advice to data
• 37:08
  engineering, data scientists, analysts, people building these
• 37:13
  things is ask the five, why's right?
• 37:16
  There's a served interrogation technique around asking why?
• 37:20
  And then when they get an explanation asking why that is
• 37:22
  true or why they feel that way. I think there's there's a
• 37:26
  there's a sort of lost art for business requirements.
• 37:29
  We used to put so much emphasis on it pre agile right, where
• 37:33
  we'd really unpack and understand and really document
• 37:36
  requirements not just at a surface level but why and how
• 37:39
  agile broke it. Agile is like so popcorn it's
• 37:41
  like I'm just going to move fast I gotta just go build stuff that
• 37:45
  you know it ends up sending us down rabbit holes sometimes
• 37:47
  there's sends us down forks in the road that are that are in
• 37:50
  the opposite direction of where we're actually trying to go.
• 37:52
  And I think, you know, data engineering and indeed a science
• 37:55
  teams can do a better job of just stopping and asking what is
• 38:00
  it you're trying to get to as an outcome?
• 38:03
  What are you trying to achieve with this?
• 38:06
  With an ideal state? How would the business operate
• 38:09
  more effectively if I were able to solve some of these things
• 38:13
  for you? Because what you start to find
• 38:15
  very quickly is they're coming with requirements and they're
• 38:18
  actually asking for the wrong things, right?
• 38:21
  Or they're asking for something and they're articulating it
• 38:23
  really poor right. The other side of it though, I
• 38:26
  push on the business teams all the time to also and do a better
• 38:31
  job of leading with those things, but then also ask to a
• 38:34
  better questions about what is possible and what's available,
• 38:37
  right. I I just went through this at a
• 38:40
  Tab 9 where we have a really great data team that tracks a
• 38:45
  tonne of really interesting data about usage.
• 38:49
  And you know how are people actually live in the product and
• 38:53
  you know what they're applying it for and where they're
• 38:55
  successful and where it's all anonymous, but it's, it's really
• 38:57
  robust And we used to tune the product.
• 38:59
  And I I just, I just sat down with them and instead of coming
• 39:04
  in directed and saying well I'm going to do some product LED
• 39:06
  growth initiatives and I'm looking for XY&ZI didn't start
• 39:09
  there. I started with what are you
• 39:12
  tracking, why did you start tracking that?
• 39:15
  What does that tell you how much data?
• 39:19
  Because by me understanding even those foundational elements, it
• 39:24
  starts to give me better questions to ask and better,
• 39:28
  better requirements to put back on them, right.
• 39:32
  And there's this sort of funny thing around data and insight
• 39:37
  when you're not a mathematician, when you're not a statistician,
• 39:41
  when you're not an engineer, where you don't know what you,
• 39:46
  what you want until you see it sometimes.
• 39:48
  So I encourage businesses very often to like, do things like
• 39:52
  look over the datasets they have and just play, where's Waldo?
• 39:55
  Like like what do I have here and what does this tell?
• 39:58
  That's really interesting And it starts to inform the business in
• 40:02
  a way. That is really striking.
• 40:04
  I wrote an Oracle 5-6 years ago when I was at a company called
• 40:08
  then called Mem Sequel. That single store which is
• 40:10
  operational analytics and ML AI data store and I wrote a piece
• 40:14
  for CEOs and it got published. I can't remember which business
• 40:18
  publication published it, but it basically was calling them out
• 40:21
  and saying data is the new oil. If and you as a CEO don't know
• 40:27
  what data you collect and how and why you're failing, like
• 40:31
  you're absolutely failing as a CEO because if we're in the 90s,
• 40:34
  we'd be talking about supply chain.
• 40:35
  If we're in the 2000s, when we talk about digital channels,
• 40:38
  both of those are required you to have a level of insight
• 40:41
  around what was happening in a portion of your business that
• 40:43
  was critical to its success. Today, that's data that's 100%
• 40:48
  data. If you don't know the data
• 40:49
  sources and what you have and where it resides and how it can
• 40:51
  be leveraged, you're going to make bad choices.
• 40:54
  And I say that for the CEO, I'll manage and if you were the head
• 40:57
  of marketing or the head of sales or you're somebody in the
• 41:00
  head of product and you have all this instrumentation around
• 41:03
  usage and your user, what do you do with that stuff?
• 41:06
  So I do think it's a two way St. Man, there is one.
• 41:11
  There's a lot to unpack there. Let's just start with that.
• 41:14
  I really appreciate all the insights.
• 41:16
  And Speaking of insights, I wrote this one down because I
• 41:19
  felt like it was a powerful phrase that you said.
• 41:22
  And it is data is not insight, insight is not action.
• 41:26
  And so it's like the the end goal if we're looking at it like
• 41:32
  what's the outcome we're looking for, what's the result we're
• 41:34
  looking for here, We want the action.
• 41:37
  The data is not it's like a means it's a stepping stone to
• 41:42
  get us to that action but we can't get caught up in that data
• 41:45
  and so they're. The other piece to this though
• 41:48
  is I almost have like a how do you juxtapose these two ideas
• 41:54
  that just earlier we were talking about how you were able
• 41:57
  to build a feature in two sprints?
• 42:00
  Get it out there. You've got people using it,
• 42:03
  iterating on it, but then you also just said that sometimes we
• 42:09
  need to slow down and really make sure we're asking the right
• 42:12
  questions and not go deep into the how.
• 42:15
  Yeah, yeah. You know, it's it's a tough one.
• 42:17
  It's the way I think about it is, is how much effort does it
• 42:21
  require for you to pull back the bow before the arrow fires,
• 42:25
  right. And you know, I think agile,
• 42:29
  agile encourages a ready fire aim mentality when even if
• 42:34
  you're moving fast, it needs to be ready aim fire right.
• 42:39
  Not because you know you should have a perfect answer.
• 42:42
  Don't, don't, don't mishear me. I'm not saying you need to have
• 42:45
  a perfect answer before you proceed.
• 42:46
  In fact, quite the opposite. Like it's OK to move forward
• 42:50
  when you only have, you know, a limited view of of what's ahead
• 42:53
  of you. But two things are critical.
• 42:56
  One is you don't want to waste energy on stuff that's not going
• 42:58
  to go anywhere, right? If you and actually cause
• 43:00
  there's opportunity cost or it's not just the cost of time, but
• 43:05
  now you're pulled down the rabbit hole and you're trying to
• 43:07
  unpack it. And then you end up in
• 43:08
  situations where people want to justify the work they did
• 43:11
  before, even though the evidence is saying it needs to be
• 43:13
  somewhere else because they invested so much time and effort
• 43:16
  in things. And that stuff is dangerous,
• 43:18
  right? You know, we talk in software
• 43:20
  all the time fast. You know, we should fail fast.
• 43:24
  I've never met a single company in Silicon Valley that actually
• 43:27
  was comfortable failing. So how he would have failed
• 43:29
  fast. It couldn't be OK Like, it's
• 43:31
  really uncomfortable to say you went down a bad path.
• 43:34
  And it's a dumb idea. It's really uncomfortable to say
• 43:36
  that. Just admit it.
• 43:37
  It's really uncomfortable to say it.
• 43:39
  Right. And so I think there's a lot of
• 43:41
  value in just asking the whys enough that everyone feels like
• 43:46
  they're on the same page and everyone feels like they
• 43:49
  understand the problem and maybe not perfectly right.
• 43:52
  You say, well, I understand the problem well enough to do my
• 43:55
  initial foray. I know it just well enough.
• 43:58
  And then what I'm going to do though is I'm going to assume
• 44:00
  that this is written in crayon, not in stone, and we're going to
• 44:05
  use it to continue to unpack the requirements and maybe we throw
• 44:08
  it out. And and that's something that
• 44:09
  engineering teams are really terrible at.
• 44:12
  Engineering teams get emotionally invested in their
• 44:14
  code. It's it's bits.
• 44:18
  Uh. Delete it like it's OK You can
• 44:20
  throw it out like if it got you to where you are.
• 44:23
  There's more than enough value in the information and education
• 44:28
  it gave you that you don't have to maintain the code.
• 44:32
  And that's something that is a product leader.
• 44:34
  I feel like it was probably my biggest success was early days
• 44:38
  and things saying no, no, no, this isn't where we're going to
• 44:41
  land long term. This is just to solve this one
• 44:43
  customer problem, to deny we're going to reject this thing from
• 44:46
  scratch right away. But we know we can get this out
• 44:49
  fast. We know we can solve close
• 44:51
  enough of the problem that we're not going to like, freeze
• 44:54
  ourselves in place and say, well no, I have to wait two years
• 44:57
  until the final product is done. No, I've got a near term answer,
• 45:01
  I've got a medium term answer and I've got a long term answer.
• 45:03
  And you know you've got you've gotta kill your loved ones.
• 45:09
  I hate to say it, but like, if you really want to be successful
• 45:12
  as an entrepreneur, you have to be willing to kill the things
• 45:14
  you love, right? If there's something better on
• 45:16
  the other side of it that a customer is going to be more
• 45:18
  successful with it and you're going to end up having a better
• 45:20
  result from it, then don't be afraid to throw away the things
• 45:23
  that you created. That's it.
• 45:25
  Don't be afraid to delete that code.
• 45:27
  It's all ChatGPT written. Anyway, it's so that's be
• 45:31
  honest. That's that's it's OK.
• 45:34
  Yeah, incredible. Now there is there is something
• 45:40
  else that I think is important to call out too when it comes to
• 45:44
  the interrogation of the data and almost like the
• 45:47
  interrogation of what you have. I've noticed that with myself,
• 45:52
  when we are doing things and I can sync with people on what we
• 45:58
  have, it gives me strokes of inspiration on what we could do.
• 46:04
  O It's like being able to just to understand, OK, there's this,
• 46:08
  there's this, there's this. And then it's like you're able
• 46:11
  to put two and two together and say, you know what?
• 46:13
  If I could know that, it would be very exciting.
• 46:18
  And I think about it a lot with like.
• 46:24
  I'll try and give a concrete example here.
• 46:26
  For example, if you're if you've got your cloud platform that
• 46:31
  you're trying to get people to use and you can get information
• 46:35
  on who is using the cloud platform, but you want to know
• 46:39
  what the journey is to activation, you want to see like
• 46:45
  hey, what's the biggest metric that shows us that if people do
• 46:50
  these things then we're able to say like you know, 8090% of the
• 46:55
  time they'll go on to becoming a paying customer.
• 46:58
  So we need to optimise for that person within the 1st 30 days to
• 47:02
  do these things. How can we just give them all
• 47:05
  the information that they need? How can we make big pop-ups on
• 47:08
  the product so that it points them to doing that?
• 47:11
  And all of that is huge. But you don't know what.
• 47:15
  Like if you're starting from scratch, you don't know what the
• 47:18
  journey is. You have to figure out, do we
• 47:20
  have that data? How can we find that data?
• 47:22
  How can we know what those metrics are that we should be
• 47:26
  tracking? All of that is like, sometimes I
• 47:29
  think when I look at it, I'll get overloaded because there's
• 47:34
  so many potential things. And then it's choosing those one
• 47:38
  things and saying, OK, we're going to make a bet that this is
• 47:41
  the metric we need to be looking at.
• 47:43
  So I guess there's not really a question in there, but I I feel
• 47:46
  like you may have some wisdom and you've done that before.
• 47:50
  And actually I think you you you you just threw out a really
• 47:53
  critical insight which is you don't know what you don't know
• 47:57
  right you. But if you approach it with the
• 47:59
  right mindset of I'm gonna start off with with a with a potential
• 48:05
  answer and I'm gonna be observant and really pay
• 48:08
  attention to what it's telling me, I will start to understand
• 48:11
  what I don't know that actually has value.
• 48:13
  And so our union example this so the most common modelling use
• 48:18
  case that I do in marketing. So we talk about marketing as
• 48:21
  the right message at the right time to the right audience.
• 48:23
  But the irony is it's inverted and value.
• 48:27
  You know we spend all our time thinking about create a very
• 48:29
  right message, but actually like 85 to 95% of the value is
• 48:33
  targeting the right. If you just target the right
• 48:36
  person, even with a mediocre message, you tend to get more
• 48:38
  sales, you tend to get more, more, more product growth.
• 48:42
  Right time is the next like 5 to 10% of that the the last few
• 48:47
  percentage points is the right message.
• 48:49
  So we always spent our time on the on the modelling side, on on
• 48:53
  ideal customer profile. How do I identify the right
• 48:56
  customer for me? Because you don't want 2% of the
• 48:58
  market, you want 100% of the customers that you're going to
• 49:02
  deal fit for. You want to dominate something.
• 49:04
  And so I've worked in a bunch of, you know, I built these
• 49:08
  models myself from scratch from those management consultant for
• 49:11
  for companies. I've used tools to do this.
• 49:13
  My my wife works at a company called Makutu that does this for
• 49:17
  sales and marketing professionals around like
• 49:19
  enterprise sales. And what we've always discovered
• 49:22
  is the same thing. So when you first build one of
• 49:24
  these predictive models like I'm trying to figure out who my
• 49:27
  ideal customer is and the sales and marketing asked is simple.
• 49:30
  I want to put 100% of my time at a customer base that is the
• 49:34
  highest likelihood to convert. I don't want to spend my time
• 49:37
  on, you know, 100 people to convert 5.
• 49:40
  I want to spend my time on the five.
• 49:42
  That's the ideal state, but it's a spectrum, right?
• 49:45
  So when you first build these models, the best thing to do is
• 49:49
  collect as many characteristics about the customers you have
• 49:53
  been selling to as is humanly possible.
• 49:55
  Because you'll know what is what's going to generate lift.
• 49:58
  You don't know which ones are actually the indicators.
• 50:00
  And I remember when I first started doing this price
• 50:02
  software, I would just collect everything firmographics, hiring
• 50:06
  data, technographic profile, you know anything about their,
• 50:10
  their, their product set, the verticals they sell into, any
• 50:14
  piece of data I could collect in.
• 50:15
  A pattern started emerging in all of this has been consistent
• 50:19
  for all the software companies that worked at in emerging
• 50:21
  technology that the technographics of the things
• 50:24
  that always are the most important.
• 50:27
  So literally what else do they have in their stack, right?
• 50:29
  That was always the most important.
• 50:30
  So give me an example. That's a cockroach we landed on.
• 50:34
  You know, Copper DB is a high performance, high scale,
• 50:37
  unkillable system of record, right?
• 50:40
  So it's a system of record database, very traditional OLTP,
• 50:44
  the dead giveaway of a customer that was an ideal fit for us,
• 50:48
  had Oracle and Mongo, they measured data centre and they
• 50:54
  ran Cougar days. Those four things would feel
• 50:57
  like their opposites to each other, right?
• 51:00
  But they actually worked because this was somebody who had high
• 51:03
  scale, which is why they had their own data centre.
• 51:05
  They had legacy applications which they didn't want to mess
• 51:08
  up, but they were really forward thinking they were modernising.
• 51:12
  They were, they were. Their business kept evolving,
• 51:13
  they kept changing, which is why you have Oracle and Mongo,
• 51:16
  right? Because you had, you know,
• 51:17
  unkillable system of record in first June and the sort of
• 51:21
  unkillable but not agreed system of record that you try to use at
• 51:24
  the other end. And So what was interesting
• 51:26
  about that is when I first started doing these things, when
• 51:29
  I first started seeing technographic data popping up as
• 51:33
  high left, what you realise very quickly is how much you don't
• 51:36
  know. Like the very first time I built
• 51:38
  it, we didn't track who was running Kubernetes.
• 51:41
  We didn't track of what cloud providers they were on, We were
• 51:43
  tracking whether they were hiring for data centre roles.
• 51:46
  Like we didn't have any of these things.
• 51:48
  But once a couple of them started popping, we were like,
• 51:50
  wow, that's really interesting. If that's generating lift, I
• 51:53
  wonder if dot, dot, dot and the wonder if then gave us
• 51:59
  requirements. So people, let's go get this
• 52:01
  data. Let's go see if it helps, right.
• 52:03
  And then some of it helped and some of it didn't and it was
• 52:05
  great. And I think that that's true.
• 52:08
  Timing is true for that. Messaging is true for that.
• 52:10
  I think that's actually true for even other things outside of
• 52:13
  marketing, like when you think about the AI stuff.
• 52:16
  We're building a tab and so we use rag and context.
• 52:19
  So when you're writing code, you're working in chat and
• 52:21
  you're asking for a recommendation, I need a
• 52:23
  function that does this. We started actually giving
• 52:26
  context awareness to things like documentation to things like you
• 52:31
  know, your requirements, docs, not just code like we started
• 52:35
  giving it exposure. Other things when we discovered
• 52:37
  was interesting. Context awareness around things
• 52:40
  like your Geo instance is actually more useful than
• 52:43
  context awareness around the code, right?
• 52:46
  So you don't know what you don't know until you start experiment
• 52:50
  and then you have to be unafraid to say, OK, would add data.
• 52:53
  I'm gonna add these things, I'm gonna try it and I'm going to
• 52:56
  keep tuning. But the one thing that never
• 52:57
  changes is what are you trying to achieve?
• 52:59
  That ideal customer profile example I gave you When I roll
• 53:02
  it out with nginx, we were getting a shocking number of
• 53:06
  inbound leads, shocking like Ajax as of now, I think 70 or
• 53:11
  80% of the runs off. So we had this massive pool, We
• 53:14
  just have to keep trimming. And my CEO came to me one day,
• 53:16
  we were getting it focused down. He goes, I want to cut the
• 53:20
  number of leads that the sales team touches in half without
• 53:23
  changing the outcome. That was his goal to me and we
• 53:26
  did it in two seconds and we kept getting smaller and smaller
• 53:29
  with their conversion rates just kept doubling every time we make
• 53:32
  changes. Wow.
• 53:33
  OK, it's on. Yeah, you're just following
• 53:35
  bread crumbs in the beginning, trying to see what are the
• 53:39
  biggest drivers and what is going to give us the idea of who
• 53:45
  our ICP is And then once you can zone in on one, it's like huh, I
• 53:51
  see this, whatever it may be. Kubernetes users and Kubernetes
• 53:56
  users that are running their own data centre, that means they
• 53:58
  have scale, that means that they're doing that and then ohh
• 54:02
  add a little sprinkle of Oracle on there with some Mongo DB.
• 54:05
  We're golden. We have found our fit but you
• 54:08
  didn't get that on the first try.
• 54:10
  I really like that you're honest about saying it took you a while
• 54:13
  to dig through all of this data and maybe you had one little
• 54:17
  nugget of information and you probably found OK, people that
• 54:21
  are using Kubernetes like us. And I think for me as I was
• 54:25
  listening to that, I probably would have stopped there and
• 54:28
  been like, alright, let's go get all the Kubernetes users, but
• 54:31
  you're saying no slice down segment down as much as possible
• 54:36
  so that you can find that 100% of the market that you can
• 54:41
  dominate. Yeah, you know, so if there's
• 54:43
  another famous example of this, which was Netflix's
• 54:45
  recommendation engine, and that's something that everybody
• 54:47
  I think can really relate to, you know, Netflix's
• 54:50
  recommendation is really good. It's really really good.
• 54:53
  When it first started it was 100% based on likes, so it was
• 54:57
  like did you thumbs up or thumbs down right?
• 55:00
  And what they discovered was and I'd have to find the the report,
• 55:03
  but I'm sure your your viewers can search for it.
• 55:06
  They discovered actually that whether or not you played
• 55:08
  something all the way through and if you played it multiple
• 55:11
  times was actually more valuable data than what you explicitly
• 55:14
  said. So they discovered your
• 55:16
  behaviour was so much more useful than your stated view of
• 55:21
  something and and that's another one, that's all.
• 55:24
  Yeah, actually I was thinking about that when you were saying
• 55:29
  it, it was like features, right? Like for recommendation systems,
• 55:32
  this is exactly what your you can use that as a parallel here.
• 55:38
  What you're talking about for in this marketing sense of
• 55:40
  segmenting down and being able to slice down for the ICP.
• 55:44
  It's the same thing for like features if you're trying to
• 55:48
  build a recommendation engine. Yeah, yeah.
• 55:51
  So there is another place that I wanted to take us.
• 55:56
  It's a little bit of a change of gears, but bear with me.
• 55:59
  Because you're building you, you kind of drop some hints, but
• 56:04
  you're building a coding assistant that has access to so
• 56:09
  much more than just your code. And I know that I can only
• 56:14
  imagine how happy people are that an LM can read their epics
• 56:20
  and their stories and it can give them context.
• 56:23
  And maybe one day you can have some agents that will go and
• 56:26
  just do the epics and stories for you.
• 56:29
  But I I don't know. Are you there yet?
• 56:31
  We we're there actually we're not we're not the least but yes
• 56:34
  we have that we have, we have a lot of that stuff already built
• 56:36
  and we'll be rolling out over the over the coming months.
• 56:39
  So I'll back up and and and tell a little bit about the why this
• 56:44
  company and what they were going after and then why Julie, I've
• 56:47
  only been in the company now a few months.
• 56:48
  So Tam Dine is the originator of the AI coding assistant
• 56:53
  category. So the company actually released
• 56:55
  its first product five years ago now and it was basically in
• 56:59
  parallel with the growth of WLMS, right.
• 57:02
  So as olms started to really demonstrate their power, the
• 57:07
  company founders had been focusing on developer
• 57:10
  productivity and automating as much of the developer software
• 57:13
  development life cycle as possible through other sort of
• 57:16
  data intensive means. But Elements had the most
• 57:19
  promise and you know it's it's. Be assured if you think about
• 57:23
  large language model, what does it need?
• 57:24
  It needs you know, structure, the language.
• 57:27
  It needs repeatability and the language and needs clear rules.
• 57:30
  Like the English language is a mess, which is why it's taken so
• 57:33
  long for generative AI to work in the English language.
• 57:35
  But programming languages are very varied Rule strip, it works
• 57:40
  really well. So, so the company built
• 57:42
  originally an LLM to do code completions in the IDE.
• 57:46
  So as in type basically we'll type ahead of you.
• 57:50
  Really powerful, actually. Really, really powerful.
• 57:53
  Because when you're trying to write a phone shit and you're
• 57:56
  trying to try to add data in fields and everything, it knows
• 57:59
  where you're going based on the context of what you have open
• 58:02
  and where it's going though. And the reason why I joined the
• 58:07
  company as a I, my my belief is I didn't change every job right.
• 58:10
  Every single thing we do today, it will look a little bit
• 58:13
  different. Five years from today, it's the
• 58:14
  same thing. We went through a digital
• 58:15
  transformation. Some all scary.
• 58:17
  I think some of it is feels a little scary but my view of it
• 58:20
  is it's not gonna be it's not gonna be a job replacement.
• 58:24
  I I think about a lot of these things as like an Iron Man suit
• 58:27
  for the mind. Like you're really about
• 58:29
  accelerating you and accelerating your skills and
• 58:32
  your capabilities and automating the way the mundane that's the
• 58:34
  truth, right? That's we're actually seeing
• 58:37
  these AI tools for software development been around like I
• 58:39
  said three to five years now for the majority of the category.
• 58:42
  So we've seen this pattern and what I really like to go with
• 58:46
  tab million was do is they were looking across the entire SDLC.
• 58:49
  I'd say if I had an AI Egypt that land in the middle of
• 58:53
  everything that lived in the IDE with you, but also lived in Jira
• 58:57
  and Confluence and also lived in Data Dog and lived in your other
• 59:02
  APM tools and then could read your logs and could do all those
• 59:05
  other things. You know, how smart would that
• 59:07
  AI agent be. So when you went to write a test
• 59:10
  you know instead of saying we're going to start writing a test
• 59:12
  would say, you know we're write a comment that says I need a
• 59:15
  test that does this thing. And, and and I wanted to be able
• 59:18
  to assess for these factors and I want to be able to know it's
• 59:21
  succeeded fell based on these outcomes.
• 59:23
  It can actually do it because it has all the systems available to
• 59:25
  it, right? It's not just I'm going to write
• 59:27
  a dumb test. I actually know, oh, you mean
• 59:30
  you want to actually track the outcome from Datadog.
• 59:32
  So we'll pull the data that comes back.
• 59:33
  Right. And tell you what actually
• 59:35
  happened. Now some of the stuff is still
• 59:36
  in progress, but that's the vision.
• 59:38
  The vision is you should have an AI agent who knows how to do
• 59:42
  testing with you. You should have an AI agent that
• 59:45
  knows how to write good requirements with you.
• 59:47
  You should have an AI agent that does things like deployments and
• 59:50
  actually, you know, upsets alongside you as a DevOps
• 59:53
  professional is pushing out and tell you if it's successfully
• 59:56
  deployed based on real world data, right?
• 59:59
  And all of those things would. It works really well, It's your
• 1:00:04
  patterns of behaviour, it's the things you would normally do.
• 1:00:07
  But instead of you having to go to five different systems to do
• 1:00:10
  it, you've got this partner that works with you in actually doing
• 1:00:14
  those things. And what the vision for the
• 1:00:17
  company that we have is in order for that to happen, in order for
• 1:00:20
  that to be done, well, there were a few requirements.
• 1:00:22
  The first one is. We believe privacy matters,
• 1:00:26
  right? So unlike a lot of the public
• 1:00:29
  tools like Copilot and others, we don't, we don't collect it in
• 1:00:33
  your data. We will not share it.
• 1:00:35
  We will not change the model based on your data.
• 1:00:37
  So we are completely private. You want to deploy, completely
• 1:00:40
  air gapped, we do that. You want to go and deploy in a
• 1:00:43
  private SAS, we do zero data retention.
• 1:00:45
  So we're really focusing on privacy because that's the only
• 1:00:48
  way we earn the trust to talk to all of those systems, right, is
• 1:00:52
  if we are fully private, Fully private.
• 1:00:55
  We also really focus on copyright.
• 1:00:57
  So our models, unlike the public models have not been trained on
• 1:01:02
  any non licensable data. So a lot of these larger chat,
• 1:01:07
  JPT and others, you know, there's, I mean there's all this
• 1:01:10
  discussion or copyright, they're trained on stuff they don't have
• 1:01:12
  the right to and you can debate whether or not do they have
• 1:01:15
  rights, do they have not. A lot of that stuff will get
• 1:01:17
  settled at a court of law. But from our perspective, it was
• 1:01:20
  not even a law issue as an ethical issue, right.
• 1:01:23
  If you said that your code is GPL and anything that gets
• 1:01:28
  generated needs to stay GPL, we're not going to train on GPL,
• 1:01:32
  right? We just all do because there are
• 1:01:33
  other ways of getting to that answer right?
• 1:01:35
  And we see it. We see maybe a slightly lower
• 1:01:38
  performance for us versus others.
• 1:01:40
  But we actually have switchable models, so our platform can run
• 1:01:43
  across multiple models. We've tested it.
• 1:01:45
  So if I go to a CIO and say, well, I can give you 7 out of 10
• 1:01:49
  answers perfectly with tab 9. Or I can, using the Open AI
• 1:01:54
  models, get you 8 1/2 out of 10 perfectly, but one of them is
• 1:01:59
  legally safe and the other one is legally questionable.
• 1:02:01
  Which one do you want? Right They only always set A7
• 1:02:05
  and A10 cause the additional benefits aren't worth the risk.
• 1:02:07
  Right now I'm hoping a lot of this stuff gets resolved.
• 1:02:10
  Our view of personalization is we can get to that 8-9 out of
• 1:02:13
  1010 out of 10 by trading on our customers data, by training on
• 1:02:19
  other things. We can licence which we own
• 1:02:20
  licencing from third parties. But we think it's a mindset have
• 1:02:25
  with this stuff because AI is going to penetrate every
• 1:02:28
  function in every company and you have to do it in a way that
• 1:02:31
  really respects the requirements of the total business not just
• 1:02:36
  the the task at hand and and what that is and we believe
• 1:02:40
  that's that's really critical. One thing I find fascinating
• 1:02:43
  about the ability to ingest from all these different places like
• 1:02:50
  you were talking about. Maybe it's your data dog
• 1:02:53
  instance, maybe it's your CICD pipeline.
• 1:02:57
  You've got visibility into that. Maybe it's the JIRA, all of that
• 1:03:01
  I and a little bit of a sidebar. I've 100% understand why privacy
• 1:03:06
  is so important. Like I don't want you just
• 1:03:08
  sending out data to chat to BT about my road map and all of a
• 1:03:11
  sudden GTA5 now has like my nine months ahead road map.
• 1:03:17
  Yeah, well I guess by the time GC 5 comes out, it'll be
• 1:03:20
  obsolete by then. But I think you you more than
• 1:03:24
  anybody understand why that is. That is not cool.
• 1:03:28
  But coming back to the point is, one thing I I find fascinating
• 1:03:31
  about this is that when it comes to code, evaluating if it runs
• 1:03:36
  or if it doesn't run is so much easier than evaluating, like
• 1:03:42
  marketing. Totally copy that.
• 1:03:44
  Is that good or is that not? Is that going to get somebody to
• 1:03:48
  buy better than what I could have created?
• 1:03:50
  I don't really know, but I guess it gave me a bunch of cool ideas
• 1:03:53
  and I can riff off with those ideas.
• 1:03:55
  Yeah. And when it comes to code, you
• 1:03:58
  know it's very black and white. Yeah, yeah, very much so, very
• 1:04:03
  much so. And and even even where it's not
• 1:04:06
  black and white, like I had this conversation with a with a
• 1:04:09
  partner, another big tech company that we're we're
• 1:04:11
  starting to work with. And they said, well there might
• 1:04:13
  be 3 answers that work, but one of them is preferred, one of
• 1:04:17
  them is better. That's where context actually
• 1:04:20
  really helps. Like, you know, we can actually
• 1:04:22
  set some guardrails and say when you are at, when a developer
• 1:04:25
  asks a question about X, the answer is always Y, right?
• 1:04:29
  Because you know, for example, like I need a function that that
• 1:04:33
  accomplishes this and, you know, does it in this way, right?
• 1:04:37
  Oh, we always use this API for that.
• 1:04:40
  We always use this system for that and we always secure it in
• 1:04:42
  this way as a company. So you can actually set those
• 1:04:44
  parameters. That's stuff that we're rolling
• 1:04:46
  out later this year. We call it expert coaching or
• 1:04:49
  expert guidance where you can set some parameters for those
• 1:04:52
  things and that is possible, right.
• 1:04:54
  And now that's more of the grey area, right?
• 1:04:57
  But this is where the machines with the right context to get
• 1:05:00
  smarter and smarter. And this is actually something
• 1:05:03
  that users don't understand. I can't tell you how many times
• 1:05:06
  I see users complaining about how the system is broken and
• 1:05:10
  this is triggered what we get in for feedback in tab 9.
• 1:05:12
  We see it with our competitors, but I see it with people just
• 1:05:15
  prompting ChatGPT and getting bad answers in which you when
• 1:05:18
  you when you look at it, you say OK, the issue wasn't the system,
• 1:05:21
  you didn't give it enough context.
• 1:05:24
  It's like going up to a stranger on the street and saying help me
• 1:05:27
  find a car. Actually, wait a minute, like,
• 1:05:31
  who are you? What do you need it for?
• 1:05:33
  Who are you carrying? How often are you driving?
• 1:05:35
  Do you care about performance? Do you care about fuel economy?
• 1:05:38
  Do you care about size? Are you agreeing?
• 1:05:40
  Are you a muscle car guide? Like there's a million questions
• 1:05:43
  you have to answer in order to do that.
• 1:05:44
  Or you feel like our learning around how to use AI is still
• 1:05:48
  stuck in this era of we accept we expect an omniscient bot
• 1:05:52
  that's going to give us a perfect answer with minimal
• 1:05:54
  input. They're not there yet.
• 1:05:56
  So there's we're going to close the gap two ways, right?
• 1:05:58
  We're going to close it by being better consumers of these tools
• 1:06:02
  and giving them more to work with.
• 1:06:04
  But actually, I think this is where companies are really going
• 1:06:07
  to win over the LLM providers is we can shape that, Sure.
• 1:06:12
  So for example, when someone starts asking for a function, we
• 1:06:15
  look at the workspace and we look at what everything else to
• 1:06:18
  do and say, oh, you're writing that function, You always write
• 1:06:19
  it this way. And that's the right answer.
• 1:06:22
  So really, we're going to give you that.
• 1:06:24
  So it's instead of having like a stranger on the street, this is
• 1:06:27
  somebody who knows you, who's worked at your company, who's
• 1:06:29
  been part of your team for a while.
• 1:06:31
  So they have a different context, right.
• 1:06:34
  So I I think that's what's really exciting about a cause
• 1:06:36
  you there are three components, all these things, there's the
• 1:06:38
  models themselves as the data they train about and then
• 1:06:40
  there's a user experience. And the user experience includes
• 1:06:43
  prompt engineering, the shaping of all of this stuff, what goes
• 1:06:46
  in, what comes out, all those other things.
• 1:06:49
  I think the models are going to get democratised very quickly.
• 1:06:51
  I my my belief, and I've seen this in every other part of
• 1:06:54
  technology in the last 30 years, I think the models are all going
• 1:06:56
  to be equivalent within five years.
• 1:06:57
  There's going to be no differentiation because they're
• 1:06:59
  all open source white papers. We're all building the same
• 1:07:02
  stuff. The datasets, I think, are going
• 1:07:04
  to become more cheering. Yeah, we're gonna all licence
• 1:07:07
  the appropriate stuff. We'll have access to the right
• 1:07:09
  things. People will get better about
• 1:07:10
  protecting their own data and using their own data to train
• 1:07:13
  their models. O won't be everyone gets access
• 1:07:16
  to everything. I think really the space for
• 1:07:18
  entrepreneurs and for people building these products to wit,
• 1:07:21
  is still the UX. It's going to be in crafting the
• 1:07:24
  right experience so that you bring to it what is appropriate
• 1:07:28
  for you as a user to bring to it.
• 1:07:29
  We don't make you jump through hoops, right?
• 1:07:32
  And instead we provide that additional context and and we
• 1:07:36
  gather all that stuff and the I things are great for this.
• 1:07:38
  Like I'll ask you questions, you ask me a question, I asked you a
• 1:07:41
  couple of questions before I give you an answer, right.
• 1:07:44
  That's going to be really exciting because that's where
• 1:07:46
  the things will really start coming together.
• 1:07:47
  So it's so funny you mentioned that because I was going to ask
• 1:07:50
  you, are you baking the philosophy that we were talking
• 1:07:53
  about earlier on making sure you're asking those five why's
• 1:07:56
  into how you're dealing with the I I yeah, I figured as much.
• 1:08:05
  And earlier today I was reading the blog post about the way that
• 1:08:10
  you evaluate coding AI. And I was remembering a
• 1:08:15
  conversation that I had probably 1/2 year ago, maybe eight months
• 1:08:20
  ago now on how one of the strong evaluation metrics that people
• 1:08:25
  will use, especially when it comes to coding but just in
• 1:08:29
  general, is if you accept the output.
• 1:08:33
  And this article that I was reading today was talking about
• 1:08:37
  how when they did a study and I have to figure out what it is
• 1:08:42
  and cite the paper and stuff. But there was a study done on
• 1:08:46
  the amount of code that stayed in the code base.
• 1:08:50
  Yeah, from suggested chat to be three, yeah two weeks or four
• 1:08:57
  weeks after it was accepted. And what they've been seeing is
• 1:09:01
  there is a lot of tech debt that kids formed by just accepting
• 1:09:07
  copilots suggestions and then later on you have to go back and
• 1:09:11
  realise ohh man. That was a.
• 1:09:13
  Bad wrote this code. How did anybody let this get
• 1:09:16
  through, you know? So it I know exactly the paper
• 1:09:19
  you're talking about because I just posted it myself on social
• 1:09:22
  media. It was a great piece.
• 1:09:23
  It was a great piece of research.
• 1:09:24
  Our code. Call that is.
• 1:09:26
  And my response to it, and I'll share with you exactly what I
• 1:09:30
  said when I posted it, which was it reminds you that these are
• 1:09:34
  not a replacement for a developer.
• 1:09:37
  Once again, I use like they're they're environment and sick for
• 1:09:40
  the mind. If you are a terrible developer,
• 1:09:43
  you're going to insert terrible code whether you wrote it
• 1:09:46
  yourself or whether an assistant coached you on it, right?
• 1:09:49
  But you still have to actually make a good.
• 1:09:51
  Choice, yeah, whether it's from Stack Overflow.
• 1:09:53
  Yeah, I mean this happens all the time by the way.
• 1:09:55
  Like I can't tell you how many people copy pasta out of Stack
• 1:09:58
  Overflow when injected. So this isn't new.
• 1:10:00
  Like this isn't the AI bots are just making it faster to do it.
• 1:10:03
  That's all they're really doing. And so there's a couple of
• 1:10:06
  answers to this. First and foremost is as
• 1:10:08
  engineering managers you have to be diligent who is on your team
• 1:10:11
  and what are they accepting and what are they putting in.
• 1:10:13
  It doesn't replace code review, right?
• 1:10:16
  And the anti bot can help you with code review I suppose.
• 1:10:19
  Specifically, you have parameters.
• 1:10:20
  You can it. But if you have a bunch of very
• 1:10:23
  very poor developers, all you're doing by getting them these
• 1:10:26
  tools is making them generate polar code faster.
• 1:10:29
  Your answer here is to get rid of the port developers or train
• 1:10:31
  right? It's one or the other.
• 1:10:33
  You either need to make them more effective or need to get
• 1:10:35
  them out. The AI coding assistance by the
• 1:10:38
  way, do on average save 20% total developer productivity,
• 1:10:44
  20% total across everything. When you talk about writing
• 1:10:48
  code, writing tests, writing comments, evaluating code,
• 1:10:52
  explaining code, all these other things that all the studies,
• 1:10:55
  we've got a white paper on this and we created a Tab 9 where I
• 1:10:59
  didn't do any research myself. I just went and said what did
• 1:11:01
  McKinsey see? What did IBM see?
• 1:11:03
  What did Cambridge see? There's all these studies that
• 1:11:06
  have been done and what you see very interesting is the data.
• 1:11:09
  All is very similar and very consistent and it's about 20%
• 1:11:13
  owning up. If you have 20% productivity
• 1:11:15
  savings on your mid level software developers, that gives
• 1:11:19
  you more time to coach the junior folks or it means you can
• 1:11:22
  fire 20% of your junior staff, right?
• 1:11:25
  One or the other and you know get back to quality.
• 1:11:28
  Now the interesting thing is I do think AI has an answer to
• 1:11:31
  this above that which is you when you when the chat
• 1:11:36
  suggestions come out, like when we actually suggest something,
• 1:11:39
  we do track whether it was accepted because that helps us
• 1:11:42
  shape it. You can overweight who who is
• 1:11:46
  valued in the response. So if you know that this is a
• 1:11:49
  senior developer, this is a coach, then there we is given a
• 1:11:54
  much higher priority over others, right?
• 1:11:57
  And this is where that Coche and context I think comes in really
• 1:12:00
  helpful because we can come in and say actually this is what
• 1:12:04
  good code looks like, right? So when you're recommending a
• 1:12:08
  function, if you have three options, you know generally from
• 1:12:11
  what's available in in the wild versus this is what an expert
• 1:12:15
  developer on the team built bias towards that.
• 1:12:19
  And that's where things started. This is the UX stuff I'm talking
• 1:12:21
  about. This is where things start to
• 1:12:23
  get really, really interesting because this is not a difference
• 1:12:25
  in the model. This is the difference in what
• 1:12:27
  we do with the prompt, the context and then what we
• 1:12:31
  recommend back. And you can get very specific if
• 1:12:34
  you really want to in these things.
• 1:12:36
  So then makes complete sense that you would have these senior
• 1:12:41
  developers be weighted much higher and it makes me think
• 1:12:47
  about how, yeah, like, of course you would.
• 1:12:50
  You want that to be the golden evaluations that you want to be
• 1:12:55
  taking that back to the lab and training with what they're
• 1:12:59
  accepting and what they're saying is, OK, so.
• 1:13:03
  Yeah. Look, I'll leave you with one
• 1:13:04
  thing on on these AI tools because I don't think it's just
• 1:13:07
  for us. I think it's going to be true in
• 1:13:08
  a lot of these other categories. I think there was this, there's
• 1:13:10
  this universe of tools that are trying to be all things to all
• 1:13:13
  people, right. And and getting COVID, it's a
• 1:13:15
  great example. They don't want, they don't want
• 1:13:18
  to be focused on certain type of developer.
• 1:13:20
  They want they want to get the lowest common denominator
• 1:13:22
  because pretty as possible, right?
• 1:13:24
  Because that's what's best for Microsoft's cloud business.
• 1:13:27
  If they do that, they'll get a lot more people burning compute
• 1:13:29
  on Azure, whereas someone like us is focused on enterprise
• 1:13:33
  development teams, like high performing enterprise developer
• 1:13:37
  teams. They're going to want things
• 1:13:39
  that are bespoke. They're gonna want things that
• 1:13:41
  are a little bit more focused on them and how they work and the
• 1:13:44
  things that matter. So that's why we made
• 1:13:46
  personalization such a critical part of what we do, because we
• 1:13:48
  don't care about being all things to all people.
• 1:13:51
  I want to go and make every engineering team more successful
• 1:13:56
  as a unit, right? I want to make that entire Jerry
• 1:13:59
  team more successful, not individual developers modestly
• 1:14:02
  more successful, right? And your example or a code
• 1:14:06
  quality is a great example. I do not want to accept just
• 1:14:10
  generating more code. I want to know that the code
• 1:14:14
  that we are generating as an engineering organisation is more
• 1:14:17
  successful. That is absolutely part of the
• 1:14:19
  parameters of these things. And I think that's we're going
• 1:14:21
  to start seeing a difference in a in general, you can go and
• 1:14:24
  just tell Dolly to generate you an image or you go to Adobe that
• 1:14:28
  actually has better curated content, knows what you're
• 1:14:31
  trying to do Stylistically, what you want is going to ultimately
• 1:14:34
  focus on generating something that isn't just, oh, look, I
• 1:14:36
  made an image, it's this is an ideal image from my use case,
• 1:14:40
  right. And I think that's where this,
• 1:14:42
  this expertise and the UK side is really going to come to the
• 1:14:46
  fore in the next 5 to 10 years as companies like ours are gonna
• 1:14:49
  say we care about the outcome, we care about the output right
• 1:14:52
  there is at the right level of quality.
• 1:14:54
  That user experience is a great thing to flag, and I hadn't
• 1:14:59
  thought about it until you said it, but it makes me think about
• 1:15:04
  how that is the differentiating factor.
• 1:15:07
  At the end of the day, you have to have that strong user
• 1:15:09
  experience when everyone has access to the model.
• 1:15:14
  Unless you are just adding a feature into your already high
• 1:15:19
  performing product, yeah, that isn't leading with AI.
• 1:15:24
  If you are leading with AI though, you gotta make sure that
• 1:15:27
  the user experience is the strongest because that is the
• 1:15:30
  differentiator. That's that's your vote right
• 1:15:33
  there. Yeah, agreed.
• 1:15:34
  I want to end with one last thing, which is you've had some
• 1:15:40
  time in marketing positions and you've specifically marketed to
• 1:15:44
  Debs. You've talked about how you
• 1:15:47
  really were looking for these three different pieces of
• 1:15:51
  information, but I want to dig a little bit further and ask if
• 1:15:57
  you found anything specific When it comes to marketing to
• 1:16:02
  developers, Have you seen any insights that you can share?
• 1:16:08
  Look, I I think you know I have. Yes, I have marketed to
• 1:16:11
  developers, but I've sold enterprise software, right?
• 1:16:14
  So, you know, I think there's an important thing to note with
• 1:16:18
  large scale developer tools which is not bought by
• 1:16:20
  developers. They're bought by the boss.
• 1:16:22
  So they're bought by their bosses boss.
• 1:16:24
  And what you learned very quickly is selling complex
• 1:16:28
  technical products. You don't sell to one person,
• 1:16:32
  you sell to a sphere of influence, you sell to a group
• 1:16:35
  of people who make the decision together and they all have
• 1:16:37
  different roles. And that's probably the biggest
• 1:16:40
  breakthrough I made early in my career and something I advocate
• 1:16:44
  for anytime I talk to anyone going to market and enterprise
• 1:16:47
  software is don't think you have one customer.
• 1:16:50
  You have a champion who really wants this product in because
• 1:16:54
  they have a problem they're trying to solve.
• 1:16:56
  You have a decision maker who's gonna say yeah or nay, You're
• 1:17:01
  somebody who controls the budget, which might be different
• 1:17:03
  than the decision maker. Think about procurement in a
• 1:17:05
  large company. And then you're now users.
• 1:17:08
  And those users, they can either push you in the business or they
• 1:17:12
  can keep you out of a business If you don't, if you don't win
• 1:17:16
  their hearts and minds. And so, like selling to
• 1:17:18
  developers, a lot of my career has been, can I build a
• 1:17:22
  groundswell of trust and belief among my end user?
• 1:17:27
  Do they believe in this? Do they feel like this is
• 1:17:28
  something they want to have in their lives every day?
• 1:17:31
  Right. They'll developers almost never
• 1:17:34
  have budget right? So you with that, can you
• 1:17:36
  activate them to become champions for you?
• 1:17:39
  Can you get them to push you in an organisation?
• 1:17:41
  And then can you go to the engineering manager?
• 1:17:44
  Can you go to the CIO or the CTO?
• 1:17:46
  Can you go to the architect or the principal software engineer?
• 1:17:48
  Those people usually are decision makers, right?
• 1:17:50
  They're champions or visual makers and help them understand
• 1:17:54
  the value you add. And so even when I coach, my
• 1:17:56
  salespeople like, look, know who you're talking to and know what
• 1:18:00
  they care about. Like if you talk to a developer,
• 1:18:02
  you know, coming out of out of the AI place, for example,
• 1:18:07
  developers don't care if that code is licensable or not, don't
• 1:18:11
  care like they don't care if the course is being pushed out to
• 1:18:14
  open a I. They don't care about any other
• 1:18:16
  stuff. Most developers don't last more
• 1:18:18
  than two to three years in a job.
• 1:18:19
  So they're not thinking long term.
• 1:18:21
  They're not thinking about these, They're just thinking,
• 1:18:23
  look, I've got, I've got 10 GERA tickets I have to work through
• 1:18:28
  in the next 5 days and I really only have time to get seven of
• 1:18:31
  them done. Like they're very focused on,
• 1:18:34
  like you gave me a job, I'm trying to get to this job fast
• 1:18:36
  as possible, right? So when we talk to developers,
• 1:18:39
  we're like, look, I'm going to help you get through the job as
• 1:18:41
  fast as possible. Now I am going to do it in such
• 1:18:44
  a way that doesn't get you scolded in the future.
• 1:18:46
  That is part of it because the code quality is bad, because you
• 1:18:49
  issuing code that wasn't licensable but those sorts of
• 1:18:52
  things. But when you talk to the CIO,
• 1:18:55
  you know they don't care about developer satisfaction.
• 1:18:59
  They I mean I I hate to say that but they they don't like they
• 1:19:01
  are those are those are machine parts to them, right.
• 1:19:04
  If you have 10,000 developers like I'll just hire another
• 1:19:07
  thousand. But they do care about
• 1:19:09
  productivity and efficiency. We do remind them that you don't
• 1:19:12
  want to turn developers to sets expensive and you know so you're
• 1:19:16
  going to really know your audience and make sure that your
• 1:19:18
  value propositions resonate with each of them.
• 1:19:21
  Like, are you really solving something that they all care
• 1:19:23
  about? Because if you're solving
• 1:19:25
  something that only one of them there is about and the others
• 1:19:27
  are resistance to, you're never gonna build a business you just
• 1:19:30
  built. Shoot me straight, Peter.
• 1:19:34
  You good at golf? No, I don't golf.
• 1:19:37
  I was a skateboarder. Do you?
• 1:19:38
  The Gulf. I thought I was about to say,
• 1:19:42
  you sold out on us and now you're golfing, selling to the
• 1:19:45
  CIO now on the weekdays. I actually, you know, I thank
• 1:19:49
  God that most of the guys in charge in in software decision
• 1:19:55
  making in the United States at least are jet Xers who grew up
• 1:19:58
  the same way I did. So I actually liked biggest
• 1:20:00
  hobby I race cars that was a hobby and I probably know more
• 1:20:04
  executives in the software through resetting than I know
• 1:20:06
  through work. So, you know, it's it's.
• 1:20:11
  I feel like adrenaline sports are the new Golf for for for my
• 1:20:15
  generation at least so. I'll and be more likely to find,
• 1:20:19
  you know, fine. There we go.
• 1:20:20
  But he's on the on a keyboard or on a snowboard or in a race
• 1:20:24
  track, Then they're not a golf course, which I'm grateful for.
• 1:20:28
  So there is one thing I want to leave us with which is I know
• 1:20:32
  you are hiring right now. You've got an incredible team of
• 1:20:35
  developers that are working in. You've got an engineering team
• 1:20:41
  remotely all over the world I think.
• 1:20:43
  And then you've got a home base also you've got a team in
• 1:20:46
  Israel, you're based in California, right?
• 1:20:48
  Correct. So if anybody out there is
• 1:20:51
  looking to join the rocket ship and let us know, hit up Peter,
• 1:20:58
  tell him you heard him on the podcast.
• 1:21:00
  It's been great talking to you, man.
• 1:21:02
  Like there's so much I've been, I've been scribbling notes a
• 1:21:04
  tonne, which I normally don't do because I try and get my full
• 1:21:08
  focus and attention to the guests.
• 1:21:09
  But here there was a few that I I was like, well, I have to
• 1:21:12
  remember that so that I can use it later and incorporate it into
• 1:21:17
  my life and how I do things. So thank you for coming on.
• 1:21:20
  Here always want to be helpful.