Latest / Agentic Conversations (formally mlops.community) / AI-Driven Code: Navigating Due Diligence & Transparency in MLOps // Matt van Itallie // #275

Transcript

AI-Driven Code: Navigating Due Diligence & Transparency in MLOps // Matt van Itallie // #275 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm Matt Vandedley, founder and CEO of FEMA, and I drink my
• 0:04
  coffee black. Welcome back to another ML of
• 0:08
  Security podcast. I am your host Dimitri OS, and
• 0:13
  today we talked about code based scans, mainly because Matt has a
• 0:18
  company that scans code bases when founders are looking to
• 0:25
  exit the company or when companies or hedge funds or
• 0:29
  people with a lot of money are looking to buy companies.
• 0:33
  They're doing their technical due diligence.
• 0:36
  And I find this issue fascinating in general, but the
• 0:42
  nuances and also the eight different things that he is
• 0:47
  doing when he's scanning the code base and checking for is
• 0:54
  really cool to see that you can just automate that number one.
• 0:58
  And #2 how much thought has gone into it?
• 1:02
  He said he took a whole year to figure out the best ways to
• 1:08
  translate certain pieces of the code base.
• 1:12
  That are are very. Technical into numbers or a
• 1:16
  score that non-technical stakeholders could understand.
• 1:22
  I don't want to spoil anything. Let's just get right into it.
• 1:25
  And as always, if you enjoy this episode, share it with just one
• 1:30
  friend. This episode is not sponsored
• 1:32
  and it is not financial or legal advice.
• 1:37
  I want to say that right now I love it.
• 1:46
  But you don't have any Italian roots, do you?
• 1:48
  So the last name, it means from Italy in Dutch.
• 1:52
  So hundreds of years ago ancestors moved from Italy to
• 1:57
  Holland and then my grandfather emigrated here.
• 2:01
  OK, all right. And you kept the Italian roots,
• 2:05
  not the Dutch roots then, huh? Yeah, well, van is like there's
• 2:07
  like you've heard of von in German means from, so it means
• 2:13
  from Italy, but in Dutch. So yeah, we're a lot more.
• 2:15
  I mean, I'm not particularly Dutch, but more Dutch than
• 2:18
  Italian. But the name the name has stuck.
• 2:21
  Incredible. All right man, I want to talk to
• 2:24
  you a lot about these code based scans and maybe we can just set
• 2:29
  the scene with what they are and how you think about it.
• 2:34
  Absolutely, and again, thanks for having me.
• 2:37
  I'm so looking forward to this conversation.
• 2:39
  So our code scans product is a detailed and also summary view
• 2:48
  of the non functional requirements of a code base.
• 2:52
  So it's not what does a product do?
• 2:53
  How well does it do it, how well, what kind of product
• 2:55
  market fit there is. It is stuff quote under the
• 2:58
  hood. We categorise it into two
• 3:01
  chunks. 1 is product related risks and that is the likelihood
• 3:06
  that the current code base will be able to that the team will be
• 3:11
  able to maintain and expand the current code base.
• 3:14
  And the second is compliance risks, things that lawyers might
• 3:19
  care about, things that could get you in trouble with external
• 3:21
  stakeholders. We capture 8 components or
• 3:26
  modules that fit into one of those two categories.
• 3:29
  And I'll just briefly because I know this is such a, such a
• 3:31
  technical audience and I love the detail at least run through
• 3:34
  all through all 81 is code quality, various forms of
• 3:37
  technical debt. A second is development process.
• 3:41
  So notice I didn't say it's a code.
• 3:43
  I think we're not just looking at the code, we're looking at
• 3:45
  the code base. And to us, it's really important
• 3:47
  to think about the engineering team around it, both as a whole
• 3:51
  and individuals, because that's a huge part of the of a healthy,
• 3:56
  of a healthy code base, full stop.
• 3:58
  So development process is the consistency or lack thereof, of
• 4:02
  development activity. We were working on product X.
• 4:05
  All of a sudden we tripled the commits to that.
• 4:08
  Maybe there was a good reason, maybe not, but that's that's an
• 4:10
  example of a process metric. The team metric is looking at
• 4:14
  developer by developer subject matter experts and whether they
• 4:19
  are still in the business. And I have said this, I'm very
• 4:24
  proud of this actually. We, you know, we serve our
• 4:28
  customers, our private equity firms, our deal teams, our
• 4:31
  non-technical CE OS and CF OS. And our job is to cross, cross
• 4:36
  the chasm between making tech understandable to non tech
• 4:40
  audiences. And we try very hard to do that
• 4:42
  as one of the things I think we've done a good job of, of
• 4:45
  teaching our, our customer base. We serve most of the world's
• 4:49
  best software investors. This is used in price
• 4:51
  significantly in technical due diligence when someone's buying
• 4:54
  a company. Not, that's not the only use
• 4:56
  case, but it's, it's been, it's a predominant 1.
• 4:58
  So back to team, one of the most important things and we would
• 5:01
  say the most important of all of these 8 sections about the
• 5:05
  health of a code base is the presence of subject matter
• 5:09
  expert developers. Now everybody listening to this
• 5:12
  knows that that is true, but to non, you know, to non-technical
• 5:17
  audiences, they think, well, it's the code, it's the IP, it's
• 5:20
  the words. And so my favourite, one of my
• 5:22
  favourite quotes, I do have a few, is having a, buying a code
• 5:26
  base without having the coders, the, you know, most of the
• 5:30
  quarters who wrote it and know it in detail is like having a
• 5:34
  half written novel without the novelist.
• 5:38
  Yeah, if the novel's done fine, you have the book, but if the
• 5:42
  novel's half done it you it really doesn't work.
• 5:45
  And code is 1/2 written novel code is never done.
• 5:49
  There's new features, functionality, there's upgrades,
• 5:51
  there's patches, etcetera. And so with all these eight,
• 5:54
  while they do matter, you know, in total that's the, the team is
• 5:58
  the single most important one. And again, it's it's content,
• 6:02
  it's the subject matter expertise.
• 6:04
  So that's number three. Number four is cloud spend and
• 6:10
  utilisation. So of course that is a huge
• 6:15
  accelerator, but also can be a tax in terms of how expensive it
• 6:18
  is. And so an assessment of whether
• 6:21
  how optimised it is, whether they're savings opportunities
• 6:23
  for for public cloud providers, then there is intellectual
• 6:29
  property risk from open source code.
• 6:32
  So if any of you have thought about copyleft or Blackdock is
• 6:37
  the 800 LB gorilla of that of an individual IP risk scan.
• 6:42
  That is about if you use certain open source libraries that have
• 6:49
  certain licences, there are certain circumstances where it
• 6:53
  creates intellectual property risk for the organisation,
• 6:56
  especially around high stakes diligence situations.
• 7:01
  Next is code security and that's the security both of code that
• 7:05
  the team wrote. Think of that as a SAS or a DAS
• 7:08
  scan. It's also the security of code
• 7:12
  that open source code. So CV ES 7th is a very light
• 7:21
  touch cyber assets review. I won't call it a pen test
• 7:25
  because it's it's it's not doesn't actually intrude, but it
• 7:28
  looks at the assets around the domains and subdomains.
• 7:33
  And finally in the one that we worked on over the last year is
• 7:37
  generative AI code in the code base.
• 7:40
  I will say a minute on that. And then of course, happy to go
• 7:43
  whatever direction you want. But you think about open source
• 7:46
  code. Open source is code the team
• 7:48
  didn't write. It's a super good idea to use
• 7:52
  it. It would be effing crazy.
• 7:54
  I hope I can say that PG effing crazy to not use open source
• 7:59
  because you're reinventing the wheel.
• 8:00
  It's annoying. Like the whole point of
• 8:02
  engineering is to solve problems.
• 8:03
  If that problem is solved, why do it yourself?
• 8:05
  But open source code comes with security risk, comes with
• 8:08
  maintainability risk, comes with intellectual property risk.
• 8:11
  Well, guess what? You replace open source with
• 8:14
  Jenny I code, you know, code that it comes.
• 8:16
  Either from a different. Literally the same, the team
• 8:19
  didn't write it, the team prompted it, but the LLM wrote
• 8:22
  it. We believe that it is the right
• 8:25
  framing to think of it like open source.
• 8:27
  It is a really good idea to use it in the right circumstances
• 8:31
  because it saves time, it's better for the organisation,
• 8:33
  etcetera. But it absolutely comes with
• 8:35
  quality risk. Jen AI makes up packages, among
• 8:39
  other problems. It comes with security risk.
• 8:42
  I'm sure it's clear every code comes with security risk.
• 8:44
  So we're not saying it's better or worse for Gin AI, but you do
• 8:46
  have to manage that risk and under certain circumstances, it
• 8:50
  comes with intellectual property risk.
• 8:51
  So we started with so folks who know software bills of material,
• 8:57
  which is basically how much open source is in the code.
• 9:00
  We were incredibly honoured that some pretty amazing customers
• 9:03
  said, well, you build an S BOM, now please figure out GIN AI
• 9:07
  code. And so we invented the G BOM
• 9:09
  generative AI bill of materials to understand how much Gen AI
• 9:13
  there is and how safely it's being used.
• 9:16
  Yeah, let's get into that. But before we do the, there's a
• 9:19
  few fascinating things on those first seven that I wanted to
• 9:25
  look at because basically, if you're telling me and I'm trying
• 9:28
  to understand it to get the full picture, which is I can scan my
• 9:34
  code and I can recognise like how much I'm spending.
• 9:37
  So I have a little bit of fin OPS in there.
• 9:39
  And if there's some easy wins recommendations on, hey, you
• 9:45
  might not want to use this service or you might want to
• 9:48
  think about turning it off. You're not using spot instances.
• 9:50
  You've got this GPU over here that's just been running for the
• 9:54
  last five days. I don't know why.
• 9:57
  Think about setting more alerts or something there.
• 10:01
  How does the actual practise of it look?
• 10:07
  And do I, so basically, do I go in and do I set all the knobs
• 10:12
  inside of what you're doing or is it that it automatically will
• 10:16
  scan it and then just give me recommendations?
• 10:18
  For for cloud cost optimization in particular, is that right?
• 10:22
  Yeah, let's let's start with that one, but I want to also go
• 10:24
  down the line. It's approximately and there's
• 10:28
  two phases to cloud to the cloud cost module.
• 10:31
  One phase is an hour of set up where we just analysed
• 10:36
  historical data. So it takes about an hour to set
• 10:38
  up and then there's detailed, detailed results that come from
• 10:44
  that just that hour or less of set up.
• 10:47
  There's a second level review if you want to get to per customer
• 10:53
  level analytics. So for certainly for larger
• 10:56
  organisations who should be taking into account the
• 11:01
  profitability or the COGS at least of individual accounts,
• 11:06
  there's another one to four hours of pulling in a collecting
• 11:11
  some customer related information to merge the two to
• 11:15
  come up with per customer. You know, COGS or profitability
• 11:20
  contribution. Side note, if you're a medium or
• 11:22
  large company, your board would be freaking thrilled by that
• 11:27
  because it shows as technical leaders you're thinking about
• 11:29
  the business impact and the, the, the, the business side.
• 11:32
  But Long story short, an hour or so of setup is enough to get to.
• 11:37
  Of course, this is how much you spend in the trends and this is
• 11:39
  the maturity level of your spend.
• 11:41
  And here is a rough estimate of how much savings is possible.
• 11:46
  You can't usually you can't flip a switch in most cases to carry
• 11:49
  out some of the improvements. You are pretty quick.
• 11:52
  Some of them require some some team time but we have literally
• 11:56
  never seen it pay not pay for itself 5X1010X or more.
• 12:03
  And so there's also this idea of code quality.
• 12:07
  And I'm wondering if the code quality is just on specific
• 12:14
  functions or are you looking at the whole pipeline?
• 12:17
  Are you looking at the CICD pipeline when we're the tests
• 12:20
  everything, the robustness of it, what does that look like?
• 12:24
  Yeah. So we do not do the whole CICD
• 12:26
  pipeline yet. If I were to wave a magic wand,
• 12:30
  we would do be working on Dora metrics which we haven't, we
• 12:33
  haven't added yet. So it's not about your podcast
• 12:37
  notwithstanding, we don't do dev OPS.
• 12:40
  We don't do dev OPS in any in any meaningful way yet, although
• 12:43
  it's definitely, definitely coming.
• 12:46
  It is for us for code quality is sometimes at the line level,
• 12:50
  sometimes the file, sometimes the block and then sometimes the
• 12:54
  repository. What I like, what I like about
• 12:56
  our approach to to looking at line level warnings is it's a
• 13:03
  very high level. So if we were to look at a code
• 13:06
  base and it had 50 line level, medium sized code base and it
• 13:11
  had fifty line level warnings, almost certainly the team is
• 13:15
  linting aggressively. It's almost impossible not to
• 13:18
  get to that number without. If there are 50,000 line level
• 13:21
  warnings, the team is not linting.
• 13:24
  Yeah, linting's not the end of the world, right?
• 13:26
  We think in general it has some benefits for developers being
• 13:29
  able to communicate with each other.
• 13:30
  But for us, of course, the day by day, minute by minute, team
• 13:34
  by team, getting the right, you know, having lynchers are not
• 13:38
  making that right decision is really important.
• 13:40
  But our job is to explain the whole state of the code to ACEO
• 13:45
  or a board member who doesn't understand it, doesn't know the
• 13:49
  details of line level warnings. And so we actually, part of this
• 13:53
  is we, we translate almost all these metrics into a one to 100
• 13:56
  score. That's incredibly provocative.
• 13:59
  We didn't do it lightly. Took us about five years of data
• 14:01
  collection before we were ready. Line level warnings counts less
• 14:06
  than 1% of the total because whether or not you're using a
• 14:10
  lint or not doesn't really move the needle on whether or not the
• 14:12
  code can be delivering business outcomes.
• 14:16
  By contrast, back to another non code quality one.
• 14:19
  But just for for comparison's sake, having team retention is
• 14:22
  worth 25 out of 100 points because the code doesn't like
• 14:26
  you're screwed if you don't have the team.
• 14:28
  So line level warnings at the line level, at the block level.
• 14:32
  If we do things like unit test coverage, we don't, there's, we
• 14:37
  can only measure unit test coverage looking at the at the
• 14:40
  code itself. So there's other kinds of
• 14:41
  testing that can't come through with a tool like ours.
• 14:44
  We look for duplicate code, excessive complexity or McCabe
• 14:47
  complexity, which was written in 1976 and it's still relevant,
• 14:52
  which I love. I love when good ideas last.
• 14:55
  And then at the repository level, we look at indications
• 15:01
  that the code base might need a refactoring.
• 15:03
  That is, I didn't say this at the beginning.
• 15:07
  We love data about code. I'm actually the son of a math
• 15:10
  teacher and a computer programmer, so I was kind of
• 15:13
  trained to treat everything. As data and code.
• 15:16
  Is data, but it it ends at a point code is not coding.
• 15:21
  Coding is not purely reducible to to data.
• 15:25
  Unlike sales or sports, it is a craft.
• 15:31
  It is not a competition. You can't just look at the
• 15:34
  metrics. Let's say the number of lines of
• 15:37
  code increased yesterday versus today.
• 15:40
  Is that a good thing? No.
• 15:42
  Maybe not, always not. Always right and frequently
• 15:44
  going down would be better, right?
• 15:46
  And so just from that simple example and you spin it up, we
• 15:49
  produce this report, we make it as clear as possible.
• 15:52
  We have automatically produce a set of discussion questions.
• 15:55
  And it's those discussion questions that need to be
• 15:57
  answered to really to really tell the whole story.
• 15:59
  And refactoring in particular is not the bowl reductible to an
• 16:04
  automatic assessment. It really does need a a human
• 16:07
  review. So I want to go down the rabbit
• 16:11
  hole of bridging the gap between the technical side of the house
• 16:15
  and the business side of the house.
• 16:16
  What are some of the key learnings?
• 16:18
  Yeah, when in doubt. When in doubt, explain things
• 16:24
  using middle school concepts. Because if you can make it
• 16:30
  accessible and understandable at the middle school level, whether
• 16:33
  it's math or analytics or telling a story, you can, you
• 16:39
  can make it work as long as it's defensible.
• 16:41
  You can make it work at all these other levels as well.
• 16:44
  So our, our system, and by the way, it is transparent you,
• 16:48
  you're used like we don't of those 8 modules looking for Gen
• 16:53
  AI code, We actually use AI to detect for various reasons.
• 16:57
  It has to be a probabilistic approach.
• 16:59
  We use a deep learning model for it, but everything else is
• 17:03
  mechanical, is deterministic. And that's in part because what
• 17:06
  we do, we are, you know, make this up Demetrius, you're going
• 17:09
  to, you're the CTO of a company you've put your blood, sweat and
• 17:14
  tears into making it work. And if this still goes through,
• 17:18
  yeah, it could be, you know, 1,000,010 million, 100,000,005
• 17:23
  billion dollars, $10 million. And it rides like the thing
• 17:28
  about technical due diligence is it's one or zero.
• 17:32
  If your sales are a little bit less than one would like or
• 17:36
  there's too much churn, someone might take down the purchase
• 17:38
  price. But tech due diligence, you
• 17:40
  either pass or you don't. And so in that moment, it's
• 17:44
  incredibly stressful. It's really important to be
• 17:48
  clear, and I obviously am a big fan of AI and I use it all sorts
• 17:53
  of ways professionally, personally in the company, you
• 17:56
  know, in the product, etcetera. But what doesn't work is
• 17:59
  Demetrius, we looked at your product and the black box says
• 18:02
  it's not very good. So we're not going to we're not
• 18:04
  we're going to recommend to our clients that they don't buy it.
• 18:07
  You would have a freaking heart attack.
• 18:08
  You would have a freaking heart attack and you should, right,
• 18:10
  because that it's so high stakes.
• 18:12
  We can't we can't leave it to something that that not
• 18:17
  determined. Now I'll answer with some
• 18:20
  specifics. If your audience can envision a
• 18:24
  credit score at a personal level, it it works like this.
• 18:29
  And by the way, we're happy to send you a sample and if you
• 18:31
  listening for free, you can see a sanitised version and you can
• 18:34
  also try it out for free. We have a it's not exactly the
• 18:37
  same as the true analytics, but there's a self-serve anonymous
• 18:43
  and not saved. So you can put it in.
• 18:46
  If you want me to look at it, I will, but you have to screenshot
• 18:49
  it because we we don't save it to protect confidentiality.
• 18:53
  But so the report has a one to 100 score.
• 18:56
  You heard some of the very, very particular waiting that we have
• 19:00
  thought about and put in. Lynter results are only worth
• 19:03
  1%, developer retentions worth 20 and 25%, etcetera.
• 19:09
  On on that developer retention, how do you know if the team is
• 19:12
  still there? Yeah.
• 19:13
  So we do version control. So the sources of data for all
• 19:18
  these 8 modules are threefold cloud, the cloud financial
• 19:23
  modules are looking at, you know you're spending usage in cloud
• 19:25
  to do cloud. We look at, we scan the domains
• 19:29
  and subdomains for the light touch pen test.
• 19:32
  It's not exactly pen test, but for the cybersection.
• 19:34
  And then the rest is the version control system data.
• 19:38
  And we don't just look at the code at the end, we look at the
• 19:40
  code at all points a time series data.
• 19:45
  So how the code has changed over time.
• 19:47
  So we know on Tuesday Matt made three commits to repository X
• 19:57
  and added 100 files. So we have that kind of
• 20:00
  information. The way that we do team
• 20:03
  retention, developer retention, is we look at when was the last
• 20:08
  time that person made a commit. Anywhere into the one of the
• 20:13
  repositories, you might have 20 products.
• 20:16
  If you committed to one of those products, you're still late
• 20:19
  yesterday, you're almost certainly still around and
• 20:23
  you're available to answer questions on other repositories
• 20:25
  even if you haven't touched those in a while.
• 20:28
  And so our, our rule of thumb automatically is if the coder
• 20:32
  has made a change to the code in the last 90 days, we assume
• 20:36
  they're there. And if they haven't made changes
• 20:38
  in the last 90 days, we assume they're not.
• 20:41
  Code is contextual. You cannot solely rely on that.
• 20:44
  Someone could be not have coded, but still be there because
• 20:48
  they're now an engineering manager.
• 20:50
  Someone could have coded recently and not be there
• 20:53
  because they left in between. And so that, you know, we start
• 20:58
  with a automatic quantitative analysis and then as part of the
• 21:01
  conversation that are, you know, the advisor should use us or, or
• 21:05
  if we're doing it, we do, we then adjust the results based on
• 21:11
  based on the conversation that whether or not people are
• 21:12
  actually there. OK.
• 21:15
  So that's the, if the person is there, what are the other things
• 21:21
  that make sure that leadership understands where we're at or a
• 21:26
  potential acquire IA 100% see the value in this.
• 21:31
  If you're doing the due diligence, I also see the value
• 21:33
  in this when you just want to be able to champion for something
• 21:39
  to the executive team and say, hey, look, we implemented XYZ.
• 21:44
  And what we're seeing from the code base scans are these
• 21:49
  results, right? It's almost like another data
• 21:51
  point that you can point to to say that what you're doing is
• 21:56
  working. Yeah.
• 21:57
  No surprise, couldn't couldn't agree more.
• 22:00
  In addition to their code scam product, which you've been out
• 22:03
  in the world for seven years, we are prototyping a SAS version to
• 22:07
  make that easier on a regular basis.
• 22:10
  And if your listeners want to weigh in, find me, we'd be happy
• 22:13
  to show you wireframe. So we're pretty we're pretty
• 22:14
  excited about that. I like, I love analogies.
• 22:19
  The sales team wouldn't only understand the state of sales at
• 22:24
  a diligence. They wouldn't work in the dark
• 22:26
  until someone showed up in a diligence is to tell me what
• 22:28
  your sales is like. Oh, wow, we learned this, blah,
• 22:31
  blah, blah. No, of course you're looking at
• 22:32
  sales at an executive level version of sales every day.
• 22:37
  If you're the the chief revenue officer.
• 22:40
  One of the most ironic things, certainly the most ironic things
• 22:43
  in in the world of coding is that engineering teams,
• 22:46
  engineering leaders have built dashboards for every person or
• 22:50
  every job on earth, but have not really built dashboards for
• 22:55
  themselves. And so you can think of the code
• 22:58
  scan as a one time CTO dashboard or engineering leader dashboard.
• 23:03
  We are turning that into a SAS version.
• 23:08
  It's very easy to do that poorly.
• 23:10
  So we do not want to do, we had to, we wanted to get a right at
• 23:15
  a point in time. Once you've got the right at a
• 23:17
  point in time, we're now building the time series 1 back
• 23:19
  to it. What does it mean whether or not
• 23:21
  you're using one of our tools or not?
• 23:23
  I think one of the biggest, some of the biggest tips for making
• 23:26
  this understandable to forgetting what you want from
• 23:31
  the rest of that, from the board, from the from the rest of
• 23:33
  the C-Suite. Always put things in context and
• 23:38
  always translate things into dollars and cents wherever you
• 23:42
  can. So for context, let me go back
• 23:46
  to Demetrius, your fictitious company.
• 23:50
  Let's say there were 10 developers who've ever worked on
• 23:54
  this code base and which for us is relatively small.
• 23:58
  And let's say we looked for CV ES, which are security warnings,
• 24:04
  of course, related to open source violation, to open source
• 24:08
  usage. Every code base has CV ES.
• 24:10
  It's not a question of if, it's how many.
• 24:13
  If you had a ten person software, A10 person software
• 24:16
  team and there were, I'm going to approximate this, let's say
• 24:22
  15 high risk security warnings, it would be almost certain, high
• 24:29
  risk CV ES almost certain that your team had a CVE detector in
• 24:34
  place and your team had permission, resources, capacity
• 24:42
  to actually fix them. Because SIMA has done, we've
• 24:46
  looked at a trillion and a half worth of companies, trillion and
• 24:49
  a half dollars worth of companies.
• 24:51
  We know that 15 only having 15 high risk security warnings is
• 24:55
  very, very good compared to your peers.
• 24:58
  The answer is not zero. In case anyone's wondering,
• 25:01
  everybody has some security warnings.
• 25:02
  If you were a ten person engineering team and you had 500
• 25:06
  security warnings relative to other ten person companies,
• 25:11
  you'd be in the bottom quartile. And both you didn't have a tool
• 25:15
  and, and, or almost certainly didn't have a tool and
• 25:17
  developers weren't able to, didn't have permission to fix
• 25:22
  it. And I say that intentionally.
• 25:23
  I've never met a developer who wakes up and says, you know what
• 25:26
  I want to do? I'm going to add security risk
• 25:28
  and I want to add technical debt.
• 25:29
  Like everyone, if they left to their own devices, of course
• 25:31
  they would want like the most pristine code base that
• 25:34
  possible, the beautiful ranting a little bit or hopefully
• 25:39
  inspiring, maybe a combination of both.
• 25:41
  The incredible thing about being a software developer is that you
• 25:43
  get to create for a living and it's this amazing craft that can
• 25:48
  be really high job satisfaction, really high comp.
• 25:52
  The exchange, the get the gift for that is we have to build
• 25:55
  things that are commercially reasonable or that meet the job
• 25:58
  that meet the needs of the organisation we're serving.
• 26:01
  That is what allows engineers to be able to have these amazing
• 26:04
  jobs. And so the amount of tech, there
• 26:08
  should be technical debt, there should be security debt, there
• 26:10
  should be intellectual property debt, all of those things
• 26:12
  because it's, it's not about having none, it's about having
• 26:15
  the right amount for the size and stage of your business.
• 26:19
  So back to this, how do you, how do you explain things back to
• 26:22
  the C-Suite? Don't say we have 15 warnings.
• 26:26
  We have 500 warnings. I don't know.
• 26:29
  It doesn't like, what am I supposed to do with that?
• 26:30
  As someone who doesn't know, say, the data, the benchmarking
• 26:34
  data says we're in the bottom quartile of bottom quartile of
• 26:39
  companies of our size and stage. It would behove us to not be in
• 26:43
  the bottom quartile, yeah. And then they say why?
• 26:48
  And then they say why code security can lead to.
• 26:54
  So if it's if you're an investor back company, being in the
• 26:57
  bottom quartile on any of these dimensions will make it harder
• 27:00
  for you to raise the next round or harder for you to get
• 27:02
  purchased. And being below average on
• 27:06
  security risk increases the likelihood that you're going to
• 27:09
  have a catastrophic data leak breach, etcetera.
• 27:14
  You can't prevent them at all, but you can get out of you can,
• 27:17
  you can be, you can avoid being the lowest, the lowest
• 27:21
  performer. So that's so one of the piece of
• 27:23
  advice is put it in context. And the other piece I said is
• 27:26
  can wherever possible, can you translate it into dollars and
• 27:29
  cents? That's hard, but it's definitely
• 27:34
  doable. And you know, let's say you had
• 27:38
  500 warnings that were legit. You could say, and let's say
• 27:44
  they receive, let's say there were SAS warnings.
• 27:46
  Just make it easier for a second.
• 27:48
  On average, each of those warnings takes about four hours
• 27:51
  of developer time to fix. 500 * 4 is 2000 hours and you know
• 27:58
  that's the average year is 1500 working hours.
• 28:01
  So it's one and a quarter one and a third.
• 28:05
  Excuse me, you know FT ES worth of work to clean up those
• 28:09
  warnings. For a giant company that would
• 28:12
  be worth it. For a brand new company would be
• 28:14
  incredibly expensive road map time relative to what else
• 28:17
  people can work on. And you can do the same thing
• 28:20
  with adding tests. We can you know, because of AI,
• 28:23
  we can add five tests an hour. It's going to take us 2 tests an
• 28:28
  hour. I don't I don't want to minimise
• 28:30
  it. We in order to get from 5%
• 28:35
  coverage to 15% coverage, it's going to take X 1000 hours and
• 28:38
  that translates to this many this many FT ES as well.
• 28:42
  It is reducible to person time or tooling cost which can then
• 28:48
  get summed up, you know, And again, executives are there
• 28:52
  because they're good at making trade-offs.
• 28:54
  If you can explain it in the language of you know they have a
• 28:57
  budget, implicitly or otherwise, of how to improve non functional
• 29:01
  requirements, it makes it much easier to you know to get to the
• 29:05
  right answer for the code base. Man I love this.
• 29:10
  This is really helping me understand how do you, like you
• 29:14
  said, speak a different language almost.
• 29:17
  And I do often hear how you need to always be able to tie your
• 29:25
  work that you're doing to a certain metric.
• 29:27
  And ideally it's a metric that the business cares about.
• 29:30
  And so translating something into dollars and cents is
• 29:35
  incredible. I have found that whenever I am
• 29:39
  working with startups, it's a little optimistic to think that
• 29:43
  you can translate it into dollars and cents like that.
• 29:46
  Because I think the startups are when you're in that infancy
• 29:52
  stage, you have so many things that like so many bullets right
• 29:57
  in front of your face and you got to be Neo in the Matrix and
• 30:00
  try and dodge all of them, right?
• 30:02
  And so you're thinking about many things that are of utmost
• 30:08
  important because there's a bullet 6 inches away and there's
• 30:11
  a bullet 7 inches away and then 112 inches away.
• 30:14
  And it is easy to say, OK, let me translate this into dollars
• 30:20
  and cents and let me do this reverse engineering of if we add
• 30:25
  X amount of tests and that's going to take us this many hours
• 30:28
  and that's X amount of developer time.
• 30:33
  Where I think it falls down is everybody says that that I've
• 30:36
  taught that I've encountered, right?
• 30:37
  Everybody will say that. And then the actual, all right,
• 30:41
  well, let's act on that information or let's do
• 30:44
  something about that is where I, I tend to find it falls down.
• 30:48
  Totally. And I think I would say from you
• 30:50
  know from our experience, if you have not raised or earned $10
• 30:57
  million, I wouldn't worry about anything that Sema says except
• 31:02
  maybe developer retention and then there still could be good
• 31:06
  reasons to have to switch. The only thing that matters is
• 31:11
  getting the product market fit and not running out of money.
• 31:14
  The only two things, those are the only two things.
• 31:16
  Do not worry about code quality, do not worry about code
• 31:18
  security. Get to a place, process
• 31:22
  consistency, any of these things get make it, get out, get to
• 31:27
  escape velocity and then future dollars, whether it's customer
• 31:31
  dollars or. Venture.
• 31:34
  Dollars or whomever then they use that money to work on these
• 31:37
  non non functional requirements. The first of the list that I
• 31:42
  might care about is testing because testing can increase
• 31:49
  velocity which can then help you get faster to product market
• 31:51
  fit. But I would be incredibly
• 31:54
  sceptical about adding unit tests or any kind of testing.
• 31:58
  I'd be very sceptical. Doesn't mean the answer is no,
• 32:00
  but I'd be very sceptical for almost any business adding them
• 32:04
  in those early stages. You'd have to be incredibly
• 32:08
  intentional that there was a return on investment of the
• 32:10
  developers time rather than just coding.
• 32:13
  Not that you shouldn't be adding it later.
• 32:15
  And obviously AI has, is testing is one of the best use cases for
• 32:21
  AI because it just makes it go so much faster.
• 32:23
  But any, any engineering leader, I would be inclined to be
• 32:26
  sceptical of of adding tests that early on.
• 32:30
  You can still fundraise like investors don't care at A at a
• 32:33
  Series A. They're expecting you to work on
• 32:35
  product market fit every with the right people and with the
• 32:38
  right budget. Any problem, any code problem is
• 32:41
  fixable. You need to stay in business and
• 32:44
  you need to grow. So get those things right and
• 32:46
  then hopefully when you make it then you can come back and worry
• 32:50
  about these non functional requirements.
• 32:53
  Brilliantly said. So let's dive into Gen AI and
• 32:58
  specifically in my mind, I've had this paper that I read
• 33:04
  probably 6 to 8 months ago that will not leave my head.
• 33:11
  And it is all about how something around the lines of
• 33:17
  75% of code that was created by Gen AI is then later deleted.
• 33:23
  And so it, it really goes back to this theme that you're
• 33:27
  talking about, move fast and then come back and refactor
• 33:30
  almost. And there's a time to move fast
• 33:34
  and there's a time to do what you have to do to make sure all
• 33:38
  your keys are crossed and your eyes are dotted.
• 33:42
  What have you found when you are looking at the Gen AI code?
• 33:46
  We are absolutely bullish on using Gen AI code full stop.
• 33:53
  We think it is better for developers, we think it's better
• 33:56
  for organisations. I would say it was high enough
• 34:00
  quality at least a year ago to be more useful than not.
• 34:05
  And it's only been it's only gone up the quality curve since
• 34:09
  in terms of accuracy, usefulness, etcetera.
• 34:15
  We really like using and really recommend using Jenny out a code
• 34:20
  in in almost all situations. But it comes with an incredibly
• 34:26
  strong recommendation, which is to apply every kind of rigorous
• 34:30
  review that you normally would and then maybe then some.
• 34:34
  And of course that means putting Jenny I through any of your
• 34:40
  quality gates, your security dates.
• 34:43
  Not that anyone would do this, but you can't we you know, you
• 34:47
  don't run open source framework through a security scan.
• 34:51
  You use it right, because you're it's gone through others.
• 34:55
  This you I mean, you would run the CV ES, you wouldn't run a
• 34:58
  SAS or DAT. You wouldn't you wouldn't put it
• 35:00
  a SAS or DAS scan on your on open source libraries.
• 35:04
  You most certainly need to put security scans, quality scans,
• 35:08
  any scans you have on on Jenny I code.
• 35:12
  Most importantly, code is a craft is code review most
• 35:17
  important. So if you are you know, you
• 35:19
  don't we don't code review open source libraries we're using.
• 35:22
  We just use them. That's it's obviously a very big
• 35:24
  difference here. It should get extra code review.
• 35:28
  And I'd say the third part is, you know, when Sima scans for
• 35:31
  Gen AI code, we actually ask two questions. 1 is how much code
• 35:35
  was Gen AI originated versus not Gen AI originated, Not Gen AI
• 35:38
  originated could be written by the team, copied from Stack
• 35:42
  Overflow, copied from, you know, open source reference.
• 35:46
  If it is Gen AI originated, question #2 is was it modified
• 35:50
  by a developer? And so we think about Gen AI
• 35:53
  pure. Gen AI blended.
• 35:55
  Pure came straight untouched from the prompt.
• 36:00
  Gen AI blended has been modified at least a bit by a developer.
• 36:05
  So think of the extreme example. Imagine you were looking at a
• 36:08
  code base and you asked the developer how they built it and
• 36:12
  100% of it was Gen AI and it hadn't been modified in any way.
• 36:16
  It just came straight from the prompt.
• 36:18
  You should be very sceptical that it was contextually
• 36:21
  appropriate, that it didn't have major quality holes, didn't have
• 36:25
  any security risks, etcetera. So that would be an example of
• 36:28
  100% pure Gen AI. Now if you're prototyping, sure,
• 36:33
  right. If it just gets the idea across,
• 36:35
  it's a great, it's a great way to do that.
• 36:37
  But if you're talking about enterprise grade, you know,
• 36:39
  production ready code, by all means you should be pretty, you
• 36:42
  should be quite nervous. And so we measure Gen AI
• 36:45
  originated, yes, no. And then if Gen AI pure blended,
• 36:50
  just, you know, a code review could it could be good.
• 36:55
  So we're not saying Gen AI code has to be 100% blended, but we
• 36:59
  are saying if it's not largely blended, it's a pretty big risk
• 37:02
  factor that that folks aren't taking the the review seriously.
• 37:07
  It's basically the video that I've seen on line where a man
• 37:14
  teaches his 8 year old daughter to code with cursor and she's
• 37:19
  able to do it. It's a true feat that she can
• 37:23
  create a web page with cursor and edit things there.
• 37:27
  But if it's pure Jenny I code, you're kind of no different than
• 37:33
  that 8 year old. Yeah, you know, you can prompt
• 37:37
  it well and you know, I used Mai when I write, certainly when I
• 37:42
  research and for low stakes stuff if it can be about right
• 37:49
  or close enough and I I don't need to modify to get the point
• 37:51
  across. But for anything high stakes, I
• 37:55
  rewrite it from scratch or certainly line by line I'm
• 37:58
  editing and most things, most words are getting modified.
• 38:01
  So the the more serious the same is true for code, the more the
• 38:04
  higher stakes it is, the more you should expect that you're
• 38:08
  going to be line editing it. And the more sceptical you
• 38:11
  should be that if you see code that is high stakes and hasn't
• 38:14
  been line edited, that would be that would be not determinative,
• 38:17
  but certainly a red flag. Have you seen patterns of Gen AI
• 38:23
  code? Because as you were saying,
• 38:25
  there's, when we write, I think just about everybody uses some
• 38:29
  form of Jenny I these days to write.
• 38:32
  I notice I never liked Jenny I writing because it is so
• 38:36
  verbose. And even if you prompt it well
• 38:39
  and say short sentences, one sentence, paragraphs, it still
• 38:44
  doesn't listen and it doesn't really do it.
• 38:47
  That is a key giveaway for me. And there's some words that are
• 38:52
  key giveaways that you're using Jenny I in your writing.
• 38:55
  Are there things that you've noticed or you've seen in scans
• 39:00
  that potentially the code quality or Jenny is going to to
• 39:05
  learn more towards this? I've heard stories about how if
• 39:10
  you are writing in Python, you're good because there's been
• 39:12
  so much data and training examples in the LLM with Python,
• 39:17
  But if you're trying to write and go, maybe you get random
• 39:21
  Python spliced in there. Have you seen any of this in in
• 39:25
  the code quality side of things or is that a non?
• 39:29
  Factor. Yeah, it's a really good
• 39:30
  question. My honest answer is I have heard
• 39:34
  those stories and I have not found any of them in particular
• 39:38
  conclusive, which does not mean they're wrong.
• 39:41
  It means any individual developer out there, I would be
• 39:45
  at least once 1/4 contemplating switching tools or using more
• 39:50
  than one tool. Now if you're working in a
• 39:52
  commercial setting, you have to make sure that you have approval
• 39:55
  for those tools and they have to be at the right licence level to
• 39:57
  protect protect your company's intellectual property.
• 40:01
  But there are at least 5, arguably 10 very high quality
• 40:06
  LLMS, whether they're specific decoding or not, and more are
• 40:10
  coming that I would be experimenting pretty regularly
• 40:14
  to swap out. Do I like is this one doing a
• 40:16
  good job for this or or not? We see it as a, it's kind of an
• 40:20
  audience or, but I think you'll know where I'm going.
• 40:22
  It's a real strength for us if we're looking at a company and
• 40:24
  they let their developers, they pay for more than one tool
• 40:27
  because it's a recognition of just how powerful these tools
• 40:31
  are in terms of helping developers do their best work
• 40:33
  and an acknowledgement that we're so early in the process.
• 40:37
  You know, you could mandate everyone use VS Code does many
• 40:41
  things, right? It may not be the best in every
• 40:43
  situation, but you're not you're not losing much, you're not
• 40:46
  losing leaving much value on the table if you force everyone to
• 40:49
  use the same IDEI don't think that's true for forcing everyone
• 40:52
  to use a single, a single LLM to code.
• 40:57
  I think you could be really losing out.
• 40:59
  And certainly I would want you to make that review that
• 41:01
  decision pretty regularly. Executives listening it, your
• 41:06
  coders are using genetic code, full stop.
• 41:08
  Everybody's using it. So please buy them a commercial
• 41:12
  grade licence and please take me seriously that you should let
• 41:16
  them experiment, use more than one, have a workshop like a, you
• 41:21
  know, a, a slack channel on discussing which ones are
• 41:24
  working in different circumstances, etcetera.
• 41:27
  Because, well, it, it very well could be true that language X
• 41:31
  doesn't work today on, you know, it, it could, that could be a
• 41:34
  tool specific decision, it could change over time, etcetera.
• 41:37
  This tools at work, they're very powerful, but they're still
• 41:40
  being built. So please, please, please
• 41:43
  encourage and support that. Experimentation.
• 41:48
  Have you seen trends with Gen AI created code that is actually
• 41:55
  copyright code? So this is not legal advice.
• 42:01
  Go talk to your team's lawyer about this.
• 42:05
  If we think about Gen AI code, you know the plus side of using
• 42:10
  Gen AI code is it increases productivity, throughput,
• 42:14
  developer satisfaction, etcetera.
• 42:16
  The downsides can include the code is not high quality enough.
• 42:20
  You solve that through code reviews and tooling.
• 42:23
  The code is not secure enough. You solve that through your
• 42:26
  security tooling and and code reviews and making sure the
• 42:30
  developers have time to fix security issues, which is not a
• 42:34
  given. Third is intellectual property
• 42:38
  risk from using Gen AI to code. And 4th is what I'm saying.
• 42:42
  It is something called exit risk.
• 42:43
  We'll come back to it within. You see, I'm a very structured
• 42:47
  guy. Within intellectual property
• 42:49
  risk, there is the risk that your code can't get patent
• 42:54
  protection, that you're seeking patent protection and can't get
• 42:57
  it. We think that that risk is 0 to
• 42:59
  none. You should be able to get patent
• 43:01
  protection because the patent is protecting the idea, not the
• 43:05
  text. Another important IP protection
• 43:08
  is trade secret, which is just like Coca Cola's formula.
• 43:11
  You just can't let anyone have access to it.
• 43:13
  The way that you can prevent that is making sure that you're
• 43:17
  using the right tool with the right licence level.
• 43:21
  Please play the following clip to anybody who needs to use your
• 43:25
  your legal team. It is crazy not to give
• 43:28
  developers enterprise grade licences and implicitly I'll let
• 43:31
  them use free tools for free. You know non enterprise tools
• 43:36
  for free. Because those tools train on
• 43:38
  what people are entering in and it is just like posting your
• 43:41
  company secret code on the Internet, which you should not
• 43:43
  do. So trade secret is a real risk,
• 43:45
  but it's solvable by giving people licences.
• 43:48
  Some companies seek copyright on their code just like they seek
• 43:54
  patenting on their code There. It is not clear we would
• 43:57
  consider that a risk because copyright is you copyright the
• 44:03
  words themselves or the work itself.
• 44:07
  And in human writing, you can't say try to be to your whomever,
• 44:13
  write me a book and then go get it copyrighted because the words
• 44:16
  are not yours. The words are a computer.
• 44:18
  So if you're at a company, it's only the largest ones really who
• 44:21
  seek copyright protection. Number one, please talk to your
• 44:24
  lawyers and #2 come talk to us because copyright protection,
• 44:29
  you're going to have to take some safety precautions if
• 44:32
  you're going to continue to copyright your code.
• 44:34
  That leaves us with infringement.
• 44:37
  So copyright of your own code, once you've written it, can you
• 44:39
  get copyright protection? Now let's talk about infringing
• 44:43
  because the LLM is trained on code that then passes to you
• 44:51
  that you should not. So that infringement a little
• 44:55
  structure again, one is copyright infringement and the
• 44:58
  other is open source licence risk infringement.
• 45:03
  We we assess again, it's not, it's not legal advice, talk to
• 45:07
  your lawyers. We assess that copyright
• 45:11
  infringement is relatively low because for for it to be trained
• 45:16
  on copyrighted code, they would have had to have gotten the code
• 45:18
  in the 1st place. And it's very unlikely that they
• 45:21
  were training based on copyrighted code, which is it's
• 45:23
  going to be secret in the 1st place.
• 45:25
  That leaves infringement of the open source licences.
• 45:30
  So just like regular open source comes with licences that are
• 45:32
  risky, so too if the if the training set, you know, it was
• 45:37
  trained on open source with the wrong kind of licence and then
• 45:40
  that code made it into your own code, it generates the risk
• 45:47
  there is unclear. So I would say if your company
• 45:51
  above average cares about open source legal risk, open source
• 45:55
  legal risk, then it is likely you should apply that same level
• 46:00
  of care to open source to open source infringement risk from
• 46:04
  using Gen AI code. If you today don't care if your
• 46:08
  company, and I don't mean you as a developer, I mean your legal
• 46:10
  team, if they don't care about open source legal risks, it is
• 46:15
  likely that you're not. The the risk of open source
• 46:19
  legal risk from Gen AI is smaller than directly using open
• 46:23
  source. So if you don't care about the
• 46:24
  larger risk, you likely don't have to worry about it.
• 46:27
  Demetrius, I can only hope that was the most boring 5 minutes of
• 46:29
  your entire week. Intellectual property risk and
• 46:33
  and code. Apologies to the listeners.
• 46:35
  I hope at least some of you found it interesting and aren't
• 46:37
  like falling asleep at the wheel hearing a little IP talk.
• 46:42
  Well, I do like how you break it down as a spectrum as opposed to
• 46:47
  just like, yeah, watch out for it.
• 46:49
  It's in there and it is very nuanced and it's also what does
• 46:54
  your company care about. And then you should think about
• 46:59
  the potential risks. It sounds like the real risk is
• 47:03
  probably using some type of open source package, not in the
• 47:08
  generative AI code that could be using open source.
• 47:16
  That single topic is the Wizard's been the most vigorous
• 47:20
  debate in the legal community. And so, you know, let me
• 47:24
  acknowledge there's a debate, which is another even more
• 47:26
  important reason that you should talk to your lawyers about this.
• 47:29
  But logically, what you're saying must be true.
• 47:32
  If the company goes and uses open source with the wrong kind
• 47:39
  of licence, it's directly risky. Now how likely is that risk to
• 47:43
  be realised? Depends, but it's a direct use.
• 47:47
  If the the LLM creator uses training set data with the wrong
• 47:53
  kind of licence, that itself is not enough to trigger risk in
• 47:58
  the user. It's enough to trigger risk for
• 48:01
  the LLMS. The LLMS are getting sued over this or getting sued,
• 48:05
  but it's not enough just that the the LLM creator is using it.
• 48:09
  That code would have to make it to your code for it to actually
• 48:13
  be triggered. There's not a general risk and
• 48:16
  someone would have to find it and someone would have to care.
• 48:18
  Now this is my public service announcement given everyone out
• 48:22
  there. Your code runs on open source
• 48:26
  directly because you're using it, but also indirectly because
• 48:29
  it's been the training set for all of your Gen AI tools.
• 48:32
  Please, you have your company make a suitable investment in
• 48:37
  doing right by the open source communities you're a part of.
• 48:40
  Commiserate with what resources you have.
• 48:43
  If you're a startup, do not spend 0 because the most best
• 48:47
  thing you can do for the world is make it like you're a small
• 48:49
  start up. Don't, don't spend any time.
• 48:51
  And if you're a giant one, I mean, obviously look at the
• 48:53
  contributions that the that Fang companies have made to open
• 48:57
  source, right? They can put, I'm sure billions
• 49:00
  of dollars in over time, make an investment, commiserate.
• 49:03
  And it doesn't have to be cash. It could be letting developers
• 49:05
  code on it, etcetera. That's the right thing to do,
• 49:09
  given that you're getting this for free.
• 49:11
  And it probably also helps, I guess tactically, were you ever
• 49:15
  to be challenged by your inappropriate use of open source
• 49:18
  code, it would be good to be able to point out, hey, we made
• 49:21
  a mistake. But listen, we take, we take
• 49:23
  this seriously. We're good.
• 49:24
  We're good citizens. That doesn't literally get you
• 49:28
  out of the legal risk you might be facing, but it doesn't hurt.
• 49:32
  It doesn't. Hurt.
• 49:34
  You did mention exit risk also. Yes.
• 49:37
  So exit risk is the possibility that if your company is getting
• 49:41
  is getting sold or someone's thinking about putting
• 49:43
  investment in that a buyer or investor looks at the code you
• 49:48
  wrote and decided we can use AI to build it instead.
• 49:53
  And your company is no longer valuable to us or is less
• 49:56
  valuable to us. Wild, wild.
• 50:00
  I'm sure you've seen wait say that.
• 50:01
  Again. So basically it's like there's
• 50:03
  no Moat here. AI can do the, it's the
• 50:07
  potential. Of it's the potential of no
• 50:08
  mote. You've seen inception, I hope,
• 50:12
  right, that Christopher Nolan. Yeah, I like to explain the
• 50:16
  power of Jen AI in the those world creators where they
• 50:20
  literally think it and then the world just unfolds.
• 50:23
  Certainly with respect to prototypes, Jen AI is 95% faster
• 50:29
  than doing it some other way, right?
• 50:30
  And seeing some of that actually works with real code, it's so
• 50:33
  much faster. We there are public published
• 50:36
  examples of companies in a diligence process, red teaming,
• 50:42
  taking a group of people and saying, could you just build
• 50:44
  this yourself? And instead in deciding in at
• 50:48
  least one instance that they're not going to buy the company
• 50:50
  they were looking at. Now take a 10,000 foot view.
• 50:54
  Why do companies buy other companies or invest in other
• 50:56
  companies? Because of the financials and
• 51:00
  because of the technology. You know why they invest in
• 51:03
  technology companies, the financials, Let's take eBay.
• 51:07
  eBay doesn't matter how they're building the code.
• 51:09
  They are very, it's a great brand.
• 51:11
  They have a lot of revenue, a predictable revenue.
• 51:14
  Whatever they're doing, working. It could be they could have no
• 51:17
  open source, they could do it by hand.
• 51:18
  I know that's not true because there's a great team and they
• 51:20
  rely on open source. Whatever they're doing is
• 51:22
  working. If you are, you know, if you're
• 51:26
  building an analytics tool, that if you're building an analytics
• 51:31
  tool that takes unstructured data and produces plain English
• 51:35
  insights, well guess what? Any LLM can do that.
• 51:39
  And even if you make it to the stage of diligence, you, they
• 51:44
  might say, well could we, could we build this ourselves instead?
• 51:47
  Now, if you have a whole bunch of customers who love your
• 51:50
  version of it, then they are not likely to switch to a generic
• 51:54
  LLM tool. And so the way that we'd say it
• 51:57
  is if a investor or Enquirer is buying your book of business, is
• 52:01
  buying your sales, that is still protected regardless of how well
• 52:05
  of how you got there, not how well if they're buying the tech
• 52:08
  or they're emphasising the tech. It is less valuable than it was
• 52:12
  6 or 12 or 18 months ago. And that trend is only going to
• 52:15
  continue. That's kind of the sad news.
• 52:18
  The good news is you can get to a product that is sellable
• 52:22
  faster. So if you can build products
• 52:24
  five times fast, certainly three times faster to get to
• 52:27
  production on not just prototype, but actually get to
• 52:30
  production, well, then you can get to business outcomes, you
• 52:33
  can get to revenue faster. And so yes, there could be an
• 52:37
  uncomfortable conversation about is your code reproducible, but
• 52:40
  you're in diligence conversations in three years
• 52:42
  rather than 5 because you've you've built a valuable product
• 52:47
  that much faster. It comes back to remember the
• 52:50
  beginning of this where I talked about pure versus blended code.
• 52:54
  Blending is your friend because it shows that you are and it
• 52:58
  don't just do it for its own your own sake, But the more
• 53:02
  contextualised and specific to the problem you're solving, the
• 53:06
  more valuable that technology is relative to building it from
• 53:10
  scratch. So all of the benefits of all of
• 53:14
  the risks of Jenny I code are addressed by blending to solve
• 53:18
  security issues, to solve quality issues, IP risk for the
• 53:22
  most part, and an exit risk because it's it's not your
• 53:26
  concern. Again, imagine an extreme
• 53:27
  example. I'd like you to buy my company.
• 53:29
  Did you write it? No, it's three open source
• 53:32
  libraries. The the value of that tech is 0
• 53:35
  because people can get it for free.
• 53:37
  Now you could have an amazing customers who love it and for
• 53:40
  some reason are buying it through you, right.
• 53:42
  That's an extreme example. Jenny eyes.
• 53:45
  It's it's bringing us in that same direction.
• 53:47
  The code will matter less, and certainly generic code will
• 53:51
  matter a lot less than than than truly custom code.
• 53:57
  Yeah, it's funny. There's something there about
• 54:01
  the code, but then also everything else around the
• 54:05
  company and the brand. Is the brand strong enough to
• 54:09
  where OK, it is generic code, but because the brand is there
• 54:13
  and people go with it, like you were saying with the eBay
• 54:16
  example or even I know there, there's plenty of social media
• 54:24
  sites or templates that you can grab a news feed with a simple
• 54:30
  plug in and you have on your website a news feed that doesn't
• 54:34
  make you Twitter or Facebook or whatever name your favourite
• 54:38
  social media network, right? There's, there's those brands
• 54:40
  they're granted that's an extreme example because I'm sure
• 54:44
  the code bases of every one of those that I just named off is
• 54:48
  very complex and not at all in this same ballpark that we're
• 54:53
  talking about. But it is fascinating to think
• 54:56
  about that how how to create that defensibility.
• 55:02
  And I'm wondering since you are playing in the intersection of
• 55:07
  companies that are getting bought and basically the
• 55:12
  acquirers, right, because your tool helps the acquirers
• 55:17
  understand what's going on in the code bases of companies.
• 55:22
  Have you seen trends or do you have anything on how many
• 55:27
  companies are getting bought right now?
• 55:29
  Is there anything you know how with, I think it's Carta puts
• 55:33
  out there, this is how many companies are getting invested
• 55:36
  in Series A, Series B, that type of thing.
• 55:38
  Do you have any information on that or is it all behind NDA's
• 55:41
  that you can't share any? Yeah.
• 55:43
  I can say at a high level, MNA activity is definitely up from
• 55:46
  last year and definitely up from 2 years ago.
• 55:50
  We're not all the way there yet of we think 2025 is going to be
• 55:55
  gigantic in terms of MNA activity.
• 55:59
  Investors have so much, so much, so much funds to spend that we
• 56:04
  call it dry powder that they haven't invested yet and it's
• 56:07
  been building up. So we think so volume is
• 56:09
  definitely up. It's going to blow up sometime
• 56:12
  in 2025. Wow.
• 56:15
  And this is all for SAS companies.
• 56:18
  Primarily it's. Certainly a SAS companies.
• 56:22
  I try not to speak about things I'm not super familiar with.
• 56:24
  I would expect a general increase in investment activity
• 56:31
  because these investors turn, the investors we know well are
• 56:34
  multi fund or multi strategy. So they're doing different
• 56:38
  things. It's not just SAS, but certainly
• 56:41
  software companies. We'd expect to see a pretty
• 56:43
  substantial increase in M&A in the next 6 to 12 months.
• 56:48
  Incredible.