Latest / Agentic Conversations (formally mlops.community) / AI Reliability, Spark, Observability, SLAs and Starting an AI Infra Company

Transcript

AI Reliability, Spark, Observability, SLAs and Starting an AI Infra Company · Agentic Conversations (formally mlops.community)

• 0:00
  And our view is inferences the new transform.
• 0:03
  Data started like slowly to become more of the product and
• 0:07
  not just like a back office things that the company needed
• 0:09
  like to run the business. Platforms that exist out there
• 0:11
  don't treat these as first class citizens.
• 0:13
  It's all like, OK, you're not getting a bonus this month
• 0:16
  because your prompt was not good, dude.
• 0:17
  Who's going to be the pets.com of the LLM bubble?
• 0:28
  I'm I'm excited because I was the metaphorical Cupid of U. 2
• 0:35
  Ultimate matchmaker. Yeah, you did it.
• 0:37
  Building a company is always a journey.
• 0:40
  Usually people talk about what they've done after the fact, so
• 0:44
  they tend to like to remove a lot of the gory details.
• 0:47
  Yeah. So both me and Yoni.
• 0:51
  We, we, we, we came from similar experience of the market, but
• 0:58
  from different angles. So I was working mainly like
• 1:02
  more, let's say traditional data infrastructure.
• 1:05
  So I saw a lot of like the data engineering like work there,
• 1:08
  especially like in the enterprise Yoni was came from
• 1:11
  data infrastructure again, but more from the ML and we started
• 1:15
  like working on that. OK, how we can rethink, let's
• 1:20
  say and build new tooling around working with data infrastructure
• 1:26
  for the problems that we have today?
• 1:28
  Because keep in mind that like the dominant tools that we are
• 1:30
  using were created 1213 years ago.
• 1:34
  Completely different use cases like.
• 1:36
  Spark. Spark, Trino, even like the
• 1:39
  commercial tools like Snowflakes, they're kind of like
• 1:41
  similar. At the end of the day, it's all
• 1:44
  about how we move into the cloud and how we do like this big data
• 1:48
  thing, which the dominant use case is BI, It's analytics,
• 1:52
  right? There is ML, but ML was like
• 1:54
  always more of a niche thing. At the end of the day, nothing
• 1:58
  converts like to the BI market. If we take it in like just like
• 2:01
  the numbers out there, right? But it's like 1213 years.
• 2:04
  Many things have changed, like many new workloads came in.
• 2:07
  There's like ML, what was like Yoni was working on, OK, today
• 2:10
  we have AI. We'll talk more about that
• 2:12
  later. There were things about like
• 2:15
  embedded analytics, a lot of like product related, like data
• 2:20
  started like slowly to become more of the product and not just
• 2:23
  like a back office things that the company needed like to run
• 2:26
  the business right. And of course like these tools
• 2:28
  were not built for that. We could kind of like pass them
• 2:33
  like to do it. But as the market was demanding
• 2:36
  more and more and more, the pain was becoming bigger and bigger
• 2:39
  and bigger. And we felt that like, OK, this
• 2:40
  is like the right time to go out there and start building
• 2:45
  something in this space. Yeah.
• 2:46
  No, I think you're, you're right on there.
• 2:47
  I think now more than ever what we were seeing is teams saw the
• 2:51
  value in putting data products into production as quickly as
• 2:54
  possible because they realized the direct correlation that they
• 2:57
  have with business outcomes. The more effective they can put
• 3:00
  data products into into production, the Better Business
• 3:03
  outcomes they get. And it's not just now the
• 3:05
  Silicon Valley companies, right? If you think about it, it's all
• 3:08
  the non tech first companies also saw this value there,
• 3:12
  right, but the tooling didn't didn't enable them to do that
• 3:15
  right. So I can tell you how much time
• 3:17
  we spent like thinking back in my time at Tecton, like helping
• 3:21
  customers debug through their spark logs right and working
• 3:23
  with them to figure out, you know, we would go and onboard
• 3:26
  them and these weren't Silicon Valley companies that have run
• 3:29
  spark cluster before. So here is spark one O 1.
• 3:32
  This is how you this is a spark config.
• 3:34
  This is how you go through and like your.
• 3:36
  Boy, you're in for a tree. And, and at the end, what we saw
• 3:39
  is the main thing that's evolved from, you know, 15 years from
• 3:43
  now when like, you know, the brilliant guys from Databricks
• 3:45
  went out and built Spark, is that there's a couple things
• 3:50
  that have really changed since that time, right?
• 3:52
  And one thing like hardware's evolved tremendously, right,
• 3:55
  which is Leah Costas and I always talk about this is that,
• 3:58
  you know, it's at that time distributed workloads were a lot
• 4:02
  more necessary, right, because you're a lot more limited on the
• 4:06
  resources you can get through AWS.
• 4:07
  Now you can go and check out, you know, a huge instance on, on
• 4:11
  EC2 and, and duck DBS come in and data fusions come in and
• 4:16
  they've kind of flipped analytics world on its head
• 4:17
  where you can do a lot more in single node, right?
• 4:21
  And we saw that at Tecton 2 where we'd have these these
• 4:24
  clusters, but and you'd be running like hundreds of
• 4:27
  gigabytes of workloads that weren't really necessary to have
• 4:30
  distributed spark clusters there, right?
• 4:32
  You can really do a lot on single node.
• 4:35
  So that's one thing, one thing. And I think one of the catalysts
• 4:38
  that we were kind of identifying and other things, AIS come into
• 4:41
  the picture too, which has totally changed the nature of
• 4:44
  workloads, right? It's no longer only about
• 4:47
  structured tabular data, which we have great platforms for
• 4:50
  right now and more or less a solve.
• 4:52
  There's a lot of challenges there, but now there's
• 4:55
  unstructured data, there's a lot of text data, there's images,
• 4:58
  there's videos, all the different modalities there that
• 5:02
  these platforms just weren't built for, right?
• 5:04
  They're first class citizens are structured tabular data.
• 5:07
  They do that really, really well, but they don't have the
• 5:10
  ergonomics, the capabilities. They weren't built from the
• 5:13
  ground up with first principles having these characteristics of
• 5:17
  AI workloads in mind. The other thing that like AI did
• 5:19
  was actually acting like a catalyst to the market and the
• 5:25
  industry. Because if we take and see what
• 5:28
  was happening against like in the past 1213 years, we have the
• 5:32
  analytics market, which is huge, but still is it's a back end
• 5:37
  kind of work, right? You have reports that you need
• 5:40
  to build and deliver. These reports are consumed by
• 5:43
  people who make decisions, blah, blah, blah.
• 5:45
  But it's not a product like it's not the customer facing product
• 5:49
  of every company out there right now.
• 5:52
  These things started changing gradually, but it was changing
• 5:56
  slowly. We had the MLML was like part of
• 5:58
  that because when you have recommenders right now, you the
• 6:01
  data becomes the product itself. Like you need to use the data in
• 6:04
  order to recommend and increase your revenue, right?
• 6:08
  And it's not like BI doesn't do that.
• 6:10
  But the link between, let's say, how the data is used and the
• 6:15
  value that's created is like kind of hard like to identify,
• 6:18
  right? Yeah.
• 6:20
  That's been a traditional problem and I know that there's
• 6:22
  so many teams that have talked about that, like how do you
• 6:25
  champion for the work that you're doing if you can't draw a
• 6:29
  Direct Line from what you are doing to revenue generated or
• 6:33
  saved? 100% and especially the way that
• 6:37
  data teams are built, I think in the successful teams like take a
• 6:40
  take a company like Lyft, for example, right, or Uber, again,
• 6:43
  Silicon Valley, like data-driven company.
• 6:46
  And if you see like the data teams in there, the layers that
• 6:49
  you have of people from the Sr. ES that they take care of your
• 6:54
  AWS account to the data platform people who are making sure that
• 6:58
  there's always capacities there for your Spark clusters to the
• 7:03
  data engineers who are making sure that the data is delivered.
• 7:07
  And then the analysts and the ML people and the data scientists
• 7:11
  on top. Like the, the guy at the bottom
• 7:14
  has no idea how like his work is used actually to make a decision
• 7:20
  of how like to improve like the revenue that you have.
• 7:22
  Like, right. And that's like part of like
• 7:25
  what Yoni was saying also that the model that these
• 7:28
  technologies were built and the requirements in terms of like
• 7:31
  the talent needed to maintain and like scale them, right?
• 7:34
  It was not sustainable if you want like to make it accessible
• 7:37
  like to everyone in the market out there, right?
• 7:40
  Contrast that to how we build apps, right?
• 7:45
  If we were building apps like SAS applications the same way
• 7:49
  that we do it with data like the industry would be, I don't know
• 7:53
  like 110th of what it is today and like the people that are
• 7:56
  working there are probably will be 2% of what is today, right.
• 8:01
  So what AI did because we started getting into the data
• 8:07
  becoming more and more of a product is that the AI came in
• 8:11
  and said is everything about data.
• 8:14
  Now whatever product we are building one way or another,
• 8:18
  like consumes data and spits out data.
• 8:21
  And it's not just the data that we're used to as like beyond
• 8:24
  said, now we have also a structured data that we can
• 8:28
  process like in ways that we couldn't do like before.
• 8:31
  And there is an opportunity there to make these technologies
• 8:34
  even like, more accessible to more people.
• 8:36
  Yeah, it's easier than ever, Yeah.
• 8:38
  Exactly. So AI kind of like accelerated,
• 8:43
  that's what I'm saying. Like it's a, it's not like it
• 8:44
  wasn't happening. Like already the industry was
• 8:47
  leaving the SAS and selling the software as a service as a way
• 8:52
  like to build value and getting more into the data is becoming
• 8:56
  the driving force of the next iteration of the industry.
• 9:01
  But AI came in and accelerated like 100 X.
• 9:05
  You mean by that exactly the leaving the SAS?
• 9:08
  Because I've heard so many different versions and
• 9:11
  viewpoints of that idea of like SAS is not the way forward.
• 9:16
  Yeah. So if we think about we need to
• 9:21
  you need to be old like me, like to do that, OK, because you have
• 9:24
  to see like the how the market developed in the past like 20
• 9:28
  years. OK, I'm revealing a little bit
• 9:29
  of my age now. So what happened with let's say
• 9:33
  the, the the past from the 2010 to 2020, right, where we had all
• 9:38
  these companies like like the sales force became like really
• 9:43
  big, the work days of the world out there, right?
• 9:48
  What was actually happening is that we took all the activities
• 9:52
  that humans are were doing and we tried to automate them using
• 9:57
  like software and deliver that software over the cloud, right?
• 10:01
  The cloud was important because the cloud allowed for
• 10:05
  efficiencies like in the financial point of view that
• 10:08
  like you couldn't do it like with other delivery methods,
• 10:11
  right? And that helped like grow a lot
• 10:13
  and commoditize the software at the end, right?
• 10:17
  So we are using taxes, right? You go out there.
• 10:21
  Hey, hello. Or call someone to come and pick
• 10:24
  you up. No, now you can use an app like
• 10:26
  to do that, right? So we kind of created platforms
• 10:32
  or Esops, for example, have like Sopify, right?
• 10:37
  We created for each activity that like humans do, like in
• 10:39
  business and their personal life software platforms to make this
• 10:44
  process like more efficient, right?
• 10:47
  And that was like delivered like through SaaS, we kind of
• 10:50
  saturated that. I mean, like, OK, I don't think
• 10:53
  there's like much left out there that we haven't turned into like
• 10:57
  a software platform one way or another, right?
• 11:00
  So if one like to keep the growth of the industry the way
• 11:05
  that was like, OK, what's next? Now the magic of like turning
• 11:09
  every process into a software is that you catch a lot of data,
• 11:13
  you capture a lot of data, right?
• 11:15
  And this data is like sitting somewhere.
• 11:18
  And now the question is like, what do you do with this data?
• 11:21
  So the first thing do BI reporting, right?
• 11:26
  Like do your financials like BI, by the way, like for people
• 11:29
  don't know, like the first owner in the, in the business of BI
• 11:33
  was like the CFO because it was like primarily the, the first
• 11:36
  need was to go and build financial reporting, right?
• 11:40
  We came into like, OK, now let's do marketing reporting, let's do
• 11:43
  sales report, blah, blah, blah, like all that stuff.
• 11:45
  And then well, you know what, now that we have all the
• 11:47
  interactions of these people with our products, maybe we can
• 11:50
  build recommenders and now we can automate and send like an
• 11:53
  e-mail and be like, hey, you know what?
• 11:54
  I saw that you were looking into that stuff.
• 11:57
  I found this one that might be interested, right?
• 11:59
  And this can drive literally the behaviour of the world.
• 12:02
  Go and buy more. That's based on data, right?
• 12:06
  So we have all this data and we have also a lot of fun
• 12:08
  structured data. We have all this customer
• 12:10
  supports that people are like calling like thousands and
• 12:13
  thousands of like people to tell like their issues and all that
• 12:16
  stuff. But we couldn't like work on
• 12:20
  that before or at least like easily enough, right?
• 12:26
  So there is the opportunity out there to take all this data.
• 12:28
  It's like an untapped opportunity to turn more and
• 12:31
  more value out of that. Now the question is, OK, can we
• 12:35
  do it with the technologies that we had?
• 12:37
  And actually it was like, it's kind of hard.
• 12:40
  And then AI came in, like actually LLMS came in, right?
• 12:44
  And they offered a number of tools and most importantly, my
• 12:49
  opinion to a much, much broader audience out there than ever
• 12:55
  before. So people now they can work on,
• 13:00
  do things with their data that previously was like really,
• 13:03
  really hard like to do Like you had to be Lyft or Uber or like
• 13:08
  Apple. Yeah, I like that highlighting
• 13:11
  how mature the companies, data teams and justice overall team
• 13:17
  and vision of how to use their data needed to be before AI
• 13:22
  came. And then you relate that to app
• 13:25
  builders and how simple it is to build an app versus how simple
• 13:29
  it is to put a recommender system into production.
• 13:31
  Yeah, yeah. And I can give an example for
• 13:36
  people to understand how big of a gap there is between the
• 13:40
  maturity that let's say the app building part of the industry
• 13:44
  hasn't the data one, right? So imagine like today, you have
• 13:50
  a front end engineer built like the front end.
• 13:52
  They work in React, blah, blah, blah.
• 13:53
  Whatever they are doing there, you have a back end engineer,
• 13:56
  they're doing their stuff like with Firebase or like whatever.
• 14:01
  And you have an app that you can put out there right now these
• 14:04
  two guys, they're they are working each one of them like is
• 14:07
  working on their own thing, right?
• 14:08
  And somehow like they can merge their work and have an app like
• 14:12
  working at the end now. And you 2 probably have like
• 14:17
  more experience than me on that. Try to contrast that with what
• 14:22
  has happened, like with the data teams.
• 14:23
  Like the data scientist is going to build a notebook.
• 14:26
  And what happens next? You hand this notebook to the
• 14:29
  engineer and what is the data engineer doing?
• 14:32
  Rewriting the code to put it into production.
• 14:34
  Now take this and put it into like app building and how many
• 14:39
  sales force we would have if we had like to literally the front
• 14:42
  end developer hands the code to the back end developer and the
• 14:45
  back end developer going to rewrite the thing in order like
• 14:47
  to release that, right. So that's like the the, the
• 14:51
  difference between like the the level of maturity that let's say
• 14:55
  the one side of the industry has convert like to to the other
• 14:58
  one. Yeah, and AI is that great
• 14:59
  equalizer. Actually, it's the catalyst.
• 15:05
  I would say it's not there. And I think like a big problem
• 15:08
  that we are seeing today with AI is that getting things into
• 15:12
  production is like really, really hard.
• 15:14
  It's great like for demos, like we can build demos, very
• 15:17
  impressive demos, but getting into production is really hard.
• 15:20
  And the reason for that is because the tooling that we have
• 15:22
  and the engineering practices that we have are not there yet
• 15:27
  to deliver, let's say, the same capacity of delivering software
• 15:32
  with the quality that we need as we do like with building
• 15:35
  applications. It's it's funny, yeah, you
• 15:41
  mentioned that and how you're, you now have a new persona that
• 15:47
  is able to access that. So that is that catalyst and
• 15:50
  it's, it's like the market is demanding that you have the ease
• 15:54
  of use because you have all these back end and front end
• 15:56
  engineers who are able to use AI and they're coming in and then
• 16:01
  they're like, OK, I got something 80% there.
• 16:03
  I can show a demo, I'm good. But then you're like.
• 16:07
  So let's let's use this in our app.
• 16:10
  Yeah. And I think part of that is like
• 16:12
  kind of what you're describing is if we take a look at the AI
• 16:15
  journey that companies go through, right, Chat interfaces,
• 16:19
  that's the primary thing that came out.
• 16:21
  That is still, I think the dominant way that people
• 16:24
  interface with LMS and AI, right?
• 16:27
  And so you have, let's say, interactive.
• 16:28
  Yeah, that's where people tend to start building apps, building
• 16:31
  AI products, right? Go and grab Blank Chain or they
• 16:34
  go and grab a llama index and and really expect either have
• 16:39
  like a human in the loop or, you know, interactive on the other
• 16:42
  end, like an agent waiting for like an AI response that comes
• 16:45
  from these LMS, right? And I think that's where
• 16:49
  companies tend to start primarily because that's where
• 16:51
  most of the tooling exists now. Then they tried, they already
• 16:56
  start seeing value, right? OK, we're building this product,
• 16:59
  it's adding some value, but now I really want to go and scale
• 17:03
  this. Like how do I now do AI at
• 17:05
  scale? How do I actually build a
• 17:07
  product in production? And the challenge there is the
• 17:11
  nature of all these models are non deterministic, right?
• 17:15
  And so a lot of the concepts that were once used for data
• 17:18
  teams with structured and tabular data don't necessarily
• 17:21
  apply for unstructured data because of the nature of non
• 17:24
  deterministic. And So what, what, what our
• 17:27
  theory is, what our thesis is, is that we really want to create
• 17:31
  and help companies and build a platform that can help them
• 17:35
  create deterministic pipelines with, with using very familiar
• 17:40
  interfaces that they're already used to in concepts on top of
• 17:43
  non deterministic models, for example, right.
• 17:46
  And that's where a lot of the challenges, that's what what
• 17:48
  really the problem set that we're excited about helping
• 17:51
  teams do is how do you how do you build a lot of clarity and a
• 17:55
  lot of stability and reliability and help companies actually take
• 17:59
  that next step? Where I think about how do we
• 18:00
  scale our AI products and how do we put these pipelines into
• 18:04
  production in a resilient in a stable fashion?
• 18:07
  And that's where a lot of the tooling is kind of missing,
• 18:09
  right? They kind of start doing the one
• 18:11
  off interactive AI. And then when it comes to
• 18:13
  putting this actually in production at scale, you know,
• 18:17
  you're, you're, you're talking about concepts like context
• 18:19
  windows and tokenizing and partitioning and chunking, all
• 18:22
  things that are new muscles that data teams really need to build
• 18:26
  to be able to run these things reliably in production.
• 18:28
  Even just is Anthropic working right now is the API.
• 18:34
  Yeah, exactly. So what are the latencies like,
• 18:37
  right. And then all the new versions of
• 18:39
  the models coming and introduce new things for cost
• 18:41
  optimization, right? Inference is expensive, right?
• 18:44
  Time to 1st token. There's so many.
• 18:46
  That's so true. There's so many new metrics that
• 18:48
  you're looking at that you previously had not been exposed
• 18:52
  to. And the platforms that exist out
• 18:54
  there don't treat these as first class citizens, which means that
• 18:58
  teams that have great engineering teams are DIY ING
• 19:02
  it. They're saying, OK, how can I go
• 19:04
  and build something around these constraints that I have and
• 19:07
  around these properties of non deterministic models to try and
• 19:11
  simulate how we're currently interacting with our structured
• 19:14
  and tabular pipelines, right. So they'll, if you're like a
• 19:16
  great Spark shop, for example, you'll go and write these
• 19:19
  complex Udfs, which are brittle, hard to maintain and you know,
• 19:24
  on top of Spark and not taking advantage of all of any of the
• 19:27
  performance of Spark has to write offer because it's not
• 19:29
  running in distributed fashion. Or you'll go and build some
• 19:33
  complex logic on top of Lambda functions where you're trying to
• 19:36
  hit the APIs of these models. But you know, lots of
• 19:40
  engineering cost goes into that that's hard to maintain.
• 19:43
  And so teams are trying to piece things together to try and help
• 19:47
  them run AI at scale in production.
• 19:51
  And so that's where we see that there's a really nice
• 19:53
  opportunities. How do you build a lot of these
• 19:54
  properties really natively into an engine in a platform that
• 19:58
  helps companies go and and scale their AI work list and as much
• 20:02
  as everyone? And I think it's true.
• 20:04
  Like agents are here and you know. 2025 is a year of ages.
• 20:09
  We're going to see a lot more of these come into production.
• 20:11
  But I also think you have a lot of teams that are starting to
• 20:14
  think about, OK, like, you know, I'm collecting all these
• 20:17
  transcripts, right? I have, you know, a call center
• 20:19
  of 60 people that are doing thousands of calls per day.
• 20:23
  And I want to perform semantic analytics on it, right?
• 20:26
  And our view is inferences, the new transform, right?
• 20:30
  It can be used very much in the same way as what teams are used
• 20:35
  to when they're building struck pipelines for structure data,
• 20:38
  right? It's a very powerful form of
• 20:40
  transform. So if you're trying to do
• 20:42
  semantic analytics at scale where there's no human in the
• 20:45
  loop, how do we enable teams to really take advantage of this
• 20:49
  new powerful form of transform that's there?
• 20:51
  I like that. I've heard that before from the
• 20:54
  head of AI at Wise. He was saying, you know, we
• 20:57
  should be thinking about these different LLM calls as just a
• 21:02
  way to take structured unstructured data and turn it
• 21:05
  into structured data. That's what that's what you see,
• 21:07
  that's what we're trying to to help teams really do.
• 21:10
  And we see they can think that there's teams are doing that
• 21:12
  already on their own right. But can you build something that
• 21:16
  treats this as really the first class problem that you're trying
• 21:19
  to solve? And let's take a moment real
• 21:23
  fast to highlight the difficulty of that scale in production.
• 21:29
  Because I remember I was talking to my buddy who works at Decagon
• 21:32
  and he was saying, you know, to get the agents working for the
• 21:37
  customer service stuff. That's not really the hard part.
• 21:41
  We had a working demo up and we were able to sell within two
• 21:44
  weeks of when we generated the idea later.
• 21:48
  What's really hard and what kept us up at night is how do we do
• 21:51
  that for a company that is receiving thousands of customer
• 21:56
  service requests per minute? Yeah.
• 21:59
  And that's where batch inference becomes really relevant, right,
• 22:03
  And really prominent there. But then also what you were
• 22:06
  saying is like, you're taking all of these transcripts at
• 22:09
  scale, and there's a lot of things that you need to think
• 22:12
  about that are properties of, let's say, the AI models, right?
• 22:15
  So the context windows, right? So if I have this transcript, is
• 22:18
  it enough for me to take each individual message and pass to
• 22:21
  the LLM? No, because it turns out that
• 22:23
  you need to have the context of the entire conversation.
• 22:26
  So are you going and applying arbitrary buffer around each
• 22:29
  individual message? Well, that's going to have you
• 22:31
  increase your input token count, which is going to make things
• 22:34
  more expensive. So there's a lot of these
• 22:36
  nuances and complexities that when you're trying to run things
• 22:40
  at scale, you need to you need to give people the expressivity
• 22:44
  to be able to one experiment before pushing to production.
• 22:47
  So how do we build this in kind of a notebook environment, test
• 22:51
  out different prompts that we have to make sure that the data
• 22:54
  quality is coming back. But then also what you're saying
• 22:56
  is true is like, how do we create structure of the
• 22:58
  unstructured, right? So really long transcripts,
• 23:01
  blobs of text into nice neat structured tables, right?
• 23:05
  And so, and, and that's where the true power lies, because if
• 23:09
  you can take things and create structure out of it, now there's
• 23:12
  a lot more that you can do to control input token count and,
• 23:16
  and you can do a lot more on understanding what the context
• 23:19
  window and that you want to send to the LLM.
• 23:21
  And so, and then you get into things like rate limiting and
• 23:24
  like model latencies and model cascading's and things of that
• 23:28
  sort that are also really important.
• 23:29
  You're running things at scale. It does feel like to, if you're
• 23:33
  doing that kind of stuff in Spark, you're over engineering
• 23:36
  it. Yeah, yeah, 100%.
• 23:38
  And, and, and if you're building it on Spark, right, I think it,
• 23:43
  it just the architecture that was built and it's a brilliant,
• 23:48
  super powerful platform, right. But it wasn't built with
• 23:52
  unstructured data in mind, right?
• 23:54
  And so and so you're not getting a lot of the guarantees that you
• 23:58
  can get if you had a platform that was built for and around
• 24:02
  unstructured data processing. I also remember there was this
• 24:05
  guy Zach that was putting an agent platform into his company,
• 24:11
  and he was talking about how one of the things that they did was
• 24:16
  they were trying to allow everyone at the company to build
• 24:19
  their own agents for whatever the use case was.
• 24:21
  And I know that's a common thing that most people are dealing
• 24:24
  with. They're saying we know that each
• 24:27
  individual department needs agents, and we know that these
• 24:32
  agents are going to be better served if the subject matter
• 24:36
  expert builds them. So how do we create a platform
• 24:39
  that can allow folks to build their own agents?
• 24:43
  And one thing that he did was he created a metric for when
• 24:49
  someone is building an agent so that, hey, if you put this into
• 24:52
  production and wherever you're going to expose it, the expected
• 24:57
  traffic is going to be this much and this is the expected cost.
• 25:01
  And so, so that's a fascinating piece.
• 25:04
  But then the other side of that is, I wonder how much you have
• 25:08
  been encountering folks like the people that we talked to at
• 25:12
  Process that say we don't really look at cost in the beginning,
• 25:16
  we just want to make something that is working.
• 25:19
  Yeah. So is it possible?
• 25:20
  Then we go into how do we optimize it?
• 25:24
  Can we delight the users? And then can we actually like
• 25:27
  optimize that? Yeah.
• 25:29
  That's a great point because there is also a hidden
• 25:33
  complexity that we didn't have before.
• 25:35
  Again, with with like the thing with AI is that we really have
• 25:39
  like to rethink how we're building software.
• 25:42
  So one of the things that AI enabled is that the people who
• 25:49
  are actually the domain experts now they can they have to be
• 25:53
  part of the techno like building the technology itself.
• 25:56
  Like if you think how we were like building software before,
• 25:58
  right, we would get OK, our customer, we have like a product
• 26:02
  guy. We have a persona.
• 26:05
  Let's say like our persona is like sales people.
• 26:07
  We try to understand what the sales people need.
• 26:10
  Then we'll create some PRDS. We'll go to the engineering team
• 26:14
  like, hey, now you have to build that.
• 26:15
  How many story points? Yeah.
• 26:17
  How many story points roll roll out like the software give it to
• 26:22
  sales and the sales people somehow it will work OK, but now
• 26:29
  that's not enough because this behavior of the software depends
• 26:33
  on the salesperson directly, right?
• 26:38
  So how do we involve that person right in the process is like
• 26:44
  something that I don't think we have figured it out yet.
• 26:50
  And also how we, if we let like the engineers keep doing
• 26:55
  engineering because we still need engineering, right?
• 26:58
  Like it's not like we can't live in a world where everything is
• 27:02
  like kind of random at the end of the day.
• 27:04
  And the salespeople at the end of the day, they are still
• 27:07
  salespeople. They get paid for selling shit,
• 27:09
  not for building software, right?
• 27:12
  So. Yeah, yeah, yeah, it's fine.
• 27:15
  So. It's not like, OK, you're not
• 27:17
  getting a bonus this month because your prompt was not
• 27:19
  good, dude. Like we can't do that, right?
• 27:23
  So this is like I think like a big challenge especially for the
• 27:28
  people who are building like more customer facing products,
• 27:32
  but. Thankfully the sales people
• 27:33
  don't have to write sequel though, right?
• 27:35
  They can just write prompts, which is like and like.
• 27:39
  That's easier for them to be able to go and let's say iterate
• 27:42
  with than having to go through and try and do like the old.
• 27:45
  But but also it brings up the point of this is built for
• 27:49
  engineers, it's not built for sales folks.
• 27:51
  And so the fact that, like you were saying, playing around in a
• 27:55
  Jupiter notebook, trying to give that to a salesperson is not
• 27:59
  going to work. How do you create an environment
• 28:00
  that someone is able to natively go in and it's it's like do you
• 28:07
  focus on the lowest common denominator of someone who is
• 28:10
  technically apartment at being able to do something or are you
• 28:13
  trying to push for how do we enable our power users?
• 28:18
  The role of like the heads of AI that we see now coming in and
• 28:22
  becoming very common at a lot of companies.
• 28:24
  Most companies that we talked to have a head of AI at this point,
• 28:27
  right? And really, I think the
• 28:29
  interesting part about that role is doing exactly what you're
• 28:32
  talking about is trying to bridge that gap.
• 28:34
  They're essentially like the AI Sherpas within the company,
• 28:37
  right? If you think about it, they're
• 28:39
  going to like the head of product marketing, they're going
• 28:42
  to the head of sales, they're going to all the other business
• 28:44
  functions there. And they're saying, hey, we, I,
• 28:47
  I can, I'm the gatekeeper for AI.
• 28:49
  What do you want to try and build for this?
• 28:51
  How can we try and add value to your teams to be able to
• 28:54
  leverage AI? And so the head of sales will
• 28:56
  say, hey, look like we're losing out on a lot of deals and all of
• 29:01
  the information we have is in this sales slack channel that we
• 29:04
  have. There's a lot of data there in
• 29:06
  context around why did we not hit our numbers over the last
• 29:09
  week, over the last month and quarter, right?
• 29:12
  Can we build, can you build me some pipelines that are going to
• 29:15
  go through and leveraging LLMS, try and identify the patterns in
• 29:20
  there so that we can then go and improve and increase conversion
• 29:23
  rates, whatever it is? If I know this sales team, I
• 29:25
  know the answer and it's probably marketing.
• 29:27
  Marketing fucking sucks. But then you go to the marketing
• 29:30
  guys and they're like, OK, like look at the sales guys.
• 29:33
  With products, with marketing, they're like we're.
• 29:35
  Giving them qualified leads, right, That's kind of the, the,
• 29:38
  the thing, right? And, and, and I think there are,
• 29:42
  there will be more advances there where it's like, OK, how
• 29:44
  do we get, but at the end of the day, like, do you want your head
• 29:47
  of sales to be going in and trying to work in a platform and
• 29:51
  experimenting with different prompts?
• 29:53
  I think there's a balance there, right?
• 29:56
  But that's what these I think heads of AI are trying to do is
• 29:59
  like create the AI road map based on all the feedback
• 30:02
  they're getting from other like business functions within the
• 30:05
  company, which is like a really interesting.
• 30:06
  Sharp is a great term. So there is a the UX part, which
• 30:10
  is like a big thing that needs to be like figured out, but
• 30:13
  there is also the like the platform, like the
• 30:15
  infrastructure side of things, right?
• 30:17
  And I think and going back a little bit like to like why you
• 30:24
  shouldn't do that like on Spark, for example, right?
• 30:27
  The thing with with AI and LLMS is that the workloads changed
• 30:33
  their characteristics like dramatically because of LLMS and
• 30:38
  inference and GPUs. So in the past, again, 1213
• 30:44
  years, everything was like pretty much CPU bound.
• 30:48
  Spark is like an amazing tool if you want to crunch numbers and
• 30:51
  make sure that your CPU's are always like operating at 100%,
• 30:55
  like that's what they're trying to do first and second, OK, how
• 30:59
  we move data around because how we shuffle because we need to
• 31:03
  move from one server to the other.
• 31:05
  And if you have thousands of them, then how do you make this
• 31:07
  reliably? So if something breaks, we can
• 31:09
  resume. Now put like an LLM in the
• 31:13
  equation, CPU doesn't matter anymore because your CPU is
• 31:18
  going to sit idle there waiting for the GPU to return like a
• 31:21
  result, right? So from ACPU bound workload, we
• 31:26
  go into like more of an IO bound workload.
• 31:30
  Then OK, you're building your UDF, you're running these
• 31:34
  things, but guess what? There's zero reliability when
• 31:38
  you're talking to GPUs, right? Like a GPU because the systems
• 31:43
  are not mature yet. We're kind of I would say how
• 31:47
  kind of like the Internet was like before 2000 like literally
• 31:51
  like back then, you know like you would connect on the
• 31:53
  Internet and if your mom was like picking up the phone, like
• 31:55
  you would lose your connection, right?
• 31:57
  Like does this happen today? No, right.
• 32:00
  But that kind of what happens like you're.
• 32:01
  Aging yourself again, it's. OK, it's OK.
• 32:04
  Like we can discriminate on it. We're in San Francisco.
• 32:11
  But then with the LLM, like OK, you sent a request you sent,
• 32:16
  let's say, I don't know, let's say you have, you're using
• 32:20
  Jamini that has like 1,000,000 tokens, like context window.
• 32:23
  You said like 1,000,000 tokens there, right?
• 32:25
  It starts doing its magic there and in the middle of that it
• 32:31
  breaks. What happens?
• 32:33
  You can't resume that right? There's no that.
• 32:35
  Literally happened to me yesterday.
• 32:37
  I did that exact same thing. I'm pairing up all of the
• 32:40
  attendees for the event tomorrow and I'm sending an e-mail to
• 32:44
  them saying hey, you should know each other because I think it
• 32:47
  would be great. And so I asked.
• 32:49
  They're trying. To trying to get a new new
• 32:50
  company built I see. Yeah, I'm trying to be that
• 32:52
  matchmaker again, exactly that. And so I'm, I'm actually Gemini
• 32:57
  without me prompting it to. It says why it paired up each
• 33:00
  one, but it's taking a long time.
• 33:03
  And halfway through I said, all right, well, I'm gonna go get
• 33:06
  lunch and it should be done by the time I get back.
• 33:09
  I got back and the whole thing was just not there anymore.
• 33:13
  It magically disappeared. So I had to prompt it again and
• 33:15
  then ask it to create it again. And you're already you've
• 33:19
  already wasted cost on input token for the first time, that
• 33:21
  didn't work right. Even before you reach the cost
• 33:24
  thing, it's like the reliability thing, right?
• 33:26
  Because think that now you are doing this like 12 hours before
• 33:32
  your event and it breaks and you can't send the emails.
• 33:36
  Like you're not going to be in a very happy position, right?
• 33:39
  Imagine you're AT&T and you are going to process all the
• 33:44
  transcripts of the previous day, right, to create tickets for
• 33:47
  your engineers for the next day, like to work.
• 33:50
  And we are talking about like probably 10s of thousands of
• 33:53
  hours of transcripts that you have to do there.
• 33:55
  And that breaks, right? Where's the SLA?
• 33:57
  There's no SLA. There's no SLA right now.
• 34:00
  And data teams are used to Slas, right?
• 34:02
  That's how that's how they get rated and qualified.
• 34:05
  And like, are they doing a good job?
• 34:07
  Are they hitting their Slas right?
• 34:08
  Are these pipelines getting executed in the amount of time
• 34:11
  that they need to throw LLMS into the loop?
• 34:13
  It's a whole, it's. The Wild West.
• 34:16
  Yeah, it's the Wild West, right? And.
• 34:17
  Exactly. So you have this super reliable
• 34:20
  CPU focused technology that is Spark.
• 34:25
  You put LLMS in the equation there and it doesn't work
• 34:29
  anymore like the reliability that you expect from something
• 34:33
  like spark, it's not going to work and it's not it's fault,
• 34:36
  right or your clusters are going to just sitting there and just
• 34:41
  waiting for the GPUs to return something back.
• 34:44
  And the reason I'm saying that is because people might say, OK,
• 34:48
  why not just go and like build on top of that, right, For our
• 34:54
  like a lamps. And the reason is that because
• 34:57
  very, very soon and again, let's say you have infinite money,
• 35:01
  like you don't have a problem with that, right?
• 35:03
  You will have an extremely hard time creating reliable systems
• 35:07
  that will reliably deliver value to your company.
• 35:11
  And the moment that that happens, people will lose faith
• 35:14
  in the technology itself. And we see that like with AI,
• 35:17
  right? Like people do get the lose like
• 35:19
  faith because they don't care at the end of the day.
• 35:21
  Like they're not doing it just for the technology.
• 35:23
  Like the technology is like making their lives harder.
• 35:25
  Like why they would care, right? Not everyone is like, you know,
• 35:29
  like a tech junkie in like Silicon Valley that's have like
• 35:33
  to work with towards like advanced general intelligence,
• 35:38
  right? Yeah, I think it's a it's an
• 35:39
  important thing too, right? As you talk about that, that's
• 35:42
  that, that 70% of AI projects never make it into production.
• 35:45
  Everyone that's building or like trying to innovate in the AI
• 35:49
  space always famously claims that thing.
• 35:51
  And then they fit in somewhere under that umbrella for why the
• 35:54
  problem that they're trying to solve is the main reason for
• 35:57
  that. And, and, and it's all based on
• 36:01
  conversations, right, that they're having with these heads
• 36:03
  of a is and teams to try and figure out and, and pinhole what
• 36:06
  the problem is. And there are multiple layers to
• 36:08
  that, right? The one that we're very focused
• 36:11
  on is taking it and running AI at scale in production is a very
• 36:15
  hard unsolved problem that teams just really need to struggle
• 36:20
  with because they're used to the world of things working
• 36:24
  relatively well, right? They don't always hit their SL
• 36:26
  as, but when they have these pipelines that are running
• 36:28
  across large data sets, you know, they have on calls, they
• 36:32
  have processes for being able to run these, they have retries,
• 36:35
  they have orchestrators like Airflow that are kind of good at
• 36:37
  helping run that retry log logic and build all of that.
• 36:42
  And that's where we think that there's AI, like the new
• 36:47
  workloads that are introduced and the new paradigm that we
• 36:49
  live in now, the AI, the infrastructure behind it, needs
• 36:52
  to be rethought. Well, it's funny you mentioned
• 36:53
  that because I remember back in the day, my friend Diego was
• 36:57
  telling me about how he felt like there was going to be this
• 37:01
  new term for folks that focused on reliability, but specifically
• 37:06
  for the ML systems. And he was like, maybe we could
• 37:09
  coin it as the MLRE instead of the SRE.
• 37:13
  It's the MLRE. But now it feels like what
• 37:16
  you're talking about is the Aire, the AI reliability
• 37:19
  engineer, who is going to be 100% heads down focused on can
• 37:24
  we make this reliable? And that goes into all of these
• 37:29
  new metrics that you're talking about.
• 37:30
  It's not just does. Can we look at the logs and
• 37:34
  traces and pipe it into Datadog? Or can we like have that kind of
• 37:39
  analysis? Yeah, They want to be able to
• 37:40
  feel confident around their pipelines and be able to even
• 37:45
  think the next look, can we put an SLA on this, right.
• 37:48
  And and given the current tooling, it's very, very hard to
• 37:51
  do that based on the ergonomics and the the tooling that's there
• 37:56
  in the platform that exists for working with these models,
• 37:58
  right. And so that's the AI
• 38:00
  infrastructure kind of opportunity that we're thinking
• 38:02
  about. Something about like the roles
• 38:07
  that exist already because I think that there is a gap there.
• 38:11
  Someone has like to fill this gap.
• 38:12
  I don't know if it's going to be the like AI engineer.
• 38:15
  Although my experience so far with AI engineers is I feel like
• 38:19
  they're coming more from the data scientist side, which is
• 38:24
  great. Like you need like people who
• 38:25
  can like model and understand how to work with models and at
• 38:30
  the end of the day, like create something that delivers what has
• 38:34
  to be delivered there. But I think there's like a huge
• 38:38
  opportunity there for people, especially from the data
• 38:41
  engineering background and the metal engineering engineering
• 38:44
  background to step in and actually take this role.
• 38:47
  Because at the end of the day, their reason that they existed
• 38:51
  was to had reliability to working with data because
• 38:55
  working with data was always an unreliable business.
• 38:58
  It was always really, really hard, right?
• 39:01
  And the job of these people were to make sure that everything is
• 39:05
  delivered on time and is with the quality that we need.
• 39:08
  We have Slas, we have all these things.
• 39:11
  And if you want to understand like a group of people, you have
• 39:14
  to look into the language that they are using, right?
• 39:17
  And one of the most commonly used terms for data engineers is
• 39:22
  independency. Independency is like the
• 39:24
  attribute of a system that if you put the same input, the
• 39:27
  output will always be the same, right?
• 39:30
  That's great because when you have that, you can ensure that
• 39:33
  there is a reliable system there.
• 39:35
  Right Now that says something about data engineers that they
• 39:43
  literally breathe and leave and exist for reliability at the end
• 39:47
  of the day. The same with ML engineers.
• 39:49
  But the ML engineers, I think they have an added advantage
• 39:53
  that they've already had to take into account that we are working
• 39:57
  with systems that they are not deterministic, right and.
• 40:03
  LLMS get this like to think to the extreme.
• 40:05
  So I think these two groups of people, they have like an
• 40:09
  amazing opportunity to actually transform themselves like into
• 40:14
  something that will be extremely important for the future of the
• 40:18
  industry. And if they don't do it, someone
• 40:20
  else will do it right? Like it might be like.
• 40:22
  The SRE, yeah, I was thinking 100%, it's a lot of folks that
• 40:26
  were in the ML OPS community in the beginning came from that SRE
• 40:30
  background and they were tasked with figuring out the
• 40:33
  reliability of the ML systems and the ML platform.
• 40:36
  And then you had this, the rise of the platform engineer.
• 40:39
  And a lot of those folks were Sr.
• 40:42
  ES that were rebranded and you have the platform engineer.
• 40:46
  Now you are probably going to start seeing more and more of
• 40:49
  these AI platforms. And one of the jobs that the AI
• 40:54
  platform engineer is going to be tasked with is exactly that.
• 40:57
  How can we make sure that whoever's building with AI can
• 41:02
  do that confidently? Yeah.
• 41:04
  And I hear a lot from people especially, like, from the data
• 41:07
  engineering world because, yeah, like, if your whole existence is
• 41:11
  around determinism, right? Like when something comes that's
• 41:14
  like, so different to what you're doing, like your initial
• 41:19
  reactions like, to reject it, right?
• 41:21
  And that's like the biggest, I think danger for them right now.
• 41:25
  Like you see a lot of like rejection.
• 41:27
  It's like, oh, we're dead engineers.
• 41:29
  Like leave us alone without like LLM stuff.
• 41:32
  We're going to do our thing, you crazy AI engineer, go do
• 41:35
  whatever you want. I don't want like to know about
• 41:37
  it. No, because if you go like you
• 41:43
  don't focus on that and you are OK to feel a little bit
• 41:48
  uncomfortable, you have a tremendous amount of value that
• 41:51
  you can deliver because reliability is what is literally
• 41:53
  like what is missing to turn like LLMS and AI into like what
• 42:00
  they are promised to be. So I think there is like for
• 42:04
  both data engineers and the metal engineers like a huge
• 42:06
  opportunity here. They need to link.
• 42:09
  I mean, like it's not the existing tooling enough for that
• 42:13
  stuff, but the tooling is different conversation.
• 42:17
  They are not, it's not their job to build the tooling.
• 42:20
  The industry should build the tooling.
• 42:22
  But the most important thing is the mindset that they bring and
• 42:27
  the experience that they bring. And that's something that not
• 42:30
  tooling can do, right. So these people have like
• 42:32
  literally sitting on like gold, but they have like to do
• 42:35
  something without otherwise they will meet like a huge, huge
• 42:38
  opportunity in my opinion. Yeah, I've heard some people
• 42:42
  talk about how they can't connect the dots.
• 42:46
  You hear everyone banging and screaming from the rooftops on
• 42:50
  how AI is only as good as the data that you give it.
• 42:53
  And garbage in, garbage out. These are like the tropes that
• 42:57
  are so common. And then I saw someone say, but
• 43:03
  explain to me how that's possible because right now I
• 43:06
  just go in and I give a prompt and there's no data that's going
• 43:10
  into that. That's just the prompt.
• 43:12
  And so I was trying to put two and two together to really
• 43:15
  encapsulate why it is like that. And on one hand you have me just
• 43:21
  going in doing one off tasks with ChatGPT or Gemini.
• 43:25
  And that's used as a bit of a productivity or I'm talking to
• 43:30
  it, I'm trying to learn something new, I'm trying to
• 43:32
  understand something or I'm using it more like a browser.
• 43:35
  I'm asking it to tell me these different things.
• 43:38
  That's not necessarily a data product, but then you have
• 43:43
  products that the company uses and like you were talking about
• 43:46
  with the support and all of the data that's going to be going
• 43:51
  into the context window, there is not something that you're
• 43:54
  doing 1 offs with. That's something that should be
• 43:56
  very operationalized. Yeah.
• 43:59
  So, and so here's like a, an example that we use one of our
• 44:02
  early design partners to help kind of crystallize this to like
• 44:05
  for a use case, right. It's like we talked about it.
• 44:08
  This team has, let's say 60 call center folks.
• 44:12
  It's for all intense purposes an insurance tech company, right?
• 44:16
  And whenever you go and you get a new policy for insurance, you
• 44:21
  get this thing called a deck page is what they call it in the
• 44:24
  it's a declaration space. So it's like a summary of all
• 44:27
  your coverages and all of your policies there.
• 44:30
  Now, the problem here is that if a call center representative
• 44:35
  misrepresents what's in the decks page.
• 44:37
  So let's say there was someone, this is an example at Costas
• 44:42
  recently built out is like, you know, I have roadside assistance
• 44:47
  and it says in the deck page, it's only for 15 miles.
• 44:51
  But the call center representative during the call,
• 44:53
  the customer tells them that it's for 50 miles, right?
• 44:57
  That's a liability that they're then taking on and could
• 44:59
  potentially get sued for that because they misrepresented what
• 45:02
  the actual policy is, right? And so how do you build these
• 45:06
  pipelines that are going to let's say, and this is where the
• 45:08
  data quality portion comes in too.
• 45:10
  You want to have the the the declarations paid structured in
• 45:15
  a way and be able to manipulate it through using some nice like
• 45:18
  text and chunking and partitioning capabilities, along
• 45:22
  with a transcript side by side and be able to go through and
• 45:26
  and look for and filter all of the questions that were asked by
• 45:30
  the customer and then the answers that the support person
• 45:33
  gave them. And then match that to the
• 45:36
  portion of the deck page that they're actually talking about.
• 45:39
  Right. You don't really care about like
• 45:40
  the niceties, the hi, how are you?
• 45:42
  Like you don't want to feed all that shit, shit.
• 45:44
  It's OK to say into into like the LLM because going to be more
• 45:49
  expensive. You don't want to just take the
• 45:51
  whole transcript in and of itself.
• 45:53
  You want to be able to partition and Chuck in it, only send the
• 45:55
  relevant information that you need in order to understand
• 45:58
  whether the customer support agent represented what was in
• 46:02
  the declarations page accurately.
• 46:04
  So you're taking the question and the answer and you have the
• 46:07
  expressivity in the the tooling to be able to do that and then
• 46:10
  feed it. And that's where the data
• 46:12
  quality portion comes in to feed it into the LM to make a
• 46:14
  decision as to whether yes, what the customer support
• 46:17
  representative said is correct or no, if he actually
• 46:20
  misrepresented it. The actual amount of roadside
• 46:24
  assistance this customer had was 15 miles, as stated in this
• 46:27
  portion of the declaration phase, right?
• 46:29
  So then you want to create that report very quickly that there
• 46:32
  was something that they misspoke or they misrepresented and send
• 46:36
  that up and escalate it to the team that can then be proactive
• 46:39
  around handling this case so that they don't end up getting
• 46:42
  sued in court and then having to pay out for this
• 46:47
  misrepresentation. Right.
• 46:49
  And so this is an example of what teams are thinking and
• 46:53
  trying to do at scale, right? Where it's not like I'm sending
• 46:56
  this response immediately back to like an AI agent or there's
• 46:59
  like a human in the loop. Sure, you want to have somewhat
• 47:02
  real time, which is like an overloaded term, but like, you
• 47:05
  know, it's within the next few hours.
• 47:07
  It's totally fine to be able to do that.
• 47:08
  But you're getting these transcripts that are coming in,
• 47:11
  you know, thousands a day. How do you go and actually build
• 47:14
  these pipelines in a way that is trying to create some
• 47:18
  determinism on top of the ability to work with these
• 47:21
  models? And so context windows,
• 47:23
  chunking, partitioning, all these things.
• 47:25
  We need to arm engineers, AI engineers, data engineers with
• 47:29
  the ability to actually build these in a robust manner and
• 47:33
  then give them some of the guarantees like Kosis is talking
• 47:35
  about that they're already used to, right?
• 47:38
  Like I need each transcript, the medium time to have it being
• 47:42
  reviewed needs to be or the, the meantime to have it reviewed
• 47:46
  needs to be 3 hours, right? And so be able to give these
• 47:49
  teams those guarantees that within three hours of a customer
• 47:53
  conversation happening, we'll know if the representative did
• 47:56
  well or if we need to go and, and, and fix things on the back
• 48:00
  end, right. Yeah, I wanted to add something
• 48:03
  here. There's always data like you
• 48:06
  can't say that there's no data even if you want to keep it just
• 48:11
  to the oh, but I'm asking a question to the LLM about, I
• 48:15
  don't know like how to change like.
• 48:18
  My baby's diaper. Yeah, right.
• 48:20
  In a proper way, yeah. Like.
• 48:23
  Your prompt is the data. Actually the structure of the
• 48:28
  whole dialogue on its own, Like it's important just because of
• 48:33
  how LLMS work, right? Like LLMS, like you take also
• 48:38
  like the previous conversations that you said, like you feed it
• 48:40
  back there, like, so you create data.
• 48:41
  How do you feed to that? The LLM itself is built on data,
• 48:46
  right? And then anything that is, let's
• 48:50
  say outside of like the trivial things of asking what we would
• 48:53
  ask, like on Google, for example, it requires like extra
• 48:58
  data. Like there is a reason that
• 49:01
  tools are becoming so important. Like we wouldn't have agents if
• 49:03
  we didn't have tools, right? A lot of the work that you are
• 49:07
  doing with tools is actually fetching data.
• 49:10
  Now it might be, let's say if you are using cursor.
• 49:14
  If cursor didn't have the context of your code base, it
• 49:18
  wouldn't help you. And what is your code base in
• 49:21
  this case? It's data, right?
• 49:23
  When you use Clouds Coder, whatever it's called and you're
• 49:28
  like, hey, like, find me the file that does this, this or
• 49:31
  that in my code base and to transit tools.
• 49:35
  What does like a find in the grip?
• 49:38
  It gets data back right, Your code base again.
• 49:42
  And the outputs of these tools are the data you see that
• 49:45
  already we are creating actually something that I think like the
• 49:49
  data engineers and the remote engineers again, would be like
• 49:51
  very familiar with, like we're building pipelines of feeding
• 49:55
  data, getting data out and use that like to the next step,
• 49:58
  blah, blah, blah, blah, blah, blah, blah.
• 49:59
  Right? The deep research functionality,
• 50:04
  it's pretty much OK, I'll search on Google or Bing, I'll get the
• 50:08
  row HTML that is returned from the queries that I sent there
• 50:12
  and I'll work on that. That's data.
• 50:13
  Again, anything non trivial using like an LLM requires data.
• 50:18
  So I think the the way that we think of like LLMS of let's say
• 50:24
  these kind of like Oracle, that like it's not it's not accurate.
• 50:28
  Like yeah, you can do it. And that's like part of what
• 50:30
  made them like so successful because it was like so easy for
• 50:34
  people to experience something by just like talking to it.
• 50:38
  But the at the end of the day, what we are doing with LLMS is
• 50:42
  that, and let's go back to the to the SAS exam, right?
• 50:50
  So what we were doing was like building software and we were
• 50:53
  forcing people to learn how to think and operate the way that
• 50:58
  the machines can do it, right? And now we changed the equation
• 51:02
  there because we made the machines be more like thinking
• 51:07
  and working like the humans do, right?
• 51:09
  That's what's like makes like so accessible.
• 51:11
  But at the end of the day, we still have a machine that has
• 51:13
  like to do something. We tell it like with natural
• 51:15
  language to do that. And they're very generic, right?
• 51:18
  They can do like many different things, but still they are going
• 51:21
  to do the vacuum, right? And everything is driven by data
• 51:24
  at the end of the day. So even if you are even in the
• 51:28
  online use case where you are just like chatting with the bot
• 51:31
  and asking about things, right, you will copy paste something,
• 51:34
  you will take a picture from somewhere and be like, hey, but
• 51:37
  the CSS here doesn't look good. Like look at this picture.
• 51:39
  That's data, right? The difference is like, OK, how
• 51:44
  they're like the things that you do just you as like Demetrios
• 51:48
  and like Gemini and your Excel seats with the attendees there.
• 51:55
  And there is the company that has to do that at scale every
• 51:59
  other day for all the new leads that they have, right.
• 52:02
  So that's different approach. You can't like do it like with
• 52:07
  chat boots anymore and that's what we are talking about like
• 52:10
  in how you put these things like in the in the production.
• 52:14
  One thing that I feel like we need to hit on if we're talking
• 52:18
  about reliability is evals and how you think about reliability
• 52:25
  in the context of evals and getting that also, like where do
• 52:30
  they fit in in your world view? One of the problems that we have
• 52:34
  is that the first iteration of evals platforms were inspired
• 52:41
  primarily from, I would say more of like the engineering that
• 52:46
  happens in the application layer.
• 52:48
  So if you think about like evals, like in the common case
• 52:52
  you have model and input and an output, OK, and that's what we
• 52:56
  care about. So you're saying, OK, I put this
• 53:00
  into this model, I take this output.
• 53:02
  Is this output what I expect like to get?
• 53:05
  Now, the problem is that in my opinion with this model is that
• 53:09
  there's a lot of context that is actually missing, right?
• 53:13
  Especially in cases where you have to invoke many different
• 53:17
  models like to achieve a goal, right?
• 53:19
  So let's take the case of processing a transcript, right?
• 53:23
  What most people do is, OK, the first part is I'm getting my
• 53:28
  audio file. I'm going to use something like
• 53:31
  whisper telling you to like a transcript that has some issues,
• 53:34
  right? Like the output, I mean, it's
• 53:36
  usually really good, but still there are things that need to be
• 53:39
  corrected. You get this big chunk of like
• 53:42
  text that you have there. And then what people do will
• 53:46
  like, OK, clean it up. Maybe use an LLM to go and fix
• 53:49
  some of these issues. So let's say something is
• 53:52
  misspelled, right? LLMS are like great, like to go
• 53:55
  and find these and like fix them.
• 53:57
  But still, they can make a mistake right now.
• 54:00
  This is like on the very, very first stage of like processing
• 54:04
  that right now, the next tape is OK, let's start like creating
• 54:08
  some summaries. So we will break it down into
• 54:10
  some pieces, create a summary for that, store the summaries.
• 54:15
  Then on another level, we'll take all these summaries, create
• 54:18
  a summary of the whole thing, and we'll end up, let's say,
• 54:22
  with the summary that you will put on your like websites when
• 54:26
  you put the podcast episode out there, right?
• 54:29
  Now, if you consider, let's say the eval just as individual
• 54:37
  steps there, right? You're having like the problem
• 54:41
  and you can't evaluate the whole pipeline of creating like going
• 54:45
  from the audio to the end result, which is the summary
• 54:49
  that you have there, right? To do that, you have to trace
• 54:52
  all the calls and you have to consider all the calls and you
• 54:56
  have to see maybe the LLM that corrected some references in
• 55:00
  there made like a huge mistake and that changed completely all
• 55:04
  the summaries, right? I'm exaggerating, but The thing
• 55:08
  is that a step at the beginning, right, can affect the result at
• 55:13
  the end. But if you take like a call in
• 55:16
  the middle, it might still be perfect, right?
• 55:19
  But the data was like wrong. So of course, like the output
• 55:22
  was like wrong. So the question is like, OK, how
• 55:24
  we can work on that? How we can build these more
• 55:28
  complicated workflows? And I think with, if you take
• 55:32
  into consideration like the like, like agents, for example,
• 55:35
  it's like even worse because like the agents can make 10s of
• 55:40
  different calls go back, like run calls again.
• 55:46
  And how do you evaluate the output at the end?
• 55:50
  Because that's what I see at the end, right?
• 55:52
  Like I ask Claude to write some code for me.
• 55:55
  It can take like a few minutes. Who knows what it's doing there.
• 56:00
  But definitely there are like many back and forths and calls
• 56:04
  like to the LLM. So what do I eval, right?
• 56:09
  So I think this is like an important things that is missing
• 56:13
  and we'll get there. But I think we need to rethink
• 56:16
  also like the infrastructure that we are using for that.
• 56:18
  Because now we're talking about like a lot of data and data that
• 56:22
  it's, it's not like a unit test. It's more of like how let's say
• 56:28
  in the observability world, we were doing like traces over like
• 56:32
  distributed systems, right? So that's what I I want to see
• 56:40
  out there and I think that is going to change a lot like how
• 56:44
  people work and how they can build actual reliable systems
• 56:48
  and incorporate and. I think as we talked about that
• 56:50
  theme of 70% of projects don't make it into production.
• 56:53
  That's one things that we hear from the AI leaders out there
• 56:56
  right is great. I built a lot you're you tell me
• 57:00
  inference is a new transform, right?
• 57:01
  So now I'm counting on the results of these pipelines to be
• 57:05
  mission critical. They're making business
• 57:07
  decisions for me. So so they say, how do I know
• 57:11
  that this pipeline, this multi stage pipeline, like COSAS is
• 57:14
  saying this multi stage pipeline that's going through a bunch of
• 57:16
  different inputs and outputs into LLMS ended up making the
• 57:20
  right decision for me, right? And that's really what top, what
• 57:23
  is top of mind for that? And now these eval platforms
• 57:26
  that have come out, there's lots of them.
• 57:28
  They all have different flavours and and different angles.
• 57:31
  I would say very important part of building trusts in your AI
• 57:36
  pipelines, right? But really what we need to be
• 57:40
  able to do is if we have these multi stage pipelines that are
• 57:43
  running in production, we need traceability all the way up.
• 57:46
  So how can you traverse up from the final decision?
• 57:50
  What was the what was the input for getting there?
• 57:52
  But then also what were the stages before that?
• 57:54
  And how do we provide visibility and observability to AI leaders
• 57:58
  to really build confidence and understand which step of the
• 58:01
  pipeline was actually the wrong? Did the model not perform well,
• 58:06
  which then propagated down to the next stage?
• 58:08
  The same thing for like AI agents, right?
• 58:10
  Like they have 10 steps that they need to go and do which
• 58:14
  step of that of that AI agents workflow does it?
• 58:20
  Was it not good? Do I need to go and tweak the
• 58:21
  prompt? Do I need to go and modify and
• 58:23
  and iterate on right. And so that's where I think this
• 58:26
  next stage is. It's also for agents, also for
• 58:28
  these multi stage production pipelines that we think are very
• 58:32
  important for businesses to run to.
• 58:33
  And you need, you need to provide them that level of
• 58:36
  confidence so you can do things like give confidence scores and
• 58:39
  things like that that models have.
• 58:41
  But what Costas is saying is true is you need to have all of
• 58:44
  the outputs and the reasonings behind these models and how they
• 58:46
  made a decision and give them a really nice interface and
• 58:49
  tooling to be able to then go and review it very quickly and
• 58:52
  know what they need to go and iterate on order to get that
• 58:54
  output data quality to be feel that good.
• 58:57
  Like maybe it's not 100% of the time they have confidence, but
• 59:00
  when they're running these things in batch at scale, if we
• 59:03
  can get 99% confidence that the output for this mission critical
• 59:07
  pipeline is good, that's probably good enough for us.
• 59:10
  But there's there's tooling and info that needs to get built for
• 59:12
  that. And to add something here,
• 59:14
  because that's, I think that is, is relevant also with the last,
• 59:19
  the previous generation, let's say of like data infrastructure.
• 59:22
  So like a term that every data engineer and probably like every
• 59:27
  data practitioner is like familiar with is like data
• 59:30
  lineage, right? So everyone's thinking like a
• 59:34
  very important part of ensuring the quality of our data is like
• 59:38
  keeping track of the lineage. Like how, OK, I have this end
• 59:42
  result here, this report, how these reports came into what it
• 59:47
  is when there are literally hundreds of tables that we have
• 59:52
  to operate on, like to get that right.
• 59:55
  And it appears that when we're working in a fully deterministic
• 59:59
  world, just keeping track of the column level is enough, because
• 1:00:03
  knowing the data type. Is enough to reason about what
• 1:00:08
  is happening, right? And it fits also well like with
• 1:00:11
  the columnar, let's say like nature of these systems, the EO
• 1:00:14
  lab systems that we have. But you can't do that anymore
• 1:00:18
  like slightly differences, small differences in like the input
• 1:00:24
  that you put and the prompt that you have there, which you didn't
• 1:00:28
  have before, right? As like an additional piece of
• 1:00:33
  data there contains dramatically like the output that you get on
• 1:00:37
  the other side. So what you need now is more of
• 1:00:39
  like role level in it, which is a very hard problem.
• 1:00:42
  And it's not something that has been developed primarily because
• 1:00:51
  it was hard enough and not needed enough for people like to
• 1:00:54
  invest in that. But I think that that's
• 1:00:57
  something that we are also like working ourselves from that I
• 1:00:59
  think this is part of how you can make people create like a
• 1:01:04
  traceability there that it will definitely change completely how
• 1:01:09
  like the quality of the eval starts that you are doing.
• 1:01:12
  I'm not sure I fully understand role level lineage.
• 1:01:15
  So the role level lineage is that let's say you start, let's
• 1:01:21
  take again the example of the transcripts, right?
• 1:01:24
  The output of your like whisper model is going to be a BLOB of
• 1:01:29
  text. Right now you might do a few
• 1:01:32
  things like, OK, break it down into pieces, chunk it da, da da,
• 1:01:35
  whatever. Now the next step is like for
• 1:01:37
  each one of these chunks that you have this, each chunk is a
• 1:01:42
  row, right? This is going to be fed into an
• 1:01:45
  LLM with a prompt that says, do you find any references there
• 1:01:51
  here for the names of the participants that they are like
• 1:01:54
  mistakenly transcribed? And if yes, fix them right now
• 1:02:01
  your first row got into the LLM and the second will do the same,
• 1:02:06
  the third will do the same, blah blah blah blah blah.
• 1:02:07
  Right? Now the next step is that you
• 1:02:10
  are going to take each one again of them, right?
• 1:02:13
  And create a summary of its chunk, right?
• 1:02:17
  So you see that each row goes through steps of processing, but
• 1:02:22
  because of like the differences in like the data that you feed
• 1:02:25
  at the row level, right? And the how the prompt might
• 1:02:29
  change for each one of them. You might have like different
• 1:02:33
  results, so you want to be able like to track that something
• 1:02:37
  that you didn't you didn't need like to.
• 1:02:39
  Do and like a tree traversal exercise, you, you wind up in
• 1:02:42
  this leaf node, but then what? What were the the the nodes
• 1:02:46
  before that that led to that right Where?
• 1:02:48
  So that's what you want to kind of traverse back up this multi
• 1:02:51
  stage pipeline and see. So if let's say you have your
• 1:02:55
  end summary, right? Because at some point you get
• 1:02:57
  all these like mini summaries and you put them all together
• 1:03:01
  into a like an LM and you put the output which is like your
• 1:03:06
  summary and you see the summary there.
• 1:03:07
  And you're like, no, I don't like that.
• 1:03:10
  So now you want to go backwards, right?
• 1:03:13
  And you're like, OK, what contributed?
• 1:03:15
  What data contributed in this particular output that I have
• 1:03:18
  here, right? And then you say, oh, it's like
• 1:03:20
  this five summary mini summaries that I have.
• 1:03:24
  You want to be able like to track these five and recall
• 1:03:27
  them. So you as the human or whatever,
• 1:03:29
  like a valve machine, you're machinery you're using can have
• 1:03:32
  access like to this particular data with the lineage as it was
• 1:03:36
  before. You can't do that because the
• 1:03:38
  lineage is just keeping track of the metadata of like the columns
• 1:03:41
  that participate, right? But now you can't do the same
• 1:03:47
  thing because the actual data has like a big effect.
• 1:03:51
  It's not just like the data type.
• 1:03:53
  It's not that something, let's say a joint breaks because I was
• 1:03:57
  expecting a data type of integer and it was a string, for
• 1:04:01
  example, something like that. Like, OK, I'm just like making
• 1:04:03
  things up. Now it's much more like you have
• 1:04:08
  to get into much more detailed views of the data itself to
• 1:04:12
  understand how the LLMS at its stage operate to give a good or
• 1:04:16
  a bad result. And you have to be able to
• 1:04:18
  navigate that. And the data infrastructure does
• 1:04:20
  not keep track of that. So it's not an information that
• 1:04:24
  you can recall. And that's like one of the
• 1:04:25
  things that I'm saying that it is a big problem to solve
• 1:04:29
  because it's not just like the evals from point of view of like
• 1:04:33
  how scientifically do an eval that has valid, it's valid.
• 1:04:38
  It's also how to capture all the data, because if that's
• 1:04:42
  overhead, store all the data which adds overhead and process
• 1:04:46
  all these evals that now like explode in terms of like the
• 1:04:49
  number of evals that you have like to do there, right?
• 1:04:52
  What you're talking about is the logs and this data lineage that
• 1:04:58
  we have is not sufficient. It's not painting good enough
• 1:05:04
  picture for us because even if we know that, yeah, this call
• 1:05:09
  went through successfully or this data was transformed in
• 1:05:12
  this way, we can't get that granularity that you're talking
• 1:05:16
  about. And so you don't see that as
• 1:05:20
  something of a job of an observability tool?
• 1:05:24
  It's like the same thing as we have like if you think about
• 1:05:26
  like the data platforms, right. So you had something like data
• 1:05:30
  bricks that or Spark that would go and actually execute whatever
• 1:05:35
  logic you deploy there, right? Then you most probably would
• 1:05:40
  have another tool that is doing like analyzes the linear ads or
• 1:05:45
  does like the quality checks, blah, blah, blah, like whatever.
• 1:05:49
  Take the QA of data and the lineage there as inputs and
• 1:05:55
  substitute the names with evals instead of QA, right?
• 1:06:00
  And the lineage still has to be there.
• 1:06:03
  I think what people, I think one of the problems is that we,
• 1:06:08
  we've been building applications around the lens for SAT
• 1:06:15
  modalities primarily, right? So of course, when you do that,
• 1:06:20
  it's all about, OK, that's the prompt of the user.
• 1:06:22
  That's like the output. Is this like a good one, right?
• 1:06:25
  But when you start getting into an environment where the
• 1:06:29
  interactions with the LLMS for an outcome become much more
• 1:06:32
  complicated and they have dependencies between them,
• 1:06:35
  right? You have to expand like your
• 1:06:39
  understanding of that to the whole pipeline that is built.
• 1:06:43
  Like to do that right, again, you might need, it's probably
• 1:06:49
  like a different tool to do that, but still this different
• 1:06:52
  tool we need needs to access data.
• 1:06:55
  That's the engine that does the processing has to capture, which
• 1:06:59
  on it's own is like a hard problem and then it has to work
• 1:07:03
  with, it will face like the same problems that observability in
• 1:07:08
  the app world have, which is there's like a lot of data.
• 1:07:13
  The value per piece of data is not that high.
• 1:07:16
  So we have to be extremely good at storing this data to make it
• 1:07:21
  affordable. And then you have the additional
• 1:07:26
  problem of like a vault being slow, so and expensive.
• 1:07:28
  So again, how do we pick the right vault like to do and what
• 1:07:35
  kind of tooling we have like to give to the users to build the
• 1:07:37
  Splunk of LLMS at the end of the day, right?
• 1:07:42
  I don't know, I'd love to see how this is going like to come
• 1:07:49
  out because I think it is like a pretty lucrative space to build,
• 1:07:54
  Although I know they're like very like as in any other LLM
• 1:07:58
  related activity like thousands of companies trying to do it.
• 1:08:05
  But I think I, I still believe there's like a lot of noise and
• 1:08:08
  not that much signal. And again, that's what I'm
• 1:08:11
  trying to say is that I think there's like tremendous value
• 1:08:13
  for like the data people to go and build solutions for LLMS
• 1:08:19
  because primarily is driven by more application engineering
• 1:08:23
  people. And there's a lot of
• 1:08:27
  foundational stuff that comes from the data world that they
• 1:08:32
  are needed if we are going to be building with LLMS. And I wonder
• 1:08:35
  in your time talking with folks, it feels like right now, because
• 1:08:39
  there's so much open space and there's so many new pieces that
• 1:08:43
  we're trying to add to our platform and our ecosystem of
• 1:08:47
  putting this into production, there's a lot of things that you
• 1:08:51
  could do. Where have you seen people
• 1:08:54
  focusing on what absolutely needs to be done before we can
• 1:08:58
  do anything else? Like what are the main
• 1:09:00
  bottlenecks? Do I need to go out and get an
• 1:09:02
  evals tool? Do I need to go out and get a
• 1:09:06
  proxy or like AAI gateway? Yeah, the eval tools are
• 1:09:09
  important, right? You need that feedback loop,
• 1:09:11
  right, to be able to help you understand if you're building
• 1:09:14
  effective AI, if the output of what you're doing, if it isn't
• 1:09:18
  what you're expecting, why isn't that right?
• 1:09:20
  So this eval tools provide you that feedback loop to you all to
• 1:09:23
  go and do that. You know, something that I
• 1:09:25
  always think about when you're talking about tabular and
• 1:09:28
  structured data world, right, is engineering teams were really
• 1:09:32
  good at like building like Canary builds, for example,
• 1:09:35
  right, that go back and try and sense any sort of regression
• 1:09:38
  that happens in these pipelines that are more or less
• 1:09:41
  deterministic because you're dealing with structured and
• 1:09:43
  tabular data, right? And so we'd run these nightly
• 1:09:46
  Canary builds and the output of that would be like, oh, there's
• 1:09:49
  like 4% drift or like 5% drift because you introduced some
• 1:09:54
  regressions by adding some application code that actually
• 1:09:58
  caused some drift from what we're expecting to have the
• 1:10:01
  output of these pipelines being right.
• 1:10:02
  That same kind of mindset needs to be applied now and thought
• 1:10:06
  about. And that's where I think we're
• 1:10:07
  talking about the opportunity here is like, how do we take
• 1:10:10
  that same concept and allow teams to be able to build these
• 1:10:14
  kind of Canary pipelines, for example, on top of output for
• 1:10:19
  non deterministic models. And it's a very big challenge
• 1:10:21
  because the nature of the data is totally different.
• 1:10:25
  You're dealing with lots of text, right?
• 1:10:27
  These evals are very expensive to store and process and, and
• 1:10:31
  build inside on top of right? Now the more complex the
• 1:10:35
  problem, I think the bigger the opportunity from an engineering
• 1:10:38
  standpoint. And I think the more fun it is
• 1:10:39
  to go and solve it. And that's why I think it's kind
• 1:10:41
  of wide open for helping teams build that, right?
• 1:10:44
  Like imagine if you're able to like the transcripts example,
• 1:10:47
  right? You, you, you're able to sign
• 1:10:50
  certain scores and confidence levels to the output of the
• 1:10:53
  pipelines that you ran on a daily basis based on certain
• 1:10:56
  properties that you consider to be successful outcomes of the
• 1:10:58
  pipelines, right? Which is what kind of Canary
• 1:11:01
  builds were were built for, right?
• 1:11:03
  It's a very deterministic way of being able to evaluate your
• 1:11:07
  software and your pipelines. And so applying that same
• 1:11:10
  concept to unstructured data processing is, I think going to
• 1:11:13
  be very important in a huge unlock.
• 1:11:15
  For now, AI and data teams feel confidence around putting these
• 1:11:20
  and leveraging inference as a new transform in production.
• 1:11:24
  Everyone would like to do something with AI, right?
• 1:11:26
  Everyone's like, oh, like, OK, we should invest into that.
• 1:11:29
  That's like why I think that's one of the big reasons behind
• 1:11:32
  having the role of the head of AI is because there's a mandate
• 1:11:35
  like from the board or like whatever that we need to do
• 1:11:37
  things like with AI, we have no idea what to do.
• 1:11:42
  But it's a race because all of our competitors are doing it.
• 1:11:44
  We need to be on that. Too.
• 1:11:45
  So we'll bring this poor guy here whose job is like to go and
• 1:11:49
  find use cases. So he will go, as Johnny said,
• 1:11:53
  like to the marketing folks, but like, hey, guys, like, how can I
• 1:11:55
  help you? I have budget, by the way.
• 1:11:57
  So that's great, right? Like you don't.
• 1:11:59
  So let's let's do it. Sales people the same thing,
• 1:12:03
  like engineering the same thing, like product the same thing.
• 1:12:07
  And then they have like to go and build something.
• 1:12:09
  And so the first thing is like, OK, many companies are like
• 1:12:13
  still up the stage, these where, where should we focus, like from
• 1:12:18
  all the different things that we can do, like where we should
• 1:12:21
  focus, right, which project like we should run.
• 1:12:25
  And these like typically companies that are like early in
• 1:12:29
  the journey, then you have cases where they build the demos, but
• 1:12:39
  you get it's hard to deliver consistently the value to
• 1:12:43
  whatever like the stakeholder is because again, salespeople don't
• 1:12:48
  care about your signing technology.
• 1:12:49
  Like, OK, if you're going to help them, like you have to help
• 1:12:51
  them, right? Like, and you're talking about
• 1:12:54
  business lines here that they are like as sponditive as they
• 1:12:57
  can be. It's like, I have a quota, dude.
• 1:12:58
  Like, oh, I'm going to lose my job.
• 1:13:00
  So like, can you help me with my quota or not?
• 1:13:03
  Like that's it. Not if you help me 30% of the
• 1:13:06
  times I have to think, is that like worth the time?
• 1:13:09
  Like doing it I. Yeah, and potentially you can
• 1:13:12
  mess up what I'm doing 30% of the time because you're sending
• 1:13:15
  the wrong emails or you're sending some hallucinated jargon
• 1:13:18
  A. 100% And then there's I had like some, we had like some
• 1:13:22
  interesting conversation with folks that they were saying,
• 1:13:25
  well, we run like a few experiments and our experience
• 1:13:29
  was that like, I'll give like an example.
• 1:13:31
  Let's say we have tickets, right?
• 1:13:34
  And like support tickets and we want to be able like to extract
• 1:13:38
  and like first of all, label them somehow like into
• 1:13:40
  categories and then extract some information from there that can
• 1:13:44
  drive like our products, like decisions, right?
• 1:13:48
  So what they usually do is like they find a company that
• 1:13:52
  provides labeling with a lens or whatever as a service.
• 1:13:57
  They go there, they're like, OK, I have to take all my tickets.
• 1:14:00
  I have to upload the tickets here.
• 1:14:03
  I'll run this thing. Something comes out.
• 1:14:05
  Well, it's not exactly what I expect.
• 1:14:07
  Kind of a black box. Yeah.
• 1:14:09
  Like iterate, iterate, iterate. Actually what he was saying like
• 1:14:12
  we did like a lot of prompting gymnastics to make it work.
• 1:14:18
  And actually one of the very interesting feedback on that was
• 1:14:21
  like we need it. Like when you are doing
• 1:14:25
  classification, you still have the problem of like you have to
• 1:14:27
  tell to the LLM, like what's the classification scheme of that,
• 1:14:31
  right? Or using LLM like to figure it
• 1:14:33
  out. But still you have to figure out
• 1:14:34
  what are the classes there, right?
• 1:14:35
  It's not like, so the Oracle will just come out and be like,
• 1:14:39
  yeah, that's what you should do. And like, shut up.
• 1:14:40
  You don't have an opinion on human, you know.
• 1:14:43
  So we did that took like some time.
• 1:14:47
  We ended up getting like an output which was like a data set
• 1:14:50
  in CSV that we could download. And I was like, OK, that's like
• 1:14:55
  too much work to start. Because here's the thing, like
• 1:14:59
  when you get your labels, that's when like the real work starts,
• 1:15:02
  right? Like you still have to go and
• 1:15:04
  figure out insights out of these like labels.
• 1:15:07
  So the guy was like, OK, like I, I can't do that.
• 1:15:10
  Like I can't like being like in the process of like downloading,
• 1:15:14
  uploading, getting CSVS, put them like somehow like on my
• 1:15:19
  Snowflake. And then have an analyst also
• 1:15:21
  like to go and analyze these things like, OK, that's not
• 1:15:24
  going to work. And that's your work.
• 1:15:26
  Exactly. So it's a lot of like I think
• 1:15:30
  like the the kind of it's like a problem, like more like a
• 1:15:33
  product mistake. But again, people need to
• 1:15:36
  understand like they need to meet the users where they are
• 1:15:40
  and not try like to force them to do things that do not fit
• 1:15:45
  their like work flows because a product person is a product
• 1:15:48
  person. Again like he's getting just by
• 1:15:53
  the business based on the product work that they are
• 1:15:56
  doing, not the label things that they are doing right and this.
• 1:16:01
  Yeah, it's potentially a huge distraction.
• 1:16:03
  Yeah. And it is.
• 1:16:04
  It is. And then you have the companies
• 1:16:07
  that they manage to get to the point where they have things
• 1:16:12
  that they work, but then they're like, OK, how do we put these
• 1:16:16
  things like into production? It's on it's own is like a big,
• 1:16:19
  big conversation, like what that means and what risk it puts to
• 1:16:23
  the project itself. Because again, like the people
• 1:16:25
  on the other side, they're waiting results, right?
• 1:16:29
  And then there are like a very few companies, I think like, OK,
• 1:16:33
  it's like usually either like Fortune 20 type of companies or
• 1:16:37
  very like Silicon Valley high tech companies, which by the
• 1:16:41
  way, many of them, they say they're AI.
• 1:16:42
  They are not really AI, right? But there are some of them that
• 1:16:45
  they are doing like they put things in production, like
• 1:16:47
  seriously. But we are talking about, I
• 1:16:49
  don't know like maybe Tensor like low hundreds of companies
• 1:16:54
  out there that have like successfully do that and.
• 1:16:56
  The other thing we see too is a common theme is everyone gets
• 1:16:59
  wide eyed when like the head of AI comes to them.
• 1:17:02
  Be like, what can I build you with AI?
• 1:17:04
  Like just tell me I have a whole team will go and build this free
• 1:17:06
  budget. It'd be great.
• 1:17:07
  I got budget, I got everything going, whatever you want.
• 1:17:10
  We'll top it off with a cherry for you and they'll go and
• 1:17:14
  they'll come up with these ideas.
• 1:17:15
  They'll create their AI road map for the quarter.
• 1:17:17
  The team will go and execute on it and then he turns around and
• 1:17:20
  says, yeah, we built it, we spent the time on it, we gave it
• 1:17:23
  delivered to them with a cherry on top and a nice bow and they
• 1:17:26
  don't use it, right. And so I think a lot of it also
• 1:17:30
  has to do with you really need to go in and identify the high
• 1:17:33
  value use cases that are business critical for you,
• 1:17:38
  right? If you get something that's like
• 1:17:41
  kind of supplementary and not really easily easy to integrate
• 1:17:46
  and you're like in the day-to-day workflows of like a
• 1:17:48
  product marketing manager, for example, they're probably not
• 1:17:52
  going to use it, right? And so, and so I think it's very
• 1:17:57
  important when you're thinking about considering and taking on
• 1:17:59
  new AI projects that the business outcomes and there are
• 1:18:03
  key success metrics that are there, right?
• 1:18:05
  Like, OK, I, you know, like the content moderation, for example,
• 1:18:10
  is another big example besides like the transcript thing,
• 1:18:13
  right? These, these companies that are
• 1:18:14
  building communities where the users are applying content and
• 1:18:19
  they're going through and spending tons of money on
• 1:18:22
  content moderation teams, which means humans are going through
• 1:18:26
  and reading every single message in order to understand whether
• 1:18:30
  the the the meat of whether the message was safe or not to be
• 1:18:35
  published into the community. It's all about building very
• 1:18:37
  safety. So if there's any like racist
• 1:18:39
  undertone or sexism or things like that mean they want to
• 1:18:42
  disqualify profanity, things like that too, right.
• 1:18:46
  And so when you have a use case like that and you're spending
• 1:18:49
  hundreds of thousands, sometimes millions of dollars on hiring
• 1:18:53
  this workforce that's just literally sitting there reading
• 1:18:56
  messages manually. Now there's a business outcome
• 1:18:59
  associate. How can I reduce my cost, right,
• 1:19:02
  'cause that model doesn't scale. Now if let's say I want to
• 1:19:05
  expand to different locales, right?
• 1:19:07
  I want to now offer this in Portuguese and I want to offer
• 1:19:10
  the same community in Spanish or French, I have to go and build
• 1:19:14
  out these content moderation teams that are French speaking
• 1:19:16
  and all of that, right? And so LMS are great at that
• 1:19:19
  kind of use case too. And the value is, is very
• 1:19:24
  apparent there, right? Like now no longer do I have to
• 1:19:27
  go and spend millions of dollars on hiring these folks.
• 1:19:30
  I can dedicate that cost towards inference, build these really
• 1:19:33
  robust pipelines that are helping me do content moderation
• 1:19:37
  and even get better performance outcomes and metrics.
• 1:19:39
  Like my, my, my mean time to review a message or a
• 1:19:42
  conversation is no longer 8 hours because I need to wait for
• 1:19:46
  the content moderator shift to start in France or whatever it
• 1:19:50
  is, right. And so I think that's one of the
• 1:19:53
  key lessons that we're seeing a lot too is as your company
• 1:19:56
  thinking about it bringing on building on AI use cases, make
• 1:20:01
  sure that it's getting that there are there are key success
• 1:20:04
  metrics and business metrics that you're targeting and that
• 1:20:06
  you're able to track as you're putting AAI into production.
• 1:20:10
  For it, I heard it like on AX and Y axis and I I think the
• 1:20:16
  guy's name was Sergio that said it.
• 1:20:18
  It was when I was at a the Gen. AI conference in Zurich and
• 1:20:22
  basically he said put on an X&Y access the impact and then
• 1:20:28
  confidence that you can actually implement it and whatever is the
• 1:20:33
  highest up in that top right quadrant, that's what you should
• 1:20:38
  start with. And the, the, the conference you
• 1:20:40
  can implement comes down to the tooling and infrastructure that
• 1:20:43
  you have to do it right. And that's where I think we're
• 1:20:45
  seeing a lot of innovation going into now.
• 1:20:47
  It is helping build that confidence for teams to be able
• 1:20:51
  to say I, I know I can put this into production.
• 1:20:53
  This is a very common. Issue now the only other, yeah,
• 1:20:55
  Now the only thing question is, is there business value and
• 1:20:58
  that's potentially up to the stakeholder to be able to decide
• 1:21:01
  that, right? Yeah.
• 1:21:02
  And there's something else that's like, I want to add to
• 1:21:04
  that because we see, you know, having budget is kind of like a
• 1:21:09
  blessing and the curse at the same time, like for these teams,
• 1:21:13
  especially like in the market as it is right now, because one of
• 1:21:16
  the patterns that we see is that, OK, usually, like the head
• 1:21:20
  of AI is not necessarily like a technical person themselves,
• 1:21:23
  right? So they rely a lot on like the
• 1:21:27
  teams like to find the tooling that they need, right?
• 1:21:29
  And obviously there's like a lot of offerings out there.
• 1:21:33
  Now we are still at the stage of the industry where like there
• 1:21:38
  are like so many vertical eyes, like very specific tools, you
• 1:21:42
  know, that do like one thing and they're pretty much like a black
• 1:21:48
  box, right? So they have money.
• 1:21:51
  So they go and like get that stuff and it's good because in a
• 1:21:54
  way like it helps you kick start like your projects.
• 1:21:58
  But there's like a huge trap in that you can't engineer systems
• 1:22:03
  with black boxes. You can't do that.
• 1:22:05
  And people need to understand that like, no matter what, like
• 1:22:08
  a little lips are like there's true engineering that needs to
• 1:22:10
  happen to make them like robust and deliver value at the end.
• 1:22:15
  So my advice, like to the people is that you should try and
• 1:22:19
  invest more on the infrastructure and your
• 1:22:22
  knowledge of like how to build the right things and what
• 1:22:26
  practices will drive you there instead of like, oh, you know
• 1:22:30
  what, like, OK, I need to OCR something here, let's go.
• 1:22:34
  And like use every black box OCR thing there that says that they
• 1:22:39
  are 5% better than Mistral or like whatever out there, right?
• 1:22:45
  And do it for like prototyping 100%.
• 1:22:48
  But I guarantee you that if it's successful, you'll get to a
• 1:22:51
  point where you'll be like, OK, now what do we do with this
• 1:22:54
  thing? Because it's either like too
• 1:22:55
  slow, it's not reliable. Oh, now like we are getting
• 1:22:58
  outputs that like we can't really understand why we're
• 1:23:00
  getting the outputs that we're getting.
• 1:23:02
  Or oh, now we're getting into a different use case of like
• 1:23:05
  different documents that these models that they are using here
• 1:23:07
  are not probably like as good as they are we used to be.
• 1:23:11
  So now we're are we adding another tool that we have to
• 1:23:14
  manage and who's going to do that, right.
• 1:23:16
  So I think engineers should keep thinking like engineers invest
• 1:23:21
  like in tools that they are, let's say good with like their
• 1:23:28
  infrastructure for their work and they will build the value on
• 1:23:31
  top of them. And one last thing on that,
• 1:23:35
  there is a reason that these like revolution of AI goes
• 1:23:42
  through like the engineering practices first.
• 1:23:46
  The reason for that is that the work that we are doing as
• 1:23:49
  engineers is, is to validate, right?
• 1:23:51
  So you can add like if I asked like the LLM like to build me a
• 1:23:56
  function, It's almost like you can almost automate figuring out
• 1:24:01
  if it's like working or not. Like you run it, you just run
• 1:24:04
  it, you compile it, right now there are like many super
• 1:24:10
  impactful problems to be solved with the lamps that you don't
• 1:24:14
  have that if you are doing the things that like you only was
• 1:24:16
  describing with the transcripts, right?
• 1:24:19
  How do you validate like with like 100% confidence that this
• 1:24:24
  thing is going like to work and data.
• 1:24:27
  And that's like something, again, I'll go back to the data
• 1:24:29
  practitioners and how important their knowledge and experience
• 1:24:33
  is like to make this successful because they know that data
• 1:24:36
  drifts. That was always the case.
• 1:24:39
  Like there's nothing that you can just build it once, put it
• 1:24:42
  out there and it was going like to be working forever.
• 1:24:45
  And that's going to become even more true like with LLMS. So you
• 1:24:49
  have to build systems. You can't just like throw black
• 1:24:52
  boxes in there and make things like work.
• 1:24:55
  You have to be an engineer. You have to engineer this.
• 1:24:57
  And you have like to keep iterating on it as like the data
• 1:25:00
  drifts as like the needs drift also like from the users and
• 1:25:05
  with other lens it's going to happen like in the much more
• 1:25:08
  accelerated base. So again, like focus on core
• 1:25:12
  skills and infrastructure. I wanna connect the dots on two
• 1:25:17
  things that it feels like we're dancing around, which is that
• 1:25:22
  business value. And the skill of being able to
• 1:25:27
  sniff out that business value and understand how to properly
• 1:25:31
  implement it is one thing. But then going back to since
• 1:25:36
  2020, I've heard almost everyone that has presented at an ML OPS
• 1:25:42
  community event talk about in some way, shape or form in
• 1:25:45
  different words, saying how I can build the best model I can
• 1:25:51
  build with the highest accuracy score.
• 1:25:53
  I spent 5 weeks tuning it, so it went from 98.1 accuracy to 95 or
• 1:26:02
  whatever that like metric is inside of it.
• 1:26:05
  But then I gave it to the people that were going to be using this
• 1:26:10
  model and they didn't use it and it fell flat on its face.
• 1:26:13
  And all of that time that I had spent on it was for nothing.
• 1:26:17
  And so the whole idea of being able to make sure that what
• 1:26:21
  you're building is the right thing and that you're spending
• 1:26:25
  ample amount of time getting it into production as quickly as
• 1:26:28
  possible to know if it is the right thing or where you need to
• 1:26:31
  tune it is so important. And that's like what you're
• 1:26:34
  saying Costas here is get it in there.
• 1:26:37
  Start working with it, engineer it in a way that you can then go
• 1:26:41
  and debug it when you need to. Figure out if it's not working
• 1:26:45
  because we've absolutely missed the mark on the product or we
• 1:26:48
  missed the mark on one of the steps in this pipeline.
• 1:26:52
  Yeah, it's not about the models anymore.
• 1:26:54
  The models are going to keep getting better, keep beginning,
• 1:26:56
  more accurate, probably cheaper and a lot more performance, less
• 1:27:00
  hallucination, all that kind of stuff.
• 1:27:02
  It's all around the infrastructure you have around
• 1:27:04
  it now, right? And your ability to have that
• 1:27:06
  really tight feedback loop, to be able to know and build
• 1:27:09
  confidence around the outputs of the pipelines that you have and
• 1:27:13
  be able to trace all the way back, to be able to iterate and
• 1:27:15
  make improvements incrementally on, on that.
• 1:27:18
  And that's where we lie now. And like the AI innovation
• 1:27:22
  spaces, there's lots of great innovation happening towards
• 1:27:25
  that. And I think in the next couple
• 1:27:28
  years, it's going to be a lot more prominent where we're
• 1:27:32
  starting to see teams feel like you're talking about the X&Y
• 1:27:35
  axis. The confidence level is going to
• 1:27:37
  be going way up because of the infrastructure and tooling
• 1:27:40
  that's being produced. Now.
• 1:27:41
  I want to give a quick shout out.
• 1:27:43
  Demetrios. You, you are.
• 1:27:45
  Yeah. You know, you're the ultimate
• 1:27:47
  community builder. And we, Costa Sinai, are always
• 1:27:50
  super impressed and admire. Like you just seem to be
• 1:27:53
  everywhere all at once. And we're like, Oh my God, I
• 1:27:55
  don't know how he does it. So and, and it's like amazing
• 1:27:59
  seeing the prize of the community and like how you've
• 1:28:02
  been able to grow it and all the conferences that you're doing.
• 1:28:05
  So like lots of respect, man. And, and it's been been really
• 1:28:09
  fun, just like over the last few years seeing all of it happened.
• 1:28:12
  I'm. Glad that we got to make this
• 1:28:13
  happen. Yeah.
• 1:28:14
  And thank you, big thank you for making type def.
• 1:28:18
  And our company happened like it was one of the things that's
• 1:28:21
  super hard when you're building a startup is like finding Co
• 1:28:24
  founders. And we know we're both second
• 1:28:26
  time founders. We put even more emphasis in it.
• 1:28:29
  So we don't take for granted the fact that you made us meet up at
• 1:28:33
  some random Blue bottle in San Francisco.
• 1:28:35
  And then, and then we're 5 minutes into the conversation,
• 1:28:37
  decided that you had better things to do.
• 1:28:39
  It rarely works, but this time it did, so I'm going to put that
• 1:28:43
  on my resume now. Yeah, so, so thanks for
• 1:28:46
  everything, man. So I'm saying then communities
• 1:28:49
  have have always been, it's been like very important in building
• 1:28:53
  technology and they will always be.
• 1:28:57
  So that's another kind of like service that you provide
• 1:29:03
  bringing like the people together, especially like with
• 1:29:06
  when you're solving problems of like they're not even well
• 1:29:08
  defined. Like at the end of the day, it's
• 1:29:11
  all about like emerging patterns like through like people
• 1:29:14
  interacting who are passionate about what they are doing and
• 1:29:17
  like trying like to find solutions.
• 1:29:18
  So it's like probably like the most important thing.
• 1:29:21
  And I think it's beautiful too that you mentioned that people
• 1:29:24
  come from different backgrounds, like the data engineering
• 1:29:28
  background, like the modelling background, data science like
• 1:29:30
  the SRE background. And getting this space, and
• 1:29:33
  specifically this space, getting to see how each one of these
• 1:29:38
  folks is attacking the problem is really cool, and it makes for
• 1:29:42
  fertile ground for Innovation 100.
• 1:29:44
  Percent and I think if we want to succeed, we need to somehow
• 1:29:47
  like increase the cross pollination like these
• 1:29:50
  communities and that's your job to do obviously, but there is
• 1:29:55
  like tremendous value like bringing like these diverse,
• 1:29:59
  let's say engineering disciplines like together and
• 1:30:04
  like trying to because here's like the what is like very
• 1:30:06
  interesting and kind of like why I'm excited with LLM's is
• 1:30:13
  because you know, we talked a little about my age, but LLM's
• 1:30:17
  like in a way, like they make me feel young because it reminds me
• 1:30:20
  like how technology was when technology was young.
• 1:30:23
  You know, like people complain today that like, oh, like this
• 1:30:28
  thing is like not reliable, blah, blah, blah, blah, blah.
• 1:30:31
  But they forget that in order to get our databases to be
• 1:30:34
  transactional and probably kids don't even know the day that
• 1:30:38
  like there is like transactions that they ensure that when you
• 1:30:41
  write something and I write and journey writes, it's going to be
• 1:30:44
  the correct thing. It looks like decades of
• 1:30:47
  research and development to get to that point, right?
• 1:30:51
  So we kind of like we are again in these like early states of
• 1:30:58
  like a new potentially very transformative technology and it
• 1:31:04
  feels nice. I mean, it's not easy, but you
• 1:31:08
  know, you get back like to how it was like hacking with
• 1:31:10
  networks around and like networks not being reliable and
• 1:31:12
  not like having fiber at home like each one of us and don't
• 1:31:15
  have like them to think that there is a router somewhere,
• 1:31:18
  right? And there is what's 1 of the bad
• 1:31:24
  things that SAS did was that it managed to hide from the vast
• 1:31:29
  majority of the engineers out there, the complexity and the
• 1:31:33
  efforts to make things reliable. But it was always about that
• 1:31:38
  there was no technology that the data that was introduced was
• 1:31:40
  reliable. It took a long time to make it
• 1:31:43
  reliable. The same thing will be also like
• 1:31:44
  with other lamps, right? And that's where engineering
• 1:31:47
  comes in. And but we're not like in 1995
• 1:31:52
  anymore. We're like in 2025.
• 1:31:55
  And there's like so much experience with all these
• 1:31:57
  different disciplines and like bringing these people together
• 1:31:59
  can really, really accelerate and make the things that took
• 1:32:02
  like decades to happen before, like now carbon, like in South,
• 1:32:05
  like a few years. So go out there, bring them
• 1:32:08
  together. It's top of mind and innovation,
• 1:32:11
  especially being here in the Valley, like thinking about AI
• 1:32:14
  and infrastructure and the new world that we live in now.
• 1:32:17
  But building as a community is also very important.
• 1:32:19
  Having the community one, but also being able to contribute
• 1:32:22
  together and innovate together, right.
• 1:32:25
  So part of our launch 2 for type def is we're open sourcing one
• 1:32:28
  of the libraries that's really amazing at going through and
• 1:32:32
  helping people build and let's say like Jupiter notebooks and
• 1:32:35
  interfacing with the LLMS very nicely.
• 1:32:37
  And so that's one of the things I think is a huge help in being
• 1:32:42
  able to move the pace of innovation forward is once you
• 1:32:45
  can have a project and have multiple projects and everyone's
• 1:32:49
  contributing, there's a lot of excitement.
• 1:32:51
  It helps build a lot of momentum in that.
• 1:32:52
  Space Who's going to be the pets.com of the LLM bubble?
• 1:32:59
  OK, I'll I'm not going to say like a name, but I do think that
• 1:33:06
  the prom to website companies, they are going to be something.
• 1:33:12
  To have a rude awakening. Yeah, yeah, I think so.
• 1:33:16
  I'm I think we need them like we need the page.com of the world
• 1:33:20
  like to happen for things like but you know, that's like if you
• 1:33:24
  reflect back likethe.com bubble, like people say, if you think
• 1:33:29
  like what happened back then, it was like an extremely
• 1:33:32
  verticalized solution that was built for like pretty much like
• 1:33:36
  everything crusts. And then people realize that we
• 1:33:42
  need platforms, right. And then you're on Amazon, which
• 1:33:45
  is I'm. Basically pets.com, but more.
• 1:33:48
  Yeah. And you have like Soapify and
• 1:33:51
  you have. You have a whole shoe too?
• 1:33:53
  That's doing pretty well. Where are they?
• 1:33:55
  Amazon No. No, I don't think so anyways.
• 1:33:58
  I digress that. Yeah, my my pets.com is whoever
• 1:34:04
  has that fucking billboard that says don't hire humans.
• 1:34:07
  That is on the whoever those guys are.
• 1:34:11
  There's so many things that you have to do to get a billboard
• 1:34:14
  and the fact that it got so many marketers and top level people
• 1:34:19
  to sign off on that billboard. I don't even know who it is, but
• 1:34:23
  I know that I don't like them. Like, I don't know, like I, I, I
• 1:34:27
  personally feel that anything that feels like too easy cannot
• 1:34:31
  be real. I might be wrong, Like I'd love
• 1:34:36
  to be wrong for myself at least. But like problems that they are
• 1:34:42
  valuable, they tend to be hard. You need to put efforts like you
• 1:34:46
  need to work hard like to make them successful.
• 1:34:49
  So there's no easy path to. It's not like the LLM just
• 1:34:53
  because you're doing LLMS like you're going to be reached like
• 1:34:55
  it doesn't work like that. I think there's tons of eval
• 1:34:58
  companies that are out there now too, right?
• 1:35:00
  And I think they're solving very hard problems, but we might
• 1:35:04
  start seeing some consolidation there too, right?
• 1:35:06
  As you get think about the the main observability and big
• 1:35:09
  players out there, like the data dogs of the world that are also
• 1:35:12
  very much thinking around how, how do they go and integrate
• 1:35:14
  into AI. So that's going to be an
• 1:35:18
  interesting thing to see. I think they're still cropping
• 1:35:20
  up like there's lots of different eval platforms there,
• 1:35:24
  but the ones that are solving the hardest problems, like kind
• 1:35:26
  of what Kosis was describing, I think are the ones that are
• 1:35:28
  going to be able to really stand on their own and and
• 1:35:30
  substantiate there. But it's a very important part
• 1:35:34
  of the AI life cycle, right? That problem set that they're
• 1:35:37
  going after. I don't know that we need like
• 1:35:39
  hundreds of them, but it'll be interesting to see.
• 1:35:43
  I think you know what, what happens in that space generally
• 1:35:45
  to track that over time. Yeah, I think like, although
• 1:35:48
  like I think the good thing with evolved companies is that to
• 1:35:51
  build an evolved companies requires like a baseline of
• 1:35:55
  technical competencies that at the end of the day, the teams
• 1:35:59
  that they build something, they will have some kind of like
• 1:36:04
  valuable exits, let's say, or like at least like the value is
• 1:36:07
  not going to be destroyed. There are companies that they
• 1:36:10
  will destroy value like there are like companies that will end
• 1:36:13
  up like the what was the company?
• 1:36:15
  It wasn't like a dot com era company was like much more
• 1:36:20
  recent the one that was doing the one click check out.
• 1:36:24
  Oh, fastly or. Fastly one of those which?
• 1:36:27
  Wasthe1.last.com Yeah, that had like the record in like Burn.
• 1:36:32
  It wasn't, no. Both was the one, the other one
• 1:36:35
  that was doing well, I think it was fast.com or fast.
• 1:36:38
  Yeah, the one that had like the broken record of like how much
• 1:36:42
  money they they burned in like the year or something like that.
• 1:36:46
  And the founder went on Twitter and was saying stuff about
• 1:36:49
  Stripe and how is the mafia. Yeah, of course.
• 1:36:52
  Yeah. There's always someone to blame
• 1:36:54
  if you want to. But still there's value that has
• 1:36:58
  been destroyed, right? Like, so there's always that
• 1:37:02
  like in the industry. I think it's part of like any
• 1:37:04
  fast pacing and like high rewards space.
• 1:37:09
  It doesn't mean that like everyone is like a scam or like
• 1:37:11
  anything like that. Will you have us on again
• 1:37:18
  Demetrios or is this like? A.
• 1:37:20
  There's a one time thing.