Latest / Agentic Conversations (formally mlops.community) / The Future of Feature Stores and Platforms // Mike Del Balso & Josh Wills // # 186

Transcript

The Future of Feature Stores and Platforms // Mike Del Balso & Josh Wills // # 186 · Agentic Conversations (formally mlops.community)

• 0:00
  Welcome back to the ML OPS Community Podcast.
• 0:02
  I am your host Dimitrios and I'm currently broadcasting from the
• 0:06
  island of Crete in Greece. Today we talked to none other
• 0:11
  than the CEO of Tecton, Mike Delbalso, and if you've been
• 0:16
  following along while I'm here in Greece, I am just pointing
• 0:19
  out a few Greek isms that I find when I'm here and it just it
• 0:24
  boggles my mind I don't understand.
• 0:26
  So if you work with any Greeks, or if you are Greek yourself,
• 0:30
  have a few questions for them, because this is the country that
• 0:35
  invented democracy. Yet they for some reason have
• 0:40
  not figured out how to make pipes that toilet paper can go
• 0:45
  into. What is that all about?
• 0:46
  No matter where you go in Greece, you have to throw your
• 0:48
  toilet paper into the trash. I do not get it.
• 0:51
  I do not understand it at all. Alright, so please, someone who
• 0:57
  knows about the two very or someone that is doing some kind
• 1:00
  of mill, maybe we could just upgrade the system, you know,
• 1:04
  we'll just throw some AI at it and it'll be an easy fix.
• 1:08
  Anyway, didn't come on here to ask questions about Greece.
• 1:11
  We came on here to talk about feature platforms and everything
• 1:16
  that comes with it. I have a great time talking to
• 1:19
  Mike always. He is a avid guest and Co host
• 1:25
  on the show and so today he was meant to be a cohost.
• 1:29
  But our guest of the day is a retired early retired Josh Wills
• 1:34
  and he just didn't show up because he apparently doesn't
• 1:38
  keep a clock in his house. Alright Josh, if you're
• 1:42
  listening to this, I I got to go into it a little bit.
• 1:45
  So Josh was supposed to be our guest and it was meant to be a
• 1:49
  great conversation between Josh and Mike and it ended up
• 1:52
  happening. But it happened at the end of
• 1:55
  the conversation that I was having with Mike.
• 1:56
  So Josh showed up 55 minutes late to the podcast and we ended
• 2:01
  up talking with them. We extended the podcast to like
• 2:03
  30 minutes and we had a blast having Mike and Josh just
• 2:08
  pontificate about where the future of future stores in
• 2:12
  future platforms go. But before that, when I was
• 2:15
  talking with Mike, I was asking all kinds of questions about
• 2:18
  what he sees as main barriers to entry and pain points while or
• 2:26
  or that a feature platform can solve for.
• 2:30
  And so he walked us through his history, how he was working at
• 2:34
  Google, and then he was working at Uber and why he felt like a
• 2:38
  feature platform was so important, a feature store,
• 2:40
  which then evolved into a feature platform.
• 2:42
  And he also talked about operationalizing these data
• 2:46
  pipelines and feature pipelines and why that can be hard and why
• 2:50
  it is so important. Had a great conversation with
• 2:52
  them. As always, if you enjoyed this,
• 2:54
  please share it with a friend, because that is how we will grow
• 2:57
  and continue to get the good word out about ML OPS LL mobs.
• 3:04
  Oh my God, I can't even believe I'm saying that.
• 3:06
  It's not coming out of my mouth, but last thing is I want to give
• 3:11
  a huge shout out to the Tecton team because they have been
• 3:14
  supporting the MLS community from the beginning.
• 3:16
  If you are on the market for a feature platform, check them
• 3:20
  out. We'll leave a link in the
• 3:21
  description or just hit up Mike directly in Slack and tell him I
• 3:25
  sent you. So, all right, Mike, we're here
• 3:34
  today to talk all about future stores and all about what the
• 3:40
  hell is going on these days with the feature store, but also to
• 3:44
  get into a little bit of the features and maybe clear up some
• 3:49
  questions on everyone. What is on everyone's mind when
• 3:54
  it comes to features and the new paradigm, we could say, of
• 4:00
  machine learning, AI and LM's and how those fit into feature
• 4:04
  stores, how features and recommendations and all that
• 4:08
  good stuff can be powered and play together.
• 4:13
  So I think it's probably good just to get like if you can give
• 4:18
  me a bit of an orientation on where we are right now with
• 4:22
  feature platforms, feature stores and how it's changed, how
• 4:26
  it's evolved over the past couple of years because it is
• 4:31
  constantly moving, right. Yeah.
• 4:33
  OK. So I think there's there's.
• 4:34
  Two concepts that we chat about sometimes we talk about feature
• 4:38
  stores and feature platforms and I think it's good to talk about
• 4:42
  the difference between both of those.
• 4:43
  And then it's good to also understand, maybe useful to talk
• 4:48
  through the history of kind of how we got to where we are
• 4:51
  today. But I think what's, and I think
• 4:53
  what's probably most interesting for the community is like.
• 4:57
  What do I need and when and like when is this relevant for me?
• 5:01
  I've heard a lot of like awesome companies use this stuff and my
• 5:05
  company doesn't. So should I be thinking about
• 5:07
  this kind of thing? So also disclaimer for this
• 5:10
  whole thing that we make we think of as the leading feature
• 5:15
  platform which has a feature store.
• 5:16
  You're getting one side of the discussion for me, but I try to
• 5:20
  be pretty balanced about it. And so maybe it will be helpful
• 5:23
  just to kind of talk through a little bit of like the history
• 5:26
  first, just to kind of understand some of the
• 5:27
  motivation. And so.
• 5:30
  So I used to work at Google and and we ran the real time AI
• 5:35
  models that choose which ads you see and you had D Scully on here
• 5:39
  before and that guy always PM so he was the manager.
• 5:45
  Maybe the smartest person I've ever met.
• 5:47
  These guys. Really awesome and and very,
• 5:50
  very under spoken and you know like you would never he would
• 5:54
  never tell you he's the smartest person in the room but you can
• 5:56
  figure it out very quickly. That guy's the real deal.
• 5:59
  He is where everyone that is wondering.
• 6:02
  He wrote the two seminal posts on machine learning back in like
• 6:07
  2017. No, even they wrote the They
• 6:09
  wrote the the blog post The paper Machine Learning the High
• 6:13
  Interest Credit Card of technical Debt in.
• 6:15
  Like 2013, That's when I was on his team.
• 6:18
  But like I was, I was junior mode and so they didn't even
• 6:21
  tell me they're writing a paper, so.
• 6:26
  So I wasn't in the driver's seat on the team at that time, but
• 6:30
  these guys were had been solving these problems for a long time,
• 6:33
  like true deep MLM's problem. So to give everybody a sense of
• 6:36
  the kinds of problems that Google has solving at that time.
• 6:39
  So 2013, so 10 years ago, so MLPS.
• 6:43
  In LM stuff feels like this whole new thing, and certainly
• 6:46
  LM stuff is, but you know Google 10 years ago, this was probably
• 6:50
  the the what I still think of as like the most productionized ML
• 6:54
  application, the best productionized mill application
• 6:57
  in the world at that time. And it was like a gigantic
• 7:03
  logistic regression model like the they had their own way to
• 7:07
  build this gigantic thing and it would be live training and
• 7:11
  updating every 15 minutes on all the searches in the world.
• 7:14
  And they had, they had like thousands of features in these
• 7:17
  models, constantly being computed in real time and being
• 7:21
  driven into the ads models. But these features were sourced
• 7:25
  from all kinds of different data sources too.
• 7:27
  So think of like you're a user of Google.
• 7:30
  Maybe they have some micro service that prepares some user
• 7:34
  preferences for demetrios and then they have another thing
• 7:37
  which is like calculate some statistics for each merchant or
• 7:41
  each domain or something like that.
• 7:44
  And then there's a lot of, like, complicated ads specific stuff.
• 7:48
  So the point was, this is like this behemoth system.
• 7:52
  Constantly running, launching new models on the fly.
• 7:55
  And so they were really far advanced in the mill.
• 7:57
  They're like at the mill OPS dream state where everything is
• 8:01
  live, learning and happening automatically and they're tuning
• 8:03
  the system at a high level. So they were trying to solve
• 8:06
  problems like. Here's two example problems.
• 8:09
  One is when we roll out a new model.
• 8:13
  Then that is going to start making different predictions
• 8:16
  than the old model. But that means you're getting
• 8:18
  different predictions in real time means you're going to show
• 8:21
  different ads. And so when you get show
• 8:24
  different ads. Then you know because of what's
• 8:27
  happening in real time becomes your training data.
• 8:29
  Your training data changes. And so how you were building
• 8:33
  training data 10 minutes ago compared to how you're building
• 8:35
  training data now is going to change.
• 8:38
  And then. But we didn't build this model
• 8:40
  on the new training data. We built it on the old training
• 8:42
  data. And so is there, are we going to
• 8:44
  introduce some weird feedback loop and things are gonna go
• 8:47
  wacky And how do we control that?
• 8:48
  It's a problem that probably nobody at MILFS community has
• 8:51
  even thought of. But when you solve all the other
• 8:55
  MYOPS problems, then you get to that kind of level and it was
• 9:00
  just really and that's like. Schlage the guy behind that and
• 9:04
  did a lot more there and they yeah, and then they rolled out
• 9:06
  they replace it all the deep learning and so real time
• 9:09
  learning, deep learning model fine tuning stuff like that.
• 9:12
  But the real thing that I learned.
• 9:13
  So anyway, all of that contacts to say that one of the
• 9:16
  interesting things that I learned there was that so much
• 9:20
  of the ability to build this high quality AI system, this
• 9:25
  really deeply productionized AI system was that it the their
• 9:31
  ability was completely. Based on their data capabilities
• 9:36
  under the hood they had so many. Had so many high quality
• 9:41
  datasets, they had so many great tools to access and develop and
• 9:45
  test with these datasets and that allowed them to be
• 9:48
  efficient in building high quality models and systems and
• 9:52
  build a systems that create those high quality models and
• 9:54
  roll them out and maintain them in production and stuff like
• 9:56
  that. And probably you know the one of
• 10:01
  the most important of such systems was all of the feature
• 10:04
  infrastructure, the stuff that's constantly computing and serving
• 10:07
  the new signals. So that was a lesson to me,
• 10:10
  that. This stuff is very important
• 10:12
  even for super advanced use cases they're and even after you
• 10:17
  go into the deep learning world, there's this kind of whole data
• 10:20
  supply chain kind of issue. You need to pass all the right
• 10:23
  data and the right context along to the model such that it can
• 10:26
  make the right decision and it's not something you learn.
• 10:28
  When you go to machine learning class.
• 10:30
  You don't learn like all of that, the whole data supply
• 10:33
  chain component of it. And it's also these problems are
• 10:37
  not something you learned when you go to analytics class.
• 10:39
  When you go to become a data scientist, they teach you how to
• 10:43
  make. How to use R, how to use Python,
• 10:45
  make some plots, make a dashboard, stuff like that.
• 10:48
  But it's not the same kind of stuff as the the production data
• 10:50
  engineering that happens at scale and sometimes in real time
• 10:53
  to power production AI application, so very hard
• 10:58
  domain. Then I joined Uber and Uber, we
• 11:01
  were starting from zero basically.
• 11:02
  It was so, yeah. So I joined Uber to start the ML
• 11:06
  team there, start the ML infrastructure team and really
• 11:09
  like enable ML across the company and when we did that.
• 11:14
  There was like nothing, there was no infrastructure there.
• 11:16
  And so we had to you know, build the model training, the model
• 11:19
  survey and all of the parts of the mill platform and up system
• 11:22
  became called Michelangelo. But one of the most important
• 11:25
  parts of that system was exactly that same thing, the feature
• 11:28
  infrastructure, the parts that allowed the data scientists in
• 11:33
  the company to access all of the data in the business and build
• 11:37
  models, build decision making systems on top of that data in a
• 11:41
  way that wasn't going to be impossible to productionized.
• 11:44
  So it's not that hard to go and let just download some data and
• 11:49
  then do a bunch of stuff in a kind of like in your notebook.
• 11:55
  And like some kind of error unreproducible processes that
• 11:59
  you forgot what they were, but you kind of moulded a training
• 12:01
  data set and you build a model and we've all forgot everything.
• 12:04
  You lost all of that code or it's all a jumbled mess, but at
• 12:07
  least you got the model. The real task for these teams
• 12:11
  that are building AI applications that are kind of
• 12:14
  like operationalized and productionized, like AI teams
• 12:19
  that are really focused on the true finish line for value at
• 12:22
  ML, which is this thing is operating in production and
• 12:25
  delivering. Value to the business, not just
• 12:27
  delivering insights internally to help me with the decision,
• 12:30
  but it's delivering external insights to my customer or
• 12:34
  external decisions that affect the product that my customer
• 12:37
  interacts with. That means they have to get to
• 12:39
  the production. That's the important finish
• 12:41
  line. And for them to be able to do
• 12:43
  that, they need to be able to build these models in ways that
• 12:48
  can be easily productionized. And that requires that
• 12:51
  productization step means you also need the data to be
• 12:55
  productionized you also. I mean the kind of data supply
• 12:58
  chain to to be operational not just like a one time hacky code
• 13:02
  thing that you built when you were building your prototype.
• 13:04
  And so we kind of already thought about that and I had
• 13:09
  been exposed to a lot of amazing designs and amazing solutions
• 13:12
  for that kind of thing at Google.
• 13:13
  And so that influenced a lot of our, a lot of our designs for
• 13:18
  building the feature infrastructure that was part of
• 13:21
  Michelangelo. And and the whole purpose of
• 13:24
  that was to make it really easy the way if you kind of like.
• 13:27
  Instantiate that and and and drill it down to something very
• 13:29
  specific. Make it very easy.
• 13:32
  For people to access business data, turn it into features.
• 13:37
  Store those features and serve those features in production and
• 13:41
  generate models with those features.
• 13:42
  So generate historical training data sets and do real time
• 13:45
  surveying with those features. So that's a little bit of the
• 13:48
  genesis of the feature store and that's when we came up with the
• 13:52
  term feature store. We started calling the feature
• 13:53
  store. We started talking about it a
• 13:55
  little bit and did some blog posts about Michelangelo.
• 13:58
  You got your name on that blog post though.
• 14:00
  That paper you did your name on? Yeah, yeah, yeah, exactly.
• 14:03
  Yeah. I did write that blog post, that
• 14:05
  one I was involved in. And yeah, which is another
• 14:07
  Seminole, I mean to be honest that that's one of the ones I
• 14:10
  think a lot of people were looking Uber as far as like what
• 14:14
  are they doing. And when that came out, it was
• 14:16
  huge for everyone because they got to see like, OK, this is how
• 14:20
  you guys are viewing it and this seems very advanced compared to
• 14:24
  me and my Jupiter notebook. Yeah, and honestly, I've been,
• 14:27
  like, humbled by the. Number of people who said ohh
• 14:32
  yeah we just read that blog and just tried to rebuild that exact
• 14:35
  thing because it's. I mean it's not it's honestly
• 14:38
  not a bad plan especially like a couple of years ago like when we
• 14:43
  did it at Uber it was there was not even tools you could buy
• 14:46
  that would do this stuff. So.
• 14:49
  But we had when we did the blog post, we had a lot of people
• 14:51
  reach out to us and they were saying like, I'm trying to solve
• 14:53
  the same problems. Can you either like, hey, can
• 14:55
  you come work at my company or can you open source this thing?
• 14:59
  Can we work together? Kind of thing?
• 15:00
  And that's when you knew you were like, Nope, I'm going to
• 15:02
  start a. Well, yeah, actually I was like,
• 15:05
  oh, OK, this is a problem. This is not just a sort of
• 15:07
  Google soda Uber, it's some advanced companies are dealing
• 15:10
  with this now. But if you talk to every
• 15:12
  company, they want to do production machine learning.
• 15:15
  And so this is an important thing for.
• 15:19
  And for the industry in the future, it's an important
• 15:22
  problem to solve. And so we built, so basically
• 15:25
  myself and lead at Uber, we left and started Tecton.
• 15:30
  And So what we've done in Tecton, the quick pitch for
• 15:33
  Tecton is just the feature platform that you would have at
• 15:37
  a Google or a Facebook or an Uber kind of thing.
• 15:41
  That's what we make available to you.
• 15:42
  So you don't have to build it, you don't have to spend hundreds
• 15:44
  of engineers doing that. And that means you get to build
• 15:47
  better features, you get to build, and that means better
• 15:50
  models. You get to do it much faster and
• 15:52
  you get to do it with fewer people.
• 15:53
  So the same people you have can work on doing more ML and stuff
• 15:56
  like that. So that's I'll kind of constrain
• 15:59
  the tecton pitch to just that. But that's like the kind of
• 16:01
  outcomes that we try to drive. And so over the past handful of
• 16:05
  years we've seen just to finish up the history thing, which is
• 16:08
  kind of I appreciate is now going too long.
• 16:10
  But we've seen it go from, we've seen it go from like this mode
• 16:14
  where people were like I've never heard of this feature
• 16:15
  store thing. I don't know if I'm ever going
• 16:17
  to have this problem to Oh yeah, I heard a feature story, got a
• 16:20
  couple of questions, I'm not too sure.
• 16:22
  So now people are like, hey, I've got like 2 feature stores
• 16:25
  I've built before. We hate building the feature
• 16:27
  store. It's a pain in the ass for us
• 16:29
  and we wanna we wanna buy like a more comprehensive feature
• 16:33
  platform, here's our specific requirements and can we try out
• 16:37
  Tapton kind of thing. So the industry has very much
• 16:41
  matured and I think that's there's two reasons for that.
• 16:44
  One, because we kind of like helped inform people about, hey,
• 16:48
  there's a solutions to these problems.
• 16:49
  But second, just because people have, you know, they've gone
• 16:52
  through the stages of. Animal OPS, maturity.
• 16:54
  And you know, the first thing you do is just like, can I build
• 16:56
  like a very simple model in a Jupiter notebook?
• 16:59
  But then you get to this point where, like, I'm trying to
• 17:00
  productize this stuff and do it efficiently and reliably and
• 17:04
  keep it accurate and stuff like that.
• 17:06
  And there's problems that you only encounter when you're at
• 17:09
  that stage. And a lot more people are
• 17:11
  getting to that stage. And that's when a feature
• 17:13
  platform becomes super, super useful, dude.
• 17:17
  So true. And I love a few things that I'm
• 17:20
  just gonna like, try and talk about from your summary.
• 17:24
  One is the operationalizing of the data, and that data pipeline
• 17:31
  needs to be on point. And how many people have felt
• 17:36
  the pains of that when it's not That used to be, I think one out
• 17:41
  of every 3 posts in the ML OPS community, something like, hey,
• 17:46
  so my data scientist kind of fucked things up, how do I get
• 17:49
  out of this mess? And that's that's like the main
• 17:54
  thing that we're seeing. And now I do appreciate too that
• 17:57
  a lot more education has gone out there.
• 18:00
  So people understand the ways to do it.
• 18:03
  Whether or not they follow along on these different practises is
• 18:07
  their choice. But at least people understand
• 18:10
  that, whoa, there's a better way.
• 18:11
  And look here. They can reference different
• 18:14
  blog posts and they can see that there are different tools out
• 18:18
  there and so they can understand it better.
• 18:21
  And then the last thing that I was going to say is that just on
• 18:25
  the, I'll sing your praises on Tecton too, when you were
• 18:28
  talking about how you went and you did it, and I think it's
• 18:34
  almost in a way better because you're 100% focusing on feature
• 18:38
  platforms. You're not focusing on the whole
• 18:41
  thing like you were doing with Michelangelo.
• 18:44
  You're just saying, hey, there's this really important slice that
• 18:47
  we saw was so difficult to get right.
• 18:51
  And we're going to just focus on that 100% and make sure that we
• 18:56
  can do that right. So I do respect that you went
• 18:59
  about it that way. Yeah.
• 19:00
  Thank you. And I do think we're very much.
• 19:03
  Specialist there like, I think. Like the thing I like to say,
• 19:07
  when companies come to us and say, hey, we have this, like,
• 19:09
  wacky problem because people's problems are all different, but,
• 19:12
  you know, have this weird data source you never heard of before
• 19:14
  or you've got this nuance in how we calculate our features and
• 19:18
  stuff like that data from the NSA we can't tell anybody about.
• 19:22
  The secret? We can't tell you what we do,
• 19:23
  but we need you to solve the problem and.
• 19:27
  You know what? I what I say is there's not
• 19:28
  another team in the world that spends that thinks about this
• 19:32
  problem more than we do. So yeah, come chat with us and
• 19:36
  it's not gonna be like if you talk to AWS and you talk to a
• 19:38
  sales guy and it's like. The side thing of the 1% of this
• 19:42
  time of his training class time thing, yeah, it's this is all we
• 19:47
  do every day and we know all the nuances of what all the teams
• 19:49
  have done. So anyway, so we're specialists.
• 19:51
  We focus on it and that allows us to have like that's our
• 19:54
  strategy to have a better solution here all around
• 19:57
  operationalizing the data for ML.
• 19:59
  Yeah. And I think there was another
• 20:02
  thing that I wanted to ask about when it comes to this
• 20:05
  operationalizing how have you seen people go from?
• 20:09
  Either 0 to one and how that looks compared to the one to N.
• 20:15
  Yeah. So, good question.
• 20:18
  So we kind of help with both stages and we have customers who
• 20:22
  are going through both stages of that like going through that
• 20:24
  journey, both stages of that journey and when you're OK.
• 20:28
  So let's talk about stage zero. Stage zero is you have a
• 20:31
  prototype but you don't have anything in production.
• 20:34
  You just never had anything in production at all and maybe
• 20:37
  you're. And and you can break this
• 20:41
  problem into a few different ways, but one is you're just
• 20:43
  trying to. Put you're trying to productize
• 20:45
  something and generate automated predictions, regular predictions
• 20:50
  with it, and so folks need just a way to standardise.
• 20:54
  Here. This is my pipeline that runs
• 20:57
  maybe. I have my organisation has a
• 20:59
  number of models and they want to reuse different parts of the
• 21:01
  pipeline. So I have the one feature that's
• 21:03
  used across multiple models, and then I have my model and there's
• 21:08
  this notion of productizing it. And so sometimes the
• 21:11
  productization means I'm doing predictions in real time.
• 21:15
  So a lot of our use cases are like.
• 21:18
  Like someone's logging in and we want to determine is this
• 21:21
  actually the real person, is it fraud or not?
• 21:24
  Or someone's loads the website and we have to make a bunch of
• 21:27
  recommendations, we have to compute some recommendations to
• 21:29
  show them or we want to do some personal personalization thing
• 21:33
  and that all happens in the moment, like the users logging
• 21:36
  in. So you need to be fast, low
• 21:38
  latency and that. So that's kind of just like the
• 21:41
  basic set of things is like the core things connecting to the
• 21:45
  data, computing features, storing them.
• 21:49
  And serving them and retrieving them in real time and retrieving
• 21:52
  them offline and production offline for model training.
• 21:56
  And and a lot of companies are in this stage, they're going
• 22:01
  through this journey. But there's kind of this,
• 22:03
  there's a difference between going from or there's like 2
• 22:06
  dimensions here. There's not just like going from
• 22:07
  zero to that point, but there's also like, I think of it as an
• 22:11
  orthogonal dimension of getting to that point, but being able to
• 22:14
  do it like repeatedly across many teams in your company, kind
• 22:17
  of like the enterprise thing. So we have a lot of like fortune
• 22:20
  companies, customers who they're not doing anything more advanced
• 22:24
  than that. They're not doing like real time
• 22:26
  streaming real time. Calculation stuff, super
• 22:29
  optimised performance or whatever, but they're just like,
• 22:32
  look, we just have these features and they need to be
• 22:34
  reusable across 100 teams and they need lineage and they need
• 22:38
  monitoring all based plugged into this stuff and they need to
• 22:41
  be able to retrieve it online. So that's like the the very
• 22:45
  proto feature store, the the basics and then and then it kind
• 22:49
  of branches off from there. So what I meant by like
• 22:52
  everybody has different problems.
• 22:53
  So it's kind of stage two of sophistication.
• 22:56
  There's a lot of different kind of ways that things get more
• 23:00
  advanced. Or rather, you can think of it
• 23:02
  as like people's requirements get more intense.
• 23:05
  But not everybody. It's not just like everybody has
• 23:07
  at the same thing. So some people, so I'll just
• 23:09
  talk through some examples of of requirements that many use cases
• 23:14
  have. And maybe if you're listening to
• 23:15
  this, you're going to be like, yeah, I have #2, #3, but I don't
• 23:18
  have 1/4 and five. But some things are like, I've
• 23:22
  got to plug into a stream of events and I want to turn that
• 23:26
  into a feature and have that updated really quickly.
• 23:29
  So I want to continuously calculate some features.
• 23:34
  On a stream, Another one is. I have you know this user is
• 23:38
  logging in and I've got some information from them that I
• 23:41
  only get in real time. I can't precompute these
• 23:43
  features. So these are coming in at at
• 23:46
  inference time. Imagine I'm signing up for
• 23:48
  insurance and I'm typing in all of my details into the web form
• 23:52
  and I press submit. So I need to calculate that
• 23:55
  stuff in real time and so that's another set of requirements.
• 24:00
  We have a lot of customers who are kind of like large tech
• 24:04
  companies that have are, are are basically like B to C like
• 24:08
  consumer software companies that have huge scale.
• 24:12
  So this scale thing is really interesting too, like how do
• 24:15
  you, OK you get all this data, but how do you serve this data
• 24:19
  fast enough and cost efficient way at gigantic scale.
• 24:23
  So there's a lot of the problems that we solve are around doing
• 24:27
  all of the these data pipelines that I just mentioned.
• 24:30
  Doing them in a very cost efficient way, so you're not
• 24:33
  wasting a bunch of money and so, so it actually scales the cost
• 24:37
  you have per prediction effectively makes sense for you
• 24:41
  because we have a lot of customers who are just like,
• 24:44
  hey, we're trying to add these signals to our models, we're
• 24:47
  trying to build these pipelines, we're trying to build this
• 24:48
  stuff, but it just ends up like literally costing more than
• 24:51
  getting a good prediction is even worth for us.
• 24:54
  And so there's a lot of opportunities, and these are the
• 24:57
  repeated problems, to build in such optimizations into a
• 25:00
  feature. Platform to have the right cost
• 25:03
  optimizations on the information retrieval side, the serving side
• 25:07
  with the right types of caching and the right types of using
• 25:11
  certain integrations between the data pipelines and the online
• 25:15
  stores to efficiently load them up into the online stores and
• 25:18
  stuff like that to allow for. Like a more cost efficient
• 25:24
  solution there. So those are kind of like a
• 25:26
  couple of the other domains that are areas where people we see,
• 25:31
  people like push the limits. Another is, hey, my feature I
• 25:33
  need you to go call out to another system like to assist in
• 25:37
  that provides credit scores or something in real time, get that
• 25:40
  score, do something with it and pass it along.
• 25:44
  And then of course there's all stuff that we can talk about
• 25:46
  too. But so the point is, there's the
• 25:49
  kind of the basic mode and then if you have.
• 25:52
  If you kind of go to anything that's a like a little bit, you
• 25:55
  know, once you get there, you're like, OK, I want to do something
• 25:57
  better. And then there's a variety of
• 25:59
  different ways you can go a little bit more advanced and if
• 26:01
  you go to the feature platform, the feature infrastructure they
• 26:05
  have. Meta from Google, stuff like
• 26:08
  that. You know they've got an awesome
• 26:09
  solution for all of that and that's how they power the news
• 26:11
  feed and the ad system and search and all of that.
• 26:14
  And that's what we're building to make it really easy to for
• 26:20
  anybody to adopt that. Because these problems,
• 26:23
  especially as soon as you get past that basic mode, which is
• 26:25
  by itself not super easy, but you get past that, those have
• 26:28
  become really hard data engineering problems.
• 26:30
  And we see when people are kind of like a lot of our customers,
• 26:35
  when they become our customers, they're not at stage.
• 26:37
  You know, they're at stage 1. They're like, hey, I've kind of
• 26:40
  got this basic thing in place and now, you know, it was like
• 26:43
  easy for me to get here and then the difficulty for me to get to
• 26:46
  the next stage is like way higher and now I need help.
• 26:49
  They're starting to scope their work and they're like, ohh, this
• 26:53
  is gonna take a few sprints. I didn't realise what I was
• 26:56
  getting myself into. Yeah.
• 26:57
  And it it's funny you say that because I in one of our pancake
• 27:01
  stack sessions where we basically break down what
• 27:06
  everyone's stacks are and one person will present and it's
• 27:09
  it's all off camera and off no recording.
• 27:12
  So it's kind of like a ohh, a nice little intimate thing I
• 27:17
  could say I don't want to blow up the spot of the person who
• 27:21
  was saying this, but you know them.
• 27:23
  And I can tell you after we hit recordings of who they are, they
• 27:27
  were talking about how much of A the their effort of trying to
• 27:32
  build a feature store was death by 1000 cuts.
• 27:36
  And it really felt like it wasn't like there was this one
• 27:42
  really hard problem that they had to sink millions of hours
• 27:45
  into. It was just like as soon as they
• 27:48
  took one step forward, they were getting bombarded by all kinds
• 27:52
  of different things. And then they put out all these
• 27:54
  fires and the next time they took another step forward, boom,
• 27:58
  there was 20 other things they had to focus on.
• 28:00
  Yeah, and. I and I think so I totally
• 28:03
  agree. And so like one thing to to
• 28:06
  share there is so some of our customers are like really
• 28:10
  advanced tech companies that everybody knows that are like
• 28:12
  some of the biggest scale tech companies and.
• 28:16
  And they have multiple feature feature platforms, feature
• 28:20
  stores, different systems to power their features internally.
• 28:23
  And they they're at this point where they've been like, OK,
• 28:27
  we're not going to keep building this stuff.
• 28:29
  It's just constantly no one's ever happy with the solutions we
• 28:31
  build internally. The maintenance cost is so high
• 28:34
  for it and it'll take us so much time to build, you know to the
• 28:39
  next stage and what are we going to do higher like a huge team to
• 28:43
  do that. And just it's.
• 28:44
  So the point is it's gotten to the point where.
• 28:46
  The requirements like what people are trying to do with
• 28:48
  Mill has gotten to a level where they need a really good system.
• 28:54
  And the and then the options, the commercial options are also
• 28:58
  there. So the kind of build versus buy
• 29:01
  calculus has really shifted in a big way in the past couple of
• 29:04
  years where and This is why I was saying earlier, Michelangelo
• 29:06
  yeah, it made sense for us to build everything because the
• 29:08
  only company that was around was like Data Robot at the time and
• 29:11
  and also like Uber didn't have any.
• 29:16
  Like buy buying software bone in its body at the time.
• 29:19
  So it wasn't really a question for us, but I still would have
• 29:22
  built stuff at that time. Today, I definitely wouldn't.
• 29:25
  Just the other kind of anecdote I'll say to back you up on that
• 29:28
  is I talked to a lot of companies, I talked to a lot of
• 29:31
  platform teams. So the ML platform teams where
• 29:33
  they I just remember this distinctly, this one
• 29:35
  conversation with the ML platform lead and he goes, look,
• 29:38
  we have three models running on our platform.
• 29:40
  We're supposed to get to 30 by the end of the year.
• 29:42
  But I'm afraid to take on #4 because it's just.
• 29:46
  Gonna be like such another burden for us and the whole the
• 29:49
  kind of like point there that he was saying is his his people.
• 29:53
  The platform team doesn't have the leverage to scale with to
• 29:57
  scale with the business to scale with the customer demand because
• 30:00
  they're doing so much manual stuff on a per customer basis.
• 30:04
  So they would have to grow their platform team proportionally.
• 30:07
  But a lot of people who run platform teams are in this
• 30:09
  position where they're not going to get TENEX headcount.
• 30:11
  You know you're not gonna get one head this year.
• 30:12
  So what can you do to get more value and enable and get more
• 30:17
  leverage? From the few people you do have.
• 30:19
  And so we can talk a little bit about how we do that, but that's
• 30:22
  a really big focus. That's like a lot of our
• 30:24
  conversations, dude. It goes back to that Z axis you
• 30:27
  were talking about, how there's not only going forward, but it
• 30:31
  is looking at it in this way of, OK, if I'm going to do this, how
• 30:35
  can I make sure that I can do this repeatedly?
• 30:38
  Yeah. Yeah, yeah.
• 30:39
  And people care a lot about we talk about getting accuracy,
• 30:42
  getting reliability in your production system and then
• 30:45
  getting repeatability in your. You know your MO workflows.
• 30:50
  So you mentioned that there was like multiple feature platforms
• 30:54
  that some of these companies you talk to have.
• 30:57
  What does that even look like? That sounds like such a nuisance
• 31:00
  and it also sounds like how did someone let it get to that
• 31:03
  point? But what does it look like in
• 31:05
  practise? When?
• 31:06
  Good question that. So I so we're kind of now at a
• 31:10
  point in the Teton journey where we've.
• 31:14
  Been through we've we've really deeply understood a lot of what.
• 31:20
  When these big companies have done in their multi year history
• 31:25
  and here, So here's my best understanding of it and I'll
• 31:27
  tell you like the story. And then if you're going through
• 31:31
  this story, if you're listening, going through this story, maybe
• 31:34
  you can have the foresight to just to kind of connect to some
• 31:38
  stage of this story and be like oh that's kind of where I am,
• 31:40
  maybe let me try to avoid making certain mistakes.
• 31:43
  Yeah, hit the parachute now. Yeah, So I think.
• 31:46
  The first thing that happens is like where does how does a
• 31:49
  platform team happen? Because we always try to talk to
• 31:52
  platform teams, but but like what is a platform team?
• 31:56
  And so the way I see platform teams evolve, I kind of think
• 31:59
  about it like in biology there's like Cell and you have meiosis
• 32:02
  and like split into two. No, I will not recommend your
• 32:07
  podcast as the MLE Engineering TQB Bank.
• 32:11
  I'm definitely not wanting other engineers to have all this
• 32:15
  relevant information and I will not give my name which is Lena.
• 32:19
  So definitely I do not recommend this podcast.
• 32:28
  So what happens is, you know, you start with like one person,
• 32:31
  one data scientist, and they're building the model and then it's
• 32:34
  like, oh, this is like the early stages of your company or your
• 32:39
  business unit or whatever. And then it's like, OK, we
• 32:41
  should try to actually, there's something here.
• 32:44
  We should invest in this machine learning thing.
• 32:46
  OK. We need some engineering.
• 32:47
  So still on the same team, there's two people and then and
• 32:50
  then it kind of grows and then OK, well these people are going
• 32:53
  to kind of like. Because in building the the
• 32:55
  production side of it and these people are going to focus on the
• 32:57
  modelling side of it. And then you have maybe the
• 33:00
  modelling people start focusing on 2 problems and so then you
• 33:03
  have the modelling and the production people and then they
• 33:06
  make the production people like infrastructure team and so or a
• 33:10
  platform team and some fortune companies is kind of like it.
• 33:14
  So it splits off. And then that infrastructure
• 33:17
  team might be like the imagine all these problems are like
• 33:20
  fraud that we were talking about.
• 33:21
  It's within the business unit, but then you have, then you have
• 33:24
  a personalization unit, business unit and you have your payments
• 33:27
  business unit or whatever across multiple business units and they
• 33:30
  go through the same thing, right.
• 33:31
  And so they have these platform teams and then these like
• 33:34
  business units, specific platform teams go like, hey,
• 33:37
  we're all building a training platform, we're all building
• 33:40
  basic data pipelining for ML models.
• 33:43
  Let's have kind of like a central team and sometimes.
• 33:45
  All at a centre of excellence too.
• 33:47
  And so those infrastructure groups pull out and you have
• 33:50
  multiple layers of platform, team and then you get to and so
• 33:54
  it's just kind of like different layers of specialisation.
• 33:56
  And this is not bad, I'm just saying like how it goes.
• 33:59
  And so, so that now to turn into, so to turn to how like
• 34:06
  feature infrastructure kind of gets to a messy state and
• 34:11
  there's probably a very similar thing with data infrastructure
• 34:15
  as well, but. What happens with feature
• 34:17
  infrastructure is you know that first team that that fraud
• 34:21
  infrastructure, ML infrastructure team, they're
• 34:24
  like ohh, we got this like specialised feature nobody else
• 34:27
  has. It's we've got to process things
• 34:28
  in real time. So they build a little like
• 34:30
  thing that runs Python in real time and then just like their
• 34:34
  little fraud modellers use it in production and then and then you
• 34:38
  have the personalization people like six months later say hey I
• 34:42
  want to do real time features too.
• 34:44
  Instead of me building this thing, let me just use the.
• 34:46
  Nice thing. And so then the Frog guys thing
• 34:49
  was never designed for being a really nice like horizontal
• 34:53
  system. It was just the one engineer who
• 34:55
  built it in some hacky way, using some weird programming
• 34:58
  language maybe. And so the net and so then, you
• 35:01
  know, maybe they start depending on that other business unit or
• 35:03
  it gets kind of pulled out here. But that process kind of happens
• 35:07
  in parallel for different types of requirements.
• 35:10
  So you end up, I'm thinking of a big payments company that have
• 35:14
  one kind of feature processing infrastructure that's like for
• 35:17
  the streaming stuff that everybody hates now because it's
• 35:21
  like old and it was built for like this one team and they have
• 35:24
  to do all this weird munging their data in a weird way and
• 35:27
  they're all their modellers have to lend this weird.
• 35:29
  System to be able to create features for their thing.
• 35:31
  And then there's another one for real time features, then there's
• 35:34
  another one for features that get pulled from Snowflake.
• 35:37
  And then there's another one for features that are processed
• 35:40
  using their data lake. And so there's this big like
• 35:43
  kind of consolation of like different tooling that don't
• 35:46
  work well together. And so it didn't.
• 35:48
  When you look at it as a whole, you didn't really solve the
• 35:50
  problem of like building a training, training data set and
• 35:53
  serving simply in production. You just kind of changed the
• 35:57
  problem from a bunch of disparate data sources to a
• 36:00
  bunch of. Chunks of feature infrastructure
• 36:02
  and when they get to that point, they're like, OK, we could
• 36:06
  either do this thing where we take on a huge multi year
• 36:08
  project to rebuild all of these things in one thing or we can
• 36:10
  just consider buying it. Yeah.
• 36:14
  So if you're kind of. Going through that or you're
• 36:15
  seeing the early stages of that, you know, at least come chat
• 36:19
  with us, yeah. I think a few people that are
• 36:23
  listening are probably getting PTSD after you explaining that.
• 36:27
  Because I know certain people, engineers, who have left their
• 36:32
  jobs because it's just like, oh, this has gotten to be too much
• 36:36
  and leadership doesn't understand the value of us
• 36:41
  spending the time to go and make this right.
• 36:44
  They're just like, it's working right.
• 36:46
  Let's just keep going. Yeah, and I found,
• 36:49
  interestingly, when we were starting Tecton, I spent some
• 36:52
  time talking to data scientists at a bunch of insurance
• 36:55
  companies. And this one insurance company
• 36:57
  told me that 11 out of 12 of their core data scientists left
• 37:03
  in the past year. And all of them cited poor data
• 37:07
  science infrastructure at the time.
• 37:09
  There's a couple of years ago. But like, that's how much this
• 37:11
  stuff matters. It's not like a side thing for
• 37:14
  people and it really affects people's productivity.
• 37:17
  So anyway, if you're listening to this, you're probably
• 37:20
  preaching to the choir. You want to have good tools so
• 37:22
  you can build stuff, go fast kind of thing.
• 37:23
  And that's the point here, exactly.
• 37:26
  So dude, I want to get into this online offline stuff and the
• 37:32
  play between the two and what are the most common choices
• 37:37
  here? Because I understand it as this
• 37:40
  that the online stuff, the very low latency that you need are
• 37:46
  things like fraud that you're mentioning.
• 37:48
  I mean I guess loan scoring or maybe not.
• 37:52
  Then you have recommender systems like got to know what's
• 37:56
  on my Netflix page or my YouTube page or like my TikTok for you.
• 38:01
  And then I imagine there's a few very clear use cases for this
• 38:05
  low latency like real time stuff.
• 38:07
  And I'm talking real, real time because when we get on here,
• 38:10
  some people say real time and they're like, yeah, like every 5
• 38:12
  minutes. And that's not the real time
• 38:14
  that we're talking about. What is the real time that you
• 38:16
  talk about and what are those use cases?
• 38:19
  And then what are these offline store use cases?
• 38:23
  Yeah. OK, so.
• 38:23
  I think like. So your question is kind of
• 38:26
  like, hey when you say real time, what do you mean by real
• 38:28
  time and what what fits into that like band of use cases.
• 38:33
  And so you can think of it as like a from a spectrum of like
• 38:36
  don't need real time at all. I make one prediction per year
• 38:42
  and don't care about real time at all, and I feel like those
• 38:46
  are the ones where it's like in. Organic biology or stuff like
• 38:53
  that where you're just exploring different protein folds or
• 38:57
  whatever, like I OK, so let's. Actually make it even further
• 39:00
  distinction then. So I think there's, I think
• 39:03
  there's analytical I break it into like analytical machine
• 39:06
  learning or analytical AI where you're doing exploration, you're
• 39:09
  not really productizing and putting a thing on a schedule.
• 39:12
  You're not building a system that will live without you being
• 39:15
  there that day. And then there's operational
• 39:19
  model and so operational ML. The way I think about it is
• 39:22
  like, it's something where you know, it's productionized.
• 39:25
  It's like it's running, you know you're sick that day.
• 39:27
  It's still going to run, right? And.
• 39:31
  And so, so an example of the analytical one is, OK, we're
• 39:35
  doing like drug discovery or something, Let's write this new
• 39:38
  script, let's press run and see what comes out at the other end
• 39:41
  or let's, let's analyse finances for our company and run it and
• 39:45
  see what happens, right. And that's OK because if it if
• 39:48
  it fails, you just press retry and you're right there, it
• 39:51
  doesn't. And that's great and there's a
• 39:53
  lot of tools specifically for that.
• 39:55
  But when you're building like a product that is powered by.
• 39:59
  AI powered by machine learning you don't want and it's an
• 40:02
  important business process you don't like.
• 40:04
  You know, if you're writing down your business process, there's
• 40:06
  not like and then you ask the guy to go run the script.
• 40:09
  You'd have it all kind of like productionized in some way.
• 40:12
  And so there's a spectrum though of needs with respect to speed.
• 40:18
  So you can go from there. There's, you know, on one end
• 40:21
  it's, hey, this thing runs once a year.
• 40:23
  So I'll give you a good example of one of our customers for that
• 40:25
  they compute. A bunch of predictions for they
• 40:29
  renew their insurance company, they renew their users, their
• 40:34
  customers once per year, their policies.
• 40:36
  And so they have a big batch job they run for however many
• 40:39
  millions of people at that point of the year.
• 40:42
  And they want to make sure it works at that point in the year,
• 40:45
  but they don't care what happens in other days.
• 40:48
  They kind of do because they do like 1/3 hundredth of people
• 40:52
  every single day because it's evenly split out.
• 40:54
  But it's a batch job and if it goes slow it doesn't really
• 40:56
  matter On the other side of things, the real time side.
• 41:01
  Think of like, um, well actually let's get there.
• 41:04
  So then another kind of stage is like a workflow where you do an
• 41:08
  action and you have to kick off something.
• 41:11
  Imagine I created an account in your company and you want a
• 41:15
  whole business process to get started.
• 41:17
  We're going to, you know, apply for a mortgage for you and you
• 41:22
  know it's not going to happen. You get an instant response, but
• 41:25
  kicks off a process, talks to this other company, whatever.
• 41:28
  It can take 5 minutes and maybe it'll take an hour, but it's
• 41:30
  kind of an asynchronous process. And then another one is 1 where
• 41:36
  you are making a prediction, someone, someone logging in and
• 41:40
  you want to show them some recommendations.
• 41:43
  So this is even more real time. You want to show them some
• 41:45
  recommendations or let's say it's a purchase.
• 41:48
  You want to score a purchase and say, hey, is this fraud or not
• 41:51
  fraud? But maybe there's not a lot of
• 41:54
  signals that you get in real time, but you have a whole bunch
• 41:56
  of historical, like recent historical data.
• 41:58
  So you've gotten these streams of like, hey, how many purchases
• 42:01
  has this guy made in the last five minutes, right.
• 42:03
  And so maybe you're always computing the that that, that
• 42:07
  count for the last five minutes. And when they're logging in,
• 42:10
  you're not computing that live, but maybe you're serving that
• 42:13
  value. And so it's not super, super
• 42:15
  real time. It's near real time.
• 42:16
  That's how we talk about it. And then all the way to the most
• 42:19
  real time is imagine on Visa or something.
• 42:23
  So I have the credit card, I need to know like where is the
• 42:26
  merchant, how much is the purchase.
• 42:28
  I got a feature, is that all in real time, it has to happen in
• 42:30
  less than 50 milliseconds so I can generate a prediction and
• 42:33
  get back to you. And so that's like super real
• 42:35
  time. And so you can imagine we can
• 42:38
  take like any use case and you can at home think of like, hey
• 42:42
  like what is the use case that I have for my ML problem?
• 42:46
  And like which bucket does it fit into?
• 42:48
  And sometimes there's not just one bucket.
• 42:50
  Sometimes a bucket like you're you start your problem here, but
• 42:53
  it's maybe it's a fraud problem. And you're like, you know what,
• 42:55
  my fraud model would be better if I moved it over here to make
• 42:58
  it more real time because I could incorporate better signals
• 43:00
  and be more accurate and stuff. It's just some kind of natural
• 43:03
  journey. But the interesting thing about
• 43:06
  this spectrum of going from this batch to super real time, and
• 43:10
  you can think of it as like different speeds of decisions
• 43:13
  and different requirements and different kinds of decisions
• 43:15
  like that, is that there's there's sometimes different
• 43:19
  under the hood, you know, data processes that power and you can
• 43:22
  take two approaches, right? You can take an approach, you
• 43:25
  can take two approaches to solve those needs.
• 43:28
  So you can take an approach where you go, OK, I'm going to
• 43:31
  do 1 system, I'm going to build 1 system using just my data
• 43:35
  warehouse just from my batch stuff.
• 43:37
  And then I'm going to do some really custom thing from my
• 43:39
  super real time thing. Then I'm going to have another
• 43:41
  system here. And that's kind of how you get
• 43:43
  the patchwork of different systems.
• 43:46
  And the challenge is, OK, now you have a number of things to
• 43:50
  maintain. But secondly, it makes it really
• 43:53
  hard to, for your use case to upgrade in that journey.
• 43:56
  So you went from stage two and you built it on your system that
• 43:59
  doesn't talk to streaming systems.
• 44:01
  Now you want to start using streaming, OK, You've got to
• 44:03
  integrate with a whole new system.
• 44:05
  Maybe it doesn't talk to all the stack you are already using.
• 44:08
  That's a gigantic project. So a lot of companies that we
• 44:12
  talk to, they say, look, we actually don't do a tonne of
• 44:14
  real time stuff right now, but we want to, it's actually
• 44:18
  competitive. It's like our industry is going
• 44:21
  more real time and that's a competitive dynamic for us.
• 44:24
  For example, like loan decisioning.
• 44:25
  It used to be OK, I apply for a loan, get back to me and tell me
• 44:30
  if I'm approved or apply for insurance and we'll come back to
• 44:34
  me with a quote the next day. And then when your competitor
• 44:37
  goes cool like instant loan approval, then you've got some
• 44:40
  pressure to compete with them and go more real time.
• 44:42
  So a lot of use cases start at some point in that spectrum
• 44:45
  trying to go one or two stages more real time And and so when
• 44:49
  those companies come to us they say look we're really we've got
• 44:51
  a lot of use cases that are here or here or here, but we want to
• 44:54
  move them that way. So we want to be able to support
• 44:57
  what we can do today and then enable migration to more real
• 45:01
  time. And the kind of like focused
• 45:03
  approach that we take is we provide one platform that really
• 45:06
  like solves all of that and it's really trying to be best in
• 45:09
  class solution for doesn't matter where you are in the
• 45:12
  Super badge to super real time perspective, you know best in
• 45:15
  class in terms of like cost optimization, scale best
• 45:19
  practises built in and shareability of signals across
• 45:23
  the whole spectrum. So people say OK cool, I'm over
• 45:25
  here, but I know if I use this as my centralised platform I'm
• 45:29
  gonna, it's gonna be trivial for me to build these other things
• 45:32
  in the future. That makes sense for sure.
• 45:35
  And. I can totally see this idea of.
• 45:38
  We want to get to real time, but considering the way that we
• 45:42
  built things out right now in order for us to bring on that
• 45:46
  capability, it's going to be again, it's a whole nother
• 45:50
  undertaking because our offline store is not able to talk with
• 45:54
  the online store and we can't connect those two.
• 45:56
  It's like there's just a gigantic wall there and we can't
• 46:00
  figure out a way to hop over that wall or breakthrough.
• 46:04
  Yeah, and that's the thing that should just be it's a thing that
• 46:08
  she should be a problem or a requirement that's automated.
• 46:12
  There's best practises around it.
• 46:14
  You don't have to solve that. There's now it's solved problem.
• 46:17
  There's really easy ways to do it and it's hidden behind the
• 46:20
  scenes with the right platform. Hey, Josh is here.
• 46:23
  There's Josh. What's up dude?
• 46:26
  Sorry, I'm. Late Hey Mike, it's good to see
• 46:28
  you in virtual life as as opposed to real life with our
• 46:32
  kids in the park. Basically.
• 46:35
  How's, how's everything, Josh? Everything is great, man.
• 46:40
  Unemployment continues to be awesome.
• 46:42
  Thanks for asking. Yes.
• 46:44
  Love it. Yeah.
• 46:45
  Yeah, I wouldn't be aspired to be unemployed.
• 46:48
  I know, man. You're lucky you've figured it
• 46:49
  out. I am.
• 46:50
  No. I I, I'm just, I'm, I'm I'm a
• 46:52
  lucky idiot. But I, I, I talked to all My
• 46:54
  Portfolio companies and they're all like, you know, it's like,
• 46:56
  you know, so like what is your life like?
• 46:57
  Well, I play enormous quantities of Pokémon Go and I research
• 47:00
  World War 2. It's a very slippery slope,
• 47:02
  guys, from like Rick, Rick Atkins books to like, you know,
• 47:05
  I really need to plan a trip to Normandy and do some primary
• 47:08
  document research. So I mean, Demetrius, I think
• 47:11
  like you know, my 2 main themes for this, for this, you know,
• 47:15
  the call here is like I'm curious about.
• 47:19
  Kind of where feature stores are going to go, like feature stores
• 47:24
  now is like you know, feature definitions and then
• 47:28
  orchestration and kind of like coordinating the workflow across
• 47:31
  the offline and the online data stores and stuff like that.
• 47:35
  Would love to see a feature store, dedicated query serving
• 47:41
  engine that is like a thing that integrates the workflow
• 47:44
  together. And I know we don't do this
• 47:45
  today for various reasons. The day this year the data is
• 47:47
  there. We need to work with the data
• 47:49
  where it lies. But I am excited for an Apache
• 47:52
  Iceberg centric future where and again bias that.
• 47:56
  I'm a Tabular investor and Tabular announced to fundraise
• 47:58
  today. Where this stuff.
• 48:00
  Is open, the storage format is open to everybody and you can
• 48:03
  bring the right query engine for the job.
• 48:05
  And I'm just like I really want that to happen and I want to
• 48:08
  understand is that on the road map what what are the
• 48:11
  impediments to making that happen?
• 48:12
  Like that's that's kind of my my first big thrust here.
• 48:15
  Yeah, for sure. I think I well first thing to
• 48:18
  say is like I agree with you on that.
• 48:20
  So talking about where things are going, just a quick kind of
• 48:24
  like terminology thing. So yeah, we think of like
• 48:26
  feature store as the system that literally stores your data and
• 48:31
  offline and will put it into an online store.
• 48:34
  And we talk about feature platform as the kind of full set
• 48:38
  of workflows and the like, enterprise sharing and
• 48:41
  monitoring and stuff like that. And the part of the system, your
• 48:46
  feature infrastructure that's doing the regular feature
• 48:48
  computation, so the ongoing raw data to fresh features and often
• 48:56
  some kind of like optimised serving systems to for example
• 49:00
  in technology we have a whole bunch of specialised caching
• 49:03
  stuff that makes that can help to make it to your proper use
• 49:06
  case. And so I think your question was
• 49:08
  like I'll try to rephrase it and tell me if I'm like getting it
• 49:12
  right. But it sounds like you're like
• 49:13
  OK, why do I care about like having all my feet, all my data?
• 49:18
  Why would I put my data in this one system that kind of like
• 49:20
  locks it into this one system, whereas why not just have it?
• 49:25
  Why not have the kind of this open type of system where
• 49:28
  there's it's getting almost like a data lake centric version of a
• 49:32
  feature store where you have feature processing workflows
• 49:36
  like all the stuff I just mentioned, but they contribute
• 49:39
  to feature you know? And then my feature data kind of
• 49:42
  just gets stored in some central location that maybe you shared
• 49:46
  with some other data that can be retrieved for other purposes,
• 49:48
  maybe even for analytics or it can be similarly retrieved by
• 49:52
  different systems for different ML use cases.
• 49:55
  Is that kind of like the main gist of it I guess?
• 49:58
  I would give beyond in a couple of ways, I think.
• 50:00
  For what it's worth, right? So for instance, I love, I love
• 50:03
  a Doordash's blog posts about the various ways they have
• 50:07
  hacked Redis. And I've got to assume Techcom
• 50:08
  has done similar things like you.
• 50:10
  Yeah, I mean, you had we hired the guy who wrote that blog
• 50:12
  post. Ohh, that was great.
• 50:13
  That's. Like that that that's that's.
• 50:15
  So sorry again, I did not. That was an unintentional plug
• 50:18
  for techno. Sorry about that.
• 50:19
  But that's a fantastic blog post, right?
• 50:21
  Yeah. And I guess what I mean is like
• 50:22
  is there a point at which we want to go beyond either hacking
• 50:28
  Redis keys and structures and stuff like that or hacking the
• 50:32
  point in time join queries in the in the offline snowflake or
• 50:36
  Spark based feature store to an actual dedicated query engine,
• 50:39
  something that is designed? From the ground U for the online
• 50:43
  surveying to do the ingestion quickly from the bash offline
• 50:47
  system and also like a dedicated query engine for the point in
• 50:51
  time joins. I don't need to like hack it on
• 50:54
  top of SQL or Spark or whatever, I can just literally like this
• 50:57
  and it does it and it's optimised to do that.
• 50:59
  Anyway, that's kind of what I'm going for is like.
• 51:02
  Anyway, it's it's it's. Yeah, I think you're saying,
• 51:05
  hey, here's another way to I'm interpreting it like, OK,
• 51:08
  feature platform stuff is a lot of platform stuff.
• 51:12
  It's not coordinating other important data infrastructure,
• 51:16
  yes, but is there going to be feature platform specific data
• 51:21
  infrastructure? For example, a special serving
• 51:23
  layer or a special point in time joining sistant work feature
• 51:28
  retrieval engine that will emerge for supporting this
• 51:33
  process that these cases better? That's why I'm hoping.
• 51:35
  I'm hoping this is the lead a product announcement that you're
• 51:38
  about to do here on the podcast. You say well, well, yes, Josh,
• 51:41
  as a matter of fact we are you know getting, you know you have
• 51:43
  to do, but that's OK. So as it happens, I think
• 51:46
  there's the answer is yes and no on the OK, so, so on the real
• 51:51
  time on the online store side certainly.
• 51:54
  And so we do feel like a lot of the online infrastructure
• 51:58
  basically caching tools and the real time databases, they're not
• 52:04
  really optimised for the use case that a lot of our customers
• 52:08
  have, but a lot of times they're close enough.
• 52:11
  So what's happens today is that doesn't become the actual
• 52:15
  bottleneck for people like making progress in their ML
• 52:18
  projects. However, we do see and our plans
• 52:21
  are to build a proper serving layer and like a specific key
• 52:27
  value store tuned to this use case in the future.
• 52:29
  But that's not a like next six months kind of thing.
• 52:32
  Got it. On the feature on the offline
• 52:36
  retrieval side, we've been going more in the direction of
• 52:40
  managing and storing the data with a data model such that the
• 52:45
  such that it's easier for different kind of like the kind
• 52:49
  of open model you're referring to or we stored data in a delta
• 52:52
  live table, a delta table or an iceberg table and we want to
• 52:58
  provide a lot of different ways for you to interact with that
• 53:01
  system to with those data to process them with the right
• 53:06
  joins. So we're kind of moving away
• 53:07
  from the use Tecton specific SDK to pull this data and we want to
• 53:13
  make it easier for you use the tools that you have today.
• 53:17
  It's interesting we're enjoying that data to kind of get more
• 53:20
  along the more towards the vision of this feature.
• 53:23
  There's nothing special about this feature data, you just need
• 53:25
  to we're going to do the preprocessing to get it into
• 53:28
  feature data format and we do a lot of intermediate data values
• 53:34
  and stuff like that. But then we do a lot to just
• 53:36
  provide like a polished published finalised values and
• 53:39
  then you can interact with that. Maybe we'll give you a little
• 53:41
  bit of a library or something to do the joins, but the join
• 53:44
  enough is not super hard in a like most engines can do it,
• 53:51
  except for if you're doing like super gigantic like performance
• 53:54
  you have like you're hitting performance limits and then and
• 53:57
  then we've kind of done some special stuff with Spark there
• 53:59
  and maybe there's a thing we can do in the future to optimise,
• 54:02
  but that's not the goal today. I got you.
• 54:04
  That makes sense. I love it.
• 54:06
  Wait, Mike you? Mentioned how that specific
• 54:09
  thing. Isn't the bottleneck right?
• 54:12
  What do you feel like is the bottleneck that you hit up
• 54:14
  against much more than that? I think people have.
• 54:18
  I think a lot of people are just trying to get their system up
• 54:22
  and running in the 1st place, so it's pretty uncommon that.
• 54:29
  So there's a lot of like setup and a lot of like, hey, this
• 54:33
  thing needs to be able to integrate with all of my data
• 54:35
  sources because it's ultimately connecting to my complete data
• 54:38
  ecosystem, integrating that data such that it can deliver this
• 54:41
  data to my model. And there's always some snags
• 54:46
  there. And so we're trying to get that
• 54:47
  to be faster. What we're trying to optimise is
• 54:50
  you decide, hey, I need to use a feature platform from that date,
• 54:53
  how many days until you have a model running in production at
• 54:57
  the scale and speed that you want.
• 55:00
  And so that's kind of like there's a variety of integration
• 55:04
  challenges that we think we can speed up.
• 55:06
  And then as just mentioned, a second one is just like
• 55:09
  sometimes you have features that you're trying to express that
• 55:12
  are in your mind, wouldn't it be cool?
• 55:13
  I think this would be a predictive thing for my fraud
• 55:15
  problem if I could do this. And we're trying to continually
• 55:19
  extend the flexibility of our system to be able to support
• 55:23
  additional types of feature transformations.
• 55:25
  That is that tends to be like more of a bottleneck than kind
• 55:30
  of like the online performance. Because for us anyway, I think
• 55:35
  online performance is a bottleneck if you were to use a
• 55:38
  lot of other systems, even if you were to use Feast.
• 55:41
  But if you use technology, we've put a lot into all of the
• 55:45
  optimizations for the online system such that we're confident
• 55:50
  we could go DoorDash is not our customer, but I'm confident we
• 55:53
  could just plug into DoorDash and potentially outperform their
• 55:57
  custom solution that they put together.
• 55:59
  There is something. Josh, you you mentioned wanting
• 56:01
  to talk about LM's and. Feature stories and I think we
• 56:04
  cannot, like we would be remiss to not talk about that,
• 56:07
  especially as you were saying where feature stories going, How
• 56:11
  are you seeing them being used now?
• 56:12
  And so all with that, like Josh, I'll let you lead into this next
• 56:16
  question and where you want to take it.
• 56:18
  Yeah. Yeah, I mean, I mean like this
• 56:20
  would just be a huge thing. So I'll try to be as
• 56:22
  parsimonious as I can here, right.
• 56:25
  But I guess first first one was in the context of like building,
• 56:28
  deploying a large language model or system that integrates a
• 56:31
  large language model. What?
• 56:33
  Is the feature. What is a feature in that
• 56:36
  context? Is that the is that the
• 56:37
  embeddings? Is the prompt?
• 56:38
  Like how do I think about that? Like what's my mental model
• 56:41
  here? Yeah, good question.
• 56:43
  So obviously we've had to, we've been thinking a lot about this
• 56:46
  and I think the kind of we've talked to a lot of people
• 56:49
  building LLM applications and a lot of people are using Tecton
• 56:54
  to do that now. And one of the we have a blog
• 56:58
  post with Lane change people that kind of shows like the
• 57:02
  minimal example for this. But but I think the the key kind
• 57:06
  of like intuition here is forget about LMS for a second, what
• 57:09
  does like the feature infrastructure do?
• 57:12
  What's its relationship with the decisioning system, the model,
• 57:15
  it's providing real time context to that model, it's providing
• 57:18
  context to that model. So it's saying here's we went
• 57:20
  and interacted with the whole world and we've given you
• 57:23
  information about the world that you can make a decision on.
• 57:26
  And I think the same looms need that same service effectively.
• 57:32
  And you know, So what is, what is that context?
• 57:36
  It's a prompt. But you know, you may not need
• 57:40
  like a complicated data pipeline to say here is what the user's
• 57:47
  ticket said, what should I respond to, right.
• 57:49
  Yeah. But you may also not get the
• 57:52
  type of performance from the LLM that you the best performance
• 57:57
  that you could get. Compare that to here's what the
• 58:01
  users complained to us about, here's what their ticket said
• 58:04
  and here's what they said last time and here's how many tickets
• 58:06
  they've submitted in the past three days and here's when they
• 58:09
  signed up. And this is the most common way
• 58:12
  they use our platform and stuff like that.
• 58:14
  And so we think about it as like we can help the context that you
• 58:19
  pass into the model. So the prompt be highly enriched
• 58:24
  with relevant information about your business, the product, the
• 58:28
  user specific transactions, stuff like that.
• 58:31
  And it's basically all the same data supply chain that the
• 58:34
  feature infrastructure powers. You're just kind of injecting
• 58:38
  that data rather than as a a kind of like structured feature
• 58:42
  vector into ML model as kind of like, you know, you're kind of
• 58:47
  instantiating it as a snippet of prompts macro effectively into
• 58:53
  the prompt and we think so that's kind of like the the how
• 58:59
  the context powers the LLM in the future I think one of or
• 59:03
  like today. But one of the kind of
• 59:05
  interesting things that I've reflected on a lot is like, I
• 59:09
  think this is a really interesting question.
• 59:10
  One of the core values of feature infrastructure, one of
• 59:14
  the key requirements that it solves is this online offline
• 59:18
  thing. And because you want to have all
• 59:21
  this data such that you can train your model and LLM world,
• 59:25
  you're not trading model usually so.
• 59:28
  Yeah, so, So what do you need to have the tooling to support that
• 59:33
  type of that requirement if you're just building alms and
• 59:38
  what we learned after talking to a bunch of people is that.
• 59:41
  So while you may not be training a model, when people are
• 59:46
  building these LLM applications, they're still doing, they're
• 59:49
  still going through an offline development workflow that
• 59:52
  requires evaluation, prototyping and eval.
• 59:57
  And so they might say, hey, which prompt works better, which
• 1:00:00
  set of signals works better. I'm going to backfill these.
• 1:00:03
  I want to try it on these thousand historical examples and
• 1:00:06
  score them and see which one I like better.
• 1:00:08
  And basically that's what the feature story does So the
• 1:00:10
  feature platform is for. So we see like a lot of
• 1:00:13
  opportunity there as well to make really.
• 1:00:15
  Crisp user experiences. User workflows.
• 1:00:17
  For that, that's awesome. That's fantastic.
• 1:00:21
  Yes, I didn't even think about that.
• 1:00:23
  No, those examples are perfect. Yeah, that makes a tonne of
• 1:00:26
  sense. And.
• 1:00:28
  It seems like that there's a lot of surface area to cover with
• 1:00:34
  like the abilities that the feature store has.
• 1:00:37
  And right now, as you both know, rags are all the rage.
• 1:00:42
  And so it's like this is in a way it's it's a rag, but it's
• 1:00:47
  not your typical like hey, grab a vector database and throw it
• 1:00:51
  at the LM. And Josh, I know you got strong
• 1:00:54
  thoughts on vector databases, which, yeah, what do you think
• 1:00:57
  about vector databases? Ohh.
• 1:01:00
  Yeah, I feel like I'm. Being I'm being set up right
• 1:01:02
  now. This is, this is, this is not
• 1:01:05
  this is this is my got a lot of spicy takes so I wanna no.
• 1:01:09
  This this is my punishment for being 55 minutes late for the
• 1:01:11
  podcast, isn't it OK? See.
• 1:01:14
  Exactly, I guess like. I don't quite get it.
• 1:01:18
  I guess is kind of like why exactly do I need a dedicated
• 1:01:21
  vector database as opposed to PG vector or all of the good stuff
• 1:01:26
  in the Lucene, Elasticsearch, et cetera, et cetera, ecosystem
• 1:01:30
  like I'm just really sort of thrown here what is everyone has
• 1:01:34
  everyone kind of lost their minds I guess like I don't I
• 1:01:38
  don't quite get it that's that's maybe the the nicest.
• 1:01:42
  Way to say is that I am I am a dumb person and I don't
• 1:01:44
  understand it. Uh yeah, that's that's I I just
• 1:01:48
  don't get it. My my guess on my take on vector
• 1:01:52
  databases is that there's going to be there's the same people
• 1:01:58
  are going to have the same kind of use case very popular.
• 1:02:00
  So users give me something similar to this other thing.
• 1:02:02
  Yeah it's basically what they help you with and for the vast
• 1:02:07
  majority of use cases any most databases that have implemented
• 1:02:12
  A vector similarity in retrieval interface are going to be able
• 1:02:17
  to support those. This is why you think that.
• 1:02:19
  But I do think that there's if you're high scale company, you
• 1:02:24
  need it to be super real time. You know you have specific
• 1:02:26
  requirements and it may justify having a specialised solution.
• 1:02:31
  But I think you'll be in a situation where that will be the
• 1:02:34
  specialised solution. And for most people you can just
• 1:02:37
  use whatever your normal database is with a vector
• 1:02:40
  retrieval interface and that should be kind of fine.
• 1:02:44
  So that's kind of my guess is where things are going.
• 1:02:48
  But there's also just don't discount the kind of like the
• 1:02:51
  workflow around it too. So I think like a lot of vector
• 1:02:54
  databases are building really nice APIs that make sense to ML
• 1:02:58
  engineers. They're saying this is either
• 1:03:01
  this is my embedding, just figure it out and I want to use
• 1:03:04
  this algorithm and it makes it really easy to retrieve it.
• 1:03:07
  Or they're saying here's my new piece of data update, my
• 1:03:09
  embedding. And that's like some other stuff
• 1:03:12
  that's kind of handled in there that might make for the
• 1:03:14
  workflows nicer, easier to adopt, so faster to get started
• 1:03:18
  and more sticky longer term. So I'm not sold either way, but
• 1:03:22
  I do suspect that like a lot of the core retrieval stuff will
• 1:03:25
  just be able to be reused across many solutions will be able to
• 1:03:30
  support common use cases. I just feel like that.
• 1:03:33
  Kind of thing is like something that there will be a super based
• 1:03:36
  style company that comes and does this on top of Postgres and
• 1:03:39
  it'll have like this awesome feature.
• 1:03:41
  You'll have two really critical features.
• 1:03:43
  One is it'll have a great workflow for machine learning
• 1:03:45
  engineers doing like vector type operations.
• 1:03:48
  Two, it won't lose my data, it won't lose, it will keep the
• 1:03:52
  data. Is one of those things.
• 1:03:53
  It's kind of like a baseline. You know, very low on the
• 1:03:55
  Maslow's hierarchy of needs of data stores.
• 1:03:58
  Yeah, Are you worried about reliability and vector databases
• 1:04:02
  that you're referring to worry? About reliability all the time,
• 1:04:04
  man. I worked at Slack, for God's
• 1:04:06
  sake. Everything.
• 1:04:07
  Anything that can break is going to break as far as I'm
• 1:04:09
  concerned, right? Even S3.
• 1:04:11
  Like, what the hell happened yesterday?
• 1:04:12
  Like we lost W 2 and East One at the same time, right?
• 1:04:16
  That was. I mean, you talk about like
• 1:04:17
  times you're happy to be retired.
• 1:04:18
  Oh my God, that was amazing. Right.
• 1:04:22
  Like everything. Everything breaks.
• 1:04:25
  And so like the S3 and the Postgres and like all it's just
• 1:04:29
  it's old, safe, boring technology that doesn't lose
• 1:04:32
  data. It just it's like my warm safety
• 1:04:34
  blanket, Man, I'm I'm an old fuddy.
• 1:04:36
  Look at this neckbeard, right. Look at that.
• 1:04:37
  That's just like this is, this is just how I roll.
• 1:04:39
  This is who I am. You've got you've got your old
• 1:04:42
  friends that you're you return to and I do they form those
• 1:04:46
  relationships over many hours. I do the keyboard.
• 1:04:48
  It is entirely. Possible that 22 year old Josh
• 1:04:51
  Wills would be all about vector databases and be like clearly I
• 1:04:55
  need to write my own vector database.
• 1:04:56
  That's the obvious solution to this.
• 1:04:58
  But 44 year old Josh doesn't. Let's just I don't do that shit
• 1:05:01
  anymore. That's just not like, absolutely
• 1:05:03
  not. Yeah.
• 1:05:04
  Anyway, well, one thing that we've seen the kind of like an
• 1:05:08
  overlap with our world is, is this kind of core workflow where
• 1:05:11
  someone goes, hey, I want to retrieve, so I want to have good
• 1:05:15
  recommendations. So what I want to do is retrieve
• 1:05:18
  similar objects. I want to get a set of
• 1:05:20
  candidates effectively from my vector database, and the vector
• 1:05:24
  database becomes a candidate generator.
• 1:05:25
  That's how we think about it. And then use a feature store to
• 1:05:28
  enrich those objects. So provide a whole like 1000
• 1:05:32
  features for each one of them such that I can then score them
• 1:05:36
  and produce a much higher fidelity understanding
• 1:05:40
  prediction of you know is some what's the what's the PCR, is
• 1:05:44
  someone gonna click this add or this object or whatever this
• 1:05:47
  item and then rank of by that. And I think they've seen a lot
• 1:05:50
  of kind of compatibility in both systems.
• 1:05:54
  And I think there's still a lot of opportunity in joining those
• 1:05:59
  workflows together in some nice way.
• 1:06:01
  And that's something we're going to be investing in the future as
• 1:06:04
  well, spend a lot outside. Work on that problem slack.
• 1:06:06
  So love, love, love working on there.
• 1:06:08
  But it was, it was all of it. Canada Generation re ranking off
• 1:06:11
  of your feature store? Absolutely, 100%, yeah.
• 1:06:14
  Well, I'm as. You're talking about workflows.
• 1:06:16
  One thing that interests me is this idea of like templates and
• 1:06:20
  it seems like there are very clear.
• 1:06:23
  Problems that people are trying to solve for.
• 1:06:25
  Have you thought about template vizing stuff, Mike or?
• 1:06:29
  We thought about template vizing stuff when there's so many
• 1:06:32
  different like, so we thought about temporising things and
• 1:06:37
  we've shield away from it and so we've tried to stay a little bit
• 1:06:39
  lower level. We would like to, There's a lot
• 1:06:41
  of cool things that we can think about right now.
• 1:06:44
  For example, what if we just had a common set of fraud features
• 1:06:48
  or common set of fraud feature definitions that you could write
• 1:06:51
  onto your own data or a common set of third party features that
• 1:06:55
  are just available in technology that you could use.
• 1:06:57
  Or like a prototype fraud application and recommender
• 1:07:02
  application that's kind of built on.
• 1:07:03
  So all of these different things, but we find that like
• 1:07:06
  it's kind of unlikely for any of those to be completely just copy
• 1:07:11
  paste and just like stamp it in my own world and those when we.
• 1:07:15
  And so I like the idea of that, but I just haven't gotten to the
• 1:07:18
  point where I think that's actually the thing that's gonna
• 1:07:20
  unlock something for our customers.
• 1:07:23
  Our customers come to us and they're like, look, we've got
• 1:07:25
  this hard problem. We've been trying to solve it in
• 1:07:26
  our own world, but we just, we don't have enough people or we
• 1:07:29
  don't know how to do it in real time or whatever it is.
• 1:07:32
  And so it comes down to like the can this technically happen And
• 1:07:35
  then we enable them rather than like we just haven't had, we
• 1:07:40
  don't have enough time to build right the features.
• 1:07:42
  Writing the features is the easy part, and I think you're
• 1:07:44
  referring to as kind of the temptation of the workflows
• 1:07:48
  across different pieces of tooling like the vector database
• 1:07:50
  and the feature store thing was just referring to and maybe
• 1:07:53
  that's a more appropriate domain for it.
• 1:07:54
  But we haven't gotten there yet. Still.
• 1:07:57
  I mean still. Feels like we don't have it
• 1:07:58
  right yet to me. I don't feel like we have the
• 1:08:00
  right. We're still kind of groping
• 1:08:02
  around trying to find the right architecture of the right
• 1:08:04
  pattern and the right abstractions and stuff like
• 1:08:05
  that, and a lot of people are trying right now, which is
• 1:08:07
  great, but anyway, but I don't see it yet.
• 1:08:10
  Yeah. And I definitely feel like we've
• 1:08:11
  gotten through It's taking time, but I feel like we've gotten to
• 1:08:14
  the right place in the feature infrastructure, like the feature
• 1:08:17
  store, feature platform stuff. Yeah, the concepts are solid.
• 1:08:21
  And so we feel like we've converged there.
• 1:08:23
  And so now we're taking on exactly what you're saying.
• 1:08:25
  What are the abstractions for the next set of workflows, next
• 1:08:28
  set of use cases that people have?
• 1:08:30
  And it's an open question. So we're learning from the
• 1:08:32
  community just as much as anybody else, dude.
• 1:08:35
  Well. Mike, I said this to Kevin, I
• 1:08:37
  don't know, three years ago when he came on one of the first.
• 1:08:41
  Podcast that we did and Kevin, your cofounder, the thing that I
• 1:08:46
  would love to see that you I don't think you've implemented
• 1:08:49
  it yet but some kind of gamification on the amount of
• 1:08:53
  money that features cost or the amount of money that features
• 1:08:56
  will save or something like who has created the most features
• 1:09:00
  and then they get badges within Tecton or something.
• 1:09:03
  It's like you can see like all this person's been working hard
• 1:09:05
  look at they created thousands of features.
• 1:09:08
  Yeah, I think. I think there's.
• 1:09:10
  So it's a cool yeah. And it's the kind of thing that
• 1:09:13
  requires some I'm not sure exactly what it is but we had
• 1:09:21
  something like that at Google. So at Google you know you had
• 1:09:23
  this one big problem that you're trying to solve which is like
• 1:09:26
  adds click through rate and trying to make a better
• 1:09:29
  prediction. And then you have all these
• 1:09:30
  different teams that are like creating better features in the
• 1:09:32
  running experiments. Is this feature make us more
• 1:09:34
  accurate or not And and I think like when if you get to for your
• 1:09:40
  business, if you have the AI problem that is so valuable that
• 1:09:45
  it justifies having different teams constantly working
• 1:09:48
  independently to compete against each other to nudge things out
• 1:09:51
  to like a handful of percent and there's not an obvious
• 1:09:54
  bottleneck. But you're kind of like what do
• 1:09:56
  we do now let's try to make let's get creative then that's
• 1:09:59
  kind of like a like a fertile ground for that kind of kind of
• 1:10:03
  competitive dynamic. But I think most of the teams
• 1:10:05
  realistically are like, hey, how do we, how do we like, get this
• 1:10:10
  thing in production And a lot of.
• 1:10:12
  People are just kind of getting getting the first thing going
• 1:10:15
  and trying to go from 50% accuracy to 80% accuracy instead
• 1:10:19
  of 99 to 99.2. Hello, we've got.
• 1:10:25
  It we've got intruders. Basically the the time basics,
• 1:10:29
  they cut me off. They said I'm gonna go back on
• 1:10:32
  vacation mode. I love you guys.
• 1:10:34
  I love to watch. This is a good convo.
• 1:10:36
  Yeah, love it. Thank you so much.
• 1:10:37
  Y'all thank you. I appreciate this.
• 1:10:39
  I'll see you all later. Thank you.
• 1:10:45
  I'm Emmanuel Mason, Machine Learning Engineer at Stripe and
• 1:10:49
  author of Building Machine Learning Powered Applications.
• 1:10:52
  And if you don't want your machine learning models to
• 1:10:55
  explode, well, you should subscribe to this podcast.