Latest / Agentic Conversations (formally mlops.community) / Unpacking 3 Types of Feature Stores // Simba Khadder // #265

Transcript

Unpacking 3 Types of Feature Stores // Simba Khadder // #265 · Agentic Conversations (formally mlops.community)

• 0:00
  Hey, I'm Simba Carter, I'm the founder and CEO of Feature Form
• 0:03
  and I like to drink my coffee Arabic style.
• 0:08
  And we're back with another ML UPS community podcast.
• 0:11
  I'm your host as always, Dimitrios talking to my man
• 0:14
  Simba today. And we got into the traditional
• 0:18
  and the new age. E Simba gave me a great
• 0:23
  breakdown on embeddings because he was one of the first people
• 0:27
  that I heard used the term embeddings back in like 2020.
• 0:31
  He told me why he was so intimate with that term because
• 0:37
  of his recommender system work that he did back in the day, and
• 0:40
  why that spurred him to go and create a feature store of his
• 0:45
  own and start selling it. And so now he is doing the thing
• 0:50
  that one tends to do when you create something cool or when
• 0:54
  you have a pain and then you figure out how to ease that
• 0:58
  pain, you go and bring it to the market to see if anybody else
• 1:02
  has the same pain. He really walked through
• 1:05
  different ways of looking at feature stores.
• 1:08
  I appreciated that. How vector stores and feature
• 1:11
  stores can play together and why they're so oftenly confused
• 1:16
  between needing and If you need one, do you not need the other?
• 1:19
  If you need both, how does that look?
• 1:22
  And then we talked about LLM Land and how personalization is
• 1:29
  a must when it comes to LLMS. I usually look at it as variables.
• 1:34
  He gave me a little bit of the light bulb going off in my head
• 1:38
  and told me, you know, it's just features that you're talking
• 1:42
  about. Variables are features, and so
• 1:45
  we can bring those into LLM prompts just as much as we can
• 1:51
  bring other things into LLM prompts, like the context that
• 1:56
  you get from a vector store. Enough for me.
• 1:59
  Let's hear it from Simba. So for your recommender system
• 2:05
  today, we've got a nice little song that I'm going to add to
• 2:08
  your mix and mix up your recommendations on your music
• 2:13
  player. This song right here is coming
• 2:17
  in from the Grateful Dead. It's called Dancing in the
• 2:20
  Street from the 1977 live edition that they've got on
• 2:29
  Spotify at Cornell. So for those who have never
• 2:50
  listened to the Grateful Dead, they had their fans recording
• 2:55
  every single show that they did, and every single show was
• 3:01
  different. There's an awesome book on the
• 3:05
  Grateful Dead and how they did things totally different by the
• 3:10
  CEO, former CEO of HubSpot. It's like Grateful Deads
• 3:14
  marketing or something like that, in case everybody wants to
• 3:17
  check that out. But let's just hear this guitar
• 3:21
  solo and cruise on into our conversation with Simba.
• 4:02
  Dude. So you were the first person,
• 4:05
  and I got to give you credit for this because before LLMS, you
• 4:11
  told me. I'm pretty sure everything is
• 4:14
  just like moving. It's all going to be embeddings.
• 4:18
  And I remember thinking I barely know what an embedding is, and
• 4:22
  that's how I know it was before LLMS, because once LLMS came
• 4:26
  out, I think everybody understood what embeddings were.
• 4:30
  And can you explain that position?
• 4:32
  Has it changed since then? Yeah, you you also have to.
• 4:39
  It's an experience to have people come to you and try to
• 4:42
  explain to you what an embedding is like.
• 4:44
  I'll be talking about something like, oh, there's this thing
• 4:47
  called embedding. So I'm like, oh, tell me more
• 4:49
  like, never heard of them before.
• 4:51
  So yeah, that's like a day in the life.
• 4:53
  Yeah, I think it's still true. I mean, part of how I got into
• 4:56
  embeddings and like, I guess into embeddings, the part of how
• 5:00
  I began to kind of get interested in that field.
• 5:04
  And kind of like the, I would say at the, at the time or the
• 5:07
  cutting edge of deep learning was I was doing recommender
• 5:10
  systems and what was really kind of new vent.
• 5:16
  And maybe now kind of actually, I'm still seeing some articles
• 5:20
  about it, like the two tower architecture, we were seeing
• 5:23
  people kind of pushing holistic understanding of items and users
• 5:30
  to the model itself and essentially turning users and
• 5:32
  items into embeddings. So this concept of like, hey, we
• 5:36
  have this holistic view of what something is the the analogies
• 5:41
  around like in in text we often hear about like the king is to
• 5:45
  queen as like man is to woman. You can get way more
• 5:48
  sophisticated than that. I remember I did a a bedding
• 5:52
  generation for an e-commerce data set and one really cool
• 5:58
  when I came up, one really cool like kind of analogy which looks
• 6:01
  like a parallelogram and and embedding visualizer was Coke is
• 6:07
  to Diet Coke as Cherry Coke is to Coke 0.
• 6:12
  Really. Wait, how does that work out?
• 6:14
  Well, like I think when I think of the flavour profiles, like
• 6:17
  Coke is a generic, like that's the standard and Diet Coke is
• 6:20
  the sugar free cherry Coke is like Coke standard, but there's
• 6:25
  almost like a slightly different like vanilla extra sugary
• 6:28
  flavour to it. And Coke 0 is like Diet Coke,
• 6:31
  but has, again, a same kind of extra flavour profile.
• 6:36
  And I can kind of explain it and we could talk about it and be
• 6:38
  like, yeah, that's how it tastes.
• 6:39
  But the crazy thing is, is that this model was able to derive
• 6:43
  that just by looking at O user X, but item Y.
• 6:47
  And the first time I started seeing stuff like that, I just
• 6:51
  was like, man, like it really deeply understands what these
• 6:56
  things are all by itself by looking at really sparse data,
• 6:59
  which if you take that and maybe expand it, you kind of start to
• 7:06
  see what LMS are. Or if you take that to an
• 7:09
  extreme level, you're like, hey, if you just read random text 1
• 7:12
  token after the other, like a model can actually start to
• 7:16
  understand ideas and concepts and and things that you wouldn't
• 7:21
  really think would be possible just by reading word to word.
• 7:24
  So this is a very long answer to your very short question, but I
• 7:28
  think the, the real answer is that yes, we're only like we're
• 7:33
  always sort of at the beginning of things.
• 7:35
  I think we're kind of, there's always these S curves of like
• 7:38
  there's a new architecture that comes out or a new style or new
• 7:44
  in this case, like a training technique.
• 7:46
  And that just pushes us to the next level and intends to
• 7:51
  plateau for a little while and then it goes to the next thing.
• 7:54
  At the same time though, every single time that happens,
• 7:56
  everyone's like, all right, everything we've ever done is
• 8:00
  now deprecated. And I remember when, you know,
• 8:04
  the deep learning became a thing and everyone was talking about
• 8:07
  even like 20/17/2018, it's like, yeah, everything else is, you
• 8:13
  know, deprecated now. Like we don't need anything now.
• 8:16
  We just everything is going to be deep learning and that's it.
• 8:20
  Like feature engineering is dead, everything else is dead.
• 8:22
  We just have to deep learning. But to this day, probably the
• 8:26
  most common model in production is some form of a random forest,
• 8:29
  like actually boosts or something similar, and I
• 8:32
  wouldn't be surprised if that still remains to be true years
• 8:36
  into the future. Yeah.
• 8:38
  And there's the stories that come up consistently of folks
• 8:43
  that go through university, they do a bunch of cool deep learning
• 8:47
  problems and then they get on the job and they're like,
• 8:49
  where's all my deep learning? What's going on here?
• 8:52
  First of all, you don't have the data for deep learning.
• 8:55
  Second of all, that's not what we do here.
• 8:58
  And so get used to XD Boost baby.
• 9:01
  It's so true. And yeah, there's a big gap
• 9:04
  between what people how people learn about machine learning and
• 9:07
  what it actually looks like in in the Natural Company.
• 9:11
  So I really like the idea that you mentioned before of this,
• 9:14
  this 2 tower architecture. I think that's cool.
• 9:18
  And I hadn't heard that before. And basically you're you're
• 9:21
  saying it, if I'm understanding it correctly, the customer is
• 9:28
  put as one type of embedding and then the item or whatever people
• 9:33
  buy is another type of embedding.
• 9:37
  And depending on how close these embeddings live to each other,
• 9:41
  you can recommend certain embeddings, correct?
• 9:44
  Yeah, actually how we would do it is we would.
• 9:48
  If I want to recommend something to a user, there's another
• 9:54
  common strategy which again has really become popularised, which
• 9:59
  was called re ranking, where we would use an embedding and we
• 10:03
  would do candid generation. So I would take, let's say I'm
• 10:06
  YouTube and the YouTube actually popularised this in their paper
• 10:10
  where they were like, hey, I have a user embedding, I'm going
• 10:12
  to take this user embedding but the model figured out all by
• 10:15
  itself just based on behavioural kind of items.
• 10:20
  And I will figure out 1000 or 10,000 video embeddings that are
• 10:25
  near this user embedding and those are my candidates.
• 10:29
  And then I'll have specialised model like one for the side
• 10:32
  view, one for the autoplay, one for whatever, where I will re
• 10:35
  rank those 10,000 candidates to decide what to show people.
• 10:39
  And so the one thing I think is that, yeah, embeddings don't,
• 10:44
  it's not just like a text thing, like we can do a betting zone, a
• 10:47
  lot of things like users and items, images.
• 10:51
  And then the other thing, which I think a lot of people miss,
• 10:54
  and this again re getting popular as I feel like
• 10:56
  everyone's relearning a lot of things that people in the
• 10:58
  recommended systems learn a few years ago now.
• 11:02
  But the other thing is that you can actually use embeddings as a
• 11:05
  drop in as a feature in a typical and a traditional model.
• 11:09
  So for example, if you're have a ranking model and you want to
• 11:14
  put an item as like an input, you can actually just put the
• 11:18
  whole item embedding in. So and do that.
• 11:22
  You can do things like average embeddings together.
• 11:24
  Like I can take the last 10 items the user watched and I can
• 11:28
  just average them together and blend it.
• 11:30
  And that that actually works with.
• 11:33
  But the other funny thing about embeddings is it works except
• 11:36
  when it doesn't. And when it doesn't, it's quite
• 11:39
  painful experience just trying to figure out what's happening
• 11:42
  because they kind of have. Debug.
• 11:44
  Abstractions on abstractions, yeah, it's which is why like
• 11:48
  actually boosts and things remain very popular because they
• 11:52
  might not always have the best performance, but they always do
• 11:55
  quite well or they almost always do quite well and you kind of
• 11:58
  kind of start to understand what it's doing and why.
• 12:01
  So you can figure out, you know, make it not do dumb things as
• 12:05
  easily as. That's such a great point.
• 12:09
  The it's, it reminds me of a video I watched on how windmills
• 12:16
  that are your typical windmills that we probably all seen,
• 12:19
  especially in wind farms, those tend to break like 10 times more
• 12:24
  than windmills that spin on the Y axis as opposed to the yeah,
• 12:30
  the typical ones. And so the windmills that spin
• 12:33
  on the Y axis and are just spinning around in circles like
• 12:36
  this. And for anybody that's just
• 12:37
  listening, you can think like a water bottle that has holes in
• 12:41
  it and it's a it's just spinning on the Y axis.
• 12:46
  Those break the, sorry, the ones that are the typical ones, those
• 12:51
  break a lot more than the ones that's been on their Y axis.
• 12:56
  And so you have to do some calculations on all right, I can
• 13:03
  generate more power with the typical windmills, but it's in
• 13:07
  spurts because it's going to eventually throw me an error and
• 13:11
  it's going to break. But I can get like 10 years of
• 13:16
  no breakage with these very simple windmills.
• 13:20
  However, it's not going to generate as much power.
• 13:23
  So it's almost like the tortoise and the hare.
• 13:26
  And that reminded me of it because you were saying, hey,
• 13:30
  you can have this very powerful architecture of setting up with
• 13:35
  embeddings on embeddings type thing and using those as
• 13:38
  features. But when it breaks, you're, you
• 13:42
  potentially are going to spend a lot of time trying to debug it
• 13:46
  and figure out where it's breaking.
• 13:48
  As opposed to, hey, let's try this simple architecture that if
• 13:53
  it breaks, we don't need to spend that much time debugging
• 13:56
  it and it is not really going to break as much.
• 14:01
  Or if it does break, we can get it.
• 14:02
  The uptime is much higher, right?
• 14:06
  Yeah, I think that's, that's spot on.
• 14:07
  It's it's, it's just more accurate in its simplicity.
• 14:10
  Like it's, it's you know what it's going to do.
• 14:14
  And depending on you just have to think of your problem space,
• 14:17
  like in some problem spaces, maybe having that really one, do
• 14:21
  you know that amazing once in a while model recommendation
• 14:26
  prediction is is worth the bad predictions.
• 14:29
  And there are other times where, hey, I'd rather have a do pretty
• 14:32
  good, you know, every time, then have a do amazing, you know,
• 14:38
  most of the time, but really badly some of the time.
• 14:41
  Well, I feel like you wanted to go down this route, but I know
• 14:45
  there's people that ask the question frequently.
• 14:50
  How do vector stores and feature stores play together?
• 14:56
  Yeah, it's a, it's a, it's a good question that it's almost a
• 14:59
  funny question because if they didn't both have this word store
• 15:03
  in them, I don't think anyone would ask this question.
• 15:07
  But they both kind of play with embeddings, right?
• 15:10
  You were saying the feature stores have the embeddings and
• 15:13
  then vector stores for sure are all about embedding.
• 15:17
  Yeah, yeah, yeah. And they both kind of touch
• 15:19
  data. I, I think, I think it's true.
• 15:21
  I think it's more like it's not things that I, I typically put
• 15:24
  right next to each other. But yeah, I mean, we, I remember
• 15:26
  like in the early days, we released embedding support in
• 15:30
  feature foreign and there was a mix.
• 15:34
  It was mostly people asking like, or being like, OK, like
• 15:38
  we, we don't understand very few people at that point or like
• 15:41
  understanding what that meant and why we like open source,
• 15:44
  something called embedding hub, which is like a really early
• 15:46
  vector database. And I think the key difference
• 15:51
  is really the value prop. So I'll tell you what the key
• 15:56
  difference is today and where I think it's going.
• 16:00
  So the key difference today is that a vector store is a
• 16:05
  specialised index, which makes it really quick to do nearest
• 16:11
  neighbour look UPS. So to do true nearest neighbour
• 16:14
  look UPS where I give you a point and you find me 10 points
• 16:17
  nearby is a very, very expensive problem.
• 16:22
  And so there are many algorithms that give you approximate
• 16:26
  nearest neighbour look UPS and that's a vector database.
• 16:31
  And so they typically build indices.
• 16:33
  A really common one is called HMSW, which Facebook released in
• 16:39
  the early days. There was a library I used to
• 16:41
  use a lot called Annoy, which was Spotify, right?
• 16:46
  Yeah, Spotify data actually, that's Eric's the guy who built
• 16:50
  annoy or I, I believe that he built it by himself, but
• 16:54
  whatever him, he's released it or I, I associate with him.
• 16:57
  He's now the CEO of Modal. So it's funny how all these kind
• 17:02
  of things he what people work on very different things.
• 17:04
  Everyone's kind of seen and play the law the same tools.
• 17:07
  But yeah, I like we. So the problem of all of these
• 17:10
  indices is they don't actually really fit well into the
• 17:14
  traditional database models. The only one that kind of does
• 17:18
  is a specific one, which is I believe it's IVF, and it's not
• 17:27
  as good as HNSW, but it's a little easier to scale and it
• 17:30
  works better and looks more like a traditional index.
• 17:34
  But anyway, so I've actually database as a way to index your
• 17:40
  vectors through embeddings such that you can do nearest
• 17:43
  neighbour lookups pretty quickly and there is a give and take of
• 17:48
  how of recall and that's kind of the difference between the
• 17:57
  different algorithms. There's also difference in speed
• 18:00
  and latency, scale, usability, UX, whatever.
• 18:04
  So that's the way the vector database is kind of fight
• 18:07
  amongst themselves. Feature stores are almost
• 18:11
  solving of quite a different problem, but has a little bit of
• 18:13
  overlap. The problem that feature stores
• 18:15
  are solving is the problem of a lot of ML is inherently so a
• 18:24
  specialised form of data engineer.
• 18:26
  Like you take data and you turn it into features, put in a less
• 18:31
  ML wordy way. Like I have raw data and I need
• 18:35
  to pull signals out of it. And so a lot of what I'm doing
• 18:38
  is just kind of iterating on the data to get the best signals
• 18:41
  that I can provide to my model for it to do a good job.
• 18:44
  Now there's a lot of problems that come into that.
• 18:47
  Some of it is organisation like versioning, lineage, governance
• 18:53
  comes up a lot and it is a production problems, which is
• 18:57
  OK. Well, even if I build this thing
• 18:59
  in my notebook and I use pullers or pandas, well, I need to get
• 19:03
  this in prod. And in prod, if it's like a
• 19:06
  recommender system, it can't go down or bad things happen.
• 19:10
  So how do I build a enterprise grade data pipeline for all
• 19:18
  these features I built? And by the way, like I'm a data
• 19:21
  scientist, like I'm not a data engineer.
• 19:24
  And So what we see a lot of is like data scientist going to
• 19:26
  data and teams and giving them notebooks like, oh, here's my
• 19:29
  notebook, can you please put this in prod?
• 19:31
  And so there's this like awkward friction between data and data
• 19:34
  science where data engineers are like, like these data scientists
• 19:38
  have thousands and thousands of things they're throwing our way
• 19:41
  and we just have enough stuff to do.
• 19:43
  And in practise, like the amount of asks that you'll get from
• 19:47
  data science versus the ROI of every specific data pipeline is
• 19:51
  actually it just the math isn't really math very well.
• 19:55
  And then for the data scientist? Wait, because of the data
• 19:59
  scientist being way more valuable like they're.
• 20:02
  The stuff that they're asking to put in production is so
• 20:04
  valuable. Is that why?
• 20:05
  Actually it's it's more of that like, hey, this might
• 20:08
  incrementally increase my model a little bit.
• 20:11
  Oh, OK. And so a lot of asks for a
• 20:14
  little bit of lift, correct? And but so data scientists job
• 20:18
  and that little lift doesn't make like a difference to the
• 20:20
  business. But from the data engineers
• 20:21
  perspective, it's just like a painful and from the data
• 20:27
  scientists perspective, it's like, how can they do their job?
• 20:29
  Like it's science, It's it's, it's a lot of hypothesis testing
• 20:32
  and throwing stuff away and evaluating things.
• 20:35
  If every experiment is very expensive, then you know, but
• 20:40
  you can't really do your job very well.
• 20:41
  And so where we find the teacher stores being really valuable is
• 20:45
  making it possible for data scientists to kind of
• 20:48
  self-serve, build an experiment and deploy their own future
• 20:52
  pipelines and have us be kind of like this harness that gives
• 20:56
  them. Yeah, the versioning, the
• 21:00
  monitoring, the scale, the up time, all the things that you
• 21:05
  would expect, incremental processing, stream processing,
• 21:08
  everything you'd expect. And maybe a really good data
• 21:11
  engineer could do. We make it possible for data
• 21:14
  scientists to do themselves. And, and so the promise space is
• 21:18
  different. Now the overlap is that will
• 21:19
  sometimes those features are embeddings.
• 21:24
  And so that's where the overlap is.
• 21:27
  I see. That's where they dance together
• 21:29
  because I think the confusion comes in when you hear folks
• 21:34
  talking about how they're using vector stores for their
• 21:38
  recommender systems. Yeah.
• 21:42
  And I, I think what it is, is I would describe it as everyone
• 21:45
  has a feature store. If you have a model in
• 21:47
  production that takes data as inputs, whatever process you
• 21:51
  take to get that data there and manage that data processing,
• 21:54
  that's your feature store. Now they could be a bunch of
• 21:58
  bash scripts. It could be a Python package
• 22:02
  that you install the dark container and just like do the
• 22:04
  processing on the fly. Like there's a lot of things
• 22:07
  that people do. But that whole kind of promise
• 22:11
  space I would define as a feature store.
• 22:13
  And a lot of what I'm doing when I'm talking to to prospects is
• 22:17
  trying to understand if they have malls of production.
• 22:19
  It's like you have something here, help me understand like
• 22:22
  what that looks like today so I can show you what the kind of
• 22:26
  ROI would be. Yeah.
• 22:28
  Do you feel like there are phases to when you need more of
• 22:36
  a bulletproof solution as opposed to those bash scripts?
• 22:40
  And because I remember we had Neil Lithia on here ages ago,
• 22:46
  and he was head of ML at Monzo Bank.
• 22:49
  And one thing that he said, I think they had their own
• 22:52
  homegrown feature store, right? And he was saying, yeah, what I
• 22:56
  see with if we wanted to buy something off of the shelf,
• 23:00
  that's us going and trying to leapfrog in a way.
• 23:06
  It's us saying, cool, we have nothing now we're going to one
• 23:11
  really quick because we can buy it off the shelf as opposed to
• 23:14
  we've already got something in there and it's working and the
• 23:17
  gradual increase of us making it better and better.
• 23:21
  And so I wonder if there's patterns that you've seen where
• 23:27
  you're like, I recognise that you're currently at this scale
• 23:33
  and you want to go and do it this way.
• 23:36
  Or I recognise that your current architecture, your, the way that
• 23:40
  you set it up, is this brittle? Because I can only imagine that
• 23:44
  we all in the back of our heads are we know our weak spots and
• 23:49
  we are all just expecting, even if it isn't, we all kind of have
• 23:53
  nightmares of how brittle our systems are.
• 23:57
  And so anyway, tangent aside, if there was a question in there,
• 24:03
  I'm sure you're going to answer it right now.
• 24:06
  Yeah, I think the question I would define it as like when do
• 24:09
  you actually need a feature store or or how do you even
• 24:12
  decide like feature aside like like how do you do that analysis
• 24:18
  of like, OK, it's time to go of a vendor or something off the
• 24:20
  shelf versus hey, we have our own home grown hacky thing.
• 24:23
  We know it's hacky, but what's actual ROI of going for
• 24:27
  something else? And I actually think that this
• 24:30
  question is was at the heart of we both kind of saw like an
• 24:36
  implosion of MO OPS startups because there was just like
• 24:39
  humongous to like everyone had an MO OPS company for like a
• 24:43
  moment in time few years ago. And then a lot of those
• 24:46
  companies don't exist anymore. And there was this kind of era
• 24:51
  where people were buying MO OPS just almost experimentally,
• 24:57
  like, let's just try this or we heard this is a good idea.
• 24:59
  I'm not really thinking of ROI, just more like we need more
• 25:02
  shiny tools. And I just think that LM OPS
• 25:06
  kind of sort of looks like this today.
• 25:08
  That's what I was going to say. I and I don't know if I fully
• 25:11
  agree with a lot of those companies are not around
• 25:14
  anymore. I think a lot have maybe pivoted
• 25:18
  to LLM hubs companies and or the the story has changed because
• 25:26
  folks feel there's a bigger market out there with the LLM
• 25:30
  use cases, which I'm not necessarily sold on.
• 25:34
  But that's a whole another conversation.
• 25:36
  So forgive me for interrupting you.
• 25:38
  No, I, I, I think you're, I think the company that, that
• 25:42
  maybe we're referencing or companies that I would say were
• 25:45
  like above the fold. So companies that most people
• 25:47
  knew about and had some there's this like this whole like for
• 25:50
  every, for every monitoring company you knew about, there
• 25:53
  was like 15 that, you know, very few of us have ever heard of.
• 25:57
  Same a feature stores for every feature company you've heard of,
• 26:00
  for every feature form, there was 15 companies that even I
• 26:03
  didn't know about all the time until I would randomly run into
• 26:05
  them in like a cycle or someone's like, oh, there's this
• 26:07
  thing that has no website but we're thinking of music and so.
• 26:13
  Yeah, I think there was that era of a friend who as he knows a
• 26:17
  thing or two about feature stores.
• 26:19
  But but also I'll let you make your point on the question that
• 26:23
  I didn't ask before that I tried to ask but couldn't get out.
• 26:27
  But I will say that feature stores feel like it is
• 26:31
  relatively strong because you do see that folks are still coming
• 26:35
  out with new feature stores. Like there's, I heard about
• 26:39
  chalk the other day and I heard about what's the one that Airbnb
• 26:45
  open sourced, right? And so the fact that there are
• 26:48
  more people coming into the space, even though it is not
• 26:55
  viewed as like the booming space that it once was, is still a
• 26:58
  good signal. And I, I know that you mentioned
• 27:01
  before we hit record that like it's just something you do.
• 27:04
  It's not as sexy as it was in 2020, but it is still something
• 27:08
  that is very useful. Yeah, I think if you talk to at
• 27:14
  least in feature store space, I don't know so much about the
• 27:17
  upper arm off spaces, but at least in our space, even like
• 27:20
  our competitors will say like, yeah, like all boats are kind of
• 27:23
  rising in feature stores. There's this way more.
• 27:29
  I think what it was, was a lot of people didn't really fully
• 27:32
  understand the problem space or we kind of understood but like
• 27:35
  we need to do this ourselves and field ourselves.
• 27:38
  That might have been driven by a little bit of like, I think good
• 27:42
  reasons in some places. We're like, hey, before we bring
• 27:44
  this vendor in, let's make sure we actually understand this
• 27:46
  problem. There's probably some bad
• 27:48
  reasons where, hey, this is like the hottest thing ever and
• 27:50
  everyone who's doing this is writing blog post.
• 27:52
  So I want to do this and not buy it, which, you know, always
• 27:56
  happens in new categories. And I think we'd move past the
• 27:58
  air where it's like, hey, the ROI, the feature store is kind
• 28:01
  of well understood. Now if we, you know, have data
• 28:04
  scientists doing ML and they're working with data, if they're
• 28:06
  always complaining about all sorts of things related to the
• 28:09
  data feature store is kind of an umbrella that solves a good set
• 28:12
  of those problems. But just get one like there's
• 28:16
  enough of them very well proven at this point.
• 28:18
  They have big customers like, are we really going to build
• 28:21
  this in house? Why?
• 28:23
  And I think everything we ran into is evo.
• 28:26
  A lot of companies could build in house, like you mentioned,
• 28:30
  like Airbnb, Like what ends up happening a lot of those
• 28:33
  companies is if I build something valuable in house, the
• 28:37
  team that builds it then leaves, starts a company and kind of
• 28:41
  leaves the company holding a bag of like, Oh yeah, this thing
• 28:45
  that we built originally that now everyone who actually wanted
• 28:48
  to drive it forward has now left and all your left with this like
• 28:51
  maintenance on it and it just degrades, which you've seen a
• 28:55
  lot of. That happened with the LinkedIn
• 28:58
  feature store Feather, I think, which is a bummer because when
• 29:02
  it came out, we interviewed the folks that worked on it and then
• 29:06
  it feels like it didn't get picked up and it didn't get the
• 29:10
  energy that it needed to be injected into it.
• 29:13
  But you made a great point on Resume Driven Development.
• 29:17
  And in 2020 for there was so many blog posts about feature
• 29:22
  stores. That's so true that yes, if I'm
• 29:25
  going to be working in the ML op space, what am I going to do?
• 29:28
  Well, an easy win is I build my own feature store.
• 29:32
  And we even had Nicholas on here a few weeks ago and he was
• 29:35
  talking about how he has Big Query as their feature store S
• 29:41
  type solution. And so it, it is like that, that
• 29:45
  thing still happens. And that was the reason that I
• 29:50
  wanted to ask you the original question that I asked you and
• 29:54
  didn't let you finish and give your answer to.
• 29:56
  So now I'm going to shut up. I swear I'll go on mute and I'll
• 29:59
  let you give your answer. Yeah.
• 30:01
  I, I also have the problem of like who also I'm like, oh, we
• 30:05
  can talk about all these other things too.
• 30:06
  But yeah, coming back to the main problem that that that
• 30:09
  mentioned, which is, hey, you know, I had big query.
• 30:13
  We have this kind of ad hoc process in the house and we use
• 30:16
  as our feature store. Do we need an actual external
• 30:21
  feature store? So first of all, answer very
• 30:26
  pragmatically of like you owned a whole on loss platform and a
• 30:29
  feature source is one of many things that you can invest in by
• 30:32
  I think in practise, there are four, call it modalities that
• 30:37
  like a ML engineer, data scientists will be in.
• 30:41
  Where's the data modality, which is like when they're working
• 30:43
  with data and putting data pop as a production.
• 30:46
  Where's the training modality, which is like experimenting,
• 30:50
  sometimes locally, sometimes in notebooks, just like building
• 30:54
  malls. This is where like ML flow, like
• 30:56
  this is the part of the the mall pipeline or using something like
• 30:59
  ML flow or weights and biases or something.
• 31:01
  There's mall production, which is OK.
• 31:03
  Now I have a small, it's in a model store.
• 31:05
  Actually to get it in production, I need to serve it.
• 31:08
  I need to have this up time to slate and see all that sort of
• 31:10
  stuff. GP us all that comes in.
• 31:13
  So there's deployment and then there is evaluation and
• 31:16
  monitoring, which is the final bit.
• 31:19
  So those are kind of the four modalities and arguably the four
• 31:21
  problem spaces. So to to say them in order data
• 31:26
  model, training model production or model serving, whatever, and
• 31:29
  then monitoring evaluation. So my first question would be,
• 31:33
  OK, talk to your user because you're a platform team.
• 31:36
  So if you think of your platform like a product, your goal is to
• 31:39
  make your ad user happy and your ad user is a data scientist or
• 31:43
  I'm all engineer or something. So what do they of those four
• 31:46
  things, what's causing them pain and what's does that pain equal
• 31:53
  like beyond just like I hate this, like is it slowing things
• 31:57
  down? Is it causing errors in
• 31:58
  production? What else?
• 32:01
  Typically the answer, the question, the answer is almost
• 32:05
  always going to be 1 of monitoring, evaluation or data.
• 32:08
  At least in practise, I've seen that those two tend to be the
• 32:11
  ones that have the most pain. And it's no surprise then that
• 32:16
  when you think of like Amalop startups, the ones that I've
• 32:19
  kind of most done, I think have done quite well are companies in
• 32:25
  both of those categories, data being feature store companies
• 32:28
  and then monitoring evaluation companies.
• 32:31
  Interestingly, in LMS, you could almost ask the same questions
• 32:35
  and you end up at the same place for most of the problems are
• 32:38
  data and monitoring evaluation, like come back to that later.
• 32:42
  But yeah, so that's the first bit is like, which one matters?
• 32:45
  If it is data, the question becomes a what about the data?
• 32:50
  And one thing about like feature fame specifically part of how we
• 32:54
  built it and why we built it, we call ourselves a virtual feature
• 32:57
  store or the premise that it sits above your existing
• 33:00
  infrastructure. So the problems that we saw,
• 33:02
  they're much more about productionizing things and
• 33:05
  operationalizing feature pipelines.
• 33:07
  So it sits above Big Query, it sits above the vector story,
• 33:10
  sits above these things. So it's the application layer
• 33:13
  that sits above your infra and it makes all work together
• 33:15
  coherently like a feature store. Gartner came out and called this
• 33:20
  concept a logical feature store. We weren't I I came up with the
• 33:27
  name logical feature sort of felt kind of like a other handed
• 33:30
  thing to say. We're the logical feature starts
• 33:32
  at a virtual, but conceptually it's like the the logical layer,
• 33:37
  the semantic layer of features above the emperor.
• 33:41
  And so if you have a data problem, the question that
• 33:44
  becomes what kind of feature store?
• 33:45
  And there's three kinds which I can get into, but maybe I'll
• 33:47
  take a second and let you ask any follow-ups you.
• 33:50
  Have Don't even get me started on Gardner.
• 33:54
  That's all I got to say about this.
• 33:56
  Yeah, but it's an interesting with the small of gardeners.
• 34:02
  Well, so the fascinating bit is the the four pieces of the life
• 34:10
  cycle. And I do like how you you've
• 34:14
  clumped together like there's the data piece, which is there's
• 34:18
  so much inside of that data piece that can be a headache,
• 34:22
  right? Like just the data acquisition
• 34:24
  or as you were talking about earlier, you mentioned a few
• 34:26
  good things like the governance or transformation or access even
• 34:34
  can be a pain depending on if you're at a gigantic company or
• 34:37
  not. And then you have the
• 34:41
  deployment, the model training is another interesting one
• 34:45
  because that one for sure feels like there's less pain there
• 34:49
  because the tools in my eyes are very mature.
• 34:53
  And so you look at like awaits and biases or an ML flow and
• 34:56
  those tools feel very mature compared to the rest of this
• 35:00
  life cycle. And maybe that's because the
• 35:04
  training has been around almost longer than that was like the
• 35:08
  first thing to kind of pop up in the life cycle.
• 35:11
  I don't know, maybe you have theories around why those tools
• 35:14
  are more. Mature, yeah, they just came a
• 35:16
  little earlier and I think the problem space is a little more
• 35:20
  well understood. I think data like you said, like
• 35:22
  what does that even mean is very dependent on the type of
• 35:25
  company. So the solution is way more, has
• 35:28
  to be way more complex and same with monitoring, evaluation,
• 35:31
  whereas like training and serving, it's like an end, it's
• 35:34
  like this lots of matrix multiplications and GPUs.
• 35:38
  Is this Yeah. I'd say just throw more GPUs at
• 35:42
  it. You're all good and so OK.
• 35:45
  There we are. Now keep cooking because I know
• 35:49
  you wanted to say something else.
• 35:52
  Yeah, I want to maybe Dad, because the other thing that I
• 35:55
  think there's a lot of confusion about with feature stores is OK,
• 35:59
  like there's, I get it, like there's this data problem, but
• 36:03
  there's all these feature stores and they all look kind of
• 36:05
  different, but they all call themselves feature stores.
• 36:07
  So kind of like understanding what's actually happening there
• 36:11
  and what you need. And one thing we've done is
• 36:15
  we've kind of broken up the feature stores into 3 distinct
• 36:19
  categories. And the categories are
• 36:22
  interesting because they all kind of came about at the same
• 36:24
  time and there were just three different solutions to the
• 36:28
  problem. One solution which was
• 36:31
  popularised by Feast, I would call the literal feature store,
• 36:36
  where you literally just store your features there.
• 36:38
  This is kind of how Databricks is feature store, Sage Maker's
• 36:40
  feature store looks for you transform it and you build it
• 36:44
  somewhere else. And they would define the
• 36:47
  feature as like a table. So the artefact of the
• 36:51
  transformation is the feature. So the benefits, it's really
• 36:56
  simple, like you just have a new sync to put your, you know,
• 36:59
  features into. The cons is everything I just
• 37:02
  said about productionizing feature pipelines, dealing with
• 37:05
  streaming data, versioning lineage, none of that stuff
• 37:08
  really gets solved by the literal feature store.
• 37:11
  So in some ways it's like a very simple add on, but in other ways
• 37:17
  it doesn't really do much with the databases feature store.
• 37:21
  It's almost like such a lightweight thing that it's
• 37:25
  almost like a specialised sub part of the UNI catalogue.
• 37:29
  So every single table in data bricks is unique catalogue, but
• 37:33
  it has a primary key they just put in the feature store.
• 37:36
  It's just like, yeah, it's the same thing.
• 37:38
  Like a table of a primary key is a feature.
• 37:39
  Like that's that's kind of their approach.
• 37:42
  And then yeah, part of the problem they solve, I should
• 37:44
  make sure it's clear, is that there's another big problem with
• 37:47
  features, which is training and inference.
• 37:49
  A training time you need to kind of batch high throughput data of
• 37:53
  all my historical feature values.
• 37:56
  And then an inference time I need to be able to just do a
• 37:59
  quick look up like, hey, Demetrius, just open up Spotify,
• 38:02
  I need to do recommendations for him.
• 38:03
  I need to look up the features now.
• 38:06
  And so having like these pre computed cash features and
• 38:09
  something like Redis or Dynamo in keeping that In Sync with
• 38:13
  your training data is another problem that these things solve.
• 38:16
  Yeah. Wait, can we, can we pause real
• 38:19
  fast right there? Because there's something
• 38:21
  fascinating for me when it comes to Databriggs and Sage Maker
• 38:29
  feature stores and these end to end platforms like Vertex or
• 38:34
  Azure, I think has one too, right?
• 38:36
  And it feels like what they did is almost like what Microsoft
• 38:40
  Teams did against Slack where they said, Microsoft just said,
• 38:46
  you know what, you get Office, cool.
• 38:48
  You can also have Teams. And so Sage Maker said, yeah,
• 38:52
  cool you you have Sage Maker. Well, you also get a feature
• 38:55
  store. And I would love to hear your
• 38:59
  thoughts on how that is like when you look at the space and
• 39:06
  you see big platforms that just kind of throw in the feature
• 39:12
  store as a part of it. Where's your head at on that?
• 39:17
  Yeah, so. Yeah, I'll answers in two parts.
• 39:21
  1st I'll answer from the perspective of data bricks.
• 39:24
  So data bricks or, or or something like data bricks, I'm
• 39:28
  just going to use data bricks just because it's they have a
• 39:31
  feature store, but that's well known.
• 39:34
  So data bricks they're NRR which frozen.
• 39:39
  No, it's kind of like they're not revenue retention, which is
• 39:41
  like if I have, you know, ex customers and they pay me a
• 39:45
  billion dollars a year, ignoring new customers, how much will
• 39:51
  those customers, the same ones pay me next year?
• 39:55
  And so it's almost like upsell, like how much more are you
• 39:57
  paying Daybreak or expand? Exactly.
• 40:01
  And so I I believe they're numbered and someone's going to
• 40:06
  probably fat to call me, but I'm going to save her number sort of
• 40:09
  kind of looks like 150% even it's 120%.
• 40:14
  Let's just say 120 if you're at like a billion revenue.
• 40:19
  That means that they're those customers paying the billion
• 40:22
  revenue now paying 1.2 billion next year if they do nothing,
• 40:27
  like they can literally do nothing and they just make 20%
• 40:31
  more dollars than they made the year before.
• 40:34
  And they are held to task on that.
• 40:37
  It's a huge revenue, It's a huge metric that like companies like
• 40:41
  Snowflake and others are held to.
• 40:43
  And so how do you do that? Well, there's one bit which is
• 40:47
  OK, like just make people use the product more.
• 40:50
  And there's another bit which is like, OK, well, like how do we
• 40:53
  make it more? How do we get in more places,
• 40:55
  make it more sticky? So, but I thought ML big use
• 40:58
  case, lots of money, very let's just throw tools at it that make
• 41:02
  it useful for ML and make a stickier for ML and we benefit
• 41:07
  from there. So that's kind of perspective is
• 41:08
  just like we need to get people to spend more money on data
• 41:12
  bricks. So very on data bricks, it's
• 41:14
  easy to get people to spend more money on a vendor they already
• 41:17
  have and to get a new vendor in. And so we'll just do that.
• 41:21
  So then there's a question of OK, feature stores, you know,
• 41:23
  data bricks has 1. And I think this is where like
• 41:26
  the space, and I think a lot of spaces have to justify the
• 41:30
  incremental value. And so feature form, we took the
• 41:33
  approach of being open source and we also took the approach of
• 41:37
  playing very nicely with data bricks, this feature store, for
• 41:39
  example, in fact, every feature, if you're using feature form in
• 41:43
  data bricks, every feature that's built with feature form
• 41:47
  runs on data bricks and appears in the data bricks feature
• 41:49
  store. So it's like data bricks, this
• 41:51
  feature store plus plus like it's only incremental value on
• 41:54
  top. And then the question becomes,
• 41:56
  OK, well, like how much is the incremental value worth to you?
• 42:00
  And you know, if it is valuable, yeah, we charge X dollars, how
• 42:06
  much ROI do you get on that? And so there's some math to be
• 42:08
  done there. But yeah, I think you're right
• 42:11
  in that a lot of companies are releasing these and taking this
• 42:13
  platform approach, this kind of idea like data gravity, or if
• 42:16
  they all lives in one place, it's, it's easier to kind of
• 42:19
  sell and like more things on top of it.
• 42:22
  And I think the question really becomes quality, which is like
• 42:26
  the AWS, like AWS has a service for everything, right?
• 42:29
  But it's kind of accepted that the AWS 1 is kind of the
• 42:32
  baseline, like it's like, OK, like it can only get better from
• 42:35
  here, but is this good enough? And I think the same is true of
• 42:39
  data bricks, but data bricks, I think to their credit, still has
• 42:43
  more of a quality component than maybe like AWS service do.
• 42:47
  So there's maybe more of an argument to be made.
• 42:51
  And anyway, I, I think it's a very long answer to what I think
• 42:54
  of like the, the, the, there's a business reason why they do it
• 42:58
  and that business reason, it's not like we're trying to be an
• 43:00
  envelops company. I don't think they really, I
• 43:02
  don't say we don't care about the quality of it, but it's,
• 43:05
  it's, it's more of a way to drive more spend of data bricks.
• 43:09
  So anything they can do, they could just kind of add on, is
• 43:11
  this a way to drive more data bricks spend?
• 43:13
  And in a funny way, if we continue to do as we're doing,
• 43:18
  continue to do well, continue growing and it becomes, we
• 43:20
  become like a de facto feature store.
• 43:22
  I don't think data bricks cares. I think they're like, cool,
• 43:25
  we'll just support feature form because they're still driving
• 43:27
  more data bricks spend. And then And that's all data
• 43:30
  Bricks cares about. Yeah, if you look at what they
• 43:33
  did with Mosaic, that's exactly to what you're talking about.
• 43:37
  They bought Mosaic and they said now you have a place inside of
• 43:42
  Databricks to do all your LM stuff.
• 43:45
  Yeah. And Tabular was an interesting
• 43:46
  acquisition where they were just like cool.
• 43:49
  We're just kind of, I mean, that one has a very fun story behind
• 43:53
  it, but the the very short of the story is like we can pretty
• 43:58
  much make a whole differentiation, not be
• 44:00
  differentiation anymore by just unifying every data format or
• 44:04
  table format under 1. And so, yeah, they they've done
• 44:08
  very well. I mean to, but not a company
• 44:09
  that's easy to compete against a snowflake.
• 44:12
  I think numbers are same. Yeah.
• 44:14
  So I derailed you when you were talking about the different
• 44:18
  types of feature stores. That was one type, which is you
• 44:21
  called it the literal feature store, right?
• 44:25
  Classic feature store like Feast or pretty sure that's what we
• 44:30
  had Nicholas on here using Big Query as that.
• 44:34
  That's where they stored features and so.
• 44:38
  And that's why the name gets and that's what a lot of people
• 44:40
  think of feature store is, which is unfortunate because it's
• 44:43
  like, Oh yeah, it's a feature store.
• 44:45
  It's a place where you store features.
• 44:46
  And this is where like the vector store comparison comes
• 44:48
  in. It's like you store your
• 44:49
  features in the feature store and the vectors in the vector
• 44:51
  store. But how, if my vectors are
• 44:53
  features, where do they go? And that's where people get
• 44:55
  confused. But I would almost argue that
• 44:58
  it's a feature store isn't really a feature store.
• 45:01
  It's not really a storage of features.
• 45:02
  It's just a term that got coined and all of us are kind of stuck
• 45:06
  using other companies like Tecton, for example, they've
• 45:09
  gone and tried to rebrand the term.
• 45:12
  We call ourselves a virtual feature store, they call
• 45:14
  themselves a feature platform, which all are just trying to
• 45:18
  show, hey, it's not just a storage of features.
• 45:21
  And so if I get to like my next maybe category, which I call the
• 45:25
  physical feature store, some would call like a feature
• 45:27
  platform. And this is where like Tecton
• 45:30
  lives and others live. And the approach there is that
• 45:34
  you do the compute. So it's not just it's the
• 45:36
  features defined as a pipeline, which I think is the correct
• 45:40
  definition. If I think of a feature in my
• 45:41
  head, the top five things the user clicked on, like when I
• 45:44
  think of that, I'm thinking of it as like almost like a, a
• 45:47
  sequel query in my head. It's not the table at the end,
• 45:51
  it's the actual definition. And so the physical feature
• 45:56
  stores approach is like, hey, data scientists can give us
• 45:59
  these definitions, we will build them and make them production
• 46:03
  ready for you so that you can very easily iterate on features
• 46:06
  using the prod, get that monitoring, get the version and
• 46:08
  get all that those nice things that you need.
• 46:11
  And we have this hyper tuned compute framework that we think
• 46:18
  is even better than Spark. You mentioned chop, that's kind
• 46:21
  of very they've mentioned they're saying like, are you
• 46:23
  tired of Spark? Like you use, you use chop.
• 46:27
  And so that's the approach. It's like, OK, we are owning the
• 46:30
  feature pipeline and we are building those features better
• 46:33
  than you can yourself and better than Spark can.
• 46:37
  So excuse me, this is what Nicholas was doing with Big
• 46:41
  Query. They were computing.
• 46:43
  And now that you say that, I'm remembering the conversation
• 46:45
  because he was saying before we implemented Big Query as like
• 46:48
  the standardised spot, we had all all these ad hoc jobs
• 46:53
  running because data scientists would introduce a little bit of
• 46:57
  feature compute into their code. And then when they would run it,
• 46:59
  it would run here or there or whatever.
• 47:02
  But once we introduce Big Query in, we had it standardised like
• 47:05
  day and night. We would compute all of them and
• 47:07
  you would get the most recent version right?
• 47:11
  Yeah. And this is like super, yes.
• 47:14
  And that's like the value and that would be I would call kind
• 47:16
  of like in the that feature platform place.
• 47:19
  Now if a company in the Teacher platform physical feature store
• 47:23
  space was to go and sell to this person, they would say, yeah, we
• 47:29
  do it even better than Big Query.
• 47:31
  So we will process those features, we will kind of will
• 47:33
  read from Big Query like a source and then we will compute
• 47:37
  them and then put them and essentially we will handle the
• 47:40
  data from there. The benefit is it truly might be
• 47:43
  faster, cheaper, whatever the con is typically adoption cost.
• 47:49
  All of the physical feature stores are proprietary.
• 47:55
  There's typically because of proprietary, you can't really
• 47:59
  tune them in the same way you could tune something like Big
• 48:01
  Query or Spark. There's no such thing as like an
• 48:03
  expert at tuning listings. It just doesn't exist yet.
• 48:07
  So the con typically comes with partly data gravity, except I'm
• 48:12
  already in Big Query. Like I already have Big Query, I
• 48:14
  already have data breaks. Like why am I going to buy
• 48:16
  another compute player just for my ML team?
• 48:21
  And so that's kind of where our approach, which we call it the
• 48:23
  virtual approach, it's like virtualizing a layer on top.
• 48:27
  So pretty much what would happen if you built that feature
• 48:30
  platform, but rather than actually building Compute
• 48:34
  Engine, we just build plug insurance to big Query, Sparks,
• 48:40
  Snowflake, all these different tools.
• 48:43
  And we do the, we tune those tools as well as we can to do
• 48:47
  the future compute. So from the data scientists
• 48:50
  perspective, all they're doing is giving us like their data
• 48:53
  frame transformation or sequel transformation, whatever.
• 48:56
  We will then handle production izing that feature on their
• 48:59
  existing data infra. And so, yeah, the benefit we see
• 49:03
  is just adoption cost is way lower for companies.
• 49:05
  Like what most companies build in house kind of looks like an
• 49:11
  ad hoc version of feature form because almost no one is going
• 49:14
  to build a compute platform in house and no one's going to be
• 49:17
  like, we're going to build a better Big Query.
• 49:19
  I mean, that's what Airbnb and Uber did back in the day.
• 49:23
  But I just don't think that the math makes sense anymore, I
• 49:26
  suppose to give in that other things exist out there.
• 49:28
  So really they end up building on top of Big Query or Spark or
• 49:33
  whatever, but that's what feature form is.
• 49:35
  And feature form is open, open core.
• 49:36
  So like if you're going to do that and you don't want to pay
• 49:39
  like, you can still just use feature form to do that for you.
• 49:43
  So explain a little bit more about this idea of connecting
• 49:48
  things and how. Then you're using these engines,
• 49:54
  whatever the compute engines are, and then you're spitting
• 49:56
  out the features to the virtual feature store, and then it lives
• 50:01
  there. And how is that different?
• 50:03
  I'm trying to see how that's different than what you were
• 50:05
  saying in the beginning. The literal feature store of
• 50:08
  like, OK, it just lives here and these are the features.
• 50:12
  Yeah, so two points. Ones, you're not ever really
• 50:15
  spitting it out to the virtual feature store.
• 50:17
  It is the whole thing. And it may put one way, it's
• 50:21
  like the literal feature store is like cash, like a database
• 50:26
  for features. Yeah, the physical feature store
• 50:29
  is like a compute and storage framework for or engine for
• 50:34
  features. So it's doing the compute, it's
• 50:36
  doing the storage. The virtual feature store is
• 50:39
  kind of more like an orchestrator and metadata.
• 50:43
  So whereas like in the physical feature store you would compute
• 50:47
  and vert compute engine and then you store in their storage
• 50:50
  engine or maybe Discord, Dynamo, Redis, but generically their
• 50:53
  compute, their storage. We do the same thing, except you
• 50:57
  plug in the compute like Big Query and you plug into storage
• 51:00
  like Redis. And then we will make all those
• 51:03
  things kind of work together. Click House has written quite a
• 51:05
  few articles about us and about using feature form and Click
• 51:09
  house because from their perspective so much daybreaks,
• 51:12
  they want to have more use cases of quick house.
• 51:15
  And ML is a big one for about. And so a feature store just
• 51:18
  makes it easier to use Click House for ML so that data
• 51:21
  scientists don't have to learn the insurance and outs of using
• 51:24
  click house and optimising click house pipelines.
• 51:26
  They can use feature form, which already has all that built in.
• 51:29
  And so now a data scientist has to think about, hey, I have this
• 51:32
  transformation, I'm going to write it, I'm going to register
• 51:35
  a feature form. Feature form will then build it
• 51:37
  and put in prod for me, keep it up to date, do stream
• 51:40
  processing, incremental processing, dew point, time
• 51:42
  preference, all the things you'd expect to be done.
• 51:45
  Feature form does similar to what a feature platform does,
• 51:49
  but it's being done on your data infra.
• 51:53
  I see. I understand now you're the
• 51:56
  connector where there's already whatever Spark cluster or you
• 52:02
  guys do Hadoop also you. We don't, we don't, We do have.
• 52:06
  It's been asked for, we've been asked.
• 52:08
  We've had some interesting because we work a lot bigger
• 52:10
  companies and enterprise. We've had some interesting asks.
• 52:14
  Well, it's funny and just a random side note on the Hadoop
• 52:17
  thing. Someone came into the community
• 52:19
  the other day and they were saying, like I've had it with
• 52:22
  Hadoop, the community, the Hadoop community is a graveyard.
• 52:26
  If you want to make any changes, your PRS don't get merged until
• 52:31
  three months later maybe. And so we're watching Hadoop
• 52:36
  just slowly crumble between before our eyes right now, which
• 52:41
  is an interesting thing as technology is constantly
• 52:45
  updating. You can see that.
• 52:46
  But tangent aside, I, I get it. So you've got the connectors.
• 52:52
  Nothing new needs to be brought in as far as the storage or
• 52:57
  compute side of things. It's just that you're able to
• 53:02
  optimise whatever is already in production or whatever is
• 53:06
  already on your stack. So you're optimising it by
• 53:09
  saying, hey, data scientist, like you don't need to learn
• 53:12
  click house. You can just do what you're best
• 53:16
  at and let us work on that optimization of click house.
• 53:21
  Yes, I think that's true. Maybe to make it like a little
• 53:24
  more clear, let's say you as a data scientist or working on
• 53:29
  like a streaming pipeline like so you have like a stream of
• 53:32
  user buys and you need to build a feature there in the literal
• 53:38
  feature store. It would be your job or your job
• 53:41
  to go to Data Henge to build that pipeline and flame current
• 53:44
  Spark streaming or whatever. And then you would have to have
• 53:47
  you have to build a production grade data and pipeline and
• 53:50
  manage it and maintain it and have monitoring and have all the
• 53:53
  things you would need yourself. That's the literal feature store
• 53:57
  and the physical feature store and the kind of feature platform
• 54:00
  approach. You would be able to do it
• 54:03
  yourself and it would work and be great and it would handle
• 54:06
  streaming quite well and it would scale well and it would be
• 54:09
  production grade, but it would have to run on their compute.
• 54:14
  So you have to bring a new compute engine in and your
• 54:16
  storage engines proprietary engine in feature form.
• 54:22
  You still just write your SQL query like you would feature
• 54:25
  form will then take that SQL query and it knows it has like a
• 54:28
  harness to get that SQL query in and run it in a way that is a
• 54:31
  production grade data pipeline. So it's really clear when you
• 54:35
  start thinking of things like incremental processing and
• 54:37
  streaming, on top of having versioning, lineage search,
• 54:41
  reuse, all the other stuff you'd expect as well.
• 54:45
  OK. Now you did mention before
• 54:49
  there's this LLM aspect to things and you were saying it's
• 54:54
  when it comes to the day to life cycle, it's also very similar.
• 54:59
  Although LLMS are this new thing, you still almost have
• 55:02
  like the same day to life cycle and you still have the same
• 55:05
  pains. And I think that's why we're
• 55:07
  seeing a million evaluation companies come out.
• 55:10
  So I agree with you there. Can you go a little bit deeper
• 55:13
  on to what you've been seeing and how you've been thinking
• 55:16
  about? That yeah, I think there's a lot
• 55:19
  there. I'm going to actually just fully
• 55:21
  focus on this RAG, the concept of RAG and the data part of
• 55:24
  ALMS. So RAG when we first kind of came out as a concept was
• 55:33
  this idea of I take my documents, I chunk them somehow,
• 55:37
  I embed them, I do a nearest neighbour look up based on the
• 55:40
  embedding of the users prompts or query and I have some prompt,
• 55:45
  the template that I feed my retrieval.
• 55:49
  So if I do a nearest neighbour look up, I get 5 paragraphs from
• 55:53
  all of my documents that most look like the question and then
• 55:58
  I'll just feed that into my template and it works pretty
• 56:02
  well like it it does. I would just like naive RAG
• 56:05
  implementation and a lot of people did that and they wrote
• 56:08
  this is it, this is cutting edge.
• 56:10
  I think we're going to look back at this and be like, what's so
• 56:13
  funny that we thought that this was like the end all be all.
• 56:16
  This is what RAG is. I think RAG at its core is the
• 56:21
  problem of I have a context window.
• 56:24
  It's either a set size again size or the bigger I make the
• 56:28
  context window, the more expensive the query is.
• 56:31
  And then just I don't want to just put fluff in the context
• 56:33
  window. So my goal is to have the most
• 56:36
  information dense context window possible, which is really an
• 56:39
  information retrieval problem. Like how do I retrieve context,
• 56:44
  the best information that I can feed into my context window and
• 56:48
  where a lot of people think of like text and nearest neighbours
• 56:51
  and etcetera. Imagine if I asked you pay
• 56:55
  Demetrius, how should I invest my money?
• 56:58
  I want to save up for retirement.
• 57:00
  You would probably ask about my age, my income, my risk profile.
• 57:05
  You asked a lot of questions about me and about how I you
• 57:08
  know my situation. Those answers are not in a
• 57:13
  vector database. Like you're not going to get
• 57:15
  someone's age. You're not going to do a vector
• 57:16
  database look up to get their age.
• 57:18
  Those might exist in like a post Crest database somewhere.
• 57:21
  But if you're. Giving that context to a prompt
• 57:23
  like it would make sense to say, hey, I have someone who is, you
• 57:26
  know, 30 years old. They have this much money, they
• 57:29
  do this at this, but no one thinks of it that way.
• 57:32
  Like everyone thinks that you have to do nearest neighbour
• 57:35
  look ups and embeddings and vector DBS.
• 57:37
  And I don't think there's any reason that that is true other
• 57:41
  than one, that it does work quite well naively.
• 57:44
  And two, it just those companies came about first.
• 57:46
  And almost all the content about RAG is written by those
• 57:49
  companies or proponents of those companies.
• 57:51
  So I, I, and this isn't even like a hit on vector DBS.
• 57:55
  I think vector DBS are extremely powerful.
• 57:56
  Like I said earlier, I just think it's part of the puzzle
• 57:58
  and really the problem spaces. And I think honestly, where a
• 58:02
  lot of vector databases are going to go is become less a
• 58:05
  vector database and more like an A contact retrieval database
• 58:10
  where I give it a prompt and maybe some other metadata.
• 58:14
  And it has all of these signals that maybe like exists in a
• 58:18
  feature store or something and it decides what to pull from the
• 58:22
  feature store and feed into the prompt.
• 58:25
  That's kind of the architecture that I think things are going to
• 58:27
  move to. We're already seeing a little
• 58:29
  bit of that happening. Oh, I love it.
• 58:32
  Yeah. So basically right now that
• 58:40
  personalization aspect in the prompts you don't see as much.
• 58:46
  I've seen people talk about you have variables and you want to
• 58:52
  have variables that you can use across prompts.
• 58:55
  And so you store those variables in some kind of a database and
• 59:00
  then you can reference them in different prompts.
• 59:03
  That's been really cool to see. I have heard of the idea of
• 59:08
  let's personalise prompts a little bit more.
• 59:13
  I feel like instinctively this works for certain use cases but
• 59:19
  not for others. But then as I was trying to
• 59:22
  think about what use cases it wouldn't work for, I couldn't
• 59:25
  think of any off the top of my head.
• 59:27
  So I was like, well, maybe it that's not true.
• 59:30
  I'm just making that shit up in my head.
• 59:33
  So I don't know. Do you have any insight on if
• 59:36
  it's more relative for certain use cases versus others?
• 59:43
  I think it probably is. I think what you'll find the
• 59:50
  most valuable use cases for alarms, because alarms are
• 59:53
  relatively slow, they're kind of expensive.
• 59:56
  You're not going to use alarms for fraud detection.
• 59:59
  Most of where you see LMS being used and where values coming
• 1:00:02
  from is making like Co pilots, like making people better at
• 1:00:06
  their job, making them move faster.
• 1:00:07
  So we're interacting with the human almost always and I think
• 1:00:12
  that's actually where LMS do best, their augmentation tool.
• 1:00:15
  I'm personally a little sceptical of what can be done in
• 1:00:18
  the automation space. True automation where no one
• 1:00:20
  touches it because it just. I don't think you can achieve
• 1:00:23
  that 99% accuracy you need. I mean, you can get to like 95,
• 1:00:27
  which sounds great until you realise that, hey, if it's
• 1:00:30
  fraud, you know, like that 5% of fraud that's getting through is
• 1:00:34
  not just like a, you know, we can't just look.
• 1:00:36
  Over that accounts, yeah, that's not super fun to think about.
• 1:00:40
  So I think that everything in some way has its level and it's
• 1:00:44
  not just personalization for the user, but I do think that's a
• 1:00:47
  big piece of it. Even just like what you're
• 1:00:49
  calling variables, I would, I mean, those are just features or
• 1:00:52
  signals or whatever. Like we can give these things
• 1:00:54
  different names, but really all it is, is I have these
• 1:00:57
  conceptual things and I want to pull numeric or textual data
• 1:01:01
  points about them so I can feed that into the model.
• 1:01:04
  So the model can do a very job. And I mean at its core that is a
• 1:01:09
  feature. Yes.
• 1:01:11
  You're opening my mind now because I was thinking of it
• 1:01:14
  just as attributes of a human, but it can.
• 1:01:19
  It's exactly that. Like the variables I know are
• 1:01:24
  describing a company. And then in your prompt you just
• 1:01:28
  put the company name in quotations and it can grab that.
• 1:01:32
  But that is exactly what it is. It's a feature, Yeah.
• 1:01:35
  So you have all these different variables that you want to be
• 1:01:38
  able to give the most context and the most accurate context to
• 1:01:42
  the prompt so that LLM can do the best job that is possible.
• 1:01:48
  And so it makes sense that you would break it down and really
• 1:01:52
  get as descriptive as possible for all these important themes,
• 1:02:00
  not just like, hey, this is company A.
• 1:02:03
  It's like if you have a very rich description of what company
• 1:02:08
  A is, you add that as the variable.
• 1:02:12
  Yes, and you're saying like rat frag and knowledge graphs kind
• 1:02:16
  of get pushed to, which is a very similar pitch of what I'm
• 1:02:19
  saying. And the really funny thing is,
• 1:02:23
  is like then you have graph frag and they're like you need that
• 1:02:26
  and not factor databases. Like there's this weird like Eva
• 1:02:30
  or philosophy even rag versus fine tuning versus like awkward.
• 1:02:34
  It's like everyone just trying to justify that they are the
• 1:02:36
  best way to tune because there's no real good data yet.
• 1:02:39
  Like no one knows. I think they're really there's
• 1:02:41
  very few enterprise grade LM products in the world.
• 1:02:44
  I think that really the only one that you can really look at as
• 1:02:47
  like gold standard is a copilot. I think everything else is still
• 1:02:54
  either is still kind of been like very valuable, but maybe
• 1:02:57
  more even consumer driven, which our consumers can handle a
• 1:03:00
  little more like, you know, it's, it's, it's like can be a
• 1:03:03
  little weird sometimes or just very experimental.
• 1:03:07
  Like in in B to B, you see a lot of things are going out that
• 1:03:10
  don't actually work very well. They have great demos, but they
• 1:03:14
  they're not quite there yet. And I think where we'll get to
• 1:03:17
  that next level is when we get to the point where it's just
• 1:03:21
  like, hey, here's your data. I have this a very specialised
• 1:03:24
  retrieval, these very specialised prompts.
• 1:03:27
  Maybe those are models themselves, which I've actually
• 1:03:29
  seen people starting to use traditional.
• 1:03:30
  I'm not models for re ranking, which is fascinating and yeah.
• 1:03:35
  That a little more. I didn't catch that.
• 1:03:38
  Sounds cool. So if I have a million signals
• 1:03:44
  that are relevant to the prompt, yeah.
• 1:03:47
  The question then becomes which ones do I feed into the prompt
• 1:03:52
  for this specific query? And how do I properly place them
• 1:03:56
  in the prompt so that the model can understand it And that
• 1:04:02
  question and how to do that? I think that is where every like
• 1:04:10
  a Llama index S type company that I think that's where
• 1:04:14
  everyone kind of starts. That's where the big hard open
• 1:04:17
  question is. I think the vector databases
• 1:04:20
  will will start to compete there.
• 1:04:21
  I think a lot of the companies will start to compete in that
• 1:04:23
  space. And I really think that's
• 1:04:25
  probably one of the hardest problems to solve.
• 1:04:27
  And if you can solve it, well, it just becomes a no brainer to
• 1:04:32
  use. But I actually similar to lead
• 1:04:34
  to feature stores, I don't think there's really a perfect way to
• 1:04:39
  solve it. So you ever get very specialised
• 1:04:40
  verticalized ones or you will get and and or you will get more
• 1:04:46
  generic ones that give you ways to tune them and specify what
• 1:04:50
  you want to do where. But really that's the that's the
• 1:04:53
  problem to solve without lines. And I think a lot of people get
• 1:04:56
  too caught up and like I need a vector database and I need to
• 1:04:59
  the nearest neighbour look up and they forget the core problem
• 1:05:01
  which is just information like packing.
• 1:05:04
  Like how do I pack the right info and the right density of
• 1:05:08
  information to achieve the best result?
• 1:05:11
  I think that's why for a lot of people, DS Π felt so interesting
• 1:05:18
  because it has that capability of like fanning out and testing
• 1:05:24
  a lot of different prompts and then saying, well, here's the
• 1:05:26
  winner of this choice, right? And so you could see that being
• 1:05:32
  something that becomes more attractive and you you start to
• 1:05:36
  see incorporated into a lot of these frameworks.
• 1:05:40
  Yeah, I think that's true. I think evaluation, I don't go
• 1:05:43
  too deep into it, but I find evaluation to be really
• 1:05:45
  interesting and kind of funny, similar to how recommender
• 1:05:48
  system evaluation used to be. Because the hard part of
• 1:05:52
  recommendations is most people don't give it a thumbs up or
• 1:05:55
  thumbs down when you give them a recommendation.
• 1:05:57
  Very rarely happens. And on top of that, there's
• 1:06:01
  almost like a certain profile of person who does that.
• 1:06:03
  And that's probably not the average person.
• 1:06:06
  So if you take both of those things into account, the
• 1:06:08
  question becomes, well, how do I decide if this was a good
• 1:06:10
  recommendation? And the answer is implicit info,
• 1:06:14
  which is like what Google does like, OK, Google, if you search
• 1:06:17
  something, did you click, how long did you stay on that page?
• 1:06:19
  Did you go back and click on another page that you can kind
• 1:06:21
  of deduce if a user had value? And I think evaluation is so
• 1:06:27
  focused on like, we're going to do this statistically or
• 1:06:31
  mathematically, or even funnier, just like we're just going to
• 1:06:34
  layer L alums on top of LLMS until you're so confused about
• 1:06:37
  what's happening that you can't understand.
• 1:06:39
  And will you ever say this is good or bad when really the
• 1:06:41
  answer is all that matters is the end user.
• 1:06:46
  And like we talked about, most of the time, the end user is
• 1:06:48
  literally a human. So did they find value in it?
• 1:06:53
  If yes, then that's good data. And if no, if and no, and this
• 1:06:56
  is actually open like ChatGPT does, right?
• 1:06:58
  Like sometimes they'll give you the two prompts next to each
• 1:07:00
  other. Like for sure, they're using
• 1:07:02
  that information to like decide and then tune their models.
• 1:07:06
  So I think that's what we're going to, you're going to always
• 1:07:08
  see these new user experiences that are driven to teach the
• 1:07:13
  model more, which again, we saw in recommendations.
• 1:07:15
  And I think that we're going to see the same thing in L1.
• 1:07:17
  So I think it's so interesting coming from Rex's to LMS and
• 1:07:22
  also having seen a lot of things in between, because I just am
• 1:07:24
  seeing the same pattern play out.
• 1:07:27
  History rhymes for sure.