Latest / Agentic Conversations (formally mlops.community) / Feathr: LinkedIn's High-performance Feature Store // David Stein // Coffee Sessions #120

Transcript

Feathr: LinkedIn's High-performance Feature Store // David Stein // Coffee Sessions #120 · Agentic Conversations (formally mlops.community)

• 0:00
  I'm David Stein. I am a senior staff software
• 0:03
  engineer at LinkedIn, working on ML Ops, and machine learning
• 0:06
  infrastructure and coffee. I like my coffee black and I
• 0:12
  love fills coffee. And I have been male ordering it
• 0:15
  and making it home since the pandemic, mr.
• 0:17
  Schuyler, welcome back. This is awesome to have you
• 0:22
  here. Today, we're talking with David
• 0:23
  Stein, fellow buddy, of yours from the LinkedIn days, and
• 0:27
  there's a lot we chatted about, I've got So many cool takeaways
• 0:31
  but I want to hear yours first, yeah.
• 0:33
  You know, one of the biggest takeaways for me was just a lot
• 0:37
  of the talk around. First of all, open sourcing
• 0:42
  feather and sort of the journey to get there and then also kind
• 0:46
  of like what's next? And it was really interesting,
• 0:49
  hearing David's perspective, because when David started
• 0:53
  building feather feature stores, did not exist.
• 0:56
  That, that was not like a term that was floating around at all.
• 0:59
  Yeah. Was it 2016?
• 1:02
  This was probably to 2015. I think, when they started,
• 1:05
  well, maybe 2016 but it was, you know, this was definitely like
• 1:11
  one of the OTG feature stores for sure.
• 1:14
  And so David has kind of been in this space and just steeping in
• 1:18
  this problem for a very long time.
• 1:21
  I loved hearing about sort of like how they got to this point
• 1:24
  of like having an open source release in using LinkedIn,
• 1:27
  really as an incubator to like build a really great product.
• 1:29
  T', that that's such a good point.
• 1:32
  How he at the end mentioned LinkedIn, he could build a
• 1:36
  better product at LinkedIn. Then spinning out his own
• 1:40
  company because he has so much access to people like you back
• 1:43
  in the day that he could just go and sit next to and read their
• 1:46
  code or ask questions and have a relationship with and bump into
• 1:51
  in the cafeteria lunch. Like that is such Insight that I
• 1:55
  hadn't even thought of because automatically you go to like, oh
• 1:58
  yeah, why wouldn't You start this on your own, you've been
• 2:01
  doing feature store since 2000 16 2015, like going and do this.
• 2:06
  And so he gave a nice little plug as to why building it at
• 2:10
  LinkedIn. Was arguably a better option
• 2:13
  than going and building it out in the world.
• 2:17
  And I personally loved when he talked about, how he does, not
• 2:24
  feel like real time is being so quickly adopted because the two
• 2:29
  Rules are not in place, and so I think that's huge.
• 2:33
  I know there's a ton of companies that are trying to go
• 2:36
  at this. Real-time problem, you mentioned
• 2:38
  chips. I know tecton starting to do a
• 2:41
  lot of their stuff with real-time.
• 2:43
  There's a few others that I see. Like it's almost every other
• 2:48
  week. I feel like somebody comes out
• 2:51
  and says, like, we're the XYZ platform for your real-time data
• 2:57
  and there's a problem there. There's definitely like people
• 3:01
  are looking for a solution and the way that he broke down, why
• 3:08
  it is so important because of the ability, if someone has
• 3:12
  confidence that the tools they're going to use are going
• 3:15
  to work and they're going to be able to use them quickly, then
• 3:19
  they get that many more Cycles on them.
• 3:21
  They get that many more iteration time or that much more
• 3:24
  iteration time to go through it. It just makes them more likely
• 3:28
  to try. Try that use case XYZ with, with
• 3:36
  real-time features and real-time capabilities.
• 3:39
  I did also like, how you push back a little and said, well,
• 3:42
  not everything is for Real Time right now, especially you've
• 3:46
  been working on it and I know this is a strong belief that you
• 3:49
  have. Like, real time might be
• 3:51
  Overkill, a lot of the time, it's not necessarily the most
• 3:56
  important thing to have these real-time features coming
• 3:59
  through. Because the features aren't
• 4:02
  going to change over the course of an hour that much so that was
• 4:06
  huge and at the risk of just babbling on a little bit more.
• 4:10
  I also wanted to talk about when he when he said the idea of
• 4:15
  making things more simple, so that machine learning Engineers
• 4:19
  can understand what they're working on and then decoupling
• 4:24
  features from code that like that kind of blew my mind and
• 4:28
  having this features Is it as something that engineers and
• 4:32
  whether its data Engineers or data scientists or machine
• 4:34
  learning Engineers, they can reason with, they can look at
• 4:38
  they can change and it's irrespective of the code that
• 4:42
  built the model. So that was a huge one.
• 4:46
  It's probably one that is not such a big surprise to people
• 4:50
  but you know I got a thick skull and it takes a while for things
• 4:53
  to enter into my mind. And this finally was he put it
• 4:58
  in eloquent way so that I So, David Stein, he is machine
• 5:02
  learning engineer on the platform, on the feather
• 5:05
  platform at LinkedIn. And we talked to him about so
• 5:09
  much good stuff today, and that's it, man.
• 5:14
  I don't have anything else. You have anything else you want
• 5:16
  to mention it. Only that it was a pleasure
• 5:18
  getting to chat with David Again.
• 5:20
  David is careful thinker. And I always appreciate
• 5:25
  regardless of how loudly I would rant at LinkedIn and give you
• 5:29
  always found a way. A to respond calmly.
• 5:31
  So really, really appreciated the opportunity to chat with him
• 5:34
  again. Yeah, it seems like he would
• 5:36
  have been awesome to work with. I'm sure you enjoyed that and I
• 5:40
  know that you told me just before we hit record like you
• 5:42
  would happily work with him again and so we do got to give a
• 5:47
  shout out to you're having a happy hour in La for the ml Ops
• 5:50
  Community. Yeah, tomorrow night, there It
• 5:54
  Is. By well, it's probably like a
• 5:56
  Week Ago by the time this Exactly.
• 6:01
  So but you're going to be doing more.
• 6:03
  So if anybody's in LA and they want to hang out with Skylar or
• 6:05
  hot or any of the LA crew then like there's a whole meet-up
• 6:10
  page, we've got meetups happening all over the globe in
• 6:13
  real life because we're sick of being behind screens and we've
• 6:17
  done enough virtual events, in the last two years, let's get
• 6:20
  out meet each other and hopefully, we're going to be
• 6:24
  giving away fun things like at certain meetups, I was just
• 6:28
  talking to the Berlin crew. Ooh, and we're going to be given
• 6:30
  away. Raspberry Pi's for different
• 6:38
  trivia contest that we're doing. So we're going to try and make
• 6:40
  the in real life meetups fun and a good time.
• 6:44
  You're just doing the happy hour maybe later on, it's going to
• 6:46
  develop into a full-blown meet up.
• 6:49
  Let's see. Alright, so now further Ado
• 6:53
  we're getting into it with David.
• 6:57
  All right. Where do we even begin with this
• 7:00
  one? I think we should start with the
• 7:03
  Skyler and David both knew each other back in the day when
• 7:08
  Skyler was working at LinkedIn, and this is a reunions of sorts.
• 7:12
  Yeah, I'm definitely. Super happy to see David again,
• 7:15
  really enjoyed working with them in the past.
• 7:17
  It's so glad to see he's doing. Well, yeah, I just got to say
• 7:22
  likewise Skyler. It's really great too.
• 7:25
  Do it be here again with you for sure.
• 7:29
  So since the time of working together you David have been
• 7:35
  very busy because you created this feature store that LinkedIn
• 7:42
  uses and then you not only created it and we're using it at
• 7:45
  LinkedIn later. Skylar can tell us if you guys
• 7:48
  were using that back in his day but you open source this.
• 7:52
  So I want to get into the feature store.
• 7:54
  I want to get into just the Infrastructure at LinkedIn,
• 7:57
  where you think ml Ops is going. And then the whole experience of
• 8:02
  open sourcing something because I know that can be a bit of a
• 8:05
  beast and how it's been so far? I just got a comment one thing
• 8:10
  before I ask you the first question.
• 8:11
  I love this background, you've got.
• 8:13
  It reminds me of like, when Neo and Morpheus jump into the
• 8:16
  Matrix right before they jump in.
• 8:17
  And they're like everything's white and then all of the
• 8:22
  different shelving comes in around them.
• 8:24
  I'm expecting shelving. To come in around you at any
• 8:26
  moment when you have to talk about the different tools that
• 8:30
  you use LinkedIn. So but anyway man why don't we
• 8:35
  start out with this? Like can you give us just a bit
• 8:37
  of background on feather and what exactly this feature store
• 8:41
  is? We probably know what a feature
• 8:43
  stories. I'm guessing a lot of the people
• 8:45
  that are listening have heard it many many times before, but what
• 8:50
  is feathers unique feature stored?
• 8:52
  Miss sure. So Feather started out really
• 8:58
  like six years ago, it had a different name, internally at
• 9:02
  LinkedIn for a lot of its history.
• 9:04
  So, Skyler would know about it, probably mostly by its name
• 9:08
  frame. That we've also talked about
• 9:09
  publicly or think it's mentioned in the blog post about feather.
• 9:13
  But yeah, we at LinkedIn years ago because we've been, we've
• 9:18
  been doing ml using it in production for a lot of LinkedIn
• 9:23
  businesses for a long time like more than 10 years.
• 9:27
  Recommendation systems for the news, feed for advertising for
• 9:31
  notifications, relevance for all different, kinds of things.
• 9:33
  And there's lots of like, you know, research papers and other
• 9:36
  blog posts that LinkedIn has put out about a lot of these
• 9:39
  applications. We've been doing a lot of these
• 9:41
  things for a long time, and a huge issue that we encountered
• 9:50
  is that preparing features and managing the features.
• 9:55
  So, the inputs to machine learning workflows is really,
• 10:00
  it's a big part of the, of the challenge in kind of
• 10:03
  operationalizing and managing machine learning workflows.
• 10:07
  So we were even saying that it's the in some of the large machine
• 10:15
  learning applications, it was even like the hardest or the
• 10:18
  most complicated burden. Something to actually manage
• 10:23
  these feature data preparation workflows that had been growing
• 10:28
  for years and different generations of people on the
• 10:32
  team. You know, like years of turnover
• 10:35
  and new engineers and researchers coming in.
• 10:37
  And adding a new feature here, a new feature with are adding a
• 10:40
  join here, join their, you get. We started to see these giant
• 10:45
  complicated workflows. Yeah.
• 10:49
  Kind of for Using like Azkaban, which is linkedin's kind of
• 10:54
  internal version of airflow, that's often.
• 10:55
  And it's also open source. Basically, like all of these big
• 10:59
  data processing workflows that have gotten really complicated,
• 11:01
  that really were trying to do something that I thought was
• 11:05
  simple should be seen. As a simple thing, which is
• 11:08
  Define a set of features compute them from their Source data load
• 11:13
  them into the context for training and then be ready to do
• 11:15
  the same thing for online and production.
• 11:18
  So I guess I could say that say that all more simply feature
• 11:25
  preparation is like the hardest thing.
• 11:28
  One of the hard part is parts of actually doing applied ml at
• 11:33
  scale for a large project for team over time and so we build
• 11:37
  feather as a solution to kind of simplify feature preparation.
• 11:43
  To make it so that you define a name set of features and then
• 11:46
  get those features by their names in the training context
• 11:50
  and in the inference context instead of having to, like,
• 11:52
  manage a whole bunch of application-specific workflows
• 11:55
  to do that. That's sort of the origin story.
• 11:59
  I'll let you steer me with more questions about what facets.
• 12:02
  Do you want to go into? Yeah, definitely.
• 12:06
  And you know, one of the things that is really awesome that I
• 12:09
  just want to point out is that you talking about how you
• 12:12
  started this project like it, Years ago.
• 12:14
  And I, I remember the very, very earliest days of frame and it's
• 12:19
  really cool to see the Milestone and landmark of like, actually
• 12:22
  seeing it become like a full-fledged open source project
• 12:26
  and mostly first wanted to congratulate you because like,
• 12:30
  obviously that's a lot of work and a lot of cycles that you had
• 12:33
  to go through a lot of learnings.
• 12:36
  And so, one of the things that I was curious as you were, you're
• 12:39
  talking about this is, you know, you have Like six years of
• 12:44
  lessons to learn, what do you think are like the lessons that
• 12:47
  you learned in building? This is like, you know, one of
• 12:49
  the things when we were talking to he and the other week, you
• 12:53
  know, LinkedIn was in this interesting position where like
• 12:56
  feature stores didn't really exist.
• 12:58
  So to speak when you started this and so you're kind of like
• 13:03
  one of the people that sort of spearheaded, you know, the
• 13:06
  creation of feature stores. And so like I'm sure there's
• 13:09
  tons of interesting lessons you learned along the way about what
• 13:11
  works and feature stores. What?
• 13:13
  And how to get people to use them.
• 13:15
  So really just curious to get your take as what stands out as
• 13:18
  important. Learnings there's so many
• 13:20
  lessons both in terms of the, how to help a team deliver.
• 13:27
  Some, you know some technology Mission especially when it's in
• 13:31
  a new space like this where we're still collectively I think
• 13:35
  as an industry figuring out what the right solution for feature
• 13:39
  management platform and for feature store should look like
• 13:43
  Leading a team to deliver it and refocusing.
• 13:49
  The needs based on what we hear from users.
• 13:52
  There's, there's many lessons in there, I think. one of the
• 14:04
  things that, I think was most important is the Importance of.
• 14:19
  It almost sounds obvious but having really clean Concepts
• 14:22
  that are going to work. Well for users, what like
• 14:25
  something like this with the feature store, we view it as a
• 14:34
  It is a device to actually help the users who are the machine
• 14:38
  learning Engineers, think more simply about the system about
• 14:43
  the machine learning platform, the thing that they need to be
• 14:46
  deploying their models into and running them.
• 14:49
  And when we're trying to make it easier for people to think about
• 14:52
  what they need to do, you need to have really good clean
• 14:54
  Concepts. You need to have named things
• 14:57
  that are that resonate with people in the work.
• 14:59
  They need to do getting those Concepts really clear.
• 15:04
  Early on is really helpful because you have to course.
• 15:08
  Correct, less later on. And when we make mistakes, like
• 15:12
  if there are, if we make things more complicated than they
• 15:15
  should have been, if we didn't polish down tightly enough, some
• 15:21
  Core Concepts, those become, you know increasingly challenging
• 15:25
  things that need that you know the cost of fixing those
• 15:27
  problems goes up later on. So I mean there's many different
• 15:32
  kinds of lessons there but The. Clean Concepts starting from the
• 15:43
  point of asking users and working together.
• 15:46
  The users to figure out. How do you want to specify
• 15:49
  things? How do you want to specify your
• 15:51
  feature dependency set of your model?
• 15:54
  How do you want to specify a feature definition?
• 15:57
  Can we use like a imagined configuration or imagined DSL as
• 16:02
  a sort of a thought device to shape how we want to think about
• 16:05
  the things we want to build the importance of those things early
• 16:09
  on in a Design process. I think contributed to the
• 16:17
  success and Longevity. I think of what we built for
• 16:22
  this project. I don't know, but there's many
• 16:27
  other angles to go here. Hopefully that covers that
• 16:32
  question. If I don't know if there's
• 16:33
  different angles in there, you want to go?
• 16:35
  Yeah, I think that's great. Yeah.
• 16:38
  And I think that's definitely a lesson that resonates with me
• 16:43
  seeing a lot of the churn that we had in trying to figure
• 16:46
  things out early on. I definitely recall like
• 16:49
  unclear, semantics like definitely made my brain hurt.
• 16:53
  But yeah. So You know, one of the things
• 16:56
  I'm sure they did a lot of listeners would be curious about
• 16:59
  because I don't think everybody has a good understanding of the
• 17:02
  scale of LinkedIn. And so, would love to talk about
• 17:05
  the the scale that you're handling from a few different
• 17:08
  angles. One, how many teams are using
• 17:10
  feather to, you know, how much data like is sighs?
• 17:15
  Are we talking about? And I think people would love to
• 17:19
  kind of like see that those bits on scale.
• 17:23
  Sure, sorry. I wish I Add my you know cheat
• 17:26
  sheet of the LinkedIn published. You know like yeah own and
• 17:30
  publicly released numbers about these things.
• 17:32
  I know that the you know, just in terms of scale LinkedIn
• 17:37
  obviously has many hundreds of millions of members and then
• 17:43
  numbers for, I assume that there are some numbers that folks are
• 17:49
  really interested in Lincoln scale, can look up things that
• 17:51
  are probably described on blog posts in terms of the scale of
• 17:55
  the The people who are coming to the website every day using
• 17:58
  different products, the number of different kinds of entities
• 18:02
  that need to, you know, the number.
• 18:04
  I mean, just the scale of the event of people using the
• 18:06
  product. And I think that most people who
• 18:11
  use the LinkedIn product who, for example, we use LinkedIn
• 18:14
  feed and no, you know, you can scroll and scroll and scroll and
• 18:17
  the diversity and variety of kinds of things that will show
• 18:20
  up there. You can think of this scale of
• 18:21
  some of these kinds of applications that we're dealing
• 18:23
  with. It's very, it's a very yard
• 18:25
  sale. A large one of the, you know, I
• 18:29
  wouldn't consider to be like at the magnitude of scale of
• 18:34
  something, like, in terms of traffic load and things like
• 18:36
  that, or in terms of the data volume, it's it's going to be
• 18:40
  like a, it's a one of the larger use cases, right?
• 18:42
  And speaking to this specific numbers of these things, number
• 18:47
  of teams using feather, this is because it's It depends on how
• 18:55
  you count in terms of sub-teams weekends, but there's like
• 18:58
  dozens of teams who are using feather, most of the internal
• 19:02
  users know a piety. Those same internal kind of code
• 19:04
  names that nice guy that you knew as particularly frame all
• 19:09
  those, a couple of other pieces inside there.
• 19:10
  Also a wrapped in the feather umbrella, the the history kind
• 19:17
  of went like this project started in 2016 and then really
• 19:22
  got implementation 2017 with with a collaboration with
• 19:27
  Engineers from various different machine learning like apply two
• 19:30
  male teens who are building different parts of the stack.
• 19:34
  We, it was a very kind of close partnership kicking this things
• 19:37
  off and that kicking this thing off.
• 19:39
  And there was a lot of early kind of Grassroots enthusiasm
• 19:44
  about this idea because it was this, you know, widely commonly
• 19:48
  understood issue that there's a big opportunity and simplifying
• 19:53
  the feature preparation aspect. Adding machine learning
• 19:56
  application workflow. So we had like, you know, many
• 20:02
  adopters in the first few years that helped us build this.
• 20:07
  And then we went through additional, you know, migration
• 20:12
  phases over the years. After that, where I am pretty
• 20:17
  sure it is a good. Majority of the applied ml, use
• 20:21
  cases at LinkedIn that are using this technology.
• 20:25
  So, I want to take a little bit of a step back because there's
• 20:29
  something that you said in that answer, the a few questions ago,
• 20:36
  where you were talking about, how feather is this device for a
• 20:41
  machine-learning engineer to think more simply about the
• 20:45
  system. Interact more simply with the
• 20:48
  system and I thought that was something that I hadn't heard
• 20:53
  before because it's like, you're taking Basically this very, very
• 20:57
  complex thing and you're serving it up to the machine learning
• 21:01
  engineer in a way that they don't have to understand the
• 21:04
  whole complexity of everything that's happening.
• 21:07
  But you're making sure that they do understand the important
• 21:11
  parts, and the parts that are pertinent for them.
• 21:16
  Am I understanding this correctly?
• 21:18
  Yeah, I guess to add a little bit to that. the idea of having
• 21:24
  a feature for like an, especially an entity feature,
• 21:29
  which is like the big important case for future stars, where you
• 21:33
  have features about users or features about job postings or
• 21:38
  features about feed items or about advertisements that need
• 21:42
  to be managed and fetched and looked up the idea that you have
• 21:46
  a named set of these things that can be captured in our registry,
• 21:50
  that can be reasoned about apart from the code that creates them
• 21:55
  or part from the models that use them was sort of a An innovation
• 22:01
  really before future stores. And before feather you would
• 22:09
  have named variables for features in Mike model code, but
• 22:13
  they would be, they were kind of coupled with the model code or
• 22:17
  coupled with the workflow that's doing the data processing.
• 22:19
  The idea that there should be like a like a named entity named
• 22:24
  like almost like an artifact or a named abstract entity like a
• 22:30
  There's a feature that is registered in a registry and has
• 22:33
  a name and that you can rely on that name.
• 22:37
  Meaning the same thing in the different parts of the
• 22:41
  ecosystem, right? Like any experimentation
• 22:45
  offline, you know, sandbox environment.
• 22:47
  And in also in the on the online production, serving environment
• 22:52
  that you can use that same name and expect it to mean the same
• 22:56
  thing and you just get to ask the infrastructure to provide
• 22:59
  that feature for You and you can rely on it being the same, you
• 23:03
  know, like in the training time and inference time, and that the
• 23:07
  underlying Machinery is going to make sure that that is the same
• 23:14
  being able to rely on that set of named features in a registry.
• 23:18
  First, you add the features and then you use, the features, is
• 23:21
  sort of a simplifier because before that, yeah, I like the
• 23:26
  engineers working on these applications would have to
• 23:29
  think, too much. Is that about the details of how
• 23:32
  this data preparation needs to work?
• 23:34
  They might have different names for feature columns in one
• 23:39
  context and you know, other names for them and our insurance
• 23:43
  online contacts and they'd have to write code that would need to
• 23:46
  remember which feature corresponding to which having
• 23:49
  these things ordered in a common registry with common
• 23:51
  infrastructure. That can provide, it does kind
• 23:53
  of simplify the things you need to think about.
• 23:56
  You're like, okay, I have a set of features, I import them into
• 23:59
  my model context. I have like you import
• 24:01
  dependencies into your code at the top of your Java or C++
• 24:05
  file, or something like that. Getting that cleanliness around
• 24:08
  named importable features does help.
• 24:10
  You think about what you need to do in terms of building and
• 24:13
  deploying a machine learning model.
• 24:14
  I love this idea of decoupling the features that you can reason
• 24:18
  with from the code to end, and then bringing that
• 24:22
  standardization. So that across the company, you
• 24:25
  understand that if you have this named thing that Going to be the
• 24:30
  same whether or not I am here or there and it reminds me this is
• 24:35
  a horrible like analogy, but I saw Skylar drinking some coffee
• 24:42
  and it's like going to a Starbucks, I can go to Starbucks
• 24:45
  anywhere around the world, right?
• 24:47
  And I know that the coffee is going to be pretty much the
• 24:50
  same. It's any kind of fast food chain
• 24:53
  that you have that dependent, or you have the predictability in.
• 24:57
  When you go in there, it's going to be the same and so that's
• 25:00
  kind of What it feels like you were trying to do is make that
• 25:02
  predictability across the organization so that when
• 25:05
  somebody is messing around with features, they know that they're
• 25:10
  messing with this thing, that is not that is the same across the
• 25:16
  org. And so I'm wondering too when it
• 25:20
  comes to open sourcing this because it feels like that that
• 25:23
  helps with the standardization peace and I actually was just
• 25:27
  recently listening to a podcast with the Of Hashi court and he
• 25:31
  was talking about how open sourcing in his mind, the key to
• 25:35
  open sourcing especially for Hashi, Corp was to drive
• 25:40
  standardization. Was that at all in your minds
• 25:45
  when you decided to open source it?
• 25:47
  He is the queen of chemicals that work and this is your daily
• 25:51
  dose of among herbs. Well, if you are dead serious
• 25:55
  about the Arabs, you have to immerse yourself in the
• 25:59
  envelopes. Content, the best way to do it
• 26:02
  is to subscribe to the email Ops Community podcast.
• 26:06
  All great people are here on this podcast soon, good luck and
• 26:10
  keep learning. So I'm not familiar with the
• 26:15
  with the example that you just mentioned.
• 26:17
  So I don't, I can't compare to that specifically but I know of
• 26:21
  other examples where of projects and systems where the open
• 26:28
  sourcing of a thing is specifically.
• 26:30
  He played a standardized like I'm thinking about things like
• 26:33
  Onyx for example, it's like let's define a common language
• 26:36
  for how model graphs can be represented in for purposes.
• 26:40
  That brings I would say. The reason why we open sourced
• 26:47
  feather is maybe a little different from that.
• 26:52
  We at LinkedIn. We have a kind of a culture of
• 26:56
  open sourcing. Our infrastructure pieces that
• 27:00
  We build that we're proud of and that we think may be of interest
• 27:04
  in use to the outside world. There are many examples of other
• 27:09
  LinkedIn projects that have been open sourced and you know that
• 27:14
  have made an impact in the industry.
• 27:17
  People talk about Kafka Pinot data Hub and there's others.
• 27:23
  So we believed with feather that we had a interesting solution.
• 27:30
  To a problem we saw and we're hearing about increasingly from
• 27:36
  Folks at other companies and so we, you know, in the same kind
• 27:40
  of spirit of open sourcing generally useful things that we
• 27:44
  built at LinkedIn. We wanted to put this to put
• 27:47
  this out there because we do think that our approach to the
• 27:52
  problem of organizing feature preparation.
• 27:54
  And so I a good one when compared against kind of like,
• 27:58
  what is commonly available Out there so we did.
• 28:02
  That's kind of part of the thinking behind you.
• 28:06
  Why, why would we do that? If there's a lot of benefits,
• 28:08
  right? Also, to open sourcing to
• 28:10
  undertaking, that process of open sourcing, internal Tech,
• 28:15
  like there's a whole effort around cleaning things up.
• 28:19
  Getting it ready for, you know, the public eye, improving your
• 28:24
  documentation a little more cleaning up some facets of
• 28:28
  things. Doing a lot of that work that We
• 28:30
  also would do for internal projects but the sort of like a
• 28:33
  motivating casian to do it. And you know, there's there's
• 28:36
  various benefits right that you get but we did believe, it was
• 28:40
  useful. And we also have a partnership
• 28:42
  with some with a team of folks at Microsoft Azure who've also
• 28:48
  published a blog post about this collaboration where they, you
• 28:52
  know, kind of got a early inside.
• 28:54
  Look at what we were thinking about open sourcing where their
• 28:57
  features store solution, they were interested in.
• 29:00
  In, you know, working with us on this.
• 29:01
  So, we've got some collaboration on this open source project from
• 29:04
  folk said at Azure. Who've done made, really awesome
• 29:08
  contributions in terms of making feather, easy to use and easy to
• 29:11
  try and Deploy on Azure. So there's a lot of stuff about
• 29:14
  that in the blog post that that they published and on our GitHub
• 29:19
  page. Folks can go there and see the
• 29:22
  how it's been really made into like a cloud native Solution.
• 29:26
  That's easy to try on Azure. So all of The, you know, the
• 29:31
  interest from our partners at Azure and interest from folks
• 29:34
  who talk to an industry, these are all kind of factors that led
• 29:36
  us to undertake this initiative of open-sourcing feather that
• 29:41
  had been in use for LinkedIn for quite some time.
• 29:44
  That's awesome. So, you know what, one of the
• 29:47
  things I was, just remembering, you know, a long time ago, I
• 29:51
  stumbled across this website feature store dot org, and I had
• 29:57
  looked through it and noticed, hey, LinkedIn doesn't have
• 29:59
  anything. Thing here.
• 30:00
  And I think David at the time, I was like, hey man, what's up?
• 30:03
  Why why, why isn't your work here?
• 30:05
  And I just went and checked and Feathers there now.
• 30:08
  And it's really interesting seeing, just how many feature
• 30:11
  stores are represented there, and I'm sure you being somebody
• 30:16
  who is steeped in that, in, that, that problem space, you're
• 30:20
  probably aware of how other people's have built their future
• 30:24
  stores. And just curious, what are some
• 30:26
  of the things that you see is that differences between Other
• 30:30
  and maybe some other feature stores, especially differences.
• 30:33
  That might be important. Maybe things that are better or
• 30:36
  worse. Sure.
• 30:39
  There are a lot of solutions out there and something I think I
• 30:42
  mentioned earlier was about how the industry we're all kind of
• 30:46
  together still deciding on like what that good feature story
• 30:51
  even really means as a you know, what is the what is the
• 30:54
  requirement set? What are the functionalities
• 30:56
  needed? It is It's still you know like
• 31:01
  in development, even the names, you know, the name features or
• 31:06
  and what it even really means like there's been some actually
• 31:10
  shift in terms of what that means.
• 31:11
  So if you look at the solutions there I think that there you see
• 31:15
  a variety of approaches, you see some that are much more like a
• 31:24
  data sync. Like I just a data store where
• 31:26
  you put Computed feature data that can then you know, make it
• 31:32
  available to me online context. You see, other solutions that
• 31:36
  provide different amounts of capability for actually defining
• 31:42
  feature views or defining features using Transformations
• 31:46
  on Raw data and then varying levels of sophistication in
• 31:51
  terms of the kinds of features that can be defined whether you
• 31:53
  have support for defining Dynamic features that I have
• 31:58
  over time windows before, you know, up until the event that
• 32:02
  you're making a prediction about support for features of features
• 32:10
  or defining features. As that are computed as a
• 32:13
  function of the values of other features.
• 32:14
  There's various kinds of capabilities, there there's
• 32:23
  There is, yeah, so it's kind of like maybe summarize that there
• 32:28
  are, there are some very simple Solutions and then there's some
• 32:32
  more complex solutions that are may be tied to specific
• 32:35
  Technologies and provide certain capabilities.
• 32:38
  Our approach is that we kind of think that there should be like
• 32:41
  a Swiss army knife as it were about like Making it easy for a
• 32:47
  machine-learning engineer to Define features Based on data
• 32:51
  assets that already exist in an organization.
• 32:55
  So that means an expression language that lets you Define
• 32:58
  featured, you know, like that do that help you with a good amount
• 33:03
  of the last mile of data munging in order to actually prepare a
• 33:06
  feature into something that would be, you know, close enough
• 33:09
  to like a tensor form that you can be easily just injected into
• 33:12
  like a context for running a model or training model.
• 33:16
  The Swiss army knife in the kinds of things that are
• 33:19
  supported in the expression language and then the apis, we
• 33:22
  have a set of those things, based on what we found to be
• 33:24
  useful for LinkedIn, use cases and we think are generally
• 33:27
  useful. And what we want to know, what
• 33:33
  we're building towards is to have That Swiss army knife be
• 33:40
  cover 80% or more of what people need to do and where there's UDF
• 33:45
  support for the remaining 20% and where you have really good
• 33:48
  management capability around those things and where you have
• 33:50
  really good real-time support for those things where you're
• 33:53
  able to properly simulate backfill and compute the values
• 33:57
  of real-time time variant features that are made with the
• 34:02
  Swiss army knife apis, and expression, language has to go
• 34:05
  to do that, really well. And then to be able to deploy
• 34:06
  those and use those things in production.
• 34:09
  Those are aspects that we are pushing on and building for with
• 34:15
  feather and some of the solutions.
• 34:18
  If you look at like the long list of things that are out,
• 34:21
  there are much more like the storage element.
• 34:25
  So in that aren't covering, you know, the Swiss army knife of
• 34:28
  Transformations and feature engineering kind of
• 34:30
  capabilities. So yeah.
• 34:34
  So in a way it's more than a store.
• 34:36
  Like, there's actually that the name Features store and that in,
• 34:40
  for some of these Solutions it does include more than storage
• 34:42
  includes things like feature engineering tool kits and things
• 34:44
  like that. We're it's a little more than
• 34:50
  the store, right? It also includes those kinds of
• 34:51
  capabilities that we see is kind of important, part of solution
• 34:54
  to the problem. I muted myself.
• 34:56
  So I didn't interrupt your your answer.
• 35:01
  Yeah, so definitely concur with, you know, we're still figuring
• 35:04
  things out. There's obviously a lot of like
• 35:07
  when somebody says features store, there's some ambiguity
• 35:09
  about what that means. And personally I've often felt
• 35:12
  frustrated at some feature stores that people are like, why
• 35:14
  don't you use this feature store?
• 35:16
  And when I look at it it's a glorified database.
• 35:17
  You know it just gives Somewhere like store the data.
• 35:19
  It's like well you know I don't know how much value add that's
• 35:22
  having it's so cool to see that. There's like such a wide variety
• 35:26
  of different features stores but you know, somebody who's been in
• 35:29
  the trenches for a long time, I would love to just get a sense
• 35:33
  of, you know, where do you think this is going?
• 35:35
  Where do you think feature stores?
• 35:37
  Are evolving towards. What, what's next?
• 35:39
  For feature stores? Or maybe more generally ml Ops.
• 35:42
  Yeah, I think that what is next? for the things that I pay
• 35:48
  attention to you, There are. There's almost like a hierarchy
• 35:56
  of needs. We started out with having named
• 36:04
  features as any of these people can reason about and can
• 36:08
  register and know what types they have and count on them to
• 36:11
  mean, one thing across contexts training and inference and so on
• 36:17
  Then we added a bunch of Swiss army knife kind of capabilities,
• 36:20
  to Define certain common features based on real data
• 36:23
  sets, but there's like a expanding set of of hierarchy of
• 36:27
  needs kinds of additional things that you want to have to support
• 36:30
  an ecosystem around that. In terms of really easy,
• 36:37
  visualization of the trends of the data Trends behind the
• 36:42
  features or anomaly detection for making, you know, they're
• 36:45
  making it easy for humans. Understand the features that are
• 36:49
  defined and make sure that you know, that they're looking the
• 36:51
  same way over time. Like there's things about nature
• 36:54
  like I understand my current installation better and better.
• 36:58
  Can we continually polish and prove that that's One Direction
• 37:02
  where things are going to improve the timeliness, the
• 37:04
  ability to have time to have a highly Dynamic features that are
• 37:09
  based on some window of time, that is in a rolling window.
• 37:12
  And make sure the support for that is really, really, really
• 37:15
  solid. Is another really huge area.
• 37:19
  Where in my opinion there's sort of a and under use in the
• 37:26
  industry of real-time signals because of the relative
• 37:31
  difficulty of actually dealing with the, you know, like stream
• 37:36
  processing Machinery, which is come a long way but it's still
• 37:40
  not quite at the point, in my point of view, we're from what
• 37:44
  I've seen, where machine learning Engineers are ready to
• 37:46
  pick that stuff up and Apply it, unless they're really know that
• 37:49
  they really need to do it for a specific problem.
• 37:51
  It's not yet at the point where like it's in the Swiss army
• 37:54
  knife, where you can like try it quickly and validate the idea
• 37:57
  that some real-time signals going to add a lot of value.
• 37:59
  So you can just like add it and deploy it, we're not let that
• 38:02
  level yet but getting there and Feathers making a big play in
• 38:06
  that direction. It's going to be like really big
• 38:07
  in terms of helping that a lot of applications in the industry
• 38:12
  move towards using real-time signals more making better use
• 38:15
  of You know, dynamic real-time Computing.
• 38:19
  Things like that. Wait, farther off.
• 38:22
  I understand if I understand this correctly, you're seeing
• 38:27
  that because it's so hard to do real-time and because there
• 38:31
  aren't the proper tools around it.
• 38:34
  People are just like and I'm not going to try for that because it
• 38:38
  seems or it is too complex. Absolutely.
• 38:42
  So I It's yes, I think that people from what I've seen,
• 38:53
  people are ready to try things that they know that they can try
• 38:58
  in a reasonable window of time and get an answer about, you
• 39:01
  know, like that they can try reliably that they know how to
• 39:05
  use it. They can build something, see
• 39:07
  what the effect is going to be within a Time window that is
• 39:10
  within their planning cycle and where, you know, because like
• 39:12
  the longer that takes the more risky it is to actually try out.
• 39:17
  Something I almost think about like, you know, I mean it
• 39:20
  digging for oil or digging for gold or something like that, the
• 39:23
  easier it is to kind of know where to look and to be able to
• 39:26
  try getting it out quickly. And then that it has a really
• 39:30
  big effect in terms of the productivity that exploration.
• 39:33
  If you have a tool that lets you survey the field and really
• 39:36
  quickly versus if it's going to take you a year to survey the
• 39:38
  field that's going to matter a lot.
• 39:41
  So definitely I think that real-time signals Are one
• 39:47
  example of something that, you know, there's been a huge amount
• 39:50
  of interest in this generally roof past many years as you
• 39:53
  have, you have open source Solutions, you have startups and
• 39:56
  other companies that are building support for just
• 39:58
  real-time data processing and stream processing as just like a
• 40:01
  thing by itself. But really the application of
• 40:04
  that stuff for use in machine learning, in my opinion, once we
• 40:07
  have better tools, like we're building with feature stores,
• 40:11
  that'll make it easy to do. That stuff, will have a big
• 40:16
  spike. I think in the interest of
• 40:18
  actually applying that stuff for machine learning problems, in my
• 40:21
  opinion, at least, that's how I see it.
• 40:23
  I'm kind of curious to know what Skyler thinks about that and
• 40:25
  feed Ruiz. Yeah, yeah, yeah.
• 40:30
  First first time questions and flipped on me but so a lot of
• 40:35
  this reminds me of I don't know if you saw some time ago.
• 40:39
  Ship had a post that got a lot of notoriety machine learning is
• 40:44
  going real time. You're not ready and, you know,
• 40:48
  I think I agree with a lot of things in that post.
• 40:51
  One thing that I in this is a bias based on the problems that
• 40:55
  I tend to work on now. I, I look at a lot of problems
• 41:00
  in. There are a lot of problems.
• 41:01
  We're actually getting more real.
• 41:03
  Time signals is not super helpful there.
• 41:07
  There are. Definitely like, I think, in
• 41:08
  most problems, there are probably at least some
• 41:11
  collection of signals that like, yeah, it's really useful to get
• 41:14
  it real time. So, as an example, I work mostly
• 41:16
  in the space of mental health and so, somebody indicates,
• 41:19
  they're suicidal. Yeah, I want to know that ASAP,
• 41:21
  you know, and I want my models to understand that ASAP but like
• 41:25
  somebody's like Mental Health. Like change instantly.
• 41:29
  You know like if I got more data up-to-date right now, probably
• 41:33
  I'm going to make a prediction that's like pretty close to what
• 41:35
  it was like an hour ago and so I think like it really depends on
• 41:39
  the type of problem you're working on and I think the
• 41:42
  reality is a lot of where the money has been in machine
• 41:46
  learning and as far as applications go those are places
• 41:49
  where yes having more real time stuff matters like think
• 41:52
  advertising like you most of the state-of-the-art advertising
• 41:56
  systems now Are like doing like nearline training even because
• 42:01
  like you there's so much more to be gained there so I definitely
• 42:06
  agree. I think it's going to go there
• 42:08
  but I almost think it's like an artifact of like the types of
• 42:11
  problems where money comes from rather than being like
• 42:15
  representative of all machine learning problems in like being
• 42:18
  representative of like what everybody needs so to speak, but
• 42:21
  I definitely could occur with the, with the view that Great to
• 42:27
  make this stuff easier because like the alternative is like
• 42:30
  probably like in most problems there's at least some collection
• 42:33
  of signals of the useful to get real-time and I think the
• 42:36
  reality is like even in places where there are those signals
• 42:40
  people just aren't using them because there's such a big gap
• 42:42
  to like, actually implementing it.
• 42:45
  Yeah, yeah, yeah. Sorry.
• 42:48
  Go ahead to meet you. Well, I was kind of going to
• 42:50
  follow that up with a question to you.
• 42:53
  David around what? Some of the you use cases, you
• 42:57
  feel L would be strong for this real-time that aren't
• 43:00
  necessarily being leveraged right now, right?
• 43:04
  Like and may, is it just more of what Skylar was talking about,
• 43:08
  like that stuff, that big money making stuff?
• 43:11
  Or are there places that you feel like we could leverage real
• 43:16
  time? We're just not, because it's so
• 43:18
  difficult. I guess it would say that we
• 43:23
  have we do have a like at LinkedIn.
• 43:26
  We do have a lot of Use cases actually better doing real-time
• 43:29
  processing. We have a lot of support for
• 43:31
  this, and we have even like, I think there's some other blog
• 43:33
  posts and things like that, that we've released recently about
• 43:36
  real-time computation real time. You know, we feature preparation
• 43:41
  or amount of training, things like this.
• 43:42
  So it is definitely used for some use cases where it is known
• 43:47
  and proven to be important, and we do get a lot of value out of
• 43:51
  doing that at LinkedIn. I think that just to clarify
• 43:56
  what I'm saying before, what I was saying before?
• 44:00
  I'm just in terms of personally, as a focusing on like, Emma laps
• 44:06
  infrastructure. I really like to look for what I
• 44:09
  think are problems where if the infrastructure was a little
• 44:13
  better, it would actually unlock capabilities and investments in
• 44:20
  valuable Investments for users because they're gonna be like,
• 44:22
  okay now that I can quickly try something, I'm going to try that
• 44:25
  thing more and then find out like gosh, this is Useful like
• 44:28
  being able to do that by providing tools that let people
• 44:31
  try and find good solutions. That would have been
• 44:35
  prohibitively complicated to try before, or at least more
• 44:39
  complicated than they were willing to spend, if that makes
• 44:41
  sense. Finding that kind of that that
• 44:45
  edge and trying to move the kind of critical point of investment
• 44:48
  into something and move it to make it easier because the
• 44:52
  tooling is better. So it's easier to try is the
• 44:55
  thing that I'm particularly interested in.
• 44:57
  I agree that there are definitely use cases, where
• 45:01
  real-time signals, for example, matter much more where they
• 45:05
  where they matter less. You know, when you have when
• 45:09
  you're predicting things that are more stable, you're
• 45:11
  obviously, it's not going to do that much to add, you know,
• 45:15
  real-time capability. But there are, you know, like,
• 45:18
  certainly for personalization where you responding to
• 45:20
  real-time user Behavior. You're trying to understand, you
• 45:23
  know, like how to help a user by putting the right search results
• 45:26
  in front of them based on. Something that they just
• 45:29
  indicated recently. You know things like that.
• 45:31
  I do think that there are a lot of places where we would get a
• 45:34
  increasing use with liquor and you know Tools around that
• 45:39
  thing. Hopefully that makes sense.
• 45:41
  Yeah definitely. Yeah.
• 45:43
  So in that that's definitely a future.
• 45:46
  I'm excited for seeing, how many people struggle with doing
• 45:51
  anything real time? I recall there were several
• 45:54
  times at LinkedIn where it was like, hey, we know that this
• 45:56
  would be a great Real-time signal to have and you know I
• 45:59
  think a lot of the real-time features that we ended up
• 46:03
  having, it was a journey to get there, you know?
• 46:06
  It's so I'm very, very excited for that future.
• 46:09
  For sure, we've got to say some Demetrius All right.
• 46:16
  So what are the things I was thinking about?
• 46:17
  You know, again, long time to get to this point.
• 46:21
  It's like having an open source release.
• 46:23
  And I'm just curious if there's anything about, like, through
• 46:26
  the process and maybe not even just feather, but maybe your
• 46:30
  work or generally or more. Broadly, is there anything that
• 46:33
  stands out as something that you're particularly proud of and
• 46:36
  that work? Well, I'm very proud of the work
• 46:43
  that the team has done in terms of being able to actually do the
• 46:47
  heavy lift of mood of getting a project that has been in
• 46:50
  existence for a long time and turning it into an open source
• 46:54
  solution that can be used outside of the context in which
• 46:57
  it was incubated and originally built.
• 46:59
  That's definitely you know not easy to do.
• 47:04
  So there was a lot of you know, it would like I said Heavy lift
• 47:10
  their folks on the team who worked very hard to make that
• 47:15
  happen. We had help from our partners
• 47:19
  that Azure that I mentioned previously who have, you know,
• 47:22
  done a lot of work to do to make this thing useful in the, you
• 47:27
  know, outside of Lincoln context.
• 47:29
  Even though we had built it for years within LinkedIn.
• 47:32
  Anyone who's built a big software project at a at a
• 47:35
  medium sized to fit company over the course of many years and I
• 47:38
  was that like, I can over time, you do get like a lot of you get
• 47:42
  layers of dependencies and relationships with other
• 47:46
  internal Tech and the ability to actually rationalize those
• 47:50
  things and lift and make a release that works outside of
• 47:53
  the context is obviously a lot of work.
• 47:55
  So I'm just super proud that we, we delivered that and that we
• 47:59
  are, you know, we're that the solution is is out there right
• 48:04
  now producing value for for other teams and companies
• 48:08
  outside link. And I just think it's, it's
• 48:10
  really, really cool to see that at this point in the life cycle
• 48:13
  of this project, that that I've been working on for a long time.
• 48:17
  So that's the thing that I'm most proud of I guess.
• 48:19
  Like if that answers switch here?
• 48:21
  Yeah. Well don't worry about it.
• 48:23
  Yeah, I love it. So, if you were to get to go
• 48:26
  back in time and do one thing differently, when you were open
• 48:30
  sourcing feather, what would you do differently?
• 48:34
  Because I'm sure there was a lot that you learned along that
• 48:37
  journey of open sourcing it. Yeah, this is interesting.
• 48:41
  I there's a lot of reasons why we do and don't do things that
• 48:50
  are, you know, based on, you know, the focus in the
• 48:54
  priorities of teams and the needs of internal users and
• 48:58
  various factors right over over time that are kind of hard to
• 49:03
  Kind of hard to fully characterize.
• 49:06
  So it's not like, you know, I could go back and change it.
• 49:11
  It's not like I would go back and take a different turn
• 49:13
  because I think we respond. You know, we went on the the
• 49:16
  timing and the roadmap we did for, you know, those various
• 49:18
  kinds of reasons. But I would say open-sourcing
• 49:20
  sooner would have been nice. Like I would have been, it would
• 49:24
  have been easier to do earlier on and there would have been
• 49:28
  fewer bumps in the road. But then again there a Why we
• 49:33
  didn't do that. We were very focused on
• 49:35
  supporting LinkedIn, internal users and building.
• 49:39
  What we thought was a good solution and supporting
• 49:43
  different kinds of migrations. You're doing these kinds of, you
• 49:45
  know, obvious things. So we didn't get to it sooner.
• 49:49
  But if I could change anything about the timeline, with regards
• 49:51
  to open sourcing or I would have done it done it earlier.
• 49:55
  I think that we it's cool that we've been in this domain for a
• 50:01
  while and That would have been one thing other than that.
• 50:09
  Yeah, I'll probably leave that one at that.
• 50:12
  No, it's a perfect answer and that's a perfect one to end end
• 50:16
  on. I really appreciate you taking
• 50:19
  the time. I learned so much, man, if
• 50:22
  you're such a deep thinker in you obviously like choose your
• 50:27
  words very carefully. I love how much thought you put
• 50:30
  into each sentence. As opposed to me, I just kind of
• 50:33
  like verbally vomit all over the place and ramble on But I'm man,
• 50:41
  there's so many great takeaways on this and Skyler and I are
• 50:43
  going to talk about him right now.
• 50:45
  I can't thank you enough for coming on here David and we will
• 50:50
  hopefully be seeing more of you when it comes to feather.
• 50:53
  I think the last question is and we can leave this at the very,
• 50:56
  very end. So, in case we need to bleep it
• 50:58
  out, we can. When are you going to spin out
• 51:02
  feather into an its own company and do that Playbook?
• 51:08
  Well, yeah, there's no plan to do that.
• 51:10
  I I think that one of the things that I tell people and that is
• 51:16
  is this is really, this is really the truth is that
• 51:21
  LinkedIn is actually a great place to incubate a technology
• 51:24
  like feather, because we have a huge variety of just businesses
• 51:30
  and teams that are doing, machine learning a large-scale,
• 51:34
  small-scale different kinds of problems.
• 51:38
  It's like there's an end the company is big but it's not like
• 51:42
  too big. You can still meet people and
• 51:44
  bump into them and the cafeteria if you're in the actual office
• 51:48
  and learn from users about what matters to them, kind of like
• 51:50
  how I know Skylar because he was really on one of the user teams,
• 51:54
  the internal users within LinkedIn.
• 51:56
  That, you know, it was making use of the feather frame, you
• 52:01
  know? And the other machine learning
• 52:04
  platform tools that we've been working on and trying to build
• 52:06
  and improve over the years, So we being able to have close
• 52:10
  relationships where they've large variety of, apply demo
• 52:15
  users is like a great place to build a really good solution.
• 52:20
  Just in terms of, like, your questions, Beatrice, I had a
• 52:22
  friend who are, have a friend who asked me like years ago,
• 52:27
  like what like five or six years ago when I was, you know,
• 52:30
  thinking of this new project, I had a question from friend like,
• 52:33
  oh, you're building a canoe project around this interesting
• 52:36
  thing. Like, why are you doing this?
• 52:37
  At least. In like why wouldn't you just go
• 52:39
  like try to, you know, talk to some, someone get funding.
• 52:42
  And I was like, well, you know, I actually think that it's like
• 52:45
  I can do a better job of building a better thing in the
• 52:48
  context. I LinkedIn because I'm have, you
• 52:51
  know, I know all these users. I know these problems, I can
• 52:53
  talk with them. I can see their code, we can sit
• 52:55
  together, we can build something there.
• 52:57
  So that's kind of the answer to your question.
• 53:00
  I would turn around and say that like LinkedIn is actually a
• 53:02
  great place to do this stuff. There's no plan to make it a
• 53:05
  another company. If folks want actually And help
• 53:08
  build really great ml Ops tools for LinkedIn.
• 53:12
  I mean like for for the industry, you can come do that
• 53:15
  at LinkedIn and I'd be happy to talk to anyone.
• 53:17
  Who wants to do that. There we go.
• 53:18
  You hiring. Yes, we are hiring.
• 53:22
  There we go. You can you can ping me?
• 53:26
  If you're interested. Awesome!
• 53:28
  Dude! We'll sweet.
• 53:29
  Will end on that. Thank you.
• 53:30
  David, thank you very much. It's my pleasure and hopefully
• 53:34
  we'll talk again.