Latest / Agentic Conversations (formally mlops.community) / RECOMMENDER SYSTEM: Why They Update Models 100 Times a Day // Gleb Abroskin // MLOps Coffee Sessions #123

Transcript

RECOMMENDER SYSTEM: Why They Update Models 100 Times a Day // Gleb Abroskin // MLOps Coffee Sessions #123 · Agentic Conversations (formally mlops.community)

• 0:00
  Okay, never broken. I work for fun core as ml
• 0:06
  Junior. Basically, I'm doing whatever it
• 0:09
  takes to deploy machine learning into production and instead of
• 0:12
  coffee, I'm drinking black, Earl Grey tea most of the time.
• 0:18
  A whole new level to this. That I like half the shit that
• 0:21
  you are were talking about. I was like whoa I never would
• 0:24
  have asked that question in a million years.
• 0:26
  As soon as he said that, I am a software engineer who is knows a
• 0:32
  little know some data science and I'm just like, whatever I
• 0:35
  need to do whatever language to get into production.
• 0:37
  I was like, okay I'm gonna get along with this guy because
• 0:39
  that's that's that's a lot of that's my job.
• 0:41
  That's what I wanted. That's what I did.
• 0:44
  Exactly. So we should probably give some
• 0:46
  background on Who exactly gleb is and also what fun Corp does.
• 0:53
  Because we jumped straight into it right, we went right for the
• 0:56
  jugular as soon as we started talking to him, we started
• 0:59
  talking about the feature store that they have created.
• 1:02
  We started talking about the recommender system.
• 1:03
  They created this whole talk was very much centered around the
• 1:07
  recommender system. So, what exactly is the context
• 1:10
  here? So as I understand it fun Corp
• 1:15
  has won 11 of one very popular app and I think they're actually
• 1:19
  maybe a conglomerate of apps, but the first ones I funny and
• 1:23
  it is a was like, a top-10 App Store.
• 1:26
  It was always a big fan. It was a very popular app.
• 1:29
  I think it has a ton of downloads and just memes so
• 1:34
  memes and then so they need recommendation system on top of
• 1:38
  that, memes are super tricky because they're user-generated,
• 1:42
  they Of really quickly. And you know they're going to
• 1:46
  live and die by the Rex. It's that's dumb and that
• 1:48
  product. Yeah.
• 1:50
  So he talks to us a ton about and even at a very, very deep
• 1:58
  level and this granularity that we don't necessarily get to a
• 2:03
  lot and that is in part. Thanks to you being here.
• 2:07
  Jake because you are, as I mentioned it I don't think we
• 2:10
  recording though. You are in my eyes.
• 2:12
  Sighs, mr. Rexis whenever I talk to you, I
• 2:14
  feel like at some point we start talking about recommender
• 2:18
  systems so it felt only right that you come on here and you
• 2:23
  talk to glad about this anyways. Yeah, I do like recommender
• 2:28
  systems. I like thinking about the entire
• 2:31
  entire Loop of them. They have, there's so many
• 2:34
  interesting challenges, it is so much bigger than just one little
• 2:37
  modeling problem and And that's an amber alert and they are.
• 2:53
  Yeah, sorry. My my do some editing around
• 2:56
  there. Don't worry.
• 2:58
  Yeah. So yeah.
• 3:00
  You love you love recommender systems II.
• 3:02
  Love them, but I like them as like I think as a software
• 3:04
  engineer, I think they're really interesting.
• 3:06
  So I highlighted there in this. I think we talked about client
• 3:11
  logging data Blinds A/B Testing. We didn't even get into too much
• 3:16
  about models serving and the actual business logic that goes
• 3:22
  on and inference, because you typically don't just use your
• 3:25
  role model scores as like to rank your candidates.
• 3:27
  You mix that in with other, with filters and shufflers and all
• 3:33
  kinds of things. Yeah it's incredible to see the
• 3:36
  maturity that these guys have. I love that gleb was so humble
• 3:41
  about it but really like when he broke it down and how he talked
• 3:44
  about a the feature store they created and be just the velocity
• 3:49
  that they have to be able to create a whole new pipeline, a
• 3:53
  whole new model and put it into production after.
• 3:57
  What was it like two weeks and then a month, I think you said
• 4:00
  about a month, what? That was what I was most
• 4:02
  impressed by their system was clearly Now I think that maybe
• 4:05
  partially they've got good experts.
• 4:07
  So if you know how to if you know, if you have people that
• 4:09
  know how to use your system but the fact that you had a
• 4:12
  combination of some ml engineers and a data scientist able to
• 4:16
  build new feature pipelines and it completely new model with the
• 4:20
  new modeling objective in a month and get that into
• 4:23
  production. And again A/B Testing is
• 4:25
  actually phenomenal. This is what dreams are made of.
• 4:30
  Well, and at you and you have to be iterating quickly because
• 4:32
  like we also touched in there and I again they have they have
• 4:35
  multiple blog post about this. You try ideas and they don't
• 4:38
  work. So it's all about the name of
• 4:41
  the game. Here is you know, how long does
• 4:43
  it take you to try to your idea? Yep.
• 4:46
  Yeah. So we are leaving all of the
• 4:49
  links for the blog post in the description, in case anyone
• 4:51
  would like to check them out because it is well worth your
• 4:54
  time to read how they're doing things.
• 4:56
  They're doing an awesome job documenting.
• 4:59
  The evolution of their platform. I think that's something really
• 5:02
  cool to see and it's kind of a random company that you wouldn't
• 5:06
  expect to have such a strong engineering, like,
• 5:10
  outward-facing engine engineering team, but as gleb
• 5:13
  told us, he has a devrel person that is doing a lot of work to
• 5:17
  make sure that they're not only shipping to the product, but
• 5:21
  they are shipping to the blog. And before we jump into the full
• 5:26
  conversation and we go at 1,000 miles an hour.
• 5:29
  As soon as As we get into it, I want to mention Jake.
• 5:32
  So a lot of people are going to be like, who are you?
• 5:35
  And we should intro you real fast.
• 5:37
  You were working at YouTube on the recommender system and now
• 5:41
  you're working at tecton on their feature store and so you
• 5:44
  are uniquely positioned and very much what I would consider an
• 5:49
  expert in this whole field. It's like you are knee-deep in
• 5:53
  recommender systems feature stores, everything that comes
• 5:56
  with it, like the difficulties and the challenges.
• 5:59
  So it's perfect. I'm super happy to have you on
• 6:02
  here, and I want to thank you for joining us now.
• 6:06
  Let's get into it with mister gleb.
• 6:14
  Let's start with this. Can you give us a breakdown on
• 6:17
  what the team looks like? You are more on the offside but
• 6:20
  you do have data scientists. I imagine you have data
• 6:23
  Engineers, you're moving really fast, you're doing lots of cool
• 6:27
  stuff with machine learning, especially around recommender
• 6:29
  systems, but what's the composition there that are free,
• 6:33
  our software Engineers, currently seeks data scientists
• 6:39
  one develops engineer. But I Say he's kind of part-time
• 6:43
  with us because we don't have this many troubles with
• 6:46
  kubernetes Rod right now. And there is also business
• 6:50
  analytic for analyzing A/B tests and all that stuff and that's
• 6:57
  our whole team for the recommendation systems.
• 7:02
  Wow. Okay, that it seems super
• 7:05
  sophisticated. I would not call it
• 7:09
  sophisticated but I would like to Give a shout out to my
• 7:16
  colleagues for doing a great job.
• 7:18
  They are all incredible at what we are doing.
• 7:21
  So even with such a small team, we are able to produce in my
• 7:26
  opinion, great results. And yeah, let's count it.
• 7:31
  Six data scientist 4343 software Engineers that sounds like
• 7:36
  that's a really heavy ratio. It must mean that you because my
• 7:41
  experience they decide to come up with pretty ambitious ideas
• 7:44
  and then it takes like a team of Engineers a quarter to try it
• 7:47
  out. So it might like my previous
• 7:51
  job. I had was like one day decide
• 7:53
  just Flex for engineers. Yes, that is true but we were
• 7:58
  able to build such a system of their data scientist.
• 8:02
  Have almost complete control on the features and they are
• 8:09
  completely unblocked in deploying the models, I guess
• 8:12
  we'll talk about it later. So we are able to maintain such
• 8:17
  a proportion of Engineering in data science and still have a,
• 8:22
  some kind of a plot for platform for them to build things.
• 8:26
  Haha. So, now comes to the juicy part.
• 8:29
  Like, what does that look like? Give us the, how did you create
• 8:34
  that for the data scientist? How did you know what to create
• 8:37
  to make basically? Just to level them up and give
• 8:40
  them what they need to self-serve.
• 8:44
  The first thing I guess is some kind of trust that they are not
• 8:49
  going to break production and it was achieved by having Quite
• 8:56
  heavy level of monitoring. It is not enough by any means,
• 9:00
  but they are quite confident in our data.
• 9:05
  And in databases that, hold this data, we know, for the
• 9:09
  deployment, how to scale them, and how to write applications,
• 9:13
  which use them, there is a airflow deployed and kubernetes
• 9:18
  and communities in isn't basically full control of the
• 9:22
  data science team. They can deploy whatever we want
• 9:24
  here. One in the schedule and by this
• 9:28
  achieving situatio and such a time to Market base Acres.
• 9:37
  So that is going to jump ahead. I was kind of hoping that asking
• 9:40
  this question later. But so you mentioned and one of
• 9:43
  your blog post about your all's feature platform, your future
• 9:45
  store, you have these declared it or you have, you know, these
• 9:50
  kuberan. These specs for offline and
• 9:52
  online, spark pipelines, it all sounded very powerful.
• 9:55
  So you're able to do joins across Kafka to your serum, a
• 10:01
  mongodb database and I thought that's not a really powerful but
• 10:07
  So maybe very technical like that.
• 10:09
  This would be something worked on by like an owl Engineers.
• 10:12
  Not data. Scientists are you are your data
• 10:14
  scientists? Actually build it whenever they
• 10:16
  want a new feature group. Are they going in building those
• 10:19
  pipelines? Sadly I have to say no they are
• 10:22
  not the way you labeled features is by writing complicated
• 10:27
  queries to The Click house and putting this data into mongodb.
• 10:31
  This is the way for them to do it but our Engineers use Spark.
• 10:36
  Mostly and there's kind of not a framework but a bunch of
• 10:39
  interfaces which allow us to write the code faster and that's
• 10:45
  our vision of the future storm, I will get into the details
• 10:49
  later, I guess, but the idea was that we wanted to give data
• 10:55
  scientist, complete freedom, and what we do, and just let them
• 10:58
  rip it out and do whatever it takes to deploy the model
• 11:01
  faster. So we have decided against
• 11:04
  single platform for all the features.
• 11:07
  And plenty of this approach. Okay, so, you're well, maybe we
• 11:12
  can get in on it. Maybe we can get into the
• 11:14
  features or down to be true so let's, I don't know.
• 11:17
  Yeah. So go straight to the point.
• 11:19
  We get right down to business, don't ya?
• 11:22
  So you all have built. So the one thing I was reading
• 11:26
  is that you are moving from Matrix factorization, just
• 11:29
  purely Matrix, factorization or access to more diverse feature
• 11:33
  sets. So, previously, you only need to
• 11:36
  basically One feature like the users history and now anything
• 11:41
  that a data scientist can think of you may need to build a
• 11:43
  pipeline for so it's getting a lot more.
• 11:45
  Diverse is getting more complicated unique, kind of
• 11:48
  difficult, aggregations custom Stuff Etc.
• 11:51
  So is is was that the impetus like moving some from something
• 11:56
  more sophisticated The Matrix factorization to know neural
• 12:01
  Nets or I think in your case of boosting model.
• 12:05
  With that the impetus for your feature store is that You can
• 12:07
  what is that was at the point at which people need to?
• 12:10
  You know, if I'm building a wreck system do I need a feature
• 12:12
  store? If I'm just a matrix
• 12:14
  factorization or do I need to do it whenever I'm trying to build
• 12:16
  a boosting model and something more sophisticated.
• 12:18
  I think it depends very much on the environment you're working
• 12:22
  in and first of all, we should Define.
• 12:24
  What is the feature store? And what are the components of
• 12:29
  it? Because intimately understand it
• 12:31
  one way. We explore the search for words
• 12:35
  and decide. It against using it, just for
• 12:38
  our use case, for doing like online stuff, with low latency,
• 12:43
  and being able to have our own data sources and controlling the
• 12:48
  queries to them. So I would say it depends on the
• 12:52
  environment for us. The approach we took to the
• 12:55
  problem was we are trying to build a lot of small chunks of
• 13:00
  code which provide single feature because as you say,
• 13:06
  Science team always want something to you something
• 13:10
  stored in the database differently.
• 13:12
  Something new input into some new inputs into the model and we
• 13:18
  were not able to govern a requirements in a such way that
• 13:22
  we will provide sophisticated and stable system on the first
• 13:27
  try. So we decided to go from the
• 13:29
  bottom up. Basically.
• 13:32
  So you say, you're a structure is fairly thin and what you, you
• 13:36
  built it in a way that's like fairly modular.
• 13:38
  So whenever you see someone who's a new feature, it's mostly
• 13:41
  new custom work. Yes.
• 13:42
  On top of a fairly thin system. Yes, right now it's basically
• 13:47
  copy paste codes of another feature and the find is that at
• 13:52
  the moment we will just abstract those feature with an API later
• 13:56
  than the know, all the requirements we need for
• 13:59
  recommendation system, and the only goal would be to it.
• 14:02
  Read as best as possible around those predefined set of
• 14:06
  features. How many distinct I'm using
• 14:10
  models. Are you using?
• 14:12
  But the same features? So, is there?
• 14:13
  You're getting like feature reuse, or is it?
• 14:16
  Well, there's one rexis model that you're trying to power.
• 14:20
  We had definitely reusing futuristic rolls, I would say
• 14:24
  two to three models at some moment but the way we do it is
• 14:29
  for a be tasting. So basically one date The
• 14:32
  scientists at the moment controls 12 to A/B test and they
• 14:37
  run them out. And after that we decided to
• 14:40
  dinner or it might be a control group if it has failed.
• 14:45
  So we don't have this many model on the runtime never basically.
• 14:53
  So, I was interested in breaking down.
• 14:56
  What you felt like were differences in how you define a
• 14:59
  feature store, and what Jake was thinking of, as a feature story,
• 15:03
  because I think the, the complexity comes into, like,
• 15:08
  wait, what are we saying here? As, as far as a feature story
• 15:11
  goes, like, when you think about that, what exactly is the
• 15:15
  feature store? Because you write it nicely in
• 15:18
  the blog, but I think it will be good for people that are
• 15:21
  listening to understand that. In my eyes, the common
• 15:24
  definition of a future store would be obliged.
• 15:29
  A bunch of job declarations for offline features and a set of
• 15:35
  tools or an hour jobs to move them to the online store by
• 15:40
  offline. I mean, the data for it's
• 15:42
  raining by online. The data for the inference,
• 15:46
  there should be a common layer which allows a person to get set
• 15:51
  of features. Or feature groups in.
• 15:56
  The store offered choosing. But it includes, first of all,
• 16:02
  the common API for getting, or setting features in one of the
• 16:07
  stores and submit information about the features, and we
• 16:13
  didn't like the idea of not documenting, but declaring the
• 16:20
  features, because it is kind of hard to get right from First
• 16:23
  Take because it took me four times, I guess to write a
• 16:29
  Attempts to write a good interface around the spark jobs,
• 16:34
  but I'm getting ahead of myself. By the definition.
• 16:41
  Okay. Okay.
• 16:43
  I think that that's actually made me not too different than
• 16:45
  I, you know, that, I think how maybe more broadly I think about
• 16:49
  a future store or a feature platform.
• 16:51
  So you mentioned you have declarative Transformations,
• 16:55
  you're managing your managing scheduling do not have this
• 17:00
  declarative Transformations and we do not have a single API for
• 17:03
  accessing both of the stores. We did it differently because we
• 17:07
  didn't want to get to manage the complexity of those Solutions.
• 17:14
  Okay, so instead of instead of that kind of unified feature
• 17:18
  store approach, you all have effectively like then people can
• 17:24
  build your own pipeline, build your own scheduling and your own
• 17:28
  store on to this. There's not as unified API.
• 17:31
  Yes that would be correct. There's a set of tools to get
• 17:36
  information from another service like an airflow operator to get
• 17:40
  the current A/B test status for example.
• 17:44
  Or interfaces and their implementation of Spike, jobs to
• 17:49
  write one code for the streaming and batch processing.
• 17:54
  But there is no platform over way.
• 17:57
  We force people to write features.
• 18:01
  Okay, and do you think so? How's that so how many sounds
• 18:05
  like you already have you all are already iterating on models?
• 18:08
  You have a very sophisticated rexis implementation with this
• 18:12
  kind of say, like, fairly ad hoc like feature set a feature
• 18:16
  pipelines and systems. Do you think that that'll
• 18:20
  continue to scale? Well for you all or do you think
• 18:22
  once you have like a second or third use case may be a more
• 18:27
  opinionated feature store is going to be the future and you
• 18:29
  all are just waiting. Build that until it's necessary.
• 18:34
  I think it will be necessary and we are moving in this direction
• 18:39
  but our approach would be to take the model with a set of
• 18:43
  features it needs for the interns and make sure they are
• 18:46
  available available in the online store after the infinite
• 18:52
  service receives them. It will it be able genomically
• 18:55
  to load them from the database and use those without writing
• 19:00
  any custom code? But to get there we will.
• 19:04
  To abstract our current solution so that would be necessary.
• 19:10
  But later, Okay. So this so it sounds like you
• 19:15
  all didn't need to jump to a full-blown future store and
• 19:18
  feature platform for your first use case and said you're going
• 19:21
  to level up your way there as needed to like because a lot of
• 19:25
  the benefits of a really centralized platform.
• 19:28
  Don't start coming into play until you have a very large,
• 19:30
  large team, and multiple use cases just.
• 19:32
  But what's exactly our thinking and the full-fledged for just or
• 19:37
  like tecton might be useful for like a hundred of data
• 19:40
  scientists but for our team of five, Would be kind of Overkill
• 19:44
  to learn all these abstractions and layers of sophistication.
• 19:51
  So I'm wondering where you're going to when you think about
• 19:54
  the feature store that you have and you think about the future,
• 19:58
  you kind of hinted at what the roadmap looks like.
• 20:02
  But what are some things that you're specifically focusing on
• 20:07
  to help bring more maturity into the way that that interacts with
• 20:11
  the data scientists? I think the first one would be
• 20:14
  mandatory. There's never enough there's
• 20:17
  board singer fauna and we are still having Major incident note
• 20:23
  outages, but incidents in our data platform or on to model
• 20:28
  side of things that will be vomiting.
• 20:32
  And the second one is just providing all types of tooling
• 20:37
  like air flow, operators, this spark framework or the IC IC.
• 20:42
  D pipelines for deploying code tester.
• 20:46
  So, yeah, I would focus on the third.
• 20:50
  Demetrius we heard a I think we but listened in on a really good
• 20:53
  talk with chip about monitoring and the space.
• 20:57
  Gleb is so monitoring. You could do it in a lot of
• 21:01
  different points in this system, you can do it whenever you
• 21:05
  produce the data to feature outputs or the, the model
• 21:08
  outputs, the final business objectives, and, and
• 21:10
  click-through rates, things like that.
• 21:12
  So we're all you're finding the most value and stability versus
• 21:16
  like, you know, what would be advice if someone's gettin
• 21:17
  anorexic system where would You advise them to invest their
• 21:22
  investor times and monitoring. I think the most important part
• 21:26
  would be the business metrics like click-through rates, action
• 21:30
  rates, and all that other stuff, which allows to receive direct
• 21:35
  feedback from users and all the other technical mentoring comes
• 21:39
  next to it because business will make decision based on business
• 21:43
  metrics. And first of all, we should make
• 21:45
  sure they work. And after that, we will add more
• 21:50
  and more dashboards for the technical part to understand
• 21:53
  Bible probable scheme from What kind of curious?
• 21:57
  Because we had problems, we had problems at this age, YouTube.
• 22:00
  What's your kind of latency for business metrics?
• 22:04
  You know, you're really good business.
• 22:05
  Metrics are typically very lagging.
• 22:07
  So, which ones are you using? And what's your latency for
• 22:10
  alerts to go off? There are different types of
• 22:14
  metrics for most important. One are on the daily basis, but
• 22:18
  the most important one is retention and the most important
• 22:21
  detention is like 30 of day or 30 days or 50 days and You won't
• 22:26
  be able to get it until you wait 30 days but there are daily and
• 22:33
  weekly metrics that allow us to check the experiment.
• 22:36
  Quite in normal ways we need, I would say And there are I see
• 22:44
  your mom sorry hourly metrics are there to just less of a
• 22:50
  okay. So you're monitoring is mostly
• 22:51
  to make sure they be test isn't going off the rails and really
• 22:55
  tanking business metrics for some fraction of your users.
• 22:57
  Is that is that a fair way to say yes but would be very
• 23:02
  Demetrius. I have some questions.
• 23:04
  So I so in my experience business metrics and choosing
• 23:08
  good metrics is like maybe the hardest problem in Texas.
• 23:12
  So I have some questions. On that but Demetrius.
• 23:14
  Yeah, I don't think yet. Did you have anything that you
• 23:15
  wanted to get to first? I love that because there are so
• 23:19
  many times where and this is probably something that comes up
• 23:22
  so much. Right?
• 23:23
  Is the metrics that you're choosing?
• 23:26
  I, we just actually interviewed the doordash team a few weeks
• 23:30
  ago, and they were talking about how one metric that they use is
• 23:36
  this velocity to production, like basically the velocity of
• 23:41
  the machine learning team, And how fast can they productionize
• 23:44
  a feature? How fast can they productionize
• 23:46
  a model? That's one metric that they're
• 23:48
  looking at. But I'm a huge fan of thinking
• 23:52
  through every metric and why certain metrics are chosen
• 23:56
  because it shows you where you're putting your emphasis, it
• 23:59
  shows you where you're putting your focus and why?
• 24:04
  So so glad specifically I'd makes it.
• 24:07
  You are mentioned I think you actually highlighted this in
• 24:09
  your blog post about you of you originally had like we're trying
• 24:12
  to optimize likes and you know your business objective is
• 24:16
  retention and it turns out that likes are not a good metric to
• 24:20
  optimize you know to for retention.
• 24:23
  So can you talked about maybe two things?
• 24:26
  How you think about how you all are thinking about metrics?
• 24:31
  Like picking a good light Target metric to Your business
• 24:33
  objective and and then what maybe what makes a good Target
• 24:37
  metric in our team. It's more of a data science job
• 24:43
  to select a metric. And more of the business side of
• 24:47
  things, we are dreaming about training models on the
• 24:51
  retention. Like with Spotify, did it, I
• 24:53
  guess to be able to move people from court of non active users,
• 25:00
  to the active ones or two. Revive them as a users.
• 25:05
  But it is very hard to build such a model and you will need
• 25:11
  Lots of not monitoring data but a lots of different kinds of
• 25:15
  stuff to build it around. So currently we are using smile
• 25:21
  or a mostly. There are different targets
• 25:23
  right now in DB testing, but I wouldn't be able to name them So
• 25:30
  as also extra my experience and maybe I've ever YouTube's
• 25:33
  mileage or veggies mileage, will vary that much from where you
• 25:36
  are going to wind up. We had the hardest time with,
• 25:41
  you know, our lot with what we want, long term metrics, we want
• 25:43
  like user growth. We want retention one ad
• 25:45
  Revenue, all those all those things, but you can't experiment
• 25:49
  on those Cycles because, you know, to measure user retention
• 25:53
  takes 30 days, right? So if you roll out at A/B test,
• 25:56
  you can't wait 30 days for metrics, also.
• 25:59
  Indeed the variance and like the Bonnet measuring, those metrics.
• 26:02
  So they're very non-sensitive, the very lagging metrics.
• 26:05
  So they're terrible for like for using for a/b testing and using
• 26:10
  to draw and to train models. Because again, you have to wait
• 26:12
  30 days for your labels if you're trying to model 30-day
• 26:15
  retention. So we we started having models
• 26:21
  that would predict retention and then that would feed into other
• 26:24
  models and it becomes this like nasty whole problem.
• 26:26
  But like so by picking something, that's easy, like you
• 26:28
  want to find the bright balance between, like, metrics that are
• 26:31
  sensitive, that are fast to get results back from, enter
• 26:35
  actually practical to model and that was like, kind of a
• 26:39
  never-ending struggle for us. One might be the time spent in
• 26:44
  the publication because it correlates with it might
• 26:47
  correlate depending on your application if retention or the
• 26:53
  revenue per user. But in our case, I do not
• 26:57
  remember a single model. Deployment of this kind of
• 27:00
  approach. We tried a lot of them and they
• 27:04
  didn't work out. Yeah, that was we also ran into
• 27:09
  Rhiannon, similar things. I think we got.
• 27:10
  I think some of our, I think some of our winners ended up
• 27:12
  being if you engage with this item, how much, how much more
• 27:19
  will you engage with the rest of the app after that, for today or
• 27:22
  something like that, which in it, but it isn't conditionals.
• 27:27
  And then, just to just to build that the Data engineering
• 27:30
  pipelines which is, you know so that's the kind of.
• 27:34
  So the kind of the the next place I was hoping to take this
• 27:38
  as talk to you about like y'all mention like measuring implicit
• 27:40
  signals and having like really good ways to measure everything
• 27:46
  that users doing in the app and structuring that in a way that
• 27:48
  you can build interesting features and interesting labels.
• 27:53
  So can you can you talk about that?
• 27:55
  How you know, I think you'll mention that Android apparently
• 27:58
  didn't have Your Android system didn't have any logging for a
• 28:02
  long time. You're in your using only iOS
• 28:04
  for data collection, and how how challenging the problem is to
• 28:10
  actually get like, deep interesting metrics from a, from
• 28:13
  a consumer app and that you can rely on for your business,
• 28:17
  you're like, back in business, stability, Well, why should
• 28:21
  compliment you on the reading through lots of blog post on our
• 28:25
  medium, I guess. And yes, it was a problem with
• 28:30
  Android Matrix. Our pipeline looks like one very
• 28:34
  reliable Java service, which collects events from users, all
• 28:41
  types of events from likes to comment, section, opened and
• 28:45
  different kind of stuff than the go through Kafka.
• 28:50
  And in parallel to the click house on the hourly basis.
• 28:55
  So in Kafka you might receive some online events and update
• 29:00
  them right now. In Click house, you may take
• 29:02
  them for the training later. There were a lot of problems
• 29:09
  with those services and we have found some bugs in them while
• 29:15
  observing AP tests, but mostly those are the Reliable.
• 29:21
  And Yes, for data from them is kind of stable but there are
• 29:28
  some issues as always. Hey there.
• 29:31
  I'm a Satanist and ever and I'm head of product a teachable hub
• 29:34
  and if you are into envelopes and want to hear more than just
• 29:37
  buzzwords, subscribe to the envelopes Community podcast and
• 29:40
  listen to the best practitioners in the field, sharing their
• 29:43
  wisdom. I like how glad you weren't
• 29:49
  ready, where you, you didn't realize Jake was going to read
• 29:51
  through all of the blog post that you all got to get quizzed
• 29:57
  on everything that you've ever said or ever written about good
• 30:01
  thing. It's not like you have like 10
• 30:02
  years of backlog Jake's like, hey in 1998, you guys mentioned
• 30:08
  you were using some form of kobol and I wanted to dig into
• 30:12
  how you could Loop that into your recommender system.
• 30:15
  What when did me When Demetria is asking you to come co-host
• 30:19
  and interrogate. Someone I was like, I'm gonna do
• 30:21
  my homework, I'm going to be ready, that's it.
• 30:24
  I love it. I was going to ask well, Jake, I
• 30:29
  mean, if you want to keep digging into this, I was just
• 30:32
  going to ask about how basically the, the bigger picture on how
• 30:37
  the feature store and the recommender system play together
• 30:42
  and then what other pieces of the recommender system you have?
• 30:46
  Because I I know you're deploying things and you put in
• 30:49
  these blogs some awesome stuff. It's kind of going up a little
• 30:52
  bit of a level from where we were at with this granularity.
• 30:56
  So Jake if you had one more that you wanted to get in this like
• 30:59
  super granular topic, we can do that.
• 31:01
  Then we can go baby gear off a less clever fakes or something
• 31:04
  very interesting here. So but I'll tell for people that
• 31:07
  are interested in our excess client logging and the data
• 31:10
  pipeline to get those logs into like reliable features, reliable
• 31:15
  a be metric. In training data and labels.
• 31:18
  And everything is a very, very complicated problem.
• 31:22
  But it went everybody. That's what it was, very, just
• 31:25
  leave us with that. I'll give my two cents and maybe
• 31:29
  glad has some thoughts to Y. So why is it a problem?
• 31:33
  Typically and especially like larger BC apps like your your
• 31:38
  consumer app is your front end at the front-end team.
• 31:41
  The pretty decouple. They don't want to think about,
• 31:43
  they want a reskin. Their app logging is This kind
• 31:47
  of Black Box. They shoot the logs.
• 31:48
  They're not positive. Always what it does and you know
• 31:50
  it's a feature into a model. No one's really sure what all of
• 31:53
  them do and so you know just just on just the Highlight.
• 32:00
  One example thing is is just the the fragility of having to like
• 32:06
  take a nap if a front-end team, make some change to how some
• 32:10
  like logging Behavior works. That was critical for computing,
• 32:13
  some feature in the back end and they're wondering Wild Are there
• 32:17
  running a new? They're running an experiment
• 32:19
  and all the sudden your site metrics are tanking, because
• 32:23
  they're front-end experiment layer that they're using, is
• 32:27
  different than your back and experimental what layer?
• 32:29
  So you can't tell. But for whatever reason, all of
• 32:31
  your models of degrading, This, like that's just one example of
• 32:36
  why this is such a hard problem, glove, I don't know.
• 32:37
  You were laughing a little bit. I don't know if you all have
• 32:39
  experienced that I have not, but I can imagine the Joy on the
• 32:47
  data science team faces when they learn about this question.
• 32:51
  Yeah, I can relate to that. It's what I do is just finish
• 32:56
  the edit Out, YouTube with, we would delay entire front end
• 33:00
  launches. So, like entire, like,
• 33:02
  improvements to the front, The app for months because we had to
• 33:06
  figure out why their front and change was degrading.
• 33:09
  Our models. It is.
• 33:11
  So it's just like, these could be like, terribly, terribly
• 33:13
  difficult, probably God. Yes.
• 33:15
  And well, let's stay on this glump.
• 33:18
  What are some things that you've seen and have caused equal or
• 33:23
  more headaches? Missing looks from Android,
• 33:28
  would be one? The second one would be missing
• 33:31
  two logs from a part of our applications.
• 33:33
  So we have two types of feeds. One for multi-armed Bandit.
• 33:37
  Swear people do exploration of the content and the looks from
• 33:41
  it. We are not getting into the
• 33:43
  Kafka, so we are not getting it to recommendation system and it
• 33:48
  was degrading slowly and the first problem would be.
• 33:52
  Sometimes, our clients might go offline and not send logs for a
• 33:59
  certain period of time. We will come to Kafka they've
• 34:02
  time steps. They would be no sense.
• 34:06
  We friends have happened yesterday or two days ago and
• 34:09
  there are some features Inspire core our tools to handle that,
• 34:15
  but it might be challenging if you don't know about it.
• 34:21
  This is the non glamour side of ml Ops or Rex's systems that
• 34:26
  don't take some valve but actually ends up making a
• 34:28
  system. You kind of it can make or break
• 34:30
  good features. Good labels and good modeling.
• 34:32
  Yes, it's absolutely true. And it but Demetrius we can pull
• 34:38
  it out back out of the weeds because I don't know if ml Ops
• 34:41
  folks if client logging is is there Jam.
• 34:44
  So like maybe you could steer us your gloves experience and you
• 34:47
  know the model deployment or what are you what else are you
• 34:50
  thinking about? Well yeah I want to go back into
• 34:54
  the weeds but I wanted to just zoom out for a second and just
• 34:57
  see because I think it's probably worth talking about the
• 35:00
  whole system as what it looks like.
• 35:03
  We really jumped into The feature store side.
• 35:06
  But what else is there? You've talked about the
• 35:09
  pipeline's a little bit. I'm imagining that you're
• 35:13
  serving things. You're doing those offline
• 35:15
  online type recommendations that you were talking about and then
• 35:19
  you have that the lagging data that comes through with that
• 35:22
  ground truth. I imagine you kind of broke down
• 35:25
  the monitoring to of that and where you're putting the
• 35:28
  monitoring, what are we missing to get the full picture?
• 35:32
  There's not much to be honest. Infinite service which source
• 35:38
  models contains the pipeline's of them, which depend on the
• 35:44
  test. It is written in code dealing
• 35:46
  with spring boot the, our stable choice for services, it stores
• 35:54
  or uses for model. In the Onyx format, they are
• 35:59
  tested by data science, team to be to produce the exact same
• 36:04
  results as a python. Regional models as a regional
• 36:07
  models written in Python. What else is there?
• 36:11
  There is a lot of caching. Neither of influence service,
• 36:17
  various memcache. And inside of it there are
• 36:19
  multiple layers of cashing in the Java hashmaps or something
• 36:23
  like this. What else is missing?
• 36:26
  We're a couple more services in the direction for user, with
• 36:30
  their caches, and HTTP and points and all that.
• 36:35
  And what else should I mention? I do not do it instead so I had
• 36:41
  science some questions around. So one thing that people Baby.
• 36:45
  Don't realize, especially if you come from a software engineering
• 36:47
  background that in these like email intensive applications.
• 36:51
  A lot of times, you try ideas and they don't work.
• 36:54
  So, I think it software engineering, we have quarterly
• 36:55
  planning, and we're going to do X and Y and Z and that the end
• 36:59
  were going to ship those and data science.
• 37:02
  And, and like, a wreck system, you have three ideas.
• 37:04
  It may be in your perfect team and none of them work at the end
• 37:07
  of the quarter. So my questions around, like,
• 37:11
  what have you built to help? Hi ideas quickly, because I
• 37:16
  think that at these kinds of Rex's systems.
• 37:19
  What makes it good Rex? It's that some is being able to
• 37:21
  throw ideas at the wall to see what sticks see you at works.
• 37:25
  So maybe curious if you have like recent examples, like
• 37:29
  what's your turnaround time? From an interesting idea to try
• 37:33
  and get out to making a go/no-go decision and like what parts of
• 37:37
  your stack or allowing you to move at that speed?
• 37:40
  That's a good question. lately, we were trying to Deploy models
• 37:49
  based on positive actions of users, like likes, comments,
• 37:54
  shares and all that stuff. They feature, engineering took
• 37:58
  probably every week with figuring out, all the Gory
• 38:03
  details of scaling Monica, to be able to handle the initial in
• 38:08
  part of a data. The model to look I would wear
• 38:13
  these all were these are new with these are new features or
• 38:15
  you re using features. You already built for another
• 38:18
  model, those who all new with new database structure, then you
• 38:24
  interface on their civic sides of the service.
• 38:29
  All this development probably took us two weeks from the
• 38:34
  initial tasks creation to the deployment into production.
• 38:39
  The model itself was developed, I would say in around the month
• 38:44
  but the data scientists who did it was working in a complete
• 38:50
  isolation. Say so, he didn't interact with
• 38:52
  soft Energy new team. He was just working on his
• 38:55
  development machine, then move the to airflow and kubernetes.
• 39:01
  It took him about a week I guess.
• 39:04
  So that would be our average not average.
• 39:07
  Probably best case. For deploying a new model, but
• 39:11
  there are always issues. And a lot of times we rerun a be
• 39:17
  tasting after finding some internal bark or the missing
• 39:22
  date or something like this. So as this data scientist, just
• 39:27
  hacking around in Jupiter. And then he came or she came and
• 39:32
  said here's what I got now. Figure it out.
• 39:37
  No one thing. It would be able to decrease
• 39:41
  time-to-market. Was this layer of abstraction
• 39:45
  between data science and the infinite service?
• 39:48
  Which Which is joining us for all the models.
• 39:53
  So we do not have to think about environment at all whereas a
• 39:57
  single operation version I guess in the Onyx and you should just
• 40:01
  make sure that they match in the on the converter site and on the
• 40:07
  user side and it allows us to give complete freedom for the
• 40:13
  data science to Just Produce this single artifact and then
• 40:16
  put it into the service, the HTTP request.
• 40:21
  So this is one thing we built to decrease time to Market.
• 40:28
  So so there's kind of want to dig in a couple things there.
• 40:31
  So number one is like it's kind of remarkable that your data
• 40:35
  scientists was able to come in any.
• 40:37
  I think anyone that has worked on like when these systems would
• 40:40
  say, if you can put a data scientist alone, in a room for a
• 40:42
  week and they're able to produce something in production, that's,
• 40:45
  that's really good. That's, in fact, I think most
• 40:47
  data scientist would be jumping off.
• 40:49
  David, love that because normally they work in a week and
• 40:51
  then they handed off to a team. It's a quarter later or
• 40:53
  something. So, what are the?
• 40:56
  So the Mets. Sounds like the main thing that
• 40:59
  you all are allowing to do is you have a your inference
• 41:03
  service, push API, Onyx. Can you talk a little more about
• 41:07
  that? Like are they setting up like
• 41:09
  continuous training pipelines or is it?
• 41:11
  They produce one artifact, pushing your service and now
• 41:13
  we're A/B Testing, We're Off to the Races, no one there to fact
• 41:16
  is Never Enough in our use case because Fields of memes, or the
• 41:22
  domain of memes is changing very fast.
• 41:26
  And we have to update models on the hourly pace.
• 41:30
  So every hour we deploy around 20.
• 41:33
  I would say models, which were trained on the data from
• 41:37
  previous hour or two hours ago, I should say.
• 41:42
  And it means that for every model we require continuous
• 41:48
  pipeline, which is written in the air, A cyclic graph.
• 41:54
  So the final result of a work for every data science research
• 41:59
  would be this graph. If some kind of documentation
• 42:03
  and the models on the S3 and features in the manga database
• 42:08
  after that, we will be able to deploy the model.
• 42:11
  But there is also some level of delay which is caused by needing
• 42:17
  to integrate the model into the infinite service.
• 42:22
  And so that was the best-case scenario, right?
• 42:28
  What you told us like this two week and then give or take a
• 42:32
  month with the data scientist and you mention how things go
• 42:36
  wrong? What's the worst-case scenario?
• 42:38
  Oh worst case you know in the previous architecture or the
• 42:43
  previous way we did stuff, I was trying to add a new feature for
• 42:49
  a month I guess. And all this time I was trying
• 42:52
  to figure Recites and optimizations, which were really
• 42:55
  low level and required a lot of verbal no test for them.
• 43:01
  And it was a Pain everywhere to get this feature into
• 43:07
  production, after that, we decided to.
• 43:10
  We decided that that's enough with this approach.
• 43:14
  We are building a new one, and this is works fine.
• 43:18
  And in most cases, we give we have this best-case scenario
• 43:23
  results. Yes, I would say that.
• 43:29
  I think hourly to have your system, sounds actually fairly
• 43:35
  mature. So for the record, good YouTube.
• 43:38
  We push models new, every 24 hours.
• 43:40
  You all are every every hour you're pushing new models memes,
• 43:44
  move fast in, your teams are means get deep fried every hour,
• 43:50
  and it just go deeper is, so you all are so.
• 43:55
  So just to kind of recap. And you're saying in a month,
• 43:58
  You're able to build a new model that depends on new features and
• 44:04
  have a data scientist kind of set up and take the last Light.
• 44:08
  Mile of that build a continuous training pipeline that pushes a
• 44:13
  new model into production every hour.
• 44:16
  You just the that took, you all start to finish about a month.
• 44:20
  Is that is that about right? It's just correct and it is
• 44:23
  realistic estimation of new model deployment.
• 44:28
  I think I'd be the Envy of a lot of Rex's teams out there.
• 44:33
  Yeah yeah it's really cool to see what you have done and how
• 44:38
  you've been able to basically take something that is so
• 44:41
  complicated and make it available for the team to
• 44:47
  iterate on so quickly this goes back to what I was mentioning
• 44:49
  like with doordash and how they measure the velocity the
• 44:53
  velocity. It sounds like you've created is
• 44:57
  very fast. It's Most light speed in the
• 45:00
  world that we're talking about, right?
• 45:02
  Like this is not the norm. I feel like when I talk to
• 45:06
  people about this, especially when it comes to Rex's because
• 45:09
  there are so many snags, but maybe it is a good sign that,
• 45:16
  you know, recommender systems are becoming so common.
• 45:22
  This potentially is the future that we're looking at.
• 45:26
  Yes. And thank You for the kind
• 45:29
  words, I would actually sometimes would like to move
• 45:33
  slower and have better monitoring and testing for
• 45:39
  stuff. So we won't be won't have to
• 45:43
  figure it out in production. Why something went wrong?
• 45:46
  And we'll know it prior to pushing the coat.
• 45:50
  But being able to run this many tests, Delights, all of us, I
• 45:57
  guess. In some way.
• 46:01
  So I have one final area that I'd like to that.
• 46:03
  I like to highlight about that I had in my notes because I don't
• 46:06
  think you can't talk about a wreck system without talking
• 46:08
  about A/B Testing. Rex is systems live and die by
• 46:12
  A/B Testing. So in, you'll mention this
• 46:16
  again, read into your blog post. I think this comes up a good
• 46:18
  bit. So are there, any are there?
• 46:21
  Any parts of, you know, do you have a system in place that
• 46:26
  helps you all make launch? No, launch decisions.
• 46:29
  Helps you slice in the user categories.
• 46:32
  Helps you get sensitive metrics things like that, that you all
• 46:35
  are fan of and like, what's going?
• 46:36
  Well, there maybe you can talk a little bit about what your
• 46:38
  system looks like. It's a single note to single,
• 46:44
  but multiple click house instances, with all the events,
• 46:47
  across all users in all the countries and a lot of free
• 46:51
  dashboards with many, many, many slices around them, we track
• 46:58
  Kinds of content. We are showing by categorizing
• 47:02
  it in different groups by we are tracking the age of a user on a
• 47:09
  platform so how well the system performs on the new users on our
• 47:14
  oh, geez out there. And there's so much going on in
• 47:20
  the testing Department, all the decisions are based on those
• 47:24
  three dash dash boards and but they are not Tomatoes.
• 47:30
  Instead we look at them, many early and try to find some
• 47:36
  correlations or the causes of certain behavior of a model.
• 47:45
  So maybe you can marry really quickly go into.
• 47:47
  So what is your because this is all this is all I thought was
• 47:50
  like one of those interesting rooms to be and what?
• 47:52
  So what are your like your note you launch?
• 47:54
  No launch committees. Like what are the what are the
• 47:57
  weird signals that your for your business specifically that you
• 48:00
  all are looking into one of your analyzing if it A/B test went?
• 48:04
  Well, you know what are? What are your biggest leading?
• 48:08
  Indicators your user slide says, Etc.
• 48:11
  That are there are different. Office, for example, we use
• 48:16
  nearest neighbor search to remove the content that user
• 48:20
  dislikes from recommendations completely.
• 48:24
  And for this test, it went, well able to decrease the amount of
• 48:31
  dislikes on the platform and we just put it aside and decided to
• 48:36
  not launch it because we learned how to do it, we proved our
• 48:40
  hypothesis but there's was no point for with Business in
• 48:44
  deploying this model. If this example, I hope to
• 48:48
  provide the Highlight that sometimes, even a proven
• 48:51
  hypothesis, won't be able won't be enough to Roll out the tests
• 48:59
  on 100% of users. So a lot of times we're just
• 49:05
  Experimenting with your behavior, trying to see what our
• 49:08
  model caused in the users patterns, and then decide what
• 49:13
  to do next. And that's why it's so so
• 49:19
  important be able to end a month to bring out a new model test
• 49:22
  the hypothesis because a lot of times even if you're right it's
• 49:25
  not going to launch your not because not worth the complexity
• 49:28
  or the business metrics or yes, it is surprising to me sauces
• 49:33
  dude, it's completely true. Yeah.
• 49:36
  Did so last one for you? What was the last book?
• 49:39
  You read the last book I write. Oh my God.
• 49:43
  So it would be the main driven design in C sharp.
• 49:50
  I wanted to learn about Newseum be sometimes.
• 49:53
  Yeah, I wanted to learn about writing better apis or services
• 49:58
  or restricting domains between the services because we can not
• 50:03
  struggles about it but we didn't have an idea around services.
• 50:08
  So I have tried to learn something incredible, man.
• 50:14
  Well, this is super cool and I appreciate you coming on here
• 50:17
  and talking to us and being very transparent being Open about how
• 50:21
  you've built the Rex's over there.
• 50:22
  Fun .ko. And so that's it, man.
• 50:27
  Jake, you got any final words? No, I wish.
• 50:31
  The, my last book I read was as as technical.
• 50:37
  I think, what was your life one? Winnie the Pooh.
• 50:41
  Well, I think, I think I was reading as I'll show you some
• 50:46
  product manager, light, some product manager light books, you
• 50:49
  know. It's mostly just feel good
• 50:50
  stories before bed. Yeah, well, no, that's my
• 50:54
  feel-good stories. Before better than ones.
• 50:56
  I have been reading to my daughter.
• 50:58
  I'm reading like a thousand words in German these days
• 51:01
  because she's she speaks better German than I do.
• 51:04
  But that's a whole nother story and a whole nother rabbit hole
• 51:07
  that we can get down. Glad man, thank you so much for
• 51:11
  coming on here and if anyone wants to hear more about what
• 51:16
  y'all are doing, I highly encourage them to read the Post,
• 51:20
  there's so much information on all these blog posts.
• 51:22
  We referenced it a ton will leave those blog posts in the
• 51:25
  description, I'm sure if you made it this far, you're going
• 51:28
  to love the blog posts and gleb, are you guys hiring by chance?
• 51:33
  Yes, we do. And for this week, but will
• 51:38
  there will be a Tolkien by me on the Divan data Fest, which will
• 51:44
  highlight those ideas, but I promised better English and kind
• 51:49
  of pet. The structure of no worries.
• 51:54
  Dude, this was perfect. So we'll link to that talk.
• 51:57
  Once it's out to in the description, if anyone is
• 52:00
  wondering where they can find it.
• 52:02
  Thanks again. Glad thanks for family.