Latest / Agentic Conversations (formally mlops.community) / Learning from 150 Successful ML-enabled Products at Booking.com // Pablo Estevez // Coffee Sessions #46

Transcript

Learning from 150 Successful ML-enabled Products at Booking.com // Pablo Estevez // Coffee Sessions #46 · Agentic Conversations (formally mlops.community)

• 0:16
  Hey everyone, welcome to another episode of ml opt coffee
• 0:19
  sessions. I'm your co-host Vishnu and I'm
• 0:22
  joined today By Demetrios as always a dimitrios.
• 0:26
  What's up everyone? I today actually have a car For
• 0:31
  our coffee session, very end. You joining me.
• 0:35
  I'm going to say thanks to our guests.
• 0:37
  Pablo for being patient with us because you all don't know this.
• 0:41
  But just before we started, we spent the last 10 minutes with a
• 0:45
  bit of a zoom hiccup in my travel set up because I am not
• 0:51
  at home. If you couldn't tell the people
• 0:53
  that are listening, definitely can't tell, but those that are
• 0:55
  watching. See that I'm in an undisclosed
• 0:57
  location, on an island in Greece, But that's not what
• 1:02
  we're here to talk about today, we are here to talk with Pablo,
• 1:06
  my man from booking Vishnu. I stole it away from you, you
• 1:09
  take it. Oh, man.
• 1:11
  Now that's okay, that's okay. I think everyone's wondering if
• 1:13
  you're in a CIA safe house or something.
• 1:15
  But today, we have the pleasure of being joined by Pablo Estevez
• 1:19
  from booking.com, where he is a senior machine learning leader.
• 1:22
  I let him give you his entire bio, but we're very excited to
• 1:25
  have Pablo join us especially after where I believe some of
• 1:29
  you have joined our In groups where we were able to discuss
• 1:31
  the paper that he wrote we're all big fans of his work and are
• 1:35
  very excited to dive into the seminal paper 150 successful,
• 1:38
  machine learning, models and the lessons learned that
• 1:41
  booking.com. Pablo, thank you for joining us.
• 1:44
  Well, thank you for the for the invite.
• 1:45
  Super happy to be here Chandler, I couldn't join the book club
• 1:48
  last time. I was in holy the time, but
• 1:50
  happy that we had an opportunity.
• 1:52
  No, I mean, now everybody can hear the book club and everybody
• 1:55
  can hear your perspective on this paper.
• 1:56
  So I think think it all worked out.
• 2:00
  I think many of the people that have, you know, listening to our
• 2:03
  podcast that have been in the community.
• 2:04
  I've had the privilege of, you know, being exposed to your
• 2:07
  work, but maybe don't know as much about you the person and
• 2:11
  your background and how you ended up at booking.com.
• 2:13
  Maybe we can kick things off by learning a little bit about your
• 2:16
  journey to where you are now jokes from.
• 2:19
  I think it's actually interesting because I just
• 2:21
  started with booking and data science, machine learning Etc,
• 2:25
  eight years ago. And even though it's not that
• 2:27
  long ago, I think it has changed a lot the way people.
• 2:30
  Go into these roles. Seen by the time all especially
• 2:32
  the transition but before that, they were not many careers and
• 2:35
  machine learning and data science for this kind of
• 2:38
  Industries. And most of the people who join
• 2:40
  at the time, they were coming from some diverse background, my
• 2:43
  case. I came from electronic
• 2:44
  engineering. So I was doing the radio waves
• 2:47
  and stuff like that, then I move into robotics robotics of been
• 2:50
  too much in their name. Then I was working with on
• 2:52
  engineering side of it, but throughout through the Journey,
• 2:56
  I got involved into Data techniques and machine learning
• 2:58
  techniques, and that's how I ended.
• 3:00
  The working on this field in the case of booking.com are you.
• 3:03
  So many of my colleagues of the time they were coming from
• 3:05
  completely wears background to help people, from aerospace,
• 3:07
  engineering from statistics mathematics bioinformatics, it
• 3:12
  was quite quite mix. And then for the last seven
• 3:15
  years, I could have been working booking mostly I was working on
• 3:18
  with us our machine learning, scientists title changes, also
• 3:21
  it with the evolution of the field you could call it.
• 3:24
  Now, machine learning scientists and I work mostly on the front
• 3:27
  and side of the of the So they decided he supposed to our
• 3:31
  users, feel for accommodations. And then now for non
• 3:35
  accommodation products booking is expanding to other markets is
• 3:38
  now doing flights attraction transportation and I'm working
• 3:42
  lazy on that on that area. Got it.
• 3:46
  Got it. Quite the diverse jury that got
• 3:49
  you to where you are today in terms of booking.com and being
• 3:54
  in the hospitality industry, very cool to hear all about
• 3:58
  that. One question, I have just to
• 4:00
  start, you know, you mentioned that you have a very diverse
• 4:03
  group of people you work with. And that your background is very
• 4:08
  sort of cross-functional. I would love to understand you.
• 4:10
  How big is the machine learning group at booking and can you
• 4:13
  give us a little bit of context that the, how machine?
• 4:16
  His run at looking in terms of a team and organizational
• 4:18
  standpoint. Sure.
• 4:20
  So let's see. We have different organizations
• 4:24
  in Booking depending on what kind of goal that part of the
• 4:28
  organization wanted to see if it's an important thing to note
• 4:30
  in there is a booking is not a company that makes or sells or
• 4:34
  its main product. Is not much in learning.
• 4:36
  Module m is one of the tools we have for chief, our business
• 4:38
  goals, and to support our partners and support our
• 4:41
  customers, right? And depending on the part of the
• 4:43
  company, the the way that you Use machine learning differs
• 4:47
  have it. We have a couple of hundred
• 4:49
  people. I don't have the precise.
• 4:50
  Number also depends everything where you put this tension
• 4:52
  between machine. Learning science.
• 4:53
  Is machine learning in your data science Etc.
• 4:55
  But there is all the couple of hundred people working on
• 4:57
  related disciplines across the company and Depends a bit on
• 5:02
  that in some areas where you have heavy product development,
• 5:04
  you have very good multi, multi discipline team try.
• 5:07
  You might have a team working, let's say on a user facing
• 5:10
  problem. Let's say the hotel page to give
• 5:13
  an example, you can imagine, there is a couple of designers
• 5:15
  and Cooper. Developers have a product
• 5:17
  manager, and if you think that there is enough material in
• 5:19
  there for machine learning to help, you might also get up
• 5:22
  machine learning science. So as one set up, you have this
• 5:24
  kind of multi skilled team, normally coordinated by a p.m.,
• 5:28
  a product manager who understand what is the end goal that you
• 5:31
  want to achieve within the team and everywhere that you've seen
• 5:33
  from the discipline. And you can see very, very nice
• 5:36
  collaborations in there. Sometimes we make a model, you
• 5:38
  have the way up in you do is cut with a copywriter who tries to
• 5:41
  find how to explain it. When you have this course, it
• 5:43
  was a copywriter you realize that the mall you were The
• 5:46
  building doesn't get translate to a user.
• 5:47
  Maybe we're trying to to satisfy or you talk to the designer
• 5:51
  designer called and say, okay. Very nice model but I kind of
• 5:53
  like the user experience that you are designing.
• 5:55
  So I need your model to give me more specific answer so I can
• 5:59
  actually put the user on the right set of all.
• 6:01
  I think that the model is in a is exposing a part of the of the
• 6:04
  journey. Well, the user still needs
• 6:06
  option so I don't need one. A tenant in a journal to Output
• 6:09
  them Alternatives, right? So they have a better
• 6:11
  understanding on that and the developer could come and say
• 6:13
  yeah the features you're requesting they take too long to
• 6:16
  So I got pellicano, deploy them all your thinking in this part
• 6:19
  of the journey, right? So that's one set up in all
• 6:22
  kinds of the company. I'm sorry you had something no
• 6:25
  just saying just saying got it got it.
• 6:26
  Yeah, I would love to understand you other setups in all parts of
• 6:29
  the company where you have much more Focus ml topics a fraud
• 6:34
  detection, you can imagine there is a very heavy component of a
• 6:37
  male in the event. You do have email organizations.
• 6:39
  When you have several ml people working, could be celibate, mix
• 6:42
  between a male scientist data scientist ml engineers.
• 6:45
  Maybe the manager of the team. He's a specific data science,
• 6:49
  skilled person but even there the is in the the main driving
• 6:54
  guideline is that we focus on what we want to build.
• 6:56
  So even if you are working a fraud or cybersecurity, you
• 6:58
  would probably have lawyers involved somehow in the
• 7:00
  organization because as part of the of it if you are working on
• 7:04
  the partner face and you will have commercial managers because
• 7:07
  whatever you are doing, will interact with article people who
• 7:09
  manage properties and you have to make sure that you have a
• 7:11
  good alignment and while you're making in there, right?
• 7:14
  But it depends how we don't. Nation.
• 7:16
  And then maybe there is a third part, which is infrastructural
• 7:18
  teams. So, this do I describe our two
• 7:21
  different type of organizations that support product development
• 7:24
  in different ways and you also have internal products, or since
• 7:27
  there are infrastructure that we all use within booking and they
• 7:30
  do have also kind of a technical organization where you mostly
• 7:32
  have developers. And if we do so much in there
• 7:34
  and available so much in Learners, who are building
• 7:37
  things like mobile engines or monitoring systems or
• 7:41
  capabilities, right? You can have, I don't know,
• 7:43
  image, recognition capabilities instantly, Yeah.
• 7:46
  Yeah, that makes a ton of sense. It's very helpful to hear this
• 7:50
  context for how booking, actually organizes, its machine
• 7:53
  learning teams and thinks about how on a, you know, human sort
• 7:57
  of people level and human resource level, how machine
• 8:00
  learning fits in. Because I think everybody that's
• 8:03
  read this paper walks away, with the sense that booking is an
• 8:07
  extremely product and user-focused culture.
• 8:10
  This is not just a functional group kind of company.
• 8:13
  This is, how can we help the customer?
• 8:15
  How does our technology advanced that in machine learning is one
• 8:19
  of those. I think that's one of the things
• 8:20
  I walked away from the paper feeling and so with that, I
• 8:25
  think it would be really cool to start talking a little bit more
• 8:27
  about the paper and it's takeaways to start.
• 8:31
  This is one of the most popular papers that people kind of come
• 8:35
  back to continuously. I'd love to understand a little
• 8:38
  bit. How did you come to?
• 8:39
  You know, what to want to write this paper?
• 8:41
  How did this effort materialize and to have so many?
• 8:45
  Any so much knowledge packed into such a short amount of
• 8:48
  time. How did you do that?
• 8:49
  And why? Sure.
• 8:51
  So, as I mentioned, I think, I would say when you have seen
• 8:53
  broken and especially the first year that I was in there, like
• 8:56
  the first two, three years, there was a lot of exploration
• 8:58
  going on. Not only, we have thought it was
• 8:59
  a whole set of people in there. I join a team that was already
• 9:02
  producing quite some interesting material.
• 9:04
  They were two people who are authors of this paper Lucas on
• 9:07
  Tammy's. They were joining also more, or
• 9:09
  less at the time. And as I said there was a lot of
• 9:10
  exploration, we're really trying everywhere within in particular
• 9:14
  in our case, the front-end part of Booking.com to try to find
• 9:17
  Value. And I was again as you mentioned
• 9:19
  is part of the company's culture, right?
• 9:20
  Every other team was also just trying to look what is the value
• 9:23
  for the company for the guests at this?
• 9:25
  Not mentions of them, but also for the partner, I mean, we are
• 9:28
  a two-sided market place and we have to get also for our
• 9:31
  properties, right? And in the case of a metal with
• 9:34
  the same idea, let's try to explore.
• 9:35
  Where can we use ml to add value to all of these components of
• 9:39
  the of the marketplace? And there was a process.
• 9:41
  I mean, is if you read a paper, it might look like a very Has a
• 9:45
  story where you think a lot of these ideas.
• 9:47
  Of course, it was a lot more organic, right?
• 9:49
  So you have pro teams, they create hypotheses normally about
• 9:51
  these are coming from user stories, so users research come
• 9:55
  up with an idea. Okay, there is a lot of people
• 9:56
  looking for flexible travel. We need to offer alternative
• 10:00
  since like this and then you start exploring different
• 10:02
  options. But it was a very, really, very
• 10:04
  productive period on this exploration and the team grew
• 10:08
  and the way start improving our methods.
• 10:10
  In a way to allow us to kind of inspiration, more effective.
• 10:15
  This paper. That was three years ago.
• 10:17
  Basically, look back and say, okay, we have done a ton of
• 10:21
  exploration, we have not owned of model, we've stated many
• 10:23
  things and we are going to be successful in any of them.
• 10:26
  I just wanted to highlight for the paper focus on 150
• 10:28
  successful applications. There are things that work it
• 10:31
  behind this. There is another couple of
• 10:33
  hundred of things that did not work and that also teach you
• 10:35
  something right, three things always are basically ways.
• 10:38
  It was to have the paper was looking but say, let's try to
• 10:40
  look at all of this work and it's like, 25 there are patterns
• 10:43
  that Can help also and other practitioners, be more effective
• 10:46
  on using machine learning to get value.
• 10:48
  Yeah, makes makes makes perfect sense Demetrios.
• 10:50
  I see you, I see you holding that Mike there.
• 10:53
  What are you thinking, man? I just I'm fascinated by what
• 10:58
  you were talking about before on the different compositions of
• 11:02
  teams and how these teams end up maybe sparking creativity within
• 11:09
  each other. And so, were you on one of the
• 11:13
  teams that Multidisciplinary, or were you on this strict ml team?
• 11:19
  Or were you doing much more of a platform side?
• 11:24
  What was your role when you were looking at that?
• 11:28
  Now? Multiply that by invoking?
• 11:29
  I've been on the multidisciplinary team so
• 11:31
  because I was working on front-end.
• 11:33
  And in there, you mean, your men client is a customer to call who
• 11:36
  comes walking.com. Direct position channel is very
• 11:38
  much driven by Design. You have certain requirements on
• 11:41
  performance. So I was always in this kind of
• 11:43
  mood, People post teams will just kill team that is trying to
• 11:46
  set up a username at some point. Also exploring with these
• 11:49
  organizational aspect, we did create a team that was mostly ml
• 11:53
  people but this ml people was a supporting for a lot of
• 11:56
  different problems. So the prices were developing
• 11:58
  worry, still move your skills organizational.
• 12:00
  It made sense at some point to centralize some of this
• 12:03
  knowledge just because the amount of demand of morals and
• 12:06
  output. Actually, if you see everything,
• 12:08
  the beginning of the story were telling in bed, one thing, we
• 12:11
  noticed you don't realize that they were to type We could offer
• 12:14
  the world very specific modes. You were with one thing in one
• 12:17
  place. There are no examples, the
• 12:18
  Germany Hotel page Gallery, you have the photos.
• 12:22
  You want to know which photos versus very specific problem.
• 12:24
  You can make a very targeted one, that's what you could very
• 12:26
  much do, if you are in the hotel page team, but we realized that
• 12:29
  we could also get value from more kinetic models.
• 12:31
  That could be serving many different things.
• 12:33
  And that's what we try to settle wearing this team.
• 12:35
  That was a machine learning thing that was interacting with
• 12:37
  all of these different disciplines.
• 12:38
  So projectwise, it was still something you would do with the
• 12:41
  other disciplines, but we said, okay, if I can maybe instead of
• 12:44
  Making a model for the gallery of the photos of the hotel.
• 12:46
  I make a photo Rancor that can be used in the hotel phases,
• 12:49
  have resulted different parts of the funnel, then I'm actually
• 12:53
  able to span the with more, my eye real.
• 12:55
  So with the performance of the model, that's one thing but
• 12:57
  because you are not targeting specific case or signature, not
• 13:00
  diluted the the relation between features and Target.
• 13:03
  Labels is a weaker but now you have a much wider application so
• 13:07
  your overall impact on the business and the customer
• 13:10
  experience is also bigger. Yeah, let's zoom into that.
• 13:14
  At the business that you're talking about.
• 13:15
  Because one thing that I took away from the paper was how you
• 13:21
  were you talking about Driving true.
• 13:26
  Business impact is amazingly hard with machine learning and I
• 13:31
  was wondering about like this idea of how you can be more
• 13:36
  clear on the problem that you're trying to solve.
• 13:39
  This is one thing that we've talked to many about and it
• 13:42
  seems like you guys learned a lot when you're putting all of
• 13:47
  these successful models into production and the non
• 13:49
  successful ones. But is there something like that
• 13:53
  you could help us? Understand like what are some
• 13:56
  strategies that you found really work, when you're trying to
• 14:00
  figure out what problem you're solving and how to help the
• 14:05
  impact? How to make sure that it impacts
• 14:08
  it more like you were talking about saying, oh well there's a
• 14:11
  bit of a trade-off. But if I can reuse this model in
• 14:13
  other areas of the business, I get more impact.
• 14:17
  Are there other strategies that you found that were helpful?
• 14:20
  In that sense? Domination of many things, I
• 14:23
  think we're going to dodging on several aspects.
• 14:27
  There is one very weak enabled, which is two of them.
• 14:29
  So one hand, mentioned a bit of these already.
• 14:31
  What in a company that has very clear objectives, as a business
• 14:35
  objective and customer experience objectives in this
• 14:37
  case. So, it's in the filling is
• 14:39
  aligning to a very understanding that as a general practitioner,
• 14:42
  you're ringing a tool, but you are not the outcome depends on
• 14:44
  the industry. I mean, if you are in a company
• 14:46
  actually sells a machine learning solution, maybe they'll
• 14:48
  come, he's the model itself. But if your company is selling
• 14:51
  or Frozen Seas have defined a different, need to realize that
• 14:54
  you are a tool to this end, right?
• 14:56
  And that means that your end goal cannot be Precision or
• 14:59
  recall or anything like that, your angle is achieving that
• 15:01
  goal. If the goal is enabling more
• 15:03
  people to find the right accommodation or maximizing
• 15:05
  profit for your stakeholder, for shareholders, or whatever it is,
• 15:08
  the goal that you're trying to achieve in there.
• 15:10
  I think there's a step then you have to stab find that Gap
• 15:14
  because I don't know why they cannot force people to
• 15:16
  accommodations. I cannot force a certain
• 15:18
  Behavior, so I have to start filling that gap of 2 - kid.
• 15:20
  What can I do? We see my A skill that will
• 15:22
  eventually lead to that in the case of booking, the way we will
• 15:27
  learn the we learn how to satisfy these needs.
• 15:30
  And that we test them is Troy, but this is testing its.
• 15:34
  I mean, if there's a whole discussion on every test
• 15:36
  instrumentation Etc using the concept.
• 15:38
  The overall concept is accurate scientific concept which is
• 15:40
  hypothesis testing. We will start with a hypothesis
• 15:42
  and the hypotheses are the ones who will try to tie your idea,
• 15:47
  your technical idea to his end goal.
• 15:49
  So if my end goal is helping people go through a Quicker.
• 15:52
  Because I seen that will save them time.
• 15:53
  It's Etc. Then I started building my
• 15:55
  hypothesis. If I put better recommendations
• 15:57
  on the ranking, they will more easily find the right solution.
• 16:00
  Therefore, they will not have that moisture to support
• 16:02
  adoption. They will get them closer to our
• 16:04
  end goal. So your eyes on hypothesis, kind
• 16:06
  of ties, the thing that I can actually affect, I can do a
• 16:09
  rancor to my end goal. So all of these negative
• 16:13
  variables on the, on the planning, and then there are two
• 16:17
  more dimensions in there. One of them is having a good
• 16:20
  tool set like Harry. Indifferent and worse in which
• 16:22
  you can affect when I say want to improve my ranking, it was
• 16:25
  nice, but are many ways in which I could try to do it.
• 16:27
  I will if you were, when I step forward, if I want to improve
• 16:29
  decision-making on the user ranking, it just one way to do
• 16:31
  it copies and other design is another, there are many
• 16:35
  different ways in which you can improve the decision-making
• 16:37
  process of the user. If you choose to go for ranking,
• 16:39
  there are many ways to do it. So having this diversity, allows
• 16:42
  you to explore many options and try to find the next one.
• 16:45
  And the last element on here is capability of measurement.
• 16:48
  So being able to know if you report, this is actually Realize
• 16:52
  this, right? So if I make a different answer,
• 16:53
  because now I try to use a online learning using some maps
• 16:57
  a top, which is a very specific impersonation of this of these
• 17:02
  hypotheses. And I actually trace it back to
• 17:05
  is one of my studies so I have to go make a decision.
• 17:07
  Do I pick help help the business achieve your business goals for
• 17:11
  that? In our case, very often we rely
• 17:13
  on our experimentation platform but the general concept
• 17:16
  hypothesis test. If I will try to use tools from
• 17:17
  a hypothesis testing to know if this specific setting I pulled
• 17:21
  in, My multi-armed Bandit is driving the whole strength in
• 17:24
  the business value that I was looking for the beginning.
• 17:27
  I love what you just said. Because I think this, you know,
• 17:30
  I was hoping that we could go through one of the six lesson,
• 17:33
  you know, each of the six lessons and it, you know, I
• 17:35
  think we've basically now jumped into for those of you who really
• 17:39
  is your through all of them but it feels a little bit like
• 17:41
  lesson for right before solving a problem design it.
• 17:44
  And you're talking a little bit about some of your, some of your
• 17:48
  tools to do that. And to me when I heard your
• 17:51
  answer, I heard that booking has learned through the process to
• 17:55
  sort of, just experimentation iteration to do three things,
• 17:59
  when tackling a problem generate a hypothesis, apply abroad
• 18:03
  toolkit, and then leverage measurement, right?
• 18:06
  Let's see how well you're doing. Yeah.
• 18:08
  I mean you said it, I'm just I, I, you know, I just get to hear
• 18:13
  it back and so, so, you know, I think for me like, when I when I
• 18:17
  think about that, you know, that it's not something that I think
• 18:21
  a lot of machine, I always think about this, a lot, you know,
• 18:24
  machine learning professionals. We tend to come from different
• 18:26
  backgrounds. We tend to come from diverse
• 18:28
  backgrounds. As you said, I came from a bio
• 18:29
  engineering background. And then we have, when were
• 18:32
  challenged to sort of solve machine learning problems.
• 18:35
  We each bring sort of what our field taught us.
• 18:38
  But there's not quite this like set of best practices.
• 18:41
  Or like clear way that, you know, you're, you know, you're
• 18:46
  supposed to approach approach. A new problem per se, right?
• 18:49
  Sure. You know, train test set but
• 18:50
  something like what? Just said, I would love for that
• 18:53
  to be tight and, you know, machine learning classes from
• 18:55
  like a hypothesis toolkit and then measurement standpoint.
• 18:58
  Is this something this sort of framework around problem design.
• 19:01
  Something that you sort of try to teach new booking machine
• 19:04
  learning employees as they come in and emphasize the thought
• 19:07
  that if I was here with the, the registration, I think we can
• 19:09
  either wait for a while work calls product data science,
• 19:13
  trying to make the focus of our product makers and other were
• 19:16
  the religious crisis machine. Learning scientists, either be
• 19:18
  part of the, of the role is focus on that were The makers of
• 19:22
  things that we present we learn agile product development
• 19:25
  because I say whether we're going to make problems with his
• 19:27
  right. So make whom we have a very
• 19:29
  strong focus on product design and product development, that's
• 19:32
  one component. In the case of ml across the
• 19:35
  company, we've already strong, focus on experimentation.
• 19:38
  And that means that everybody who comes to a company, gets
• 19:40
  trainer experimentation gets trained or hypothesis testing on
• 19:42
  basic statistical concept, and it's really across all roles on
• 19:46
  the on development. That's, it's a fancy way to say
• 19:50
  that. We're very data-driven.
• 19:51
  That's One thing that I like to mention in some cases, are the
• 19:56
  passive it on them by Roman the company, the culture in some
• 19:58
  places you have to use the fire test right eye, why would you
• 20:01
  need to make a, you make a business case to try to find a
• 20:03
  data in booking more or less. All the time.
• 20:05
  People the fishes, they will ask you is, did you make a test?
• 20:08
  I will have a very good testing, environment and infrastructure
• 20:12
  and because it's very good is also very simple to make Pairs
• 20:14
  and we have the culture of always testing right now, coming
• 20:17
  up with assumption, but if it is Sol test it, and I think that
• 20:20
  kind of development. As I mentioned for remember,
• 20:22
  it's the same for design and copy copy compensate would like
• 20:24
  to do better. Relation with our clients want
• 20:26
  to be more personal they come up with a new set of copies.
• 20:28
  You have policies, you have the implementation of our policy to
• 20:31
  make a testable setup in which you actually change the copy and
• 20:34
  call People by name. For instance, because I will be
• 20:36
  more personal and you put it in an experiment, right?
• 20:38
  You make your measurement and you verify your apoptosis learn
• 20:41
  and iterate. The Savvy, the cycle.
• 20:43
  Yeah, I think is very helpful and I think this helps us move
• 20:48
  on to new lessons. In the paper.
• 20:49
  We've talked about, you know, lesson for around problem design
• 20:52
  and now I want to talk about sort of lesson 6 and less than 2
• 20:55
  and all right folks that warning is my numbering.
• 20:58
  Might be a little bit off here but you know, you'll know what
• 21:01
  I'm talking about. Basically these lessons, talk
• 21:03
  about, you know, offline model performance is just a health
• 21:06
  check and then experiment design pays.
• 21:09
  All right, and monitoring basically.
• 21:11
  So my question to you, is, You know, I'm a, you know, sort of
• 21:16
  like, a new machine learning engineer in 2021, and the
• 21:20
  discussions around monitoring and making sure that your model
• 21:23
  is doing what you actually wanted to when you've done it.
• 21:26
  You know, when you just done your sort of like isolated train
• 21:28
  test set, but doing it in the wild it's a huge topic.
• 21:32
  Somehow three years ago and even proceeding you your team and
• 21:38
  booking as a whole was able to come up with Frameworks and and
• 21:41
  methods of implementing sort of monitoring and experimentation
• 21:45
  Frameworks for machine learning models, without the plethora of
• 21:48
  tools that are available now, can you talk a little bit about
• 21:51
  how you guys built these tools from first principles and were
• 21:56
  able to scale? Sort of the lessons you learned
• 21:59
  about, you know, monitoring models in particular cases to
• 22:02
  all of those, you know maybe 150 or having many other models you
• 22:06
  did. So I like how did you build and
• 22:07
  scale the tooling around? Monitoring experimentation.
• 22:12
  So one thing is that There is still a lot of open question.
• 22:15
  Let's put another beautiful as a career, but I do think that we
• 22:17
  have been pushing for it for a while.
• 22:20
  We have a couple of things. So one thing is pretty
• 22:22
  mentation, as I mentioned is not only animal tool, so we have had
• 22:24
  in Booking culture for a while. I think we'll start
• 22:26
  experimenting in Middle 2005, eight, or something.
• 22:30
  Like, this is really quite some years.
• 22:31
  And if you search with on our side you will see quite some
• 22:34
  papers that we have in publishing.
• 22:36
  So were quite a solid player. Experimentation where that
• 22:39
  basically means is that when a man comes into the picture
• 22:42
  anyway, is it plain? As I mentioned Is very simple to
• 22:44
  make an experiment, very simple to validate hypothesis.
• 22:46
  At least, I wanted to confess in a randomized control, trial some
• 22:49
  things. You cannot because whatever you
• 22:50
  want to make tests, that are outside of our control or there
• 22:53
  are limitations. But from the National Treasures,
• 22:56
  in Booking are very simple. Do we find it in the code, you
• 22:58
  will find your hypothesis. You write the code in there, and
• 23:00
  there is a whole platform that setups the statistical testing.
• 23:04
  The reporter is also whole culture on making sure you are
• 23:07
  reading this results in the in the right way.
• 23:09
  So I think that seeing if there is anything that has been a huge
• 23:11
  enabler for pro-development and formatting, Happening is that
• 23:14
  because once you have very cheap experiments, it allows you to
• 23:17
  understand what is the relation between offline methods and
• 23:19
  online method allows you to test, your models are doing what
• 23:22
  you think they're doing even in the early stages model more
• 23:24
  development. You make some assumptions,
• 23:26
  right? Maybe they're right seem to
• 23:27
  optimize for the smallest Pharmacy or we should be
• 23:29
  actually looking for some kind of accuracy metric.
• 23:31
  Maybe I should go for something more complex, like a deep
• 23:34
  learning season that we gave me. Maybe a 10%, how much extra
• 23:37
  business value? We get from a 10% gain on Air,
• 23:40
  Mac, for this specific, all of these questions which we are
• 23:44
  very hard to understand from from a conceptual theoretical
• 23:47
  perspective, right? To what will be the impact on
• 23:49
  business of a 10% extra messy on something that is better user
• 23:52
  experience like a recommendation, you can just put
• 23:54
  in an experiment and you have the health infrastructure to
• 23:57
  with very cheaply and very structurally, correct, right
• 24:02
  with good hypothesis. Testing with good guard rails.
• 24:05
  There are some areas where you can get a better connection
• 24:07
  prison if you're working on something like PPC or fraud,
• 24:10
  this is a much tighter connection between the quality.
• 24:13
  Of your model prediction and the business value that is ring.
• 24:16
  I mean, if you can detect a single flower, you basically
• 24:18
  save 10% more money on for out. There is not much question in
• 24:21
  there, but if I can be a slightly better, getting my
• 24:25
  energy, say, score for the rancor.
• 24:27
  How much more will I help? My people decide.
• 24:30
  They could also scroll down and find the solution by themselves,
• 24:32
  right? It's I changed hold any position
• 24:35
  15 by 16, maybe it was the same for the use of right?
• 24:37
  They were both in the same street so that but that's one
• 24:41
  element that this just to focus on.
• 24:43
  So We had the experimentation element then there is something
• 24:45
  that is to seek from a male on one hand.
• 24:47
  We started developing our, we do this exploration a Time direct
• 24:50
  mention some years ago, we start exploring.
• 24:53
  How can we better use experimentation setup that we
• 24:56
  have been working very similar to what you will get?
• 24:58
  You know, a big Ecommerce has to get the most out of ML and we
• 25:01
  realize focus on certain set of settings is complex was playing
• 25:04
  here in the podcast. But since that we were falling
• 25:05
  for a male is try to understand when different when different
• 25:09
  models when you're comparing different models, weather
• 25:11
  prediction be fair. It doesn't make sense to make
• 25:13
  up. It in tomorrow's, if they are
• 25:15
  both decent models and they were done in good faith.
• 25:18
  They probably agree very often. So, it is actually no difference
• 25:20
  in the two models in which large part of the population.
• 25:22
  So by focusing on the instances, where the mall differ, you can
• 25:26
  actually see what how much value it is adding.
• 25:29
  You can also make a statement. There's a tops that allow you to
• 25:32
  to break the performance impact on the business impact.
• 25:36
  So you might have hearings on a fancy.
• 25:37
  Never a network competing to a very simple naive Bayes.
• 25:40
  Sure, you get more performance on your offline map.
• 25:43
  Trees. But also seller is lower and you
• 25:45
  can decouple these two things. So you can realize how much is
• 25:48
  the impact of the arid climate performance of the model.
• 25:54
  If they were both having the same impact on the site
• 25:57
  performance and if you realize that there is a lot of impact
• 25:59
  but the mall is a super slow, okay?
• 26:00
  You invest in infrastructure but you already know you have the
• 26:02
  business case of this investment infrastructure is going to be a
• 26:05
  joke because I already know of the moral will perform better,
• 26:07
  right? That's in there in the
• 26:10
  experimentation site. Then specifically for a male.
• 26:13
  We also started developing some years ago, our own matching
• 26:16
  their name platform which deals with stuff like moral serving
• 26:20
  feature, engineering feature serving etcetera and a component
• 26:23
  there was monitoring it and I think what what made up maybe
• 26:28
  with the difference in our case or what?
• 26:29
  I think that was quite productive was again the focus
• 26:32
  on Final objectives. We try to understand that our
• 26:35
  goal is not to make predictions, make them all predict the
• 26:40
  outcomes success. It wouldn't happen in reality,
• 26:42
  our end goal is to affect user behavior and business metrics.
• 26:47
  So, maybe an example, makes it a bit more and more clear, right?
• 26:50
  I could make them all a try to predict if I do, you're going to
• 26:52
  my breakfast and I if I see that you're going to buy breakfast, I
• 26:55
  sent you an email. Well, if I send, when you're
• 26:56
  probably going to eat breakfast because I send you an e-mail, so
• 26:59
  we're my moral matches, the end behavior is a read of.
• 27:04
  Now, if I read above could be witnessed in him, right?
• 27:07
  I don't even really predicting your behavior.
• 27:08
  I'm affecting your behavior. Moreover, many of the things, we
• 27:11
  do get very late feedback, right?
• 27:13
  People can cancel the reservations, you can cancel of
• 27:15
  go here in advance. You actually did.
• 27:16
  Book up to a year in advance, you can cancel anytime before
• 27:19
  you receive many of our state's. So all of these things tell us
• 27:22
  that this standard just look at your output.
• 27:25
  See if it matches the prediction might not be so useful.
• 27:28
  For our end goal which is actually affecting user,
• 27:30
  Behavior, understanding user, Behavior affecting business
• 27:32
  metrics and that brings the creativity.
• 27:34
  Okay, what can we do? Can we look into featured risk
• 27:36
  and you look into concept, or if there are actually some papers
• 27:39
  or try to look at the label field and we try to work on
• 27:42
  those? So what happened?
• 27:43
  If I know I mean, if I feel is an issue but what if I know that
• 27:46
  is an issue, I know that. Ezel ABI have this illusion of
• 27:48
  the delay. I know how this delay figure
• 27:50
  will behave. Can I do something about my
• 27:52
  mother? What we saw that in the paper is
• 27:54
  one specific thing. We put it in there because I
• 27:56
  think there was a most ignored if there is you can find around
• 27:58
  me and concept. Three featured.
• 28:00
  If etcetera you can find. But we put in there, is this
• 28:02
  idea of response distributions which is just looking at what
• 28:06
  the more says doesn't matter. What is reality?
• 28:08
  Just looking at what the more is answering and you can get a lot
• 28:10
  of information, right? If you see that within a typical
• 28:13
  sample of the population in your training data, immoral very
• 28:16
  often says Depends on the model. But safety is and not
• 28:19
  circumcised know and then you put it life and it embeds.
• 28:21
  Okay. Maybe your data.
• 28:22
  Would you turn on is not very similar to reality maybe if we
• 28:25
  change from one moment, we already sort of real red flag
• 28:28
  and much more interesting. Sometimes we have that very nice
• 28:30
  situation with a huge peak of the value.
• 28:31
  We go and check and we just not cumbersome currencies rights are
• 28:34
  things that are tools that tells you know where your modification
• 28:37
  is much in reality. Just working model is working,
• 28:39
  right? I think there was a, with the
• 28:41
  focus of the book. There is so much there, The
• 28:43
  Parson what you just said also. I just want to point out all the
• 28:46
  people listening And that Pablo is actually wearing a t-shirt
• 28:48
  that says rapid experimentation. So he really practice what he
• 28:53
  preaches. Now, one of the, you know, sort
• 28:58
  of points, I want to make here and one of the things in the
• 29:01
  second half of your answer, you know, I have questions around
• 29:04
  both apps but you know you really mentioned this idea of
• 29:07
  monitoring you guys have a strong idea.
• 29:10
  Your end goal and what you're actually trying to affect.
• 29:13
  And I think what stands out to me there is a because of your
• 29:16
  and this might happen. This right?
• 29:19
  But that because your company has such a product focused and
• 29:23
  user Centric culture. You have a really strong idea of
• 29:26
  the real world interactions. You're trying to affect and sort
• 29:30
  of what the ideal real-world impact.
• 29:34
  You want to achieve it, which is actually, I find very difficult
• 29:37
  for machine learning professionals, not all, machine
• 29:39
  learning, professionals are blessed with the strong product,
• 29:43
  you know, hat on their head or mindset.
• 29:46
  You know. I think I saw Go with it a
• 29:47
  little bit to really say, like, what are we trying to do for the
• 29:49
  user? And because you guys had that
• 29:52
  from the start, you were able to then say, in a technical
• 29:55
  platform and in our monitoring solution, we have to bake in
• 29:59
  that same strong sense of the real world and the ability to
• 30:01
  capture it. Does that does that kind of
• 30:03
  resonate? You know, it's not just output
• 30:05
  versus prediction. It's really about that real
• 30:07
  world, sort of Analysis company culture.
• 30:11
  I saw the company is not only about the mail but across all of
• 30:14
  the roles is always focusing on what you want to.
• 30:17
  To achieve. But at the end, maybe something
• 30:19
  that we just kind of completing the the question before we have
• 30:23
  experimentation and we have monitoring the one part of that
• 30:25
  misses everything there is that one just of experiment, you
• 30:28
  still lose the connection with the business value, right?
• 30:30
  So you make the statement, you made the POTUS is all connected
• 30:32
  to a business value, 10 to monitoring is quite hard to keep
• 30:35
  the relation of where I'm affecting business value.
• 30:37
  And the one thing that we have time, we've been exploring more
• 30:40
  mostly lately, you set of like online learning which kind of
• 30:43
  tries to keep that logo. And I think it kind of Misses
• 30:45
  two days. You have the imitation to prove
• 30:47
  your hypothesis but it's kind of time-limited the moment in which
• 30:50
  you might use this test, then you have always monitoring tool.
• 30:53
  That tells you is anything going on.
• 30:54
  That looks weird with my myself. Travis, anything breaking is
• 30:57
  anything moving in a way that I didn't foresee and online
• 31:00
  learning tries to kind of bridge these two things, right?
• 31:02
  Size to continuously, assess if your model is fulfilling its
• 31:06
  angle angle, right? Which is not there.
• 31:08
  Some kind of business or user Behavior code.
• 31:10
  Got it got it. We just one more question.
• 31:13
  I'm sorry Demetrios. I'm really, it's all good.
• 31:15
  All the time I like No I'm enjoying.
• 31:19
  You know I think one thing that you said I loved which is that
• 31:22
  you allow your, your team teammates to Define experiments
• 31:26
  as code. Did I hear that correctly?
• 31:28
  That you can easily that you guys are basically create an SDK
• 31:31
  that allows people to run experiments in their code pretty
• 31:33
  seamlessly? And that's kind of common in
• 31:36
  biggest e-commerce, right? Do you have some companies use
• 31:38
  except and then do some comments are internally.
• 31:40
  But I would say that's big Commerce, have it and big
• 31:43
  companies have it and I think it's a very weak and I work for
• 31:45
  a company to have Very simple ways to do.
• 31:48
  Hypothesis testing, that's what makes you later even right right
• 31:51
  right. Yeah I think I you know I heard
• 31:54
  the Airbnb has a great experimentation platform.
• 31:57
  I think, you know, there's actually a book that our good
• 32:00
  friend Louise you but you know is talking about when it came to
• 32:02
  running experiments and how you know Microsoft and you know how
• 32:07
  Airbnb and others enable it. My question is could you talk us
• 32:10
  through the workflow for that a little bit?
• 32:12
  Because, you know, somebody working at a smaller company?
• 32:15
  One thing we always OS. As in, we try to do, try to
• 32:19
  learn from best company in a big companies and replicate what
• 32:24
  they're doing. But without necessarily knowing
• 32:26
  exactly what their exact workflow looks like how
• 32:28
  intensive it is and what parts of the workflow you want to, you
• 32:31
  know, borrow from. So I'd love to understand like
• 32:34
  if I'm a dataset, well, how do I drive you and your listeners.
• 32:38
  There is a very recent paper, I can try to look for the
• 32:41
  reference, but it's actually from the head of experimentation
• 32:44
  of many of these companies where is the guy who was working?
• 32:46
  When I was premeditation in Booking, I think that is, I
• 32:48
  don't know Chronicle, how he was in there.
• 32:50
  He was in my 40s in Airbnb. There is one guy from Microsoft,
• 32:52
  for sure. So it is like six of these
• 32:55
  persons who are all leaders on experimentation on their
• 32:57
  company. So they do the paper is a key on
• 32:59
  how to create on the relevant. Experimentation cultural
• 33:01
  infrastructure? We should accompany.
• 33:03
  Sup super nice paper. If you are in a company of those
• 33:05
  not have this culture and this infrastructure to understand
• 33:08
  what are the different elements. And what they are going there
• 33:10
  with is, is a read of a play between developing
• 33:13
  infrastructure and developing culture.
• 33:14
  We have to do is two things together to make.
• 33:16
  The infrastructure is invested on and is producing valuable
• 33:21
  that is connected to our investment and its values.
• 33:24
  The servitor their argument in there because of booking.
• 33:27
  As you mentioned the infrastructure, of course it
• 33:29
  happened because people investment, but he's in their
• 33:31
  world, by have quite a, quite a solid one.
• 33:34
  If you see with how the flows goes in practice, what I would
• 33:38
  highlight is the flows hypothesis test is.
• 33:40
  So the flow is basically the scientific method in practice.
• 33:43
  You write down on a prophecies that you start with a theory.
• 33:46
  Ring that we very becomes a test our hypothesis.
• 33:49
  So, your theory is that users will engagement on the platform
• 33:51
  and they will help them with a decision.
• 33:53
  Then you make a testable hypothesis.
• 33:54
  I will help them with a decision by ranking properties, with
• 33:58
  their value proposition. And then I actually Implement
• 34:02
  that into code modeling, Etc. Once you have it in there, you
• 34:06
  have to basically run experimental design.
• 34:08
  Now, enjoy area in an e-commerce, experimental designs
• 34:11
  of randomized, control trial, and more or less easy to the
• 34:13
  top, we write users. And in the case of an SDK is
• 34:16
  very Because you literally putting one line experiments
• 34:19
  that here, this is code for version a this calls for
• 34:22
  version. Be the SDK will send the message
• 34:24
  to has been patient platform, experimentation platforms, take
• 34:26
  care of the login and then it processes able to generate any.
• 34:30
  Statistically valid aesthetically values report,
• 34:34
  there is a lot of assumptions that go into having a valid sou,
• 34:38
  second result. And there goes our view into the
• 34:39
  culture part and we don't the guardrails and infrastructure
• 34:42
  sets to make sure that the result you're going to see is
• 34:45
  valid and the general. Sighs, I'm going the right way.
• 34:48
  We had to start all this in the company.
• 34:49
  We have this fermentation to putting some warnings trying to
• 34:52
  help people in their. Obviously, once you set up your
• 34:54
  sister experiment together, super nice monitoring interface,
• 34:59
  where you can see how your face is going to wait for the test
• 35:01
  results. You see all the statistical
• 35:02
  results? And the way that we see, this is
• 35:06
  the tool and I understand similar in different companies.
• 35:09
  The tool provides you information that you can use for
• 35:11
  decision-making. So, the end of the test, you
• 35:13
  know, if you validated your hypotheses, and then you can
• 35:15
  decide where you go forward, They're do a stop or well
• 35:17
  educated 8 normally, this kind of test also gives you a lot of
• 35:21
  additional data. Not only the main hypotheses, I
• 35:23
  don't know user. When Twitter California law
• 35:25
  reviews, when higher, whatever is your main metric?
• 35:28
  You can also understand how this happens.
• 35:30
  And through understanding, you can either write in your idea,
• 35:32
  may be a real workout but you realize that you reduce errors
• 35:35
  or frustration went down because you see a CS contacts went down.
• 35:39
  Then allows you to iterate over the inward provide thought but
• 35:41
  maybe I can exploit this other better I found in here, right?
• 35:46
  It's a Sorry depression or absolutely.
• 35:48
  Yeah. I mean I think what I'm learning
• 35:49
  and what I'm going on thinking now is you know this
• 35:52
  experimentation platforms are an inspiration, you know, to other,
• 35:55
  you know I'm a lapsed platforms and how, you know, how those
• 35:59
  have developed very similar in some respects, you know, to the
• 36:02
  workflows you want to enable the I'm wondering about the when it
• 36:06
  comes to this experimentation platform.
• 36:09
  And it seems like. So I have kind of have two
• 36:12
  questions here for you. One is the experimentation
• 36:15
  platform. Is there a All platform team
• 36:17
  that has to keep that intact and and was that a large undertaking
• 36:23
  to create that. And then the other question that
• 36:26
  I have is when it's not something like putting a new
• 36:29
  model into play and it's more something like we need a new
• 36:34
  tool and that is the experiment that we want to experiment on.
• 36:38
  How does that work? Because I imagine it doesn't go
• 36:41
  through the experimentation platform.
• 36:43
  Well, you could be surprised but that very often doesn't but
• 36:46
  sometimes we try Why not the you ask yourself questions in there?
• 36:49
  I mean the first one I think is very generic about any companies
• 36:52
  with engineering, right? So you have to make investment
• 36:56
  to make your infrastructure better.
• 36:58
  If you have a use case, that is generic, then it's easier to
• 37:00
  make the investment because many people can profit from it.
• 37:04
  And in the case of experimentation, I think that's
• 37:06
  usually the case. As long as your company is a
• 37:08
  scaling of experimentation, is such a powerful tool for
• 37:10
  everyone that investment is not registry 5.
• 37:13
  If you see it currently looking for most of the team that is
• 37:16
  almost Free to make an experiment because the course is
• 37:19
  divided across the whole organization and that is thing
• 37:22
  you could say about machine learning certain rights to also
• 37:24
  have much in their name infrastructure team, that
• 37:27
  provides serving and monitoring and feature management Etc is
• 37:30
  the same idea. The investment is too high for a
• 37:32
  single model but once you manage to serve the needs of the whole
• 37:35
  company, it actually becomes physical investment for another
• 37:39
  company. That is, of course a lot of
• 37:41
  providers also, right? They made a big investment and
• 37:43
  you're just used to because very little more for the service but
• 37:45
  you don't have to evaluate. Your own capability.
• 37:47
  Hold for ML. Serving formal moratorium called
• 37:50
  for experimentation to your second question, I think was
• 37:55
  about to hypothesis testing on the Italian scooter in the
• 37:57
  company. So it's not always things aren't
• 38:00
  as they throw an experiment, not always tell us who are
• 38:02
  randomized control trial, but we do try to hold us accountable to
• 38:06
  some kind of hypothesis testing process.
• 38:07
  Sometimes you really cannot write.
• 38:09
  Your people just want to us as you can because of tooling.
• 38:12
  You basically do the prophecies and you acknowledge that.
• 38:14
  In some cases you will not be able to measure it, you know.
• 38:16
  Cases, you follow. Normally, I go there and work on
• 38:18
  a project management techniques like miles and okay.
• 38:20
  Jobs, Etc. At least Trace where you're
• 38:23
  achieving your goals? But I would say in some cases
• 38:25
  you really cannot measure the impact if possible, you will do
• 38:28
  it. For instance, infrastructure
• 38:29
  will very often. If we still have the first group
• 38:32
  of life, we just asked them both and you can an experiment that
• 38:35
  was in the back end. So if you have them all running
• 38:37
  on our main engine and you have more running independently.
• 38:40
  What we serve all of them and you make a performance
• 38:42
  experiment where you can assess stuff like loading time of the
• 38:44
  page cause for company. World clock time actual server.
• 38:48
  Time, Etc. You see can experiment on those,
• 38:50
  if possible, of course, we're not the menu, you have to make a
• 38:53
  bit of a leap of faith. In those cases, last thing that
• 38:56
  I wanted to ask before we jump into deployment and serving is
• 39:01
  around what metrics you're looking at you touched on it a
• 39:06
  little bit. But I know this is a question
• 39:07
  that comes up a lot in the in the labs community and its
• 39:11
  really asking about what metric should I be monitoring.
• 39:16
  And how you can relate those back to the business value?
• 39:22
  Yeah that is the different layers in there.
• 39:27
  So on one hand we have our business or business goals for
• 39:30
  our business unit goal, which could be monetary, could be
• 39:34
  conversion, could be used to satisfaction, could be
• 39:36
  operational dry like workload for our CSS, agents, and revenue
• 39:40
  for our partners, Etc. So you have this kind of metrics
• 39:43
  possibly injuring. All right, so ideally you want
• 39:44
  to do something. You are either measuring that.
• 39:46
  At or you are measuring proxies for that or things that, you
• 39:48
  know, that are going to alert you when that is is going to
• 39:52
  write right? Whenever possible, we do that.
• 39:54
  That's normally only possible experimentation setups for most
• 39:56
  of the of cases, this very clear connection because the the path
• 40:00
  between the ml model and this business value is quite Glory.
• 40:04
  What you can do is you put in an experiment and experiment.
• 40:06
  If you have a gold from the mines control trial, if you have
• 40:08
  a another hypothesis testing setup, it kind of assumes that
• 40:12
  whatever the past we are going to see the an effect on this
• 40:14
  metric, right? That's that's a one day.
• 40:17
  What y'all was not possible yourself looking for proxy.
• 40:19
  So if my hypothesis was that I was going to affect this user
• 40:21
  Behavior by being able to run better, okay?
• 40:23
  Can at least try some ranking better if I know that's going
• 40:25
  down then I know that I have a problem.
• 40:28
  The helmet is a dimension now for instance is responding
• 40:30
  solution shark or the reef is also a way of a proxy to put the
• 40:34
  red flag. So I know that I tested is an
• 40:36
  experiment in a certain situation.
• 40:37
  Can I use brief measurement and monitoring to know what?
• 40:41
  When my current model is different from that one that I
• 40:44
  tested the time with the business metric.
• 40:45
  So if you think, If you're too much, maybe it's time to go and
• 40:48
  check it again, because I cannot guarantee that they were in the
• 40:51
  same situation as when I made my initial test and then just run
• 40:55
  into a problem and there are things that are very obvious
• 40:57
  proxies. For instance, the delay between
• 40:59
  request and response, you know, if you are doing something that
• 41:01
  is serving front end, that will have an impact.
• 41:03
  But again, you might have data allow you to connect it better.
• 41:06
  So we have make experiments in which we impose delay.
• 41:09
  The responses understand window, having an impact and that can
• 41:12
  tell you what to look for in the left.
• 41:14
  So, we know the delay of, I don't know where officer, Just
• 41:16
  leave an example. Is that coming up with a number
• 41:18
  but you can experiment. Realizable press.
• 41:20
  This feature quarter of a second, have an issue, half a
• 41:22
  second is an issue, then I know what kind of stressful and
• 41:25
  unless I have to put on my, on my back and to monitor it,
• 41:28
  right? Yeah, I think I'm a very happy
• 41:33
  camper right now, because I think we have walked through
• 41:36
  monitoring. We've talked about evaluation,
• 41:38
  we've talked about the problem design.
• 41:41
  We talked about, you know, offline model performance and
• 41:45
  and how that That is just a health check as the paper said.
• 41:49
  So we've gotten through a lot of the lessons in the paper and
• 41:51
  there's one that stands out and that's a deployment and
• 41:56
  deployment. Yep.
• 41:57
  Yep. Deployment time is money and I
• 41:58
  think everybody listening would absolutely agree that they feel
• 42:02
  that way about model deployment. One of the this is It's a topic
• 42:08
  that comes up a lot in the community.
• 42:10
  How do you deploy models? How do you do it repeatably?
• 42:12
  How do you do it? Reliably, you know, that
• 42:15
  process. It's also It's really, you know,
• 42:18
  perhaps the most nebulous component of putting a model
• 42:24
  into production because you know what does deployment really
• 42:27
  mean? Well, it's putting it in front
• 42:28
  of users and, you know, everybody's users a little bit
• 42:31
  different. And so, you know, my question to
• 42:34
  you in terms of You know, just discussing this deployment
• 42:38
  topic. How would you summarize the
• 42:42
  lessons? Learn about model deployment at
• 42:45
  booking.com. You know, I know it's a big
• 42:47
  question but maybe we can just start with a few that come to
• 42:49
  mind and it, again, if you don't have time, we have a Blog where
• 42:54
  we publish more muscle power, ML and IA content is a booking .e,
• 42:59
  artificial intelligence building.
• 43:00
  That you will find a publication from Lucas Bernardi where he
• 43:04
  explains how we do our Ma Engine rml deployment infrastructure.
• 43:10
  The concept in general is is evidence for I mentioned before,
• 43:12
  radio start making mode within your team.
• 43:14
  You hack something together with are available for one specific
• 43:17
  use case. But once you realize that you
• 43:18
  can make a better system and you scale to more themes that allows
• 43:22
  you to invest more on it and that's what happened in Booking,
• 43:24
  over the last year's we started doing this kind of localized
• 43:26
  deployment and realized, there were a lot of common use cases,
• 43:29
  could be served by a common infrastructure, which means that
• 43:32
  you can more efficiently, build it more efficient to monitor it
• 43:34
  and then develop it further. Because now, your business, Case
• 43:37
  grows from one specific module specific team to serve in much
• 43:41
  broader and it and we've been working on the other way.
• 43:43
  For several years, I will have a platform that we use for
• 43:46
  deploying, a big part of our of our models.
• 43:49
  The, this one scene that the Lucas mentioned in the, in the
• 43:52
  publication, which I really like how he faces in there, which is
• 43:55
  that we translated one of our company's core values, which is
• 43:59
  respect for diversity. We translated also endure, any
• 44:02
  plasma. So what we try to do in there is
• 44:04
  that we try to provide users tools, Allows them to be diverse
• 44:08
  on how we approach, mental problems and some companies
• 44:10
  could maybe focus in one specific synthetic.
• 44:12
  I want to use tensorflow. My holy prom is going to be
• 44:15
  focused on tensorflow or in something different, right?
• 44:17
  We try to basically predict eyes our development by how many use
• 44:21
  cases can we serve especially how many different use cases and
• 44:24
  you can have very concrete outcomes actions from it?
• 44:29
  For instance, one of the 15 that we produce can eyes was look up
• 44:31
  tables. It's a very, very simple way to
• 44:34
  put something in production. You literally pre-compute every
• 44:36
  Well, combination from your model and put it into a standard
• 44:39
  table, where people can query. This is not the most sexy thing
• 44:42
  is not super smart, but it's super gentle.
• 44:44
  You can have anymore you like, you can use any programming
• 44:46
  language. You just tell me what I have to
• 44:47
  say in every location. I put it on a table and you look
• 44:50
  it up, right? So as a very concrete action
• 44:51
  that comes from this idea of, I want to support diversity with
• 44:55
  my ml Community. Now, you said, what you see
• 44:57
  Prince and linear models. I mean, the normal sounds very
• 44:59
  simple, but you can support a very broad range of things.
• 45:02
  An amazing animals you can do one layer neural network, linear
• 45:05
  regression and logistic, Crashing a naive based model
• 45:09
  there is as broad set of things. You can do you like having a
• 45:12
  linear engine. So you make the block and then
• 45:14
  your cell phone, if you put tensorflow, can do you have all
• 45:16
  of the setups of the tensor flow allows?
• 45:18
  If you could H20 case to has a lot of differences that you can
• 45:21
  support in there, right? That's like synthesizer with the
• 45:23
  philosophy that has been followed in there, try to set up
• 45:27
  as many use cases. And of course, that also
• 45:28
  resonate with, with business, because the most use cases, you
• 45:31
  can support more return, you will get in your investment that
• 45:34
  for you can make a better investment.
• 45:37
  It is a brilliant approach. I love that story about look up
• 45:40
  tables. And, you know, I think it's,
• 45:42
  it's instructive because I think a lot of companies I know I have
• 45:46
  this mindset sometimes nowadays. It's, you know, what's in Vogue,
• 45:50
  you know, what is popular in terms of the deployment
• 45:53
  solution. Let me adopt those sort of
• 45:56
  strict Frameworks rather than thinking again from the
• 45:59
  perspective of what's the real problem.
• 46:01
  You know, I think we're actually going to remodel development
• 46:04
  challenge right now and it's like, you know, the Trinidad is
• 46:07
  like, how specifically do you engineer it and use all the
• 46:10
  tools that are available versus how generally do you want it to
• 46:13
  work? You know what?
• 46:13
  It takes, patience. If you want to go General,
• 46:15
  because you need to take a long-term view towards all the
• 46:18
  other use cases, you want to support, you know, it's harder
• 46:20
  to see that immediate return on investments think that's the
• 46:23
  challenge that I've been having. And you know, in the paper you
• 46:28
  mentioned a really interesting, sort of learning about how
• 46:32
  latency became a challenge one. Thing that we love in the
• 46:39
  community and loved hearing on podcast.
• 46:40
  So War Stories and just stories about how you got to a
• 46:43
  particular point of learning or Insight, I'd love to know.
• 46:46
  Can you tell us the story of a model where you realize that
• 46:49
  latency was becoming a challenge when it came to business
• 46:52
  performance? I don't think I mean I cannot
• 46:57
  recall a specific one right now but I can tell you're worried
• 46:59
  because it's happened many many times, so I cannot put you a
• 47:02
  specific example but what I can tell you is how we realize that
• 47:05
  it was a challenge and it goes back to us.
• 47:07
  And those who have been talking, right?
• 47:09
  We've already thrown experimentation platform and
• 47:11
  that allows to make very fancy experiments.
• 47:13
  So in one set of the we normally use but we wish that we had over
• 47:17
  based Behavior. We have a certain moral serving
• 47:19
  a certain function and we have competing model.
• 47:22
  And what we can do is we can try to isolate different So we can
• 47:25
  for infinite slow down the corn oil until they all answers.
• 47:29
  So you keep the same moral, but you usually like of the second
• 47:32
  one, or you can put the same delay on both chuckle, both
• 47:34
  models and none of the to can answer until both of them have
• 47:36
  answer. So if you put that says, do you
• 47:38
  see as when you wait for what models, you're only comparing
• 47:40
  their label the prediction, right?
• 47:43
  Because they both have the same delay.
• 47:45
  Well, if you take the first model and relay it to the other
• 47:47
  one, you're only testing the effect of the relay.
• 47:49
  So it's kind of experiment allows on very many occasions to
• 47:52
  isolate the impact of the Improvement of the production
• 47:55
  quality or whatever the mall is trying to shift from the
• 47:57
  performance. And you know, I cannot give you
• 47:59
  a brief example. We really saw it many times that
• 48:02
  you could have fun improve performance on the pole output
• 48:05
  and improve user experience. Once you put up both of them at
• 48:08
  the same performance level, which is normally no lower.
• 48:12
  Well, if you take the moral, there was a performance in terms
• 48:15
  of linear regression is not only after performing quite fast, you
• 48:18
  might still get a better result with the user.
• 48:20
  Even though the outcome is not that good.
• 48:22
  I mean, you can imagine the this kind of storage media.
• 48:24
  You go In our simple linear regression and you're moving to
• 48:27
  something that is a very fancy never and I were set up, it will
• 48:29
  be slow but you probably will go to are gonna get a much better
• 48:32
  outcome. It doesn't mean that you don't
• 48:34
  do it, but isolating the different components allows you
• 48:37
  to make the investment. So so, oh my video.
• 48:39
  Can try to look for a better way to make the neuronal.
• 48:40
  Can I deploy it in a faster? Way?
• 48:42
  Can use better hard work because now I can isolate the effect.
• 48:45
  Can I pre compute it? That was a bit of the idea also,
• 48:48
  with, with a look-up tables. Maybe I say, okay, this model is
• 48:51
  super slow, but actually, there are not so many features is
• 48:54
  million combinations. It's putting a Cassandra table
• 48:56
  and you can get it out of there, right?
• 48:58
  So at the risk of this one needing to be cut out of the
• 49:04
  episode. I got to ask any of this stuff
• 49:07
  that you've been building at booking.
• 49:09
  Is there any thought of open sourcing it?
• 49:12
  Yeah, so we can have that on open source program.
• 49:15
  Let's say, we have to set a set of components in the listing in
• 49:19
  the MLS side. We have an open source at the
• 49:20
  motion, the developers age especially where one of the
• 49:23
  biggest Parts shops in the world right now.
• 49:25
  And we are pretty maintainer of the, of the VIP packages per
• 49:27
  women. They know, some of the versions
• 49:29
  of her, in the case of a male. We also have some component that
• 49:31
  we have tried to open source. I think we have done in a couple
• 49:35
  of cases, but in many companies that you have to make sure that
• 49:38
  I think the main policy behind it, it won't go into open
• 49:40
  source, something in the name of booking have to make sure to
• 49:41
  maintain, we just we don't want to go broke and cold outside,
• 49:44
  but it's going to fail. We know the mundane.
• 49:46
  And I think that the main limitation we have to make sure
• 49:48
  that we are open sourcing, something that you are willing
• 49:51
  and able to maintain enough. There is a belief that were
• 49:54
  willing to put out When I'm behind ya otherwise you shoot
• 49:56
  yourself in the foot and you it doesn't give a good look.
• 50:01
  So playing on that question. I guess.
• 50:06
  The next question I have is, do you feel like there is when it
• 50:13
  comes to the platform that you're using at booking?
• 50:16
  Is there enough of what you're doing with the ml side and the
• 50:21
  platform? And maybe this experimentation
• 50:23
  platform or just the sir Platform.
• 50:25
  If these things were to be open sourced, do you feel like right
• 50:30
  now in the machine learning world?
• 50:32
  There's enough standards around for it to be used by others, or
• 50:36
  would it be a heavy lift? Like I keep coming back to this
• 50:41
  because a lot of people have been talking about whether or
• 50:46
  not there are enough standards or whether or not tools can plug
• 50:51
  and play with each other. Well enough.
• 50:53
  Right now because Because the the ecosystem is so young and it
• 51:00
  needs more. Maturing, do you feel like there
• 51:04
  is enough of these standards or do you think that it would need
• 51:09
  to be more like, like, would people be able to get value out
• 51:13
  of it? If that makes any sense?
• 51:15
  What I'm saying? Specifically on the platform
• 51:23
  side but what I can see is that there is some companies out
• 51:25
  there who are open sourcing things and comments were selling
• 51:28
  this kind of platforms. And in the case of organization,
• 51:30
  with our focus on diversity, we use many of these platforms
• 51:33
  already. So I, if I see this, I user J
• 51:35
  has were getting value from the community of both open sources
• 51:39
  and Commercial Ventures are delivering this.
• 51:42
  So, it seems that have some value, they do operate more or
• 51:45
  less independently many of them, right?
• 51:46
  If you're worried 2nd planning area, there are some
• 51:48
  connections. If you see things like, Pythor
• 51:51
  tensorflow. They have some common concept
• 51:53
  that you may be able to to translate, but many of these
• 51:55
  things are with isolate the receipt, right?
• 51:58
  That doesn't mean you cannot use them.
• 51:59
  They are all using the same kind of based on Evolution
• 52:01
  coordinated. You can deploy them in your
• 52:03
  Cloud, right? When you go look in the
• 52:04
  direction, it's easier than Byron is.
• 52:06
  So so you are able to use components from different areas.
• 52:09
  I haven't seen that many examples of the things
• 52:11
  interoperating so easily, but again, I'm not in the in the
• 52:14
  platform side. So, all right.
• 52:17
  So as far as the paper goes lessons, that you have learned
• 52:22
  since anything that you would add to the paper.
• 52:25
  Now that it's been some time since it was written, so that
• 52:29
  he, so I just ended up maybe two main things.
• 52:32
  One thing I already mention of it which is the focus.
• 52:35
  The light focus on online learning so many things that
• 52:38
  we're doing at the time were testing.
• 52:40
  Prepare Amanda's are kind of limited on time and then we had
• 52:42
  monitoring as much as we could. Right now we're moving every
• 52:44
  morning to online learning because it connects these two
• 52:46
  words. It allows Continuously monitor
• 52:48
  the business value and you can see the feathers, you have
• 52:51
  online their names, you have continuously training.
• 52:53
  You have all of the different dimensions of all.
• 52:55
  Try to look for a way to keep your model up-to-date with a
• 52:58
  business value. Supposed to generate only one
• 53:01
  dimension. The other one is maintainability
• 53:03
  sounds very nice to have 150 successes but it's not so nice
• 53:06
  to maintain hundred and fifty things in parallel.
• 53:08
  And one thing that we have been a working on is trying to
• 53:12
  structure together some of the solutions or some point.
• 53:15
  You look at this okay, this stem programs that actually won
• 53:17
  power. And we decided to solve 10
• 53:18
  times. Can we somehow instructor a
• 53:19
  common solution and actually using submitting of 10-day
• 53:23
  limit? And so, we're taking some of
• 53:25
  these problems and it's rooted in the in a way that we can
• 53:27
  package it into a single online learning solution and you see
• 53:30
  copying a bit more in the company gives you more mundane
• 53:32
  ability. Better monitoring more
• 53:35
  scalability again because you United to join different
• 53:39
  business cases and you get a stronger business case which
• 53:41
  allows you to put more investment behind it, right?
• 53:43
  I think that's doesn't mean less than it actually.
• 53:44
  Connects these two things, right?
• 53:45
  Try to get a structure in. These exploration phase, we have
• 53:48
  with 250 successes and try to make them more into proposals.
• 53:52
  That are more common that are more applicable in wider areas,
• 53:56
  while online learning allows you to keep them.
• 53:58
  Well performing, I love the way that you think about that and
• 54:02
  how you want to basically generalize them enough to gain
• 54:08
  more value, out of a model and reuse it and and see the success
• 54:13
  in other areas that you may not have thought they could be
• 54:16
  successful in in the Genin, as far as the maintainability goes,
• 54:19
  and this was something that I was going to ask at the last
• 54:23
  question when you were talking about how you want to make sure
• 54:28
  that there's that diversity, that is available within all of
• 54:31
  the different Frameworks that people are using or whatever
• 54:35
  they choose to use. You want to support it?
• 54:38
  Is that not a really heavy lift is, does that not make it much
• 54:43
  harder to maintain? Sure, that it's nice question.
• 54:47
  Also, So, the mention on the platform raised want to
• 54:49
  maintain, as many use cases, we want to make sure that the
• 54:51
  system is stable. So again, if you go to example
• 54:54
  of the of the lookup table is a Cassandra database, right?
• 54:57
  So, as I think the service example, because it allows you
• 54:59
  the matching their name, practitioner to use, whatever
• 55:01
  modeling approach, they like, well, the deployment solution is
• 55:05
  a Cassandra table sensing for the linear model.
• 55:07
  If whatever modeling approach we're doing ends up in a table
• 55:10
  of linear coefficient, then I make a java-based linear
• 55:13
  coefficient engine that only has a lot and I can't afford it,
• 55:16
  whatever they are. Engineers, right?
• 55:18
  And then we just go in deeper if you have something like Five
• 55:20
  Torture tensorflow. They have their own runtime
• 55:23
  engines, so you can do whatever you like, in tensorflow,
• 55:25
  sterilize it. And then we maintain tensorflow
• 55:28
  engine. We do also have, for instance,
• 55:31
  some stripper owners for python scenes like this over very
• 55:34
  carefully, applying them in production because those are the
• 55:36
  ones where you have the least got raised right by this, to try
• 55:40
  to have something you can somehow serialize.
• 55:42
  And then you have a separate droning engine that is
• 55:44
  maintained. So it actually ties into the
• 55:46
  same logic by Like to support as many use cases, while keeping
• 55:49
  the stability of the platform. Yeah, I think what I point out
• 55:53
  there that I hear is if your interfaces are simple, you can
• 55:59
  solve that problem of maintainability versus
• 56:01
  generalizability, right, like the Cassandra lookup table.
• 56:05
  It's a simple interface for any kind of any kind of model or
• 56:09
  modeling workflow can fit into and I don't know.
• 56:13
  That's just, that's just kind of the takeaway that I have for it
• 56:16
  and I just want to thank you Pablo so much for joining us on
• 56:20
  this awesome, awesome coffee session.
• 56:22
  I feel very happy. We got through all of the
• 56:25
  lessons and we got two more. That's eight lessons from
• 56:28
  booking.com on how to serve. 150, machine learning models.
• 56:31
  Thank you so much for joining us.
• 56:32
  Thanks for inviting have a nice day.
• 56:35
  Alright, that's it for now and we'll hear from you guys and
• 56:38
  we'll be talking more. I'm a lapse on the next copy
• 56:41
  session. Thanks for listening.