Latest / Agentic Conversations (formally mlops.community) / MLOps at DoorDash // Hien Luu and DoorDash Leads // Coffee Sessions #119

Transcript

MLOps at DoorDash // Hien Luu and DoorDash Leads // Coffee Sessions #119 · Agentic Conversations (formally mlops.community)

• 0:00
  This is neelu working at door Taps and leading a new learning
• 0:04
  platform team. Been here about two years.
• 0:07
  I like my coffee. You would clean and no sugar.
• 0:11
  Yeah, I'm good. Now I've been out of the market.
• 0:15
  So beginning the model development platform team at 0-0
• 0:18
  for a few months now and a peanut do dash for about two
• 0:20
  years. And yeah, I like my coffee in
• 0:24
  all kinds of medium, Blends usually black sometimes, the
• 0:27
  little bit of Light, Green Light.
• 0:29
  They Group. I'm an engineer in the machine
• 0:32
  learning platform team. I usually take a latte and see,
• 0:36
  hey of Cornell. I'm also engineering, the
• 0:38
  machine learning class. I like, the coffee black.
• 0:42
  With a little bit of almond milk as an extra.
• 0:46
  Hi, my name is our vows. I joined the human faculty about
• 0:52
  two years back. I lead the mall selling platform
• 0:56
  and I'm the only one who doesn't drink coffee here.
• 0:59
  But when I do, I And get cold. Super excited to have not one,
• 1:07
  not two. But basically the whole team
• 1:09
  it's like Christmas came early for me, I get to talk to
• 1:13
  doordash. I'm here with Skyler, who is the
• 1:16
  trustee co-host of this ml, Ops Community podcast, our coffee
• 1:22
  sessions. And we've got so much to break
• 1:24
  into the I have seen and I think like, many others who are
• 1:28
  probably listening to this. They've read the doordash posts.
• 1:32
  We've seen you guys talking in various conferences.
• 1:36
  You give so much information. You're so transparent about what
• 1:39
  you're doing with the machine learning platform at doordash.
• 1:43
  I know that when it comes to Door dashes business you kind of
• 1:48
  have to be leveraging machine learning and it is such a huge
• 1:53
  piece of the business that it is critical and so maybe we should
• 1:58
  just start With the background and hand, maybe I can go to you
• 2:03
  first on. What exactly are you doing for
• 2:06
  the platform? How does it look?
• 2:08
  Can you give us the overview? Like that, 10,000-foot overview
• 2:12
  and then we can dig into some of the specifics?
• 2:15
  Absolutely. Yeah, let you mentioned, machine
• 2:18
  learning is place. A big Corridor Dash in town.
• 2:20
  So what we do on a daily basis, it Powers many of the core
• 2:25
  infrastructure is that help delivering food and go straight
• 2:29
  to our Customers so to kind of use cases is diverse and that's
• 2:34
  one of the things I love working at door Dash, including like,
• 2:38
  you know, our core infrastructure is about
• 2:40
  logistics by delivering taking orders and then assigning orders
• 2:45
  disasters and then help them deliver those voters to.
• 2:49
  You know, standard use cases like search recommendations also
• 2:54
  adds and promotions to forecasting.
• 2:57
  For example, doesn't have a big part of how we Ml at doordash,
• 3:02
  and other kinds of use cases around fraud.
• 3:05
  Obviously, we have users on a platform.
• 3:07
  So that's a big part about it when she learned it.
• 3:10
  Well, yeah, I would love to hear others to jump in and show their
• 3:16
  process. While on this question, I think
• 3:19
  ETA is another big use case. Yes, the number that you see
• 3:25
  like how many minutes away if you order from this restaurant,
• 3:28
  that takes a lot. A lot of machine learning to get
• 3:30
  it right. Absolutely.
• 3:32
  So these are fun challenges for sure.
• 3:35
  And what exactly does the platform look like?
• 3:41
  What are the tools that you all chose to use?
• 3:43
  What's homegrown, what is stuff that you've got from open
• 3:47
  source? And our, did you pay for
• 3:50
  anything and maybe a little bit of of the design decisions?
• 3:54
  When, because I mean, it's funny enough.
• 3:57
  While I Was preparing for this, I saw the YC Interview from the
• 4:03
  founders of doordash and I think it was like, why see, 2013 or
• 4:07
  2012 batch. And so machine learning wasn't
• 4:12
  that predominant back in those days, but you all have been
• 4:14
  doing machine learning for a long time.
• 4:17
  Does that play into the choices that you've made?
• 4:21
  So, yeah. Hidden.
• 4:23
  What does it look like? Yeah, I'll start with the high
• 4:27
  level and then each of the lead, Can talk a little more
• 4:30
  specifically on each of the area.
• 4:32
  So the platform we can go from your featuring tomorrow training
• 4:37
  to model serving and for castings or add our
• 4:42
  highest-level. The machine learning platform
• 4:44
  that we aspire to do is like a world-class platform that covers
• 4:50
  n 2. N in terms of solving the apply
• 4:53
  machine learning with wood ash and at scale as well, that's
• 4:56
  another very interesting challenge that we have to deal
• 4:58
  with. You know, it spans from stuff at
• 5:03
  the beginning. My how are you fijians ringing
• 5:05
  all way to model serving and Mauser abilities in terms of
• 5:10
  like how we think about building it kind of cover little bit on
• 5:14
  the blog that I share some of the principles that we use like
• 5:19
  dream big start small one percent, better customer
• 5:22
  exceptions. All platform consists of a mix
• 5:25
  bag of commercial vendor Solutions, open source as well
• 5:30
  as at home Homegrown Solutions. And maybe you can all can jump
• 5:35
  in and maybe talk a little bit about how we do feature injury.
• 5:39
  Yeah, thanks Keen. So yeah, just question was a big
• 5:44
  loaded once. I'll try not to go too deep into
• 5:47
  the weeds just yet, but yeah, the Journey of an ml or data
• 5:52
  scientist start to start with the feature Engineering Process.
• 5:55
  And so we spend a lot of time trying to improve the ecosystem
• 5:59
  around how quickly they can build and deploy features, both
• 6:02
  for offline and online access. And so your go all the way from
• 6:07
  the users point. And with features within their
• 6:11
  own notebooks deploy those to production and again that kind
• 6:15
  of goes into warehousing which again we have a mix of both open
• 6:19
  source and Enterprise solutions that help the warehousing, trust
• 6:23
  the process. And then we go down to the
• 6:25
  compute which is then what does it take for a data scientist to
• 6:28
  build those features quickly and using a standard paved path that
• 6:32
  lets them, you know, stop thinking about, what's the way
• 6:35
  to make this really fast, but other think about what's the
• 6:37
  business logic that goes into my future.
• 6:39
  Hours. And then once we get through
• 6:42
  those process, then we kind of go towards production axis.
• 6:45
  Like it's not enough to just build features you now want to
• 6:47
  start thinking about how to serve them.
• 6:49
  And again, we've kind of surveyed a few open source
• 6:51
  Solutions over time and I think we have our boss here.
• 6:54
  Who can touch upon it. But we have a really scalable
• 6:57
  feature store that is capable of serving, a really large volume
• 7:01
  of features, and production going through somewhere around a
• 7:04
  few billion updates daily. Nothing more recently as 50
• 7:07
  billion updates a day. But again we began with reddish
• 7:10
  or elasticache and then we've been expanding that over time.
• 7:14
  That kind of goes into serving features and then lastly, one
• 7:18
  more piece I'll touch upon is that.
• 7:19
  Once you have the features ready, you have to start trying
• 7:21
  to think about building the datasets for training and then
• 7:24
  actually doing the training. And once we can in that process,
• 7:28
  like that's where we work. The data scientists was hard to
• 7:30
  build standardized models that can be served in production.
• 7:35
  And yeah, with that, I will hand it off or pause for the serving
• 7:38
  size. I'd yeah.
• 7:40
  Thanks to null fear serving side was actually one of the very
• 7:44
  first components that we built out in the machine learning
• 7:47
  platform. It was the face of production
• 7:49
  machine learning. It required us to do a large
• 7:52
  amount of computations in a small amount of time and do it
• 7:55
  reliably. So, you're talking about like
• 7:57
  four nines of latency, four nines of availability, the
• 8:02
  latency being somewhere in the order of one to less than one
• 8:07
  millisecond per prediction. And like hundred billion
• 8:10
  predictions a day. That's that's the kind of scale
• 8:14
  and we had to do it all within the confines of the constraints
• 8:18
  of the micro service architecture are do Dash because
• 8:23
  that's when all of the infrastructure level best
• 8:27
  practices optimizations come for free wanted to stay within that.
• 8:31
  So that was the challenge. We had been dealing with because
• 8:36
  of this, everything was built in-house for the Serving part
• 8:40
  the whole microservice and it's other Integrations.
• 8:45
  They're the only vendor thought we might have used at that time,
• 8:49
  was for streaming the responses, back to snowflake or a data
• 8:55
  warehouse for real-time analysis of predictions, which also,
• 9:01
  because of scale. Now we have switched to in-house
• 9:05
  Kafka solution. Yeah.
• 9:08
  And then the fist To store was a big part of us scaling up that
• 9:13
  used to be a bottleneck in our performance latency.
• 9:15
  It still is think 80% of our latency comes from talking to
• 9:20
  feature store so that has been a prime area of focus that have
• 9:24
  been multiple iterations. As Kunal mentioned, we did
• 9:27
  starting from Lost Akash moving to cockroach DB and then doing
• 9:30
  the kind of research as to what are the formats that will work
• 9:34
  best for us. Yeah, and I think it From
• 9:39
  feature, engineering the other part that integrates with is the
• 9:43
  model repository which is populated by or model
• 9:48
  development team. So Cornell here can talk about
• 9:51
  our model development framework. Yeah, so once we have the
• 9:55
  features, we want to be able to train the models and the adopted
• 10:00
  a lot of ml Ops principles there.
• 10:02
  So we want to make sure our code is reproducible and we can go
• 10:06
  back to the original source code.
• 10:09
  How the model was strained, make easier adjustment.
• 10:12
  So we adopted as principal of committing code to GitHub,
• 10:17
  repository git repository for later use and it could gets
• 10:23
  reviewed. So we have high confidence in
• 10:26
  what other our models are doing. We've also ventured into
• 10:32
  retraining models, periodically that that's going to help the
• 10:36
  models stay fresh as well. Well, and the other thing I
• 10:42
  would like to mention is once we have launched a models in
• 10:46
  production, we want to be able to understand how they are doing
• 10:50
  and so we've invested a lot into model interpretability and model
• 10:56
  monitoring as well which are really important for like our
• 11:00
  business. We want to make sure models are
• 11:04
  continuously making predictions that are high quality.
• 11:09
  And so, we like measure typical metrics a, you see log loss and
• 11:14
  so on. And the one thing that ties all
• 11:17
  of this together is we have the vision for an internal web page.
• 11:22
  We call it m, El Portal that allows us to launch models.
• 11:27
  Look at the model report, the feature repo, look at the model,
• 11:30
  interpretability and so, yeah. So our vision is to democratize
• 11:35
  ml at doordash and be able to provide this as a service.
• 11:39
  Service to anyone just by navigating through this is web
• 11:45
  UI and so maybe slope can talk about more about some of the
• 11:51
  other applications as well. Yeah, I can talk about
• 11:54
  forecasting and update of ml portal as well, which the
• 12:00
  engineers I work with. We focus on so Porto.
• 12:04
  Casting, we are using it from business planning, to internal
• 12:08
  use cases. Is like variance reduction in
• 12:10
  our experiments product experiments.
• 12:13
  So ideally would like to have product experiments, run quickly
• 12:18
  and forecasting helps speed that up.
• 12:22
  So forecasting is a slightly different Beast, it's offline
• 12:26
  training and offline predictions.
• 12:28
  And this is the kind of use case where we are building on top of
• 12:31
  the training and serving platforms that are past, Cornell
• 12:34
  and Kunal and team are building. So in this case, we use Python
• 12:41
  Daxter for orchestration database, cluster is Snowflake
• 12:45
  and we use retool for building internal admin you eyes and for
• 12:51
  the ml portal as Corner mentioned, we are trying to
• 12:54
  build it as the internal home page for data scientists like
• 12:57
  anything you need to get done on the ml portal.
• 12:59
  You can jump onto ml Porter, you know, think of two things
• 13:04
  observability. So what's happening with my
• 13:06
  modeling production? What happening with the feature?
• 13:09
  Others, can I test out a new model that I just created?
• 13:14
  And even deploying models goes through ourselves.
• 13:17
  A workflow that are bars and team.
• 13:18
  But so these are the kinds of things that we are looking into
• 13:22
  as well. That doesn't include things like
• 13:25
  experimentation. Take like one one piece that
• 13:28
  felt like an open loop. In my head was like how we
• 13:30
  experimenting and testing new models.
• 13:32
  You be speak to that a little bit.
• 13:34
  Yeah, that's a great question and actually an active project
• 13:37
  that we are taking on. So, Our bodies can add to this
• 13:41
  as well, but we have an internal experimentation platform called
• 13:45
  Theory and we are currently working on integrating the
• 13:48
  experimentation platform into our ml platforms.
• 13:51
  So that data scientist can experiment on new models without
• 13:54
  having to approach. The product engineering herb has
• 13:58
  anything. You want to add to that.
• 14:00
  So, yeah, I think the experimentation was all left up
• 14:03
  to the data scientist. So you've mentioned product
• 14:07
  engineering at to work with all these Data scientist and that
• 14:10
  renewed of velocity of iteration on models.
• 14:14
  Are people had to, you know, sometimes set up all the
• 14:18
  experimentation framework from scratch and there was an
• 14:21
  application available to see visual visualizer experiment.
• 14:25
  See the results, and all of that but some work had to be done to
• 14:29
  set it up. So what are we are doing right
• 14:32
  now is provide that out of the box within the serving layer so
• 14:37
  people can just use the The existing application that we
• 14:40
  have for experimentation. And there's a lot of blog posts
• 14:43
  already on Kure, which is our experimentation platform and how
• 14:47
  its able to run large-scale experiments throughout
• 14:51
  throughout the day. So all of this is being tightly
• 14:56
  coupled with a serving layer, so that all day, he doesn't just
• 15:00
  have to do to set up a service station is to just click some
• 15:02
  buttons set of UI, and that's that's how they get better.
• 15:08
  At experimentation. Super cool.
• 15:09
  Thanks for sharing. Hey, I think this gave us like a
• 15:12
  really good overview until all the different steps and parts.
• 15:15
  One of the things I'm curious about in case, folks, didn't
• 15:18
  realize Haines and I work together at LinkedIn, a little
• 15:21
  bit, so kind of just want to roll the clock back.
• 15:24
  And just, you know, I think there was a lot of, definitely,
• 15:28
  a lot of mistakes that we made in like building a machine
• 15:30
  learning platform while the lessons learned.
• 15:31
  A lot of things we did, right? Just curious about, like, was
• 15:35
  there anything? You were able to take away from
• 15:37
  the experiences we had in building ml.
• 15:39
  Forms at LinkedIn and apply here at doordash anything.
• 15:43
  We learned like, oh absolutely, that doesn't work.
• 15:45
  We shouldn't do that and it kind of paid off the doordash love to
• 15:48
  hear about how experiences are transferring.
• 15:50
  I think my experience at the mln fraud at LinkedIn is you know,
• 15:54
  it has gone through that evolutions of There was no
• 16:00
  light, no Solutions out there. A lot of us have to come up with
• 16:03
  the almighty has in house. There's nothing to model after
• 16:06
  back then. Right.
• 16:08
  I think for us, we started our journey about two and a half
• 16:11
  years ago and has quite a bit of resources out there that we can
• 16:15
  best practices that we can Leverage.
• 16:21
  I would say the hylas the biggest thing is that I see is
• 16:26
  not so much about technology. It's a, how do we really engage
• 16:30
  with our data? Scientist Community to help
• 16:33
  shaping the direction to the platform?
• 16:36
  I didn't feel like that what is LinkedIn and that's why I think
• 16:40
  that I really try to do very differently when I got here at
• 16:44
  the White House, is how to really engage the customers as
• 16:47
  part of a joint. A building a platform, not just
• 16:49
  ask building it. We didn't get along with them.
• 16:52
  Yeah, I think that's a great answer and would love to dive
• 16:56
  deeper into it. Because I think, you know, it's
• 16:59
  such like a beautiful proach to like you really need to engage
• 17:03
  your customers. And I definitely concur with
• 17:05
  that assessment that like, I don't think that we got that
• 17:07
  quite right at LinkedIn especially early on, and
• 17:10
  definitely has a lot of hindsight is 20/20.
• 17:12
  You know, after like it after we learned lessons like okay, you
• 17:15
  know, but it very curious and I'm sure Sure listeners would
• 17:19
  love to know, especially people building their own and More in
• 17:21
  all platforms. Like, how do you like
• 17:24
  effectively engage with your customers?
• 17:26
  How do you like make sure that you kind of bring data
• 17:28
  scientists and and essentially build the platform with them.
• 17:33
  So what are some ways that you see that playing out and how you
• 17:37
  maybe make that started high level?
• 17:39
  And then at least can feel free to jump in as well.
• 17:42
  Especially the folks, most all these folks, by the way have
• 17:44
  been with it, or with the platform since the early days.
• 17:47
  So I'm sure they have Only your perspective as well.
• 17:54
  The veteran. So how do we do that in a
• 17:55
  doordash? I think in the early today what
• 17:59
  we established was we call something called machine
• 18:02
  learning Council Which consists of representative from the data
• 18:06
  stack this community and couple the leaders from the search team
• 18:11
  that are the using ml heavily, and we sat down together and a
• 18:16
  line on the directions. And we have constant
• 18:19
  Communication in terms of like bi-weekly, we sharing updates of
• 18:23
  what we doing and, and they can bring up like, challenges that
• 18:27
  that way that they facing. If we do some kind of like
• 18:32
  vendor evaluation, it's transparent everybody and they
• 18:35
  know what we're looking at. And they also request us to look
• 18:38
  at somebody than the solutions. So that's I think that that one
• 18:43
  piece really help aligning everybody on the same page in
• 18:47
  terms of what we doing. There's no we versus them kind
• 18:51
  of mentality. So all of us, right.
• 18:53
  How do we make this successful at doing as that's a group.
• 18:57
  And then I'll continue on that is about.
• 19:00
  We do planning together like quarterly, we ask them what they
• 19:04
  need and then we go through the prioritization so process and we
• 19:10
  attend the workshops. They provide workshops on how to
• 19:13
  how did we do the models, which and those westballz, as well as
• 19:16
  giving a presentation to the dish, as community of what's
• 19:21
  being built, what's new, what a new features.
• 19:23
  So this constant back and forth, communication involvement, And
• 19:27
  the last thing I'll mention is we invite those data scientist
• 19:30
  both in leaders to come and our team and have a lunch.
• 19:35
  Have a chat over lunch, it's just to get to know them
• 19:38
  especially some of the new folks, our pause here and if any
• 19:42
  leads want to jump in and that your two cents if you print as
• 19:46
  well add but like apart from everything in the convention
• 19:48
  like a few things that also kind of help us expand.
• 19:51
  The platform is that patients from other experiences I've had
• 19:55
  with them was not from school. Or we don't tend to build things
• 19:58
  because they are cool obvious. Be as folks that have played
• 20:02
  around with ML platforms for a while.
• 20:04
  Might have an idea where some value lies for our customers.
• 20:08
  But we kind of, you know, approach the conversations with
• 20:10
  them, in all the council's that he mentioned, the kind engage
• 20:13
  with them and see if this is of immediate value to them.
• 20:16
  And that's typically not of immediate value to many of the
• 20:19
  verticals, right? So we want to find a few
• 20:21
  verticals where this aligns and then use that as an impetus to
• 20:24
  design, right? Because it like, Over designing
• 20:27
  is a common problem that platforms go through where you
• 20:29
  might end up building something that isn't particularly useful
• 20:32
  and so we try to avoid that as well by building small things
• 20:36
  first and see if that's actually solves a problem in a big way.
• 20:39
  And then take that further along the way.
• 20:41
  And that kind of helps us not over expend resources and in the
• 20:46
  white and that's where like the like Hazard was mentioned the
• 20:48
  inference service skyrocketed first because that's where the
• 20:52
  biggest problems were and we use that to identify new ways and
• 20:56
  Lowly are pieces to our platform.
• 20:59
  Hey I'm Vishnu, I'm a data scientist at first hand and I
• 21:02
  definitely think that you should subscribe to the ml Ops coffee
• 21:06
  sessions podcast. It's the best podcast out there
• 21:09
  to stay on top of what ml Ops actually is to talk to the
• 21:12
  truth, thought leaders in the space.
• 21:14
  And oh, by the way, dimitrios is absolutely hilarious.
• 21:17
  What a weird guy, you should definitely subscribe to the
• 21:20
  podcast back to that. Once I think one thing that
• 21:24
  we've been doing successfully is We have strategy sessions within
• 21:29
  our team as well as combine that with the customer, baseless
• 21:32
  process. So what happens is we we kind of
• 21:35
  come up with where do we want to be a year from now and have some
• 21:39
  consensus within the thing? But that doesn't mean we just go
• 21:43
  straight away. Start shipping it, which kind of
• 21:45
  look at the request from our customers.
• 21:48
  Every quarter, that he and sends out a spreadsheet and all the DS
• 21:52
  Milton's kind of fill in what they think are things.
• 21:56
  They Need help with next quarter or things that are blocking them
• 21:59
  this quarter and then we kind of you know match them together and
• 22:03
  come up with what are the parts that we want to build?
• 22:06
  That is impactful and useful for the customers now but is also on
• 22:11
  the line of where we want to be a year from now.
• 22:14
  Yeah. Right.
• 22:15
  I think so. As I said the other cup of
• 22:18
  couple of components as well that black to Art so he ain't
• 22:21
  talked about like raising awareness amongst our customers
• 22:24
  and can order talked about Out. How do we choose a sweet spot of
• 22:29
  flexibility? Like offering just enough kind
• 22:32
  of customizations but not too much to make the platform look
• 22:36
  all over the place I think. And sort of course talking about
• 22:40
  hand-in-hand collaboration with leads of data scientist.
• 22:46
  The other thing that comes out in my mind is the velocity at
• 22:51
  which the data scientists can move.
• 22:52
  So how quickly can they on board?
• 22:55
  A new model how quickly can the right?
• 22:56
  New feature pipeline. How quickly can the experiment
• 23:01
  on the models? So that is like a prime value.
• 23:05
  Proposition that oppression platform can offer is like, if
• 23:10
  there are lot of love, a lot of friction along the way then it
• 23:15
  gives birth to the scenario where they can be multiple
• 23:17
  machine learning Platforms in the same company and that has
• 23:19
  been the case in some of my previous experiences where
• 23:22
  people have just build something of their own because they
• 23:24
  weren't they weren't able to like me.
• 23:26
  Peace with what was offered. And so yeah.
• 23:31
  The the value that the machine learning platform provides at
• 23:33
  and the velocity at which provides that value that comes
• 23:37
  to it. One one important aspect is how
• 23:40
  can we get our customers that they are scientists onboard it
• 23:44
  to New launches. So it's one thing that we
• 23:48
  launched. Something really cool and
• 23:49
  useful. It's another thing that people
• 23:51
  actually use it. So a lot of times we have to
• 23:55
  work together with our customers.
• 23:56
  Serves to make sure they get on-boarded a lot of new features
• 24:00
  require migrations and so we have to come up with a good
• 24:03
  migration strategy. Is it something where we
• 24:07
  distribute the migration? Is it something that we do
• 24:09
  centrally or is it something that we do in a hybrid way?
• 24:14
  That's typically challenge for these new launchers as well.
• 24:19
  This is so good. So first of all, I love Colonel.
• 24:24
  I think it's hilarious. That you you keep referring
• 24:26
  going to hang as your boss. It's like don't forget guys, the
• 24:29
  boss is on the call, so make sure we say everything
• 24:31
  perfectly. We got to button it up, make it
• 24:35
  all nice and sound sweet. So just out of curiosity though,
• 24:39
  while I'm thinking about everything that you all are
• 24:42
  doing, what are the teams like? Because you're all the leads on
• 24:45
  these different projects and initiatives.
• 24:47
  How many people do you guys have working with you for and like,
• 24:51
  what is the platform team look like as in terms of size?
• 24:54
  Yeah, we Start out small and and I've been evolve and the team
• 25:01
  size has increased at what time? So we have organized ourselves
• 25:07
  and into three sub teams. A common model development
• 25:12
  platform. Team.
• 25:13
  This is responsible for like feature engineering and model
• 25:16
  training and the right path of the feature store as well.
• 25:21
  And then we have the model serving team which is
• 25:24
  responsible for the online predictions of blind predictions
• 25:28
  and we path of the feature story.
• 25:32
  And then the third team, we call the machine learning
• 25:34
  applications, which is responsible for the forecast
• 25:37
  platform. That's what we mentioned, as
• 25:39
  well as other pieces infrastructure that are on top
• 25:43
  of the pillars that we build, for example, demo portal, which
• 25:47
  is a web-based application. That's group mention that
• 25:50
  chassis provide a unified experience for our data
• 25:54
  scientist but it leverages all the underlying infrastructure
• 25:57
  that the other teams provide I love that and I think it's a
• 26:01
  nice No, Nuance there that the read and write team for the
• 26:04
  feature store are different, or it's not combine to the same
• 26:08
  team, that's an interesting choice, and they don't work
• 26:14
  together because they, they end of day.
• 26:16
  So one single features are, but each decent-sized have their own
• 26:19
  requirements and needs, but they do need to work together for put
• 26:23
  a whole feature. Start to work and scale.
• 26:25
  Yeah, yeah, completely. And I want to dig into that
• 26:29
  internal tool that you've got in a minute.
• 26:31
  That that Each board that helps you track and have better
• 26:35
  visuals into what is going on. But it's also probably I guess
• 26:40
  by way of that we can talk a little bit about the metrics
• 26:43
  that you use for what you're looking at is as far as how, you
• 26:48
  know, data scientists are successful, there was mention of
• 26:51
  velocity and the ability to get something out, like, create a
• 26:56
  new feature or deploy a model. What other metrics?
• 26:58
  Are you all looking at? Wow.
• 27:02
  How will that Corral talk about is he's, he's our metrics guy or
• 27:08
  engineer Thunder. Team numbers guy, helpdesk
• 27:10
  cracking, and then everything. So could I tell you wanna share?
• 27:13
  Yeah yeah. So on the high level, we care
• 27:15
  about how our platform is doing. So we track things like number
• 27:19
  models, the availability of our online services, the the scale
• 27:26
  of the features that we create and upload things like How how
• 27:32
  much people use our internal ml portal and just General Health
• 27:38
  of the system, as well as scalability.
• 27:39
  So what is our cost per production across per delivery
• 27:44
  and things like that in terms of model metrics?
• 27:49
  We typically look at number of predictions the availability.
• 27:58
  So like the latency of those predictions.
• 28:02
  And then we look at qualitative metrics, as well as how that
• 28:07
  model is doing. We look at the law statistical
• 28:09
  information of the features so like the inputs to the models as
• 28:13
  well as the outputs which is the prediction.
• 28:15
  So we look at percentiles average number of zeros, things
• 28:22
  of that nature. There's another qualitative
• 28:25
  aspect which is the customer survey Colonel.
• 28:29
  You want to talk about that as well?
• 28:31
  Yeah. So Typically, I think once, once
• 28:36
  every six months, we send out a survey and to see how our
• 28:39
  platform is doing in the eyes of our customers.
• 28:42
  So we asked about like how satisfied are you with this
• 28:46
  service or that service component of the platform and so
• 28:51
  that doesn't really good. Good opportunity to reach out to
• 28:55
  a customer's on understand if there's anything that if there's
• 29:01
  A mismatch and and what their expectations are.
• 29:04
  We get a lot of interesting text feedback as well in that survey.
• 29:09
  So we engage with our customers this way as well.
• 29:15
  Yeah, incredible. So now I didn't really have a
• 29:20
  specific question on this internal tool the metrics but I
• 29:23
  just want to know more about it because it seems like something
• 29:27
  that if I understand correctly you have a visual into the Whole
• 29:32
  life cycle. I think, from what I've gathered
• 29:34
  the question seems to be around is their internal tool just
• 29:38
  making sure that the metrics are correct, or is it doing
• 29:40
  something more as well, right? Yeah, I do high level.
• 29:45
  We are trying to simplify things, right?
• 29:48
  It's it's basically of not store of velocity.
• 29:51
  So imagine the case of a data scientist has to look up the
• 29:58
  workflow, orchestration status and look Cup the metrics
• 30:04
  dashboard. Now, all of these are in
• 30:05
  separate places. So for our data scientists, we
• 30:08
  are seeing, you know, you want to look at this information, go
• 30:10
  to this URL, you want to look up this information, go to this
• 30:12
  URL, and then it's left to them to interpret what that what they
• 30:17
  are looking at. So that was the situation before
• 30:20
  the situation that we are trying to get to is you want to know
• 30:23
  about your model, go to your model home page and all the
• 30:27
  basic information you need is there, right?
• 30:30
  So, for example, Canal and you have been working on lineage.
• 30:34
  So like, what are for a given feature?
• 30:37
  Like, what is the Upstream data? Where is that located?
• 30:40
  Now, the basic information is in our model home page but if they
• 30:43
  want to dive more, if they have the links to jump to the to the
• 30:47
  full tool that we assist our teams are providing.
• 30:51
  So that is one way we are trying to simplify things for our data
• 30:53
  scientist. So if there is an alert, when we
• 30:55
  can show it in what homepage if there is things that we know are
• 31:00
  problematic, then we can highlight Like those and not,
• 31:04
  not always having to leave it to the data scientist to interpret
• 31:06
  what they're looking at. So that's kind of one way that
• 31:09
  we are trying to make this improve the velocity for data
• 31:12
  scientist and the other end the other way our workflows so
• 31:17
  Arbors. Can you talk about like,
• 31:19
  deploying models? What was the before and after?
• 31:22
  Right? Right.
• 31:23
  Yeah so I think like apart from seeing the current status.
• 31:27
  We also started to invest on building these workflows or
• 31:31
  Triggering these work flows from within this tool.
• 31:34
  So things like being able to train a model, being a able to
• 31:38
  deploy a model. So it has these dialog boxes
• 31:42
  that have all these various options that would otherwise
• 31:45
  require creating separate, you know, pull request to our
• 31:49
  repositories. Now, we data scientist can do it
• 31:52
  all within the UI, which is about, by following these
• 31:56
  workflows, which has helped tremendously and it has actually
• 31:59
  been the matter of preference. Hence, for the data scientist,
• 32:03
  we've seen that 70% of data center prefer working with the
• 32:07
  ui's versus the 30% who are actually power users now like
• 32:12
  play with the Json, so the low level formats but that's
• 32:15
  awesome. Yeah, another way the ml Purdue
• 32:19
  has been helpful is the number of support questions that we get
• 32:23
  in our slack channel has reduced as well.
• 32:25
  So that's another way we kind of in iterate, on or ml portal that
• 32:30
  if there is a question that keeps coming.
• 32:31
  Up, can we answer this in a so that, you know, the future
• 32:34
  questions don't pop up on the same lines and it's like, just
• 32:38
  to refer to add one more piece of context here.
• 32:41
  I think one thing that's changed in the life cycle of the
• 32:44
  platform is that as we've grown and a data scientist Community
• 32:48
  has grown, we've added a lot of tooling for ourselves, right?
• 32:50
  Like and we as a platform actually sit within a broader
• 32:54
  data organization, which has a lot of tools of its own and it's
• 32:57
  a lot of burden to think of data scientists to on board.
• 33:00
  Each new thing we build We'll add the do descale like you, you
• 33:04
  added Griffon, out of the mix. You are chronosphere to the mix
• 33:07
  as we add each tooling for ourselves.
• 33:10
  It's not accessible to a data scientist in the same way and so
• 33:13
  we are and it's still a continuing Vision, but the
• 33:16
  vision is to have this central place Kool-Aid, all information
• 33:20
  that they can use now or in the future without us expecting them
• 33:24
  to onboard to it. It makes our life easier as well
• 33:27
  when we think of bringing that information to them.
• 33:29
  Yeah, self-service huge for our platform.
• 33:31
  We want to make sure every step of the way their scientists can
• 33:38
  launch models and everything. We don't want to have to hold
• 33:41
  their hands and we want to make sure our team scales well with
• 33:45
  the rest of the organization. Awesome.
• 33:48
  In a, you may have mentioned. This is very explicitly but it
• 33:52
  sounds like using re tools that are you using that to build
• 33:54
  these kinds of like this app essentially?
• 33:59
  For users. So the way we use retool is more
• 34:01
  of a zero to one kind of a tool. So something that needs to be
• 34:05
  prototype of something needs to be built.
• 34:07
  That does not have a wide audience, we kind of use retool
• 34:11
  to build a first card and start testing it with users ml portal.
• 34:16
  Specifically is a regular python class create application?
• 34:22
  What for forecasting? We are used.
• 34:23
  We are building. The UI is on top of Fritos.
• 34:26
  So interesting. Cool to the other Jules.
• 34:29
  I think I heard you mention that uses data bricks and snowflakes.
• 34:33
  That is that correct? So, you know, it's interesting
• 34:37
  to hear that mostly because like in a lot of the data space is,
• 34:41
  there's always a lot of head-to-head of snowflake versus
• 34:43
  data bricks, so it's very interesting to hear that, like,
• 34:47
  yeah, yeah, it's interesting to hear that.
• 34:49
  Hey, we're using both and I think like, there's a lot of
• 34:52
  people in the community that, like, when they're thinking
• 34:54
  especially of, like, hey, we need a sort of data warehouse of
• 34:57
  some sort, they're thinking, It's snowflake versus data
• 35:00
  bricks. And so would love to just hear
• 35:02
  about how do you use these tools?
• 35:04
  But really together, you know, it kind of like educate the
• 35:07
  community that like, hey, it's not an either/or.
• 35:10
  Yeah, I guess staff and I started with snowflake as the
• 35:19
  data warehouse in early days. And continue to use that and
• 35:25
  then they need a way to potentially how to run
• 35:27
  computations on top of snowflake.
• 35:30
  And that's, that's why I'm data bricks was brought in and to
• 35:33
  help with some of the early data scientists work.
• 35:37
  And it has this values in terms of providing, you know, the
• 35:41
  noble service, the ability to launch computer compute at no at
• 35:46
  a click of a button. So those are the things that we
• 35:48
  really like about it, and we don't have to spend a lot of
• 35:50
  engineering effort and building those infrastructures So we have
• 35:55
  e the way we use data. Breaks has evolved a bit over
• 35:59
  the last couple years. We use their apis to launch
• 36:03
  someone to training job for example and other regions during
• 36:07
  compute jobs as well and it is time to just continue to use the
• 36:11
  noble service for their model. Accelerations So that's where we
• 36:17
  add. And that's how our journey was
• 36:19
  started of combining those two together.
• 36:22
  However, we going to move forward with using data Lake
• 36:25
  move forward. As we doing no complications
• 36:29
  during a scale, we need to go through a little leverage data
• 36:33
  link instead of snowflake without purpose.
• 36:38
  Maybe you want to ask him. Oh yeah, I like this in two
• 36:42
  parts. I think the journey is
• 36:44
  interesting because it's easy to think of these Technologies are
• 36:49
  synonymous but there is two parts which is the compute and
• 36:52
  the data, right? And so when we began with
• 36:54
  storage and warehousing Snowflake and was and still is
• 36:58
  one of our dominant data stores but compute doesn't necessarily
• 37:02
  need to follow the same speed and that's where database has
• 37:04
  helped our data scientist scale. The part of bringing data out
• 37:08
  and correct into their models in one plane of execution.
• 37:13
  And so for us when we began our journey, it was still fine to
• 37:16
  collate. These two worlds where we want
• 37:17
  to switch our data, scientists towards a faster storage layer.
• 37:21
  Once in a while, depending on the use case when it comes to
• 37:24
  arrays embeddings and so on, and that's kind of also where we
• 37:28
  stepped into thinking in more declarative sentences where,
• 37:32
  like, if you design a framework that can be agnostic for which
• 37:36
  storage we Evolved into n which computer you want to evolve into
• 37:39
  the data scientist experiences can stay the same.
• 37:42
  And so that's kind of where we've been heading into as a
• 37:44
  platform which is to try to change the conversation for the
• 37:48
  data scientists from. I want to use data breaks, I
• 37:51
  want to use snowflake to, I want to do X for my models or for my
• 37:55
  features. And I want to store this
• 37:57
  particular thing faster or at this much scale.
• 38:00
  And if you move to those questions, we can possibly help
• 38:03
  them make those choices better as we evolve our platforms to So
• 38:07
  those are kind of the two sides of our platforms Journey.
• 38:10
  Awesome. One of the questions that since
• 38:13
  Demetrius is Mike, is acting up that hitting me that it was
• 38:17
  curious about. You just kind of rolling a back
• 38:19
  a little bit talking about the you is that you have and sounds
• 38:24
  like most users are using that. But you have some power users
• 38:27
  and Demetrius was really curious about, like, what do you do to
• 38:30
  support power users? And how do you go about thinking
• 38:33
  about that? Yeah I think the the power users
• 38:38
  are like folks who just want to interact with our you know the
• 38:43
  raw data the GitHub repositories.
• 38:46
  And the reason why the prefer doing that is because it is more
• 38:51
  like what you see is what you get is like this.
• 38:55
  There's no magic happening behind a UI workflow and there's
• 38:59
  like some more integration that could have provided which are if
• 39:04
  you are does Things like, you know, Version Control and
• 39:09
  history, and all that, the other kind of things that if we were,
• 39:13
  if theoretically, we were able to build it, it into our UI,
• 39:16
  then it would satisfy all their needs.
• 39:20
  But yeah, this is something that you have to cope up with that.
• 39:23
  There are some people who would like to get into the weeds
• 39:26
  rather than just like work with the top level abstractions.
• 39:31
  And these are the people really who drive the next generation of
• 39:34
  features. In the platform, they are the
• 39:36
  ones who will come up with these wish list items that.
• 39:38
  Hey, how about customizing this particular feature to be able to
• 39:43
  also do this and they really Drive what a typical customer
• 39:50
  would look like in the next couple of quarters.
• 39:54
  So, yeah, he mentioned about the machine learning Council.
• 39:58
  We have a tech Steering group that involves, or these people,
• 40:02
  which we interact with on, like, That Cadence and try to
• 40:07
  understand what are their pain points?
• 40:09
  Couldn't pinpoint. What are their fixed darkened.
• 40:11
  Point of friction, friction. And then take it from there.
• 40:16
  Yeah, another example is like a canal added the ability for
• 40:21
  folks to contribute their own modules to the feature
• 40:24
  engineering. So people can contribute their
• 40:27
  python code that can be reused and shared across the rest of
• 40:30
  the data scientists as well. So you know, if they have
• 40:33
  specific needs then, They can contribute to the that module.
• 40:39
  Awesome because we have a few minutes left.
• 40:41
  And so one of the things I'd love to just calls us out with.
• 40:44
  Obviously there's you have like a lot of Rich experience in this
• 40:48
  space. Both from building the platform
• 40:50
  or doordash but also it sounds like most of you from previous
• 40:53
  experiences and what not and there's definitely an eager
• 40:56
  community that of people really trying to either break into the
• 40:59
  space. They're starting to build their
• 41:00
  platforms and whatnot. And so like, if you had to give
• 41:04
  an or One piece of advice or wisdom from your experiences of
• 41:08
  building machine learning platforms.
• 41:11
  What would it be? And let love to hear from each
• 41:13
  of you. Your take on that I can go
• 41:16
  first. I would say the biggest learning
• 41:18
  for me is how do you balance velocity and reliability?
• 41:23
  These are opposing forces but as I have it in my background I do
• 41:27
  Dash. We try to think of and not or
• 41:30
  so. How do you do both velocity and
• 41:32
  reliability is something that You should start again from day
• 41:36
  one. And yeah, one thing that I have
• 41:38
  learned is that, you know, we always think of machine learning
• 41:41
  platform as a suite of offerings and try to build towards that
• 41:45
  sweet by. But that flexibility is what
• 41:48
  brings the long tail in the machine learning platform and
• 41:52
  not disintegrated into multiple different pieces.
• 41:56
  So, like offering that little bit, little bit of flexibility,
• 42:01
  is something that can bring you a lot more inclusiveness into
• 42:05
  Back home. Yeah yeah.
• 42:06
  One of my key takeaways is to like start small and and dream
• 42:11
  big and iterate on the blockage and and adopt as far as the
• 42:17
  needs change and the other and also be able to scale as the
• 42:28
  customers grow has been one of the other learnings.
• 42:34
  Yeah. Yeah, it's kind of like a
• 42:35
  mixture of what's being said. I think we all share similar
• 42:38
  lovings. I think for me it's like one
• 42:40
  takeaway has been that if you start with the generalist
• 42:43
  platform stayed out of the bat, you're going to actually
• 42:45
  struggle a lot with adoption. Like we've done this in the past
• 42:49
  we're like a lot of features come in because they all sound
• 42:52
  great to have on the platform but that solves not a single
• 42:54
  problem and so my learning is actually take a single problem
• 42:59
  and take it to the Very extreme of the solutions.
• 43:01
  You can and that's one way to get the customer all Depending
• 43:05
  on the platform for their primary use cases.
• 43:09
  And that's when you start to think of expanding how to take
• 43:11
  them further, and I think that's really helped us scaled
• 43:14
  audacious platform. A platform quite well over time
• 43:18
  without losing customers faith. I guess.
• 43:21
  Nice. Yeah.
• 43:22
  First one, Jerry thing. Everyone, though, it was saying
• 43:25
  two things you want to share. One is at the high level.
• 43:28
  It's really about organizational alignment with your customers is
• 43:35
  I hear companies that have challenges of building a mobile
• 43:39
  platform is that it was no alignment between them El
• 43:43
  platform team and the rest of organization.
• 43:45
  That's probably one of the biggest thing to make sure that
• 43:48
  everybody's lying on the same page to is, besides the dream
• 43:52
  big start, small is about what we call one percent better every
• 43:55
  day. It's about how do you
• 43:57
  consistently it? I am what you built and make it
• 44:00
  better and better every month quarter because you probably are
• 44:04
  going to get a right in the first try.
• 44:07
  But you got to make sure I stay with it and prove on it.
• 44:10
  And that's I believe has been helped us quite a bit because we
• 44:14
  focus on that so good, so amazing.
• 44:18
  Hopefully, my mic is working again and everything is.
• 44:21
  Alright, if it's not Skylar will tell me.
• 44:25
  Fellas. This was awesome.
• 44:27
  I cannot thank you enough. I was joking before we hit
• 44:30
  record that this is the most expensive call I've ever been
• 44:33
  on. I still think as far as
• 44:36
  knowledge being dropped I appreciate you all taking the
• 44:38
  time to come and talk to us. I mean it's so cool to hear you
• 44:44
  all. You've been in the trenches for
• 44:46
  the last. I don't know.
• 44:48
  Like, dare I say a decade building platforms and maybe not
• 44:53
  at doordash. But before that you were In it
• 44:55
  and the wisdom shines through. So I thank you so much.
• 44:59
  There's so many great quotes that I am going to take away
• 45:02
  from this and that's it for now. We'll see you all later.
• 45:07
  Hopefully, everyone enjoys it. If they want to get ahold of you
• 45:12
  will leave. Everyone's contact details in
• 45:14
  the description to reach out to them on LinkedIn or on the MLS
• 45:17
  Community. Are you guys hiring by chance in
• 45:20
  case? There's some ambitious people
• 45:22
  out there that want to join the team we are still.
• 45:24
  Hiring. So yes, the last now.
• 45:27
  There we go. There we go.
• 45:29
  Cool. I love hearing that.
• 45:31
  Thank you again, appreciate it so much.
• 45:34
  And can I spend fine?