Latest / Agentic Conversations (formally mlops.community) / Platform Thinking: A Lemonade Case Study // Orr Shilon // MLOps Coffee Sessions #79

Transcript

Platform Thinking: A Lemonade Case Study // Orr Shilon // MLOps Coffee Sessions #79 · Agentic Conversations (formally mlops.community)

• 0:00
  That was awesome man. We just talked with or and or
• 0:04
  she loan, for those who want a full name, he's one of the ml
• 0:09
  platform engineers at lemonade. What do you think of the
• 0:12
  conversation, Vishnu? I thought it was great.
• 0:15
  I thought it was It's, you know, it's always rare.
• 0:19
  I think to find the kind of person who is both in the weeds
• 0:24
  as a talented individual contributor, but also has to
• 0:28
  think strategically about company and business needs.
• 0:31
  Right. Usually those kinds of people
• 0:33
  are either are plucked very quickly to the top or are off
• 0:37
  doing their own thing nowadays, right?
• 0:39
  Yeah. And it was cool to see someone
• 0:42
  like or who has stuck around at lemonade for a while and has
• 0:45
  really pushed the kinds of em All platform infrastructure
• 0:50
  that, you know, I think a lot of companies would love to have and
• 0:53
  it was a great conversation, that's true.
• 0:55
  Like when he was explaining the platform that they have and
• 0:58
  basically the conversation that we have coming up for you right
• 1:01
  now is centered around, how lemonade created their platform
• 1:05
  and what it looks like and what are some things that they keep
• 1:08
  in mind this platform thinking mentality?
• 1:11
  And when he was talking about it, I just kept thinking like,
• 1:14
  wow, this is pretty advanced. This is cool.
• 1:18
  Like I hope this becomes the standard as opposed as opposed
• 1:22
  to like the exception and so hopefully when everyone's
• 1:28
  listening to it you get some good feedback or some good
• 1:33
  nuggets of wisdom that you can bring into your job.
• 1:36
  And I wanted to mention something else.
• 1:38
  Man, there's some things to call out.
• 1:40
  He was with one other platform engineer.
• 1:44
  Serving 20 data scientists, that's pretty incredible.
• 1:47
  Some point. Now the team has since grown to
• 1:49
  for but to 220 is a yeah, one platform engineer for every 10
• 1:55
  data scientists that's pretty pretty big one.
• 1:58
  And then they have a slack bot that they can train different
• 2:01
  models with. They have a slack bot that they
• 2:04
  can like push models to production with that's pretty
• 2:06
  wild. To, I've never heard of that.
• 2:08
  Yeah, the emphasis on Automation and platform thinking and
• 2:13
  efficiency was pretty impressive to see because As you don't get
• 2:18
  that level of. Leverage in terms of two people
• 2:22
  being able to support that many people very often unless you've
• 2:26
  made long-term investments in the efficiency and productivity
• 2:29
  of those people. And so true the fact that they
• 2:33
  were able to do that shows that they have really done that for a
• 2:36
  long period of time and with a lot of vision and it's a I think
• 2:41
  everyone listening will learn a lot about how if you make the
• 2:44
  right kind of Investments up front over time, they can really
• 2:47
  pay off in terms of efficiency. There's one thing that I wanted
• 2:51
  to point out and we were chatting on slack while he was
• 2:54
  talking about this with, he goes over, there are five different,
• 2:56
  like pillars of their ml platform.
• 2:59
  The first one he talked about was how they built their own
• 3:01
  feature store. And it reminded me of when Jesse
• 3:05
  was on here. Like, what a week ago or two
• 3:08
  weeks ago, and Jesse was talking about this conundrum that a lot
• 3:11
  of these companies who started doing ml early they have, and it
• 3:15
  was exactly the narrative that Jesse laid down for us.
• 3:19
  And if you Listen to the conversation with Jesse go back
• 3:23
  and listen to it was a fascinating one, not a lot of
• 3:26
  people, I don't, I'm surprised why it didn't get as many views
• 3:29
  as or listens. As I thought it would because
• 3:32
  the conversation with Jesse was just incredible, but I think we
• 3:36
  must have messed up on the title or something.
• 3:38
  People who aren't looking on it, it's called scaling biotech.
• 3:41
  So people probably thought like, well I'm not in healthcare, I'm
• 3:43
  not doing anything with biotech so it doesn't apply to me.
• 3:46
  Jesse had amazing wisdom and And so I highly encourage you to go
• 3:51
  listen to that, but the premise of it was in what Jesse talked
• 3:55
  about was how these companies they start building their own
• 3:58
  tools internally because there is nothing on the market but
• 4:02
  then and they all start doing that around the same time
• 4:04
  because they start seeing the need.
• 4:06
  And there's these bottlenecks that are happening.
• 4:08
  So everyone's kind of building their the same internal tools
• 4:11
  say like a feature story. We can use that because that's
• 4:13
  like the canonical example, but it's not just for feature
• 4:16
  stores. A lot of them like monitoring or
• 4:19
  deployment Forms. We can use any of these but what
• 4:23
  happens is that Ubers building their Michelangelo and they have
• 4:26
  a feature story and then they spin it out and it becomes
• 4:29
  tecton and or was talking about how they were building a feature
• 4:33
  store and it's the same thing. Like they, they have this
• 4:39
  problem now because tecton wasn't out before they started
• 4:44
  their their whole journey on that feature store.
• 4:47
  But now it's so customized their features.
• 4:50
  Dorrit lemonade is so customized to their needs.
• 4:52
  That he was saying, I have a really hard time seeing seeing
• 4:56
  us go out and buy something off the market because we have
• 4:59
  something that is so customized to what we do.
• 5:01
  We're just going to keep doing it and Jesse was talking about
• 5:03
  that to like he called it exactly how it was.
• 5:06
  And that's what I really appreciate the conversation with
• 5:10
  Jesse more after hearing or talk about this.
• 5:12
  But anyways, I think we've talked enough about what you're
• 5:15
  going to hear and just get into the conversation with or right
• 5:19
  now. Any last Thoughts From You
• 5:21
  Vishnu real quick? Do you want to just go through
• 5:23
  his bio? Read it out.
• 5:24
  I'm happy to do it just so that everybody gets what he does.
• 5:28
  Yeah, that's probably a good idea.
• 5:30
  Yeah, let me do that real quick. So, or or Salon is an ml
• 5:33
  engineering team lead at lemonade currently developing a
• 5:36
  unified ml platform. Trust me, it's really cool.
• 5:39
  His team's work aims to increase development, velocity, improve
• 5:42
  accuracy and promote visibility to machine learning at lemonade
• 5:46
  previously, or worked at to wiggle on semantic, search at
• 5:49
  Verona. That Intel he holds a BS in
• 5:51
  computer science and psychology from Tel Aviv University or also
• 5:55
  enjoys trail running and sometimes races.
• 5:57
  Competitively last thing before we jump into the full
• 6:00
  conversation, we're looking for people to help us edit these
• 6:05
  videos. Basically you don't have to know
• 6:07
  anything about editing, you just got to tell us where the nuggets
• 6:10
  are where are the gems are in these conversations so that we
• 6:13
  can trim it down and get like a 10-minute highlight reel for
• 6:17
  people to watch on our ml Ops clip.
• 6:20
  Secondary Channel if you're interested in that because we
• 6:23
  can't hire a producer, you're probably thinking well, get
• 6:26
  somebody that's actually like a pro at this.
• 6:28
  The problem is that producers and podcast producers or video
• 6:31
  producers? They don't understand machine
• 6:34
  learning and so they don't know what's a gem and what's just
• 6:37
  like us talking and it's not that cool.
• 6:40
  So if somebody out there who is knowledgeable in this space and
• 6:43
  passionate and a listener wants to help out, we would love some
• 6:46
  volunteers and that's it. Let's get into the conversation.
• 6:49
  Station made me dress. How's it going?
• 6:54
  What's up, man? I'm stoked to be here as always,
• 6:58
  we've got another incredible guest.
• 6:59
  What's going on or hey, man. Hey guys, I'm doing great.
• 7:03
  Thanks for having me. It's a pleasure to have you on
• 7:06
  or we're really excited to talk about ml platform at lemonade.
• 7:11
  You know, it's I'm excited to hear more about what the company
• 7:14
  does, how you guys have built a team you have and and get to
• 7:18
  learn from you in terms of really what this really fast
• 7:24
  growing and hot company in and very interesting industry is
• 7:27
  doing from an ml platform standpoint.
• 7:30
  So to kind of kick things off, can you tell us a little bit
• 7:32
  about what lemonade does of what?
• 7:35
  Ml platform team at lemonade is supporting in terms of use
• 7:38
  cases. Sure.
• 7:41
  So lemonade, I'll start off with lemonade a little bit.
• 7:44
  Lemonade is a full stack insurance carrier and I guess
• 7:47
  we're sort of like powered by Ai and social good, but we do
• 7:54
  basically four kinds of insurance.
• 7:57
  The first is like PNC insurance for homeowners and renters
• 8:01
  second would be pet health insurance and third is life in
• 8:04
  Insurance and the fourth is, we recently launched car insurance
• 8:10
  and so we do this in mostly in the United States but in several
• 8:13
  countries in Europe as well, depends on which kind of
• 8:15
  insurance. And like I mentioned before,
• 8:17
  we're a full stack insurance carrier, which means, we're not
• 8:20
  like an aggregator we actually are the underlying insurer got
• 8:24
  it. And, you know, it's funny.
• 8:26
  I'm actually a lemonade customers so I yeah, I'm from my
• 8:31
  bias. Definitely it, that's what
• 8:34
  that's going. Here, I'm really happy with the
• 8:37
  product and with that in mind I so you guys are full stack
• 8:42
  insurance carrier, you have all these different lines.
• 8:44
  Where does machine learning fit into what lemonade does?
• 8:49
  Why do you need machine learning?
• 8:51
  So I would say, sort of like, for its on this, like line with
• 8:56
  tension between two different things, the first would be like,
• 8:59
  improving the product. So we want to improve our users
• 9:04
  are users lives by improving the product that being something
• 9:08
  like, maybe I don't doing intent prediction for a chatbot for
• 9:13
  customer service before you actually reach a customer.
• 9:16
  Like, if you, you could probably handle something before reaching
• 9:20
  a representative, or maybe doing things like other things,
• 9:25
  automatically for customers. And then the second kind of big,
• 9:30
  big chunk, would be trying to improve our business with things
• 9:34
  maybe like Predicting lifetime value for users, things like
• 9:38
  that, which are, you know, can be done in any business.
• 9:42
  That makes perfect sense. It's, I think always helpful for
• 9:45
  us to set the context for why machine learning is a part of
• 9:48
  the business. And with that in mind, I have,
• 9:52
  you know, seen from the outside a little bit, you know, how this
• 9:56
  business is grown. And also, even just like how ml
• 9:58
  engineering at at lemonade has grown, you know, from the job
• 10:02
  posting you guys have from the descriptions of your platform
• 10:06
  that I see online and talks, can you tell us a little bit about
• 10:10
  what the end-to-end process from going from data to model and
• 10:15
  production looks like at lemonade on your ml platform.
• 10:20
  Yeah, so the platforms are the provides what we call point in
• 10:27
  time data or basically provides data and on our data warehouse
• 10:31
  directly in Snowflake and if researchers want to do, maybe
• 10:37
  exploratory data analysis, they can do that directly on the data
• 10:41
  warehouse on this, like on these giant Dimension.
• 10:44
  Tables, with hundreds of features that have already been
• 10:48
  created for them or Can do it on Raw data.
• 10:54
  I think at this point, we have maybe 1500 features in our
• 10:56
  feature store. So we're at the point where
• 10:59
  hopefully researchers will be able to like not, not have to
• 11:05
  create features for new models, or maybe have to create only
• 11:09
  several and then they'll start their, they'll do their modeling
• 11:13
  in a notebook. And then like the most important
• 11:16
  step for us is that we translate, which It's like
• 11:23
  modeling code both for training and for inference into our like,
• 11:27
  internal platforms framework, which kind of democratizes
• 11:32
  training. So anyone can kind of train
• 11:34
  anyone, elses model, we have a slack bot to be able to train
• 11:38
  models with Cloud resources. So we can kind of run training
• 11:41
  their configure configure like periodic training as well.
• 11:48
  That's super cool. Yeah and then finally they'll
• 11:52
  use that same slack bot. To deploy service the
• 11:55
  production. Since we usually do online
• 11:57
  inference and then that service will be exposed to like
• 12:02
  developers which will integrate with it at lemonade.
• 12:05
  And finally we use a third-party for monitoring.
• 12:08
  We use a poor area which I think alone has been on here at least
• 12:13
  once or twice. And I think what we've learned
• 12:17
  over time is that we really want our data scientists to We can
• 12:23
  figure monitors further machine learning models.
• 12:27
  Like we don't want anything automatic out of the box, so we
• 12:29
  don't get like this alert fatigue and it's something that
• 12:32
  needs to be, constantly tuned all the time.
• 12:35
  Like so they'll configure both. They're like their data drift.
• 12:40
  Monitoring concept of do need monitoring and the performance
• 12:43
  drift monitoring as well what sort of like these very specific
• 12:48
  alerts per feature. In order to be able to monitor
• 12:52
  something that like that matters won't like high high Precision.
• 12:58
  Even if the recall, isn't that great?
• 13:01
  Can we double click on that real fast?
• 13:04
  Because I'm wondering how much of this, as you said some of the
• 13:08
  stuff like the features you can recycle.
• 13:11
  And then when it comes to the monitoring, how much of that has
• 13:15
  to be custom or bespoke every time or how much can you
• 13:18
  recycle? So, I mean like I mentioned
• 13:22
  before we hopefully recycle very many features, we have dozens of
• 13:27
  models running in production like each one of them Has
• 13:31
  thousands of features. So we definitely recycle
• 13:35
  features and then we actually have several kinds of monitoring
• 13:39
  will monitor features like separately from Models, what
• 13:44
  kind of monitor maybe the amount of null values will monitor if
• 13:49
  features are equal between training and inference time,
• 13:51
  this is like the feature store will do it completely separately
• 13:54
  from with if a model even uses the feature.
• 13:57
  And then model monitoring like features within models, are we
• 14:06
  had a time where we did a lot of like Auto.
• 14:10
  We try to do like Auto monitoring to take this burden
• 14:12
  off of data scientists and we kind of reached the conclusion
• 14:17
  at least at this point in the platform that were not able to
• 14:21
  provide that service like in a very good way.
• 14:24
  So we're at the point where Data scientist like manually.
• 14:29
  Configure monitors per feature for model and that's like part
• 14:33
  of our process of reaching production and then they'll get
• 14:37
  these alerts and slack and whatever.
• 14:39
  But and but that's like a big part of the process of reaching
• 14:44
  production and we do these iterations once in a while where
• 14:48
  we go over the monitors. Like we make sure that
• 14:51
  everything's up to par and everything.
• 14:53
  How did you guys decide on a slack bot as being the right way
• 14:57
  to Have this model training process work.
• 14:59
  Yeah. Actually, this slack bot is
• 15:01
  really intriguing, isn't it? That's probably the first time
• 15:04
  I've heard like, about a slack bot, bring being brought into
• 15:08
  the whole ml Ops equation or like just training and machine
• 15:12
  learning equation in general. So I think I'm lucky enough to
• 15:16
  be working on an organization where automation is like top
• 15:20
  priority and so this the slack but there's like a lemonade
• 15:24
  slack bot with blog post about it's called Cooper like from
• 15:27
  Colin Cooper and it's really easy to integrate with this
• 15:33
  platform like this lakh bahts platform.
• 15:36
  So it's kind of like standard lemonade practice to do things
• 15:39
  with the slack bot. And so we kind of have our like
• 15:43
  our own service in our own commands that integrate with
• 15:46
  this platform and do a bunch of different things.
• 15:48
  So maybe we'll have commands to bring up Sage maker notebook or
• 15:53
  take it back down, we'll have commands to maybe remind people
• 15:56
  that their notebooks are up. So we don't spend a lot of
• 15:58
  money. Yeah.
• 16:03
  And then we'll have commands for like, managing the machine
• 16:05
  learning lifecycle for training models with God, resources for
• 16:09
  deploying them. For taking things down, I guess
• 16:14
  several other small pieces, You said something really
• 16:17
  interesting there that I would like to dive deeper into, which
• 16:23
  is automation as a top priority at lemonade and that you're
• 16:26
  fortunate to work at an organization that does that?
• 16:31
  Why do you think automation is so important to eliminate in
• 16:35
  particular? What is it about the company
• 16:36
  that makes you guys want to really prioritize that I want?
• 16:39
  You feel like you're lucky to in that context?
• 16:43
  So I think we prioritize automation.
• 16:46
  From a business perspective like one of the top metrics that we
• 16:51
  try to look at is maybe the amount of customers that we have
• 16:54
  per employee or the amount of AR are we called ifp per employee.
• 17:02
  That's like a very important metric that we look at.
• 17:05
  So automation is like a big part of the company and I think I'm
• 17:11
  lucky to work at a place like that because it makes it I guess
• 17:17
  it's a priority for everyone to automate things, which is really
• 17:21
  nice things. Like, then we have cool things
• 17:23
  like slack Bots. Which deploy machine learning
• 17:25
  models are trained, machine learning models?
• 17:28
  That's really interesting to hear that you guys actually.
• 17:32
  Apply almost business level metrics to.
• 17:35
  What is usually considered just a technical imperative of
• 17:39
  automation? I think a lot of times when we
• 17:41
  talk about, you know, automation mindsets and such at different,
• 17:46
  Talks and podcasts that we've been in, it's mostly was like,
• 17:51
  well how do you automate more? So the engineering team can be
• 17:54
  more efficient internally and can kind of get rid of
• 17:57
  bottlenecks. But to hear that, it's such a
• 18:00
  business focus that trickles down into everything you guys
• 18:02
  do. I think that's a very unique
• 18:04
  approach that clearly yields, some pretty interesting results.
• 18:07
  Like the slack bot that serves. It seems like a lot of customers
• 18:11
  in the company. Mmm.
• 18:12
  Yeah and I know like when you're looking at that automation side,
• 18:16
  I'd of things. What are some points where it's
• 18:19
  gone wrong? Like it had not gone wrong per
• 18:23
  se but just where you try to automate something and it
• 18:27
  shouldn't have been automated. You mentioned them monitoring
• 18:29
  before where you tried to take that through the automation step
• 18:33
  but then you had to dial it back.
• 18:35
  Has there been a point where the slack bot was trying to be
• 18:39
  implemented and you realized whoa actually it's not it's not
• 18:43
  the best use case for a slack bot.
• 18:44
  We need people to be doing this. I don't think I've seen I guess
• 18:52
  maybe not all flat commands that end up being implemented.
• 18:56
  Are utilized 100%. I don't think I've seen metrics
• 19:00
  on it but I don't think I have a really good story there.
• 19:06
  I think like a great story is like we talked about about
• 19:09
  trying to automate machine learning monitoring and kind of
• 19:12
  failing. They're just sort of having like
• 19:16
  low precision. Hi, Rico and having, you know,
• 19:21
  dozens of alerts a day without anyone being interested in them.
• 19:27
  And that's how you knew, this isn't working.
• 19:29
  It's like, yeah, I mean, I'm probably that it wasn't that it
• 19:32
  wasn't working for us to to kind of configure monitors
• 19:37
  automatically. Yeah.
• 19:39
  Okay. Last question about the slack
• 19:41
  bought and then we can move on but it's just so fascinating to
• 19:44
  me. Is there someone that is looking
• 19:46
  at metrics on which commands are being used with the slack bot?
• 19:49
  Like do you have a team that just babysits the slack bot and
• 19:53
  decides what to put in there? I think there's a, there's maybe
• 19:58
  a person. You definitely doesn't babysit
• 20:02
  this left, but it's a platform. So, like, technically, they
• 20:05
  don't even know the commands that that we've written to like
• 20:10
  integrate with this platform. Like I said, I'm lucky enough to
• 20:14
  work at an organization that has like automation key.
• 20:17
  So platforms is something else that's like really big.
• 20:22
  So, the centralizing development in every Of way is also a big
• 20:27
  priority that is fascinating to hear.
• 20:32
  I think, with that in mind, I want to ask a little bit more
• 20:36
  about what tools you guys use in the context of your ml
• 20:41
  development stack. You mentioned snowflake.
• 20:43
  You mentioned notebooks, are you using any announcements on top
• 20:46
  of notebooks and you kind of manage service, they mentioned
• 20:48
  in Port for you? I would just love to kind of
• 20:50
  hear what your tool Universe looks like.
• 20:52
  So I mean, I think Think you guys have seen at least this
• 20:58
  blog post by Ernest renan where there's like five main building
• 21:00
  blocks. So maybe I'll describe kind of
• 21:03
  each building block that we have a beer.
• 21:07
  So we have an internally implemented feature store.
• 21:11
  We kind of started before tecton was public or work with before
• 21:17
  they exited stealth mode. And Where are the feature
• 21:24
  stores? So customized, towards our
• 21:26
  specific needs that I'm having a really hard time like seeing how
• 21:30
  we can kind of use a, any other party that even though, like,
• 21:35
  people are developing things, that are much better in much,
• 21:38
  more comprehensive, we have some like, super specific needs on
• 21:42
  are very specific like data and use cases.
• 21:45
  So we have an internally implemented feature store.
• 21:48
  It's implemented in Python It has eight uses like different,
• 21:54
  they're different context that it runs in it.
• 21:56
  Does it reads streams with Lambda with like AWS Lambda?
• 22:02
  It runs like it serves real-time features from kubernetes service
• 22:08
  with fast API framework. We recently reported like all of
• 22:12
  our code to asynchronous feel like an event Loop which has
• 22:17
  been very successful for us and Model starting, latency online,
• 22:25
  feature stores, back by dynamodb and offline.
• 22:29
  One, like I said, it's backed by Snowflake and then we also run
• 22:33
  like different ATL's over kubernetes as well, and workflow
• 22:39
  management that we use as air flow. which is also does like
• 22:43
  our periodic training if we need and then it also managed like
• 22:47
  the different etls if we want to do batch ingestion into the
• 22:50
  feature store, We use Mo flow both for experiment tracking and
• 22:57
  as a model repository and we're very heavy on like
• 23:04
  infrastructure as a code. So, so we don't use like the mo
• 23:09
  fo feature, where there's like a model production model, maybe
• 23:14
  like a Bluegreen deployment or however, they do it there.
• 23:18
  We actually literally take the model ID.
• 23:21
  And stick it in our code. Like in get everything is
• 23:25
  version, didn't get that memo like all.
• 23:29
  All provisioning code is is version didn't get.
• 23:34
  So like I said, we're very heavy on platforms and like the
• 23:37
  centralizing development Sorry. Go ahead.
• 23:43
  No, I was going to keep describing different building
• 23:45
  blocks, but I'd love to hear a question.
• 23:48
  Well, I mean, I'm just sitting here as an engineer and I'm kind
• 23:50
  of like, well you have all these different sort of nicely.
• 23:56
  You could say configured or architected building blocks,
• 23:59
  right? It's pretty clear to me.
• 24:00
  How you guys have solved. All the different sort of,
• 24:03
  natural friction points. That a lot of other companies,
• 24:06
  including my companies have faced in building.
• 24:09
  Productionize in machine learning models and then also
• 24:12
  monitoring and maintaining the production.
• 24:14
  So what are the areas of friction that still exists?
• 24:18
  I'm curious in terms of maybe your tool selection or just in
• 24:22
  terms of putting more and more models into production?
• 24:29
  I think maybe the biggest challenge we face in general is
• 24:35
  like a people challenge. So the market is very hot like,
• 24:41
  for employees at the moment. And yeah, and I think we try to
• 24:47
  solve problems that we want to like enable data scientist from
• 24:55
  all different backgrounds to be able to use this platform and
• 24:58
  that's still our biggest. Like enabling people that have
• 25:01
  only ever delivered notebooks to reach production and at the same
• 25:06
  time enable data scientists who have been like software
• 25:09
  Engineers for 15 years and are very opinionated on Frameworks
• 25:14
  and would like to do everything. You know, don't want to know
• 25:16
  what's going on under the hood and we sort of keep like the
• 25:21
  interfaces that we provide is really the biggest challenge.
• 25:24
  It's not like the underlying code, it's really deciding on
• 25:28
  those interfaces. Aces, and keeping them current,
• 25:31
  and having them work for everyone, from like the least
• 25:35
  experience to the most experienced.
• 25:39
  Well, I love that and I love hearing about how this spectrum,
• 25:46
  you're serving this whole Spectrum from the data
• 25:48
  scientist, who has been only a data scientist and loves
• 25:52
  jupyter. Notebooks and doesn't want
• 25:53
  anything to do with anything else.
• 25:55
  And then the data scientist who is transitioning from being in
• 25:59
  the software engineering world and is very opinionated.
• 26:02
  I'm thinking about when you're opinionated like that, or you as
• 26:06
  the platform, You have to ultimately make some decisions
• 26:11
  and you have to be opinionated opinionated about some things.
• 26:15
  Like, how does your choice? How does that look and you go
• 26:19
  and are you surveying, the different users of the platform?
• 26:23
  Or is it something that you just say?
• 26:25
  All right, we can't let this happen anymore because we've
• 26:28
  seen the downstream effects. So I think we're kind of at the
• 26:32
  point where we've gone full circle.
• 26:35
  It's kind of a weird. We started with something that
• 26:37
  was very open. And then like our second version
• 26:41
  of our model servingframework which is like the main area
• 26:45
  where data scientists work, like the model training in serving
• 26:47
  framework, where those sort of implement an interface basically
• 26:52
  to like, fit a model and to predict on the model and give a
• 26:56
  list of features. So that framework was very open
• 27:00
  in the beginning where you could sort of make all these different
• 27:02
  decisions. Then the second version was kind
• 27:05
  of closed where it was really good. 80% of the use cases but
• 27:10
  did not work very well for about twenty percent and we've kind of
• 27:14
  like gone for a slow cooker. So full circle in a way that we
• 27:18
  can we sort of allow both at this point but there's nowhere
• 27:22
  in the middle. So some people will kind of make
• 27:26
  all of these decisions about where their features come from
• 27:31
  and like specific queries and do maybe like right custom queries
• 27:34
  to bring their, their features from Snowflake and a some way
• 27:38
  but they'll kind of have to take responsibility for that and
• 27:41
  others will just like provide a list of features and it all
• 27:45
  happened for them. I think your statement about how
• 27:51
  you think about your platform work Really is the clearest
• 27:56
  articulation. I've seen of the sort of
• 27:57
  customer mindset that has to happen for platforms and
• 28:02
  internal platforms to work where you're saying, you know, you
• 28:06
  mentioned. The hardest problem I have is
• 28:08
  not picking the tools, it's not setting up the architecture.
• 28:10
  It's really figuring out what my customers need in the form of
• 28:14
  data scientists. And it's designing those
• 28:16
  interfaces thoughtfully enough that I'm serving all their needs
• 28:19
  without making my work impossible to do.
• 28:21
  And I think that is like, the central challenge that we talked
• 28:23
  about Much on this podcast. It's kind of fun to hear that
• 28:27
  frame really elegantly by you and your husband, Vishnu sorry,
• 28:31
  to interrupt there. But another point that I wanted
• 28:33
  to add was that it sounds like or what I understood to is that
• 28:37
  if you make a really cool platform that someone can have a
• 28:41
  great time using and these data scientists enjoy using, you're
• 28:45
  going to be able to attract better talent because they enjoy
• 28:49
  using the platform. They enjoy the problems that
• 28:51
  they're working on. Is that another piece of It or
• 28:54
  did I just make that up in my head?
• 28:57
  So I think that's a big piece of it of having people.
• 29:02
  I like to talk about ownership a lot and like, like, I think it's
• 29:09
  quite obvious from the way that I've described the platform that
• 29:12
  were not a throw it over the fence, kind of team, where data
• 29:15
  scientists have ownership, and to end, and it attracts a
• 29:20
  certain type of data scientist. But I think that's I think maybe
• 29:25
  people that have worked on teams that were like throw it over,
• 29:28
  the fence have experienced the frustrations on like both sides
• 29:33
  both with like different ownership models and lagging
• 29:37
  delivery. And I think that's like
• 29:41
  something that's very important to us, having this like clear
• 29:47
  ownership there. Obviously got gray areas in the
• 29:50
  middle but There's clear like engineering ownership and clear
• 29:56
  data signs ownership and data scientists will write the code
• 30:00
  that runs in production and that's so important to us.
• 30:03
  Do they get the call? Also at 3:00 in the morning if
• 30:07
  something goes wrong. They're tagged in slack.
• 30:10
  Yeah. Like they're on their own or
• 30:13
  tags on their models. Cooper's letting them know.
• 30:17
  Yeah, exactly. So I mean we're definitely auto
• 30:21
  tag done more things. But they do get to have done
• 30:27
  different types of alerts on their models.
• 30:29
  Whether it be like applicative alerts just like exceptions or
• 30:34
  date and concept drift or no features things like that.
• 30:38
  Going back to something you said before about The people
• 30:42
  Challenge and hiring in general. And this in this industry, job
• 30:45
  markets Global job market that we now you really are
• 30:49
  experiencing can you tell us as a team lead, what parts of the
• 30:53
  hiring process are so hard right now?
• 30:55
  Is it finding qualified candidates for this type of work
• 30:58
  in terms of ml platform and the combination of software data and
• 31:02
  machine learning or is it closing on candidates?
• 31:05
  What are the some of the challenges you're facing?
• 31:07
  So like you said, I'll speak only two challenges on like
• 31:10
  specifically. Machine learning platform
• 31:12
  Engineers. Sure.
• 31:17
  I think I'm having, I think we're having more trouble like
• 31:20
  closing qualified candidates, definitely the pipeline, like
• 31:24
  the candidate pipeline is not what it was, two years ago.
• 31:28
  Like I see much much less candidates and we have to, like,
• 31:33
  I don't know approach them more than they would approach us.
• 31:37
  Like Lemonade's engineering brand.
• 31:39
  Is quite good. Yep.
• 31:42
  And I also think this is like a very interesting role in
• 31:45
  general. Machine learning platform.
• 31:48
  And yet we're still having to approach candidates at this
• 31:51
  point in time, where, as before like we didn't have to at all.
• 31:55
  And I have a theory about that just before I'd love to hear.
• 32:00
  Think it's I think it's because you know what, I think it is is
• 32:04
  there are so many new startups that are just getting an influx
• 32:08
  of cash, the amount of VC money that's going into all of this
• 32:12
  stuff that has to do with machine learning.
• 32:15
  And now, these machine learning Engineers are being brought on
• 32:18
  to all of these different startups and maybe they're the
• 32:21
  first engineer for the machine learning platform.
• 32:24
  Or they're being tasked with a lot of responsibility for them,
• 32:28
  for all of these different, all this Talent that's out there.
• 32:32
  There's a lot of opportunity, right?
• 32:34
  And so the reason I think, if you go back a notch as to why
• 32:40
  there's so many job openings for machine learning Engineers, it's
• 32:42
  because of that, but like the amount of VC money, It's going
• 32:46
  into all of these startups that have anything to do with machine
• 32:49
  learning or using machine learning in their product in the
• 32:52
  day and like the core piece of the product has just gone up
• 32:57
  drastically. And so that's why it feels like
• 32:59
  there's not enough Talent out there but that's that's my
• 33:02
  theory. No I completely agree like two
• 33:04
  years ago there were maybe five like Tech unicorns in Tel Aviv
• 33:08
  and now there's 50 Wow. I mean Tel, Aviv.
• 33:14
  I know tell you was a Boomtown. It's really cool to see but I
• 33:16
  did not know the scale was that, you know, massive.
• 33:21
  Yeah. So there's like, there's a lot
• 33:23
  of easy money being poured in at the moment or there that I don't
• 33:26
  know what's going on. Specifically like now now, but
• 33:30
  in the past year there was yeah, you're seeing the repercussions
• 33:33
  of it now. Yeah.
• 33:34
  You're seeing people that have taken jobs, maybe three months
• 33:37
  ago or five months ago because they got the VC money a year.
• 33:40
  Go and now they finally found someone to take that machine
• 33:43
  learning engineering position and so it's harder to find those
• 33:47
  people but visually I know you had a question that I cut you
• 33:50
  off or sorry. I cut you off with that theory.
• 33:53
  Like tell us more if you can remember what you were talking
• 33:56
  about. Well good or no.
• 34:00
  I was going to continue like with the building blocks but
• 34:04
  this conversation is more interesting.
• 34:07
  No, I mean I think I think I think the one last question I
• 34:10
  had before, maybe we can jump back to the to the building
• 34:14
  blocks piece is, you know, you've described a really
• 34:18
  interesting process in terms of like what intend ml looks like
• 34:22
  at lemonade. What how many people do you have
• 34:25
  on your team right now and how many people do you support?
• 34:27
  Like what does that or chart and sort of headcount look like so
• 34:30
  we're currently four people on our team and we support like
• 34:34
  just over 20, they This, I think like, one of the things that I'm
• 34:39
  most proud of is that there were several months where we were two
• 34:43
  people supporting over 20 data scientists, which kind of like
• 34:48
  I'm really proud of the, like, the engineering to data
• 34:52
  scientist ratio that we've had there.
• 34:54
  And we still kind of managed to compare compartmentalize all
• 34:58
  this like from the organization. And I think it's like it's a big
• 35:04
  part of the platform thinking at lemon, Made in the way that even
• 35:08
  like our devops organization has exposed building blocks for us
• 35:12
  to use and the Simplicity of us, being able to like get up and
• 35:17
  running with anything open source, even within like a
• 35:22
  quote, like medium to large sized organization at this
• 35:25
  point. That is first off.
• 35:28
  I'm glad to hear number one what you're proud of because I think
• 35:31
  that that is always an interesting sort of lesson in
• 35:36
  terms of what Your team lead right here, later on, on your,
• 35:39
  on your platform, and your company.
• 35:42
  And it's always interesting to hear what leader, celebrate
• 35:44
  right those you about their value.
• 35:46
  So that's cool. And number 22, people supporting
• 35:49
  20 data scientist for a company of lemonade size and, you know,
• 35:53
  just customer base. That's, that's pretty
• 35:55
  remarkable. And it does speak to the quality
• 35:57
  of the vision behind your engineering and operations
• 36:01
  internally. That's that's really awesome to
• 36:03
  hear. And like we're really kind of
• 36:06
  standing on the shoulders of giants.
• 36:09
  Both you know, cloud computing in general and then like
• 36:14
  Lemonade's infrastructure about that.
• 36:18
  I'm glad that people are getting to hear this.
• 36:20
  Hear about this as at the through this podcast because you
• 36:24
  know, I work in the health care sector, you know, I work at
• 36:27
  early stage startup and I think for us in an industry that we're
• 36:33
  in in Care, there's a lot more sort of we deal with a lot of
• 36:37
  administrative bloat. In the US Healthcare System,
• 36:38
  right? As a by Design, and there's not
• 36:42
  as much emphasis on efficiency and as not as much understand
• 36:45
  understanding almost philosophically of the power of
• 36:49
  leveraging tools and infrastructure to make one
• 36:54
  person the equivalent to people five years ago.
• 36:57
  And I think Industries and companies like yours or is
• 37:01
  showing the way to people like mine where I'm I'm sort of a
• 37:04
  first, I've been a first machine learning engineer higher, or
• 37:08
  first data scientist higher. I looked companies like,
• 37:10
  lemonade or Pinterest or companies like that are a little
• 37:14
  bit further along in Industries where that level of efficiency
• 37:17
  and you know, infrastructure leverage, I guess you could call
• 37:21
  it, is his prized. So thanks for sharing that
• 37:26
  infrastructure. Leverage.
• 37:27
  That's a new one. I like that might have to coin
• 37:30
  that I don't know if that's AI, don't know if that's quite the
• 37:34
  word that we're a very young age.
• 37:35
  I want to go with but we'll keep that there for now.
• 37:38
  Well let's talk for a minute about the the other building
• 37:42
  blocks that we like kind of cut you off and we derailed, we went
• 37:45
  on a little bit of a tangent and so we got to the first to write
• 37:49
  like the feature store and then m/l flow, but there were three
• 37:53
  more that you mentioned. So I also mentioned airflow is
• 37:56
  like training orchestration and then the I've kind of mentioned
• 38:02
  In beforehand, which we use aporia.
• 38:05
  And the final one is model serving so we have this like
• 38:10
  internal model training and serving framework.
• 38:13
  Which I also sort of ended up discussing that the interface is
• 38:18
  there. Like some of the most important
• 38:20
  things that we that we handle, and there's like a set of
• 38:26
  methods that if they're implemented.
• 38:28
  Then we get like the platform will guarantee a bunch of
• 38:31
  things. Like a highly available service
• 38:34
  and different several different types of monitoring.
• 38:37
  I mentioned before that we have like you he's applicative
• 38:41
  monitoring and then like generic monitoring of features and then
• 38:46
  we have some specific specific model monitoring for like data
• 38:51
  drifting concept wrist will get like alerts on those things.
• 38:56
  See ICD, you know, all these engineering guarantees, if we
• 39:02
  implemented this set of interfaces also get batch
• 39:05
  inference out of the box which is kind of nice if people just
• 39:09
  want to go and like, do badge in friends on this model, even
• 39:12
  though it's an online model, And there's something that has been
• 39:17
  coming up quite a bit recently with the ml Ops Community.
• 39:20
  Not only in slack, but also on the people that we have on and
• 39:24
  it's all centered around testing and how to do testing for ML and
• 39:28
  specifically, like, what kind of things do you look at?
• 39:31
  How have you guys cracked that nut?
• 39:34
  So I would say that's still one of the biggest challenges that
• 39:38
  we have ahead of us. I mean, on the data side, we do
• 39:44
  testing like since we have a feature store were able to do
• 39:49
  testing of feature unification like features even even if
• 39:54
  they're generated from different sources, will still be able to
• 39:57
  test that they're correcting it like in France versus training.
• 40:03
  But on like the model side I would say we're we're still at a
• 40:07
  manual phase in this process like it's still something that's
• 40:12
  done in notebooks. Yeah that's that's a really
• 40:16
  classic one and I feel like that's probably why we put out a
• 40:21
  few videos on testing recently and they've gotten a ton of
• 40:24
  traffic and I think it's just because most people are hitting
• 40:28
  that bottleneck right now. And they're saying, how do I do
• 40:30
  this? What Are some best practices who
• 40:33
  can I learn from? There's not a lot of literature
• 40:35
  out there when it comes to testing and some people
• 40:38
  depending on which space you're in, depending on your use case,
• 40:41
  you're going to do testing differently and you need to
• 40:44
  think about different things, like keep certain things in mind
• 40:47
  when you test. So, and not to mention like
• 40:50
  there's a ton of different kinds of tests that you can do, and
• 40:54
  which ones do you focus on? So yeah, but because kind of of
• 41:00
  the opinion that That like just like in monitoring where we kind
• 41:04
  of thought we could do it automatically, and we found out
• 41:07
  that it's something that at least at this point, in our, in
• 41:11
  our platform, we have to have data scientists work on this as
• 41:15
  part of the process. I think it were at the we're
• 41:20
  still at the point where it's the same for testing, like we
• 41:23
  can't autotest something. I don't know like what we could
• 41:28
  auto test to have someone feel safe.
• 41:31
  Deploying on new model to production.
• 41:35
  Like it's still something that people have to take
• 41:37
  responsibility for at least currently on our team.
• 41:43
  Yes. So I kind of want to zoom out
• 41:45
  here from the technology. They kind of go back to the big
• 41:48
  picture. We've talked about some of the
• 41:50
  winds that you guys have experienced in terms of
• 41:53
  efficiency, right? We talked about some of the
• 41:56
  lessons you've had in terms of Automation and its upsides and
• 41:59
  downsides with monitoring we talked a little bit about what
• 42:03
  the future looks like in the sense that interface has
• 42:07
  continued to be a A challenge that you are trying to think
• 42:13
  through from a platform Vision, I want to go back to the mission
• 42:17
  statement that is in your bio that you as a team lead at lap,
• 42:23
  lemonade and ml platform team. Writ large focus on development,
• 42:29
  velocity, improving accuracy, and promoting visibility into
• 42:32
  machine learning. How did you guys get such a
• 42:35
  clear mandate? And can you talk us through?
• 42:39
  Historically? What That looked like, was it
• 42:41
  sort of the CTO kind of saying? This is the way I want them a
• 42:44
  platform to work or was it a little bit more Dynamic.
• 42:46
  Like how did you get to such a Clear Vision that's then
• 42:49
  translated into all these results?
• 42:54
  I definitely would say that it's Dynamic.
• 42:59
  I kind of wanna I want to say that my like the head of data
• 43:04
  science at lemonade. I think you made a good decision
• 43:07
  by bringing in engineering quite early in the Like that assigns
• 43:11
  lifecycle lemonade and bringing dedicated engineering, and quite
• 43:14
  early. So there was like a time where
• 43:17
  obviously lemonade brand models and within the service and
• 43:21
  features were like sent to it automatically like our first
• 43:24
  version of running machine learning at lemonade.
• 43:27
  But then when you like, I was the third person on the data
• 43:30
  science team. Which is I think yeah that's
• 43:35
  quite early to bring in engineering and I think that's
• 43:39
  like a decision that sort of paid off with building the team
• 43:43
  in general bringing an engineering to like I guess look
• 43:50
  to start with platform thinking early within the process.
• 43:54
  And then these are, these are goals that have kind of
• 43:59
  developed along the way. I mean development velocity is
• 44:03
  like the biggest thing, that's it's very easy to stay but then
• 44:08
  you know improving accuracy is something that a platform can
• 44:13
  can allow and then visibility both internally in the team and
• 44:17
  externally our priorities for us.
• 44:19
  So we want like data scientist like to be able to see what
• 44:23
  other people have done and we want model training to be
• 44:26
  democratized. So anyone can train anyone elses
• 44:28
  model with different hyperparameters, And then can I
• 44:34
  go to the organization is also like something that's been
• 44:39
  understood along the way that we kind of need to explain what
• 44:41
  we're doing. Organizationally and the
• 44:44
  platform is maybe a place to start.
• 44:49
  I want to tell you just real fast.
• 44:50
  There's a quick, funny story. I have about the head of data
• 44:54
  science at lemonade. I can't remember his name right
• 44:57
  now for the life of me. I really tried in English, it's
• 45:00
  the Faneuil, but it's not the affected in Hebrew.
• 45:03
  Yeah, that's it. So, back in my DOT science days,
• 45:06
  when I was in sales, I reached out to him and like tried to
• 45:10
  sell him, the dot science platform and I think I remember
• 45:14
  just asking something like, hey we do this x y&z with thought
• 45:17
  science, you want more information And he was like,
• 45:19
  yeah, I was like, oh my God, lemonade.
• 45:22
  The guy said. Yeah, this is amazing.
• 45:24
  And then he ghosted me for the rest of my time at da science.
• 45:28
  So I always whenever I look at lemonade mango.
• 45:32
  We hire me like at the same time, probably it.
• 45:36
  Yeah, it was like 2019, was it? Yeah, that's when I started.
• 45:43
  Here we go. Man Demetrius.
• 45:46
  I have or now. Yeah.
• 45:49
  I mean, he made the better choice, to be honest.
• 45:51
  That signs went out of business, but what are the maybe they
• 45:54
  wouldn't have gone out of business and lemonade was a
• 45:56
  customer, who knows? I think a lot of people going
• 46:01
  back to what you just said about you coming in early.
• 46:04
  I think a lot of companies struggle to embrace platform
• 46:07
  thinking early because they're not sure if they're ready for
• 46:10
  the expense and the investment, and it's definitely one of those
• 46:14
  things that pays off long-term, but you have to be committed to
• 46:18
  it. You can't flip pull the plug
• 46:20
  early and I see that now at the company, I'm at now where you
• 46:23
  know, I have to Advocate a lot for really thinking about.
• 46:27
  We want to invest in our our limited time energy and
• 46:31
  resources into building a data platform, that'll be really good
• 46:34
  in a year and not focus on like generating a bunch of one-off
• 46:37
  reports right now you know about different sort of analytics or
• 46:41
  insights questions. So it's good to hear about a
• 46:44
  story where this does work and I'm definitely gonna send this
• 46:46
  podcast to my boss and say hey, Look what happened?
• 46:50
  Kind of feels like this was in the culture though, right?
• 46:53
  Of lemonade already before you got there or I agree with
• 46:58
  platform thinking is something that's kind of big and that
• 47:02
  lemonade in general, but it is a gamble like you you know right.
• 47:06
  A premature of Tom optimization is like the one of the evils of
• 47:10
  engineering. Yeah that's true.
• 47:14
  I want to close with a quick question about your talk a tap
• 47:19
  like on which I highly recommend.
• 47:21
  We're going to put it in the show notes.
• 47:23
  Everybody should check it out. It's just 10 minutes.
• 47:25
  Great. Overview of how to engineer to,
• 47:28
  real business problems. You had a quote, which is
• 47:33
  verbatim if you're making online prediction consider making the
• 47:36
  business point in time, part of your machine learning platform,
• 47:40
  can you quickly tell us what business point in time is?
• 47:43
  And what it's allowed you to do and why you think other people
• 47:46
  should adopt it? Yeah.
• 47:48
  So I'll start with the fact that it really depends on the
• 47:51
  product. So there will be companies with
• 47:55
  products that this is completely Elia relevant for and then
• 47:58
  companies where it may like, kid a spot where, oh, this is
• 48:01
  perfect. So the business point in time is
• 48:05
  basically, when you make most of your predictions.
• 48:07
  So we'll make a lot of predictions that lemonade during
• 48:10
  like these specific times during our business flow like maybe
• 48:14
  when a user creates a quote or when they purchase policy or
• 48:17
  when they make a claim. Since for an insurance company,
• 48:21
  I think it's quite easy to understand this flow and then we
• 48:25
  kind of want to know how data Looks during this like this
• 48:29
  specific point in time because that's when we're making
• 48:31
  predictions. So this is like a training data
• 48:33
  notion and creating training data for these very specific
• 48:38
  point in time instead of like having this open way where data
• 48:41
  scientists can say where they want.
• 48:44
  Like when they want data from for each data point and have it
• 48:47
  be like totally open and open to issues just as much as it's open
• 48:51
  to like to anything is something that we found.
• 48:57
  Is just not needed like people don't need that at our company.
• 49:01
  They want to know how they'd eloped when someone purchased a
• 49:04
  policy and that's all they want to know like, in most most
• 49:07
  cases. And so, providing this providing
• 49:11
  this data, both allows us to like, test it very well to make
• 49:15
  sure that it's unified between training and inference.
• 49:19
  And then also too just like minimize the amount of mistakes
• 49:22
  and the amount of engineering that goes into making decisions
• 49:26
  because It's kind of only done once like people create data for
• 49:30
  this point in time, so I couldn't help but notice
• 49:33
  something there. When you talked about this, and
• 49:36
  it felt like you got a little bit passionate.
• 49:38
  When you started talking about the way that data scientist,
• 49:42
  want their data or the the different ways or the rainbow of
• 49:47
  choices that they have, what, what, what's behind that?
• 49:52
  Why are you so passionate about that?
• 49:55
  Have you seen it Go the wrong way?
• 49:56
  I think. This is like just how it sort of
• 49:58
  worked out organizationally organizationally at lemonade
• 50:02
  like this is what the customers have wanted and they were very
• 50:05
  adamant about it. But I mean I could totally see
• 50:10
  why some customers would want to be able to you know choose data
• 50:15
  from any time. It just kind of depends when
• 50:18
  you're making predictions. Like you have to sort of look at
• 50:23
  what's going on. Like in most of the use cases,
• 50:26
  amazing man. This has been it blew my
• 50:30
  expectations out of the water. I knew we were going to have a
• 50:32
  great chat but I didn't realize it was gonna be this good.
• 50:35
  I want to thank you for coming on here and just yeah.
• 50:38
  My mind. Vishnu has always great time,
• 50:41
  hanging out with you too and is there that's about it.
• 50:44
  You got any final words, Vishnu? I mean we're gonna you know take
• 50:48
  a second afterwards and think all think about it and then come
• 50:51
  up with lots and lots of quotes which I we have from here.
• 50:54
  But you know I think your emphasis on platform thinking
• 50:57
  the lessons you shared with us and the engineering quality at
• 50:59
  limited really stand out or thanks a lot for joining us and
• 51:03
  thank you guys for having me. I really enjoyed it.
• 51:06
  Your team is hiring, right? Oh yeah, there we go.
• 51:12
  If anybody is in, is it in Israel only or anywhere from my
• 51:17
  team or hiring only until Aviv? Okay, Tel, Aviv.
• 51:20
  There are quite a few people in Israel in the community.
• 51:23
  So if you want to go work with or and get some of this
• 51:27
  incredible way of looking at the ml platforms, hit him up.
• 51:32
  You're in the community slack or just I imagine people can get
• 51:35
  ahold of on LinkedIn and all that good stuff.
• 51:37
  Thank you.