Latest / Agentic Conversations (formally mlops.community) / Setting up an ML Platform on GCP: Lessons Learned // Mefta Sadat // MLOps Coffee Sessions #71

Transcript

Setting up an ML Platform on GCP: Lessons Learned // Mefta Sadat // MLOps Coffee Sessions #71 · Agentic Conversations (formally mlops.community)

• 0:16
  Hey everybody, welcome to another episode of ml opt
• 0:20
  Community Coffee sessions. I am bishnu and I'm here with
• 0:23
  Demetrius. I do in Demetrius, what's going
• 0:27
  on? I am pretty stoked to talk to
• 0:29
  meth Today because there is a whole line of grocery stores in
• 0:34
  Canada that I had no idea existed and apparently, they are
• 0:38
  pretty popular. Yeah.
• 0:41
  I mean I'm not an expert on grocery stores, but I'm
• 0:45
  definitely interested in hearing more about this particular chain
• 0:49
  and how they are leveraging, ml in production Loblaws.
• 0:54
  Apparently, which is omnipresent in Canada today, we have nefta
• 0:58
  Sadat, a senior mln, Engineer at Loblaws here to join us.
• 1:02
  To tell us all about how this multibillion-dollar business
• 1:05
  uses ML and production, leverages different tooling, and
• 1:08
  share his experience and perspective on where I'm a lab
• 1:11
  stooling is going to go in the future.
• 1:13
  Thank you so much for joining us Matha.
• 1:15
  Yeah, thanks for having me. Looking forward to the station.
• 1:20
  So it's a pleasure to have you here.
• 1:23
  And I want to start off by asking you first.
• 1:25
  Tell us about yourself and what you do at Loblaws and what
• 1:29
  Loblaw Is yeah. So by now you probably know how
• 1:34
  big lava is that Canada. We have.
• 1:36
  We are one of the largest grocery store chains.
• 1:39
  There is literally within 10 minutes or every Canadian.
• 1:42
  There is our one of our store. So yeah, we have read we have a
• 1:46
  big presence here and we didn't la blah.
• 1:49
  Enterprise am I my company's lot of the digital which is like a
• 1:52
  smaller take part of a la blah. And our main focus is to build
• 1:57
  our digital experiences. We have different kind of
• 2:02
  experiences, like we have our online grocery business, very
• 2:05
  sell groceries online. You could order your groceries
• 2:08
  from your app or website and go to our store.
• 2:12
  Just stay in your car and they will deliver to your car, all
• 2:16
  your groceries, and we also do deliveries, we could order and
• 2:20
  get your deliveries deliver to your at your home.
• 2:23
  And there are other other side of the business.
• 2:26
  We have Pharmacy. We have Joe Fresh.
• 2:29
  We also have a It's a program called is the optimum which is
• 2:33
  one of the largest loyalty programs as well in Canada.
• 2:37
  So yeah, so my my team is within love literature.
• 2:40
  Our group is there are signs and my king is the ML and data
• 2:43
  platform team. So we basically build our ml
• 2:47
  that from for our data, scientists to use and deploy
• 2:50
  their older machine learning models or their experiments and
• 2:54
  also take care of all our data ingestion, their engineering of
• 2:57
  those kind of things. Yeah.
• 3:01
  Awesome. And how did you get into ml
• 3:04
  engineering? Yeah.
• 3:06
  So I was, I was doing my masters at Ryerson here in Toronto and
• 3:12
  my, my study was mainly focused around solving computer science
• 3:17
  and software engineering problem using machine learning.
• 3:19
  And I really liked how this field was evolving.
• 3:23
  So right after I graduated, I started at a start-up hearing
• 3:28
  Toronto and the product was great.
• 3:30
  Like they were building television experience, electric
• 3:34
  trees, like in there. Looks like experience for
• 3:36
  television content. So, they're building an app
• 3:38
  where we are regular TV content to go and you could see like a
• 3:42
  Netflix, right? So so your TV providers,
• 3:46
  basically give you this experience and my job was to
• 3:50
  build out a recommendation and personal educational experience
• 3:54
  for our users and I got to do it from scratch.
• 3:58
  So it was pretty cool. We are a small team but we have
• 4:02
  some good content and we ended up building.
• 4:05
  At application and I will be available for all the consumers.
• 4:08
  So that's kind of where I started my journey.
• 4:10
  And after that I just moved to La Fleur where I'm here for over
• 4:15
  three years now. Yeah.
• 4:18
  Got it pretty, pretty classic journey in terms of arriving to
• 4:22
  the machine learning world. I feel like a lot of our
• 4:24
  listeners will empathize with starting off from master's
• 4:27
  program getting Hands-On with the particular application and
• 4:33
  the rest being history. I want to start by understanding
• 4:38
  what some of the sort of use cases for machine learning are
• 4:42
  at Loblaws. It sounds like Loblaws is a
• 4:45
  traditional, pretty traditional grocery chain, which I think is
• 4:48
  one of the most interesting businesses in the world.
• 4:50
  When you think about all the different kinds of products that
• 4:52
  grocery stores have to have the intent to supply chain and the
• 4:55
  Really the diverse consumer base.
• 4:57
  So can you tell us a little bit about how ml is applied and what
• 5:01
  kinds of things Allah wants to apply an 02?
• 5:04
  Yeah. It's a very question.
• 5:06
  So there's so many use cases you can think of when you're dealing
• 5:09
  with a grocery business. Starting from, let's say, our
• 5:14
  e-commerce experience that you have your typical search, your
• 5:17
  typical recommended to use case where you want to show your
• 5:20
  users, the best products they want to buy your the next items
• 5:23
  they want to buy. You also going to give them some
• 5:25
  personalized experience, right? Like you want to predict what
• 5:29
  they're going to buy. Next time.
• 5:30
  They go to the store, right? Or list and the order online and
• 5:34
  there is also A whole lot of other use cases at the store
• 5:38
  level where we want to optimize how our orders are getting
• 5:42
  picked up, how we better pick our orders starting increasing
• 5:46
  efficiency. Also, their use cases, where we
• 5:49
  have to be regular. We have to predict inventory.
• 5:52
  Think about when you go to grocery stores to takes a
• 5:55
  product from the aisle is actually until you check out at
• 6:00
  the cashier. That product is not in your
• 6:02
  inventory, but we don't really know, right?
• 6:04
  Like if These minor directors from our actual inventory
• 6:08
  number. So we have to predict all these
• 6:10
  things in real time. And yeah, so there are so many
• 6:13
  things going on yet, love them. That is fascinating, especially
• 6:19
  that part about protecting inventory.
• 6:20
  I hadn't thought about it, but in an age of instacart and so
• 6:24
  many other different kinds of Shoppers and your own delivery
• 6:27
  service, it's more important than ever to know what's
• 6:29
  actually on the shelves and in stock, not to mention all the
• 6:32
  other things that you mentioned are where I'm now.
• 6:34
  Systems are applied. And so you have this like super
• 6:38
  diverse set of use cases that you want to use across your
• 6:42
  business. And I'm sure that your company
• 6:44
  has a lot of data scientists and machine learning professionals
• 6:46
  that are Applying for casting or analytics to help solve these
• 6:50
  business problems. How does your ML and data
• 6:53
  platform team? Work with those data scientists
• 6:58
  and to empower those data scientist, I'd love to learn
• 7:00
  more about how your team operates.
• 7:03
  Yeah. So the goal of my team is to is
• 7:08
  to make that it will be a mix of the data science lifecycle
• 7:12
  Quakers. So think about starting from
• 7:15
  exploratory data analysis with in your model, validating your
• 7:19
  model to go all the way to production.
• 7:21
  This is a very complicated process, right?
• 7:23
  And we want to make this process as smooth and as easy as
• 7:26
  possible and as fast as possible as always.
• 7:29
  So we build a lot of different tooling we build a lot of
• 7:34
  example, projects for data scientists, we collaborating
• 7:37
  data scientists on. On individual projects, they
• 7:40
  will pair up with them, go over kind of doing system design with
• 7:44
  them and kayak. Kind of highlighting all the
• 7:48
  best practices, or like different tools that they could
• 7:51
  use to solve a given problem. Yeah.
• 7:53
  Got it. How many people are on your team
• 7:56
  and how many people are users of your platform?
• 7:59
  Yeah. So we have, we have another six
• 8:03
  to ten people in my team. We have, we have Engineers, we
• 8:08
  have our team lead, and then we have project manager.
• 8:10
  And some some folks from a sorry who will just like maintain
• 8:16
  production Stains in production and we also have a lot of data
• 8:20
  science. Things working with us.
• 8:22
  We have more than 60 people in Regional group so they're quite
• 8:26
  a few data size change that we have to internet on a day-to-day
• 8:29
  basis. They're like, what data science,
• 8:31
  teams are divided, across different business functions.
• 8:34
  For example, a forecasting or let's say, surge or
• 8:38
  personalization. So there are different different
• 8:40
  group within the data science Division, and all of us kind of
• 8:44
  work together. And, you know, platfrom, We host
• 8:47
  a different Services built by. These teams got interested in
• 8:54
  that and how you go about deciding what you're going to
• 9:00
  work on. And this comes up because I was
• 9:04
  just reading a pretty awesome article from Eugenia and about
• 9:09
  technical design docs, I think in and then I put it in the MLS
• 9:14
  community, and Laszlo commented and It said, oh yeah.
• 9:18
  Well these are just like prds and I was thinking about it and
• 9:23
  I was wondering if people have specific things that they do for
• 9:29
  prds when it is ml related. And if there's anything that
• 9:35
  when you're trying to figure out what you're going to work on
• 9:39
  next. Do you specifically call out or
• 9:43
  do you have processes that are different than say a normal?
• 9:47
  Devops or software engineering process.
• 9:52
  Yeah, so I think we have to kind of prep purchase one approach.
• 9:57
  We have instead up, a lot of example, designs or projects,
• 10:01
  like with the whole life cycle. So you could start with them.
• 10:06
  So we use a lot of template projects.
• 10:09
  For example, a template project is something you could just
• 10:13
  thrown into a repo and then you have the whole pipeline starting
• 10:18
  from building training, testing your pipeline running low.
• 10:22
  If and deploying your pipeline so that kind of have a it has a
• 10:27
  specific design and deployment architecture.
• 10:31
  So you could start with that. The other one is we kind of get
• 10:36
  together and collaborate work on different aspects of the design
• 10:40
  and normally data scientists did sometimes take input from
• 10:44
  software Engineers because without email platform team is
• 10:47
  mostly built with software Engineers, animal Engineers.
• 10:50
  So we kind of collaborate And give some of our insights on.
• 10:54
  How let's say we design go. If there is anything that we can
• 10:59
  do better, we just give our thoughts on that.
• 11:01
  Yeah, got it. So I noticed that you mention
• 11:05
  that your team is not just the animal platform team but ML and
• 11:09
  data platform team. Can you talk to me about what
• 11:12
  constitutes the data platform and what differentiates it from
• 11:14
  El platform. Yeah.
• 11:16
  So so we started building our data that from first and they
• 11:21
  don't go in there. Thumb, because as, you know, we
• 11:24
  don't data, we cannot really do Emma, but it's so we out data
• 11:28
  platform comprises of the crucial things like we have, we
• 11:33
  have been input data. Coming through all screaming,
• 11:35
  Kafka pipelines, we have, for example, like data coming from
• 11:40
  point of sale devices of data, coming from our inventory.
• 11:44
  Those data are coming into our internal injury streaming
• 11:47
  systems. We also have some data coming as
• 11:49
  feeds, so all these data, they Go into our data warehouse.
• 11:53
  We have the query as our data warehouse where, which is kind
• 11:56
  of the unified place, where all our data leaves, right?
• 11:58
  So it was a very powerful tool for like analytics or electron
• 12:03
  you favor an aquarium. We could, you know how fast it
• 12:06
  is to run your queries there. So yeah, so all of these data
• 12:11
  pipelines we kind of orchestrate using different tools.
• 12:15
  Like, for example, I get a chair flow was wrong of the tools we
• 12:18
  use and we have Sobeys, Kafka pops up.
• 12:22
  Really to build on this platform.
• 12:23
  So, so we have, we have data warehouse.
• 12:27
  We also have some skimming topics that we expose for other
• 12:31
  constituents using toxic about Kafka and yeah.
• 12:34
  So basically once we have the data or use cases to mainly just
• 12:39
  need to use this data from more getaway house and build stuff.
• 12:42
  Yeah. Fascinating this is it's
• 12:45
  interesting to see the sort of overlap between the data
• 12:51
  professional. Sort of traditional data
• 12:52
  professional and you and then your team and the sort of ml
• 12:55
  professional. Because I think a lot of times
• 12:57
  what happens in a lot of organizations is you have one or
• 13:00
  the other, you know, where, you know, you'll have an
• 13:03
  organization that gets excited about, you know, machine
• 13:06
  learning and maybe hires like a data scientist or machine.
• 13:08
  Learning fresh bones, okay? Go do machine learning or you
• 13:11
  have a very traditional analytics stack or analytics
• 13:14
  focused or reorganization then over time, maybe starts to say,
• 13:18
  let's let's add machine learning but the group is in in Sort of
• 13:22
  analytical Focus, right? And it's driven by that.
• 13:25
  I'm curious from your standpoint.
• 13:27
  How did this all evolve, you know, at Loblaws in terms of
• 13:32
  what was the timeline for starting to really leverage
• 13:35
  machine, learning in your products and in your business?
• 13:39
  Yeah, so when I started, we were a very small team compared to
• 13:44
  what where we are now we are at now that that time our Focus was
• 13:49
  just to build out our data platforms first and start with
• 13:52
  very simple set of Emily's cases.
• 13:55
  So we already had a legacy recommendation system.
• 13:58
  We had some Legacy other little project, so we kind of take
• 14:03
  those projects and modernized than using Cloud.
• 14:06
  So they were running in our in our Hadoop cluster in our data
• 14:10
  center at the beginning. But we, once we started to move
• 14:14
  to Cloud, we first met make effort to build a data warehouse
• 14:19
  and then kind of take these for this Legacy projects.
• 14:22
  Build out our first Cloud able projects and once we get to gain
• 14:27
  more experience on that. And we kind of showed the
• 14:30
  business that we could deliver in ml products and impact.
• 14:34
  We can create really very good impact on our business.
• 14:38
  Then they started to buy in, right?
• 14:41
  And once, once we iterate over this design over and over when
• 14:44
  we gain the capability to shape, in a modest, much faster, and
• 14:48
  much quicker before you could take, like, six months to put a
• 14:51
  little System into production. But now we could be like two to
• 14:54
  three months just because we have everything in place.
• 14:57
  So yeah so you put like one example I could give you is like
• 15:01
  this a simple lever forecasting models could give you a lot of
• 15:04
  efficiency. Right?
• 15:05
  C-could. You couldn't still see the
• 15:07
  Matrix, the numbers and you will see how much money you could
• 15:10
  say, were, how much time you could stay.
• 15:13
  Right? So, those, those impacts were
• 15:15
  very important for us. Yeah, I want to know.
• 15:20
  What did it look like when you made that transition from these
• 15:24
  Legacy systems to the cloud and what are some top or key things
• 15:32
  that you would like to let everyone else know who may or
• 15:37
  may not have to go through that? What are some gotcha moments
• 15:40
  that you might have had or anything along those lines?
• 15:43
  Yeah, I mean, one of the things I can tell you is with the, with
• 15:49
  the Legacy, Legacy data center approach.
• 15:52
  We haven't a huge spark cluster and the people who were working
• 15:56
  at that time, we're given all the resources all the time but
• 16:00
  it's because the cluster was appalled that time.
• 16:02
  So so when they were writing applications, they were not
• 16:05
  thinking about efficiency efficiency in terms of cost or
• 16:09
  resource allocation, right? But with Cloud, you could create
• 16:12
  resources on demand. Like, you could spin up our
• 16:15
  kubernetes cluster, we could speed up our data, proc spark
• 16:18
  cluster, and shutting down. Once you are Done.
• 16:20
  So you could scale up for like, 30 minutes to your things
• 16:25
  quickly, and shut it down. So, when you think about scale
• 16:30
  and on-demand resource allocation, those kind of things
• 16:33
  you have to, you have to use. If the Clinton clouds, Plumbing,
• 16:37
  teased out, very efficiently. So that's one thing to think
• 16:40
  about, think about how you are going to be transitioning, like
• 16:45
  used tools that are easily adaptable to Cloud.
• 16:47
  For example, you are using each base and Googled out into this
• 16:51
  big table. So we do there is a very easy
• 16:54
  transition, right? So for that first part, it was
• 16:58
  just two data Prague so those are very different Visions.
• 17:02
  Yeah. I really want to dive deep into
• 17:04
  this whole you use case here because the recommendation
• 17:08
  systems use case, it's I've never really worked on
• 17:12
  recommendation systems but it seems like it's well it is the
• 17:16
  field in which I think there has been the most sort of production
• 17:19
  ml work. Look, that's a that's happened,
• 17:21
  right? Even more so than I think deep
• 17:24
  learning systems like recommendation systems are
• 17:25
  everywhere, right? It's the most profitable
• 17:28
  business model in the world than Google.
• 17:30
  So you leveraged Cloud tooling. As you mention right?
• 17:37
  The shift from going from sort of an on-prem Legacy stack to
• 17:41
  using an on-premise oracle Cloud stack for your recommendation
• 17:46
  system, can you talk to me in a little bit more detail about
• 17:50
  What the actual use case was that you were architecting, this
• 17:54
  recommendation system around? What tools you're using for now
• 17:58
  and sort of what the impact has been on the business.
• 18:02
  Yeah, definitely. So, so, this recommendation
• 18:07
  system was about showing you. The users can most purchased
• 18:13
  items are the next next basket, kind of use case.
• 18:17
  So so this latest style is that Was using mapreduce, it was the
• 18:22
  village base and it was everything was based on our data
• 18:27
  center, really moves to Cloud. There are several things we
• 18:31
  wanted to do better. One of them was performance of
• 18:35
  this system. So this Legacy system was taking
• 18:38
  a long time, just running mapreduce algorithm and loading,
• 18:44
  the data. Like it was a batch production
• 18:46
  system so you don't need to run the algorithm.
• 18:48
  We took page, recommendation, and then load.
• 18:50
  All the data to each base. But when you move to Cloud, we
• 18:54
  wanted to improve the performance of the system.
• 18:56
  So let's say you're sticking one second to get an eternity of
• 18:59
  recommendation. We wanted to bring it down to
• 19:01
  less than 100 milliseconds which was which was a very big drop
• 19:05
  and great improvement in our customers experience.
• 19:08
  So what we did was we use big table and we also deployed this
• 19:14
  system into kubernetes. So we we have our other back in
• 19:19
  teams. They're running in kubernetes as
• 19:22
  a micro service architecture. So once we have, we have our own
• 19:27
  recommendation micro service. They could easily easily call
• 19:30
  them micro service. Instead of just getting the
• 19:33
  recommendation from the database directly.
• 19:36
  So that will, it took a couple of things so more and more teens
• 19:38
  could use recommendations because now we offer a
• 19:40
  recommendation as a service. So that shift to the
• 19:44
  architecture, was was really great.
• 19:46
  And then then also the performance.
• 19:48
  So now that we are using Cloud Big Data, Which is because nosql
• 19:52
  guy kind of database offering and it has very high performance
• 19:56
  and throughput. So so just by using big table as
• 20:00
  a as a offline storage for a recommendation.
• 20:04
  Sorry, I'm not like straight for an oak recommendation could
• 20:06
  could improve our free moral agency by great deal so that
• 20:11
  kind of being down our Target to less than 100 millisecond.
• 20:16
  And we are also using Cloud composure which is which is the
• 20:19
  orchestration. The cloud.
• 20:21
  And without composure, we are able to run this Pipeline on a
• 20:26
  schedule C, when it fails, we had, we can, we could retry our
• 20:30
  pipelines. We could have alerting of set up
• 20:32
  on the pipeline, that's the one time it fails, it will just
• 20:34
  directly, there are so your pipeline phase so this overall
• 20:37
  kind of observability gives mix our pipeline more resilient.
• 20:41
  So so that was another another upgrade and we also have
• 20:47
  different type of monitoring setup.
• 20:50
  We had, we are monitoring our or micro service to a 3-2 collab
• 20:56
  Dynamics. So that would also give us real
• 20:59
  time monitoring Matrix like the latency throughput.
• 21:03
  And we also have a lurking, right?
• 21:05
  So so those kind of instability tooling made our services, much
• 21:10
  more resilient. We knew when stuff was going
• 21:13
  wrong, could break, you get proactive and succeed.
• 21:16
  So, yeah, that's really is really interesting actually
• 21:19
  because this is This is a large scale, Enterprise sort of
• 21:22
  offering of ml, right? Internally, of course, but I can
• 21:27
  imagine now that you've exposed this microservice, as you said,
• 21:30
  a lot more people can consume it.
• 21:32
  It's much more performant, it is, you know, it's a better
• 21:35
  engineered system. But along the way, is I have to
• 21:39
  imagine that there are a lot of bumps and that there are a lot
• 21:42
  of growing pains as you staffed up to add more expertise
• 21:46
  experience to your team as you figure it out, you know?
• 21:49
  The As with different tools because I know that, you know,
• 21:52
  as cool as bigtable and Cloud composer and stuff are that, you
• 21:55
  know, sometimes if they're a lot of different quirks and pain
• 21:58
  points in the process of adopting new tools and getting
• 22:00
  them right for you, use case, can you talk to me about how
• 22:04
  long this whole project took? And what some of the biggest
• 22:06
  challenges you faced were Yeah, so this is this was one of the
• 22:11
  earliest projects that we need and we invest a lot of problems
• 22:14
  in terms of permission like how we will set different type of
• 22:19
  identity for your wrap, you need to Grant permissions in Cloud
• 22:22
  for different things. So there are kind of toil around
• 22:26
  that so you want a specific app to have specific permission for
• 22:30
  you to like read access to your database or were cloud storage
• 22:34
  or something. So there are still a lot of
• 22:36
  color on that there is a steep learning curve.
• 22:38
  Our data site is for air flow and blood composure because as
• 22:42
  you know, it's not very user-friendly to some extent,
• 22:46
  they're like some issues, you often see, and any long-term
• 22:51
  users of a particular flow, they will tell you, there are some
• 22:54
  some, some bleeding around the figure on the tombstone.
• 22:57
  So there was a steep learning curve around that and overall
• 23:01
  like Cloud tooling the general. There are some learning curve as
• 23:04
  well for data scientists. So with, with all this Things.
• 23:08
  We kind of started to folks started to shift our Focus
• 23:14
  towards building, a reusable that from.
• 23:16
  So so now we have, we have use all these tools.
• 23:20
  We have deployed a service, but how can you do better?
• 23:23
  So we use here with a few new tools and we incorporated them
• 23:29
  into our platform. For example, one of these tools
• 23:31
  are seldom co-worker. So cell doesn't Accord is our
• 23:34
  models are being told that very thing right now.
• 23:37
  So previously I think I'm really getting our custom microservice
• 23:41
  as an architecture that we're eating certain as a mortal
• 23:44
  serving architectures. So so sale done.
• 23:46
  I'm afraid of being enslaved. Unlike pay anything, you build,
• 23:49
  you have to build now, it's all done.
• 23:51
  So so once you know, salary is not getting much, much easier to
• 23:54
  serve your model. And the next step was just
• 23:57
  creating those example, those reusable, libraries dose,
• 24:00
  template projects, and set up everything using this, these
• 24:05
  tools and that kind of paved the way for a day.
• 24:08
  Data scientists to start using this platform, right?
• 24:12
  So now that we have a fixed set of tooling, we know what we are
• 24:17
  doing and we have all the alcohol of the examples and
• 24:21
  templates and reason codes available for data scientists.
• 24:24
  They could start deliberating this platform or so that was
• 24:28
  kind of our approach really, really fascinating.
• 24:31
  I think that there are a lot of listeners that are hearing what
• 24:34
  you're saying and empathizing in terms of the same Colleges or
• 24:39
  the same sort of maturity process that your team and your
• 24:42
  company went through and are seeing it happen or has happened
• 24:46
  at their colonies. The question I have listening to
• 24:50
  what you talked about in terms of how you have coalesced around
• 24:54
  a specific tooling stack around a process that is heavily
• 24:59
  informed by your experience, working with these tools and
• 25:02
  putting in place a production recommendation system is, how
• 25:07
  did you You get data scientists upskilled on these tools.
• 25:13
  And did you face any pushback in terms of your choices for what
• 25:17
  tools do they should use? Yeah, so that's a good point.
• 25:22
  So first, when we are we were doing a lot of POC projects, to
• 25:28
  start out different tooling. So we use open source tools,
• 25:33
  such as Feast, for example, or email flow, and tell don't go.
• 25:38
  Then we also use some Cloud tools, so we know doing the puc
• 25:42
  we open source tools that said, I mean this year and I think
• 25:46
  make this year Google announced vertex here.
• 25:49
  So which which kind of being opened a new door for ML
• 25:54
  practice on Google, right? So with vertex.
• 25:56
  There you also dwc are leveraging, the whole part a
• 25:59
  Vortex. A platform to build the same
• 26:01
  thing. We built using open source to
• 26:04
  do, right? So with with the vertex, I will
• 26:06
  also use some Of the open source but mostly is in what makes a
• 26:10
  truly. And once we have done that, we
• 26:14
  kind of asked the data scientists.
• 26:16
  Okay, so here is here, is the blood from?
• 26:18
  Can we? Can we do a PLC now with your
• 26:21
  data science project? Can we use one of your project
• 26:24
  as an example and run this on our platform using these tools,
• 26:28
  start onboarding ml. So, for example, our start
• 26:31
  onboarding feast in your project and we kind of help them guide
• 26:35
  through this whole process where we give give give Give them our
• 26:39
  blood from and then we are on board one of their projects,
• 26:42
  right? So after we do that, do that, we
• 26:45
  just gather feedback from The Design District what went?
• 26:48
  Well, what we're very good do better.
• 26:52
  So those kind of questions of kind of discussions, those
• 26:55
  really helped us kind of decide on what to use and how to use
• 27:00
  them, for example, with ML. So one of the feedback we
• 27:05
  received was there is there is no Access Control in open-source
• 27:09
  version, right? So so let's select.
• 27:11
  There is no role-based access control like everyone will use
• 27:13
  the SMS or server. They could kind of have the same
• 27:17
  level of access they could edit in anyone's experiment so which
• 27:20
  was one of the issues we found so.
• 27:23
  So we wanted fine-grained access and so in our case it would be
• 27:29
  best if everyone we can grant rules to every user.
• 27:33
  But that wasn't available with open source and also version.
• 27:36
  So that was one of the issues that Found out and just as the
• 27:40
  experiment with different tools and gather feedback more and
• 27:43
  more features or other kind of issues just come out.
• 27:47
  Yeah, so actually I was going to ask about the access and more
• 27:53
  specifically like data access and how you dealt with that
• 27:57
  because I know you mentioned you you caught a few tough times
• 28:04
  when it came to the transition and learning and then having to
• 28:09
  upskill into the cloud. So what did it look like?
• 28:13
  What were some of the ways you got around the data access
• 28:16
  problems? And if you're watching on video,
• 28:19
  you can see my daughter wants to show her little baby in the
• 28:24
  screen right now. She wants her doll to get a
• 28:28
  cameo appearance. But yeah.
• 28:30
  Like like these data. And the I guess also, what you
• 28:34
  were talking about the role access and being able to have
• 28:37
  like our back was very important for you.
• 28:41
  So how did you get around that? Like what did what was the
• 28:43
  solution that you ended on? Yeah.
• 28:46
  For for this is the reason we kind of preferred Google
• 28:51
  offerings of artists. They I has virtually no haste
• 28:55
  all the Google goodies, right? So Google has removed as access
• 28:58
  control. It gives you a very fine grain
• 29:00
  excess. So when you are using the BART
• 29:02
  XML metadata Services, we could still use Google's Google's like
• 29:08
  permission and access to Leading, so, so that was one of
• 29:12
  the reasons we decided to Google.
• 29:16
  Metadata instead of ml so and features store instead of fist.
• 29:21
  So that was one of the reasons just because you're using Google
• 29:24
  to read from any dog of first that Community these days
• 29:27
  somebody's already beating to the pet from and there is also
• 29:30
  data access patient, right? So when you're talking about
• 29:33
  data, we use a different things. Such as policy tags in bigquery
• 29:38
  that could restrict access. For example, if you have pii
• 29:41
  data, you don't want anyone to see those data.
• 29:44
  So you could, you could Have three I get blocked using policy
• 29:48
  tanks on Google bigquery and there is also like that there's
• 29:52
  a platform called TLP that animal noises your that would
• 29:55
  regularly and then is unstructured data.
• 29:57
  So let's say you have user reviews or like you have a
• 30:00
  chatbot in your website where people could input free from
• 30:03
  tips, right? So all those kind of data
• 30:06
  sources could have been a potential source of pii data and
• 30:09
  you want to make sure this guy's see, cubes.
• 30:12
  So there are tools such as real defective DLP.
• 30:15
  Ready to lose to kind of scan this kind of data sets and
• 30:21
  create access control open top of the state of data.
• 30:25
  Let's make sense. I mean having to do that in the
• 30:31
  open source, world would have been cumbersome and it probably
• 30:35
  would have taken you a lot longer than you could have just
• 30:38
  gone out and bought it. And so that that trade-off is
• 30:44
  always. Fascinating to think about and
• 30:48
  and what you're going to optimize for and what you should
• 30:50
  be building yourself as opposed to just going out and buying so
• 30:55
  sticking on this theme of gcp and tooling and the staff that
• 31:00
  you built. What's been your experience thus
• 31:03
  far? I know there's a lot of people
• 31:05
  out there that I almost and I hate to say it.
• 31:09
  But sometimes when people come to me and they ask me, how
• 31:12
  should I get started where we're starting to set up.
• 31:14
  Our ml Ops, Initiative, what should I do?
• 31:18
  And a lot of times I say, just go to the big cloud vendors and
• 31:26
  by what they have their offering because it seems like it's the
• 31:30
  easiest way to get set up but you didn't.
• 31:33
  It's not your story but you have been using it for a while.
• 31:36
  So I want to hear The Good, the Bad and the Ugly.
• 31:39
  Yeah, so it's the one example I could give you is, you have to
• 31:45
  be very About your cost when you are using Cloud, right?
• 31:48
  So things could start to go wrong pretty easily.
• 31:53
  So you have to have to set up like sort of monitoring or
• 31:57
  you're costing practices. One example where we are using
• 32:01
  car to do some sort of Readiness check, and it was really having
• 32:06
  some luck message, which was a pretty bit longer than usual and
• 32:10
  it was growing up our costs just because we are working more and
• 32:15
  we had a logging things that are between our communities and our
• 32:19
  big query and it was costing us a lot of money.
• 32:21
  So we had to immediately kind of fix that.
• 32:24
  So there is one example. So let's see this.
• 32:27
  These tools have auto-scaling right?
• 32:29
  So it could it could if you do if you're not careful, for
• 32:32
  example, if you're using data flow and you have auto-scaling
• 32:34
  turned down and you are training some image image classification
• 32:40
  model, right? And if you are not careful and
• 32:43
  set up a scaling set, your number of workers.
• 32:45
  Stand off limits, you could we could start talking scale very
• 32:48
  quickly and you could be could burn up a lot of money.
• 32:52
  So there are some some some problems around not problems, I
• 32:57
  would say like configurations. You have to be very careful
• 33:00
  around costing and resource allocation.
• 33:02
  These kind of things they are. You did you set up alerts for
• 33:07
  that? Like did you figure out a way to
• 33:08
  make sure that that doesn't happen?
• 33:10
  And yeah, so you're away. So our team did normally have
• 33:16
  building a large set up at project level for every project.
• 33:18
  This is kind of an organization organizational thing.
• 33:21
  So we are we are aware what's going on in terms of calls here.
• 33:26
  Got it, this has been a great conversation.
• 33:29
  I really feel like, you know, one of the things I appreciate
• 33:34
  when talking to people ml engineers at companies, tell me
• 33:36
  about what, you know what they've done is is hearing how,
• 33:40
  you know. They saw a problem and were able
• 33:42
  to try different approaches and settled on something and what
• 33:45
  trade They were able to make and and learning what technologies
• 33:49
  are most effective in different contexts.
• 33:50
  For example hearing from you about how you use big table, as
• 33:54
  always it's always a plus because it's something that I
• 33:56
  can kind of take back to my own work and say okay this is a
• 33:59
  reference point you know for technology that can work in a
• 34:02
  given situation and it definitely gives me a lot to
• 34:06
  think about when I'm doing my day-to-day work.
• 34:08
  Now with that said a question, I want to ask you is you know, you
• 34:13
  have had this great experience working for a Two years at
• 34:16
  Loblaws working on production production, Rising ml in a
• 34:19
  highly relevant business case. What part or problem in the
• 34:24
  process of production? Izing ml, are you most
• 34:27
  passionate about and would in that you know, you would like to
• 34:30
  continue developing expertise. Mm.
• 34:33
  Yeah. So for me personally, I like I
• 34:38
  like such a recommendation space as a product so so that is one
• 34:44
  thing I really enjoyed my My Master's was also I use
• 34:48
  recommendation technique to solve software engineering
• 34:50
  problems and in my first company, I did a recommendation
• 34:53
  projects as well. So I'd like to see the
• 34:58
  recommendation problem or, you know, it's a, it's a very talked
• 35:01
  about problem, right? But I want to see, I want to see
• 35:04
  working on making this problem more generate.
• 35:07
  So you don't really have to be recommendation.
• 35:09
  Every time you have a problem you can you could just leverage
• 35:14
  the recommendation as a service. Or something like that.
• 35:16
  I know some some Cloud already offered that as a survey I think
• 35:19
  Amazon does one but but there that could be like that would be
• 35:26
  more available and also when we were talking about
• 35:29
  recommendations, you're only talking about the prediction but
• 35:32
  but we need to think about how all the models are performing.
• 35:35
  So we have to think about the feedback loop how our
• 35:38
  recommendations are actually performing in production.
• 35:41
  So there are not a lot of tooling around around Both
• 35:46
  deployment or post Post building your model.
• 35:49
  So, how how we do, you know how the recommendations are working
• 35:55
  as a, as, as you want or what is the, what is the engagement?
• 35:59
  So, so there is a, there is a disconnect I think between like
• 36:04
  like web analytics and machine learning.
• 36:07
  So I would want to work on bridging that Gap.
• 36:11
  So between web Analytics tool. Lets them motion machine
• 36:14
  learning here. That is the really fascinating
• 36:18
  way to put it. And one thing coming out of this
• 36:22
  Conversation, I'm going to test you with Demetrius.
• 36:25
  Is that Master has got to talk to iacopo Aiko.
• 36:29
  Vo who is also thinking a lot about recommendations?
• 36:33
  And recently came out with a beta version of a testing
• 36:36
  library for recommendation systems, I think you'd find that
• 36:39
  really interesting master and to go deeper on that topic.
• 36:43
  I want to know. How are you guys thinking about
• 36:45
  post-deployment monitoring of your recommendation systems?
• 36:49
  Yeah. So should I know?
• 36:52
  We're like I said I used to say don't go for models serving and
• 36:56
  we also have a feedback loop using certain because they don't
• 37:01
  supports like multi and then the another type of complex in
• 37:04
  friendships where we could really time use customer
• 37:08
  feedback or other type of spell. It is feedback to tune your
• 37:11
  models like which one is serving and we also have some analytics
• 37:16
  that from with Adobe analytics and snowplow.
• 37:19
  So we use, we will cover is that Data sets, coming directly from
• 37:23
  our consumer applications and kind of build dashboards or
• 37:29
  graphs, using that data and using career.
• 37:32
  He's looking for that that and then a look at we set the
• 37:37
  dresser alerting. So let's say your new model
• 37:40
  isn't performing or your dealer is problem are adrift in your
• 37:44
  data or model performance, we you detect that from those data,
• 37:49
  so yeah. So we are still Evaluating
• 37:52
  different Alternatives and how we are doing around those
• 37:57
  practices. But up to that point, our
• 38:00
  platform is very fixed for now. Well, that is a great way.
• 38:07
  I think to wrap up our conversation, this has been an
• 38:11
  excellent use of an hour. Thank you so much for joining us
• 38:15
  nefta. It is it is always a great
• 38:18
  opportunity for me and I think I'm speaking for Demetrios on
• 38:21
  the 10.20. Yeah, speak to a great engineer.
• 38:24
  Like you, you're just gonna sign off on it, but really just speak
• 38:28
  to a great engineer. Like, you.
• 38:29
  Somebody that really thinks deeply about, you know, what
• 38:32
  kinds of trade-offs to make has expertise in tools and really
• 38:34
  cares about impact and use case. So, thank you so much for
• 38:37
  joining us and sharing your wisdom and, and letting us know
• 38:40
  that Loblaws is within 10 minutes of every Canadian blew.
• 38:43
  My mind. Thank you so much force to be
• 38:47
  reckoned with, in Canada. And I gotta give a shout out.
• 38:49
  People that are listening, do not Sighs.
• 38:51
  But you have a guitar in the background so as a fellow guitar
• 38:56
  player it's awesome to see and I really appreciate you being very
• 39:00
  transparent and candid with us about everything that you've
• 39:04
  gone through in your journey at Loblaws to create a better ml
• 39:10
  system and the whole ml Ops process that's around that
• 39:14
  thanks a lot and for everyone listening still give us a like
• 39:18
  give us a thumbs up, whatever it is if you are Listening on
• 39:22
  Spotify or YouTube or iTunes. Or I guess it's Apple podcast
• 39:27
  now. It's not I Dunes anymore.
• 39:29
  You can hopefully leave a review or follow us.
• 39:31
  Whatever you do on all those platforms.
• 39:34
  We would really appreciate it. That's all for today.
• 39:37
  Everyone closing words map time. Yeah, just wanted to thank you.
• 39:41
  Both of you again for having me or do.
• 39:44
  It has been a really great experience talking to you today.
• 39:46
  Yeah. All right.
• 39:47
  The pleasure yourself. We'll see y'all later and I
• 39:52
  think by the time that this actually goes out, it will be
• 39:56
  past Christmas. So happy New Year, everyone and
• 40:00
  we'll see you in the new year. Ciao.