Latest / Agentic Conversations (formally mlops.community) / MLOps Meetup #26 // How to Leverage ML Tooling Ecosystem // Mariya Davydova - Head of Product at Neu.ro

Transcript

MLOps Meetup #26 // How to Leverage ML Tooling Ecosystem // Mariya Davydova - Head of Product at Neu.ro · Agentic Conversations (formally mlops.community)

• 0:18
  We have a great talk today. I am very excited to announce
• 0:25
  that Maria is with us and she's going to talk about how we can
• 0:29
  Leverage This ml2 in ecosystem, how we can make the best out of
• 0:35
  it? Because I imagine a lot of you
• 0:39
  are like me and you've maybe played around with a few tools,
• 0:44
  but there's so many and how can you make them work together?
• 0:47
  What can we do to make them play nice?
• 0:50
  Right? Just a bit of background of
• 0:51
  Maria she comes from a software development background and she
• 0:58
  started her career as a Java developer.
• 1:00
  Soper. And then switched over became
• 1:02
  more and more interested in ml Ops and AI for the last couple
• 1:09
  years right now. It's been it's been a few Well,
• 1:16
  Paul a little bit of a year. Pizza.
• 1:29
  Four. I'm not sure if it's me or if
• 1:34
  it's you but I think one of us is breaking up.
• 1:40
  So look. If you what we can try to do for
• 1:48
  now is we will turn off video see if that helps with the
• 1:56
  bandwidth for you. I know it's not as cool but yeah
• 2:04
  we'll see if that helps. We all liked the video and I'll
• 2:09
  just finish this intro. We'll see if the Your connection
• 2:14
  stabilizes but really came to Ai and ml.
• 2:20
  I think she said she's been doing it for over a year over a
• 2:24
  few years. If anyone else caught it, please
• 2:27
  correct me and she's finished a lot of books, read a lot of her
• 2:32
  done. A lot, of course has finished a
• 2:33
  lot of books and wrote a fictional series on AI which is
• 2:38
  very cool. Now she believes that proper
• 2:42
  tooling. NG and decent development of
• 2:46
  operation practices is essential success for machine learning and
• 2:52
  I think we all can agree with her on that one.
• 2:54
  That is something that we are here for, right?
• 2:58
  And if you've been in our slack, which Shameless plug, we have a
• 3:02
  slack, and go ahead and get it. If you're not in it, Chris can
• 3:07
  drop a link to that in the chat. Maria is Super Active on there
• 3:12
  and very helpful. For.
• 3:14
  She was the one who had the initiative, she created the
• 3:18
  initiative to start a questions and answers channel.
• 3:22
  So if anyone has questions about different things that are coming
• 3:27
  up, when you're in your ml, Ops life cycle, she's there to help.
• 3:32
  There's a few other really smart people that are in their way
• 3:34
  smarter than me. Yes, you can imagine and they
• 3:40
  Are very generous with their information, their knowledge.
• 3:43
  So if you have any questions about anything, ml Ops related,
• 3:47
  get in there, ask questions. I'm sure.
• 3:49
  Maria will be happy to help you. Now, let's see.
• 3:53
  Here is the million-dollar question, has your internet
• 3:59
  stabilized, Maria Well, do you hear me?
• 4:05
  Yes, that is a first. Success, we do here?
• 4:08
  Yes, success. Yeah, because I had all your
• 4:11
  last talked last several minutes.
• 4:14
  So probably it's more or less stable.
• 4:16
  Now, you know, it's not sound like a problem with Internet and
• 4:20
  overloaded mobile systems as themselves, but also, it is a
• 4:24
  problem of a lot of people coming to the countryside in the
• 4:29
  middle of summer. And it's yeah.
• 4:33
  And it's also. Also, it's raining.
• 4:37
  Another kind of trouble. Yeah, baby.
• 4:41
  Can we try with the video on? And if it gets too crazy, will
• 4:44
  shut it off again. Yeah, let's do it.
• 4:48
  Let's do it. Let's see if it works.
• 4:58
  Oh, I can't stand his myself. Wow.
• 5:01
  So please do wait. Do I need to turn your video and
• 5:05
  yes. Yeah, because they call sculpted
• 5:07
  icons. Let's see if we can figure this
• 5:13
  out. There we go.
• 5:15
  I think I turned it off D. So I got it back on.
• 5:17
  All right, we got it. This is okay, thank you all for
• 5:22
  sitting with us in these hiccups.
• 5:26
  I appreciate it. Now, let's get into it.
• 5:28
  You wanted to talk to us today about leveraging, the ml tooling
• 5:33
  ecosystem. What exactly?
• 5:36
  Which tools are you going to talk about?
• 5:37
  What do you want to talk about? I know you have some slides
• 5:40
  prepared, joint, just jump into the slides.
• 5:44
  Yeah, well, let me start with small intro about it and then
• 5:48
  they will jump to the slides so well and it hurts about this
• 5:54
  talk. And when I Was preparing for it,
• 5:56
  I thought, okay, we have a lot of experience in ml operations,
• 6:01
  and I will show you an ideal project who is ideal pipelines,
• 6:06
  which we use for our clients. And then I started preparing the
• 6:13
  deck and diving into the details because actually I'm head of
• 6:18
  product. I know what's going on, but I'm
• 6:21
  not the person who writes this by blinds and all these
• 6:25
  Integrations. So I started diving into the
• 6:28
  details. I also watched several talks
• 6:32
  from previous meetups and I suddenly realize that this is
• 6:37
  not ideal. I will show you what we have now
• 6:43
  and I will describe briefly, which directions of improvement.
• 6:48
  We see and where are we going with it?
• 6:51
  So anyway, I want to show you how we built a complex system,
• 6:59
  integrating a lot of various tools in the water system on the
• 7:06
  basis of some generic or or and let's jump to my slides.
• 7:11
  Yeah. And as I'm doing that I think
• 7:14
  what you had told me and correct me if I'm wrong.
• 7:17
  I'm sure you're going to get into it.
• 7:19
  The tools that you are using are using a bit of DVC.
• 7:24
  Get Selden you're also using airflow.
• 7:29
  Is that correct? Yes, Griffon.
• 7:32
  Uh. Yep, and I think there's a few
• 7:36
  more in there that I may have forgotten, but that is the whole
• 7:40
  thing, so we can't go. There we go.
• 7:43
  Now, we see your screen. It's why I'm trying to run
• 7:46
  present, but it's oh, that's what it is.
• 7:51
  Awesome. So you see I like the colors,
• 7:54
  which he used, we have the same a metaphorical this.
• 7:57
  Oh, yeah. Yeah, we are. well, I know
• 8:04
  that's represent a vendor here, so I need to be very cautious,
• 8:10
  but still, I will start to little bit about our product
• 8:15
  without there are 30,000 kids and let me start first of all,
• 8:22
  with a task, which were going, which we are going to solve in
• 8:27
  this particular example, which I wanted to show So we are solving
• 8:32
  a very applied problem. One of our departments is doing
• 8:37
  a mail services for different countries around the world and
• 8:42
  we are doing a very applied pro wrestling, a very very applied
• 8:45
  program. We need to develop and deploy an
• 8:48
  object of ignition system or retail.
• 8:51
  So you see to the right the image, what our clients want to
• 8:58
  see is have this all stuff to be found and recognized.
• 9:12
  So the problem here is mostly in the production rather than in
• 9:18
  the machine learning box. So new data arrives daily.
• 9:25
  We need to retrieve the model very often as You're dead to
• 9:28
  rise and we need to redeploy this new models as soon as we
• 9:32
  get one from addiction. Well, the nature of the task is
• 9:35
  that the people which want to have a system, they go around
• 9:40
  different shops, make a lot of new photos.
• 9:42
  You get new kind of soldier trips that say that about you
• 9:46
  have to retrain. So yeah, that's that's the main
• 9:50
  problem. So, To solve it.
• 9:56
  We need some core on top of this core, we built Integrations.
• 10:03
  And as a core, we have our own product which is Euro platform.
• 10:09
  The main part of it is a resource of castration is a
• 10:14
  platform which is based on top of kubernetes.
• 10:16
  Its infrastructure agnostic. You can write it in clouds in
• 10:20
  premise in hybrid installations of Five attempts of the clouds.
• 10:27
  It has some additional perks over kubernetes like SSO, single
• 10:32
  sign-on, role-based access control, it has Ingress out of
• 10:36
  the box so that you don't have to set it up and it has unified
• 10:42
  simple interface and allows me rating between various
• 10:47
  infrastructure becomes very smoothly.
• 10:49
  So like you can, for example, if you are starting Top, which you
• 10:54
  got a lot of credits from Google.
• 10:56
  For example, you can install europea, then you will run out
• 11:03
  of this credits and install our own.
• 11:06
  Your Adobe has installation with another ton of credits and
• 11:09
  migrate very smoothly without changing interface or any habits
• 11:12
  or something. But that's not the point of this
• 11:15
  thing, it's hides their infrastructure from you and
• 11:19
  gives you a bunch of abstractions which can be used
• 11:22
  to build. High levels of of the product.
• 11:28
  So the concepts actually are clusters storage.
• 11:33
  Look at images and drops, four more four main Concepts.
• 11:37
  We work with clusters aggregation of your compute,
• 11:42
  CPU, CPU storage and Docker. Registry.
• 11:45
  Private registry on the platform storage is an obstruction over
• 11:54
  network storage or object storage or distributed file
• 11:58
  system. If we speak about on-premise
• 12:01
  installations, it's just an obstruction that gives you
• 12:08
  another, another view of your file management.
• 12:14
  You do not have to think about particular, emotional particular
• 12:18
  Drive, where your data is or your files.
• 12:20
  Are you just have a uniform. Unified interface to access them
• 12:28
  as parts of your personal name, space or the name space of your
• 12:32
  team like you again. You don't think about
• 12:34
  infrastructure, you just deal with storage which does
• 12:37
  something behind the scene, but you don't have to think about
• 12:40
  it, everything which happens on the platform.
• 12:44
  Kevin's insides Docker containers.
• 12:47
  And we can use images from public Registries as well as
• 12:50
  private version of the platform and in execution, On hearing it
• 12:54
  is called drop. So in the example to the right,
• 12:59
  you see how we run a job, which has a name trained in the second
• 13:05
  level two. Yeah.
• 13:06
  Which has a name train which actually runs Project drink by
• 13:15
  Script, it's is executed on GPU small percent which is a parts
• 13:21
  of cluster. So when you're on a job, you're
• 13:25
  not think again about the particular machine that you just
• 13:27
  need to think in terms of what kind of resources I need to
• 13:32
  these unit to execute. So we're on it on GPS.
• 13:37
  Small preset, which is enclosed usually.
• 13:42
  Katie and with the Tesla would be you, you run it with some
• 13:47
  parts of volume, some Williams, some parts of the storage
• 13:52
  mounted, and on some public, or private, in which, which is in
• 13:56
  this case, your mission base. So that's that's actually these
• 14:03
  are four main Concepts if you want to.
• 14:08
  You can build actually, whatever you want on top of this
• 14:12
  Concepts. If you have enough drive and
• 14:16
  energy and time and decided to do that, you can build anything.
• 14:22
  But the main question is that Enough to solve our task.
• 14:30
  Of course, it's not because it's generic platform which doesn't
• 14:38
  refer to machine learning or machine learning operations.
• 14:41
  To be honest, is just a resource orchestrator, in this part of
• 14:45
  it. So what you need, you need
• 14:48
  tools. And here's an interesting
• 14:51
  question, which I guess any company like ours has The
• 14:59
  customer would like somebody to see all the tools in one place.
• 15:04
  Like, have a unified interface for everything they need.
• 15:10
  For machine learning off of machine learning preparations
• 15:13
  data registry will be the same interface, which is model
• 15:17
  registry with the same interface as pipelines or something else,
• 15:20
  whatever they do not want to have a bunch of different tools
• 15:24
  and to set them up manually. But at the same time, you need a
• 15:29
  tremendous amount of resources to create a product which does
• 15:34
  everything and probably you won't be able to do that after
• 15:38
  all because the tools in this field are changing so fast.
• 15:45
  And you may think about integrating or writing your own
• 15:50
  engine for hi Roberta tuning and then someone like Facebook
• 15:55
  releases and You cool Library, which does the same, but 10
• 15:59
  times better. So you will be, you will already
• 16:04
  be behind. So we realized that we clearly
• 16:09
  are not capable of doing everything by ourselves.
• 16:13
  But why we went into the direction of Integrations and
• 16:19
  then I want to show you some examples of tools how we
• 16:23
  integrate them for which purpose, So if we split it, I'm
• 16:32
  sorry if your splits at their email project process into two
• 16:37
  stages, two major stages like development and production.
• 16:43
  In development, we use these number of tools we use cookie
• 16:48
  cutter for scaffolding and actually, it is hidden from the
• 16:54
  customer, we have a cookiecutter template, which is public but We
• 16:57
  have a commands in our see a lot of which restaurants this
• 17:00
  cookiecutter template if someone is unfamiliar with cookie
• 17:04
  cutter, it's a python tool which allows you to define the
• 17:12
  structure of files and directories, which you would
• 17:17
  like to use for a project like you run cookie cutter from this
• 17:22
  Guitar Hero and you get structure.
• 17:27
  Open your project with some variables replaced with, with
• 17:32
  the ones which you provide W, Carter.
• 17:34
  So it's trans ask you a bunch of questions and creates a template
• 17:37
  for you is extruded popular. And B, then is used for Jungle
• 17:43
  very actively and it's also used in data science, the
• 17:46
  preservative popular data science.
• 17:50
  I don't remember exactly how it's called, but again, very
• 17:53
  easily. Find Cookie.
• 17:53
  Cutter data science, templates, and we created a separate one
• 18:01
  for us, which actually integrates all these other, not
• 18:07
  the tools for the development so that you can run them in a short
• 18:12
  amount from a make file, we run jupyter notebooks for
• 18:16
  experimentation, we're on Dental Board.
• 18:19
  Vote for experiment, tracking by Chan, professional is normally
• 18:24
  used by our developers for photo editing and debugging and for
• 18:30
  the quite sharp professional. Which is I think only not open
• 18:35
  source tool in the list. Table show you, you need to have
• 18:39
  a license first and then you need to have a small setup, but
• 18:44
  it's also very useful and We used to get about management
• 18:52
  because you get this template from kids at you are expected to
• 18:57
  push it. And there's a git, ignore file,
• 19:00
  and the specific structure. So we imply that our customers
• 19:03
  going to use get and for data and Model Management in
• 19:08
  development stage, we use simply our storage.
• 19:13
  In these in these parts. Neither the data.
• 19:17
  No models are very short because we spoke to a lot of people who
• 19:25
  run many experiments in development stage like the place
• 19:29
  when you just try different models and see how it goes, they
• 19:33
  rather interested in running at Jupiter notebook or running
• 19:38
  training with one command rather than dealing with It's a version
• 19:43
  in his mother versioning because actually you are not really
• 19:48
  interested in all this production rate, stuff until you
• 19:54
  get a first decent model. So in this particular
• 19:58
  development template, which is afforded by Cookie Cutter, we
• 20:02
  just use our storage as applicants and that's it.
• 20:06
  So project templates months just to give You some understanding.
• 20:13
  As I said, you run in it, you answer a couple of questions
• 20:17
  from we gotta and you'll get a structure and make file, which
• 20:24
  contains very short commands to run on Europe.
• 20:27
  As you saw, the commands actually pretty long You saw the
• 20:33
  example on the previous slides, but my file adjusts brings
• 20:38
  together all the variables and all the volumes to mount and
• 20:44
  hides all these details from you.
• 20:48
  So that's actually interesting, but not really, because if you
• 20:55
  want to develop, you still can simply go for call up or you can
• 21:00
  go for I don't know, uploading your code to remote server.
• 21:05
  If you have access to it still it requires some ate it.
• 21:11
  We may make it more convenient, but it's not the main problem in
• 21:17
  the world of machine learning today.
• 21:20
  And here, where the problems start real problems, when you
• 21:24
  want to make it in the production.
• 21:27
  And for the production, we decided to integrate.
• 21:32
  Is cool and proven tools from open source world.
• 21:39
  We write pipelines real fast. I was just going to ask you, you
• 21:48
  mentioned. Yes, this here.
• 21:50
  The problem starts when you want to make it into production, why
• 21:54
  is is that Um, because in production, you have to bother
• 22:05
  about so many things, I have to bother about the versions of
• 22:10
  your data. I have to bother about the
• 22:13
  stability of the deployment actually, you have to deploy
• 22:19
  this stuff, somehow somewhere you have to balance the load.
• 22:25
  You have to Monitor if the model behaves.
• 22:28
  Well, after some after sometime, after a lot of new data, right?
• 22:36
  It's just too many things to think about, when you develop
• 22:42
  your not concerns about the quality of the court, you are
• 22:46
  not really concerned about whether it will.
• 22:52
  I don't know. I think that it's where you just
• 22:59
  develop the model you mostly get fun but to any you come to
• 23:03
  production you have to make it profitable for your business and
• 23:10
  here's where the fun stops. We have to make it work.
• 23:15
  Automated stable as a clock and here you're a really needs a lot
• 23:22
  of additional tools. Great point.
• 23:28
  So so we are, there are actually two slides with a production of
• 23:35
  apples. This light is devoted to the
• 23:38
  ones which are platform level, like, not even small tools that
• 23:45
  you can run on the platform, but entire huge, whatever that you
• 23:50
  have to run along the platform. We use air flow for pipelining.
• 23:56
  And technically there is a separate installation which runs
• 24:00
  on the same kubernetes cluster and knows something about the
• 24:06
  storage which has some specific components has taught kids,
• 24:13
  whatever. I'm not sure that.
• 24:15
  What are the questions? Our ml Engineers Emma loves
• 24:18
  Engineers. So it has to run nearby the
• 24:21
  platform. Will you sell them for
• 24:24
  deployment? It's also separate installation
• 24:27
  and sell themselves, a whole bunch of problems for us related
• 24:32
  to not only providing the API in the development mode, we can
• 24:40
  host job on a platform which provides an API using, for
• 24:45
  example, first server. But when it comes to production
• 24:49
  deployment to your needs to balance loads, you need to
• 24:52
  capture a bunch of metrics, you need to monitor all this stuff.
• 24:57
  Seldom does exactly that for us seldom does even more.
• 25:00
  We just, to be honest, don't use all its functionalities.
• 25:04
  We just exploring and we'll use more obvious criticism, like,
• 25:08
  A/B Testing of models, and it's that kind of stuff.
• 25:12
  And we use grow fonder for monitoring, mostly because it
• 25:16
  comes together seldom, but they are already integrated fine.
• 25:21
  So your Instagram fun and Southern Brides looks to permit
• 25:24
  has refineries this. Flocks and that's go smoothly.
• 25:29
  And I can show you how the dogs, the pipeline's for this
• 25:37
  particular use case. Look, we have three pipelines,
• 25:40
  three main pipe lines, one for data pre-processing.
• 25:44
  When you have a new bunch of data you potion and you want to
• 25:53
  merge it with existing data set and Make us bleed to train
• 25:58
  validation and test as usually like just just mix mix and make
• 26:03
  and your data set so this pipeline or separates and your
• 26:06
  data portion. And the base data set to use and
• 26:10
  Grace, if you did the sets, another pipeline takes a data
• 26:16
  set and the model coat and base model.
• 26:21
  In our particular case, we retrain the model.
• 26:24
  So we use some model too. To build another model top of
• 26:28
  it. So it takes a model version
• 26:31
  data, version a cold version, and creates a new model, and
• 26:35
  puts it puts it into the registry.
• 26:39
  And just for you two to understand the most, the most of
• 26:47
  these rectangles, the steps in airflow, they happen right on
• 26:51
  the platform. So that's an elephant
• 26:54
  aggression, which we have with our fault, we have our custom
• 26:58
  plugin, which knows how to work with a platform.
• 27:01
  Despite the fact that are forced out of the stands there beside
• 27:05
  it, besides then your installation are for knows how
• 27:09
  to talk with a platform and how to round trips and they eat and
• 27:13
  how to read storage. And so on so forth.
• 27:17
  I mean we could use vitamin C line but this kind of
• 27:20
  integration is more is more powerful because we can do more
• 27:24
  cool stuff there. So these rectangles I guess all
• 27:31
  of them which are white Yeah, yeah I guess that's why it's
• 27:36
  rectangles are full of has some weird scheme of coloring.
• 27:39
  That's the only problem, which I have those two players are
• 27:42
  Floyd, don't remember, it's their color scheme but I guess
• 27:44
  that's white rectangles around on the platform.
• 27:47
  So training job happens on the platform and all the validations
• 27:52
  before training after training committing the model.
• 27:55
  Everything happens on the blacktop And then the last
• 28:01
  pipeline is, to get a model and to deploy it to inference.
• 28:08
  And you can see that these dogs are pretty large, not only
• 28:13
  because there are a lot of steps involved.
• 28:15
  But also, I mean, a lot of logical steps involved but also
• 28:20
  because there are a lot of checks, like you're on a dog,
• 28:24
  but you have to provide some arguments and if you A good,
• 28:28
  forget them, or if these arguments are invalid, them talk
• 28:31
  should fail and should tell you precisely what happens.
• 28:35
  So if you forget to, if you're on this Dock and forget to give
• 28:40
  a model, then the it will fail and you will clearly see that it
• 28:45
  fails because you forget to provide the model.
• 28:47
  Like the most simple example here.
• 28:51
  And like you get a model you find your validated your run.
• 28:59
  Inference here are monitoring, you can open up the interface of
• 29:05
  each of these tools right from the dark.
• 29:10
  So, for the production, it became the major source of
• 29:14
  information. And also it's worth mentioning
• 29:19
  that to we use meat for keeping the dogs themselves.
• 29:27
  So airflow ducks Stored in a separate key, triple.
• 29:33
  And every minute if I'm not mistaken, airflow installation.
• 29:39
  Pulls the docks and redeploys them if it's needed.
• 29:44
  So the three main ducks and a couple of other ones, which open
• 29:50
  up the ends of the boards and a couple of other tools, Again, I
• 29:57
  see a question here in the chat from Yvonne is asking how many
• 30:02
  models you have in production so far?
• 30:06
  well, in this particular example, 1 So we speak about
• 30:10
  what a bunch of that for one model which is being updated
• 30:17
  perfect, but actually each duck, each each task requires another
• 30:27
  bunch of ducks and another resources.
• 30:34
  And in this particular case, we run all this stuff on a
• 30:37
  dedicated cluster, which is used By our internal team.
• 30:42
  And in this class the right now we're you know we're under this
• 30:46
  model. Our approach is basically to run
• 30:50
  a model or a bunch of similar models per cluster, making a
• 30:57
  cluster in the euro is pretty simple.
• 31:00
  So we opted for splitting splitting resources for models,
• 31:06
  it's easy to Monitor and to understand what happens with
• 31:08
  them. So this these are three major
• 31:15
  pipelines and this is example of graphing, a monitoring.
• 31:18
  You may see that, maybe this graphs are not really
• 31:24
  interesting because in this particular case our Lobster to
• 31:29
  the years around, all these pipelines on a copy of the
• 31:33
  project, not the customer project but on But they but they
• 31:41
  used cocoa dataset and some basic model just to demonstrate
• 31:44
  the stuff. So not not much data.
• 31:49
  Can be seen on this on this Grand Final screenshot.
• 31:54
  Also also lets the first part that's when you need huge tools,
• 32:02
  which run together with the butter amend have to somehow
• 32:06
  communicate with it. Another bunch of those is the
• 32:10
  ones which will use you use for management of various species.
• 32:17
  And the first one of the most complex one, I guess, is data
• 32:21
  management for which we use the VC.
• 32:25
  And we use it together with our storage and what actually
• 32:29
  happens we keep DVC files. You know the DVC upon the normal
• 32:38
  workflow we still see is when you have it but in our case we
• 32:43
  decided to start with a version without get.
• 32:47
  So when you commit something to do see you get a bunch of DVC
• 32:52
  files cases and we kept We keep these files on the platform
• 32:58
  storage, and we have a bunch of directories one version for data
• 33:04
  set version. And we can always grab these
• 33:09
  divisive files and copy them to a predefined location and
• 33:13
  restore the data set. It's actually pretty similar to
• 33:17
  get, but instead of having Russian control stuff.
• 33:23
  You have a bunch of folders which actually works because
• 33:27
  these files are very small and this is not a problem to give
• 33:30
  them alongside the the data. So the pipelines which was does,
• 33:40
  which do the processing. They pull data, according to the
• 33:44
  CDC files, they do whatever they want to do with it.
• 33:49
  And the data is the Your data is at this point, is also kept on
• 33:54
  the storage and the volumes with these data are mounted to the
• 34:00
  jobs. It's works surprisingly fast.
• 34:05
  I must say because This is this is high for both network
• 34:11
  storage, which we normally use and it gives you a high speeds
• 34:15
  and it's not that huge problem that we expected it to be.
• 34:20
  I mean when we have some additional storage some like
• 34:24
  network storage, we would expect that it'd be way too slow to
• 34:30
  make any data modifications but in case of, in our case, it
• 34:35
  turned out to be just fine. So We do all the stuff with data
• 34:39
  on storage and we update the data sets and copy.
• 34:44
  The Visa files to getting a location.
• 34:46
  That's our current workflow. Is it for the code management
• 34:51
  again? Right now, we use storage
• 34:56
  itself, we don't use get, and the code is copied the storage
• 35:01
  during the development stage. The reason for that, at least,
• 35:06
  at the current point, Is that the court changes, very seldom.
• 35:13
  At least point of the development, the data changes
• 35:15
  but the codes have the changes at all.
• 35:18
  So there is some model, which is being retrieved.
• 35:22
  I mean, the court has some water, which is being called
• 35:24
  trained team. Doesn't need a lot of attention
• 35:26
  from ml Engineers, for the model management for use Thunderbolt.
• 35:31
  I will show you a screenshot in a moment.
• 35:34
  And all met information is also kept on storage.
• 35:37
  On the mat. I mean stuff like which model
• 35:41
  which base model and which data set were used to create this
• 35:44
  particular model so that you can trace back how this model was
• 35:49
  created and reproduce it and environment management is done
• 35:56
  through their platform, registry and pre-built, images are copied
• 36:01
  again, the images do not change very often so we just Greed,
• 36:08
  then during the development stage and keep them there.
• 36:11
  So every time you need to change environments environments, I
• 36:16
  mean, Dakka Dakka, images are barriers and call it.
• 36:20
  You just need to copy it from yellow, perfume from your local
• 36:24
  machine to the bathroom storage. That's how it works now.
• 36:30
  So the data registry in this toy example which well Ops Engineers
• 36:36
  prepared for me for this presentation, looks like this.
• 36:39
  You can see the bunch of folders in file browser.
• 36:44
  Well, browser is another tool that we use inside the platform.
• 36:48
  It's very handy way to monitor the files on your network
• 36:53
  storage. So, you can see that there are
• 36:59
  several versions of data and these to create this data
• 37:06
  versions, We have a special script which looks through the
• 37:10
  content and features out the number of the next version of
• 37:16
  the data. So that, I mean, this is
• 37:19
  important because we started with using the current time The
• 37:24
  timestamp times them. But when you have a time stamp,
• 37:29
  it's really hard to compare something.
• 37:32
  And when you have D 4, d, 5 D, whatever.
• 37:36
  It's becomes so much easier. So we decided that it's worth
• 37:40
  writing small script which going to help us to monitor this stuff
• 37:46
  and for more the registry right now, we use pencil boards which
• 37:51
  runs on top of the folder where all the All the locks are kept.
• 37:57
  So we can always compare different versions of models.
• 38:00
  Like, in this particular example, I have three and zero
• 38:03
  and one and two and I can see the Matrix I can see the number
• 38:07
  of it will blow sent, all that kind of stuff. so, That's
• 38:16
  definitely not the end of it because we understand that there
• 38:22
  are a lot of issues in this particular Pipeline and this
• 38:29
  solution and the most important thing that we want to do and we
• 38:33
  will do pretty soon. We want to make a prophecy, ICD
• 38:38
  someone recently, shared a nice story in the chat in Slack.
• 38:45
  Of the community which, which showed three types of Emma loves
• 38:53
  maturity for a team when you do everything manually, when you
• 38:58
  have some pipelines, but you're still around the manually.
• 39:01
  And when you have CC ICD, like in this particular case, which I
• 39:05
  show you, we run pipelines manually.
• 39:09
  And what we want to do, we want to have a CI CD, of course.
• 39:13
  So we want to use get. And so we want to have a proper
• 39:19
  automated Behavior. Like new data comes, great racer
• 39:24
  train and no new data. Chunk comes, let's make a new
• 39:28
  data set. Your data set appears in get
• 39:32
  through DVC lights, retrain, the modern to keep it inside the
• 39:36
  registry and you model. It appears again, let's return
• 39:41
  to keep it into the registry and you modal.
• 39:44
  Register a let's deploy in death and Waterman's production
• 39:48
  environments is still a question and we think that it should be
• 39:52
  done Mary on the manually, but you have to understand what you
• 39:57
  are doing. When you are trying to deploy
• 40:00
  production model. But for these for these
• 40:05
  development stuff, we actually have to implement this this
• 40:13
  computer continuous integration computer continuous deployment
• 40:16
  stuff. And also, there are some
• 40:19
  improvements which we can Envision right now and I think
• 40:22
  that there will be more that we oscillate like we could build
• 40:27
  Docker images right on the classroom, instead of copying
• 40:31
  them. Being with them because well
• 40:34
  it's somewhat makes it harder to reproduce because you may lose
• 40:40
  the image in the prior charges through after a while you may
• 40:42
  delete it. But if you build an image from a
• 40:46
  Docker file, every time you want to run the pipeline and then we
• 40:49
  are just fine. You can reproduce in exactly the
• 40:52
  same environment which you had two months ago or seven months
• 40:57
  ago, whatever. And we are not really satisfied
• 41:01
  with our Moderator. See solution.
• 41:04
  We are looking into something like a velcro and we'll probably
• 41:08
  use it or maybe something else. There is plenty of modern
• 41:13
  register tools on the market right now.
• 41:16
  So we're just creating current looking for a best one, for this
• 41:20
  particular case, So that's basically it which I wanted to
• 41:26
  show you today. Thank you so much for that.
• 41:31
  I have a few questions. I'm sure other people have some
• 41:34
  questions too. Yeah, and I really think it's
• 41:38
  fascinating how you've been able to build one, level up, and take
• 41:44
  all of these tools and put them together so that they work
• 41:48
  seamlessly and people don't have to think about, you know, Can
• 41:52
  standing everything together. So my question comes in.
• 41:57
  How long did it take you to put this whole pipeline together and
• 42:02
  create this whole product? Because I know there's a lot of
• 42:05
  people out there that are going through the whole build verse by
• 42:08
  question. And when they think about the
• 42:11
  build I'm sure it would look or some people have thought about
• 42:16
  this this exact solution, right? Like oh well, maybe I could
• 42:21
  piece together DVC. See.
• 42:23
  And put some Selden and then I monitor with profound.
• 42:27
  No, but I'm wondering. What, how long have you been
• 42:31
  doing it? What were some mistakes that you
• 42:34
  learned along the way? Well, all I have to, I'm trying
• 42:44
  to remember, actually, I remember that initiative when we
• 42:49
  wanted to make this particular pipeline, it was like, okay,
• 42:53
  well a lots about these tools. Let's make a pipeline for these
• 42:58
  tasks are two weeks enough. Yeah, fair enough.
• 43:03
  Let's do it. It was a guess for months ago,
• 43:08
  something when we started doing it, actually, the sad thing
• 43:13
  about amylose is that it takes so much time to set this stuff
• 43:18
  up. Really a lot of time and we had
• 43:22
  to experiment. We had to think about stitching
• 43:29
  all this stuff together and make a I think about making it
• 43:34
  useful. Because one thing is what our
• 43:39
  envelopes Engineers want to see. Another thing is what clients
• 43:44
  want to see because they are going to use this by blinding in
• 43:47
  relatives. We need to provide some
• 43:48
  additional stuff for them, which is not there yet.
• 43:51
  Like, they clearly do not want to upload a data chunk to the
• 43:57
  storage and then run a pipeline manually when it's so prepare
• 44:01
  some cool web form for Them to upload and do everything else,
• 44:07
  they're magically, you know. And another think what am L
• 44:12
  Engineers, which create the model itself want to see.
• 44:16
  So we are trying to collect all these opinions and improve
• 44:20
  Improvement and growth and I think that it's gonna take
• 44:23
  another couple of answer, to the very least, to make it
• 44:27
  something, which I will probably finally call Ideal And what do
• 44:32
  you think was some of the stuff that held you up and you because
• 44:39
  you budgeted for two weeks? But then it's taken four months
• 44:44
  or so? What was it?
• 44:46
  That that came up that you didn't foresee all well, it took
• 44:54
  a lot of time to understand how data versioning works.
• 45:01
  Because it was not really clear from the first from the first
• 45:05
  view. Another serious issue was how
• 45:11
  to, how to save information for being able to reproduce an
• 45:19
  experiment because you need to drag the exact versions of
• 45:26
  everything called data model itself environments.
• 45:31
  And stuff to be able to reproduce it and with that for a
• 45:37
  while how to do it correctly that we tried a couple of
• 45:41
  different ways and this version management stuff also took some
• 45:46
  time. Like when you start with
• 45:49
  timestamps, it looks so obvious. But as long as the amount of
• 45:55
  versions gross, you said, no understand that you can deal
• 45:59
  with it because You just don't do, can't read all this stuff
• 46:03
  through. It's just not human friendly.
• 46:05
  Yes. Oh yeah.
• 46:07
  And maybe some other things. Oh, to be honest, it's I'm
• 46:11
  speaking about what I participated to you, but there
• 46:15
  is also a staff from infrastructure site like how to
• 46:19
  install seldom appropriately. How to make seldom work with our
• 46:24
  storage, how to make airflow and We'll just install air flow
• 46:33
  provides correct level of permissions and make it.
• 46:41
  I don't know what exactly was the problem.
• 46:43
  I don't remember, but there were some installation issues, quite
• 46:47
  a lot of them, so that was another team who was working
• 46:50
  with that. But still, it took a lot of
• 46:51
  time. Yeah, I believe it and that's
• 46:54
  something that I know it's. It seems trivial, would you
• 46:59
  looking at you go. Yeah, we just piece these few
• 47:03
  open source tools together and in two weeks we'll have it
• 47:06
  working. That'll be fine.
• 47:08
  But we already have them. What's a problem?
• 47:11
  We can do. Like, we also don't use the
• 47:13
  vcu's mo flow of the, of course, yeah, it should be easy, but
• 47:20
  it's never that easy, right? Absolutely not absolutely not
• 47:26
  and it's not that I would assume that this is our team's problem.
• 47:32
  Like, maybe we are stupid, we can't deal with it.
• 47:35
  But actually we talked with a lot of other teams and most of
• 47:40
  them, they need several months after half a year to create a
• 47:44
  proper pipeline in different tools.
• 47:47
  People, use cute flow, people, yours other stuff.
• 47:51
  It's just hot, it just as it is for now but today.
• 47:56
  Yeah. And is there any specific reason
• 47:59
  that you feel? It's is so hard, is it because
• 48:03
  it's not mature? The tools aren't as mature or is
• 48:06
  there something else that you think is it?
• 48:08
  Because just machine learning is so difficult and there's so many
• 48:13
  pieces involved in it. What are your thoughts are on
• 48:15
  that? I think that the main problem is
• 48:18
  in the domain is in much today. I think because in traditional
• 48:23
  develops, you just, I know that it sounds, so it would sound
• 48:29
  strange 20 years ago, but actually you just need to deploy
• 48:32
  the code and here you need to like, you need to train a model.
• 48:39
  Now, you need to keep your data somewhere and somehow and it's
• 48:44
  updated data sets. You need to train your model.
• 48:47
  You need to wrap this model into API.
• 48:51
  You need to host this API. You need to balance the load and
• 48:55
  monitor their health of the model which I don't think that
• 48:59
  devops ever had this problem. That your API make bright not
• 49:04
  break with become worse because all the time it's a specificity
• 49:08
  specificity of machine learning so it's just harder it just
• 49:13
  involves way more pieces. Ah and additionally, you may
• 49:17
  have expensive GPU resources for all.
• 49:21
  Stuff and do you need to somehow optimize the usage that RGB or
• 49:24
  resources? Otherwise you will just get were
• 49:28
  huge bills for infrastructure so it's just way too many pieces
• 49:34
  and way too many issues. Yeah.
• 49:36
  And as you say that expensive GPU, have you thought about for
• 49:41
  the product? Because I know this is something
• 49:43
  that we were looking at dot science being able to monitor
• 49:49
  like Which models are are using the most resources and, and
• 49:55
  who's, who's working on each model and being able to like
• 49:59
  graph that out and say, okay, this this bottle is, you know,
• 50:05
  is it really effective because it's taken up this much Well, we
• 50:11
  can do something like that. We can build reports for the
• 50:16
  development stage because we okay, we can understand which
• 50:23
  project, and which people spend the most of GPU resources for
• 50:27
  training for deployment support, actually, we didn't dive into
• 50:34
  that yet but think am. That's a good point will
• 50:38
  consider that No problem. I've got slutty other ideas if
• 50:43
  you want to talk to me later. Oh yeah.
• 50:45
  Oh yeah. Please do perfect.
• 50:49
  Well, does anyone else have any other questions of about let
• 50:56
  little space for someone if they want to ask anything and If not,
• 51:04
  we will. Go ahead and I think we're,
• 51:12
  we're going to call it good there.
• 51:13
  I want to just reiterate big. Thank you.
• 51:16
  It looks like our internet worked.
• 51:19
  And oh yeah. We had success and if anyone is
• 51:23
  onerous. Oh yeah.
• 51:26
  I just realized it's I didn't answer the question about the
• 51:29
  Betances. Yes.
• 51:33
  Actually I wanted to tell about 22.
• 51:37
  Connects which we used to deal with dependences with
• 51:40
  dependencies hell in Python in particular, like two things are
• 51:46
  the first one is that we run everything in to occur.
• 51:50
  So, in the case, when you have, I don't know.
• 51:56
  Some model, which requires skewed and nine Gudda Gudda 9,
• 52:02
  and another model, which requires good 11, and it doesn't
• 52:04
  matter, which requires, I don't know what else, dog had just
• 52:08
  saved your life and we have seen such kind of pipelines and we
• 52:15
  know that it helps. So dull care is way to go to
• 52:20
  separate the environment for different steps of the pipeline,
• 52:24
  the development process. And Another thing is that you're
• 52:28
  still needs to keep a lot of dependences for the tools, for
• 52:33
  the Frameworks. First of all, inside one
• 52:36
  environment. And in this case, for the
• 52:39
  development purposes, we use exact versions.
• 52:46
  So we keep all the versions with eagles, eagles in requirements
• 52:52
  60. And we use Dependable to update
• 52:55
  them. Because without dependent board
• 52:57
  you will you will hardly be able to update these requirements,
• 53:02
  txt file, as soon as you created, but if you have it, at
• 53:06
  least you can always grab all the dependencies on the packages
• 53:11
  which you installed and create requirements, txt file with
• 53:15
  everything. You have in this particular
• 53:18
  environment with exact versions and at least be sure that this
• 53:23
  stuff works and to won't break. Model because new package
• 53:27
  arrived. So we use this.
• 53:30
  And depending on what is an essential tool for that, for
• 53:33
  updating this stuff, but you updated.
• 53:36
  When you want to do that, not when the library's maintainer
• 53:40
  want to do that. So it depends on boards.
• 53:44
  Plus Docker, we are fired for this.
• 53:48
  Thank you so much. That was my burning question and
• 53:52
  I am happy to hear about that. I had load about it it's
• 53:57
  Dependable words is what you said but but Dependable what is
• 54:03
  GitHub tool actually which monitors the dependences if they
• 54:10
  updated and it automatically creates pilar's with an update
• 54:16
  for the package and for the related packages for its
• 54:18
  dependencies. And so you can always first, be
• 54:24
  sure that your tests run the It creates the pr and the other
• 54:27
  strong on top of that. And you can check it with your
• 54:32
  eyes with your hands. And only, when you merge the pr,
• 54:36
  then you're 100% sure that you did it.
• 54:40
  Nice. It's really much.
• 54:46
  Perfect. Well right on time.
• 54:49
  I think that's all we got for today.
• 54:52
  Thank you very much Maria. And for anybody, that is not in
• 54:56
  our slack. Go ahead and jump in there.
• 54:59
  Maria is on there. She's very active helping us all
• 55:03
  learn more. If you have any questions about
• 55:05
  anything ml Ops in general you can drop them into the ml Ops
• 55:09
  questions and answer slack Channel and otherwise we will
• 55:14
  see you out there. Go ahead.
• 55:15
  Had. Yeah, big round of applause for
• 55:19
  you. Maria.
• 55:20
  I appreciate you coming on here and joining us today.
• 55:24
  We will see you next week with a bit more of a deep dive into
• 55:30
  bottle monitoring. We're going to be with Arpita,
• 55:33
  and she's going to talk to us about how she does model
• 55:38
  monitoring. I do.
• 55:39
  We did it last week and we're going to do it again next week,
• 55:43
  so it's a big theme this. As days and yeah, it's a very
• 55:49
  hot topic and I think it's something that people really
• 55:52
  enjoy learning about so that's it.
• 55:56
  Take it easy. Everybody have a great night or
• 55:58
  day or wherever you are enjoy. We'll see you later.
• 56:02
  Thank you. Bye.