Latest / Agentic Conversations (formally mlops.community) / Our 1st MLOps Meetup // Luke Marsden // MLOps Meetup #1

Transcript

Our 1st MLOps Meetup // Luke Marsden // MLOps Meetup #1 · Agentic Conversations (formally mlops.community)

0:00
So the talk I wanted to present today is really an introduction
0:02
to what is interlocks. And obviously, if you came to
0:05
this Meetup, you probably have some idea and you probably have
0:08
an interest in it. But I wanted to sort of start
0:11
start this study sessions with with an introduction.
0:17
And I also wanted to put a little spin on it, which is how
0:21
will a my lips properly done. Actually help me work from home
0:25
because it's, it's now becoming The New Normal the especially
0:28
for well, but it's becoming the new normal that everybody in the
0:32
world is working from home which is kind of crazy kind of
0:34
interesting. And in particular if you're a
0:39
data scientist or someone managing data scientists then
0:43
been working from home, working remotely, and doing a
0:46
synchronous collaboration can be particularly challenging because
0:50
of some of the complexities. So I'll also talk about that and
0:55
then I'm going to do a live demo so please could everyone pray to
0:58
the demo gods, Going to attempt to live demo of some of our
1:02
collaboration tooling and then hopefully we have a good chunk
1:06
of time afterwards for Q&A. So to jump straight in.
1:11
Then what is ml Ops? Ml Ops is the intersection of
1:16
these three disciplines, devops software, engineering, and
1:20
machine learning. And so I'm going to talk for a
1:23
short time about each of those disciplines in turn.
1:25
And then we're going to talk about how the intersection of
1:29
them is Is kind of interesting and complicated.
1:33
So if we look at how software engineering has been done, I'm
1:36
not going to spend too long on this, but the basic Point here
1:39
is that software. Engineering has changed a lot in
1:44
the time that it's been even a discipline.
1:48
And we're used to all log into big mainframes, we used to email
1:53
patch files around to each other.
1:55
And the way that, that collaboration happened for For
2:00
engineers used to very much be around manually, integrating
2:05
changes to source code. This was prior to Version
2:08
Control prior to things like subversion.
2:10
And for the longest time, the Linux kernel, for example, was
2:14
developed by people just emailing patch files around and
2:16
maintaining their own local copy of the source tree and that was
2:20
painful for obvious reasons and it took a long time to integrate
2:23
things and then subversion came along and we got to the point
2:29
where we had Had Version Control and you could check in and check
2:32
out things. And that was a great
2:33
Improvement. And then the additional sort of
2:36
improvement beyond that is this notion of distributed Version
2:39
Control where you can sort of do, social coding on GitHub.
2:45
And that's kind of, in my mind, from a collaboration
2:50
perspective, where, where we've got to from a software
2:53
engineering perspective. The next piece is devops.
2:58
And so, if software engineering is how we create software than
3:01
devops is, how we deploy it and operate it.
3:05
And devops has also undergone some significant changes in the
3:09
last few decades and years. I am personally responsible for
3:16
being one of the people who used to edit code live on the server.
3:19
I don't do that anymore, but but that used to be the way that at
3:24
a lot of things were done. People also used to build
3:29
binaries and email them around. So people used to, for example,
3:33
send War files, email a file to us to a sysadmin and get them to
3:39
deploy latest version of the software on the server and
3:43
obviously that had problems. And then this thing happened
3:47
that was called devops and devops is about breaking down
3:50
the silos for between Dev teams and Frances and and and so it
3:58
was kind of a change in the way that teams are organized and
4:01
it's also a change in the way that the software is deployed
4:05
and managed. So now, the people who write the
4:09
software are responsible for shipping software and operating
4:13
it in most World Forum, Dev Ops teams and continuous integration
4:18
as a thing. So software gets continuously
4:20
automatically tested. Whenever changes are pushed to
4:24
version. Control.
4:24
There's also been a big move away from running your own
4:28
Hardware, towards public cloud, and more.
4:32
Recently, there's been a big move towards using a beautiful
4:35
infrastructure like Docker kubernetes and even this notion
4:38
of get Ops, which is that you should also store the
4:41
configuration for how your software is operated in Version
4:44
Control as well as the source code for it and then Source
4:52
control becomes your source of Truth for Or the software that
4:57
the infrastructure, the operating, and that has some
5:00
obvious benefits. And then finally, the third
5:04
discipline that's intersecting here is machine learning and
5:09
machine. Learning is basically how we use
5:10
data and maths to train models to make predictions about the
5:14
world. And I won't go through the whole
5:17
history of machine learning here.
5:21
But just to say that sort of There was this thing that's
5:27
called the AI winter, which is where there was a lot of hype
5:33
around a, I kind of the first time that I think it was in the
5:35
70s people in the audience May well.
5:38
Correct me on that and there was a lot of hype around Ai and it
5:46
turned out that the, both the algorithms, but also, the
5:48
hardware wasn't wasn't good enough was a little bit like,
5:50
the.com bubble in a sense that the Internet speeds back in
5:55
2000, weren't good enough and the harder, I wasn't good enough
5:59
to the connectivity wasn't good enough to support all the hype
6:02
that was going on about the about the web.
6:06
And and this is AI winter. Let's do this sort of huge peak
6:13
in high penis around machine learning and AI that then kind
6:18
of collapse from people realize that it wasn't actually going to
6:21
be able to deliver on all the hype.
6:23
And we've kind of gone around that.
6:24
Cycle once already and now we're in another hype cycle around Ai
6:27
and machine learning. And now this time around, it
6:29
does seem like it is it is set to seriously change the world
6:37
really and to change the way that and it's starting to get
6:42
adopted very seriously inside lots of large Enterprises which
6:46
is which is interesting and I also just wanted to have an odd
6:53
here too. The picture in this slide comes
6:56
from a local Bristol company called graph core and they make
7:00
some very beautiful pictures that are representations of the
7:05
visual. Representations of how neural
7:07
networks actually work which, which are pretty cool.
7:11
And so basically deep learning is now becoming and machine
7:13
learning in machine learning and deep learning is becoming
7:16
computationally feasible. And there's this huge rush that
7:22
all the Enterprises around the world are in the kind of Act in
7:24
an arms race to get value out of AI.
7:29
And so we end up in this place where machine learning is
7:33
effectively emerging from research and it's being
7:35
productionize door, it's like companies are attempting to
7:38
productionize it and this means that ml is emerging from
7:42
research and that these disciplines of software
7:46
development devops and machine learning needs to converge so
7:50
that people can build models and they can deploy them.
7:54
And they can have the same level of rigor and reliability that
7:59
software has in devops. And and so this convergence of
8:04
these things will be called and is already called ml ox and
8:08
that's why we've named our community.
8:11
After this new emergent term I personally, believe that Ai and
8:16
machine learning has the potential to reinvent the global
8:19
economy and hopefully in a good way because I think it has the
8:24
has the potential to eliminate a great deal of boredom, that is
8:28
currently suffered by humans in doing very, tedious jobs.
8:33
It's kind of outside of the scope of this talk to talk about
8:36
the political changes that are going to be necessary to to make
8:40
that non-disruptive. But anyway, So I think we can
8:45
all agree that AI has the potential for good.
8:49
And as a discipline, what we found is that it's extremely
8:54
immature so it's kind of like the wild west out there and
8:58
we've seen damaging levels of chaos and pain as companies go
9:02
from what we call playing with AI to actually trying to use AI
9:06
to deliver business value, which is often referred to as
9:10
operationalizing, AI And so some of the things that we've seen
9:15
that cause these pain points, things like models being blocked
9:22
before they can even get to deployment.
9:25
So so we spoken with with many companies that are trying to do
9:31
AI in production and one of the things that is a very common
9:35
theme is that we just can't, we can develop the models on our
9:38
laptops but we but we just can't get them deployed into
9:42
production. And of course, that sucks
9:44
because that, then the time to value is very, very high.
9:48
If it takes a long time, or if it's not even possible at all to
9:51
deploy models, or even if it takes three or six months to
9:56
deploy a model, then then by the time you've got the model
9:58
deployed, the data is already stale and and often AI projects
10:03
in companies, get grounded as a failure just because they can't
10:07
get out the door. Another problem that we've seen,
10:11
is that Models aren't monitored as actually item number 6 on
10:18
this list and so we had one company tell us that they
10:21
deployed a model to production and then the model went wrong
10:25
and they lost an unmeasurable amount of money as in.
10:28
They actually couldn't figure out how much money they lost
10:30
because the model of gone wrong because they weren't monitoring
10:33
it. And there are unique challenges
10:36
around monitoring models that are different to monitoring
10:39
regular software microservices that I'll talk about in a
10:42
minute. Some other problems that we've
10:45
seen that people end up wasting a lot of time.
10:48
So if there are multiple data scientists or machine learning
10:51
Engineers, collaborating on trying to trying to work on a
10:56
Model together, then a lot of time is wasted just trying to
10:59
set your development environments up so that they are
11:02
similar enough to that. Your work is that the work with
11:08
someone does is compatible with your environment or Spending a
11:12
lot of time, copying data sets around between different
11:15
machines. It's also kind of ties into the
11:20
third Point here which is a lot of inefficient collaborations.
11:23
So we've heard of people saying things like, oh, we just keep
11:28
our data science team small and we keep them in the same room
11:31
so, so they can collaborate by talking.
11:36
And obviously in the current climate that is particularly
11:39
challenging because you don't want to put people in the same
11:41
room anymore. And, and even using paper
11:46
notebooks to keep, That gives the work that they're doing in
11:48
the results that they're keeping track of.
11:52
And, and that relates to the fourth point, which is manual
11:56
tracking. We've seen an awful lot of
11:58
companies that they put the code, like, the, some reference
12:04
to the version of the code and the data and the premises, they
12:07
use when they're training a model along with the whatever
12:09
like metrics like accuracy score for that model, they either put
12:13
that in a Wiki or on paper or on a Or in a shed spreadsheet.
12:18
We've seen an awful lot of Shad, spreadsheets kicking around.
12:22
And there's another problem with those with those spreadsheets is
12:26
that there's a lack of reproducibility and provenance
12:29
and what this means is that you can't go back from a model
12:32
that's running in production to well, what data was a trained on
12:36
and wear? What parameters were used like,
12:40
who even created this model? Do they even still work at this
12:43
company like being able to keep track of?
12:47
The work that went into creating a model causes lots of
12:50
complexity, when when when you're trying to scale up a team
12:57
or when the number of models that you have increases.
13:00
But the key Point here is that if you can't deploy your models
13:03
into production at all to begin with, then you're not going to
13:06
get to the later problems of a lack of reproducibility and
13:10
provenance. And so actually the most
13:12
fundamental thing we need to solve in ml Ops is making it
13:15
easy to get models into production in the First place.
13:20
And so this is kind of oddly familiar, because we've been
13:23
here before, and if you look at how software engineering was in
13:26
the 90s, it was very siloed, not everything was in Version
13:30
Control. There is no continuous delivery
13:33
and it took a long time to ship software when it was being
13:36
manually integrated and when they were these long like Q&A
13:40
and release cycles and now software ships in visit in
13:43
minutes in sometimes, in some cases seconds Modern tech
13:47
companies that are shipping software, will often say when
13:50
they on board, a new employee. Well, on your first day, your
13:52
job is to make a life, change the products or small one, but
13:57
push it through the continuous delivery Pipeline, and get it
13:59
out. We make dozens of changes every
14:03
hour in some cases, two different microservices, but but
14:08
AI is kind of stuck in this pre devops world of collaboration.
14:15
And a large part of that is because the tool.
14:17
In isn't up to scratch. So we have this kind of
14:21
Manifesto. So this is what we care about
14:24
and why we get out of bed in the morning is because we are we're
14:29
working on trying to solve this ml Ops Manifesto and the
14:35
manifesto is in the form of four tests.
14:37
So you can kind of apply these tests to your own ml Ops
14:41
pipelines. And you can still form an
14:44
opinion about how mature you are against.
14:48
The different, the different requirements here.
14:51
So the first requirement is that your model training and
14:58
deployment pipelines have to be reproducible.
15:00
Well, that means a good test. For this is if I can come along,
15:05
nine months later if someone else can come along, nine months
15:09
later. And retrain, a model that was
15:12
trained by somebody else who without even talking to them
15:16
with. Let's say an old version of
15:18
tends to flow on an old data set with on on Hardware that is
15:26
sufficiently equivalent that they can retrain the model to
15:30
within a few percentage points. Then you've got a reproducible
15:33
ml Ops Pipeline and if nine months later you can't because
15:37
you upgraded the version of tensorflow on your development
15:40
machines and the data's gone somewhere and you don't know
15:43
where the data is gone, the data has changed in your production
15:46
data. Is then you failed the
15:49
reproducibility test and if you fail the reproducibility test,
15:51
then you're in trouble from a governance and compliance
15:56
perspective in some Industries. The second test is is your ml
16:01
Ops pipeline accountable and we talk about accountability from
16:05
the same perspective that we hold humans accountable for
16:08
their decision-making process. And one of the ways in which you
16:11
do that is you say on what basis did you make your decision and
16:15
the on what basis Kitchen with machine learning as a minimum
16:21
requirement, not even going into the whole area of explain
16:25
ability, but as a minimum requirement, you have to be able
16:27
to say what version of the data was the model trained on.
16:30
So you need to be able to track the model, back to the
16:33
provenance of where that model came from or what data was
16:36
trained on by whom and and so on the next point and it's
16:41
especially pertinent at the moment, is this collaboration
16:45
requirements. So, It has to be possible to do
16:48
asynchronous collaboration, and this is something that software
16:51
devops has got sorted and ml Ops doesn't yet, mostly in this
16:56
means that I need to be able to, if, for example, if my colleague
17:05
Chris, is working on a Model, I need to be able to make a fork
17:10
of that model and I need to be able to make changes to it
17:14
without treading on Chris's toes.
17:16
So we both Both need to be able to collaborate asynchronously
17:19
and and get useful work done. Now, this has kind of influenced
17:26
the design of what we're building to a large extent
17:29
because because we believe very much in the sort of GitHub.
17:34
Pull request style of collaboration that the data
17:36
scientists are familiar with. And and there are some
17:40
challenges in making that possible for 4 ML, and then
17:44
finally, the model development, Has to be continuous.
17:48
And so there are a couple of things that I mean by this.
17:51
The first one is that the development process must be
17:55
automatic, the deployment process.
17:56
Sorry must be automatic. So it must be possible to
17:59
automatically deploy a model into a staging environmental
18:04
production, environment without manually.
18:06
Emailing jupyter notebooks or or tensorflow files, serialized
18:13
test flow models around because as soon as you Start doing
18:17
things manually. Then it introduces this
18:20
possibility for the human error and the other piece is that you
18:24
have to be able to statistically monitor your models.
18:27
And this is interesting because monitoring models is
18:32
specifically is quite different to monitoring regular software
18:36
that you might, deploy it as microservices.
18:39
And the reason for that, is that When you monitor software, you
18:44
can monitor things like Layton sees an error rates.
18:48
But when you monitor Microsoft, sorry, when you monitor models
18:52
machine learning models, they can be giving you perfectly
18:55
normal Layton sees and perfectly normal error rates and the model
19:00
can have gone completely haywire.
19:03
And the reason for that is basically, if you already knew
19:07
the right answer for what the model was predicting.
19:12
Then you wouldn't need the model.
19:13
In other words, the production data is unlabeled and so this
19:19
means that it's challenging to understand the behavior of some
19:23
of your model once it's running in production.
19:25
So an example might be that I might have deployed a model for
19:34
for autonomous vehicles, that classify road signs.
19:37
And so you might have a bunch of Cars driving around with models
19:44
running on Hardware, in the cars and sensors cameras, basically
19:48
on the cars that are looking around for the road signs.
19:51
And if you already knew what road sign, the sensor was
19:54
looking at, then you wouldn't need the model, right?
19:58
But at the same time it means that it's hard to understand the
20:01
behavior of the model of production.
20:02
There are some solutions to this including looking at the
20:06
statistical distribution of the classifications.
20:08
The model is making if it's a classifier. then you can say
20:12
well if the actual distribution of classifications drifts vary
20:17
significantly from my expected distribution like the
20:20
distribution that I used in training in the training set
20:25
then maybe page a human like fire and alert and get a human
20:28
to look at what's going on because either you deployed a
20:31
bad model in which case well you need to know about it so that so
20:37
that you can roll back and so that you can figure out what
20:40
went wrong with the new Floyd model or the world changed and
20:47
especially with things like computer vision.
20:49
It's it's often surprising like how the models actually
20:54
distinguish features in the data and and you can get stupid
20:59
things like the computer. Vision model might never have
21:04
classified any, or never never be trained on any stop signs in
21:07
the snow. And for some reason, it can't
21:09
classify stop signs in the snow. So suddenly a snow Is over a
21:13
large part of the country and then your stop sign classifier
21:16
stops working and obviously you're in trouble.
21:18
So you need to have that statistical monitoring So those
21:24
are the requirements. And so I'm going to talk about
21:28
how how we can try and address those requirements using ml Ops
21:33
tools. And and just before I do that, I
21:37
want to talk a little bit about how the the software life cycle.
21:43
Sorry the ml, the ml Ops life cycle is just fundamentally more
21:47
complicated than the software development life cycle.
21:50
So, when you're doing software development and Life is easy,
21:55
kind of it's easier anyway, because all you need to do is
22:00
you need to think about versioning, specific versions of
22:03
your code because as long as your doctor Rising things, then
22:09
you ought to only need to know exactly which version of the
22:13
code was used. Plus the doc file and
22:15
dependencies in order to reproducibly, build the same or
22:22
Effectively the same Deployable artifact and so you can build,
22:27
you can create code, you can test that code in CI, you can
22:32
deploy that code into production and then you're monitoring
22:35
system. Can tell you how.
22:36
Well, it's performing, maybe you need to upgrade a database.
22:39
Maybe you need to go and optimize a certain code path,
22:43
and you can go around this loop as quickly as you can.
22:46
And that's basically what we're doing with a box, but with
22:48
machine learning in ml Ops, you've got this fundamentally
22:51
more It thing going on which is that you've got data coming into
22:55
the system, which is a major form of entropy and you've got
23:01
code. That's that's being written to
23:04
train the models and you've got parameters, which are being fed
23:07
in to the training. The models, the model runs the
23:12
training model run, the model training run, sorry and all of
23:15
these things combined and before you even trainer model, you've
23:20
also got data runs which is when you're doing Feature
23:22
engineering, or your filtering, or splitting the data.
23:27
And so, you're executing a certain version of a piece of
23:29
code, against a certain input dataset, and creating an
23:32
intermediate data set. That's what a data runners.
23:36
Or you're, you're running a certain version of a piece of
23:41
code against an input data set like a test training, validation
23:47
set, and then you're creating a model as an output.
23:51
And When you're doing ml, when you're developing machine
23:55
learning models, you are doing these data runs and these model
23:57
runs whether you're keeping track of them or not.
23:59
These might be the cells that you running your jupyter
24:01
notebook, or the Python scripts that you run on your laptop.
24:04
And these runs are happening. You just haven't given a name to
24:08
them. And so what we've realized is
24:11
that it's really important to introduce this notion of run as
24:14
a kind of a building block. It's a fundamental object in the
24:18
system. When you're doing ml odds, it's
24:20
this version of this piece of code grand.
24:22
In this environment with this input data and these parameters
24:25
and it created this output and one of those types of outputs
24:29
like I said is models and then the model itself, the serialized
24:32
model file is the Deployable artifacts, it has some metrics
24:35
associated with it like how well it performed against the
24:38
validation set and then it's that model that you're deploying
24:40
into production and then your monitoring in the way that I
24:45
described the monitoring models is different to monitoring
24:48
microservices. And then you're monitoring my
24:52
Actually, cause you to go back around all sorts of different
24:54
Loops through this diagram so that's kind of the flow that
24:59
were looking at here. So I'm a little short on time.
25:03
So I'm going to skip over the life cycle quickly.
25:09
The life cycle is really just a visual representation of how the
25:13
diagram. I just showed you is cyclic and
25:16
so you do data engineering need to keep track of the data runs
25:20
when you're doing things like feature engineering.
25:22
And then you train a model, which means you take a certain
25:24
version of the data and you, you train the model and out of the
25:27
model development process. Come serialized models,
25:32
typically Docker images, with serialized models, baked into
25:35
them and then those serialized models get run in production or
25:39
staging and then production. And then there are these kind of
25:43
to feedback loops that's the statistical monitoring feedback
25:45
loop which is the first one where you can say, show me the
25:49
behavior of my model in production in real time based on
25:52
The distribution of classifications it's making.
25:55
And then there's also this slow feedback loop which is that you
25:57
retrain the models with new data as you get new data coming in
26:00
and of course the model making decisions can change the world.
26:06
And so can also impact on the data that is being recorded in
26:10
the database. So so that's kind of the, the ml
26:16
Ops life cycle. And, and hopefully I've
26:18
adequately answered the first part of the talk title now,
26:22
Which is what is ml Ox my opinion is that ml Ops is not
26:27
just about operating models. It's actually about the entire
26:30
life cycle of doing data, engineering, training models,
26:34
and then getting models into production and it's about being
26:40
able to implement this process in in your company, or any
26:43
organization with the same level of rigor, that sort of
26:48
best-in-class, devops and software engineering teams.
26:53
Managed. And so let's kind of take a step
26:57
back that and say well how can how can properly implemented ml
27:01
Ops help me in in the current Global situation that we have
27:07
where we're now all suddenly working from home and in
27:10
Brackets. If I'm a data scientist or a
27:12
manager of data scientists, well the answer to that question
27:18
comes down to the collaboration piece that I mentioned earlier.
27:21
So that's kind of do a deep dive on.
27:22
Collaboration. So there are two.
27:28
Wait, there are two fundamental modes of collaborating between
27:31
different people doing doing work there, synchronous and
27:36
asynchronous. And synchronous collaboration is
27:38
when people are sitting in a room together and then
27:40
interrupting each other. When they have a question and
27:44
when they need to get something done, and in particular with
27:47
machine learning, it might even be that they're sharing an
27:49
environment. So they might even be pair
27:51
programming sharing a text editor.
27:55
So they take it in turns to use the keyboard, they would be
28:00
working on exactly the same data set and exactly the same
28:03
environment and then you're kind of time slicing it.
28:06
Well hopefully pair programming is is effective and okay so
28:11
people can't make changes to this thing at the same time but
28:13
we just take it in turns or there's an asynchronous approach
28:18
which is to say that different people should be able to do.
28:25
Welcome different copies of things, and problem with an
28:29
asynchronous approach, of course, is that you then need to
28:31
be able to cope with conflict. So, merge conflicts in get,
28:37
which takes me to the second Point here, which is how how to
28:40
software and devops teams, do it well with GitHub.
28:44
Basically and tools that are like GitHub, like it lab and
28:49
bitbucket and and all of the tooling around that and the the
28:54
way Way that that works as I'm sure most of the people in the
28:57
audience know, here is that you can Fork someone else's project
29:05
effectively, or if you're or you can make a branch from Master,
29:10
you can make some changes in your branch.
29:12
And while you're making the changes in your branch, you're
29:14
not trading on anyone else's toes and when you then proposed
29:17
those changes back to the master Branch, that's when you can pull
29:21
in new changes from the master branch and integrate them into
29:24
your bra. And that's when you have to deal
29:27
with merge conflicts and then you can propose a version of the
29:31
change which is up-to-date with respect to the master Branch.
29:34
That's basically commonly known as get flow and it's been very
29:40
successfully used in pretty much every team on the planet.
29:45
People have tweaks to that approach with multiple Master
29:49
branches and so on. But it's all fundamentally the
29:51
same idea of asynchronous collaboration and then we're
29:54
dead. Ops teams adopting things like
29:56
get Ops where your source of Truth for what's running in
29:59
production is also in a git repo.
30:02
Well, you can use tools like terraform or you can Version
30:05
Control your communities. Yeah, yeah, malls.
30:09
And then Happy Days. You can use the same
30:11
collaboration approach pull requests when when you're
30:16
deciding to scale up the cluster or deploy a new database and
30:22
doing the other things that devops teams do, So, so how can
30:29
you? So, what are the challenges
30:31
applying asynchronous collaboration to machine
30:33
learning? Well, they are numerous.
30:37
The first problem is that is the jupyter notebooks, don't Version
30:44
Control very well, and a lot of data scientists, use jupyter
30:47
notebooks. Now, the challenge is the data
30:53
versioning and data sharing is Uh is difficult in in machine
31:00
learning context? Sorry in in a collaboration
31:03
context because you can't very easily put your data in.
31:07
Get there is a project called git lfs but it has some
31:11
significant restrictions and and so what we find is that teams
31:18
normally just don't bother the data versioning and they just
31:22
rename files or rename folders and Have folder names with like
31:27
underscore final and underscore final, final V2.
31:31
And all these funny little strings that refers to the
31:33
things that have been done and then they have to share those
31:36
folders around and it becomes quite messy.
31:40
The other thing is metric and Paramus tracking, you didn't
31:43
have to do that when you're doing software but you do have
31:46
to do with machine learning. You have to keep track of which
31:48
parameters you used in? Which accuracy score you got?
31:52
Because models are either. Sort of green or red are not
31:58
either working or broken their kind of somewhere in the middle
32:01
and the metrics. Like the accuracy school will
32:03
tell you how good a model is against a certain test set.
32:09
And so you need to keep track of that.
32:10
And you could put that in the in the git commit message but then
32:15
you have to have a human remembering to do it.
32:17
People are bad at remembering to do things or you can use tools
32:21
that help you with experiment tracking.
32:24
And there are challenges with with using a combination of
32:29
local development environments. So you might have a GPU or ipu
32:35
in the machine in your desk or you might be using machines in
32:43
the cloud and with gifts, it's quite easy to switch from a
32:47
local machine to machine in the cloud or a machine in your data
32:50
center. But doing it effectively, and
32:54
with machine learning, Where you've also got data that you
32:57
need to move around and you've got to keep track of metrics and
33:00
parameters and maybe you can't even really use gets with your
33:04
jupyter notebooks and still do effective collaboration.
33:07
It makes it a lot more challenging with respect to kind
33:09
of moving around where you're where you're doing the work.
33:15
And so there are some tools that that help to solve these
33:20
problems for machine learning and it's actually a very
33:23
exciting space and There's lots of new innovation happening
33:26
around this and so obviously I'm from dot science.
33:31
I'd love it if you started using dot science.
33:34
But I also wanted to give a acknowledgement to the fact that
33:37
there are lots of other tools out there.
33:38
And so ml flow is quite strong in the experiment, traffic
33:41
experiment, tracking space, weights, and biases is very
33:45
strong in like comparing relationships, between metrics
33:49
and hyperparameters. DVC is a promising project in
33:54
terms of. Data Version Control as a
33:58
project called Pachyderm and then in terms of Dipping and
34:01
merging jupyter. Notebooks, there's an open
34:02
source project called embiid. I'm in fact, many of these, many
34:06
of these projects are open source.
34:09
And and so what we've tried to do a DOT science is bring the
34:14
capabilities from these kinds of capabilities into a single tool.