Latest / Agentic Conversations (formally mlops.community) / Hierarchy of Machine Learning Needs // Phil Winder // MLOps Meetup #3

Transcript

Hierarchy of Machine Learning Needs // Phil Winder // MLOps Meetup #3 · Agentic Conversations (formally mlops.community)

0:00
So joining us for this third episode, is Phil winter of
0:09
winter research? Phil is a multidisciplinary
0:13
engineer who creates data-driven software products his work
0:17
incorporates data, science, Cloud native and traditional
0:21
software development using range of different languages and
0:24
tools. So regularly speaks on cloud,
0:28
native data science. Shoes at conferences around the
0:31
world and aside from public and private training.
0:34
He is also a certified trainer for O'Reilly of Pearson.
0:38
He's currently writing a book about reinforcement learning for
0:42
O'Reilly media and today he joins us to talk about the
0:45
hierarchy of needs within ml Ops enjoy.
0:59
So joining As for this third episode is Phil winter of winter
1:03
research? Phil is a multidisciplinary
1:07
engineer who creates data-driven software products his work
1:11
incorporates data, science, Cloud native and traditional
1:15
software development using range of different languages and
1:18
tools. So regularly speaks on cloud
1:22
native data science issues at conferences around the world and
1:25
aside from public and private training.
1:28
He is also a certified. Trainer for O'Reilly of Pearson,
1:32
he's currently writing a book about reinforcement learning for
1:35
O'Reilly media. And today he joins us to talk
1:38
about the hierarchy of needs within ml Ops enjoy.
1:48
Hey Phil, I appreciate you joining us today.
1:51
Wanted to start from the beginning and ask you to give us
1:55
a bit of background on where your love for technology came
1:58
from. Yeah, I'll try and be very quick
2:00
then and I wouldn't call it love.
2:02
I call it love hate as well because there's a lot of hate in
2:04
there as well as love. But now, I started off in
2:07
electronics. Actually, I find it really
2:09
interesting when you speak to Engineers, their backgrounds are
2:13
often not computer science and not, you know, not not software
2:16
engineering there. I'm from physics, backgrounds
2:19
are math background, mine was electronics.
2:21
And yeah about a decade ago, there was a big push towards
2:27
writing software and chips microprocessors and over time,
2:31
it became clear to me that the software was eating the
2:35
electronics world. And yeah, I mean, these days,
2:39
really Electronics is all about just buying a mini computer
2:42
stuffing, it inside, something small and then writing software
2:45
for it. There's actually not not as
2:47
much. Electronics going on as there
2:51
was. So I made the jump eventually to
2:54
to software engineering via signal processing because I was
2:57
doing a lot of DSP, work and chips.
3:00
And I jumped across the software through signal processing, which
3:03
is is where I, you know, I really started working with
3:07
data. And that's that was kind of
3:09
before data science existed or even sort of machine learning.
3:12
Really wasn't really a term that was used that often and slowly
3:17
over time. Learn how to do software, you
3:19
know, followed all the data stuff as well.
3:22
And I kind of ended up here where it's, you know, half
3:25
software, half data science, building products for people.
3:29
So, yeah, that's the history. Perfect, thank you for that.
3:33
And so as people saw in the slide, when they were coming in,
3:38
you're the CEO of window research.
3:40
What? What do you do there?
3:42
What does that entail? Can you give us a bit of info on
3:46
that show? Yeah.
3:47
So I am gonna add a science consultancy based in the UK, but
3:52
we work throughout Europe. We work primarily on two types
3:56
of projects at the moment, but that is very market-driven.
4:00
One is on Raw applications. So there's quite a few companies
4:05
which need help developing their machine, learning /, data
4:09
science applications. So we help them build that
4:12
productionize it. And yeah, ship it.
4:16
And the other side of the This is working with larger companies
4:19
and Enterprises where they know that that data science.
4:24
And machine learning is becoming a bigger and bigger part of
4:27
their business. But all of the the tooling and
4:31
the processes are not mature. And so they're reliant on using
4:36
all processes built for software engineering that don't really
4:39
fit data science. So we go in there and help them
4:43
improve processes. Help develop new day.
4:47
Te science platforms if that's what they need.
4:50
Are basically help them go from idea to production in a shorter
4:55
time as possible. Cool.
4:57
Yeah that sounds sounds really interesting.
5:00
And my question with the the big and the small companies have,
5:05
you noticed the difference between those two on how agile
5:08
they are at putting models into production or the big companies
5:12
a bit more slow to act when it comes to building out data
5:16
science teams. And and putting models out
5:19
there. Yeah certainly but it's not
5:22
quite as clear cut as that because it's fundamentally
5:25
driven by the resources that each company has and the
5:29
resources that they can afford to put into to that effort for
5:33
smaller companies. They don't have the resources of
5:36
a don't have an operations team for example.
5:38
So they're really focused on building building a fully
5:42
automated end-to-end system, whereas a lot of these big
5:46
companies still have Very large operations departments and
5:49
they're still quite reliant on them.
5:51
So they actually don't mind building a more traditional
5:54
operations, heavy kind of workload, and that has knock-on
6:00
effects as well. So that means that the small
6:02
company is turning to use more managed and hosted Services,
6:06
whereas larger companies attending to build a little bit
6:10
more than they buy because it they well basically they can
6:14
afford to to build something. That is more specific to their
6:20
requirements which are those has been successful.
6:26
Set against all right which of those routes are being
6:29
successful in those contexts. Well I mean I mean both the
6:32
successful but they both have their both have challenges so
6:35
like if you're a small company trying to develop a single
6:38
thing, they'll probably work well to begin with.
6:40
But eventually you get to the point where your managed service
6:44
doesn't do what you need it to do and then it starts to become
6:47
a bit of a Frankenstein. As you started patching stuff on
6:52
whereas the larger company, you get the opposite problem.
6:54
You've got, you've got so much flexibility.
6:57
It becomes very hard to focus and so you end up building a lot
7:01
of stuff, but actually delivering very little.
7:06
That's a very, very easy trap to get yourself into a lot of that.
7:11
Yeah, I think there's two great points there with the the
7:13
Frankenstein piecing together. Lots of different parts and then
7:17
I'm wondering about with these big companies.
7:19
Do you see a lot of tools being used together where they're
7:25
taking maybe three or four of the same tools that do
7:29
ultimately they don't do the same thing but they do.
7:32
They have a lot of overlap. Oh yes.
7:38
And quite often you get a lot of opinions in the larger companies
7:44
from Engineers as well as Engineers are great for
7:46
opinions. Where they think that a one
7:49
particular product is, is Is better than another faux fur and
7:53
then maybe it maybe it's right for a particular use case, you
7:56
know it could be better. And so you do end up with
7:59
competing approaches and when two teams Collide, you know, you
8:03
no one's using 11 using another. That's that's kind of okay to a
8:08
point. But a big a big part of the push
8:12
for a more organized data science strategy in from from
8:17
within a big business is the idea of unification is trying to
8:21
unify high on a single set of tools, ideally a single set of
8:25
tools because it then becomes much easier to manage and
8:28
operate. Yeah, completely and so can you
8:33
give us a few case, studies of different tools that you've
8:39
implemented and how you've gone about that?
8:43
Yeah, so I can try. So every technology under the
8:46
planet is under makes a sign, a massive NDA before we start any
8:50
work. So I can track at.
8:52
Let's talk specifics about you know specific companies and
8:55
stuff back and I could talk in general.
8:58
So yeah. Are you interested in a specific
9:03
part of the stack? When you ask that question or is
9:07
it really what I've been hearing a lot?
9:10
Come up recently is monitoring So if you, if you have any use
9:16
cases about about how you build out something with monitoring
9:20
pieces, I'd love to hear about that.
9:22
Yeah, so most of the work we do when we do build a more custom
9:26
solution is usually based upon kubernetes.
9:30
So even the larger companies now have got, there's a lot of
9:33
traction for kubernetes inside those companies.
9:35
And so, newer newer software is being built upon kubernetes.
9:40
So the the The yeah, all of the tooling the monitoring tooling
9:46
that is currently being used for for communities.
9:51
Kind of finds itself Landing in the data science seen as well.
9:53
So there's lots of Prometheus lots of graph Anna you know
9:58
maybe even using sort of the elastic stack to do some of the
10:02
work they're especially for the logging and then yeah, I mean
10:07
there there are other proprietary companies as well
10:10
and I don't want to really name any But there are, there are
10:14
proprietary Services out there that you can also hook into
10:17
kubernetes which work really well.
10:19
And I think it's a it's a very important part of the problem
10:24
and and in fact, I think the valuations and the amount of
10:28
money that some companies have got for just doing monitoring
10:31
kind of highlights. The fact that everybody needs
10:35
monitoring. Yeah, I think, I think in terms
10:38
of gaps at the moment, we're kind of retrofitting.
10:42
In software engineering tools to to do the monitoring.
10:46
So you're very reliant on basic statistics and also working with
10:52
very low resolution monitoring data, you know, kind of like
10:57
single summary, statistic type data to monitor your
11:03
applications, and sometimes that's just not good enough.
11:05
You know, quite often we're interested in in distributions
11:08
and representing distributions and comparing distributions and
11:12
things. That.
11:13
And if you do want anything like that, it ends up being a lot of
11:17
custom code to be able to implement that.
11:19
So yeah, I think there's there's a gap in the market for a
11:22
slightly more focused and monitoring Solution.
11:25
That's more focused towards data science and their mouth.
11:28
So my question is and by the way, I'm also coming from
11:32
Electronics, so a cool. And okay.
11:38
So my question is, what are the main main friction points
11:41
between doctors? Scientist in software engineer.
11:44
Because as operations as a operations issue with that with
11:50
background, I see this fight again between software engineers
11:54
and operations. Yeah.
11:56
Well, what's new? Yeah.
11:59
Well, nothing's new is the same old.
12:01
Same old story. Totally agree.
12:05
So, so just to paint the picture, you know, ten years
12:09
ago, maybe not even 10 years, maybe less than that.
12:12
There is a big divide between operations staff and developers
12:16
developers with develop software.
12:18
I mean, you've all heard this before, right?
12:19
It's the whole reason why devops became a thing was to try and
12:25
attempt to smash down that wall between the developers and the
12:29
operations teams and that has worked somewhat.
12:32
I would say in larger companies, there are still very big
12:34
operations teams so they still exist in there.
12:37
Still there primarily because there's also a lot of old
12:40
software still burning so they do Need to keep running that
12:44
software, so they see ya. And so, when out, when I sort of
12:50
started doing data science ml more full-time, which is a kind
12:55
of, about 2010. The same thing was emerging with
13:01
ML, I would write algorithms, I would develop, you know, dead
13:06
science Solutions. And then I would hand it off to
13:08
a software engineer and expect the software engineer to
13:11
implement that. Properly, because data scientist
13:14
Cartwright proper software, and then that would then go off to
13:18
up to the operations. And so you had these, you know,
13:20
multiple jumps and it was a daily a daily exercise of
13:25
complaining about the other person.
13:26
So the software engineer would complain to the data scientists
13:29
saying, you know, I don't understand this.
13:31
What's going on here? It's crazy, it's too
13:33
complicated. And the data scientist is going
13:35
to the software Engineers like it's not complicated.
13:37
It's dead easy. How can you not understand it?
13:40
So I Fully agree with the push of devops up into data science
13:48
as well. And I guess that's that's kind
13:51
of falls under the banner of ml Ops or dated devops or whatever
13:56
you want to call it but basically trying to bring the
13:59
data scientists into the software team into the
14:02
operations team so all together as a team they can actually they
14:07
can they can be responsible for and they could deliver good ml
14:11
Solutions. The downside is that in many in
14:19
many sort of roles that scope Falls towards a single person
14:26
it's becoming more and more expected that a single person
14:29
would be skilled of doing all of these things.
14:32
And so that makes the stack, the full stack is becoming Fuller.
14:36
Like the word full stack is is pretty funny because it implies
14:39
that there's nothing else to add.
14:41
But in fact this the stuff You got it all the time so it's
14:44
we're going, you know, Uber fall.
14:46
Now if you're adding data science on the top of that and
14:49
there's obviously a limit to that there's only so much that
14:52
someone can learn and yeah so that's an organizational
14:57
challenge. Yeah, the the full stack is now
15:00
overflowing stack. Yeah, right.
15:04
And I read your blog article about that.
15:07
I thought that was a great point about how people are expected to
15:11
know such a wide variety and if you call yourself a full stack
15:15
now you have to go from here to here and it's no longer just
15:19
just this. Yep.
15:21
But pivoting a bit more into ml Ops and how you see it in the
15:27
hierarchy. Needs.
15:30
We had Luke on here two weeks ago and I think he's in in here
15:34
right now, he's in the in the chat somewhere, I saw him
15:40
earlier but he was talking about this ml Ops Manifesto and he
15:47
says that everything needs to be reproducible accountable
15:51
collaborative and continuous. And so we had a good talk.
15:58
Prior to this, but I wanted to get you to reiterate that on
16:04
your take on that. How you feel about it?
16:06
What? What you think and and what your
16:09
general thoughts are? Yeah.
16:11
Yeah. Okay, great.
16:12
So when I sell it to in the first thing I thought, when I
16:16
saw that was like, yes, great, they're all best practices but
16:22
what about everything else? You know, there's that was my
16:25
immediate thought that there's a lot more to it and especially I
16:28
like, from my experience in my day-to-day role, like, the vast
16:32
majority of my work is, is what I would consider software
16:35
engineering. I wouldn't even really consider
16:37
it like ml or data science because it's just yeah, software
16:41
engineering. And that II think that that all
16:45
needs to be in place. Before you start considering
16:49
these higher-level best practices, these data science
16:52
best practices. Because like you know, if things
16:57
are being Trained continuously. Like it doesn't really make any
17:02
sense to do that. If you don't even have Version
17:05
Control in your software, like believe you're not still some
17:08
companies, don't have Version Control, they're still making
17:11
that transition and, you know, there's a lot of old software
17:16
out there that don't have anything like this.
17:18
So there's kind of a little Point focusing on this when
17:22
you've got the basic needs that need to be met before you can do
17:27
that. So, That kind of started started
17:30
me, that's sent me down a path thinking.
17:33
Well, okay, if if, if there is something underneath this, then
17:38
then what is that? And I think I, in fact if I just
17:43
share my screen, I've got a couple of slides here to kind of
17:46
talk through this. I think this is probably a good
17:48
opportunity if that's all right. All right, let me just present
17:54
this. All right so hopefully everyone
17:57
can see that. So yeah so I started looking at
18:01
that and there's more and more best practices being developed
18:05
every day and they're all sort of good best practices.
18:07
It becomes quite difficult to to actually abstract and categorize
18:15
those best practices a little bit because there tend to be
18:18
quite sort of super focused on a single task, but still many best
18:22
practices. Imagine every day as people gain
18:24
more experience with with running.
18:26
There's science in ml projects in production, but I proposed
18:31
that this, this, these best practices kind of imply whole
18:35
range of basic needs. And so that got me thinking
18:39
about about about needs about analogies and how this relates
18:42
to the software and data science in general.
18:46
What are those needs? And is was there any sort of
18:50
Prior work on defining, what those needs are and the first
18:54
one that you come across as something called, Maslow's
18:56
hierarchy, Rocky of needs. And so this is a chap, Abraham
18:59
Maslow, who in the 40s and 50s, developed a psychological
19:04
theory. Suggesting that in order to
19:08
attain cultural Enlightenment, you need to have a set of Base.
19:18
You need a set of that. There is a set of needs that
19:21
have to be in place before you can graduate.
19:26
To the next level. And so he came up with the with
19:29
this, this hierarchy of needs here where the lowest level you
19:34
have like, you know, fundamentally like basic human
19:38
needs. If you didn't have these then
19:39
you wouldn't be alive. And then on the next level,
19:42
we've got a safety layer where you need the security and you
19:48
need to be able to live your life then.
19:51
And then the layer up, we're getting a little bit more subtle
19:54
psychological and you I need to belong when you need to have
19:59
friendships and have relationships and things like
20:01
that. Once you have that, then you can
20:04
start building confidence and self-esteem.
20:06
And finally, you get to these sort of high-level cultural
20:10
Concepts that they're the top level morality and creativity.
20:14
And he was arguing that if you, if you didn't provide for the
20:17
things below, then you never going to get the things above.
20:19
So there's yeah, there's no point trying to be trying to
20:23
make a site, a society, moral or Corker Native if they are being
20:28
threatened, if there's a war going on at the time like since
20:34
then subsequently research more recent research is kind of found
20:37
that it's not it's not strictly this all that.
20:41
So for example you can be in a warrant or a war-torn country
20:44
and build relationships with people so it is still possible
20:48
to kind of shortcut but I think in general this this is this is
20:53
kind of accepted as a A rough rule of thumb in Psychology.
20:59
If you have problems lower down, then you have problems above
21:04
That makes sense. Yeah that's perfect.
21:06
That's great. So as far as ml Ops hierarchy of
21:11
needs, what do you think is? Is the the base and how does it
21:19
go up? Exactly.
21:20
And that's, that's the tricky question.
21:22
It's like so I started thinking, yeah, what, what is really
21:27
required? What is required to do a project
21:30
to build an ml products. What do you really need?
21:32
So I kind of started thinking About that.
21:34
And this is just is just a bit of a first draft but I tried to
21:38
draw a triangle and no matter which way I spun it.
21:42
I couldn't really fit into a triangle because there was there
21:46
were sort of two dimensions and maybe could have done it if it
21:49
was a pyramid actually but I thought that a pyramid might
21:51
have been beyond my my Graphics ability.
21:55
So I went with a table instead as a data scientist, we love
21:59
tables and so I kind of had these two axes where I was
22:03
Trying to think of like, well, you can't really do data science
22:06
without software engineering and you can't really use software
22:09
engineering without having some kind of framework in place.
22:11
So, all right, that's, that's kind of one dimension.
22:13
And then there's another dimension where I kind of saw
22:16
that some things are required and some things are not like,
22:20
you don't need like full automation to be able to build a
22:24
product. Like you don't even really need
22:26
Providence in a an ml pipeline to build a product.
22:31
You really can just get away with, you know.
22:34
Being a model and they go, you can just about make it work, but
22:38
there was this level of there was this kind of idea of, but it
22:41
is valuable, it becomes valuable over time when you have more and
22:45
more and production, you have more to manage things start
22:48
going wrong and need to go backwards and forwards in time,
22:51
then that becomes really valuable.
22:53
And so I came up with like, these three, these two axes
22:55
we've got on the on the x axis. We've got the kind of the value
23:01
of having a particular Euler Concept in place and on the the
23:08
y-axis and in the Rose there, we've got kind of three sort of
23:12
key parts of the problem as I see it.
23:16
So, from from top to bottom, we've got data or data related,
23:20
Concepts data, science ml fit in there underneath that, we've got
23:24
software software engineering. And then, finally, the bottom I
23:27
couldn't really come up with a decent name for this, but I
23:30
eventually went with with governance and this is This is
23:34
the idea that in order to, you know, in order to deliver a
23:40
product, you need some structures in place to be able
23:45
to do that. Like even if we just ignore the
23:49
data part for a second and it was just a pure software
23:51
project. If you are trying to develop and
23:54
deploy a software project that you wouldn't really be able to
23:56
do that without having the backing of a team or a company
24:01
or a process to be able to develop that.
24:03
So I kind of thought that there is this like process layer that
24:06
was required at the bottom. Yeah.
24:10
And then going across on the x-axis, I tried to sort of
24:13
segments the the best practices into sort of three, three three
24:17
values. One where it's just absolutely
24:19
required. If you don't have this, then you
24:21
can't do anything. You don't have a product, you
24:24
could kind of think of that as, like, mvp-type stage to be
24:29
viable, it kind of has to, you have to have all of those
24:32
things. Then there's a second layer
24:36
where things start to add value and the only add value when
24:40
things start to scale and then finally, there's like this, this
24:43
golden optimal region. And I think that the thing I'd
24:47
like to point out in this is that in all three boxes are put
24:50
the word Automation. And that's kind of the Holy
24:52
Grail of of of everything that we do and that is to automate
24:58
everything so that we don't have to do it anymore.
25:02
And it's that's that It's a good thing or a bad thing, depending
25:05
on what you have it, how you look at it.
25:08
But if you can automate all of the processes and all of the
25:10
government's, then that will make everybody's life a lot
25:13
easier. If you can automate all of the
25:14
software, you don't need to do any any software related to us
25:19
anymore. That's great.
25:20
And finally, if you're automating the models and the
25:23
things relating to the data itself, then, you know, well,
25:28
we're out of a job, I guess. But I don't think that will
25:30
happen very, very quickly, but that's kind of the golden.
25:34
So does that all make sense, any questions based upon that, and
25:37
maybe we can talk about the individual topics a little bit
25:40
more in a minute completely. Yeah, that that's great.
25:44
And if anyone has a question, feel free to cut me off because
25:47
I'm doing a lot of talking but I find it really interesting that
25:52
you put governance in there because last week we just had a
25:55
talk with Charles and I think he's in here in the chat also
26:00
and so I'm wondering what? Why you've chosen to put
26:06
governance there at the bottom? Yeah, well, I mean, originally,
26:11
it came back to this this hierarchy hierarchy of needs.
26:14
It's like, well, okay, if I've got so starting from the top, if
26:18
I'm trying to do a data science project, what do I really need
26:20
underneath that? Well, we need software.
26:23
So, most of those signs you just software engineering, so I'm
26:26
going to put the software engineering underneath there.
26:27
Okay, what do I need to do software?
26:30
I mean it's a pretty pretty sort of philosophical question, but
26:36
what do I really need in order to build Build software and then
26:40
I kind of started thinking that well we need you know we need we
26:45
need processes and structures in place to be able to do that like
26:50
it and that applies for any way. If you're if you go and work on
26:53
an open source project, the bigger that open-source project
26:57
is the more processes you will find embedded within that
27:00
project and they're there, because they're there, they're
27:03
required to orchestrate, the the distributed work.
27:08
You go to a big company, then obviously there's all sorts of
27:11
processes and most of them are horrible it.
27:13
But even if you go to a smaller company, you know, there's
27:16
always a boss, there's always a customer, there's always, you
27:21
know, probably fellow Engineers working on a project.
27:24
And so there's, there's some level of process that is needed
27:27
to do that in the first place. So that was basically, the
27:30
reason for it. I thought that that is the base
27:33
layer that is this physiological safety layer at the bottom here.
27:38
And And I guess what that means is that software engineering is
27:41
love and belonging and self-actualization is, and the
27:45
steam is maybe data if you want to make some direct analogies
27:51
perfect perfect. So as as you go back to that
27:57
other slide and you look at it, And you see products, I imagine
28:03
you're working with a lot of tools and products each day.
28:06
How do products fit into this? Yeah.
28:09
Yeah, good question. But I also thought of a very
28:11
interesting question because there's quite a lot of.
28:14
Yeah, there's a lot of sex related stuff in here so I don't
28:16
know if this is where and where does software for into sexual
28:20
intimacy? I'm not sure there's probably
28:22
companies out there that are focusing on that right now but
28:25
it's probably not safe for work, so yeah.
28:30
We're back on topic. The yeah.
28:33
So, that's a good question. And, and this is where the line
28:38
starts to get a little bit blurred as well because it acts.
28:42
I had a lot of best practices relating to, I mean, I mean that
28:47
there are a lot of best practices relating to monitoring
28:53
and logging and what I term visibility.
28:56
So basically visibility of your product visibility of your Your
29:00
users and using those tools in in a, in a way that maximizes
29:07
the product value and that can be achieved, depending on the,
29:15
depending on the goals of the product.
29:17
But I didn't quite know where to put that visibility, because it
29:20
is kind of inherently a software engineering tasks, but it was
29:25
kind of felt like more like a. We only do it as part of a Major
29:30
process as part of a larger procedure.
29:32
So therefore a kind of shoved it into the governance bit of the
29:35
bottom. So for example, in that case you
29:39
would use your monitoring stack here and that's what that of
29:47
best practice would surround. So if you're using your
29:52
kubernetes stack, then this is all of the, you know, the
29:56
elastic staff, efk or yell. And Prometheus and Griffon are
30:04
and things like that. So that that would kind of fit
30:07
into this visibility box at the bottom and you would use those
30:10
tools to basically make sure that your software is running.
30:13
Because, you know, it pretty much is required because if it's
30:16
not there and you're not aware that your product is not
30:19
running, then you won't have customers along in terms of like
30:24
the software engineering. This, I've kind of got this big
30:26
software, engineering Banner here.
30:28
There are a lot of fundamental All software engineering.
30:33
Well, you know, there's it's an entire discipline, there's an
30:40
entire industry and there's all sorts of best practices and I
30:43
think surrounding there. So I mean I won't really go into
30:45
that. I think cloud engineering is
30:48
kind of emerging as a is, a slightly different approach to
30:53
to achieving the same goal and it requires a slightly different
30:59
tool set. I will still If if I like one of
31:01
my employees, I would I would kind of still expect them to
31:05
know both. I would expect them to be you
31:08
know, knowledgeable about software engineering principles
31:13
in terms of the you know the good old best practices and
31:17
stuff but also increasingly I would expect them to be
31:21
efficient and productive with cloud-based environments by
31:27
default. So I think that that is Is
31:30
becoming more and more required. So insert your favorite Cloud
31:33
vendor here and then it then on the top level we've got the what
31:38
do we really need to build a single data science product.
31:44
So I think actually I've got a slightly different slide here
31:46
and I kind of started to expand on some of this where a picked
31:51
the top layer of this this data layer and kind of expanded it a
31:55
little bit. So on the left hand side, we've
31:57
got raw data and models of. So what do I mean?
32:00
So in order to build a data science product, we need some
32:04
raw data. We need some raw materials to
32:06
start from. And once you've got that, that
32:10
raw data those raw materials, then we need to, you know, have
32:14
a structure or a set of skills to be able to analyze and
32:22
understand that data in order to solve a problem and you know,
32:28
that's the classic data science. You know, like, crisp crisper
32:34
model where we're doing, you know, figuring out what the
32:37
problem is, analyzing the data building models, evaluating that
32:43
trying to solve the problem and then deploying it.
32:45
So you kind of need all the skills just to just to build a
32:48
basic data product and then we've got maybe a little bit on
32:53
models more specifically like we need to figure out what what
32:57
Paradigm will work with working in before we start.
33:00
Is this a is this a supervised problem?
33:02
Is this? The reinforcement problem?
33:04
Is this a unsupervised etc etc. Because that you know, really
33:08
directs the the process that you would go through in the Ultimate
33:12
model that you would deploy. And then we've got questions
33:16
about like model architecture and how do we design it?
33:19
How do we build it? Is this a is this is this, a
33:22
simple thing? Is this, a complex thing?
33:24
Is it an ensemble of things do? We need to split up the problem
33:26
to make it easier to solve things like that?
33:29
And then We've got kind of you know, training serving
33:32
evaluation, those kind of things.
33:34
I would suggest that you need all of these things in place to
33:37
release a single product. If you didn't have any of these,
33:40
then you wouldn't have a product.
33:42
And I would also suggest that you'll, you need everything
33:45
underneath this to be able to do that, you need some software.
33:49
You need some infrastructure to be able to host that.
33:53
And you also need the backing of a, a team in a company to be
33:56
able to actually build that in the first place.
34:03
So, we sold try to deviate a bit.
34:07
So the conversation run tools. So what's it's harder to build
34:12
an Mo name? Real platform capable of
34:16
deploying muscles to production or ml team capable of to deploy
34:21
a model in production because we already talking about the full
34:25
stock. But it's impossible to have that
34:29
one person are worried about the challenges of building that
34:33
team, that dream team that capable of either.
34:37
The ownership from formula idea to production Yeah.
34:44
So if you are ready in any our daily or professional experience
34:50
you already try to how to build that team.
34:58
So that the kind of thing that you can't really compare them
35:02
because it's a bit apples and oranges.
35:05
So like building a platform is difficult for obvious reasons,
35:10
but building a team is kind of a kind of a different problem.
35:13
So We tackle the first one first building a platform is is is
35:17
difficult because there's a lot of engineering and things that
35:21
are required to get that operational but but those
35:24
barriers are being lowered every day.
35:26
So you know products and services are making that process
35:31
easier and easier as time goes on.
35:33
So I'm eventually you can imagine a situation where it
35:37
would just be default. It's not even something you
35:41
really need to build any more because it's built into
35:43
everything. Already on the team perspective
35:47
though that that on the team front that's a little bit more
35:49
that's a bit more difficult because like we said that the
35:54
stack keeps getting bigger and bigger and bigger over time.
35:57
The way companies generally solve that especially bigger
36:00
companies is by having teams full of people have different
36:06
expertise And, you know, that's, that's a good strategy.
36:10
I would kind of go a little bit further than just saying,
36:13
expertise, it's important to have teams with different teams
36:22
with want more diversity. I think it's trying to is what
36:25
I'm trying to say. Like a particular problem at the
36:28
moment is that, you know, if you're white and male and speak
36:32
English, then that's that's pretty much the only the only In
36:36
that you will meet when you meet those scientists and machine
36:39
learning and and and and that kind of limits A team's Vision
36:46
it limits the team's Vision. So that that diversity is really
36:50
important, as is as important as the diversity and experience,
36:55
but I think it is it is possible to find people that that do sort
37:01
of naturally thrive in this slightly more fuzzy.
37:08
Knowledge weld. I mean, it comes down to two
37:12
specific personalities and that's like the classic sort of
37:15
Personality. Split is like the difference
37:17
between a jack-of-all-trades. I'm not sure if that translates.
37:21
Well, as the phrase, that means somebody that is is quite good
37:26
at a lot of things, but not really an expert in one
37:30
particular thing. So I would consider myself as a
37:33
bit of a jack of all trades because I Enjoy jumping around
37:38
and learning about different new things.
37:39
So, new ideas and new things. There are interesting to me,
37:44
they're not, they're not something I worry about whereas
37:47
other people are much more interested in the detail and,
37:51
you know, focusing on a specific subject and they become a true
37:55
expert in that very specific Niche.
37:57
And I think there's, I think there's room for both of those
38:02
types of people, but they're kind of suited towards slightly
38:05
different things. So somebody that's very detailed
38:08
is probably more suited to a project that is a bit more
38:11
longer term and requires more focus in order to solve that
38:17
problem where a jack-of-all-trades type person
38:20
would be more suited to something that's moving, very
38:22
fast and India's and Industry. That's moving fast or a product
38:25
that's that's moving fast. See, how does that?
38:29
Does that make sense? Yes.
38:32
But for example, I can show you the my experience one of the
38:37
challenge that I have with that the scientists each.
38:40
The first thing was to understand the kind of language
38:44
they use. So the in my career within the
38:46
base, the first step was what they value in terms of, how can
38:52
I get information? Because, as you as you said
38:54
before, each one ever different set of skills specializations,
39:01
Unless you're a big Enterprise and you have departments for
39:05
dealing with that and that way you can throw away software
39:11
above the wall. But when he was small team use,
39:15
we need to go and be that jack of all trades.
39:20
But considering that at the same time it's a bit more complex
39:26
than software and operations. For example, so, So one thing
39:31
that I started to it was, okay, let's try to understand the find
39:36
common language that works for both directions from that assign
39:42
operations in software, and from software and operations, to that
39:46
the science. So, basically, it's a bit of
39:50
sharing the pain. So when something breaks in
39:52
production or and when something is not an X from the data
39:59
scientists, some Scientists suspected.
40:02
Sometimes they don't understand how hard it is to spin a new
40:06
environment, for example, what that means.
40:09
So that is the base of we are trying to build its the common
40:17
dictionary that everyone understands, you know what that
40:20
means or that terms. So that's how first we are
40:24
trying to go the communication pipeline from the tree.
40:29
Basically, it's that Because the team it's really hard.
40:32
It takes a lot. When I said at the farm, I want
40:37
to meet that writing the technical or the tools that is
40:43
they, each needed to put in production.
40:45
So the common here, it's to our, how hard it is to take in, in in
40:50
the technical side input, something production or building
40:53
the team to do that and put that modern production, from my
40:58
experience, of course, the teams, building a team is always
41:03
more challenging and harder because it's very like, very
41:08
sensible thing in in any kind of organization, big small because
41:17
it is in small and big is the one thing that always tracings
41:21
is teams and organization, even they, if they have, like, old
41:28
old technology I work I was, I used to In the bank and the
41:33
themes change so fast, it was incredible.
41:36
So sometimes he's not the technology, it's more that out,
41:40
the team changes. Yeah.
41:43
Yeah, I totally agree. And I think I think trying to
41:49
divert that into a talk about management and how to build
41:54
teams and how to manage teams is slightly off topic but to try
41:58
and keep it on topic. I think the clap, the classic
42:00
devops answer is multi multi faceted teams of teams with
42:04
different experiences and if you can, if you can get a good team
42:07
of people that work well together, I people that respect
42:11
each other and get along with. To that then over time just
42:14
through osmosis you know people start to pick up and learn from
42:18
the other people. So the data scientists learn
42:20
from the software engineers and software Engineers, learn from
42:22
the data scientist and that team becomes more and more productive
42:26
over time, but I, yeah, I totally agree.
42:29
It's not an easy thing and it's very, it's a clear problem for a
42:35
especially larger companies going to say something on this.
42:38
Yeah, I've got. So again, from the classic
42:41
devils, Perspective, one thing we miss there is that each
42:46
organization has its own unique challenges so there is no
42:49
template that applies and the way that you achieve that is by
42:55
creating a great culture that facilitates continuous
42:59
Improvement, facilitates experimentation and
43:02
understanding what works? And what doesn't towards the
43:05
goals like cross-functional teams.
43:08
If that's what you need in your organization and a lot of us
43:11
understand. Values cross-functional teams in
43:13
a larger scale in the might be much less cross-functional more
43:17
huge operations, who knows? But that devops perspective of
43:24
continuous Improvement and and the culture part that I think is
43:29
key, you talk to you, spoke about diversity and you know in
43:33
different organizations, these localized things that we do,
43:38
maybe all part of it and our company, we have a Very strong
43:42
culture around diversity, so we don't have that issue of every
43:45
data scientist being a white wall.
43:48
But then we have other challenges, we have very strong
43:52
software, engineering culture and data, science culture, and
43:58
to bridge those that historic classical software engineering
44:03
is not necessarily geared towards data.
44:07
Scientists data science is, it's going to be over the next couple
44:10
of years. And we always need to keep
44:12
Looking to change what the environment, that's changing
44:16
really rapidly as well and the notion of what good is to a
44:20
developer to software engineer in your table.
44:25
You had Cloud engineering, and software engineering is
44:28
different things. Yeah, but at some level, they
44:32
are the same but they understand what you're saying.
44:34
There are two software Engineers, deeply cares about
44:37
things like solid principles and things like that, but then Just
44:42
to say to them be a little bit crappier and move fast in the
44:46
cloud as a difficult conversation because they're not
44:48
the values they carry. So there's lots of in each
44:50
organizations, knots diverse issues, but come back to the
44:54
devops principles of continuous Improvement and what you're
44:58
doing today. The things you do might not be
45:02
the things you need in six months time.
45:07
Yeah. Tell you agree with all that.
45:09
Yep. So I've got a few more questions
45:15
about how you specifically implemented this idea of optimal
45:24
your optimal data science, right?
45:26
And how you've gone in, can you share a story of going into a
45:32
company? I know you don't have to mention
45:35
any names or mention which tools you used if you don't want to.
45:38
But I'd love to hear. Just how you went into a company
45:46
and you were able to create an optimal functioning data science
45:52
or ml Ops pipeline models and production anything that you can
45:57
share around that. Yeah, I'll try.
46:00
So I'm let's just go back to that slide again then.
46:05
So, this is again, this is back to the sort of the top layer of
46:08
that. And we're, we're sort of really
46:10
talking about like an end to end.
46:13
Example, and Irene the, the example isn't particularly
46:16
exciting. I mean, the example that I got
46:18
is, you know, going into one of these big businesses, and
46:21
helping them do all of this. And that that is that is the
46:25
example. But I mean, we can talk a little
46:27
bit about the, the tooling and things that used in order to do
46:31
that. So you know, these steps on the
46:33
left hand side, a very, very local to a developer, a single
46:37
developer would do this, you know, by themselves or maybe in
46:42
a small Mm, it's not something that that you need that much
46:45
scale for depending on. So, some companies that I've
46:49
worked with have had very strict.
46:51
It policies, especially after all of the ransomware attacks
46:56
that happened a year or two ago, where they're not allowed to
47:01
install any software on their laptops, some developers data
47:04
scientists, anybody not allowed to install anything on your
47:07
laptops so that that forced them to move to hosted.
47:12
This has a lot, a lot, a lot more quickly than they maybe
47:15
would have done if they hadn't have been implemented that
47:17
policy. Where was I going with this?
47:20
Yeah, so I was I was trying to suggest that the the left hand
47:23
side, the gray box can can can be done locally.
47:25
But even in some companies that can't be done locally anymore,
47:28
you need to have a service to do that.
47:31
Once you start going to the, you know, the the value will pass.
47:35
All the optimal parse though. This is, this is really this.
47:38
You only sort of get into this when you start trying to scale,
47:42
You have, you know, multiple or many products.
47:46
So the companies, some of the, the large companies that I work
47:49
with we're talking in terms of hundreds of projects, hundreds
47:52
of data related projects here. And so you can imagine that when
47:57
they got all get deployed to typically the the same
48:01
environment, you know, the same production environment, at least
48:04
then it is a bit of a management headache.
48:06
So you do need a good set of best practices.
48:12
In there for auditability and reliability.
48:17
So we're talking a lot of pipelines here.
48:20
So a lot of build pipelines, a lot of tooling around testing
48:24
and automating the the validate e of the things that are going
48:31
into production and there's yeah, lots.
48:34
We can talk about in that particular Minefield but there's
48:38
also working with data as well. So actually streaming or Are
48:43
passing the the data from wherever the sources and moving
48:47
it through pipelines into a place.
48:49
Where it's actually used can actually be if that's actually a
48:51
really difficult architectural task because it's very likely
48:55
that this data lives in in different places.
48:58
So we are, we've had to build a lot of customer customizable and
49:03
flexible. What I call pipelines to to to
49:08
kind of move all of that data around and put it in the right
49:11
place. Providence Providence is the
49:15
it's a bit of a fancy word. Actually, I'm not a huge fan of
49:17
the word because it's a bit, it's a bit confusing.
49:20
But provenance is the idea that, you know, where you know, the
49:26
origin of the thing that it's running in production and that's
49:30
the origin of the software, the origin of the data, the origin
49:34
of the model that's been trained and all these things.
49:38
So you can go back to a very specific point in time.
49:42
When things go wrong and they do go wrong and when they do go
49:45
wrong, we can ask the question. Okay, why why did that that
49:49
particular, you know, observation get a completely
49:53
wrong prediction when it's, clearly clearly should be good.
49:58
You can go back and look at the data at a very specific point in
50:01
time. Look at the code, a very
50:02
specific point in time, you know, all that good stuff.
50:06
And in order to do that, you need to make sure that your, you
50:08
know, packaging the the the artifact that's being deployed
50:14
into production. Well, which relies on having all
50:16
of the pipeline's in place to be able to build it properly
50:19
repeatability. He's kind of, you know, very
50:21
similar, really. I think that the thing at the
50:25
end there, the CI CD for a male, that's another thing that's
50:28
really, really missing at the moment.
50:31
There's, there's there isn't much standardization and there's
50:34
a big gap there for doing CI CD, but for 4ml from models and for
50:40
data as well. Because I The vast majority of
50:44
projects I come across do not validate their data, they do not
50:49
validate the data as it's being used within the system, so they
50:53
get fairly of all the time, but that kind of accept it as normal
50:57
because this like, oh well, that's normal because they've
50:59
just passed. There's some missing data there.
51:01
There isn't any ending that data, so obviously fails like it
51:04
shouldn't felt it shouldn't, you know, you should have.
51:06
You should have something in there to validate the data
51:08
before it is your system. And again, it's very much a
51:11
software engineer in test but So important for a data science.
51:15
And when I get to the the far, right hand side automation,
51:20
there's obviously Lots, you can automate but there's a big focus
51:22
at the moment on how you can re trigger and retrain your models
51:28
automatically. So some but not all applications
51:33
are very lucky to have a natural feedback mechanism.
51:37
If you're lucky enough to have the situation where your users
51:39
are actually providing you with feedback, Either in the form of
51:44
new data or in the form of you know yes that was a good result.
51:49
Yes that was a good prediction something like that.
51:51
You can use that information to to retrain your models and
51:56
improve them over time. And you can then automate that
51:59
process to make sure that these new models are at least as good
52:04
as the old models. And once you've got that
52:06
feedback cycle in place, then you kind of don't need to touch
52:09
it anymore, until you need to make some large architectural.
52:12
Changes to your model. So I think that, you know, that
52:17
certainly optimal but it's not, it's not the place where most
52:20
companies are at the moment. Most companies are still, you
52:24
know, either retraining manually or sort of retraining based upon
52:27
a cron schedule like once a day or something and that actually
52:32
works. Okay.
52:33
You know, I haven't haven't come across the specific example
52:38
where it's been, it's been it's obvious that Doing this
52:43
continuously is really valuable, but I'm sure it is.
52:46
So for, for the, for the people. Like, you know, the Amazons of
52:50
the world or the Google's of the world that are working with,
52:53
with, with algorithms that are used to, at such a scale could,
53:01
well, believe that these are really important but for a
53:04
smaller company. Yeah.
53:06
So can you describe there's not doing it.
53:09
Yep. Why they're not doing it.
53:12
It's not. The lack of availability of
53:13
tools or all the while, they're not doing it, it's comes down to
53:20
a value. It comes back.
53:21
Like, how valuable is it? Like, if you don't retrain that
53:24
model, every imagine just two simple cron, schedule.
53:28
If we don't retrain that model every day, then you know, what's
53:31
going to happen like nothing, we're going to lose a little bit
53:34
of performance. We might lose a tiny bit on the,
53:39
you know, some whatever metric you're measuring to kind of
53:42
measure. These things, but the end of the
53:44
day, we've got bigger problems, elsewhere, that are far more
53:47
important than doing that. But then also in most cases, no,
53:50
no, I totally agree with you, you know, just to do that by
53:55
because somehow it's a best practice.
53:57
Actually it might not be delivering a Valium.
53:59
Yeah, exactly. Yeah, but there might equally be
54:02
cases where there's loads of value and remains unidentified
54:09
and and even if some person believes it, Can we articulate
54:13
that to the organization surrounding or something like
54:16
that? Yeah, yeah.
54:17
Totally agree. Yes, I think I think it's kind
54:21
of so I mean that's not. So let me try and give a
54:24
specific example and so you know it's like I said most of the
54:27
work we do is kubernetes base for the larger companies.
54:31
The data scientists will come in.
54:34
He will use a hosted jupyter notebook, probably to be able to
54:39
do his data analysis, you'll get access to his data and
54:43
eventually decide upon a model. Once it's decided on the model,
54:46
then he would use a Action to, you know, something like Coupe
54:51
flow or something like that to actually kick off a training,
54:55
run, given a set of code that he's produced or she's produced.
55:00
And the, the end result of that will be a pickled model usually
55:04
that will get stuffed into an artifact registry.
55:09
One of the ones we used quite commonly is something called ml
55:11
flow and will flow, is a tool. That is a has the capability of
55:17
storing. Model parameters.
55:20
And then that, that is then served by a set of tooling that
55:31
reads from the model repository, spins up a container as and when
55:35
it's needed and then serves the user and then over time that
55:41
that model gets used and then later down the process of when
55:47
the person is kind of interacting with that.
55:48
Idle and providing their feedback that feedback gets, you
55:52
know, saved into the. I mean it's various words for it
55:54
at the moment but the call it like a continuous learning
55:58
pipeline. Effectively, you storing the
56:01
responses somewhere in a database or in a bucket or
56:04
whatever. And at some point in time it
56:09
will be the continuous learning pipeline, will be triggered and
56:12
it uses that new information to go back to KU flow retrain and
56:16
you model. Deploy it back to Mr. Flow And
56:19
then the new version gets deployed and then you have like
56:21
the A/B Testing style stuff in there to verify that you've not
56:26
broken anything. So this kit several kind of
56:30
loops and feedback Cycles going on.
56:32
So it gets a little bit complicated at time.
56:35
That's great. That's perfect.
56:38
Thank you for that. Looks like we have hit the our
56:42
bar