Latest / Agentic Conversations (formally mlops.community) / Reinvent Yourself and Be Curious // Stefano Bosisio // #264

Transcript

Reinvent Yourself and Be Curious // Stefano Bosisio // #264 · Agentic Conversations (formally mlops.community)

0:00
Stefano Bazizio working for Synthesia and I'm I'm Italian,
0:06
so a strong homemade coffee with my own machine.
0:12
We are back for another ML OPS Community podcast.
0:15
I am your host as always Dimitrios.
0:17
Today talking with Stefano, we got into some of his biggest
0:22
failures as he has learn how to be an ML engineer and learn how
0:28
to create ML platforms. He went from the world of pH DS
0:34
in chemistry to building ML platforms at financial companies
0:39
AKA banks, to starting from scratch and having to build out
0:46
what he called a Ferrari of an ML platform only to realise that
0:52
it got no internal adoption. Let's get into it with Stefano
0:56
and hopefully you do not make the same mistakes as he did
1:00
because you learned from what he talked about today.
1:05
I will also call out that he recently released a course on
1:11
the ML OPS Community Learning Platform.
1:14
Highly encourage anyone who is looking to learn how to ask the
1:19
right questions when you are building out your ML platform to
1:23
go check out the course enrol. Stefano is a class act and in
1:29
this course he specifically goes through the one O ones of
1:33
building an ML platform with open source tools and then with
1:37
a tool like Vertex AI. Let's get into it dude.
1:47
We'll talk to me about moving from academia to the industry.
1:51
I know you have a strong background in chemistry and then
1:55
you've done a whole wide array of different things now since
1:59
you've moved out of the academia industry, but what has that been
2:04
like? Everything starts back like more
2:07
than 10 years ago when I was a very little Italian engineer and
2:12
with the green biomedical engineer.
2:15
I was so deeply low over the time.
2:17
At the time, AI, as everybody's saying, it wasn't a thing.
2:23
But it was a thing though. Like if we see the conversation
2:25
with guns and so on. It was a thing since 1991 and
2:29
even before. But no one really applied very
2:32
well. Like Tense of All was the first
2:34
package. I think it was released.
2:37
And but yeah, to make it short, I was, yeah.
2:40
The passion of research was totally firing me up.
2:45
I was on fire. I want to do research.
2:47
I want to discover things. Yeah.
2:49
Was in love with genetics and so on.
2:51
So and physics and so say wonderful.
2:54
I wanted APHD and that's how everything started.
2:57
Unfortunately. Unfortunately.
2:59
Maybe so. That's how I landed in in
3:02
Edinburgh, Scotland, for a PhD in computational chemistry.
3:06
It was 2014. If I know wrong.
3:12
Amazing so something I always recommend to everybody and so if
3:16
you feel yourself you're have the passion for research, want
3:20
to discover something new, you want also to continue to study,
3:24
give yourself a chance and try to maybe apply for a PhD.
3:29
Who cares if you're not earning money?
3:33
Still remember I was getting. Come later.
3:35
Yeah, You know, I always get to like something about, I can tell
3:38
you numbers, we have problems, £1000 a month.
3:41
I was very, very stressed the situation because you got you
3:44
got all the expenses. The British pound was very
3:48
strong compared to the Euro, so coming from Italy wasn't helpful
3:51
enough. But apart from that it was an
3:55
amazing experience really. So I always be thankful to my
3:59
supervisor because when you're trying to do the search, at some
4:06
point you, you start coding up, maybe especially computational
4:10
research coding up, which means maybe my level of coding was
4:15
more or less zero. And then you start to get more
4:19
around Python CC plus plus at the time as well and say, cool,
4:24
I can do serious stuff from here.
4:27
And well, we said the level of details you can get through.
4:31
You can get very deep, deep inside, like multi processing or
4:36
crazy stuff and statistics as well, because there was the
4:39
theme of the PhD at the end that you got, you're going to acquire
4:45
a lot, a lot of skills. The only problem is maybe that
4:50
you're too attached to academia, so you're too attached to a way
4:53
of thinking, so you think everyone knows what you're
4:57
talking about. You really don't have a lot of
5:02
soft skills. You don't know exactly how to
5:04
interrupt with people. This might be a problem with
5:06
many supervisors as well. Not mine, thankfully.
5:09
He was a he's a great guy. But of course, of course, that's
5:13
that's this part of the game. And and from there, what
5:18
happened? Well, yeah, the PhD is finished.
5:22
So that's ring me a bell saying it's time to find a job.
5:27
A job went waterful. Well it was 2018 if I'm wrong.
5:31
So it was the since 2016 there's been a huge hype for data
5:37
scientists. So since I did a lot of coding,
5:41
a lot of physics, a lot of maths and statistics, I said what the
5:45
hell, I can do this thing, I can give it a try and then jump on a
5:50
joint venture. So I say that was the very first
5:54
contrast in my life, passing from academia to a real company.
5:58
The 1st jump I've done was working as a data scientist for,
6:03
let's say, a fintech company. So it was a bank.
6:07
We were selling products like mortgages, loans, credit cards,
6:10
whatever. Whatever a bank can sell, we
6:13
were selling it. But it was a good time to join
6:17
the company because they were trying to build out this new
6:20
data science team. So what's data science?
6:23
At that time, the question was, what is data science?
6:25
What can we do? What are data scientists can do?
6:30
But yeah, Dimitrious, it was an amazing opportunity.
6:34
Good memories, but very harsh to understand what I had to do
6:40
passing from APSD to a company. I remember many times these
6:45
people were just talking about about acronyms or we are the
6:50
financial staff and say what, what, what and I was trying to
6:55
say guys, this is like simple stuff like introducing GitHub,
7:00
this is GitHub, this is how it works.
7:02
So you see, can you see, we can keep track of the code and so
7:05
on. And but it was very hard to give
7:08
people to deliver this message maybe to people as well as
7:12
building models as well as the way we were building models
7:15
because you know, I was used to run my stuff on a Ubuntu
7:20
machine, thrilling to whatever I want, as many GPUs as I want,
7:24
right? And while here we were very
7:28
constrained on Windows laptop you can't install anything
7:32
because of course there might be all, all the packages must pass
7:35
through some security cheques and so on.
7:38
So it was a really a heart attack for me.
7:42
But it was a good opportunity, a very good opportunity.
7:45
If I see now back in time, that gave me all the a lot of initial
7:51
skills that many PhD students I can't acquire, unfortunately.
7:59
You know, the PhD environment is very protected.
8:02
And although I'm strongly a supporter for PhDs, sometimes I
8:06
say yes, but please make sure you can get also some soft
8:09
skills course or do think of these soft skills because
8:14
they're not as useless as you might think initially.
8:18
They're super useful and once you learn them, you'll open up a
8:23
lot of those. So that's more or less how
8:26
everything started and how I changed my fire for the search
8:32
to a fire for data. Yeah, there's been, of course,
8:37
many familiar thoughts. Like I would say many, many
8:42
reasons why I didn't continue with academia, like not only the
8:46
fact that you always have to look for grants and so on, but
8:51
it's, it was merely because my wife and I loved Edinburgh and
8:56
wanted to stay here a bit more. So say, OK, I can, I can have a
9:00
stop, a break from academia and jump into this data science
9:04
thing. But but it was super.
9:07
It was super amazing, really. Well, you have since evolved,
9:13
basically taken the evolution from academia to the industry,
9:18
working with fintech, and then you started to recognise that
9:24
just data science isn't enough, I imagine.
9:27
And if you want people to have value from the work that you're
9:30
doing as a data scientist, you need to learn a little bit more
9:34
of the platform side of things. Can you talk about standing up
9:40
different platforms and when that realisation hit you?
9:44
Yes, yeah. So totally, you're other percent
9:47
right. And also here there's a nice
9:49
story. So.
9:50
Oh yeah. So that's awesome.
9:52
That was in this fintech and gosh, we were developing models.
9:58
And now that I think I start laughing because we're
10:00
developing models on a Python notebook.
10:03
Of course, you know, there's this amazing theme of pushing
10:08
Northwoods to production. Nothing against a solar, right,
10:13
deploy whatever you want and totally fine if you manage to
10:16
deploy by some point. Of course we were struggling the
10:22
the level of going forward and like at that time, I think there
10:27
wasn't a real understanding yet of the envelopes practises.
10:32
So what happened is that we had DevOps team and that it was
10:38
like, call it data OPS. We were developing model and
10:42
they were trying to do something with this models bar something I
10:45
didn't clear. Like I remember the time we had
10:49
some weird EC2 instances. There weren't even EC2 instances
10:52
on AWS kind of mixture. And these guys were taking the
10:57
binary file, the model and trying to make it run by it
11:01
couldn't even run on real time. It was a very a convoluted
11:05
system to get to something and get a little bit of value out of
11:09
what we were doing. So as, as you can imagine, the
11:12
value itself was very hard to deliver.
11:15
Because there's also something I'm telling everybody, if you
11:19
can't exploit as much as you can the business value out of the
11:24
your data, out of your work, then unfortunately there's a lot
11:28
of things to do because we do need, we do need the business
11:31
value. So my manager at the time had a
11:36
good guy take me for a meeting, say Steph, there we go, this is
11:45
AWS. Why don't you develop an ML
11:48
platform? Say, I don't know anything, but
11:49
I can try to see what you're talking about.
11:52
So in those, I think 3 months, I had a huge team inertia into AWS
12:02
trying to understand what was going on and try again to
12:07
reinvent something for for my career.
12:09
So I said initially I said, OK, I'll be a data scientist, I'll
12:12
develop a nice models. And then I was starting to look
12:15
at ML plasma, say these ML engineers know something that I
12:19
don't know. And so initially there wasn't
12:24
really I think also many resources because it was still
12:28
2019. It's true the I can't remember
12:32
when the ML OPS community was founded. 2020, we weren't.
12:37
We weren't around yet, Yeah. Where's the other thing?
12:39
So I say, where do I find all these resources?
12:42
There was there were all good, good, good books about AWS and
12:45
how to maybe make things communicating.
12:49
And so the first thing was a lot of philosophical discussion of
12:53
what we really need to do, what we mean by deployment, how we
12:57
can decompose all the problems we had.
12:59
So it was really nice because it was reattaching myself to my
13:03
engineering background, so giving a problem, try to split
13:07
into sub problems and try to reconnect everything in this
13:10
weird jigsaw. And so yeah, from there, like I
13:15
started thinking, OK, we need something for data processing.
13:18
And then I said, OK, we need something for model training.
13:21
OK, how can how can we really train the model and what can we
13:23
do with the binary files that we have or the model themselves?
13:29
How we can really spin them up? So it was an amazing journey
13:32
because allowed me to really start from simple to develop
13:36
your ML platform with Lambda functions and step functions and
13:40
try and then try to make up something more robust.
13:43
So it has the kind of time for it was the time for glue, for
13:48
data processing, using a proper database, create a data
13:53
structure. Bad people starts to communicate
13:57
better with people and make them understand what we need to do
14:00
with data, what we need to do with the model, why we need all
14:03
these weird, weird stuff. I remember it was very hard,
14:06
very hard. I think nowadays it's super hard
14:08
as well to talk to senior manager, very senior managers
14:11
about these things. Like that was a lesson.
14:16
Of course, they don't need to know all these nitty gritty
14:18
details about what an ML platform is.
14:21
They just want the business value and what I was trying to
14:24
do was to give them the beast a smile and say watch this, this
14:27
is amazing. But apart from that, it's.
14:29
On the demo for an hour. Give them an hour demo and show
14:34
them how amazing it is. It's what I tried to do, but it
14:37
was very hard really. And, but apart from that, and it
14:41
was an amazing, an amazing opportunity really to try to
14:45
understand how an ML platform could be developed and try to
14:49
think really of naive terms of what ML OPS could be.
14:53
So trying to get more into the ML OPS perspective.
14:56
So what we're trying, what we are trying to do, what we're
14:59
trying to achieve. And, and especially if you think
15:03
with very limited resources, because still I was working with
15:06
my limited Windows laptop. So developing coding was the
15:09
worst experience ever. And even pushing code to AWS was
15:13
a real nightmare. We did needs also IT to, to be
15:18
in the middle to say, OK, well, this, this data must be pushed
15:22
over there. OK.
15:24
But it has definitely moved a lot of conversation also in I
15:28
think we were one of the first banks where we were trying to
15:31
develop for real an ML platform. So really think of an entire ML
15:36
journeys, really think how ML models should be created and
15:41
constructed. So it was an amazing thing and
15:46
really, and this time my partial for ML OPS start to find out
15:50
what the hell I say whoa. And and it wasn't ML OPS, it was
15:55
engineering. ML engineering in total, say,
15:57
Oh, I could develop a model, but I could also push them to
16:00
production. I think something similar.
16:02
So that was the way. Maybe the kicker heads, it was
16:06
really an as you know, maybe it's the fate, whatever that
16:12
pushed me down this direction and trying to navigate a bit
16:15
more into the MLOP sector and give me also the like.
16:20
At some point I was fed up with all the financial things, good
16:25
financials, but I'm not getting a bonus as I as the senior
16:28
manager, so I'm leaving. And so I started to deeply focus
16:33
really on MLOPS and that's also where I got to know the MLOPS
16:37
community because I so did work really for an as an ML engineer
16:42
for another company and browsing an interest spotted envelopes
16:46
community, what are disguised doing and that's open up an
16:50
entire series of greedy things. Dude, well channelling my inner
16:55
Bezos. Because you're talking about AWS
16:56
and you're talking about things that were hard back then and are
17:01
still hard a la talking to senior management.
17:06
I would love to know what you feel like over the years hasn't
17:10
changed when working with MO OPS.
17:13
Hasn't changed. Yeah, that hasn't changed.
17:16
A lot has changed, right? But what hasn't?
17:19
Hasn't changed stakeholders. So what the the first thing that
17:23
popped into my mind OK, stakeholders.
17:24
OK, two things dude hasn't changed stakeholders and selling
17:28
products. What I mean is this what
17:31
stakeholders is the major things, I think, and this is
17:33
something again, it's very hard to grasp, especially I think if
17:36
you're changing career, if you're moving from academia to
17:41
companies and so on as the way you're communicating.
17:45
So like every time, like as an engineer, it's very easy for us
17:51
to create a new product, right? Do we want an experiment
17:53
tracker? Boom, there we go.
17:55
Do you want a super cluster with 13 nodes, 100 nodes, 1000 nodes,
18:01
There we go. Do you want us to scaling?
18:03
There we go. But what doesn't change is the
18:06
way we're communicating. We're doing a magnificent job,
18:09
right? But it's very hard to
18:12
communicate and express this business value out of it to more
18:17
senior managers, of course, or even our stakeholders.
18:21
Like many times it happened to me, I, I was working for this
18:26
company where we were really creating the very first ML
18:30
platform for the entire company. And gosh, one day had the
18:34
realisation, the epiphany that we made an amazing job.
18:38
We had an automatic way to train models.
18:41
All the models were register, all the models could have been
18:45
trick keep track of all the experiments.
18:48
It was everything very neat and pulling.
18:50
Monitoring, you know. Yeah, even monitoring which is
18:54
like it was the Holy Grail this time, you know, if your data are
18:57
saying are a bit, are a bit shady, that's why we have
19:01
monitoring place. We even had data pipelines for
19:03
validating data. Thank gosh.
19:05
People were not using it had like two weeks of depression or
19:09
even a month. And I say why people are not
19:12
using this thing because it's amazing.
19:14
They say the people should jump in here and say oh, oh, this is
19:18
an amazing product. Yeah, you were expecting an
19:21
award. You were.
19:22
How the hell? How, Or at least to go and give
19:25
the talk? Yeah, yeah, yeah.
19:27
I started like I was expecting fireworks because something like
19:30
that, but nothing. It's we're really bouncing
19:34
against the wall and that's the maybe a naive, but the
19:38
sterilisation I had the same. I didn't communicate properly
19:42
with my data science team like, which means it's not only having
19:46
meetings where you're explaining what you're doing, it's not
19:48
having stand ups where you're showing the Jita board.
19:51
And so it's giving demo tutorials, taking time with
19:54
them. And it's a huge effort.
19:57
It's a huge amount of time, but you must dedicate this to these
20:02
people, otherwise you're not going anywhere.
20:05
And like I remember I started to record myself and giving
20:09
tutorials. To, for example, show how to
20:12
ship a new pipeline for training or how these things are
20:16
automatic and how they can improve our way of doing right.
20:19
At the same time, they were also deploying instantaneously if the
20:23
model was behaving well and so on.
20:26
So writing documentation as well was another thing.
20:29
Like I spent hours on all our packages writing documentation.
20:34
I say, OK, if you want to use this, you need this, this in
20:37
this. If it doesn't work, of course we
20:39
are here to help. So like being more like a 24/7
20:43
support team, we are here to help making sure everyone knows
20:47
that we are here to help so that people could say, oh, OK, so I
20:52
want to create a new model. I can go to Stefano's team and
20:56
ask what are the best practises and do because yeah, we were
21:01
getting hiten by by everybody basically because no one knows
21:07
what we were doing, why we were doing some stuff, what was the
21:10
body we were giving to them. So I think the communication
21:14
something that I had to spend a lot of time and as well as well.
21:20
Wait, before you get into the next point, you basically had
21:24
every founders worst nightmare, which was launching to crickets,
21:30
right? And you should have told me
21:32
because I would have given you an award for sure of best MO
21:36
platform that's never been used or something along those lines.
21:39
We would have in the ML OPS community yearly awards.
21:42
We could have totally given you that.
21:43
But the the thing that I am fascinated by is how you
21:50
attribute the failure to your own lack of evangelising for the
21:56
product that you're building. And so it was almost like you
22:00
did not build enough in public, like internally in the company,
22:06
you weren't showing people, like you said, you weren't doing
22:09
demos, you weren't properly documenting or having
22:14
potentially like pair programming sessions or just
22:16
like learning sessions with the data scientists.
22:19
Were the data scientists your only stakeholders, or were there
22:22
other folks from different teams too?
22:24
Yeah. Also product managers are, yeah,
22:28
the user experience team, data, data analysts, a lot of data
22:32
analysts as well, because we were actually, we were still
22:35
giving tools to data analysts. So that's was amazing, like the
22:38
ability to query real time data through simple SQL quiz.
22:43
What the hell, Dimitris? We made an amazing job, but you
22:46
also received correctly. But you're right.
22:48
You know what? What a platform.
22:51
What a platform. Oh my God, you all right?
22:54
Well, I'm patting you on the back a few years too late.
22:56
Like, I think back in time, it was an amazing job that we've
22:59
done. And Mike, it was really a good
23:04
case study to start to learn how to do ML OPS for real and write
23:10
this stuff on the ML OPS community's Live Channels.
23:12
Yeah. Yeah, you got a lot of questions
23:15
answered in there. That's funny.
23:16
That's. Awesome.
23:18
But yeah, as you said, I think the you do need to reel into an
23:23
effort to show yourself to the business.
23:28
That's a bit like to show what you're really doing so that
23:32
people can have an idea at least.
23:35
And and that's not easy. Also, some people will hate
23:38
this, unfortunately. But gosh, please try to put
23:43
yourself into someone else's shoes.
23:47
Try to speak up a little bit more because it's very
23:50
important. Unfortunately, many people I've
23:52
worked with are very are very shy, introvert and so on.
23:55
So the doll like they love coding an amazing quarters, but
23:59
I'm not the best quarter ever. Actually, I'm very I'm very bad,
24:04
but these people are amazing. Like I say, OK, OK, OK, OK, but
24:07
wait, wait, wait. Try to push a bit more on this
24:11
thing. Also, one thing that is very
24:13
important sometimes is the emotional intelligence.
24:15
I think you do need to understand what are the feelings
24:19
of other people while you're talking.
24:21
So be empathic and so on. That's even another level of
24:24
communication, but it's very important so that you can not
24:27
shop by the right communication skills.
24:30
And so for example, I know that we Tom, have to talk in this way
24:34
with Frederick, I have to talk in this other way and so on.
24:38
So being like how to say a space knife where you get all the
24:43
tools that are heating properly where you want, because
24:46
otherwise, unfortunately, if no one is using your platform might
24:49
be like this shiny beast by, you know, no one will use it.
24:53
Your company's actually spending money because the infrastructure
24:56
is up. So the at any way flowing and at
24:58
some point the CEO will say, oh, hey, honour this money.
25:01
So that's a, that's a huge effort.
25:05
Yeah, it's like you have a Ferrari and you're taking it to
25:07
the supermarket. That's what I'm saying.
25:10
Indeed, when I remember we had, I was used to organise for this
25:13
company what's so-called ML clubs.
25:17
So like I was inviting guests from externally from the company
25:22
to show their data science cases and so on.
25:24
And one day we tried to make something internal like inviting
25:27
all the company and show what the data science department is
25:31
doing. So data scientists will do
25:32
these, data analysts will do that.
25:34
ML OPS is doing this. And I think the best definition
25:37
I had for ML OPS was we were trying to build up a Ferrari.
25:41
Data scientists are the drivers and we are trying, we are the
25:45
mechanics. So we're trying to make the
25:47
Ferrari even better so they can go as quick as possible and win
25:51
all the races. But that's, that's the that's
25:54
the thing. And if you want bonus points,
25:56
another thing that isn't changed.
25:57
And it's like people selling you Adobe bespoke solutions, which
26:03
are super expensive, but they don't work, but we can closely.
26:08
Partner, it's a hot take. Wait, wait, wait.
26:09
You can't just stop talking after that.
26:12
Tell me. More so I remember many times we
26:15
had chats with either external companies or cloud providers.
26:20
I would not say which ones. That's I mean, at that time the
26:24
MLS wasn't still quite a thing. So that's why that's why.
26:27
But they were trying to say, you know, you could use the, they
26:31
were just showing an infrastructural diagram like so
26:35
nothing really complicated, something that you could do
26:38
yourself. But of course they were creating
26:41
this for you. And they were asking like more
26:43
than £100,000 a year just for for it plus access to all your
26:49
data. Same.
26:51
OK, wait a SEC, wait a second. This this didn't change.
26:54
I can still still nowadays I can see a lot of solutions which are
27:00
OK, amazing effort. But I mean this can be
27:03
replicated by a good engineering team.
27:06
If you don't have money for an engineering team, then it's OK
27:08
to go for these solutions. Otherwise just develop
27:12
internally. That's my $0.02.
27:15
So this is fascinating to think about because there's always
27:21
that ROI trade off that you're going to be looking at.
27:25
And is it best for your strong engineering team to go and build
27:29
this themselves or to just buy it and then it unblocks them to
27:34
go and build other cool stuff, Right.
27:36
And so that's the that's the fun thing that the leadership I'm
27:42
sure love to hear from you about and say like, wait, so what's
27:47
this and why would we need it? Why?
27:49
Why do you think it's important or whatever?
27:52
Yeah. Well, you hit a good point,
27:54
though. I think it depends.
27:55
Like if you want to deliver immediately something you do
27:59
need to why OK, Like many start-ups at the beginning of
28:04
the on the race of the journey might decide to go for external
28:12
products just because they're, you know, they're ready then
28:15
they just cost some money. But you don't you can't afford
28:19
an engineering team. But if you have your team, I
28:23
think to give them a chance, try to try to understand their
28:29
potential, what they can build, what they can do is also like
28:34
very satisfactory from the human point of view, right?
28:38
And sometimes it's just egregious what some like if you
28:42
look at Sage Maker and the prices on the Sage Maker versus
28:46
if you were to do it yourself, the endpoints are like 8X more
28:50
expensive, right? Yeah, exactly, Exactly.
28:53
Yeah, under a good point. So there are things where you
28:57
can and that's why I'm a huge fan of of fin OPS and figuring
29:02
out are there ways that we're doing things now that with just
29:06
a little bit of tweaking, we can ideally save a lot of money.
29:12
So is there low hanging fruit that we did when we were trying
29:17
to go fast, but we now can go and almost like refractor our
29:26
vendors and who we are supporting and make sure that
29:33
we're just getting the best and the most out of every tool that
29:37
we're using, otherwise we can cut it?
29:40
100 percent, 100% agree. Yeah, absolutely.
29:43
So then all right, you built a few platforms, you learned the
29:48
value of that internal evangelism.
29:54
You also had the experience of learning what exactly ML OPS was
30:00
do you feel like? And this is a a question that I
30:03
constantly am thinking about because you've had many
30:07
different use cases. Are there pieces that you can
30:12
generalise and say in general, I'm all, I'm almost always going
30:18
to need these parts of the pipeline or of the platform.
30:26
If you are doing something with computer vision, it might be
30:30
totally different. And that's kind of why I wonder
30:33
if you can generalise on this like you're still going to be
30:37
having, you're dealing with data, right?
30:39
But a computer computer vision problem is way different than a
30:45
Fintech fraud detection problem. Yeah, yeah, you're right.
30:51
Hundreds $1,000,000 question, but this is this is something
30:57
like I'm bouncing on right now in my current job because we're
31:01
dealing with a lot of computer vision problems like so now the
31:05
our data are videos and not tab on our word and this is a
31:11
drastic change. So like I would say that on the
31:16
paper like you need to carefully subdivides all the all your
31:23
problems. So like the the starting point
31:26
is always the data, right? So how can we make sure we have
31:32
a data platform? How we can make sure that state
31:36
engineers know what we want and people can create their data
31:40
easily? What does it mean querying a
31:43
video? As you know, it might be
31:45
querying like a path to a bucket where you have a video.
31:50
I might be creating metadata of the video.
31:52
So if you're seeing this way, it looks like the tabular problem
31:56
and the computer vision problem might be similar.
31:58
It's a query, you need metadata, you need to know where your
32:02
video are. OK, this can be, this can be
32:06
maybe generalised. So overall data pipelines are
32:09
always there. And maybe for computer vision
32:12
problem they yeah, also for computer vision problem, for
32:15
example, thinking of general case, they could validate data.
32:18
For example, you might have a video whose length is 0 seconds.
32:22
You see what's a video, how old is 0 seconds?
32:26
Or you can extract like audio information, right?
32:29
And from there you can make up of some validation tools.
32:33
What for example is not generally is the way of course
32:37
you're treating the video. So you must be critic enough to
32:41
understand what's the is this case you're investigating over?
32:45
Do you need a yellow detect because you want to detect
32:48
something or you want to, as you know, general description of
32:51
videos and so on. So, but here we're going on to
32:54
deeper level. So I think on the general point
32:56
of view, maybe data side, we could say, OK, there's a
33:00
database. The database may work
33:03
differently. But overall, what are we going
33:05
to use? I don't know, Athena, Postgres,
33:08
Big Query, Bigtable, yes, these tools are always on the standard
33:14
side of data OPS. And then you, you move forward
33:19
and you start to think, OK, the next problem is how I can make
33:22
people training a model. So I think personally for
33:26
training models, we're getting more, we're converging to more
33:30
best practises and standard ways.
33:33
So like, OK, I know that you're lost by him.
33:36
I'm a big fan of cube flow law for cube flow, but the matrix I
33:40
had. Got nothing against Cube.
33:41
Flow all the power of AI loss for cube flow.
33:45
Look at that. This is AI for Mac.
33:49
So like I think overall what you need is to create a training
33:55
pipeline, right? A pipeline that is able to
33:58
retrieve your data and then crunch this data and generate a
34:04
model, whatever. Now let's not take into account
34:09
the lens for just for a second. If we need to create a training
34:12
pipeline, this problem can be easily generalised.
34:16
Also also, yeah, in my opinion can be easily generalised
34:19
because what's the difference between creating a random forest
34:23
and creating a model that maybe is running a, a yellow algorithm
34:30
or an object detection algorithm?
34:33
At the end of the day, the pipeline will retrieve the data
34:37
that you need. It will process the data that
34:39
you need, it will train your model, right.
34:41
So more or less, and This is why I'm a strong supporter of cube
34:45
flow as because if you're also reducing the learning barrier to
34:49
your stakeholders and cube flow is very easy to to be used
34:54
because you just see it at component at pipeline.
34:57
The engineers will create us SDK for you.
34:59
So you can just ship whatever you want.
35:02
OK, we're a step forward. So we're reaching a great level
35:07
then different, different points for MLMS. LLMS of course are
35:13
huge models. So the way we are training also
35:16
these models is just slightly different.
35:19
You need different infrastructures.
35:21
I don't think cube flow may support any LLMS. I never heard
35:26
of it but. Yeah, I haven't heard that
35:28
either. In principle, but better not to.
35:32
There are better methods to treat modestly.
35:36
So OK if you go down the truth, but the matter is a question for
35:39
you. How many companies are using in
35:41
training elements? Yeah, very few.
35:44
OK, like so 90% of the companies nowadays will still enjoy music.
35:49
And sorry if I'm saying logistic regression, Randall Forest,
35:52
super deep forest algorithms, right?
35:55
So guys, think carefully about the cost you're injecting in
35:59
your company because it's meaningless like spending
36:02
$1,000,000 a month for training on NLMS from scratch when you
36:06
can do a 95% accurate and amazing job with a random
36:13
forest, whatever forest you want or whatever arguments you want,
36:18
right? The last step is of course the
36:21
deployment. So deployments, I think we are
36:23
not converging yet. In my personal opinion, there
36:26
are tonnes of way to deploy models and this is really
36:32
depending on any company. So like there might be companies
36:36
who would like to use a Vertex AI endpoints, OK, pay the money
36:42
and we can deploy them. There are companies which say
36:45
no, no, no, we have, we're going to hit an EC2 instance, we're
36:50
going to have our own Kubernetes infrastructure.
36:53
So we go to the endpoints we want.
36:56
OK, computer vision. That's another terrible problem.
37:00
You need machines which are of course able to do the video
37:04
rendering as well. So here I think the navigation
37:09
is a bit complicated but I hope we are converging to specific
37:15
specific. That action, I think you know,
37:19
Andy came on the podcast a few weeks ago and he was talking
37:24
about how for him, the change to LLMS versus traditional machine
37:32
learning and also deep learning or computer vision and all that
37:35
fun stuff doesn't really change much about how he thinks about
37:39
the platform. What it does change is how he
37:44
builds or the tools that he needs to add to the platform,
37:49
right? And so it's what he was saying
37:51
and I really like this idea. I, I like most of Andy's ideas,
37:55
but I, I thought his idea around creating an ecosystem, allowing
38:01
your platform to be looked at as an ecosystem where if someone
38:06
wants to use an LLM, they can. If someone wants to use that
38:08
traditional ML, they can. If someone has a computer vision
38:12
problem, they can. And obviously that is like
38:15
maturity level 500 versus your maturity level like 1:00.
38:21
And you're just trying to get a few models out and make sure
38:24
that they are reliable and consistent.
38:27
But for me, what became very clear is that LLMS are very much
38:35
a part of the conversation and what people want when we talk
38:40
about doing ML and AI these days.
38:43
And so how can you set up the platform now to leverage the
38:49
gigantic transformer models that are out there, whether it's an
38:53
open source llama that you grab or it's you're hitting GPT 3 or
39:00
GPT 4? Amazing question again.
39:04
So I, I think I partially agree with what Andy said because this
39:09
is the dream platform. So like it's something we're
39:13
always trying to create, right? For any company like you, you
39:16
got this metaverse where you can ping the service you want and
39:21
this will do all the rest when it comes through the LLMS. Now
39:25
the discussion is like as you said, we could actually using
39:29
like Hugging Face for example, APIs or calling public APIs to
39:34
use these LLMS. Nothing against, that's totally fine.
39:38
And it's not so complicated to have them deployed because all
39:42
you would need is just a simple micro service that spins out for
39:48
your request. So your engineering team could
39:51
even create a simple package where you're just importing all
39:55
the models you need. And these models can be freely
39:59
being from the the score from the Internet.
40:02
The only cabin sometimes that I'm thinking of these solutions
40:05
is what if there's downtime on the external services?
40:09
So you're blocking your business.
40:11
OK, the what's the name of the guarantee service?
40:19
SLA's or SLA's or proxies that you?
40:22
Can SLA the SLA's like 90% of usage of the year.
40:28
OK, but you need to consider it. You don't want to block the
40:31
business. And what's the cost for pinging
40:35
these models and what's the number of customers that you're
40:39
dealing with? Because now, like for Synthesia,
40:43
when I'm working out, we using external models would be pretty
40:49
hard for us because we are serving thousands, thousands,
40:53
thousands of people and we do need GPUs, right.
40:56
So what could be the cost that we should support an external
40:59
provider will charge you a loss say, aha, you need GPUs.
41:02
Yeah, don't worry, here's the bill.
41:05
And of course you need to start thinking something that can be
41:10
also money saving. So what if you can use spotties
41:15
and says, what if you can use maybe sharing the same process
41:21
across multiple GPUs? Sorry, different process across
41:25
same GPUs and so on. So it really depends on your
41:29
business. That's why I say you do need to
41:32
put things into them and OPS perspective and start to
41:34
understand really what's your business value?
41:36
What's your what's your business want to do because the step from
41:40
going from spending zero to spending $1,000,000 is very
41:44
narrow nowadays, especially with this alliance because everyone
41:46
was used all that Yeah, so but a little hard to say, yeah, I'm
41:51
I'm actually a love and this idea It's definitely amazing.
41:55
And it's also easy to to realise it's not something impossible.
41:59
I think nowadays we can do it in a not seen a week, in a month we
42:05
just some terraform in the middle and you also have
42:10
everything as a code. Overall, it really depends on
42:13
the business. So that's why it's very hard to
42:14
give you a neat answer on this. It's funny, I've been trying to
42:20
craft the perfect tweet around this idea of you want to know
42:25
how to be really valuable at your job as a data engineer, ML
42:32
engineer, data scientist, get really good at understanding
42:39
auto scaling and make sure that you have that on point and you
42:45
are going to be very loved in your job.
42:48
I was just scaling and also let's go for some multi
42:51
processing which is always useful.
42:53
But auto scaling doesn't come for free, which means it's true.
42:58
You need to understand it, but try to understand as deeply as
43:03
you can. Like, because I think there's
43:06
always a surprise when you're using auto scaling mechanisms.
43:10
We, we spotted a lot of times when we were using our
43:14
Kubernetes infrastructure that auto scaling was either going to
43:20
skyrocket level or or wasn't working at all.
43:24
Why? Because all the, all the
43:25
settings were right, right? We set up everything correctly.
43:28
We say don't use these. And you was using you say oh
43:30
shit, so out of scaling is game mechanism.
43:35
But as you said, started deeply. What what how to use it exactly
43:39
Because they can also skyrocket your bill as well.
43:42
Just go back on the bill discussion.
43:45
So I think only if it's spinning up A1 hundreds under A-100
43:50
instances are the same type. So yeah, but by your rights,
43:54
you're right. This is something that nowadays
43:57
it's getting more and more important.
44:00
Well, that's why I was saying that you're super valuable if
44:03
you can be the expert in the auto scaling, because it's so
44:07
easy to lose money so quickly. Yeah, if you don't configure
44:11
everything properly or you don't set up the alerts or the limits
44:14
or whatever it may be, then I get the feeling that everyone's
44:19
gone through that story that you just told where you're like,
44:22
huh, I wonder why it's so expensive all of a sudden.
44:26
Yeah, yeah, exactly. So bear in mind, but very good,
44:31
very good advice really. Yeah, that's what I've heard
44:35
some people put it, that when you have those mistakes, like
44:41
the auto scaling mistake, and it is a little bit expensive, all
44:45
you did there is you paid a dumb tax or a stupid tax, and that's
44:50
the tax you have to pay for being stupid.
44:52
Yeah, well. That's a different price, isn't
44:55
it? Now firstly, we, we, but let,
44:58
let's pass this message. We're not perfect.
45:01
So, so to everybody, especially to managers.
45:05
If engineers do mistakes, it's OK.
45:09
We're human. So there's no way I, I made
45:12
tonnes of mistakes. Actually, when I was delivering
45:14
the shiny amazing ML platform for that week, our main models
45:19
was model wasn't running correctly.
45:21
Why? Because I forgot to add a flag
45:24
to the model. A simple flag like just a a
45:28
dash, dash something text to the model.
45:30
I say, Oh dear, so so it's mistakes are there even in
45:36
production cost some money, but what can we do?
45:42
What are some other ways that you or some people that you've
45:46
worked with have had to pay that tax?
45:50
Many people, the people and they're working with this, I'm
45:53
doing this and they're always, always huge problems in
45:59
production. Like there's has people that's
46:03
left some even simple Lambda functions running in the
46:08
background for, for standard systems, right?
46:12
And, and gosh, unfortunately they were spinning up God only
46:15
knows what's because the setup was from and it's been like
46:19
$50,000 in a weekend. Oops.
46:22
Well. Yeah, you log out on Friday.
46:25
My job's run here. It's not.
46:28
Come on. But like, gosh, I have some
46:34
stories, but I can tell you, but there are many stories of people
46:37
like even the simplest how to scan you can think of, like data
46:41
flow itself. That's also is is a problem that
46:45
I like because allows you to create the easy data pipelines
46:49
with Apache Beam. It has an auto scaling
46:51
mechanism, right. But as you said very deeply how
46:56
it works because just mess up with a number puts like 100
47:01
nodes as a maximum whatever and auto scaler doesn't recognise
47:04
this number and I do. It has happened that it's going
47:09
to scale up to 1000 nodes. And of course you say why
47:11
there's a spike in the cost there all idea and it's a stupid
47:15
problem again. So these stories are I think
47:19
they're always easy. I I actually remember OK, this
47:23
is I don't know if I should say this by wheel.
47:25
When I started my career as an ML OPS engineer, the very first
47:29
thing I've done was to also transition from AWS to GCP.
47:34
And I didn't know how exactly this company was working with
47:36
GCP said OK, these are my keys in Jason.
47:40
OK, as you know, maybe it was it was during the lockdown and so
47:44
it wasn't certainly my right shape.
47:47
I just you were. Watching videos on WhatsApp
47:50
while you were deploying. I just do a git about
47:54
everything. I even push my GCP keys publicly
47:58
on git apps so anyone could have actually access to GCP.
48:02
Our GCP environment, said warps. Thank God.
48:06
Here it is a matter of now, what's Enelop's sacking for?
48:11
Sex security, right? Immediately spotted to say oh oh
48:14
oh oh, here there's a huge, huge stereo of black weight and so,
48:20
well, I'm the first one to say we're human, things happen.
48:26
Yeah, Yeah, that's fun, man. Well, another thing that I'm
48:29
really excited about is the course that you just put
48:33
together for the ML OPS Community Learning platform.
48:36
And I think it's pretty special because you basically walk
48:41
through the process of standing up a platform and it's soup to
48:46
nuts. It's very thorough and you're
48:49
using one of my favourite platforms.
48:50
You show people how to use cube Flow, right?
48:54
Yeah, exactly. And you know how to install Cube
48:57
Flow locally, which is an iser, Yeah, so don't you said.
49:03
It I'm glad you said it and I didn't because I.
49:06
I've even said this in the course, but don't try this at
49:08
home, Alex, you're following my course.
49:11
And so yes, yeah, I so the, the idea of the course is just to be
49:18
an interface to students or people who are interested in ML
49:22
OPS. I want to give it a try and
49:24
start to learn really. Hamel OPS And I think this
49:28
refers to the discussion we were having about how we can
49:30
standardise envelopes. And so if we can find common
49:33
patterns for most of the tasks which we have to do.
49:36
So it covers like data processing, model training and
49:40
model experimenting or tracking, not model deployment.
49:43
Because I said there are multiple solutions.
49:45
It it wants to be a very first gentle course for everybody.
49:49
The the idea is to just guide students and people throughout
49:56
what the industry wants and what I've learned so far, which might
50:04
be a very simple course to follow to have the right
50:08
transition into the MLS perspective.
50:10
So I'm spending a lot of minutes of the course explaining what's
50:14
the ML OPS perspective, so how we should subdivide each problem
50:17
into the MLS perspective. So data processing, what it
50:20
means and why, what's the modern training from the data science
50:23
point of view and from the ML OPS point of view and why we
50:27
need to track experiments and so on.
50:30
So it's like me guiding with my hands all throughout the course
50:33
and showing said, hey, be careful here and giving a lot of
50:36
cameras, but also a lot of freedom.
50:38
Like I don't want people to get crazy with quiz or tests and
50:43
say, oh, don't have time. I want people to enjoy and
50:46
practically on their own. We're using the GCP platform
50:51
which comes also for free if you want to start with initially and
50:58
that's a good case for you to learn, have your hands dirty,
51:02
which is my preferred approach and start really to make up.
51:05
I also give bonus pipelines like a pipe and the Skype data from
51:09
Google or and gives you clouds of words.
51:16
So there are many little things, but it wants to be really what
51:20
is missing, like a gentle introduction to this ML OPS.
51:22
Word. Well, the cool thing about the
51:27
ideas that you propose there where you're saying there's the
51:31
tech part, which is great, but inevitably that's going to
51:36
change. That's going to constantly be
51:38
changing as time moves on. There's going to be better ways
51:42
to do it. But the mindset and the way that
51:47
you approach these problems and understanding what the motives
51:51
are behind the data scientists, the data engineers, the ML
51:55
platform engineers, what each job and responsibility and how
52:00
these folks think about it is going to be invaluable as you
52:03
progress in your career. Exactly.
52:06
So what? What's a sorry?
52:11
What? Also, to give to people is a
52:15
conceptual or critical way of thinking, right?
52:18
So the technology stack is that one.
52:24
We can maybe make it sound or whatever, but the cool thing is
52:28
be critical about the tools that you're going to use.
52:30
And this is the first also to all of our conversation.
52:32
So what I'm showing as well is by Apache Beam, why we should
52:36
use it, why we should not use it, What are the pain points of
52:40
using it, pros and cons and so on.
52:42
For all the packages, for all the products that we have a good
52:46
engineer as maybe a structured way of thinking saying, OK, my
52:52
business has this problem. I can see on Internet that we
52:56
have these tools to solve it. Why should I use tool A rather
53:01
than tool B and so on. So then this refers also to all
53:06
the soft skills I was speaking about before.
53:10
And it's all part of a big soup that's it has a huge value.
53:16
Well, and just thinking about the questions that you should be
53:19
asking, what, how should I examine this issue that's in
53:24
front of me? I'm it's funny you mention that
53:27
because I'm literally reading this book right now called
53:29
Bulletproof Problem solving. It's by a few ex McKenzie
53:34
consultants bigwigs, but it talks about breaking down very
53:39
complex questions or problems into decision trees and, and
53:45
understanding them and being able to tell where you need to
53:49
do research and what kind of outcomes you're looking for and
53:53
defining the problems. I'm really loving it.
53:56
I've I've heard that some people are like, ah, but it takes the
53:59
creativity out of things. I don't know so far.
54:03
I think that might just be the the way that you approach it and
54:07
creative people are going to be creative in general, but the the
54:13
idea is sound like how do I approach a problem?
54:16
How do I know which questions to ask and when?
54:21
I'm looking at AML platform and I'm thinking about different
54:25
tools that I want to add in here.
54:28
What kind of questions should I be looking at?
54:31
I think that's one of the reasons that so many people like
54:33
Jeresa's book, because in each section of the data engineering
54:40
life cycle that he maps out, right, he has what you should be
54:45
thinking about. What question should you be
54:47
asking in this life cycle piece, right?
54:51
And so if it's ingestion or if it's transformation and by how
54:56
you answer that, it gives you a few constraints and then it
55:00
helps you make your decisions a lot easier.
55:03
Yeah. Because if you're just going off
55:05
of, oh, well, it's new, it's got a lot of attention around it.
55:09
I've heard, I've heard good things from people about XYZ
55:12
tool. That's not going to be the most
55:15
adequate way of doing it. And so, yeah, man, I appreciate
55:19
you creating the course. Hopefully a bunch of people out
55:22
there go and they check it out and they learn a tonne from you.
55:27
This has been really fun. Oh, it's really nice.
55:30
That was really nice. I hope people will like it.
55:34
And also they can write a comment and say, Stephanie,
55:36
you're being too serious. Please add some fun to the
55:39
course step. Maybe some point.
55:41
I was very serious where when it comes to explain cube flow, how
55:44
to install it, you need to embrace yourself.
55:48
But. So this if this is the one O 1
55:51
course, what's going to be the two O 2 course?
55:53
What's the next? Edition because all he knows,
55:55
but definitely something about OK, everyone is trying to do
55:59
something about LLMS. So it might be training LLMS dealing
56:02
with LLMS so we can explore that path.
56:06
It might be even something either LMS or something very,
56:09
very, very technical like infrastructure as code.
56:12
Nice case, Terraform. First a play guys.
56:16
Terraform. We do need it.
56:19
We do need it. Having our.
56:20
ML platform being screwed up so the majors we can talk about it.
56:27
I love it and if anybody has any other ideas on courses, just hit
56:32
us up in the comments. And because we're building out
56:35
the ML OPS community learning platform, and I think it is a
56:39
great excuse to get some of the incredible people like yourself
56:44
in the community to create courses for the platform and
56:48
disseminate the knowledge amongst the other community
56:51
members. Two kinds.
56:53
It's a way to give back to the community because there's some
56:57
mutual benefits to learn. Also from the Slack channel.
57:01
Boom, there we go.