Latest / Agentic Conversations (formally mlops.community) / Building an ML Platform: Insights, Community, and Advocacy // Stephen Batifol // #178

Transcript

Building an ML Platform: Insights, Community, and Advocacy // Stephen Batifol // #178 · Agentic Conversations (formally mlops.community)

0:00
Say I'm Stefan, beautiful. I'm a developer relations
0:02
engineer at Vault and I drink my coffee with like old latte in
0:07
the morning after I'd run my beans.
0:09
And yeah, I have my own coffee to love it.
0:13
Hello and welcome everyone to the ML OPS Community Podcast.
0:17
My name is Dimitri Ose and I'm your host for today and I'm not
0:22
with anybody but I am with somebody that likes to come on
0:27
here and be a Co host. This time he is my one and only
0:30
guest and I get to talk to him all about the internal platform
0:35
that they have created at Vault. He was an ML platform engineer
0:41
and since has graduated per se. To an internal developer
0:46
advocate, so we get into how he goes and advocates for the
0:52
platform use and how they allow the teams and the data
0:56
scientists at Vault to use whatever they want.
1:00
And that makes his job real interesting because he has to
1:05
advocate real hard for them to use the platform that he
1:08
created. I loved hearing about all of the
1:12
nuts and bolts on the platform and how he actually brings the
1:16
data scientists onto the platform and creates the
1:19
internal community of users. And then we get into a little
1:23
bit of a tangent. On the tech scene in Berlin and
1:27
how fun Berlin can be. So hopefully you enjoy.
1:31
Let's get into the conversation. And last thing before we go, in
1:36
case you haven't already, it would mean the world to me if
1:38
you like, subscribe, review on whatever platform you are
1:43
listening to this on. And a huge win, absolutely huge
1:49
win would be if you could share this with just one friend that
1:54
you think would find it interesting.
1:56
So now let's get into the conversation.
2:05
Let's go because I have to go see Bobby at 8:15 tonight.
2:10
Ohh, who you going with You with A friend of mine actually.
2:14
Alright, so I heard it was actually really good.
2:16
And so yeah, I did not expect it.
2:20
I did not expect it to be that good.
2:22
But I heard I was listening to a podcast, funny enough, Smartlist
2:27
about the actually the director. They interviewed the director
2:31
and they were just praising her on how good she is with scripts
2:35
and how good she is with making scripts sound natural.
2:40
OK, so let's see. Yeah.
2:43
You're not doing the Oppenheimer on top of it, though.
2:46
No. Tonight's only Bobby.
2:48
Only so always. You know I'm getting old.
2:50
Already struggled to to like, stay awake in the cinema.
2:55
Ohh, that is hilarious. So yeah, that is so good.
2:59
Well dude, you're doing all kinds of cool stuff at Volt.
3:02
Yeah, and you are. You were previously on the
3:05
platform team, right? That's when we first met.
3:08
You were killing it on there and having lots of fun giving talks
3:12
at like Cube Con about what you are doing on the ML platform
3:15
team. But now you've since joined a
3:19
different team, which is a team that you made-up.
3:24
You created your own team. Basically, tell me about how
3:26
this happened. Yeah, basically creating world
3:29
role now adults. And so yeah, I've been in full
3:32
form, was three years and was the first missionary engineer
3:36
and working on the management platform.
3:38
It was fun, but after a while, you know, there's like so much
3:42
YAML you can do good to a bad out or something.
3:46
And I was starting to be a bit annoyed, like border platform
3:49
work and everything. And also like, yeah, I gave talk
3:53
at coupons. I help a lot of data scientists,
3:55
you know just try the regular life easier.
3:58
So I would probably like, yeah, we should have a devil, like we
4:02
should have an internal device that evolved because we create
4:06
our own machine learning platform.
4:07
But the way I see it is that at this, my opinion in my view is
4:11
that we compete against the laws directly and they're platform
4:14
because we don't force anyone. Yeah, like Sage Maker and the
4:17
whole, you know the whole offer because we don't force anyone to
4:21
use our platform. They're getting still use it.
4:24
So like, yeah, when I competing against it.
4:27
So data scientists have to be like aware of what we can do
4:30
with it. And I'm trying to make their
4:32
life easy, you know, like have like think of like
4:36
documentation, think of getting stalled and things of
4:38
factorials, They're like basically they're well, but it's
4:41
just internal. So.
4:45
What does Deborah mean to you? What is that?
4:48
Very good question? I had to think so hard for it
4:51
because I had to create the job ad as well.
4:54
Ohh I bet it says it's at least for me is like there is some
4:59
kind of community we have adults which is also small data
5:03
scientist that is already like to take care of that you know to
5:07
to to foster it to really make sure that they're all happy and
5:10
we like we can share knowledge as well.
5:12
That's one idea. The other one is also to make
5:16
sure that's on boarding guides are good, like every
5:19
documentation is good if you can.
5:21
If you have questions or anything, we should have talked
5:24
about it. And it's also really important
5:27
about the developer experience. I don't know.
5:30
I care a lot about things being simple, and I hate gatekeeping,
5:35
but her word. So I'm always like trying to
5:37
explain you things in a simple way, even if like things are
5:40
supposed to be complex. So yeah, that's really tried to
5:44
to basically like you know, our whole platform is running on
5:46
communities for example. Yeah.
5:48
And I don't believe that data scientists should have to know
5:51
about capabilities if they use a platform, if they want to, they
5:56
can. If they don't want to, they
5:57
shouldn't have to. So yeah, it's like, it's those
6:00
things. And it's also then talking at
6:03
conferences a bit. But that's the part that people
6:06
think we always do all the time. And I'm like reading.
6:08
No, no, no. Coming in influencer, Yeah,
6:12
gonna take selfies now, you know, every day at the office.
6:16
Yeah, exactly. Making everybody get jealous?
6:19
Exactly. You don't work at all?
6:22
Exactly. Enjoy votes.
6:24
Food deliveries, Yeah. Like I go to all of our
6:27
countries. You know, it'd be like, Ohh,
6:28
yeah, how to go to Israel for conference, obviously.
6:32
Yeah, and talk to the teams there.
6:34
And for those who aren't listening, Vault is basically
6:37
the DoorDash of Europe. Exactly, Yeah, exactly.
6:42
We go from, we go from Norway to Japan now with live in 25
6:46
countries. Wow.
6:48
Actually, now it's Iceland to Japan.
6:49
I have to to change its surf. Well, even more north, Yeah,
6:53
exactly. And but it's not only food
6:56
delivery, right? And that's the other thing.
6:57
It's like not only delivering restaurant food, it's delivering
7:01
like groceries. Yeah, like we started this food
7:04
delivery and now like we cannot deliver everything.
7:07
Depending on the country, like in Finland where I was living,
7:09
there we could deliver Christmas trees for Christmas.
7:13
Also jewellery. Like I've seen jewellery pieces
7:16
that are worth more than €2000 on Volt, but yeah, it's like
7:21
more than for the right now. So yeah, groceries.
7:25
I think. All the bulkheads.
7:26
So lots of. Like estimated time of delivery
7:30
use cases, lots of real time use cases.
7:33
I imagine that you are having to support with the ML platform and
7:37
you're hoping the data scientist can level up on what is the
7:42
actual platform look like that you're going around in
7:44
evangelising and like why should anybody use it?
7:47
I guess give us our pitch. It's amazing, you know, No.
7:50
Bias. Yep, You sold me.
7:52
That's all I needed to hear. And that's so also, yeah, one of
7:57
the the thing we have with my platform that is fully open
8:00
source, so we use open source tools.
8:02
That's because it's free, because just because Gold
8:05
believes in open source also contribute to open source, It's
8:08
not only using it. And then yeah, the whole stack
8:11
for now is flight for like ETL, latest transformation and
8:14
everything that was created like lift back then.
8:18
Then we have the good old evil flow in the middle.
8:21
You know it's still here. It's the part of every text
8:23
type, almost. And anyway.
8:26
That's true. Yeah.
8:27
Got in early and then got stuck in there.
8:29
That is so true. And there's.
8:31
Yeah. And there's also one thing and I
8:33
I think those articles stopped now, but those Amanfro versus
8:36
cube flow at one point, I just can't see them.
8:40
I think people just got sick of both of them.
8:42
Yes, And nobody cares. Everybody's like, I don't give a
8:44
fuck about either of these. Get them out of my life, please.
8:48
Yeah. The thing that have in common is
8:50
flu in the name and that's it. Flow and lots of hatred on the
8:54
developer experience side of things.
8:56
So yeah, I guess the reason it was so popular probably was
8:59
because how painful it was to use them.
9:02
I mean ML flow less so than cube flow, but that's a.
9:07
Yes, up for discussion, right. Yeah, I guess for tube flow for
9:09
me is like I always sell myself as the hate club, hate club of
9:14
health club. Like even at conferences people
9:15
always ask me like, ohh, why didn't you go with cube flow?
9:18
And it's like, have you tried using it?
9:22
And I really like what they did actually like the idea, you
9:25
know, I really like the idea. It's more like the indication
9:27
like from a bit more serious, you know, like I was the
9:31
multiplatform trying to deploy something and I tried to use it
9:34
and then I was struggling, I would.
9:36
Struggle with like just to like run the TSL, you know, like
9:40
compile everything, like those things.
9:41
And it was like, OK, like you have to learn the whole language
9:45
and also how it works exactly. And I feel like it's thought
9:48
that I'm like amazingly small. But if I'm the one pushing for
9:51
it and I'm struggling, imagine the data scientist they'd be.
9:55
What is that? Ohh.
9:56
That's like the whole reason the MLS community grew in 2020 when
10:02
all these data scientists were trying to figure out cube flow.
10:05
That was it. They came into the community and
10:07
they were like, what is going on with cute photos?
10:09
Anyone know what I can do to make this easier on me?
10:13
And then people were like you should learn Kubernetes and yeah
10:16
4 date and time. Then they became SR's instead of
10:20
data scientists. Exactly.
10:22
But well, you're you're talking about.
10:25
Keeps the pipelines right? Or are you talking about the
10:27
whole package kits or why it was mostly capital pipeline in case
10:31
it was not a thing back then? It was like they only like
10:34
separated it. I don't know like a year and a
10:36
half ago or. Something, yeah, it wasn't too
10:38
long ago. So yeah, back then it was like
10:41
the whole, but it was mostly pipeline, you know and I tried
10:43
to run things in parallel and and I just like that was like
10:47
really struggling. And then I think back then we
10:50
were like evaluating different things, but we really wanted it
10:53
to run on on competitors and then we went with flights.
10:58
I will buy a like. It's not like it was easy with
11:00
lights, you know like kind of triggers back then there was
11:02
still young as well. It's still fairly young project,
11:07
but yeah, like they were both bugs and stuff, but they've been
11:09
really helpful and yeah, if you're right by it then then you
11:14
basically don't have to change your code, you know?
11:16
So that was, that was the main thing.
11:18
I was like, yeah, that's what I'm hyped about.
11:21
So it was like Kubernetes Python match made in heaven.
11:25
Let's go with that for our pipelines exactly.
11:27
But are you like, break it down for me because this is the ML
11:31
platform? Is there a data platform
11:33
underneath that you're dealing with or you're interfacing with
11:36
and how much? Say, did you have and how that
11:39
gets created? Yeah.
11:40
So before I joined, there was a whole data platform which is
11:44
flow. I mean, we'll have Snowflake as
11:46
our warehouse and then we had the outflow for like, you know,
11:50
like, yeah, all those pipelines and we had a lot of M&M and
11:54
things actually running on our floor.
11:55
The only problem is that once you do ML, then you install CUDA
11:59
and then everything is 10 gigabytes.
12:01
So. So he worked, he worked quite
12:04
well. Like did you know voters that
12:06
ceases all bow or something? Yeah, and those things were
12:10
running for like 2 years. So it's also like was not
12:12
optimal, but it was working without flow.
12:14
And if you understand however it works, you know, like how to
12:19
like with the deck definition and everything, you know that
12:21
it's not too bad. They're just at one point we
12:24
would like, yeah, like let's have something else and now we
12:27
have nothing running on our floor, at least like nothing
12:30
about no e-mail workflow, but. You're OK in the MLS side of
12:33
things? Alright yeah I'm outside is
12:35
everything is removed but yeah like we good to look snowflake
12:39
to airflow and then after you have got some nice dashboards
12:43
other places you know like those local dashboards they look at.
12:46
I've never I tried to write Luke ML wanted to know I was like no.
12:52
That didn't work out for you, you know.
12:54
But you Sorry, I cut you off. You didn't explain everything,
12:57
so ML flows in the middle. But then what is deployment and
13:01
what is like? Yeah, reality, all deployments,
13:03
we use Solid Core, which is also open source.
13:07
And for those who don't know, it's like basically based on the
13:10
model. That's a fact, create
13:11
microservices on capabilities and it supports a lot of
13:14
libraries and it creates endpoints.
13:19
You would need to create yourself.
13:22
Symmetrics. It supports shadow deployment,
13:24
AB testing. You know what things you would
13:27
like need at one point, but then if you were to do it yourselves,
13:31
you actually write a lot of code, listen.
13:34
And I didn't want the other platform to have to write the
13:37
same code all over again or to create something central if I
13:40
had to avoid it. Makes really lazy.
13:43
So you know, like I'm like, if anyone has done it for me that
13:45
I'm happy, yeah, so that's what we're using and it's the one
13:50
that have been praising the most.
13:52
Actually, it's my favourite too. I don't like other people in the
13:54
team. Sometimes they are like that,
13:56
and unless it's the one that I like the most because it's like
13:59
it's really my vision of things. Like you just define a model,
14:03
the library you used to train your model, and then behind the
14:06
scene it creates so many things. It's a data scientist.
14:09
If they don't have to think of, you know, the routes like the
14:13
monitoring, the streaming sort of deployment, those you don't
14:17
have to think about those if they don't want to and that's
14:20
what I like. There's.
14:22
The idea of, well, I guess observability is baked into it.
14:26
Is it? What is it?
14:27
Alibaba or yeah, there's alibi alibi that Alibaba is.
14:32
Yeah, that's a funny name. I always get it confused with
14:35
Alibaba for some reason but. Yeah, it's the Alibi.
14:39
Yeah, they have like they have those.
14:40
Would only use those yet to be honest I kind of feel like.
14:44
But you're doing some kind of monitoring or no, Yeah, we do
14:46
have money trying but it's like you have monitoring on
14:49
performance wise but we don't use alibi, you know just like
14:52
her. We compare the grand truth with
14:55
the predictions. We do like everything is pushed
14:57
to Kafka on our side and then you have some nice dashboards
15:00
where we compare everything, but we just don't do it like in
15:04
real, real time basically because we what else related
15:08
just influence. Yeah.
15:10
Well, I mean, yeah, breakdown the real time because if you're
15:13
doing all of these estimated time of arrival type things that
15:18
needs to be real time, right? But like how real real time
15:22
exactly. Yeah, it's always the thing.
15:26
So for us, I think it's around 5 minutes.
15:31
Well, 5 minutes late for now and like we're working on improving
15:36
it that I can't really go into the details on here because
15:38
that's the most important part of all, basically.
15:41
But we are like around yeah around 5 minutes which is like
15:46
fine like in most cases, you know like phase not really
15:50
changed, like traffic is predictable except if you have
15:53
something like an accident. But then we used I think we have
15:57
like data from different problems as well like Google
15:59
Maps, Open St Maps and like we used those as 1.
16:02
So it's also like a lot of non annual things.
16:06
Yeah, you're getting tipped off about that, yeah.
16:08
And Confluent is in there somewhere?
16:11
Or Kafka. Sorry, maybe not Conka.
16:13
Yeah, yeah. Kafka.
16:15
How is that being used? So Kafka is, well, almost
16:20
everything we get from the app. You know, like we push
16:23
everything. All the app is pushing
16:25
everything and then we just Africa and then will have the
16:28
features for some M models they're coming through Kafka
16:32
directly like those that that's real time for example like.
16:35
Are you using Flink to transform them?
16:37
No, not yet. At least Kafka and yeah then
16:45
better process. Yeah, we're really looking into
16:48
Flink actually to get transform that, transform that, but it's
16:52
like we only starting a mess. There's nothing using it.
16:56
Yeah, like we haven't started like deploying Flink as one of
16:58
the clusters. One thing that I'm fascinated by
17:01
is your. What you originally said on you
17:04
want the developer experience to be nice for these data
17:06
scientists because at the end of the day, these are the users and
17:09
so you're trying to champion and advocate for the data scientists
17:13
to use this. What is it that you're telling
17:16
them? Like, what's the selling points
17:18
when you go to them and you're like, hey, there's this
17:21
incredible tool that we've created and it's using all of
17:24
these technologies and they're like, I don't really care what
17:26
technologies it's using. What does this mean for me?
17:30
Yeah, I think usually how I start.
17:32
So it depends on where they are. It's going to depend on do they
17:35
want to train the model or do they want to deploy something
17:38
they've trained on, you know, on their laptop or something.
17:41
If they want to trade a model, usually one of the best selling
17:44
point I have is that ohh yeah you're running track like train
17:47
it on 25 countries. How you're going to do that?
17:50
Are you going to use edge maker and then you go through a list
17:53
and then you do them one by one. Are you going to try to make
17:55
that in parallel by writing some spark?
17:58
I don't know. I don't know like what do you
17:59
have in mind? And then I'm like yeah, you know
18:01
what we have flights. You had decorator or your
18:04
function, You had dynamic decorator and that's it.
18:09
Wait and like. Then they can train the model
18:12
with 25 countries by just saying like, yeah, you just say like,
18:15
Oh yeah, this workflow is going to be dynamic workflow for
18:17
example. And then you give the list of
18:21
model where they can be dynamic that you can change like you
18:23
read the list and then you be like please look through that
18:27
list, train 25 models because we're 25 countries, one example,
18:32
we have one model per city, we have a lot of cities.
18:37
So, so they do that. For example, they train over
18:40
like itchy, they don't even know how many they have because they
18:43
don't have to define anything. So like 1 selling point that
18:46
like that I have usually it's like, yeah, OK, like if you need
18:50
like some real training and some real model probably want that
18:54
there were hundreds like it's out-of-the-box.
18:57
Same for like resources. You want some GPUs?
19:00
Ohh yeah, crazy things you know. Well we haven't done.
19:04
So maybe you can go on stage maker and do your thing.
19:07
Good luck trying to get somebody like, you know, there's reports
19:10
like you're on your own. Yeah like have fun.
19:14
Yeah, exactly. Man from about GPUs.
19:17
So that's the line. Get in line on their GP.
19:20
Yeah, you gotta wait for a bit. You know the during the daily to
19:23
will be like I waited yesterday. But your GPUs, what?
19:27
You just have GPUs running all the time?
19:29
Sounds expensive now that we don't like, we have some debuts
19:32
running all the time, but we have like auto scaler.
19:35
So then when no one is using them then you know they go down
19:38
and then they go back up. We also don't use like 800 or
19:42
something, you know, we're not doing like L trainings you have
19:46
for six weeks or something, so. Not yet.
19:49
Those data scientists, we're gonna come back and they're
19:51
gonna be like, actually, we need to train one big model instead
19:54
of these 25,000,000 models per city, you know?
19:59
Yeah, yeah, actually, I have a cold chamber.
20:01
Let's talk about that. So let's see.
20:03
Really. Yeah.
20:04
I mean, that's the obvious. Next question is like how have
20:08
the data scientists started asking you about that?
20:10
Yeah, like it happened. And the good thing is that.
20:14
Well, we don't really need to do other things special except like
20:17
maybe you'll have some specific resources or like more GPU or
20:20
something, but the whole platform can just take it can
20:23
just scale things U. The only problem is that
20:25
sometimes, you know, we scale things up.
20:27
And everybody else is like, hey, we don't have GPS available in
20:31
Europe, you gotta wait and then it's like, OK.
20:34
But always for other limbs on all sides, we have a whole like
20:38
AI Gallery internally as well that is like showcasing the
20:42
capability capabilities of the platform.
20:44
You know, like we have stable diffusion running internally, we
20:47
have like some LMS running internally and it's just for the
20:50
sake of. Luke, the e-mail platform can do
20:53
those. OK, might be a bit of work
20:55
obviously, but it's possible like you don't have.
20:59
You'll have to worry about the capabilities of the platform so,
21:03
but it feels like the current use cases that you're using
21:06
machine learning for are like the exact opposite of what you
21:10
would want to use LMS. For yes, exactly.
21:14
So I think it's like if we were to use the lamps and stuff, I
21:19
still think we could but for like support but not support to
21:24
our consumers, but more like support internal support.
21:27
You know, like when we have all these people and then you know
21:33
like to tag for example the messages that are coming or you
21:36
know to maybe suggest them things.
21:38
I think that's, I don't know if we would, but that's at least in
21:41
my opinion, what we could use it for, because yeah, that makes
21:46
sense. For direct support, I mean the
21:47
whole support is also famous for the quality.
21:51
There's this quality of that quality.
21:53
Good quality. So I think that's what people
21:54
tell me, you know? And then they come to me, they
21:58
send me WhatsApp and I'm like, hey, my food is late and I'm
22:00
like, cool, let me go and get it for you.
22:04
Well, that's the, you should have told them, wait 5 minutes
22:07
because the ETA probably has an updated Ohh.
22:09
Yeah, Yeah. It feels like there's a tonne
22:13
of, I imagine there's probably a lot of product engineers out
22:17
there at Volt that are trying to figure out like, OK, cool, we
22:20
could probably incorporate some LM's into this piece of the
22:24
business. Yeah.
22:25
And that it feels like the data scientists are probably thinking
22:29
about that stuff too. But right now all you care about
22:32
is can we support that? And you've already proven that
22:35
out. So you just got to go and talk
22:37
to whoever it is in the company to show them that you can do
22:41
that? Exactly like on all sides about
22:44
platform wise we we were pretty good that it just depends on
22:47
like yeah what do you want from a business point of view and
22:50
everything because also you know with LM's like the big problem
22:54
there is usually is that the latency.
22:56
You know, you like as a customer, like, it's cool.
22:59
I used to GPT or or cloud or whatever.
23:01
It's cool. But then I got a customer.
23:03
I don't really want to wait. Yeah, and that's the that's the
23:07
thing. So it's like, yeah, what do we,
23:10
how do we deal with that? Also like what is suddenly the
23:14
the model starts dreaming. You know, like it starts
23:17
dreaming. Is in.
23:20
I don't know. No way.
23:21
Even though you live in Germany, they're like, yeah, cool, That
23:25
sounds about right, Yeah. Says this are like those things.
23:29
But yeah, I think for like internally I think we will.
23:32
For the rest, let's see, let's like.
23:36
I have, yeah. Well, let's see.
23:37
Tomorrow I'm gonna meet data centres to talk about it and.
23:40
Let's see if they have ideas. I think we will use it.
23:43
But it's also like the thing, you know was like, I don't think
23:45
it's going to replace like every human being.
23:48
And you know, it just depends on.
23:50
How you use it, you can be a compliment.
23:53
Yeah, completely. So dude, how often are you
23:55
getting feedback from data scientists or users of the
24:00
platform? And then you know the platform
24:03
so well, now you go and you just update the platform or you look
24:08
at trying to update it and yeah, incorporate their feedback.
24:11
So we have a lot of feedback because we received a lot of
24:13
feedback on Slack as well. But they're just like really
24:16
nice like ohh, They'll be like Ohh, I'll try to do that.
24:19
Is it possible to do it on the platform?
24:22
Or maybe like the dock is out on outdated?
24:25
That's also a big problem. We write documentation would
24:28
change the whole platform. That's completely outdated and
24:30
you're like, yeah, let's go. But so yeah, we have, we have
24:34
that feedback and then also every two weeks argument with
24:37
data scientists. So then we can also like
24:40
showcase other cool things that other people have done.
24:43
No, that's part of the community park, community work.
24:45
Sorry. So like, yeah, like I don't
24:47
really work too. They deployed models like they
24:49
train models and like all other countries they use GPU's like
24:52
crazy. Look how cool it is.
24:55
You can do that as a showcase and I'd like to do it like that
24:58
because then I'm not the one selling it, it's our users
25:01
selling it. And then they send it to other
25:04
users and then they work together.
25:05
And then I'm basically done for work, you know?
25:10
You just the back light up a cigar.
25:12
Yeah, for your feet on the table he's got, he's saying my work
25:17
here is done. No, obviously not.
25:20
But that's like some kind of idea was also like we try and.
25:25
To not always be present on Slack.
25:28
And he would try like all other users to like, We'll give them
25:33
the time to like reply to other users as well, because otherwise
25:36
people become ML platform dependent.
25:38
Yeah. And I don't really want that.
25:41
Yeah. And you just become more of a
25:43
support channel. Exactly.
25:44
And then you become the bottleneck to be like, Oh yeah,
25:47
we're waiting on another platform because of blah, blah,
25:49
blah. So it's also like something it's
25:52
really hard to balance actually another year.
25:56
Archibald, do you offer and? Before you become an support
26:00
team. So, so it's the thing.
26:03
But we try. We try you know well.
26:05
So another thing for people who do not know Vote was bought by
26:10
DoorDash right. And we've had hang on the
26:14
podcast the head of the ML platform for DoorDash.
26:18
I also got to hang out with him last time I was in San Francisco
26:21
and blast there was so much fun man it the guy just cracks me
26:26
up. He was making me laugh.
26:28
Best story ever is we went to the what was it?
26:32
We went to the mother Duck, so the like Duck DB, the paid
26:37
version of Duck, maybe they had a launch party when we were
26:39
there and we went to their launch party and they gave us
26:43
shirts when we were walking in. And so they asked what's you
26:46
know, what's your size And you filled out your shirt size and
26:50
so they asked hey and you was like medium.
26:52
And then they asked me, and I was like large hadn't looked at
26:55
me. He's just like large.
26:58
What are you talking about? You're going to wear it as a
27:01
night shirt as it's just like, dude, that was kind of a diss.
27:05
I think that's funny, but I don't know how I feel about
27:09
that. I crack up every time I put that
27:14
shirt on and I wear it not to sleep in.
27:18
I just think about him and like, now I gotta go hit the gym and
27:22
just fill into that shirt, yeah. Yeah, maybe that's like can be a
27:26
goal of yours, you know, to be like, Oh my God, I really want
27:29
to fill it and that's it. So I can prove Hain wrong.
27:32
But it's not a night shirt. It is a day shirt.
27:35
Heinous. Anyway, the reason I brought him
27:37
up was because when he came on the Pod and DoorDash, Doordash's
27:42
ethos is incredible on how they're platform is doing some
27:48
amazing stuff and I think they probably, I know that you two
27:51
talk quite a bit, yeah. And one of the things that they
27:54
mentioned that they do a lot is the user surveys.
27:57
So I'm wondering if you're doing any of those and if so, how are
28:00
you collecting feedback from users and what are you doing to
28:03
get that? It's one of the things that I
28:06
have to work on actually as part of the new rule, because we
28:09
never had those before. And yeah, it's like we really
28:12
want to collect service and and make sure that people are happy
28:16
and then it's probably going to be divided.
28:18
Different sections. But then the problem with that
28:20
is that people have to reply to it.
28:22
Yeah, it's like, yeah, you always create it and you're so
28:25
proud and then you have 3 replies.
28:27
You trust me, dude, I know. But I think at the end of the
28:30
day, if you can basically create the culture of it, the more that
28:35
you do it, the more often it becomes a normal thing and
28:40
people get used to doing it. And then if you see that there
28:44
are quick wins, where, OK, I just filled out this survey and
28:47
I said XYZ and then I see that the platform team listened to me
28:51
and they incorporated that feedback and now I don't have
28:54
that problem. It's like, hell yeah, I'm going
28:57
to fill out the next survey because I'm going to tell them
29:00
more stuff that's happening to me, right?
29:02
Yeah, that's true. Yeah.
29:04
It's like, I like that idea of having like those quick wins to
29:06
like gain the trust of people. It's usually.
29:09
It's how I started at Bolton was like I need a quick win so that
29:13
when we send and call and making the deployment easier.
29:16
But then like for all users, one thing that I like to do is that
29:20
sometimes they give us feedback on the platform and I'm like,
29:23
oh, what else you create the PR like we can even do it together
29:27
if you want. And then they're really happy
29:29
because they'll contribute to the platform directly and they
29:32
don't see us as those like like platform guards or something.
29:36
And then they're like they feel like.
29:38
And then they're like, ohh I'm gonna use the platform for sure.
29:40
I created one PR like yes, let's go.
29:44
It's sad, is it? Now they're part of the team.
29:47
It's like this is your platform too.
29:50
Ohh by the way, is Super Bowl safeties tomorrow?
29:52
Yeah, like I expect you to be answering questions in Slack
29:57
from 9:00 till four. Make sure we have an ETA or
30:02
what? Is it our SLA?
30:03
Or like NSA, you know, reply to this.
30:09
That's classic. But I I also love the story and
30:12
I wanted to bring it up because for me it was just so clear how
30:17
cool. Evangelising and getting out
30:20
there and giving talks actually provides value to your company
30:28
and how cool it is to do that. So you mentioned to me, I think
30:33
last time I was in Berlin for the meet up, that the last like
30:37
three or four people that you hired on the platform team all
30:41
came inbound because they saw one of your talks.
30:44
Whether it was at the MLS community meetups or it was at a
30:47
Cube Con or it was online somewhere, It was all inbound
30:53
because of that. And they saw how cool the
30:56
platform was and they thought, well, now I want to work with
30:58
that guy and I want to work on those problems.
31:01
Yeah, exactly. As it happened, like, yeah,
31:04
exactly. I think 3 last three or four
31:06
people were hired. I mean, you also went from the
31:08
team of three to seven Nice. So it's.
31:11
Actually, everyone know how. But it's yeah, they all been
31:15
like, ohh yeah it was cool to like to see those videos or like
31:19
the tool queue you also shared. And also the blog post actually
31:22
like people like also wrote a blog post and people saw it and
31:26
they're like the tone of the blog post as well.
31:28
Because I mean, I tried to also make it like humbling and not
31:31
being like, look at how fucking cool I am.
31:35
It's also like you had tried to share those and need help to
31:39
either attract people or convince them, basically because
31:43
they were like, yeah, actually the way you describe the
31:47
platform and you know, the idea you have to make life of data
31:50
centres better and easier is usually it's something that they
31:53
all care about. So yeah, like I think one of one
31:57
member of the team, he contacted me after my first cube come talk
32:01
and then I was like ohh, sorry, we're not hiring and then a year
32:04
later we hired like we opened positions and then he applied a
32:07
year later because like yeah, I was still keen.
32:10
Feels like, yeah, not this. And that's some strong inbound.
32:14
Yeah, that's really cool. So yeah, it helps.
32:17
It's been really helpful actually to give those talks and
32:20
meet ups as well. And also one more host meet ups.
32:23
You know, like it's one thing when we host the meetups, which
32:26
happens from time to time, is. You know, people come to the
32:30
office and they're like, wow, it's like such an incredible
32:33
office. That yeah, it helps, but that is
32:35
also like just talking to the engineers, you know, because you
32:38
can have a nice office and everything else sucks.
32:41
Boring work. Yeah.
32:43
For those who don't know too, I will preface this with the
32:47
office that Volt has in Berlin is on the river.
32:50
And I guess you just bought the whole office from We work.
32:53
It used to be a Wework. Yeah?
32:55
And then you converted it all from Wework into Volts office.
32:59
Exactly. Yeah, exactly.
33:01
And it is right across the river from kind of famous nightclub
33:06
night. Well, I mean, night club is a
33:09
funny word to say in Berlin because it's not only a night at
33:13
the club, they're going on weekend clubs.
33:17
Yeah, we can club. It was.
33:20
I had the experience and I swear to God, this was when I was like
33:23
19 and I was going to Berlin for my first time.
33:26
I went to that club. I do.
33:28
You know what's it called? It's something river or
33:30
something. What is the club?
33:31
There's so many of these, Which is the boat?
33:34
No. Then they slept.
33:35
Our vision now? No.
33:39
Again, that's it. Watergate had something to do
33:41
with water, river, whatever. Yeah, like.
33:46
So I went there and it was getting late.
33:48
It was getting really late and the sun was starting to come up
33:51
over the water. And this was the first time that
33:54
I ever had the experience that as I was then walking out and
34:00
this was on a Sunday morning at probably like 8:00 or 9:00 in
34:03
the morning, I saw people that looked super fresh and like they
34:08
had just woken up and they were going to the club as like their
34:13
morning ritual or routine. Some people go to brunch.
34:16
These people, they go to the club on a Sunday morning.
34:20
And so I thought that was fascinating.
34:21
They just wanted to dance to techno at 8:00 in the morning on
34:24
a Sunday. It is actually.
34:26
It's quite fun. Sometimes it's on the Sunday to
34:28
do like a day day party. It's a very different vibe to
34:31
the might be very different crowd as well, but it can just
34:33
be fun, you know, like you go there on the Sunday morning and
34:36
you leave, I don't know like 5 PM, 6:00 PM, it's quite, quite.
34:40
This you know like the only really good yeah was like if you
34:44
party the whole night. Although the next day, given the
34:47
day after, you're like Oh my God now.
34:50
These days, definitely I'm out for a week.
34:55
Those days it was a different story.
34:57
And so that Watergate, I remember when I was at the MO
35:01
community meet up in the vote office and I saw that building
35:06
and I'm like, I'm pretty sure that is the building that I went
35:09
to. And when we passed it after that
35:13
after the meet up I went and the cruise around and I checked and
35:16
it totally was and so lots of fun memories.
35:19
Anyway, that was a complete tangent from the platform that
35:23
Bolt has. One thing that I I wanted to
35:26
dive into too is Speaking of Berlin and Speaking of the
35:29
culture that they have there. I think it's fascinating to get
35:33
your opinion on the tech hub that Berlin is becoming and what
35:38
you've seen just in the I guess you've been there for what, 3,
35:42
almost three years now? Ohh believe no.
35:44
I've been here for a bit more than one year.
35:47
I was in Finland before the year before, so in the over one year,
35:52
what made you go to Berlin? What was what do you think about
35:55
this tech scene? It feels like there's meetups
35:57
happening all the time there now, and there's all kinds of
36:01
cool tech companies. What's your take on it?
36:04
Yeah, so, so yeah, for for the context, I was working for fault
36:07
before in Finland and I left Finland because I was missing
36:11
the tech hub part actually of of of city.
36:13
Before that I was living in Paris and I was missing that
36:17
part. So then we have an office in
36:19
Berlin and it was like, OK, let me move to Berlin.
36:21
You go back to the take out. Then I joined moved to Britain
36:25
in April 2022, right after COVID.
36:28
Restrictions and Everything. And they didn't.
36:32
Everything was talking to us. That's perfect timing too.
36:36
Yeah. And then it was like, OK, like
36:38
the city used to be a big take up.
36:40
It's still a big tech hub, but it used to be like a lot of meet
36:42
ups, a lot of things happening. Like, I feel like you know,
36:44
after COVID. Lot of meetups died, people
36:48
moved on, people moved out of Berlin as well.
36:52
When I was like, you know, I was like, I was OK, I wanna, I wanna
36:54
create this meet ups again. I wanna I wanna do meetups.
36:57
In the city, because that's I learned a lot from me.
37:01
That's when I was younger, you know, when I was a student, I
37:03
was good to meet up in Paris. I learned a lot from there.
37:05
So I was like, ohh, wanna do those again?
37:08
And then we started talking to organise the community one in
37:11
Berlin we like. I don't know if we were the
37:14
first ones to to organise meetups again, but we were like
37:17
among the first really kick the thing again.
37:20
And now? They're like metres.
37:23
They're like 3 metres per week and like back and really feel
37:25
like Bellerin coming back on that side.
37:27
There's also. I feel like that's just my
37:30
opinion, but. Everything is focused on San
37:33
Francisco at the moment, I mean the US and San Francisco in
37:35
particular. And I feel like for billions it
37:38
can be a quick win, you know, to be like if you have a company or
37:41
companies that are investing probably still in San Francisco,
37:45
but still investing a bit in Berlin.
37:47
You can win so much for like. For spending, I don't know, like
37:51
a 10th of it, you know. Well, my people are sleeping on
37:55
Berlin, man, that is very true. They don't realise.
37:58
As far as tech hubs go, I fully agree.
38:01
I do worry though, and I hope this doesn't happen.
38:04
Is that too many tech people come to Berlin and then yeah,
38:09
the the Sunday morning party at 9:00 AM vibes.
38:13
Yeah, yeah, it's already happening.
38:15
I mean, now mostly part of the problem on that one.
38:17
You know, I've moved to Berlin as a tech person, you know,
38:20
start wearing Patagonia jacket. You know, like the oldest they
38:25
said has the jackets and they're like, but they got their jackets
38:28
and they're like, no, I can't wear that.
38:30
It's like it doesn't work in Berlin.
38:35
You can't do that. Yeah, yeah, definitely cannot do
38:37
that. Unless you DJ on the side.
38:41
You can't move to Berlin as a techie.
38:43
I think that idea. So dude, before you go to
38:46
Barbie, one thing that I wanted to ask about was Kubernetes was
38:52
like a a strong, it was a deal breaker for you.
38:55
You had to be using that and it feels like you're very much
38:59
bought into the Kubernetes lifestyle and work those and
39:04
ethos and all of that good stuff.
39:07
What are some hard parts about creating a platform on
39:11
Kubernetes for specifically for machine learning that people
39:14
should be aware of? Yeah, I think the first one is,
39:17
is all about like, you know, like if you don't want people to
39:20
learn about communities, like what is the pods, what is the
39:23
deployment to, what is the kind, you know, what is blah blah,
39:26
blah, blah, blah, blah. It's actually quite hard to
39:28
abstract everything away. You know, like you have to make
39:31
sure all the logs are available. You have to make sure like, OK,
39:35
I don't want data scientists to have to add some labels so they
39:39
can get GPUs, you know, like those things.
39:40
There's so many annotations like communities, annotation based
39:45
things. And I'm like, you know, I ask
39:47
our infrared team and they're like, yeah, just get annotation.
39:49
I'm like, yeah, but I don't want to say that to my data
39:52
scientist. So how do I do that?
39:55
It's like that's the first thing.
39:56
And then also like how are you gonna share resources?
40:00
Because usually they are, you know, permanent space.
40:03
With light it's they do it per project and put domain.
40:06
So they are like abstracts lot of things away.
40:09
Let's say like how I'm going to allocate resources as well to
40:11
your users, like are they gonna compete for GPU's, are they
40:14
gonna compete for memory or whatever?
40:16
And then the last one is data. You know that it was good is
40:21
like when you said maker, you can just like dump everything.
40:24
Like with with communities it's a bit more tricky because yeah,
40:27
where do you put the data as like that's one thing you
40:30
usually have to think about. We used to have, like, heavily.
40:34
So we just read from Snowflake, put everything back in
40:37
Snowflake, and we're like, yeah, we don't think about that.
40:40
We don't care. You don't do anything on Grace
40:42
because actually that's funny that you mentioned that one of
40:45
the other communities I started back in 2020 was the data on
40:50
Kubernetes community and it was all about, hey, can we do?
40:56
Stateful workloads on Kubernetes, yeah.
40:59
Is it possible to do that now? And so you just circumvented
41:03
that whole thing and you said, we're not going to worry about
41:05
it, we're not going to try. And figure it out.
41:09
It's, it's one of the things that, yeah, I remember.
41:11
It's both that is running. Get married.
41:13
Yeah. Yeah.
41:14
And yeah, remember like seeing talks about it and everything.
41:18
And I'm like, yeah, that's cool. But just if I could just read
41:22
it, you know I'm happy. That said, you figured out a way
41:26
to make sure that you wouldn't have to go through that headache
41:29
no matter what exactly. It's kind of the kind of thing
41:32
for now is like, yeah, we don't like we don't have that on our
41:36
clusters. And then for the rest, yeah,
41:39
those are the main things. You know, it's also like, yeah,
41:41
how do you we have different clusters, you know, being on the
41:43
environment and everything since we how to make it easy for data
41:47
scientists and I don't know if you if you know there's a
41:50
specific tool which is called K9 which is my Lord and saviour,
41:54
always praise it. Yeah, what is this tool?
41:57
K9 S and it's a tool that runs in your terminal, but it
42:01
basically it is very visual. You don't have to remember like
42:05
those very hard commands. I use it every day.
42:08
Every time I like, I introduce you to this and said it's the
42:12
fall in love with it and they're like, Oh my God, I don't need to
42:15
remember those commands And it's like, yeah, it's it's an amazing
42:19
tool. So it's.
42:22
Like and it makes it visual just by like the yaml file that
42:25
you're right it's more like you can see the cluster and then you
42:28
know like if you want to have pods then you just see the names
42:31
and then you like on your keyboard you can use the arrows
42:34
to browse the resources. You can check logs by just
42:38
pressing enter and then like you know it's like you have to learn
42:42
like couple of shortcuts and you know new keyboards but once you
42:46
do it's it's like a swear like they they every data scientist
42:50
wants to introduce it to them. They never go back to the
42:53
terminal. Yeah, that sounds incredible.
42:55
And how much are you interacting with the SRE?
43:00
To. Make sure like the platform has
43:04
the right uptime or it just doesn't go down.
43:07
And are you getting pinged or are the are the SRB's?
43:10
No, you have to go on call. So it depends on so forth
43:13
training for now like we don't have everything like we still
43:17
have models deployed. So you know if in training fails
43:20
then it's fine for like at least a couple of days.
43:23
But then for models that they deployed, we do have on call,
43:27
but it's not a platform specific, no data scientist
43:31
because you know, my opinion is that yeah, it's a machine
43:35
learning like machine learning model, sorry running, but it's
43:38
just a piece of software, you know and then once it's deployed
43:41
it's just a piece of software as you should just treat it as a
43:44
piece of software. So it's just our normal ankle
43:46
rotation. That is like checking it.
43:49
Checking like, yeah, well, it's assembling deployment instead of
43:52
being a deployment, but everything is the same, you
43:54
know, like they just check like what's up with the deployment?
43:58
Is everything fine? It's maybe because of Kafka?
44:01
Is it maybe because the database?
44:03
It's usually those things, you know.
44:06
But as far as the. ML platform itself.
44:10
We have a silos and assemblies for the e-mail platform like
44:14
location that you know the form is always available.
44:16
It's like all three. It's like 99.95% would have for
44:21
the for the year and then yeah, otherwise we don't have on call
44:27
for the Emma platform directly, at least not for now.
44:30
And how much are you interfacing with the SRB's?
44:33
Are they coming in there and helping you with that or is it
44:35
something that the platform team needs to take full ownership of?
44:40
So we do it. I'd say the other way around.
44:43
We have. I used to be a member of the Sr
44:46
Squad as part of the machining platform and then we basically
44:50
have an Sr squad where like, you know, we have the data OPS team,
44:54
then they send someone to their squad.
44:56
Then they're like you know defend their data form and
44:59
everything and be like OK for data OPS, those all Israelis.
45:03
And sometimes it was the same for me for the platform allows
45:05
which forces someone else is doing it and that's the way we
45:09
do it. We don't really have an, we
45:10
don't have an art have the squads and then people are
45:15
coming from different team. They assembled together and
45:17
they're like really happy fusion and and it's more like OK and
45:24
then yeah, that's that's the way we do it, dude.
45:28
This is awesome. I am going to let you get to
45:32
your Barbie showing and I appreciate you coming on here
45:36
chatting with me. It's always a pleasure and I
45:38
look forward to seeing you again in Berlin.
45:40
Yes, this is see. See you.
45:41
And thank you.