Latest / Agentic Conversations (formally mlops.community) / The AI Dream Team: Strategies for ML Recruitment and Growth // Jelmer Borst and Daniela Solis // #267

Transcript

The AI Dream Team: Strategies for ML Recruitment and Growth // Jelmer Borst and Daniela Solis // #267 · Agentic Conversations (formally mlops.community)

0:00
So I'm Danilo Salis. I'm the product owner of a
0:02
machine learning team here at Picnic.
0:05
And I like flat white. I like strong coffee, but always
0:09
with milk. So my name is Yilmar Borst, I'm
0:13
a Dutch guy. I I am the domain lead for
0:18
everything analytics and machine learning.
0:20
So from data engineering to real time data to to machine learning
0:24
here at Picnic, I like my espressos black double
0:30
espressos. I usually I think I'm at 8
0:33
espressos today but I can keep going.
0:37
Folks, we are back for another ML OPS Community podcast.
0:41
I am your host Dimitrios and talking to Daniela and Chandler
0:45
today. I am so glad that I
0:48
procrastinated on recording this intro until I was in Greece for
0:53
the very fact that I get to tell a joke right now about.
0:56
Some. Of the other people that I see
0:58
in Greece. And secondly, I got to listen to
1:03
the podcast that I was not a part of talking about a global
1:06
feature store at Delivery Hero. My man Stefan recorded that.
1:09
And if you have not listened to it, it is incredible.
1:13
I am so thankful that Stefan could record something like that
1:18
and you will hear at Delivery Hero they went through different
1:25
iterations of. Do they centralised or
1:29
decentralised their teams? What does the structure look
1:32
like? Do they create a centralised
1:36
platform that every team and use case uses or do they
1:41
decentralise? And ultimately you'll see that
1:46
one of the TLDRS is they decentralised after trying to be
1:50
centralised for a little bit. This podcast right now with
1:54
Picnic and Danielle and Helmer. Talks all about how.
1:59
They did the opposite approach and for them they were
2:03
decentralised and then they went and they centralised and all of
2:08
the benefits that they got from that.
2:10
So I appreciate both of these perspectives.
2:14
I also want to draw your attention to the fact that
2:18
Picnic right now is 18 data scientists that are using the
2:23
centralised platform. Delivery Hero has a lot more
2:26
than that and they have a lot more use cases.
2:29
And so it just brings me back to you really got to work with what
2:37
is best for you, your use case and recognise what the
2:41
trade-offs are. We did a great job in this
2:44
podcast about talking through how they ultimately arrived at
2:50
the decision of centralising their MO platform.
2:55
And now time for the observation that I've seen in Greece.
3:00
While I am here, there are some people that tan wonderfully.
3:10
And what is so surprising to me is that there are some people
3:16
that are very white and they tan incredibly.
3:20
And then there are the British. And for whatever reason, they
3:25
did not get that gene. I'm just going to go out and say
3:28
it because I have seen so many red people here in Greece.
3:32
I feel bad for him sometimes. It's like they look like a
3:35
lobster, literally look like a lobster.
3:38
And so sunscreen, huh? That's what we got to preach.
3:44
The good old sunscreen. I also want to share before we
3:47
jump into the full podcast. We'll send it off by listening
3:53
to one of the Greek jams that I've been enjoying.
3:55
And as always, if you like this episode, please share it with
3:59
one friend. Let's jump right into it.
4:45
Let's start with use case or use cases and where you were.
4:51
What was the snapshot 12 months ago versus where you wanted to
4:56
be? How is that evolution happened?
5:00
So let's let's let's first talk maybe about where where we're
5:04
coming from. So picnic.
5:07
So we're in online supermarket. And what do we?
5:09
Do we deliver groceries to the people to fill up their fridge
5:14
for like a sort of a low price? It's super low margins.
5:18
And one of the main things is to get really accurate results in
5:24
how many items do we need to buy, right?
5:27
And some some of the things we can very accurately do because
5:30
customers buy bread and then we bake this bread overnight and we
5:33
ship to ship these bread to you in the morning.
5:36
So that is quite, quite easy. But for quite a few other items
5:39
we need to, we need to predict is a little bit ahead.
5:42
And these are the first sort of main use cases that you start
5:45
out as a company in simple of simple sort of forecasting type
5:48
of type of problems, which is kind of the sort of a logical
5:51
use case to start with. But then as you as a company
5:55
grow, there's many more sort of use cases that start appearing
5:58
right. So around deliveries, around how
6:00
many trips that we need to drive?
6:01
How many seconds do we stop in front of your door?
6:04
How, how long does it take us to park?
6:06
But then that is all sort of still very focused on just the
6:10
operations. But of course on the towards our
6:13
customers, we also need to, we also want to improve.
6:17
And then you start to personalise things more and
6:20
more. So you want to have better
6:22
search results. You want to recommend items to
6:24
people, you want to make sure that they're not forgetting
6:26
anything to buy. You want to make sure that we
6:29
also sort of are getting more and more into the space where we
6:32
are offering sort of meals and recipes towards our customers.
6:35
So we want to help them recommend so that you don't eat
6:38
the same things every single week.
6:40
Because research shows that people generally cook like 7
6:44
recipes, just every unique recipes on a yearly basis.
6:48
People are quite boring in in what they cook.
6:51
But we can help with that, right?
6:52
Because you but you want to sort of help them explore whilst at
6:54
the same time still recommend sort of relevant stuff.
6:57
If you eat Italian very often, that's maybe not recommend you
7:00
Thai food because maybe that's too much out there.
7:03
But do we still? But at the same time, you still
7:05
want to help them explore in new areas.
7:06
And I think focusing more on all those different sort of use
7:11
cases that sort of popping up, that becomes quite tricky to
7:14
figure out. How do we make sure that we're
7:17
running this on a large scale in production?
7:19
Well, at the same time, hey, how do we help sort of easily build
7:22
and iterate to set this up whilst at the same time maintain
7:30
a good culture of learning from each other, a good culture of,
7:35
of deploying, let's say, sort of best practises from sort of an
7:39
engineering point of view as well.
7:42
And where? And then also from sort of an
7:45
organisation, a leadership point of view who, who should be that
7:48
right. And you can, I think what we see
7:52
sort of around us is where there's a big push generally of
7:57
trying to fit everything is like multi disparity teams.
7:59
And that seems good on one hand because hey, we have one team
8:04
all together focusing on one single problem at the same time.
8:08
That means if you have a junior data scientist in a single team
8:12
with no data scientist around him or.
8:14
Her. How is this person going to
8:15
learn right? And how, how do you still
8:17
connect with the rest of, of, of, of your peers and how do you
8:20
make sure that you're actually building the best models out
8:23
there and, and get challenge from, from from others?
8:26
So hence we we went on to a little bit of a journey of how
8:31
we best organise and I think in the end still go for quite a
8:34
sort of a centralised approach for this, which is maybe not so
8:39
common in, in, in many companies, but there's
8:42
definitely quite a few out there.
8:43
I think DoorDash and Netflix come come to mind.
8:47
I think especially the well, both have actually also spoken
8:51
about this sort of quite clearly on how this helps them and how
8:54
this also helps them set sort of a very high bar for for your
9:00
people as well because you don't have people who doing the
9:04
recruitment and who's doing basically, we're running these
9:07
teams with a lot of data science experience.
9:10
So if I'm understanding it correctly, you had one narrow
9:17
use case, you proved value with that use case very quickly and
9:21
then it started to balloon. And what you recognised was that
9:26
all right, this forecasting model is really cool.
9:29
We can also try and optimise routes.
9:31
We can try and recommend things to folks.
9:34
We can try and see what other ways we can plug in the ML into
9:41
our systems. But instead of opting for each
9:46
business unit having its own ML team, you said, how do we
9:52
centralised this so that we can get the most out of it, to make
9:55
sure that all of the learnings across the entire business are
10:00
able to be reincorporated into the business as quickly as
10:03
possible. No, 100%.
10:05
And I think these first use cases started already very
10:07
quickly after we launched. So we launched in 2015.
10:10
So that's already almost 10 years ago.
10:15
So initially we started actually very early of utilising ML and
10:18
of course, as many organisations you start with hey, companies
10:22
start with heuristics, use ML more as of additional sort of
10:25
insight and additional way of hey, does it actually beat maybe
10:28
the the other heuristics. But we have more and more of
10:31
these things that actually are just operationally critical
10:35
outperforming that sort of heuristics and humans are are
10:39
doing. And given that you're sort of
10:40
seeing the value for that you start, you kind of want to do
10:42
this almost everywhere. So we get way more requests to
10:45
build all kinds of ML models where we have way more models in
10:48
production than we have people building these models.
10:51
And then how do you keep, well, first of all, maintenance law,
10:54
but how do you also keep outperforming that, right?
10:56
Because each of those models we know we can do still so much
10:59
better. I think it also happened because
11:03
when Picnic started, I think we went against what most companies
11:06
were doing. We were in the area where like
11:08
in the time where everyone would have a data lake and then we
11:11
will figure it out. And I think Picnic went against
11:14
that and we started building the data warehouse and we had so
11:18
clean data that I remember when I first joined as a machine
11:21
learning engineer, it was one of the things where I decided I
11:23
really want to work to this company.
11:25
And actually when they asked me like, do you have any questions?
11:29
I was really asking, is it really true?
11:31
Is it really true that I have clean data?
11:32
Because to be honest, I was working in consultancy before
11:35
and I've seen a whole lot of not clean data.
11:38
That's that. Yeah.
11:39
How how clean can your data be, right?
11:42
And I really think that that really changed things and how we
11:46
grew because we had this say where like data scientists take
11:51
like 80% of your time recreating data and then only 20% building
11:54
the model. And in our case, that was not
11:56
true. Like the moment that we wanted
11:58
to build something, we just like throw some SQL queries and we
12:01
were up to go like with experimentation and modelling
12:05
and, and building new models. So we, we came to like, we were
12:09
a team of maybe five data scientists.
12:11
We grew up to 10. But if you see the amount of
12:13
models we were building, we were already in 2020 models and then
12:18
data scientists. So then we scaled up quite,
12:20
quite quickly and we were able to do a lot with little people.
12:23
But then of course, you see the value of it and you want to grow
12:26
even further. And that's when we started going
12:28
to this rabbit hole of how do we scale it, but at the same time
12:31
maintain the high quality and, and really finding the best
12:36
possible way of, of, of building machine learning models.
12:40
No, I imagine there's a lot of complexity that comes with scale
12:46
of models, not just on the organisational side or the
12:50
platform side, but I've heard about you, You have different
12:55
companies that have deployed models and they've gotten to a
12:58
point where they've deployed so many models, they don't know
13:00
which models are actually affecting anything.
13:04
And it's like they're zombie models out there and people are
13:07
afraid to take them off of the platform because they don't know
13:13
if it's really affecting ROI or not.
13:16
And so they don't want to be the one that takes a model off.
13:19
And then all of a sudden you see it, revenue take a hit.
13:23
And I can imagine there's this question that you all think
13:28
about quite a bit, which is how do we tear the models to know
13:33
which ones are in experimentation phase?
13:35
And we don't really have a clear sense of the ROI on these and
13:39
which ones are very high, high priority models that we want to
13:46
make sure are always performing their best.
13:51
Which is on one end on the performance side and then also
13:54
actually on how do you actually use them, right?
13:56
And I think they're it's really important as an engineer to
13:59
rehab that end to end view. So not only your boundary
14:04
doesn't stop when the model is live and running predictions,
14:08
right? The battery stopped.
14:10
Like are you delivering the business value that that you're
14:13
aimed to do? That's why you have the model in
14:14
the 1st place. And one of the things that
14:16
relates to that is like, what if the model is unable to predict?
14:21
Or what if it's done? Or what if it's giving you like
14:23
wrong predictions or what are sort of additional sort of
14:25
safeguards you can have in place, right?
14:27
What are potential fallbacks you can use, say it has issues,
14:32
let's say operationally, what other models would you then use?
14:36
Or is there sort of a heuristic you can fall back to or what
14:39
will you show your user otherwise?
14:40
And I think thinking about these questions even up from before
14:44
you even maybe deploy the first model helps a lot in in defining
14:49
automatically. Can I even?
14:52
Take it offline later or what, what is sort of the phase of of
14:55
of this model? Is it operationally critical or
14:57
is it not? And, and that's why, and that's,
15:00
I think they're having that connection with the other
15:04
product teams. I'm in connection with the
15:07
business teams is, is incredibly important because you, you need
15:11
to understand how your model is being used because otherwise you
15:15
get a need into, into this phase that you're just spinning these
15:17
up. You're producing some results,
15:18
but nobody really knows what it does, how it does it.
15:22
It's putting out some results, but doesn't actually work or not
15:24
work. And that's, that's it's, it's
15:28
super easy to fall into the trap of just to, to spin up new
15:33
experiments and spin up new, new ways and just see what it does.
15:37
But yeah, I think we tried to find sort of the right balance
15:41
between between that versus being careful in, in, in, in how
15:46
we sort of skilled it up. And in the beginning, you might
15:47
sort of very quickly say, ah, this kind of seems to work like
15:50
you skilled it out to, to everyone, but taking a bit more
15:53
of a gradual approach of, hey, does it really work on this
15:56
small segment of users? And does it work not only on
15:59
like the average, right, the average performance or the
16:02
medium, but literally going to like for which customer segments
16:04
is this really working for which customer segments is it not
16:07
working? Because maybe we should only
16:08
roll it out for, for this particular segment and skill it
16:12
up there, whereas actually for the others really sort of
16:15
develop and into it. So we have, I think sometimes a
16:18
bit more of a careful approach there to, to really make sure
16:21
it's, it's doing what it what it's supposed to be doing.
16:24
And also having this end to end mindset, right, like we're full,
16:28
we call ourselves full stack, like we're really thinking of a
16:32
solution up until the end of maintaining it and, and, and the
16:35
impact that it will bring from the moment that we're starting
16:38
to design and to do experiments. So you mentioned something,
16:42
Danielle, that I don't think I've ever heard anyone say
16:45
before, which is that clean data.
16:47
And you were almost in awe of the clean data and how you could
16:54
basically do so much and create so much with it.
16:58
I would love to know about what you did to make sure that that
17:01
data was so clean. What does the rest of the
17:03
platform look like? And that will help us also set
17:07
the scene for then later when we talked about the organisational
17:11
piece, how different teams can pull from the platform.
17:16
I, I feel extremely lucky in that, that regard and and well,
17:20
Lockwood sound as if it's sort of it was sort of unintentional.
17:23
But what I think has helped us a lot is that from from very early
17:28
days, we started with a very centralised approach and at the
17:31
time these data legs were all the hype.
17:34
But the challenge is, is that so we have around, I think 150 or
17:41
so sort of analysts slash data scientists picnic who are using
17:44
this data on the daily basis. If you're shifting basically all
17:47
the cognitive load and the effort of using this data and
17:51
cleaning this data to everyone, there's a lot of duplicate work.
17:56
So that's where we started with a more of a centralised approach
17:59
so that you have a view of what's happening towards your
18:02
customer, not only what happens, how they did shopper journey in
18:06
our app, went to what happened during the delivery of the
18:10
groceries and basically relating all those points to each other.
18:15
And I think for us, given that we are in a business, so a
18:19
supermarket business that is known for like super low margins
18:23
and also in a quite operationally heavy business as
18:26
well. The only way how you can make
18:29
this skill and how you can make this work is to basically get
18:32
efficiency out of every single small piece.
18:35
In order to do that, you need to collect data.
18:37
You cannot purely rely on that feel on what to optimise.
18:41
You need to use the data on how what are the processes, how
18:44
efficient there's certain things, how often does a certain
18:46
edge case actually happen and relate them to each other to to
18:50
really optimise across the entire supply chain and having
18:56
there a centralised data engineering team to basically.
18:59
Pull data from I think now 5. 100 or so sources on a day on on
19:05
let's say throughout the day, cleaning that and exposing that
19:08
was was very painful at the beginning, right?
19:10
Because it's a lot of work and investment that you're doing to
19:12
get to this particular state. And once you are once you have
19:17
that, trying to maintain it and update it, it's quite quite
19:20
tricky because every time your data model actually changes,
19:23
it's super painful because you need to migrate from one from
19:26
from 1:00 to the other. So it requires to have quite a
19:31
bit of a future looking view as well like how can this business
19:34
evolve in order to that these data models actually still
19:38
support that particular directions and we don't need to
19:41
revamp and restructure everything every every single
19:43
time. So there also in terms of
19:47
hiring, I think we looked very much at people not only have
19:50
like a very strong hard skills in terms of data engineering,
19:54
but also I think most of them come from more of a consultancy
19:57
type of background. So they have a lot of experience
20:00
in working with different type of companies and different type
20:03
of segments. And having that approach to
20:06
connect and talk to many people across the business to really
20:09
understand not only the current use cases, but think along what
20:12
could be the future use cases. And that's whilst initially we
20:17
sort of started with one person who just built on a Postgres
20:19
instance locally, sort of the V1 of this.
20:23
Then later going to redshift and now running on on on snowflake
20:28
has worked actually quite, quite, quite well for us and and
20:31
paid off there. It is a forever battle because
20:35
it's it it it is a slower approach, even though it skills
20:39
better, right? And it means where people might
20:41
not might actually have a new use case, they wouldn't have
20:43
gone large and get done today, right?
20:45
This is always a big conversation that you would end
20:48
up having. You might.
20:50
Sometimes slow things down to go faster in the future.
20:52
And that's, that's, that's always tricky.
20:53
But I think in the end, whilst it has worked quite well for,
20:57
for, for many of our analysts, I've, I've been in many
21:00
conversations as I was having a discussion with somebody, I
21:05
simply wrote the sequel select statement.
21:07
As we were discussing to get the data to then actually directly
21:10
answer and then solve it straight away, right.
21:12
That's I think the the mode that is that that you're in, which is
21:15
so much better than let's discuss this.
21:17
Oh, I need a week time to analyse this data and come back
21:19
to you. And then what I think maybe we
21:23
didn't even foresee maybe at the time, but we really absolutely
21:27
realised in the last couple of years that from an ML point of
21:29
view this is amazing, right? Because then.
21:32
You don't need to spend all the time to figuring out how how
21:35
this data is structured because you have the engineering team,
21:38
whoever really basically prepared that for you.
21:41
Of course you still need to. Do your feature expiration in
21:44
your extraction engineering and try to and experiment with all
21:47
kinds of different models and and see what works well and what
21:49
the work doesn't work. But that's already, I think,
21:52
part of the process of what an MO engineer does anyway.
21:55
But are you just building directly on Snowflake?
21:58
Is that all of these models are done in Snowflake or do you have
22:02
the data platform and then the ML capabilities and the ML
22:07
platform built on top of that? Goes actually, yes.
22:10
However, there's no, I know there's no spark pipelines or
22:14
air flows or whatever. There's just you write the
22:17
sequel care query to get your data and then you train your
22:20
model and to do your predictions.
22:22
You do your you call or data warehouse to get your your data
22:26
to do inference and then you run it.
22:28
Now of course, if you run this in production for
22:30
recommendations to our customers that you need to do on demand,
22:34
then it's a combination of like historic data you can get from
22:38
there versus more real time and on demand data you get from from
22:41
other, other sources. But for example, of course, we
22:44
also now have a picture story and other other components that
22:47
are that are there. But I do think that's how
22:50
machine learning has evolved. It's been quite different from
22:53
other companies because it, our infrastructure has been quite
22:56
lightweight and we have been using the same infrastructure
22:58
that other tech teams use. And then of course, we have been
23:01
adding components as we need them, but we are the, the
23:05
components and our MO platform has grown with us and with the
23:09
complexity we will have. And we have this mindset where
23:12
we, we build on top of the solutions that are out there.
23:14
We have something that is open source.
23:16
We have something that is used in the rest of the of our, of
23:19
our tech company. But we, we have a lot of support
23:22
from like just having really clean and, and and good tooling
23:27
that has allowed us to to scale up quite, quite easily without a
23:32
lot of overhead of how to even start building things.
23:37
And so you added the component of a feature store.
23:40
I imagine you have to have some kind of model registry,
23:43
especially as your models scale. What?
23:46
What other components did you add as time went on?
23:49
Yeah. For example, for the model
23:51
registry, we have something really lightweight.
23:53
We build it on top of an S3 and some, but like really
23:56
lightweight. We have the feature store, the
24:00
jobs and the services are using of course Kubernetes and all the
24:04
like tech infrastructure that we have here at Picnic.
24:08
So we use MO Flow for. For experiment tracking.
24:12
Experiment tracking as well. I think so for if we look at
24:17
sort of back end engineering here, Picnic primarily is Java
24:20
and and and Python. So we already have sort of a
24:21
platform team that really provides a lot of standard
24:24
tooling out-of-the-box around CICD, around LinkedIn
24:28
formatting, all of that. So your Python formatting, your
24:32
SQL formatting, etcetera is already sort of taken care of by
24:35
default. And then given that most of
24:37
these, all of it actually just runs on top of communities as
24:41
just a Python service or a Python.
24:43
Sort of Chrome job. That allows you to use a lot of
24:46
these off the shelf components that are already there now.
24:51
That still means there's quite a few ML specific tools that you
24:54
might still be missing. So for example, even feature
24:57
stores already, I don't know, five years ago you already sort
25:00
of think about, hey, we might actually need something like
25:02
this. At the same time, you're,
25:04
you're, you don't want to sort of, you want to figure out what
25:08
is sort of the right solution sort of out there.
25:10
And and sort of, I think for a long time, every time we sort of
25:13
evaluate, we're still in this mode where there's not an
25:16
amazing solution. Really out there.
25:18
So we sort of keep rolling a bit our own and yes, it's sort of
25:21
sort of a centralised, easy to use off the shelf component.
25:24
But the moment you, you and which is fine if you only have
25:28
one or two models that that that needed, but at some point you
25:30
get to the point where many models needed and then you start
25:33
sort of investing in it. So for these type of all major
25:37
components, we we go a little bit slower, maybe.
25:41
So we kind of let everyone just roll bit of their own.
25:46
And yes, there's going to be duplication, but to generalise
25:49
too early has also kind of a risk, right?
25:52
Because then you have generalised very early on and I
25:54
figured out that certain use cases do not even fit the bill.
25:57
So we'd rather see a bit of duplication happening in a few
26:02
places. And then you definitely see all
26:04
the different patterns and the different options and struggles
26:07
in the various places so that you can from there actually
26:09
figure out this is actually what we need and how we can really
26:13
solve, solve this. And then, but then having a few
26:16
people dedicated on the XAML platform to rebuild and maintain
26:20
a rule out is is quite key for for this.
26:24
I've heard that explained as the whole method as opposed to the
26:28
push method, right? You're getting, instead of you
26:32
pushing some certain type of technology onto the teams,
26:37
you're getting pulled in a direction.
26:39
And at a certain point, ideally what happens is you have this
26:44
critical mass that tips over and you realise, OK, enough teams or
26:48
enough folks are using this or trying to create their own or
26:53
hacking something together. It shows there's that demand and
26:56
we have that pool for it. 100%, But I think one of the key
27:00
things is, is to have that discussion, right.
27:02
So for example, we have we have what we call an MO weekly, which
27:06
is basically where all the MO engineers just come together and
27:08
just discuss pain points and issues and challenges or maybe
27:12
something that they've learned in a different state.
27:13
But one of the things that just pops up is that if more and more
27:16
people starts complaining that certain things do not scale or
27:20
do not work well or struggle with something that at some
27:23
point, you know, there's value in in trying to sort of solve
27:25
this. So that helps you a lot in in
27:27
figuring out which direction to to go.
27:31
That means adoption becomes very easy as well afterwards, because
27:34
this was the pain point of everyone.
27:36
And you rather want to sort of meet everyone where they are
27:39
right, with the pain points they are instead of trying to sort of
27:42
prematurely optimise for things and then come off with, hey,
27:45
here's a feature story and they're.
27:46
Like yeah, why do I need it? Rather so wait a little bit
27:49
longer in that case, yeah. And they feel the need, they're
27:52
asking for it. So when it comes to monitoring,
27:55
you mentioned before that you want to monitor like how it's
27:58
doing on the business side of things.
28:00
You also want to monitor the actual model.
28:02
And I imagine you have the dev OPS folks that are trying to
28:05
monitor the system. There's a lot of tasks and
28:10
functions that are involved in monitoring an ML model.
28:15
Do you have each person monitoring something?
28:20
Is there one person that owns all of this different
28:23
monitoring? How did you set up the team so
28:25
that you can monitor these things and get that information
28:27
out there? Is it a little bit more
28:29
dispersed? What does it look like and how
28:31
did you arrive at that decision? So I think like we, we, we
28:39
started calling ourselves a full stack data scientist, but now we
28:43
rename ourselves as machine learning engineers.
28:45
But I think where it comes from is we, we own the entire
28:48
solution. So that means we also deploy it
28:50
and we also monitor it once, once it's up in production.
28:54
It is tricky, as you said, because of course you need to
28:56
make sure that the model is performing good, but also that
28:59
you're actually delivering the business value that that you
29:04
want to bring. I think what has really helped
29:07
there is to be very like having a central organisation does not
29:12
mean that you don't have the same goals.
29:14
So you work really closely with your business teams and you
29:17
really like what we've seen. That really helps a lot is to
29:20
have this really short feedback loops where every time that
29:24
you're deploying a new version, there's an analyst on the
29:26
business side that knows a lot of the domain knowledge from the
29:29
area where you're working on that is also working towards the
29:33
same goal and trying to improve and gathering new insights to
29:36
come up with new features or understand what's going on
29:38
there. That's one side that I think is
29:40
really key to make sure that you you're building the right things
29:44
and, and you're keeping track of them.
29:46
And on the other hand, more on the tooling side, we build also
29:53
at, at a tool called model performance monitoring that is
29:56
actually using DVT to run tests on top of our predictions to
30:00
sort of like keep track of, of, of of the performance of our
30:03
models and make sure that we're not drifting.
30:05
So again, it's very lightweight way of building stuff that we
30:09
just built when we when we see things that are needed.
30:11
And we also saw that we needed to keep track of that and the
30:15
performance of our models also on a more automated way.
30:20
But I do think that in order to do it properly, you need both
30:23
things. You need to have this tooling
30:26
set in place and have the right test on on your data, but you
30:29
also need to make sure that you work together with the business
30:33
and to really work towards the same goal.
30:36
What do those tests look like? That's we we make tests against
30:42
actuals when we have the actuals, but also making sure
30:44
that our predictions are not too far off from previous
30:47
predictions, that they're actually there to make sure that
30:51
the model will actually run. Yeah, that's the first step.
30:55
Yeah, is it actually? Picking anything.
30:57
It sounds quite easy, but it's, it has happened in the past that
31:01
you don't realise and then you don't have the predictions up,
31:04
right? Yeah.
31:05
And I think what is really nice is that because we collaborate
31:08
so closely with business analysts, then also the
31:11
knowledge that they bring or the kind of things that they like to
31:14
they start looking at. We can also automate them and
31:18
and and write tests with them together with them to see what
31:22
kind of things we need to to to monitor.
31:24
Yeah. So if, for example, if you if
31:26
you have if you have a model more, for example, on the that
31:30
predicts how many items we we need to buy right on one end, we
31:33
we're monitoring things like like way B2 to figure.
31:36
Out how the model was performing.
31:37
But the end that doesn't really matter, right?
31:41
In the end it matters do we have products available for our
31:43
customers and how much waste do we have by by not be able to
31:46
sell it? Yeah.
31:47
And by bringing that back to also in, in the model
31:51
performance and evaluation as well, that helps by bringing
31:55
everyone on the same page of what is it that we want to
31:58
optimise. Well, we all want to optimise
31:59
sort of the balance between the availability of our products and
32:02
and ways to find sort of a good balance between that and not so
32:05
much in the the underlying model metrics.
32:08
Of course, we we monitor them and if we figure out every
32:11
experiment, then we look at past data that helps us directly
32:16
already to sort of see, hey, this other model version, will
32:19
it perform better? Yes or no?
32:21
But the end sort of the final proof in the budding is the is
32:27
actually when what it's doing in production on the final sort of
32:32
business metrics and getting everyone sort of aligned and
32:34
getting your, your, your testing your, your monitoring on those
32:38
as well helps a lot in in, in iterations.
32:43
It helps a lot in. Optimising for for business
32:48
value. So can we just geek out on this
32:51
idea of how many, let's say apples, how many apples do we
32:56
need to buy for the next week or two weeks, Right.
32:59
And you're predicting something. You have the model that's
33:01
predicting something. And at a certain point, I
33:06
imagine you're going to have apples that go bad.
33:09
First off, how do you know those apples have gone bad?
33:11
Are you getting, is there some computer vision that's there?
33:14
Or is it just the workers that are in the factory or wherever
33:17
it may be that are throwing away apples and you have it on a
33:20
scale? That's the first like
33:22
interesting piece on that I'm thinking about.
33:25
And then second off, that seems like it is a long time.
33:29
I don't know. I, I buy apples and it takes
33:31
like a week or two weeks for them to go bad.
33:33
I guess if it's organic, it's less time, but that's a long
33:40
feedback loop, right? So you can't just buy, predict
33:43
it and then buy them and then know the next day if you got the
33:46
right or wrong score. So how does that all look in
33:49
that one little use case? So no, we don't use vision,
33:55
although I think we will go into the direction of also actually
33:59
using it. So for example, we launched
34:01
about a year or two ago an automated warehouse where a lot
34:05
of it's usually robotics. And what is amazing is you can
34:07
of course put a camera on top of a conveyor belt.
34:10
Then you can actually monitor also your well the the quality
34:14
of your apples or also the quality of of what you ship
34:17
towards our customers, right? Yep.
34:19
So that's something we we can definitely do in in our thinking
34:22
about. But overall, I think.
34:25
Long story short, usually it's it's you're you're inspecting
34:28
the stock or you're trying to pick something and you figure
34:30
out how this is actually not good and gets gets thrown away
34:32
unfortunately. But we do actually actively
34:35
measure how much we're. Throwing away to really minimise
34:37
that. And fortunately if we compare
34:38
this to other supermarkets, it's roughly I think a factor of 10
34:42
lower. But that doesn't mean everything
34:44
that goes away is to still waste, right?
34:46
So we still should optimise as much as possible to to further
34:49
reduce that. But then to your point, think
34:52
one of the very early feedback loops we can look at is
34:55
availability of items. So if, if, if we're predicting
35:00
because the availability that people buy, in many cases you're
35:02
buying an item we haven't even bought yet, right.
35:06
So even if you're buying something today to get delivered
35:09
tomorrow, we haven't bought this item yet.
35:11
So, So what we do is basically make predictions continuously to
35:15
figure out how likely is it that we will have this on stock given
35:19
supplier availability, given based on our customer demand,
35:24
based on seasonality trends, based on maybe other items being
35:28
unavailable. So for example, if one pasta
35:30
brand is unavailable, people start switching to another pasta
35:33
brand. So we need to figure out that
35:34
actually a people will probably start buying more of this one
35:37
because not because of historic demand, but just because the
35:40
other one is unavailable or maybe the suppliers unable to
35:43
deliver these items to us. So trying to take that all into
35:45
account, then we do measure in the end in, in your app when
35:50
people are trying to buy this, we still show these to, to, to
35:53
to our customers. But then the moment they
35:56
actually try to buy this, we show us, sorry, we don't have
35:58
this available. And that helps us to sort of
36:00
give back this feedback to Liverpool.
36:01
Hey, these, these are the amount of customers that would have
36:03
bought this item, but we're not able to do instead, which is
36:07
also definitely required in the end for your evaluation because,
36:10
well, if you look at actual demand, but you didn't have it
36:12
available, nobody bought it, right?
36:14
So it's basically things about your historic data that you need
36:16
to need to tweak. And I do think that in that
36:20
sense, availability is a bit more important for us than other
36:23
supermarkets that have physical stores.
36:25
Because in some cases, if you don't have an item, but you're
36:28
already in the store, you're going to buy your groceries.
36:31
But if you don't have certain items and then you, then you'd
36:34
rather just go to the store because you need that, then why
36:37
would you place an order with us?
36:38
So for us it's, it really, it really brings that like way more
36:43
pressure into having a, a, a proper forecast because we
36:47
really want to minimise the waste as much as possible, but
36:50
we also cannot afford not having availability of our items.
36:54
Yeah, it's a real just in time situation and it's cool to think
37:00
about all the different ways that you want to make sure
37:02
you're tracking the signals that folks are giving you through the
37:06
interactions on the app. So let's go back to that point
37:11
where you joined Daniela and it was, I think you said 10 machine
37:15
learning engineers. And then you realised, wow, we
37:17
need to scale up because we've got 20 models in production.
37:20
But pretty soon it's going to be way more than 20.
37:23
And there you hit this speed bump, we could say of how do we
37:30
even go about hiring? Who do we need to hire?
37:34
What skill sets are we looking for?
37:36
How does that look? And when you did some research,
37:39
you couldn't quite find any literature out there, so you
37:43
almost had to wing it. And you've learned in the
37:46
process. What would have been some of
37:48
these key learnings along the way as you scaled up, What did
37:50
the scale up look like? And then what did you learn
37:52
along the way? Yeah, I think when when we
37:55
started scaling up, we we had a small team, but we were really
37:59
delivering value fast. We had, if you look at other
38:02
companies, we were still keeping up with state-of-the-art models
38:06
wherever it's needed, not just because of the hype.
38:09
We had a lot of knowledge sharing.
38:11
And I do think that that helped us streamline our development.
38:17
You know how in, in, in machine learning, there's a lot of
38:19
experimentation and that can, you can really go into a rabbit
38:21
hole. So by really sharing from other
38:25
use cases, you can sort of learn from that and, and, and iterate
38:29
faster. So we had a lot of benefits
38:31
there and we didn't want to lose those.
38:34
You also want to stay close to business because at the end is
38:37
the main goal, right? Like that your models should
38:40
deliver the value that business wants.
38:43
But at the same time, how do you keep all the advantages of a
38:47
centralised model while also scaling up?
38:50
How do you make sure that we keep on having the best
38:53
practises we can come? We keep on having amazing
38:56
people. Because I really think that I,
38:58
I, I like to to call our team like really Unicourse because we
39:02
do find the best possible people that are really good at solving
39:07
very challenging problems. And that's really pushing the
39:10
limits. So how do you keep on attracting
39:12
talent? How do you keep on building them
39:14
the best machine learning solutions while scaling up and
39:16
making it Yeah, also suitable for, for, for the new challenges
39:23
that come with with the business.
39:25
So then you see a lot of out there and you have like purely
39:29
central teams, hybrids, hub and spokes, purely decentralised.
39:35
And you, you looked at all these blog posts and what companies
39:40
are doing. But everything, it's always the
39:42
same, right? Like 3 pros, 3 cons, but not
39:46
real insights of what are the real struggles or how is it
39:49
actually organised from the inside to make sure that
39:53
whatever structure you choose, that really works.
39:57
And I think that was quite a challenge.
40:00
It was all too high level, basically.
40:02
Yeah. Exactly.
40:03
And I think we're especially in, in, in, in many large companies,
40:10
you see also the, the combination of this sort of
40:12
matrix management style, right. So maybe you're, we are in a
40:15
team, but then maybe you're the only data scientist or MLA
40:19
engineer, but then you have an engineering manager that sort of
40:22
overseas many of these different teams.
40:24
Like the other week I was talking to somebody who has like
40:27
30 reports she was an engineering manager for many
40:31
sort of different teams. But then.
40:34
How are? How are you going to?
40:35
Support this person where you're you're not in the same team.
40:38
You have no idea what this person is actually.
40:40
Working on how can. You of course you can help this
40:42
person by an appointing to some conferences or blog post or
40:46
trying to sort of spar and think along and and etcetera or if
40:50
there's like personal issues, of course you can help this person,
40:53
but what becomes super hard if. You want to really.
40:56
How people grow into their ML skill set, right?
41:00
And it's a, it's a still, I think still also a bit of an
41:03
I'll defined skill set, But right, you need, you need data
41:06
engineering skills, you need analytic skills, you need ML
41:10
skills, But even that is quite specific for the various areas
41:13
that you're working on, right? Vision experience is completely
41:16
unrelated to like recommendations experience.
41:19
And then you need a lot of software engineering and a lot
41:22
of sort of how to use the certain tools.
41:25
And you if, if, if you if you join or work here, you also
41:30
expect from from your lead that that that person will help you
41:33
to grow. Right.
41:34
And I think. That's what what anyone should
41:36
be able to expect, so. We need sort of a model that
41:39
that allows to do that whilst at the same.
41:41
Time. World model, which sort of this
41:44
one single team kind of worked, but at some point that it simply
41:48
doesn't fit in a single team anymore, right?
41:50
Yeah. So 10 people was already sort of
41:51
pushing it, and then we knew we had to.
41:55
Grow a lot in terms of people. How do you still make that work
42:01
whilst there's and as you sort of opening this good discussion,
42:05
there's a lot of opinions and I think also partly pressure from
42:09
from from other teams to to completely decentralise in the
42:12
met. Why?
42:13
Because if you conceptual feels, if we just have one team that
42:17
thinks about search, it feels much better that there's just
42:21
one team and one person that I can talk to that will do
42:23
everything search related. Whereas if you put, let's say
42:27
all the ML models that we use in search in a different team.
42:30
Ah, but now. I need to talk to two different
42:32
teams and that's that that becomes harder in terms of maybe
42:36
communication alignment. But the moment you, you, you are
42:40
able to find sort of a good collaboration and working mode
42:43
for that, I do feel that the end result it's going to be much
42:48
better by by having, let's say, a lot of people with the same
42:52
skill set in the way, same knowledge together in, in, in
42:55
sort of a single team. Now for hiring wise, this is
42:58
quite tricky, right? As you already sort of
42:59
mentioned, how do you how do you sort of do that?
43:02
Wait, I, I didn't get it. So did you decide to go with
43:08
that embed model or did you decide because you went from 10
43:12
to how many people are you now? So we are, I think, soon to be
43:18
18. 18 And what was the decision was it?
43:23
How did you break up the team By use cases?
43:25
By. So, so we, so we started with
43:28
one, let's say or two cases sort of split out where they were
43:34
actually embedded in, in, in a team and then sort of reviewing
43:39
is it, is it going well? So we had basically sort of a
43:43
hybrid setup where we had some parts centralised and some parts
43:46
embedded where we said, hey, these are this is like a
43:49
critical major model. We have 2-3 people working on
43:53
this, collaborating on this. A single use case is being
43:57
embedded of this sort of larger product around this use case and
44:00
that I think whilst there was definitely benefits it, it
44:06
didn't end up working for us. So after quite some discussion,
44:12
we, we ended up actually going back to a more centralised way
44:15
of organising, but still into multiple, multiple teams.
44:19
So there's you where you define sort of a domain or an area that
44:23
that team is responsible for, but it still is a team
44:27
consisting of Emma engineers all together responsible, let's say
44:32
for multiple models across an entire domain.
44:34
And as we'll grow, we'll probably start cutting it up
44:37
into smaller pieces and we might actually end up to a case where
44:41
we get again closer towards the the the product teams.
44:44
But I think for now keeping this relatively more centralised seem
44:50
to better option. Yeah.
44:52
Well, it does make sense going back to what you said earlier
44:55
with the if you have a junior data scientist embedded into a
44:59
team where there's no other data scientists on that team.
45:02
And then their manager is where they have two types of managers
45:07
or the project manager. And then you have the data
45:09
science manager who doesn't really know about that specific
45:13
team and that use case, but they're trying to help them
45:16
grow. There's a lot of places where
45:19
that can fall apart. So I fully buy in to the idea of
45:24
let's organise it with all the folks that want to geek out
45:27
about the ML use cases. Let's get them here together,
45:32
put them even at the same desk so they can just ask questions
45:35
to each other and things can move faster.
45:39
And you have the ability to, when you're stuck, hopefully get
45:44
unstuck really fast because somebody is right there.
45:47
No, 100 percent, 100%. And I think that's where the
45:50
moment we sort of sort of reviewing again, sort of decent
45:52
meta set up, after a while, we started looking at what what how
45:56
do we progress sort of over time.
45:58
And actually one of the the the scary part was, holy crap, we
46:03
actually did not actually, the model started performing worse
46:06
and worse and worse, and we haven't been able to actually
46:09
keep up and actually improve as we should have compared to
46:12
actually with fewer people that we did in the past.
46:15
So whilst it conceptually sounds like a great idea, the actual
46:19
business impact was actually really lacking.
46:22
And then having, let's say this closer with each other, where
46:26
people are really also keeping each other accountable for
46:29
things, right? Because you see people around
46:31
you where you see them doing are maybe more knowledgeable about
46:36
certain topic etcetera. Also makes use of strive for for
46:40
for learning about that and and and and become better and grow.
46:45
And also the way you design your solutions, right?
46:48
If you're really focused on a business area, you're only
46:50
focusing on, we want to improve this specific thing and we're
46:54
going to tailor like a road map and the way we approach this,
46:58
solving this problem towards the, the actual pain point that
47:01
we have. Whereas if you have more of a
47:03
holistic view or like how the field of ML is going and how
47:07
some certain problems behave similarly, even though
47:11
intuitively won't look like at the end, we're still struggling
47:15
with the same type of things. You you can approach it in a
47:19
more future robust way or pushing forwards innovation and
47:23
things like that. That would another thing, you
47:25
would only just be stuck into your specific problem.
47:29
Yeah, that knowledge sharing comes much easier as opposed to
47:34
if you're more isolated. I could see that too.
47:37
So. So at the end of the day, you
47:40
said, all right, let's get the team together and we've got
47:45
different areas that the team is working on.
47:48
So the 18 folks aren't all thinking about all the different
47:53
use cases. You've got maybe the recommender
47:56
systems or, or how did you break it up?
47:58
How did you buy use cases or buy something else?
48:01
So for now, we didn't want to go to break up to too many small
48:04
teams. So for now we have sort of
48:08
organised one very much out of on our consumer side where
48:12
everything happens and let's say withdraw the personalization
48:15
recommendations etcetera. And then another team very much
48:18
focused on the world operational supply chain use cases.
48:22
Even that is still actually very broad domain.
48:25
And I do expect that at some point we'll split it up more
48:28
fine grained. But we didn't want to sort of
48:31
prematurely cut it up into too many, too many sort of smaller
48:35
pieces. Because then at some point you
48:37
might also have, let's say, I think it's, it's good to keep in
48:41
mind sort of a good balance between sort of execution versus
48:45
sort of leadership. And if, if you get to too many
48:48
fine grained teams, then then it gives a lot of overhead, right?
48:51
And, and more alignment and more discussion and more whatever.
48:54
Not. Whereas if you're just a single
48:56
team, it just becomes a lot easier.
48:57
To to to to execute. And the platform on the other
49:01
side. I was going to say, do you also
49:03
have a platform team? So there's there's the platform
49:05
team, and then there's the use cases team.
49:07
To use like two big domains and the platform team.
49:11
Makes sense. OK, When you were searching for
49:16
how you can organise the teams, what are some things that you
49:21
wish other blogs had talked about?
49:27
I think first of all it starts with.
49:29
Validation around around this. So it seemed that everyone was
49:35
saying, well, if you're a small company, it makes sense to have
49:39
like a central sort of data science practise.
49:41
But as you grow, surely that doesn't work anymore and you
49:44
need to decentralise. You need to put everyone across
49:47
the entire company. That's I think where you see
49:50
this in in in many. But I think which sort of two
49:54
notable exceptions was I think there is actually a lovely
49:57
podcast with the CEO of Netflix. I think of the Lenox podcast, I
50:00
think it was, and there's one. With the data science lead of
50:05
DoorDash. We're actually both actually
50:07
going quite against that that that mono.
50:09
I think that helps at least us to validate this quite a bit
50:14
even though we were already sort of in the whole process of
50:16
having that earlier would have helped tremendously.
50:20
Then again, the second part is what is extremely difficult to
50:26
articulate is that if you are performing actually with a very
50:31
small team to do actually lots of things.
50:35
If if the rest of the company doesn't know any better, it
50:38
becomes super hard to articulate what you will lose.
50:40
The moment you'll you'll change that.
50:42
And some of it you kind of maybe need to accept by just doing and
50:47
just see will that work or will that not work?
50:51
But I think a couple of things that would have been helpful to,
50:55
to, to see from sort of other models is I think the, the, the,
50:59
the, the, the, the downsides of, of, of embedding.
51:03
I think there's many areas where people are articulating why
51:05
embedding might work better, but not necessarily articulating the
51:09
challenges of, of, of an embedded model.
51:13
Embedding always makes it a bit reminds me of of, of Jenny I as
51:18
well. So it becomes a bit of a similar
51:20
of the term features. I think it's always a bit of a a
51:22
ambiguous word, but that aside. I think vector database as soon
51:27
as you say embedding, yeah. Exactly.
51:32
But I think articulating the downsides and the and the
51:34
trade-offs whilst it it's easier to skill that the question is
51:42
are you accepting the downside of performance and of the
51:44
knowledge sharing and of the well, I think yeah, primarily
51:48
the the the final business performance.
51:50
Articulating that is quite hard, but getting more clear cases and
51:54
proof of how that leads to lower performance in companies would
52:00
have been extremely helpful. But most companies don't want to
52:04
share that right? Because most companies in the
52:05
blogs and. Things are sharing their
52:07
successes and other failures. Yeah.
52:09
Yeah, it's a rare day when you see a company writing on their
52:13
engineering blog how bad we brought or how we brought down
52:17
the whole the whole service for 24 hours.
52:22
Yeah, that's true. Like we need to see more of that
52:25
type of engineering blog because.
52:27
I would love to is where you learn the most and I think we're
52:31
only sharing our successes. So it, it makes a lot of sense.
52:36
One thing that I wanted to ask too, with the organisational
52:41
well, is there anything else that you wish you would have
52:45
known while you're reading those books?
52:47
Yeah, I think you always look at the pros and the cons, right.
52:51
But what I really like more now looking at in hand side, what I
52:54
would like to know is like, OK, you want the best of both
52:57
worlds, what kind of structured allows you to really get them?
53:03
Because I do think that for example, we're now going towards
53:06
a centralised one. But if you want, if you were
53:09
like embedding the business teams, you would have more
53:12
alignment with with your team or, or like try to strive for
53:16
the same goals or stuff like that, that kind of thing you can
53:19
still achieve in a centralised model, right?
53:22
But if you decentralise, how do you know, do the knowledge
53:26
sharing or how do you really push for innovation?
53:28
It's a bit harder. And I do think that looking at a
53:31
more high level, you lose this, you win this, it's a bit hard to
53:35
suit to, to really know what the best structure is for your
53:39
company. And I really think that's sort
53:41
of finding solutions to get the best out of both worlds.
53:47
Looking into how your organisation is, how your
53:51
companies organise can really help because I don't think that
53:55
there's a unique model that helps for for every company.
53:58
I do think that you need to find how to apply the a certain model
54:03
to you and how to really make sure that you you lose the less
54:07
by creating disconnections or or or making solutions around it
54:10
that can really help. And our key I guess to your
54:12
organisation that that sort of innovation or or pushing it,
54:15
right, in some cases just decent performance might actually just
54:22
be good enough, right? Because maybe you have a
54:25
business model where you have huge margins and you are, you're
54:28
making, making bank anyway, then doing a little bit of, of, of a
54:33
good forecast or recommendation or whatever is really sort of
54:36
good enough. Whereas in the other case, like
54:38
in our case, we, we kind of need to make this whole thing work.
54:42
And, and in terms of margins, maybe supermarkets, really one
54:46
of the worst things you can get into.
54:48
So we kind of have to really push the boundaries on all these
54:52
things because otherwise it just, it would just not work,
54:54
right. So I think then you you kind of
54:57
want to maybe put a higher emphasis on the sort of
55:03
innovation or going for more state-of-the-art models
55:05
etcetera, rather than maybe applying a multitude of more
55:11
simpler regressions, which maybe for quite a few other companies
55:15
would already be be good enough. You're playing the game on hard
55:21
mode right now. Yes.
55:24
And I think then and, and I think to, to your other point
55:27
on, on sort of learnings on sort of the hard mode, I think
55:30
especially sort of a year ago, trying to go into this embedded
55:34
model, it really didn't work where people were quite unhappy.
55:37
And then we had more multiple people leaving actually the
55:40
company as well, which actually only stretches even more.
55:45
And I think that's really sort of increased playing on hard
55:48
mode even more so whilst at the same time you're trying to
55:51
recruit people, but you're trying to recruit these unicorns
55:54
that are sort of super friendly, energetic, fun to work with, but
55:58
also actually extremely knowledgeable, have to write
56:00
software engineering skills and have the right ability to think
56:04
along on the business side as well.
56:05
And hey, if shit hits the fan, you just you, you just fix it,
56:08
right? You have that full end to end
56:09
ownership and trying to find these people.
56:12
It's, it's, it's, it's not easy, but I, I do really value that in
56:16
the end, waiting longer in, in, in hiring people, but hiring the
56:21
right people pays off so much. It's it's very painful in the
56:26
short term, but in the long term it it pays off very well.
56:31
Yeah, what a strong signal that the embedding model wasn't
56:34
working. The folks are leaving, they're
56:38
telling you they don't enjoy it. Your model performance is going
56:41
down. It feels like you were doing
56:42
better with less people. There's so many things that were
56:45
pointing to that wasn't the way to organise it, at least at this
56:51
point in time, going from the 10 to 18.
56:53
Maybe later you will re tackle this and you'll recognise when
56:57
you're at 200 data scientists, it's good to have one or two
57:00
embedded in each one. But from 10 to 18, it's not the
57:04
moment. And I think that's a awesome
57:07
learning that you can share with the rest of us, because I can
57:12
imagine there's a lot of folks that have been in your shoes and
57:15
they've thought like, OK, is it me or is it this company the the
57:20
way that we architected this thing isn't really working.
57:24
And yeah, I wonder how many people listening have left their
57:28
jobs because of that very same reason.
57:32
I think I would not be surprised, right?
57:33
So also people around me that I should have talked to.
57:35
Then there's there's, there's so many also in other companies
57:38
where they're like, yeah, we've organised it like this, but
57:40
it's, it's just, it just doesn't work.
57:42
I really don't like it. But well, some people decided
57:45
to, to organise it like this. And I think that's, that's such
57:49
a bad sign. I think you, you need to like,
57:52
if, if everyone in your team is telling you this is not a good
57:54
idea, then then there's probably some truth through it, right?
58:00
But of course, sometimes you need to change things.
58:02
And as you scale and as you, right, we, we started 10 years
58:05
ago with, with only a handful of people, the entirety of picking.
58:08
So we're now with like 1300 in total.
58:12
It's skilled, right? At some point you need to
58:13
basically reorganise in order to, to, to make it work with,
58:16
with more people and as your business skills along.
58:19
But it does need to go in a way where we we were able to work
58:23
effectively. Awesome.
58:26
I think that's it. That's all I got.