Latest / Agentic Conversations (formally mlops.community) / Lessons on Data Science Leadership // Luigi Patruno // #185

Transcript

Lessons on Data Science Leadership // Luigi Patruno // #185 · Agentic Conversations (formally mlops.community)

0:00
My name is Luigi Petrino. I'm the vice president of Data
0:03
Science at Two You and I like to take my coffee in the morning,
0:09
just single shot, black espresso.
0:11
I drink it 90 minutes after I wake up the the Lex Fridman,
0:17
another Lex Fridman, the Andrew Huberman sort of advice for how
0:22
to drink your coffee. And I I do intermittent fasting,
0:25
so it's black, there's no milk, there's no sugar.
0:28
That's that's how I do it. Welcome back to another MLBS
0:32
Community podcast. I am your host Dmitry iOS, and
0:36
today we're talking to none other than Luigi Petru.
0:39
I'm flying solo, but this is now the third time that we have
0:43
talked to him and every time I talk to him, I.
0:47
I just love it. That's all I can say.
0:50
It's incredible for me. I get energised.
0:53
You'll hear me get stoked at a few of his answers he has.
0:57
Been moving up the ladder in the corporation.
1:00
So the first time we talked to him he was writing a newsletter
1:03
called MLM Production, which some of you OG's out there may
1:07
remember. He had some really cool thoughts
1:10
on what he was learning. He was documenting his journey
1:13
as a data scientist and what he was learning and how he was
1:16
getting into ML OPS and how he was starting to look at
1:19
platforms and look at how platforms can be useful for him
1:23
as a data scientist and what he needs to do.
1:25
Since then he climbed the ladder.
1:28
And now? As you heard his VP of Data at
1:31
Two you and this conversation really centred around being a
1:35
manager and how he's had to change from that individual
1:38
contributor role into managing people and what he's learned
1:42
from that. Specifically managing data,
1:44
products and data. How that can be a little bit
1:48
different than your typical management and how the team is
1:53
composed, how he's able to balance experimentation with.
1:59
The confidence that he can get something done.
2:02
And also how to make sure that your data team is not a cost
2:07
centre, it's a profit centre. He's got the mind for business,
2:12
which as you hear he talks about, it wasn't always that
2:15
way. So we went over the evolution of
2:19
how he went from being someone that was just really into the
2:24
technology in. And then he graduated and went
2:28
into someone who had to learn how to speak other people's
2:32
languages and one of the things that.
2:35
I think I will quote and remember always is that.
2:41
He said your CEO doesn't care about machine learning, so you
2:47
have to speak to them in a language that they care about.
2:50
Figure out what that language is and you will be much more
2:54
successful. Without further ado, let's get
2:56
into it with my man Luigi. I was thinking about this before
3:06
we hit record, and I was saying you have one of those names,
3:10
that it's a miracle you're not famous because it just rolls off
3:15
the tongue. Yeah, yeah, exactly.
3:17
That's what. That's what's coming.
3:19
I imagine they'll probably be from this podcast and all five
3:22
listeners that are going to blow you up on social media.
3:27
But man, it's great to have you back here.
3:28
I think you are the first person to actually come back.
3:31
Three times. And.
3:34
Yeah, that's saying something. Yeah.
3:37
Thanks for having me. I I, I love chatting with you
3:39
and I really enjoyed our our last conversations and all the
3:43
chats that we have in between those podcasts that aren't
3:45
recorded, at least not that I know of.
3:48
So yeah, super happy to be back here and I'm really looking
3:51
forward to chatting. They aren't recorded.
3:55
I've been actually training a model on the wisdom that you've
3:58
been giving me when we are not recording you, supposedly.
4:02
So now I can have my own little Luigi that I ask for advice when
4:06
I get stuck sometimes. And dude, honestly, when we chat
4:10
I find it immensely valuable because I love the way that you
4:14
look at things and I also love your practicality when you look
4:17
at things and it helps me realise that I potentially am
4:23
overthinking things. Or over engineering things in my
4:26
head and you get straight to the point.
4:28
I feel like that is something that is innate in you.
4:33
Do you think you have that quality or is that just
4:35
something that others pick up on?
4:37
No, I definitely think that's one of, I don't know if you'd
4:42
call it a strong suit, but that's definitely an asset I
4:45
rely on. I think I do have some ability
4:50
to sort of see through things that could be related to a
4:56
problem but aren't necessarily like the crux of the problem.
5:00
And kind of maybe put those like on the side for a moment, right?
5:06
Like, I never forget about them, but I say let me just put them
5:08
over here for a minute and zoom in on this other thing.
5:11
And yeah, I definitely think that part of who I am.
5:16
I think generally speaking, I've gotten better at being polite
5:21
about putting things on the side, you know, rather than just
5:26
saying, hey, like forget about all that other stuff you're
5:28
saying, let's just focus on on what I want to talk about.
5:31
But I think it's no, I definitely think I appreciate
5:33
you calling that out, but I definitely think it's part of
5:35
who I am and I don't know if I've probably grown at that, but
5:39
I could remember it for for, you know, sort of a long time.
5:43
Well, it seems like it is something that is a strong suit
5:48
when it comes to the articular profession that you find
5:51
yourself in, right? And you came from the data
5:54
science world and now you are managing and you are
5:58
understanding the valuable pieces of how to find.
6:04
Business returns on investment when it comes to data and I just
6:11
think about when. I talked to so many different
6:16
people doing mill. There are these pains,
6:20
especially at the beginning of any project, where you're trying
6:23
to figure out what is really important here and what is
6:27
valuable. And then you go down a path and
6:29
maybe at 2 weeks later you realise that that's a dead end
6:33
and you have to go and backtrack or you have to try and go down
6:37
another branch and explore. And so when it comes to you
6:42
being able to see what things are.
6:47
Important and recognise them. I can only imagine how helpful
6:52
that is having you as someone that is leading A-Team and I
6:57
would love to get into some specifics of when you recently
7:02
have felt like. You've been exploring a new
7:05
territory and you had to make trade-offs on decisions and wise
7:10
in the road that you've gotten to and how you made those
7:12
decisions. Sure.
7:15
Yeah, makes a lot of sense. And I think you know, now I see
7:20
a connection that was well done. There are great right into it.
7:25
Yeah, yeah, yeah. You know, it's interesting
7:29
because, um, I've had certainly as my career has progressed, my
7:35
interests have changed. And you know, I started out as
7:40
an individual contributor, as I think most people do, but in in
7:45
data science specifically. And what brought me into data
7:49
science was my technical background, but also my sort of
7:53
sense of wonder and awe at what was possible that we could learn
7:59
from experience, or that we have algorithms that could learn from
8:02
experience rather than needing to be told explicitly what to
8:05
do. And that's what got me into the
8:08
field. But as I, you know, even with
8:12
Mill in production, when I was writing on that blog and there I
8:15
was really writing from the perspective of an individual
8:19
contributor. Even there, I think what
8:21
motivated me to write that was that I found that what most
8:25
people were talking about in the data science realm wasn't
8:29
actually what the data day looks like.
8:32
Yeah, right, you. There was an emphasis and focus
8:35
on the tooling and the algorithms and while that's a,
8:41
you know, an important component in data science, it wasn't what
8:45
my day-to-day looks like. And I think at the time that was
8:48
sort of a more novel opinion perhaps and it was encapsulated
8:53
by that forget which paper from Google.
8:57
But you know, I don't want all the boxes and the ML part is
8:59
just a small part and and now more people believe that.
9:03
But as I've grown more and more into leadership and management,
9:07
what I find is that that's just the sort of a general, I think
9:14
that's that's what separates very successful projects from
9:16
projects that could be more successful is that the
9:19
successful projects know exactly what they're aiming for and they
9:25
keep that goal at the forefront of the project.
9:30
And then how you go about achieving that goal.
9:35
Those are all tactics which are important, but they're not the
9:38
focus of the project. Right?
9:40
So you could have, and these are all motivated by real business
9:44
problems. Or at least they should be,
9:46
right? So you have a real business
9:48
problem that you're seeking to solve, and once you describe
9:54
what that problem is, you want to figure out what is your
9:58
actual sort of measurable objective for this goal.
10:02
How do you determine whether or not you've successfully, you
10:07
know, solved this problem or achieved this goal?
10:10
And typically that you know you'd want to measure that
10:13
through some metric, and then everything else is just in
10:17
service of improving that metric and hence solving that problem
10:22
or achieving that goal, right? And so machine learning, in my
10:26
opinion, is just one tool, right?
10:31
And the thing that's more important is what's the problem
10:34
and how do you know if you've solved it?
10:35
And then sure, you could use machine learning.
10:37
If it makes sense to use machine learning, you could use, you
10:41
know, just traditional software engineering to solve the
10:44
problem, if that. If that's a more cost effective
10:46
way of solving the problem, right?
10:48
Because you don't want to just say the problem is solved, you
10:50
want to do it as efficiently as possible.
10:51
That's what's going to generate the real return on the project.
10:56
OI tend to see, you know, my teams work as a potential way of
11:03
solving some business problems. And where you want to look for
11:08
those problems from the perspective of a data scientist
11:12
is what are the types of problem, business problems that
11:16
can't be solved with more traditional software engineering
11:21
and can potentially be solved with learning from data?
11:26
Because those are sort of the, you know, that's the pool from
11:32
which you want to be sort of fishing for potential impact
11:36
from machine learning and other sort of data science related
11:40
techniques. And there's all sorts of
11:42
problems and they span all types of business functions and
11:46
functional areas. And my team specifically, we
11:49
were sort of an internal optimization shop and we look
11:53
across the different areas of the business for problems and
11:57
when we're picking out which problems should we go tackle, I
12:01
sort of have two criteria. One is, if we solve this
12:06
problem, will it have outsized return in general, right?
12:10
Does it really benefit the company to solve the problem?
12:14
Right. That's number one, and I think
12:16
that's important regardless of what technique you're using,
12:19
right? And then #2 is what is our
12:22
chance? What are our chances of
12:24
successfully solving the problem using data science techniques,
12:29
Be that machine learning, experimentation, optimization,
12:35
et cetera, et cetera. So if I see a problem that has a
12:39
huge potential impact and for which we have a decent chance of
12:42
making that impact with our methods, then that's where I
12:45
think our team should go play. Such as?
12:49
Well, I mean, we could talk about all kinds of specific
12:51
problems, but. What's one that comes to mind
12:54
recently that you've had that you're like, oh that was a W for
12:57
the team. Yeah, I I mean I don't want to
13:00
get too specific into like things.
13:02
I'm working on it to you just for, you know, proprietary
13:04
reasons and that sort of thing. But I will say that there are,
13:07
for instance, a lot of problems in the marketing domain and we
13:13
do a lot of paid marketing work right, where your focus is
13:17
really on trying to find those prospects who have a higher
13:22
chance of interacting with your brand successfully, be that
13:26
becoming a customer, returning as a customer, that sort of
13:29
thing. And there you have a lot of
13:31
information, both demographic and behavioural, right?
13:35
And you can use that information to build all sorts of propensity
13:39
models, trigger, you know, tailor your outreach strategies.
13:45
But basically how I see how I generalise from those problems
13:48
is there's a big potential impact because it scales, right?
13:51
You're running a lot of paid marketing campaigns, for
13:53
instance, or you're interacting with a lot of prospective
13:56
customers and you have a lot of information on those people.
13:59
You have a lot of data on those people.
14:01
That's essentially impossible to beat an ML model with hardcoded
14:10
logic in terms of identifying somebody who's more likely to do
14:13
something, for example, right? So room scale.
14:16
So there's a big potential business impact.
14:18
My company, you know, we, we have a significant amount of
14:22
cost related to marketing, let's say.
14:24
So I know that there's a big financial opportunity and
14:29
there's a lot of data, right. And without and you have to use
14:32
that data and automated ways to do your targeting.
14:36
Otherwise there's no way to really solving the problem.
14:39
So those sort of marketing problems are a big focus area
14:41
for my group. It's like it's the perfect
14:43
storm. You not only are looking at hey
14:46
do we have the data, Is the data there so that we can go and
14:50
tackle this problem. But.
14:52
Is there that outsized return that we can look at And how much
14:57
are like the ways that you measure if there's an outsized
15:00
return, I imagine is how much are we spending on this?
15:04
How much do we intentionally make from this if things were
15:08
changed And one piece that I love and just knowing from
15:13
talking to you, I can imagine that you do it.
15:17
It feels to me that you're not one that likes to reinvent the
15:21
wheel and so. You when you're looking at these
15:25
problems and you're saying, can we fix this with data science or
15:29
just with machine learning or something in our toolbox as you
15:33
were saying? You're not trying to figure out
15:36
like could we throw an LLM at this and make it like super
15:40
complex and then just have this whole new way of doing it?
15:44
You're like, whoa, what's the again?
15:46
What's the most efficient way of doing this to see a return and
15:51
get the win so that we can go and show?
15:57
Our work and show the impact that we're having.
16:01
Yeah I mean it's that's that's exactly right.
16:03
And this was true back in ML production days as well, right.
16:06
I would, I would literally say that there are those days like I
16:09
don't care if it's linear regression, right.
16:11
It's if it works, it works. And I think how I've sort of
16:17
really translated that into how our team operates is in a few
16:22
ways. But, and this I think is very
16:25
different from other data science teams, at least with
16:28
individuals I've had conversations with in terms of
16:31
how I measure our team's success and I try to do this
16:35
quantitatively and I'm always sort of updating it and thinking
16:37
about it. But I think if you talk to you
16:41
know, I, I, I tend to learn a lot from books and from just
16:45
general business practises And because data science is new, but
16:48
business is not right, Data science is newish, but it's a
16:52
new capability that businesses have, businesses have been
16:55
operating for a long time and really if it wasn't for a
16:57
business existing like we wouldn't be doing the work we're
17:00
doing as data scientists every day.
17:01
I think that's something that people tend to sometimes some
17:04
people tend to forget and you know what?
17:09
What what do you care? What do you care about there or
17:11
what what what are some businesses care about?
17:13
And one of the things that you know, important business people
17:17
or or or famous business people have said is to focus on the
17:19
incentives. And if you get the incentives
17:21
right, the behaviour is going to follow.
17:25
So as I've thought about that, I've thought through how do you
17:28
incentivize a data science team to do the right thing.
17:32
And in my mind that comes down to how do you measure, how do
17:36
you measure it right. And and and for me, a lot of,
17:39
you know, sometimes we used to do this as well where we would
17:42
build a model or something and say, ohh, this model achieves X
17:45
percent accuracy. We're awesome.
17:47
All right. And the my thoughts on that now
17:50
are great, but what's next, right?
17:54
And by that I mean unless we've solved the problem.
17:58
The real business problem. I don't mean unless we've built
18:00
the model, unless we've solved the problem and there is clearly
18:03
metrics that demonstrate the problem exists.
18:07
No longer. We've capitalised on some sort
18:09
of challenge. I consider our our efforts as
18:13
purely cost to the business. We've paid salary, we've used
18:18
time, we've spent time, perhaps we've learned some things and
18:21
that's great. But at the end of the day what
18:23
counts is that you've solved the problem for the business.
18:27
And oftentimes the data science that requires is a lot of
18:31
helping other departments, at least my team, because we help
18:33
other teams internally helping other teams change their
18:36
operations, right, or and that's what typically called like
18:40
change management, right. And so I consider data science,
18:44
my data science team really a team of change agents where we
18:48
go to different business units and we help them become more
18:51
efficient or capitalise on new opportunities.
18:54
And how I measure our success is whether those teams have
18:58
actually changed their operations and are seeing the
19:01
results of our investment and that's measurable.
19:05
And if we do not have that, I consider our work either not
19:09
finished or a failure, right. We may be learning some things,
19:13
but in terms of the tangible impact, we don't have that
19:15
tangible impact. And what this has allowed the
19:17
team to do is really see themselves as wanting to bring
19:23
into reality a complete change in the operation of how some
19:27
other team operates. And that means that they have to
19:29
be excellent partners. They have to think about things
19:33
through their partners perspective rather than just
19:36
from the perspective of oh I need to build this model that
19:38
does this right. Some people do that, but it
19:41
allows our our management layer especially to always be thinking
19:44
about things from the customer's perspective and we evaluate our
19:48
team in through those lenses and part of that calculation where
19:54
you talked about the efficiency, right, you know your question
19:57
about or your your statement earlier about not bringing LLM's
19:59
and that sort of thing unless necessary.
20:03
It's not just like, do we solve the problem, but do we solve the
20:05
problem efficiently? So how much time and effort has
20:08
gone into solving the problem? And then it comes down to just
20:11
an ROI calculation, right? What did you win over?
20:13
How much did you spent to get that, to get that return And
20:18
that's how I evaluate our team success.
20:22
So I want to go down the path of this change management and how
20:25
receptive the teams are when you approach them and you're tacked
20:29
for approaching them. But I also want to just pull on
20:33
that thread of. When you were a individual
20:38
contributor, were you also coming to your boss and saying
20:43
look? Here's the ROI of these
20:45
projects. And were you always thinking
20:49
about these types of things or was this something that you just
20:52
realised is really important and it's going to be something
20:56
that's always going to be important, so you need to?
20:59
Make it a habit and the sooner the better.
21:03
I definitely was not going to my boss with, you know the the ROI
21:06
on this needs to be such and such.
21:09
I did always though have a desire to solve things from end
21:12
to end. I was never satisfied with just
21:15
working on a one piece and I don't think that's a data
21:18
science specific thing. I think I'm just a generalist,
21:20
I'm not a specialist and I like to see problems from sort of
21:24
start to end. And I think that's also what
21:27
motivated me to go into management at to you because I
21:30
thought that in order to have the real impact and to change
21:34
how things were done, I need to have a seat, a sort of higher
21:37
level tables and be talking about things in business terms.
21:42
Now I don't have a I didn't have, I don't have a business
21:44
background, right. I didn't go to Business School,
21:46
I don't have an MBA. I came from a technical
21:48
background, but as I progressed in my career, I for whatever
21:55
reason started seeing more and more that it's necessary to
21:59
understand things in financial terms.
22:02
I think actually this is really, it's one of our sort of offline
22:05
conversations where you know we're talking about investing,
22:07
we talk about investing. Dude, I will always remember you
22:12
being like, yeah, so I know what is important to this business
22:16
because I read their 10K and I was like, ohh, you're the only
22:19
person that I know that does that.
22:21
You can actually, when I read 10K's, I just go like, oh this
22:25
is perfect. I'm going to go to sleep so well
22:27
right now. Yeah, yeah.
22:30
No, I, but I I mean, I started, I got interested in investing on
22:32
the side, nothing to do with data science.
22:34
And I started, you know, reading financial statements and
22:37
learning how to read financial statements due to that interest.
22:41
And then I started connected it with my day-to-day and I
22:44
thought, wait, why don't I think about the data science projects
22:48
and sort of these terms? And also at the same time, I got
22:51
a new manager who's still my manager, who was similarly came
22:56
from a technical background and then went into marketing,
23:00
specifically went into marketing.
23:01
But we've had a great relationship and he's really
23:04
helped me to see things in more sort of business specific terms.
23:07
So I think it's a combination of my interest and then my
23:09
mentorship. Mentorship I've received from my
23:11
manager but that's what's allowed me to and I've certainly
23:16
gone further away from technical background and technical work to
23:20
more business oriented business facing but always with the lens
23:23
of how do we do this with Data Science.
23:25
O to answer your question and I'm being a little long winded
23:30
in. In the past when I would go to
23:32
my manager I always wanted to do things sort of end to end and
23:35
that's what sort of prompted me to do the modelling and then
23:38
productionized the thing and then talk to the customers.
23:40
I just was interested in it all and I think that motivated me to
23:44
continue to grow. But on that journey I've
23:47
realised that nobody, at the end of the day, you know, if you
23:51
talk to your CEO, your CEO doesn't care about machine
23:53
learning and nor should he or she, right?
23:56
He or she cares about the results that the business has
23:59
gone to get. So I put myself in like, OK, how
24:03
would I explain the work I'm doing to the CEO in terms that
24:06
the CEO cares about? I think that's one thing that
24:10
does stick with me all the time, and it's that you need to talk
24:14
to people in their language, not in your language, right?
24:17
The other person doesn't speak your language.
24:18
Why would you talk to them in your language?
24:20
You talk in their language. So be it the CEO or some other
24:23
team that you're working with, or somebody who speaks a
24:26
different language entirely, right?
24:28
If you speak to them in your language and they don't
24:30
understand it, you're not going to do much communicating.
24:34
Yeah, nobody cares about your F1 score.
24:37
Yeah, especially if they don't understand what an F1 score is.
24:41
Yeah, I mean, typically, you know, typically they're going to
24:44
say sometimes they might say that they don't understand it.
24:46
You all might also have a lot of people who pretend to understand
24:49
it, right? So you know, you're not, you're
24:52
not really doing anybody any favours.
24:54
And that's an action that you can take, right?
24:56
It's on you to do something differently and you will
25:00
increase your chances of succeeding through that work and
25:03
through those actions. And that's another thing I try
25:05
to stress a lot to my team is, what do you have control over?
25:08
Yeah, well, this feels like a perfect segue into that idea of
25:12
you going and doing this change management if if I'm
25:18
understanding you correctly, you're.
25:21
Customer. Is still an internal employee at
25:25
the company. That's right.
25:27
So you're making their lives easier and so that's that allows
25:31
you to go and talk to them and make sure that a.
25:35
You're giving them what they want and you're helping them
25:40
along their journey and helping them hit their goals And so if
25:44
you frame it in that lens, I imagine.
25:47
It's an easy sell. Now, have you encountered
25:50
experiences where it isn't as easy as of a cell and you know
25:55
the right thing? Or at least in your opinion,
25:58
what the right thing is is to change things up.
26:00
But people are stuck in their ways and they don't want to.
26:03
And how do you approach that? Because it's very messy, like
26:06
people Problem, not not at all tech.
26:09
Or data science, machine learning oriented whatsoever,
26:15
right. Yeah.
26:16
And it's a great question. And I think the keyword you said
26:21
that I want to focus on for a second is sell, right, which
26:26
essentially I think implies that you're sort of trying to
26:31
persuade somebody from doing something differently that
26:33
they're than they're doing today, right?
26:37
One of the things that we've learned, my team and I have
26:41
learned is that again, I try to think about things
26:44
probabilistically as much as possible and really in terms of
26:46
what's the probability of success.
26:48
O if I think if I take a step back and I say all of the types
26:51
of projects we can take on which projects prior, we'll have a
26:56
higher likelihood of succeeding and its projects where you don't
27:00
have to sell realistically, right.
27:02
So if there are people who are knocking down your door for
27:08
help, they're saying please help us.
27:13
We have a problem. We are doing our best to solve
27:16
it, but we can't solve it and we're desperate.
27:20
Those are the best projects potentially in terms of you
27:25
don't really have to sell your solution, like these people are
27:28
desperately in search of something to solve their
27:31
problems. Yeah, as soon as you do some
27:35
ideation on your own and your thought is to go to a team and
27:40
say, hey, I have something that I think you could use, that's
27:43
where things get much more complicated.
27:45
Yeah, right. And it's worth separating sort
27:51
of two factors. And I was having a conversation
27:54
yesterday with a colleague and he said things in this way and I
27:56
think it's worth repeating. So credit kudos goes to John
28:00
Callahan, but the you know there's there's a buy in for we
28:05
should do this and then there's a commitment for OK, we're
28:10
invested in helping you do this. And what we find is that we need
28:14
not only a buy in not only statement that this is an
28:18
important problem that we need to solve, but also commitment
28:22
from our stakeholders that they're going to invest time in
28:25
helping us solve their problem and helping us to get their team
28:29
to operate differently. Because without that commitment
28:33
you still may not succeed. And we have a number of product
28:37
managers on the on our data science team and it might be
28:40
worth talking about like how our data science team is sort of
28:42
organised because we have product managers, engineers and
28:47
data scientists. So I tend to think of us sort of
28:49
as like a full shop because we have people who understand
28:52
stakeholders and whose job is to work with stakeholders.
28:55
And then we have teams that are responsible for modelling and
28:57
then teams responsible for building end to end solutions.
29:00
And that makes our job significantly easier to do
29:05
because we have all of these personalities.
29:08
But our product managers have called out in our retros time
29:12
after time how important it is to sort of get this step right
29:16
in the beginning Or we've had projects that have failed and we
29:20
thought to like why have these projects failed?
29:22
And whenever our best guess seems to be that our customers
29:29
weren't as invested as we needed them to be, basically our
29:33
solution has tended to like push the problem further and further
29:36
upstream or earlier into the life cycle of the project.
29:39
Where we're looking for indicators of how invested is
29:42
the stakeholder group in solving this problem and committed to
29:45
spending time with us. Because if we find early on that
29:48
they're not, for whatever reason, we can try to solve that
29:52
problem before investing an individual contributors time
29:55
into doing any modelling. Well.
29:59
Dude, so it's almost like you have.
30:02
So many different variables that you're weighing out again when
30:05
you're going back to choosing the right problem to solve.
30:08
It's not only is this a does this have an outsized impact on
30:13
the business? Is this feasible right?
30:17
Can we solve this? How likely do I think we are to
30:21
solve this? But it's not just like
30:23
technologically how likely we are.
30:25
It's also organizationally how likely are we And when you're
30:30
thinking about this going back to that idea of how you.
30:34
See these experimentation phases and how you basically have to
30:38
have this north star and if you get that then all the other.
30:43
Roundabout ways that you get to that North star are just part of
30:47
the adventure and the journey. And.
30:51
I'm wondering what does that look like?
30:53
Do you just write a bunch of documents so that everybody's on
30:56
the same page? Are you making sure that your?
31:01
Like, how do you clarify that this is our North Star, this is
31:04
what it looks like And we're gonna get there by trying these
31:08
methods, but when they go off track and we're gonna come back
31:11
like always coming back to that. Is it a a document like A1
31:14
pager? What does that look like?
31:17
Yeah, yeah, that's a really, I think you're sort of asking like
31:19
how do I manage the team in general, right?
31:21
It's a pretty, pretty broad topic and I can talk about a few
31:24
specific things that I think have been really helpful for our
31:27
team. Hey, Laszlo here.
31:30
If you're serious about Milos, you hit subscribe right now.
31:40
One is that the people, we have a very committed group of people
31:46
who I think trust me to, to lead the team.
31:52
And that's very, very important. And one of the things I love
31:55
about my team, amongst many things is we have a really sort
31:58
of long average tenure. So I was actually just today
32:04
congratulated somebody in our Slack channel for being at the
32:06
company for eight years. Ohh, well and we have folks
32:11
who've been on our team who've been the company for 10 years,
32:15
right. So the we have a smallish team.
32:17
We have about just under 20 people and for some people to
32:23
have 10 years at the company is huge, right?
32:26
So we've really built these relationships over a long period
32:29
of time and I think people have seen how invested I am in their
32:35
growth and in the team success that I've been able to develop
32:40
trust with a lot of people and specifically with my management
32:43
layer and you know they're willing to do things
32:50
differently. And so that's key.
32:54
I think there are two things specifically that really leads
32:56
to our success in my opinion and they both help us.
33:01
One of them helps us stay focused and the other factor
33:05
helps us stay disciplined and the focus part is.
33:11
Both of those virtues, by the way, I am sorely lacking in, so
33:16
I'm all ears right now when you say that I love the focus and
33:19
discipline you've got my full attention with.
33:23
So I'm on the on the focus part. We do very strict quarterly goal
33:29
setting and we use Okr's objectives and key results.
33:35
And at the sort of two months into any quarter or a month
33:39
before the start of the next quarter, I kick off a planning
33:42
process with my sort of project leaders.
33:45
I call them individuals who lead projects and we have to define
33:50
objectives and key results for the following quarter.
33:54
And key results are very tangible.
33:56
You have to know whether or not you're succeeding right at the
33:59
end of the quarter. You have to know whether or not
34:01
this was done or not. Typically it's tangible, it's
34:03
measurable, that sort of thing. Now each key result is basically
34:11
led by a single person who is accountable for the success of
34:15
that key result, and we call that person a DRI or directly
34:19
responsible individual. And so the project lives or dies
34:23
with that person, right? And that level of accountability
34:26
makes it clear to that person and to the team who's leading
34:30
the effort. And if the project succeeds,
34:33
fantastic. That person has done a great
34:35
job. If the project does not succeed,
34:38
well, you know it's on that person, right?
34:40
So that brings a level of focus to the team because everybody
34:46
knows what we need to do. There's no confusion as to what
34:49
our goals are for the quarter and we've been doing that for
34:52
several years now. The process has evolved a bit,
34:55
but our setting goals in this manner we've been doing for
34:58
several years. And I think one of the more
35:00
specific instances where I saw just how useful it was for our
35:05
team was earlier this year when I went on vacation for three
35:08
weeks. Right now, I went on vacation
35:10
for three, did not log in at all, and I lead the department.
35:14
I lead the data science department to you.
35:16
And I just remember thinking, you know, I wonder what things
35:19
are gonna be like when I go, when I come back.
35:21
Like I hadn't been on a vacation that long and something like
35:24
four or five years, right? Because in the in that interim
35:27
I've had a kid. I moved COVID right, so the
35:32
longest vacation I've taken in five years was like a week for
35:35
sure. And I was going away for three
35:38
years, three weeks, excuse me to no, I was going to Italy.
35:41
Different time zone, everything. So.
35:43
And when I came back, I checked in, everything was proceeding as
35:49
planned and I was amazed. And when I asked people talk to
35:54
people, you know, everybody's, everybody likes, you know, I
35:57
think we have a good sense of humour on team.
35:59
But everybody said you could have away for another two weeks.
36:00
We were doing our thing, you know, everything is going fine.
36:03
Yeah, they're like, actually, we don't even need you.
36:05
Just come back yards and then right, right and number of
36:10
people said we knew exactly what we needed to do.
36:12
There was no confusion as to like, well, we were supposed to
36:14
be working on and we were marching towards succeeding on
36:20
these goals, right? So I think that just brings a
36:23
lot of focus. O when it comes to Okr's, the
36:26
thing that I notice that I tend to have this bad habit of when
36:30
I'm trying to set them and I really have to remind myself a
36:34
lot to not do this is OK Rs are not just A to do list.
36:39
Yeah. And that is where I think you
36:42
can fall into like, OK, we just want to see like this much
36:47
action. And at the end of the day you
36:50
realise there is a bit of an art form to setting OK R because
36:55
it's not what are we going to do, it's what's the outcome that
36:59
we're looking for and we can do anything we need to do as long
37:04
as it's legal I imagine or to get to that outcome right.
37:08
And so you really have to look at it that way.
37:11
And and. Go into it with that mentality.
37:15
And then on the data science side, what I also would like to
37:19
know from your perspective again is going back to the amount of
37:24
experimentation and variability that you have on projects.
37:29
How can you comfortably set OK R's?
37:33
Knowing that. You may scrap a whole project
37:36
because it's just not viable or it's not not that interesting
37:40
Like does this go back to that metre that you have that
37:44
internal gauge where it's like, what is the value of this
37:48
project, what's the likelihood of success?
37:50
And then is that how you like tie it back to it?
37:54
Yeah, good question. In terms of, you know, one of
37:59
your earlier comments about OKR versus just two dudes, I think
38:02
that's my main job then, right, is to make sure the leader's job
38:07
is to make sure that the team is rowing in the right direction,
38:10
right. So if we don't set goals that
38:13
are ambitious enough, that's my fault, right.
38:17
So I typically don't lead many of the key results.
38:22
I lead a very small number of them, if any then and but my job
38:27
is to ensure that the larger set of them, you know if we achieve
38:32
some probability of success against these initiatives that
38:37
the investment in a data science team is worth it to the
38:40
business, right is worth it to the organisation.
38:43
So it's my job to make sure that we have a good mix of projects,
38:48
all of all of which could have big impact on the company.
38:53
But there's going to be variability in terms of the
38:57
likelihood of success in any single project, right.
39:00
So if you put all your eggs in the classic diversification,
39:03
right, if you put all your eggs in your basket and that basket,
39:08
you lose that basket, you have no more eggs, right?
39:10
So we need a good mix of projects whereby if we succeed
39:15
in some number of them, we've done a good job.
39:18
But if we fail in any individual project that's not going to kill
39:22
the team. So you know that means working
39:26
with a a diversity of stakeholders, right.
39:29
So folks in marketing, product, finance, other operational teams
39:35
working in a mix of and that mix of projects should include areas
39:39
where we have good reason or confidence to believe that will
39:42
succeed. Maybe because the problem is a
39:44
problem that we've we're just continuing to work on from some
39:47
period of time or something that's close enough to a
39:50
previous model we've built that we believe there's a
39:53
considerable likelihood that will succeed.
39:55
And then there will be some projects that are, you know,
39:59
brand new and maybe in those projects we'll we'll want to run
40:02
sort of like a POC. Where before promising the moon,
40:07
we'll say, hey, let's we'll devote 1/4 to working on this
40:10
and we'll see what we can do. And then we'll come back at the
40:13
end of the quarter and determine whether or not the organisation
40:15
should grow its investment in this particular initiative.
40:18
So there's ways of hedging against the sort of
40:24
unknowability or the, you know, harder to predict success of
40:30
data science initiatives because they are closer to sort of
40:33
research than just pure development.
40:36
Earlier in my career I thought it was a hard problem but as
40:39
I've spent a lot of cycles and a lot of projects I realise it is
40:43
a challenging problem but there are straightforward ways of
40:45
going about it. Yeah, I like this again.
40:49
It's going back to. The idea of likelihood of
40:53
success. And if it is similar to
40:56
something that you've done in the past or if it's optimising
41:01
something that you already see is working and it is creating a
41:05
lot of value for the business, then you.
41:08
Are. That much more confident that
41:11
it's going to be successful. Absolutely.
41:14
So, yeah. And that, you know I can I can
41:17
comment on that discipline piece if you're.
41:19
Interested. I cut you off completely, dude.
41:21
That no other part. See focus.
41:24
See what I mean? I got distracted before you even
41:27
got to discipline. It's still, it's all good.
41:29
No, you had good questions. So yeah, so the goals help us
41:32
set focus, help us focus, right? And then everybody has to be
41:35
invested towards solving those goals.
41:38
The discipline part means for me, how do we?
41:44
If you do the same task multiple times, I want to get the same
41:47
excellent result then, right? So if we're going to build a
41:51
model, we have different data scientists.
41:53
You have different teams. You might get slightly different
41:56
results. You might get different results
41:58
in how the model is documented or how the pipelines are built,
42:01
or how the stakeholder communications go and how many
42:04
meetings you need to have in order to get the project
42:07
complete. And I want that down, I want
42:11
that variability driven to 0 basically.
42:14
And the way you do that is by developing processes that people
42:18
need to follow, right? So you know, earlier in my
42:23
career I thought we don't need any processes and I've realised
42:26
that's a a good way to fail as your team grows and how we've
42:32
gone about solving this problem. Actually totally borrowed the
42:37
solution from GitLab which is a company.
42:42
GitLab has has a public facing handbook of how their company
42:49
operates and if you it's online you can Google get lab handbook
42:54
or I'm sure you'll put in the show notes.
42:56
But you can find how every one of their teams runs their
43:01
departments in a static website. So I saw this and I thought this
43:09
is a great idea and I also love the format.
43:11
It's a. It's a markdown file stored on
43:14
GitHub, so their version controlled which has a lot of
43:18
benefits because you can see how processes have evolved overtime.
43:23
And so we borrowed this idea and we have a data science handbook
43:27
and this handbook We're constantly updating and contains
43:31
all of our processes. We document exactly how we do
43:35
quarterly planning, exactly how we pick projects, exactly how we
43:39
develop data science products. We have expectations for how
43:45
data analysis should be done O For instance, we have a
43:49
categorization system of how much if this analysis is for
43:56
another technical stakeholder versus a technical internal
43:59
teammate versus a non-technical stakeholder versus an executive.
44:02
There are different requirements for how the analysis should be
44:06
completed in order to be deemed sort of acceptable for that
44:11
stakeholder. We have all of our technical
44:14
guides, so you know, we use AWS, we use a number of other systems
44:18
and we describe exactly our bias in how we use these systems,
44:23
right. We expect people on the team to
44:25
use the system this way, and anybody on the team is allowed
44:30
to contribute processes to the handbook as as simple as opening
44:34
a pull request, writing the process and then having it go
44:37
through a review process with individual contributors.
44:40
And then myself, sort of as the final.
44:42
I call myself the star of the handbook.
44:44
I'm the last to say, but anybody who's allowed to contribute and
44:48
everybody's is not only allowed but expected to contribute.
44:52
I think the most surprising part about that is data scientists
44:57
are using Git. Yeah, I mean that that's, that's
45:00
that's sort of like a table sticks on my on my team.
45:05
Yeah, you don't let any of that mess around, so I think that.
45:11
There was the last time that I talked to you were just starting
45:16
that. And I remember distinctly one of
45:18
the questions that I asked was, you know, how do you prevent
45:22
this bus problem, right? Where if one person knows how to
45:25
do something and you have them get hit by a bus, then it just
45:30
doesn't it down that whole project?
45:33
Yeah, write it down. And you were.
45:34
I think it was the beginning of this handbook phase because you
45:37
were like, we're starting this new thing and I'll get back to
45:40
you on how it goes. Are you fully bought in on it?
45:42
It's been a lifesaver or what, Yeah, it's been like very
45:46
incredible actually, because it takes time for to bootstrap the
45:50
system, basically. But we now have, I don't know
45:54
how many pages on the handbook. Let's say somewhere 30 to 40 to
46:01
50 pages on the handbook and everybody on the team has
46:07
contributed at least one page nice and individuals are often
46:10
talking about. I need to add a new process to
46:13
do this and I think it's been one of the most impactful things
46:19
on the team because I often get comments from individuals and
46:24
they say you know, I I just a number of questions that they
46:30
have to answer for. How they do things has just
46:33
dramatically fallen and they can just point somebody to a
46:37
particular page in the handbook and you know you write something
46:40
once and it's red many times. So the number of times you have
46:43
to repeat yourself drastically reduced and at the end of the
46:47
day this is just, again, this is just a standard sort of business
46:51
best practise, right? This is not different.
46:55
Maybe the fact that we're using markdown files on GitHub and
46:58
hosting it as a static site is different, but I think that's a
47:02
sort of incremental improvement on the system, right?
47:05
The the real win is that there are things that you do multiple
47:11
times and you want to make sure the quality of those actions or
47:17
tasks remains very high and consistent.
47:21
And the best way to do that is to specify as explicitly and
47:26
directly as possible how that work should be done, or at least
47:29
what the expectations are for the quality of that work.
47:33
I don't know if you've ever heard of the book at Emeth.
47:35
Yeah, it's a popular sort of entrepreneurship business book,
47:39
right? And if you if you've read The E
47:41
Myth, one of the main points that the author drives home is
47:46
that you want to create this sort of turnkey business that
47:51
you can sell to somebody and the thing just operates well.
47:55
And in order to create this turnkey business, you need to
47:58
have your processes very well defined so that if you hire new
48:02
people, they know exactly what to do and they believe in the
48:05
process. And I I've read this book
48:08
recently but it very very recently, so it didn't influence
48:13
my idea to build this handbook and our teams subsequent doing
48:17
doing over that. But I see that now and I think,
48:20
OK, I'm trying to build a turnkey data science operation.
48:24
I'm trying to build, you know, a machine, a data science machine
48:29
that runs. And when we add new people, they
48:31
know how to our brand of data science and how we do things.
48:35
And that's what's going to keep. That's the discipline.
48:37
That's why I call it discipline, right?
48:39
We do things in a certain way and we do this because we expect
48:43
a certain level of outcome. So if we know, if we have a
48:47
focus on what we need to achieve which is comes from our goals
48:51
and then we have disciplined way of working towards our goals,
48:56
again we increase our chance of success, right.
49:00
And that's just generally how I see the I see in this sort of
49:04
process here. Yeah.
49:07
When I listen to you, I I can feel the conviction and it makes
49:13
it sound so easy and I'm like, dude, I'm ready to go.
49:17
That's why I always get pumped up when I talk to you.
49:19
I'm like, I'm, I'm out of here, man.
49:20
I gotta go like take action before this high wears off so.
49:26
One thing I want to talk about though before before this high
49:29
wears off is how you interact with the data platform, the
49:37
machine learning platform, How that even looks like
49:40
organizationally because is there a platform team?
49:44
Is the data science team You mentioned a bit about how you
49:47
have like kind of your own unit and there's product people in
49:51
this unit, there's engineers and there's data scientists.
49:55
Are you building? On top of a platform and then if
49:59
you are I'm, I get the feeling that the platform team is doing
50:03
to you all and especially the data scientists what you do to
50:08
the change management when you go into the business, right?
50:11
There's like some, there's some ways that you can compare notes
50:15
or whatnot, so you break down how it looks.
50:17
Yeah, so this is great. It goes back to how like the
50:20
team is organised and we everything that our Mill
50:26
platform is internal to our team.
50:29
So let me talk, let me take you through a couple of the teams.
50:33
So let me start with the data scientist.
50:36
Basically, we have a data scientist organised into a
50:38
couple of sub teams. So there's a sub team that's
50:41
focused on marketing problems. For instance, there's a sub team
50:44
that's focused on our learning marketplace.
50:47
So it's more sort of product related, so to
50:50
youacquirededx.org and X is a product and or tech platform.
50:54
So there's data scientists who are focused on that.
50:57
So a little bit of specialisation, but not much.
50:59
But the major difference between those two teams is how they
51:02
operate on in one team we have a much more sort of centralised
51:06
project based model where we have a bunch of data scientists
51:10
who work with mill engineers on our team and product managers on
51:14
our team to deliver a project for some of the group, right.
51:18
So in those models, the product managers act as the interface to
51:23
other internal teams, and they work to scope out the project to
51:27
understand the challenges. Then those product managers work
51:31
with the data scientists and engineers to design a solution,
51:35
and then the data scientists and engineers build out that
51:37
solution. The other data science sub team
51:40
is an embedded model. So those data scientists
51:43
actually go work within the product squads that are working
51:46
in the business, right? So one is decentralised and one
51:49
is completely centralised, and it makes sense for the
51:53
stakeholders with whom they're working.
51:56
The milliners have two roles. One is to work within projects
52:00
in those project teams, right? So they work within a project in
52:04
a sort of vertical fashion. Maybe they need to
52:07
operationalize a model, build some monitoring onto a model,
52:10
deploy some pipelines, that sort of thing.
52:13
That's that. They're vertical function and
52:15
the Mill engineers also play a horizontal function, meaning
52:18
they are responsible for building and maintaining the
52:22
mill platform that the data scientists and mill engineers
52:26
use within projects, right. So we actually use a third
52:32
party, third party solution. So they're responsible for, they
52:38
just actually do this. We got a new platform this year.
52:41
They went through a procurement process and and they did for the
52:44
data scientists exactly what you described right where they had
52:47
to understand how data scientists work on projects and
52:51
what they need in a platform. And then they went out and got
52:55
the solution and tested it and ran.
52:57
They did an excellent job and we have a new solution now that the
53:00
data scientists are raving about early days, but they're really
53:04
excited about it so far. And then the last group, the
53:07
product managers are really responsible for understanding
53:10
projects, understanding our goals to setting setting,
53:13
helping to set our Okr's and ensure that the projects succeed
53:17
in achieving the business objectives.
53:19
So basically these, the ones that I'm really interested in
53:23
hearing about when it comes to the platform and how they work
53:27
with it are the decentralised ones.
53:29
Do they still have the ability to make choices or are they also
53:36
coming back to this platform when they need to do things?
53:41
So everybody's using the platform, be it, you know, on a
53:46
centralised or decentralised. And because you know, the
53:49
platform gives you like notebook environments, it lets you host
53:52
models, lets you, lets you build datasets quickly.
53:55
But there are all sorts of features in the platform that
53:58
any data scientists can use. So there's some auto ML
54:02
functionality. So as part of the procurement
54:05
process, the vendor review, you know, the ML engineers collected
54:09
requirements from every data scientist on the team and then
54:12
presented potential solutions, ran POC's, that sort of thing.
54:16
So everybody was involved there, but everybody's expected to use
54:18
the platform and that's that itself is in our handbook.
54:23
Nice. OK, so I get it.
54:25
It's only organizationally that the centralization and
54:28
decentralisation happens. It's not necessarily on the like
54:32
tooling side, the standardisation of of what
54:35
you're doing. So that makes sense.
54:37
Now as far as the way that that platform links to the greater
54:42
data platform, is that something that?
54:47
Like how how does that look? Take me through that cause I I
54:50
always see it kind of as you know, I'll tell you it's very
54:53
simplistic way of seeing it, but it's like there's the data
54:56
foundation and then you have different columns that are built
54:59
off of that be it the analytics or the machine learning or
55:03
etcetera etcetera. Yeah.
55:05
So Data at two you Data Science and Data Platform are two
55:10
different teams. And Data Platform encompasses
55:15
Data engineering, Business intelligence.
55:19
Yep. Some, some sort of like some
55:22
analytics engineering. Yeah, that sort of thing.
55:25
Yeah. And then data science as my, my,
55:28
my team, which compasses machine learning, experimentation,
55:34
exploratory data analysis, that sort of thing.
55:38
Specifically, we use Snowflake. So we snowflake.
55:42
We use things like Tableau for BI, so our mill platform, we've
55:49
hooked it up to be able to plug into all of these other tools O
55:53
we can query data directly from Snowflake or from our other data
55:57
stores we can. I don't.
56:00
I'm not sure how much we've done with Tableau directly, but you
56:05
know my team doesn't really use Tableau very much as a
56:09
visualisation tool. But you know we're we're our ML
56:13
platform is connected in our virtual private cloud to our
56:17
data platform. And another responsibility for
56:20
the machine learning engineering team is they're really our
56:23
conduit data scientist connection to the rest of the
56:26
engineering org. So they're responsible for
56:29
integrating our solutions into the greater software ecosystem,
56:33
including our ML platform. Yeah.
56:35
Are they the ones that get pinged if the model goes rogue
56:37
at 3:00 in the morning? Well, they're the ones who set U
56:42
the systems so that they won't have to get pinged.
56:46
There you go, right? So they have the monitoring
56:49
solutions and the drift detection systems and that sort
56:54
of thing. So that you know if there is a
56:57
problem, they know ahead of time.
56:59
Yeah, yeah, yeah. That's not to say that it always
57:02
happens that way. We've had plenty of problems in
57:05
the past and you just got to stay flexible and sort of solve
57:08
things and come up with new processes to make sure the same
57:11
mistake doesn't happen in the future.
57:13
Yeah, the last thing that I wanted to finish up on is.
57:18
Closing the loop on something you said earlier around.
57:23
The idea of incentivizing and like getting your incentives
57:26
right will get you a long way. And I know that because I've
57:31
seen it first hand and I think particularly coming from the
57:35
sales world, sales is all about incentives and.
57:41
It's very much like it's a fast feedback loop and if you can
57:46
step back, you can see how certain incentives will drive
57:51
you to do certain things. And so I think about it like if
57:55
you're. An SDR that's trying to book a
57:59
meeting because you get paid more.
58:02
If you book meetings, then you're gonna book low quality
58:04
meetings and you're not gonna care because it's not your
58:08
problem. You're getting paid for these
58:10
low quality meetings. If you're an AE that gets
58:13
incentivized on how much deals you close, how many deals you
58:18
close, and the. Revenue count, you're going to
58:21
close deals even if they're not the best fit, it doesn't matter
58:25
because you're like, well, I'm still getting paid and now it's
58:28
CS's problem, so I don't care. So when you're looking at it
58:33
from the data aspect of things and incentivizing your team, you
58:38
said it earlier, like you've got ways that you like to
58:42
incentivize the team, which I think if I'm remembering this
58:45
correctly, is along the lines of making sure that they
58:49
understand. When things are successful and
58:53
how things can be successful, and then they know that well, I
58:59
have the power to effect change here.
59:01
I have the power to actually like do something that has these
59:06
outsized returns. Are there any other ways that
59:09
your? Incentivizing and you've found
59:13
that incentivizing the team works well.
59:16
You know for first one comment on sort of the quality, you know
59:21
you mentioned booking you know SDRS, just booking pointless
59:24
meetings or ages closing bad deals.
59:30
And again I I I just read a lot so I I consider myself a very
59:35
unoriginal person sometimes. But I'm good at borrowing good
59:37
ideas, cloning good ideas and I'm shameless about it.
59:40
The best management book I've read and reread.
59:44
It's called The High Output Management by Andy Grove, right?
59:48
And in this book he talks about KPIs and measuring your success.
59:54
And his point is you should have your indicator and it should be
59:58
paired with the quality indicator.
1:00:01
So if you're measuring something, you should measure
1:00:04
not only the thing, but also the quality of the thing.
1:00:07
And by pairing those together, you have a tighter loop to make
1:00:10
sure that you're doing the right stuff right.
1:00:12
So you know, if you look at the, you know in your sales example,
1:00:19
UH or your SDR example, like the number of meetings that you've
1:00:25
booked that actually leads to a next action could be a good, a
1:00:29
better way of incentivizing just than the number of meetings.
1:00:32
So the number of meetings that led to, I don't know, a call
1:00:35
with an A, right, because that led to some actual action.
1:00:39
It is a higher, higher quality call than just a call that you
1:00:42
had, right. So having quality indicators is
1:00:46
useful. In terms of the incentives, I
1:00:49
think the philosophy of how to manage like what you need to do
1:00:53
is actually like pretty simple, but it doesn't make the job
1:00:56
easy. I think it's just like a sort of
1:00:58
straightforward to describe a hard and practise and another
1:01:03
book that I recommend. Sort of.
1:01:06
There's a book I recommend to all managers and then there's a
1:01:10
book I recommend to all Everybody and the All managers
1:01:13
is High Output Management. It's the best book on management
1:01:16
I've read, And the book that all people should read, all
1:01:20
employees should read is called The One Minute Manager by Ken
1:01:24
Ken Blanchard. And you know, it says basically
1:01:29
it's good for managing as if it's your first management
1:01:32
experience and it's also good for being knowing how to be
1:01:34
managed. And there's sort of three things
1:01:38
that a manager does. One is make clear to the
1:01:43
employee what the what that person needs to do.
1:01:46
So specify the goals. That's number one.
1:01:51
Number two is call that person out for doing a good job when a
1:01:57
good job is done well, right? So positive reinforcement for
1:02:03
successfully doing the job and then #3 is tell that person when
1:02:10
they're doing something wrong immediately, right?
1:02:15
So in my opinion, the job of the manager is to tell the employee
1:02:21
where he or she needs to go, and then to act as the signal for
1:02:26
course correcting that person's actions, right?
1:02:29
So here's where you need to go now.
1:02:31
It's your job to go there. I can't do your job for you.
1:02:35
Yep, but I'm gonna act as the bumper lanes to tell you when
1:02:39
you're far off course or when you're absolutely on course.
1:02:44
And when you're far off course, you need to tell the person
1:02:47
because they may not know and it's your job to tell them.
1:02:52
So you need to deliver that critical feedback to get them
1:02:54
back on course. And when they are on course, you
1:02:57
want to positively reinforce that because it's going to
1:03:00
psychologically give that person pleasure, right.
1:03:04
And it's gonna that person is going to want to get more and
1:03:07
more of that positive reinforcement.
1:03:09
O If you enforce the good things and tell somebody when they do
1:03:12
something wrong, you're basically training them to be
1:03:15
more and more productive. So you know, to answer your
1:03:18
question about incentives, like I think those are the key
1:03:20
things. Set the great goals, let them
1:03:24
know. We'll let people know when
1:03:25
they're doing a great job. And it doesn't need to be like a
1:03:27
project completed. It could be minor steps along
1:03:30
the way, or maybe not minor steps but steps along the way
1:03:33
towards completing a goal. Let them know that they did a
1:03:35
great job on that pull request for review because you saw their
1:03:39
comments and you thought it was a very thorough job.
1:03:42
That person should know and then you should tell it.
1:03:44
You should say publicly because other people see that behaviour
1:03:48
and then they want that kudos, right?
1:03:51
And they know OK, what for the receptive person.
1:03:55
This person did a great job on this pull request.
1:03:57
What It made it a great job. I want to do that great job next
1:04:00
time. And when you delivering critical
1:04:02
feedback, I believe you deliver that critical feedback in a
1:04:04
one-on-one way. You don't make that public, but
1:04:07
you just pull the person inside and you say hey this was wrong
1:04:10
for such and such reason. Either you know the work wasn't
1:04:13
so the quality I expect it to be or your behaviour was off or
1:04:17
whatever it is right. It doesn't need to be a
1:04:18
monumental thing, but you need to tell that person immediately,
1:04:22
timely, and, you know repeatedly if necessary.
1:04:26
Her. Yeah.
1:04:27
It reminds me a little bit of. Of course, we both have kids,
1:04:31
and I am. Learning a lot with my daughter.
1:04:36
And one of the things that I am doing now that she's four is I'm
1:04:41
trying to get good at giving her positive reinforcement.
1:04:46
I don't want to just give the basic great job, oh, that's
1:04:50
you're so good, you're so you're so smart or whatever.
1:04:54
And some things that I've seen is that people say it's much
1:05:00
more effective if you can call out specific things.
1:05:06
And say why those things are whatever The thing is, if it's
1:05:12
good, then why is this good instead of saying ohh good job.
1:05:17
It's like, wow, you drew a house and a bird.
1:05:22
That. Shows me that you are creative.
1:05:25
Absolutely. And getting excited about that
1:05:28
instead of just hey ohh, you're so good at drawing cool and so
1:05:33
being able to make it very detailed in the positive
1:05:37
feedback is incredible, I think at least for the receiving end
1:05:42
because if it's just. Pie in the sky?
1:05:47
Ohh, good job. Yeah, great.
1:05:48
You're doing great. Then it's kind of it's it
1:05:51
doesn't mean much, right? A lazy compliment, I would say.
1:05:55
I agree. I agree.
1:05:55
Yeah, you want the compliment to mean something.
1:05:58
That's why I don't think you should compliment somebody for
1:06:00
everything they do. You shouldn't compliment
1:06:01
somebody for doing their job right.
1:06:05
They're being compensated for doing their job.
1:06:08
You should compliment somebody for going above and beyond, and
1:06:11
for demonstrating behaviours that you think are exemplary,
1:06:14
exemplary, and that you want other people to imitate.
1:06:17
Yeah, and this. Nipping it at the bud when it
1:06:20
comes to the bad parts is another huge one.
1:06:23
And I just realised, I mean when you go back to the one minute
1:06:26
manager, I learned so much from that book and not that I even
1:06:31
manage that many people right in the MLS community.
1:06:34
We don't have a tonne. But still what I realised is
1:06:37
there's there's one, there's like a million of these one
1:06:39
minute manager books. I think now he's written.
1:06:42
Yeah, they're all rubber thin. So you can read them in like,
1:06:44
yeah, it's made an hour. Yeah.
1:06:46
You feel like, wow, I'm ready again.
1:06:49
Is that high? I'm like, I'm gonna, I'm not.
1:06:51
Nothing's going to conquer me. Now I'm the manager.
1:06:54
I can do this. And one thing that I always
1:06:57
stick with. From one of those books that I
1:07:00
read is the Monkey, how he talks about your your employees will
1:07:06
come and they'll they'll have this monkey on their back and
1:07:08
they'll throw it on your back and then all of a sudden you're
1:07:11
the manager. But it feels like you never have
1:07:13
any time to do anything because you're doing your employees
1:07:16
work. And it strikes me as you not
1:07:21
being the type of person that takes on other people's work.
1:07:25
And again, I don't know what it is.
1:07:27
I think it might just be part of your personality.
1:07:29
But I think you're able to draw clear boundaries of Nope.
1:07:32
That's your shit. You take care of that because
1:07:35
that's your job. And I'm going to be over here
1:07:38
guiding you, but I'm not going to do it for you.
1:07:42
Yeah, I read that article. That's actually a separate
1:07:44
article. Yeah.
1:07:46
Yeah, Harvard. I think a Harvard Business
1:07:48
Review article from like the 80s or something in the 90s, it was.
1:07:51
It was written a while back and I actually read that at in my
1:07:54
current job. It was like sent to us by HR
1:07:58
department at some point, but I loved it and I think now I've
1:08:01
gotten much better. But I used to when I started
1:08:03
because I like, I'm a sort of perfectionist.
1:08:05
I definitely have that quality. When I started managing, I was
1:08:09
it was very hard for me to be hands off.
1:08:13
Not only when somebody came to me with a problem, but even if
1:08:17
they didn't come to me with the problem, right, And I would just
1:08:20
want to jump in. But you're doing the wrong thing
1:08:23
there, right? Because you need the person to
1:08:24
be productive. So it's on you to as a manager
1:08:28
to coach but not do then. And that's, you know, that's a
1:08:35
very, very important job. So there are times where I
1:08:41
actually explicitly tell myself I've reminded myself to do that
1:08:45
sort of thing. So recently I somebody who
1:08:50
reports to me came to me and said hey the stakeholder is
1:08:52
having this issue. What should I say?
1:08:56
And I just looked at the person and said, what do you think you
1:09:01
should say? You know what to say and then
1:09:04
the person said it. Maybe a bit, you know, less
1:09:08
organised because I asked that person on the spot and the
1:09:11
person said I think I should say that but more coherently.
1:09:15
And I just said I think that's a good idea.
1:09:19
And that was the, you know, that was the coaching, right.
1:09:22
So, but in the past I would have said, oh, you should say this,
1:09:25
this and this. And This is why.
1:09:27
But that doesn't. Let me write you an e-mail, all
1:09:28
right? And that's allowed and right.
1:09:30
That doesn't force the person to think at all, right?
1:09:33
You know, one of the things I say is you should always come to
1:09:35
me with a solution, even if you have low confidence in it.
1:09:38
Because we can work on the confidence part, we can fix it.
1:09:41
But I want to see that you've put thought towards what the
1:09:44
solution should be. Yeah.
1:09:48
Dude, I love this. So before we completely wrap up,
1:09:51
the last time that we talked, I asked you a question.
1:09:54
I thought you had a. Very poignant answer.
1:09:59
And I'm going to ask you the same question and see how things
1:10:02
have changed. I asked you about your blind
1:10:05
spots and. What you think are certain blind
1:10:09
spots that you may have right now and.
1:10:12
When you're going through. Your day-to-day.
1:10:17
In data, in management, however you want to interpret this, what
1:10:22
are some things that you're constantly trying to look out
1:10:24
for and get better at because you feel like maybe you're not
1:10:27
that strong at or you have a blind spot in that area?
1:10:31
I don't remember what I said last time, being completely
1:10:33
honest, but I would be curious if I say the exact same thing,
1:10:35
maybe you should. Maybe you can call me.
1:10:37
Out. That's it, I think.
1:10:39
I think it's the blind spot is being too conservative meaning
1:10:45
because you want to succeed you may I may not take on as many
1:10:49
bigger potential projects for which I don't have that prior
1:10:53
data to help me understand what our actual rates of success are.
1:10:57
So I may just, like, hold them to the side.
1:11:00
Um, so if you tend to, if you want to, you know, increase your
1:11:06
rates of success, you tend to go towards areas where you already
1:11:09
have a lot of data. And so projects that are similar
1:11:12
to projects you already worked on, stakeholders who are similar
1:11:14
to stakeholders already worked with techniques that you use
1:11:17
that have already shown themselves to be successful.
1:11:20
So for instance, like LM's and that sort of, you know, there
1:11:26
could be a lot of big, there are a lot of areas of high impact at
1:11:29
2 ULM and there's a lot of, there's a couple of teams
1:11:31
working on these things. And actually, we ran an internal
1:11:34
hackathon earlier in the year to build some products with LMS
1:11:37
that was very successful in a couple of different dimensions,
1:11:43
but we haven't yet pulled the trigger on like an actual POC
1:11:47
with LLM. So I think that's something that
1:11:49
I'm thinking through not just LMS but in general, but like how
1:11:52
do you not get too conservative and remain sort of visionary
1:11:58
with your bets. I think that's the biggest one.
1:12:00
So it's yeah, it's trying to get on base as opposed to trying to
1:12:04
hit a home run. Yeah.
1:12:07
It makes sense, yeah. Or or really just like taking, I
1:12:10
would say even earlier upstream taking the swing, like taking
1:12:14
the swing versus just looking at watching the ball come past the
1:12:17
plate? Yeah, yes.
1:12:20
Incredible, man. I appreciate you coming on here
1:12:24
again and doing this. It's always a pleasure talking
1:12:26
to you. We we are probably going to have
1:12:30
you on for a fourth time and forward.
1:12:32
So when that day comes, until then people want to connect with
1:12:36
you. We'll leave your information in
1:12:39
the description, along with all these great recommendations that
1:12:42
you talked about. I I think it's awesome that
1:12:47
you're coming on here. You're helping me a tonne and
1:12:50
I'm sure you're helping a lot of other people out there, so
1:12:54
thanks dude. Awesome.
1:12:55
Thanks so much for having me. Really enjoyed it.
1:13:01
Hey, I'm Vishnu. I'm a data scientist at first
1:13:04
hand and I definitely think that you should subscribe to the ML
1:13:08
OPS Coffee Sessions podcast. It's the best podcast out there
1:13:11
to stay on top of what MLP's actually is, to talk to the true
1:13:15
thought leaders in the space. And oh by the way, Demetrius is
1:13:18
absolutely hilarious. What a weird guy.
1:13:21
You should definitely subscribe to the podcast.