Latest / Agentic Conversations (formally mlops.community) / MLOps: A leader's perspective // Stephen Galsworthy // MLOps Coffee Sessions #39

Transcript

MLOps: A leader's perspective // Stephen Galsworthy // MLOps Coffee Sessions #39 · Agentic Conversations (formally mlops.community)

0:16
We are back with another edition of the ml Ops Community Coffee
0:20
sessions. This is a special one today.
0:24
Talking about it's another edition of what our friend jet
0:29
likes to. All the Kung Fu is in the
0:31
culture. We are joined by none other than
0:37
Steven goldsworthy. And this conversation came up
0:43
the idea for this conversation actually came up because of a
0:47
LinkedIn chat we had and it's kind of becoming a theme a
0:52
general theme to these coffee sessions, where we have great
0:55
conversations on LinkedIn and then I just say, you know what?
0:59
We got. To bring this to a coffee
1:01
session because these ideas are so deep and they're so important
1:04
and we need to properly unravel them and we need to look at them
1:10
under a microscope and see if we can gain more insight from all
1:15
of your experience that you've had Steven and also help us
1:21
navigate this dark and lonely road.
1:25
We could say. So it's a pleasure to have you
1:26
here. Of course I'm also joined by my
1:28
man Vishnu. Welcome to you both.
1:31
Hello. Hello.
1:33
Hey, thanks a lot. It's a pleasure to be here,
1:34
guys. So I think we can start with
1:38
just a bit of background about yourself and how you got into
1:42
Tech. Yeah, so I'm, I'm now based in
1:48
the Netherlands, just outside Amsterdam.
1:50
I'm originally from the UK, and I kind of got into Tech through
1:55
my degree course. So, I studied Applied
1:58
Mathematics at University. When Went on, and did a PhD in
2:02
applied maps and what always appealed to me, was this, the
2:07
fact that you could use the same techniques to look at multiple
2:10
problems. So, I jumped around in the PhD
2:13
from looking at biological problems than moved a little bit
2:18
into epidemiology for a while aerodynamics before entering
2:23
more of the tech field. I originally got going in the
2:27
energy sector about Wow, 10 years ago now time flies in a
2:34
scale-up environment in London and of stayed in sort of the
2:38
energy Tech sector since then in a couple of different roles and
2:43
for the last five years or so been based here in after them.
2:48
Yeah. And if you've seen quite a lot
2:51
of transformation in that time, you know, as I've taken those
2:56
steps through from, I wasn't data scientist before it school
3:00
data scientist. And we were sort of looking over
3:04
to the west coast of the US to figure out.
3:06
Okay, what do we call this role? What is this and Vern of slowly
3:10
sort of seeing things come in a little bit more.
3:13
And now I mean data science is such a broad spectrum of roles.
3:17
I mean it's a little bit like I don't know, 10 15, 20 years ago
3:22
when we called everyone a programmer, you know.
3:24
And I just did did coding stuff and now you're kind of you get
3:29
all of the different This of software development and all of
3:32
the infrastructure around that you're seeing that sort of
3:35
specialization coming in from from data science with machine,
3:39
learning engineer's ml Ops, engineering AI engineer into the
3:45
analytics translator. So it's really you seem quite a
3:47
huge ham change that. And I've grown a centrally from
3:51
being a, an individual contributor into a team lead.
3:56
Ultimately, the position I've had for the last few years.
4:00
Has been on the board of directors of a company.
4:03
So it's a yeah, seeing things from all different angles.
4:07
Nice, roller coaster ride. I'm sure there's something that
4:11
you said to me on LinkedIn that I found fascinating, and it was,
4:15
and it wasn't in any way tooting your own horn, but it was like,
4:21
I feel like I'm someone who gets it because of the background
4:24
that you have, I imagine you understand the Just implications
4:31
that trying to do ml at scale and productionize ml has.
4:38
And so when you take up a new project you have all of that
4:42
experience with you. And so you are not so naive
4:46
going into things. Can you explain that a little
4:48
bit? Yeah, definitely.
4:51
I mean, I think my experience is come from particularly on that
4:54
sort of combination of the, the algorithm side of stuff that
4:58
statistical mathematical, thinking, with the, with the
5:02
subject matter expertise. And I've always been a little
5:05
bit less on that real software engineering side of things.
5:08
If you look back to that that data scientist, Venn diagram,
5:12
that everyone was sharing. But five years ago now or so
5:15
I've always been that sort of algorithms subject matter
5:19
expertise what What I've seen is that I've been through all those
5:24
stages of working on a, on a, on a notebook, working on a
5:28
problem. And what you want to do is you
5:30
wanted to prove that you got a certain level of accuracy.
5:33
You're solving a business problem.
5:35
And then then what happens, then you get to this notoriously.
5:39
Putting things in production. What I've learned from the
5:44
process of going through, that is in production is not the end.
5:49
I We're certainly probably three four years ago.
5:54
I was giving a lot of talks and this was the the time that
5:57
people were like, if you heard of a data scientist, putting
6:01
something in production, like this was really, this was a real
6:03
special event. And then we were from our
6:06
experiences that QB, we were really sharing a lot of them
6:08
knowledge. We had it like.
6:10
Okay, guys, we've got, we've got multiple models in production
6:14
now, really, there's a lot of other stuff that comes along
6:16
that you now need to worry about.
6:19
So that's sort of how I've grown into this.
6:21
My perspective is always been a little bit more on their, the
6:25
technical side of stuff. So I have been in the trenches.
6:28
So to speak is being a data scientist.
6:30
I know what it means to Well, to start with nothing to start with
6:36
data, to start with, trying to understand the business
6:38
problems. And to get to make that step
6:41
from 0 to 1, which can sometimes be the most difficult thing,
6:45
you've got it, you've got a feeling of what might happen and
6:47
you make that, that transition across, I certainly like to what
6:54
I think I could make good use of that experience in the role that
6:57
I have. Now more of an executive-level
6:59
role of I can die if a few levels deep levels deeper when
7:06
needed. I mean in the same way that good
7:08
ctOS can dive into the code base and really get to the Crux of
7:13
the problem. I think that you're seeing with
7:17
data leaders, that have that more technical background,
7:20
they're able to, they're able to jump in.
7:22
They can appreciate the the technical challenges that the
7:28
come from from doing machine learning at scale.
7:30
And certainly That's something that I've tried to keep a good
7:34
balance between being enough Hands-On to understand that the
7:38
current challenges because they change, they change every every
7:42
few months. But as well as letting the
7:46
letting the people in my teams, have that space to grow and to
7:49
take on those responsibilities themselves.
7:52
Well it's funny. You mention these technical
7:54
challenges and one other thing that we talked about prior to
7:59
hitting record was the idea that the difficult Parts aren't
8:07
really the technology parts. Can you explain that a little
8:11
bit more? Yeah exactly I mean I think I
8:15
think you're talking in terms of those three real things you need
8:18
to deal with in machine learning at scale.
8:21
One, when you got the technology, To you've got the
8:24
people component. And thirdly, you've got all of
8:26
the processes that are required within an organization to
8:29
actually make the organization gel and to work as one and the
8:34
technology side of stuff, we're very fortunate that we've got
8:38
these companies that are extremely Advanced with using
8:42
data and AI. So that Google's Amazon's Ubers
8:46
Facebook's that have. There are a few years ahead.
8:49
They've been able to have put the effort into building some
8:55
technology that was when they face the problem, they were that
8:58
they were coming up to The Cutting Edge.
9:00
They needed to create something. There was nothing off the shelf.
9:04
Companies that are wanting to work with AI.
9:06
Now, that's a lot of things to choose from.
9:09
You've got, I mean Cloud providers, they offer a lot of
9:13
things that you can just you can hook up and you can get quite a
9:16
long way. I mean things like kubernetes, I
9:19
mean that like if you didn't have that and you were having to
9:22
build that from from scratch, every time, I mean that would
9:26
that would take a lot of development effort.
9:29
So in that sense, I think that the technology certainly in
9:32
comparison To five years ago, that is, that is not the big
9:36
stumbling Point anymore. There's a lot of things that you
9:39
can use sort of off the shelf. You can, you can partner up.
9:43
I mean, to be in the last few years we've been working closely
9:46
with AWS and data bricks. So using some of that tooling
9:51
and that really helps a lot of just allowing the data
9:55
scientists and Engineers. You have in the organization to
9:58
focus on where they can really add the most value and that is
10:02
always going to be. much closer to your specific use case, your
10:06
specific business problems rather than Reinventing the
10:11
wheel or even I mean, I have a background working in tech
10:17
companies that this in companies with a coach like that, you have
10:21
very capable engineers and you know, those Engineers, they
10:25
could build those Solutions and they know they could build those
10:28
Solutions and a bit of them would really like to build those
10:31
Solutions. But you need to you get that
10:35
that balance and you you focus your efforts on where you can
10:39
really make a difference that it's at the business.
10:43
Yes. Yeah, I think I we hear this as
10:48
a recurring theme. This organizational notion of ml
10:51
Ops and you know, I know Demetrius is published in great
10:54
blog posts that everyone has checked out about.
10:57
Why am I Labs is really organizational problem?
11:00
And, you know, I think it's so fascinating to get your
11:02
perspective as you said, because you've really, I guess you could
11:05
call it matured through the entire process of the.
11:09
I guess what we would call the data science, career ladder.
11:11
And now you're at this interesting executive position
11:13
An interesting moment where data science is really crossing
11:16
crossing the chasm and and is starting to become a part of
11:19
business models, a little bit more aggressively.
11:25
Yeah, I mean I think for so long you had sort of Sort of your
11:30
typical organizational setup was to have the, the standard
11:33
organization and been data. Science, comes along in some
11:36
kind of Center of Excellence, or some kind of place where data
11:41
science data. Engineering teams are trying to
11:44
come up with a proof of concept, something to prove the value to
11:47
the business, to bend, get the funding to take that to the next
11:49
level. And you get with that, you get a
11:52
certain amount of putting particularly data scientists on
11:56
some kind of Podium or some kind of pedestal of how If I think
12:01
that he goes a little bit but also needing to do that to build
12:05
that momentum and to get that investment from the rest of the
12:09
business. What I've seen is going through
12:12
that stage. Once you've once you've made
12:15
that investment, you've got a data science and engineering
12:17
team on board. You've done those proof of
12:20
Concepts. You've taken some of those proof
12:24
of Concepts, put them into Pilots, you've put them into
12:26
production. You've demonstrated that those
12:28
value for the business. You've already gone through
12:31
that. You've got that sort of conveyor
12:32
belt built and running and you can you can bring on new use
12:37
cases, tackled new business problems.
12:40
And you've got a way of solving that then what you see quite a
12:45
lot is that than that the challenges they shift away from
12:51
as we sink shift away from technology as the problem, a lot
12:55
more into. How do you, how do you organize
12:59
that? How do you Data science as an
13:01
integral part of the organization rather than just
13:05
some kind of supplementary function.
13:07
That is Yeah. Adding minimal value.
13:11
I mean you often see like the the stats of what?
13:15
80% of there's a don't even know.
13:19
I mean it yeah - don't make even the models that would have value
13:26
if they were made into production.
13:30
Such a challenge of overcoming overcoming that.
13:33
So I think that the organizational challenges come
13:36
from how you embed. A i as a core part of the
13:40
organization, what I've seen in my experience so that the
13:44
company is working out. Until recently QB, we've really
13:49
started on Having data science, Thursday, designs models, with
13:55
something like their work or product features.
13:57
They weren't they weren't focusing on sort of how do we
14:02
assist the CFO in making predictions for the journey for
14:06
the coming six months? They were really focusing on
14:09
taking the data that was coming from a smart thermostat, product
14:13
running machine, learning, algorithms on that.
14:15
And then offering those Services back to those, those end users.
14:19
So then you, you have to, and that essence, you have to get
14:24
data scientists in particular, very close and embedded within
14:27
those product teams. So over time, we made that shift
14:31
from sort of a data science, sort of Innovation Center, sort
14:36
of Center of Excellence into a more of a data science and
14:41
engineering agile, team sort of one sort of team of a product
14:45
owner into then ultimately taking Eventual or typically a
14:51
couple of data, scientists plus a data engineer and distributing
14:55
them between different product teams.
14:57
So you're you're getting those data scientist is close to as
15:01
possible to those business problems and you've got the
15:04
engineering support there to make sure that you can you can
15:08
tackle those challenges. That's that's a tricky shift to
15:12
make from an organizational perspective.
15:14
You need to You have to have a certain level of maturity across
15:20
the organization. This is not only at their sort
15:25
of the executive level used for sure, you have to have the
15:27
buy-in of the CEO. You need to have the the CEO or
15:30
the CTO. Whoever is going to support that
15:34
understanding what AI is and yai is different than software
15:37
development. But you have to have the
15:39
developers themselves software developers across the business
15:42
understanding like, okay, if we're running some machine
15:46
learning code in this In this next product, it could change.
15:50
If it's if it gets retrained automatically on new data, the
15:53
results that we're going to offer could be new, you need to
15:57
have product management. They need to understand that the
16:00
trade-offs between things like, precision and recall.
16:03
And what that means for the kind of service that they're offering
16:08
as an example from from QB, we were looking at Services of
16:13
using a thermostat data within a Home to identify when people
16:18
were, were present at home to alert them, whether they should
16:22
turn the heating off or not. You want to be You want to be
16:27
pretty accurate when you making a recommendation and you'd
16:29
prefer to have less recalls, the less coverage in the population
16:34
of your whole user base. But when you are making a
16:37
recommendation you want to be a certain as you can.
16:39
But that's that's a true bigotry.
16:43
You don't want to be having other things coming in there.
16:46
So you need to have a lot of Education across the whole
16:50
product team of how, to how you need to change things to have ai
16:56
and machine. Money based in that product.
16:58
Yeah, I totally see what you're saying.
17:01
Because the way that I describe it and I were, you know, work at
17:04
a medical device company as our as our listeners, likely no
17:07
verbatim and you know, as we try to infuse machine learning into
17:14
our product. It's so crucial that we
17:16
understand the outputs of what other teams do and really try
17:20
and emphasize to them how much we depend on them, you know, to
17:24
actually get good data and Really, really make sure that
17:27
our algorithms work in their context.
17:30
Yeah, I think definitely the quality of the day sir is
17:34
something that's really important because you, you've
17:37
often sort of have this concept to let the data Lake where just
17:41
anyone can generate data and it gets thrown into the data Lake
17:44
and anyone else can use it. But then you you soon realize
17:47
that this partisan organization that the generating data someone
17:52
else is consuming that in the organization but particularly in
17:56
Organizations you can you can have teams that are generating
18:00
data and then if they don't even realize it's being consumed and
18:03
relied upon by another team. So I think it's really important
18:06
that you that you do have that communication.
18:10
You do have that understanding of How that all works and links
18:15
together, I think, particularly for machine learning problems, a
18:19
lot of the challenges in collecting labeled data, if you
18:22
want me to make a breakthrough in a machine, learning problem,
18:26
it's that labeled data that is going to set you apart.
18:30
I mean, because if you're comparing yourself to
18:33
competitors in space, I mean everyone has access to the same
18:37
technology. You're going to have competition
18:40
for for talents. But ultimately if you've got If
18:43
you got hiding the top level Talent, top level technology,
18:46
what's going to make the difference, is the access to the
18:48
data in particular, that labeled data.
18:51
So, making it clear and letting the, the whole of your
18:55
organization and particularly the teams that are collecting
18:57
that data, be aware of the value, that immense value of
19:02
collecting collecting data, that can be used for labeling is, is
19:06
really important. I mean, art is another example
19:09
from from Key be time, we we were offered a service.
19:14
Ultimately that was detecting a using electricity data,
19:17
inefficient appliances within homs.
19:21
If we'd have just started off doing that, we were not got
19:23
anywhere. But we actually ran a proof of
19:26
concept that our pilot project with about.
19:28
I think hundred and fifty two hundred households that I should
19:32
be installed what's called a smart plug with all of their
19:36
major white good appliances so that the washing machine, the
19:39
dishwasher, the refrigerator, So we could really see when those
19:44
appliances were being used. So this was super important
19:47
labeled data for the for the challenge of identifying.
19:52
When those appliances were running that initial scope of
19:56
that pilot, we were looking at something to do with her
19:59
demand-side response, which is controlling switching devices on
20:03
and off relative to a a price signal that that proved to be
20:07
far too early for the for the domestic customers in the in the
20:11
Netherlands. But the label data, we collect
20:14
really allowed us to make that step forwards into machine
20:18
learning algorithms, particularly that shift from the
20:22
very simple machine learning error as well simple.
20:26
But, like, the traditional sort of random Forest approach, is
20:30
more into the deep learning practice.
20:32
That's what that the label data. Pays back a hundred or thousand
20:35
fold in comparisons is of some of those ever approaches.
20:39
I love how you talked about the Idea of the center of excellence
20:44
and how that's how it starts. And it's like the gateway drug
20:48
into ml through that and you also mentioned something.
20:55
This is going back a little bit, but you were talking about this
21:01
idea of getting everyone on the same page and how important it
21:05
is for everyone. Every stakeholder involved to
21:12
stand the implications. Do you have a special way of
21:16
doing that or what are some tricks that you found can help?
21:21
Yeah I mean it's really important for different levels
21:26
of stakeholder and what we found is the different approaches work
21:30
best for some rather than others.
21:32
So what we QB we actually set up something more like a an
21:36
internal data University, we call it data data science 101.
21:41
This will be Set up. And we essentially used examples
21:44
from within the business to demonstrate a little bit of what
21:48
was going on under the hood. So people could understand what
21:51
was box happening. We offered that to everyone in
21:55
the business. So all everyone from product
21:59
development Finance admin receptionist HR.
22:04
Like everyone was able to access that and very enjoyable to dig
22:09
beneath the sort of scrape beneath.
22:12
The surface, what we found there that was really sort of three
22:15
strands that needed a bit more attention.
22:18
One was the, the product management side of stuff and
22:21
that's it was talking about a little bit earlier, that's
22:23
really understanding concepts. Like, Precision, recall
22:26
accuracy. What that means in the context
22:28
of creating an AI products, then you've got sort of the
22:32
development side of stuff. You've got developers that are
22:35
very happy to just take a little bit of sort of a toy example.
22:42
We'll take some codes and python codes a bit of SQL play around
22:47
with it. They learn the best by getting
22:50
really Hands-On very quickly. So, we we created a few
22:54
notebooks that were able to be spun up by anyone in the
22:56
organization. And they could, they could play
22:58
around with it. They could just, they could
23:00
train a very simpler male model and they got a feeling for what
23:03
that, what that means. And I think the third bit is
23:06
really more of that sort of executive leadership level.
23:11
And that's when your You needing to understand the consequences
23:17
of ail pain, the strategy of the organization.
23:21
I'm particularly the differences.
23:25
Between what it means, but to do AI in comparison to what it
23:29
means, just do software development because I think what
23:33
we've seen is, I mean, if you look sort of 20 years ago, maybe
23:38
even 25 30 years, like when the internet was coming up, you've
23:42
got like, people feel like giving this feeling, okay?
23:47
The internet has a huge potential for changing
23:50
everything. We do the, the companies that
23:53
were able to Adapt to make that shift to it.
23:57
They're the ones that we see that are really leading the way
23:59
now. I mean, I still remember when
24:02
amazon.com just sold books, you know, I mean, they've really
24:06
been able to adapt it. I think we're seeing a similar
24:08
kind of Step change with AI. I think if we look back at this
24:14
in 10 years time, we're going to see the organizations that have
24:20
across the board, understood the implications of what it means to
24:23
do AI. Embedded that within that
24:26
strategy, those are the ones that are going to come out on
24:28
top. So, I think I like to make sort
24:32
of that comparison to some of the CTO role emerging, 20 years
24:36
ago, when you needed you were sort of shifting from that CIO
24:40
role. So someone who had basically
24:41
been there to sort out their, the computer systems and the
24:44
hardware, so everyone could could work on a computer back in
24:47
there in the 80s and early 90s Suzanne.
24:51
The CTO role coming in a little bit more of like, basically,
24:55
Being the tech, the voice of tech in the Board Room.
24:59
So they could they could make that translation between
25:01
technology and business and particularly the impact of the
25:06
internet coming in. I think we're going to see a
25:09
similar shift again with the impact of a.
25:13
I think there's a huge role for sort of the the chief data
25:17
officer or the. I think the CTO is got a good
25:20
role in that as well, but I think we're going to need to see
25:23
a basic Level of understanding from from any executive of what
25:29
it means to work with AI, that's tricky.
25:34
I mean, that's, that's, that's really far outside of the
25:38
comfort zone of many of these Executives.
25:41
I mean, if you're coming in from a financial or operations
25:44
background, you know, your stuff and you're very good at what you
25:49
do. But to make that To be able to
25:52
understand the stuff that you need to know, in the stuff that
25:55
you can ignore is a tricky is a tricky one to do, because
25:59
there's so much noise around the technology.
26:02
There's so much focus on the conversation is upon the latest
26:08
thing. That's rolling out on the AWS or
26:11
Azure or this latest open source project that could disrupt
26:15
everything. And yeah, I mean that there's a
26:19
lot of techies in this field They get excitement from that
26:22
they want to try out the latest thing.
26:24
If you're coming in with the perspective of the CFO you just
26:28
want to know if you're like I don't need something for the.
26:33
You need to be distracted a little bit more.
26:35
Get onto those strategic consequences rather than into
26:37
the nitty-gritty details. Yeah.
26:40
Yeah. No.
26:40
I there was a lot in what you just said and it may also it was
26:44
very, very useful. I really like how you have
26:48
broken down this whole process of data Science Education.
26:51
And in two different personas, right?
26:53
That makes sense. And think about what those
26:56
personas really need in order to be successful.
26:59
I think that, that is, that's a very powerful lesson, not just
27:03
at QB but you know, for any organization that they're
27:06
different personas and how they interact with this fundamental
27:09
concept is important. A question that I have just to
27:12
kind of level set you know it sounds like there was some
27:15
really interesting work at QB going on.
27:17
Can you give us a little bit of sense of like what your role
27:20
was? What your I sort of mandate was
27:23
and what, how that fit into what your company was doing?
27:25
I think that would be some yes good good definitely to get a
27:28
little bit of a flavor. So security is an organization
27:31
was it was a tech company around about the hundred and fifty
27:36
people Mark. So not not a small garage that
27:39
up not a massive multinational, super midscale organization.
27:44
In that sense my role in books of the last couple of years have
27:48
been Chief data and product officer so, I essentially grew
27:53
into that role as The focus of the organization shifted from
28:00
Hardware developing Hardware. So, it QB when I joined was very
28:05
well known for creating a white labeled smart thermostat device
28:09
that's been sold across many European countries, I think, is
28:14
around about half a million of those devices out in the field
28:16
right now. That's shifted into sort of
28:23
software services and then ultimately is into data-driven
28:27
software services. So my role thanks for the last
28:32
couple of years has been that figure out sort of in the
28:35
boardroom alongside my counterparts from commercial
28:39
Finance operations of being that voice of data product and
28:45
technology. And so the title Is always a
28:48
real tricky, one to turn to stick on because it's more than
28:53
just what a chief data officer is doing in one.
28:56
Many organizations. It was really getting core into
28:59
what the product means and the product strategy and ultimately
29:02
the company strategy and making that that transformation.
29:06
So I am I led the hot what could the data Guild?
29:09
So that was all of the data scientists.
29:12
They were engineers product owners, sort of bi analysts were
29:17
all sort of Anyone who sort of had a stomach data, kind of
29:20
flavor and was sort of sitting in that, that group for me work
29:23
closely together there. But also, my responsibility was
29:28
on overseeing the strategy of the organization and
29:32
particularly their interaction with out with our customers.
29:36
So that we have a business model that was a B2B b2c business
29:40
model. So we were manufacturing, a
29:44
hardware devices, smart thermostat, and offering
29:47
services that aren't That we would not offering that directly
29:51
to domestic customers. We were always partnering up
29:54
with a, another company typically, an energy provider,
29:58
the utility company. So, a lot of the challenges we
30:05
get of bringing that to Market is you want to utilize that
30:09
energy utility to assist them in their transformation.
30:13
They want to move away from selling electricity and gas into
30:17
offering a Services. They want to be able to, to
30:21
shift them adapt, their business model, but they are ya.
30:27
Large corporations. They come from a utility
30:30
background, where historically, their role has been to keep the
30:35
lights on and to send a bill at the end of the month, you know,
30:38
it's very different than working in a fast paced agile, tech
30:42
company, right? So a lot of the stuff that we
30:45
did there was actually working with those with those partners.
30:48
And helping them to adapt to different ways of working
30:52
speeding up ways of working. What ultimately happens is
30:57
adoption utility called an echo. That had been our largest
31:01
customer for a number of years and majority shareholder.
31:05
We actually decided to integrate the big q b operations so that
31:09
the hundred and fifty people from the tech company into that
31:12
larger utility. So a lot of my focus for the
31:15
last 18 months has been on How do you make that work?
31:20
Howdy how do you take these two different cultures and put them
31:25
together in to get something that there's ultimately strong,
31:29
that's set up to formally. Kicked off at the start of this
31:33
year, early this year. And that's where I took a bit of
31:36
a step back and decided to turn to let that organization.
31:41
So that get up and running. And for me, it's I'm currently
31:45
exploring what my next challenge in the Now that makes makes
31:48
perfect sense. I'm sure there will be a lot of
31:50
interesting interesting things coming your way and it sounds
31:54
like you know it sounds like Cuba had a reasonably
31:57
sophisticated business model recently, switched Gator product
32:00
and and things that you had to you take into consideration as
32:05
you as you went through an overall technology
32:07
transformation, right? From Hardware to software today,
32:10
driven software as you know, eloquently put it I'm curious,
32:14
if you had to pick a project that really drove home for you,
32:17
the ml Ops challenge, right? Organizational project in
32:21
organizational challenge that we talk so much about.
32:23
Can you tell us what? That might be to the extent that
32:25
you can disclose? Yeah, I'm just trying to I think
32:32
the best example is a say it's not the first project that we
32:37
did is the the first one, your everything's new.
32:40
You're always learning. Like it always takes longer than
32:44
you expect. I mean, I think I first project
32:46
probably took Twelve months from proof-of-concept towards getting
32:51
things into production but that's because you need to build
32:55
the entire data platform and LML infrastructure.
32:58
So offer that first use case and this is always the highest at
33:02
that point to. Yeah I mean you start off you've
33:06
got for the first time you've you've got a use case, you've
33:11
water solution to it and you can't wait to scale that up and
33:15
to get that feedback from your entire customer base.
33:18
To understand, okay? Every really stumbled upon
33:20
something that works or how much does it need to evolve from?
33:23
Where it currently is? Hmm, I think, when we really got
33:27
a feeling for what it meant to work with ML Ops was a sort of,
33:33
I guess we're talking. This is probably like the fourth
33:35
or fifth, major use case that we were becoming from and we
33:39
actually started from all these cases up to that point.
33:43
We're taking data coming in from the smart thermostat that was
33:46
produced in-house. We were in troll of collecting
33:49
all of the data. We actually shifted over towards
33:52
collecting data from a, from a new source.
33:54
So directly from a smart meter that's installed within a
33:58
without a home. So this is distinct from a, from
34:01
a smart thermostat. This is something that is
34:03
instant installed by the electrical utility and allows
34:06
you to get access to electricity and gas data.
34:10
What we were able to do then is we will really creating that
34:16
product quite we didn't have so many of the constraints that
34:21
were perhaps imposed from the rest of the, the technology
34:25
stack of the organization. And we could really start things
34:28
from build things from scratch and to integrate some of those
34:33
pipelines coming in, from a completely new data source with
34:36
those existing pipelines. And the benefit comes from, As I
34:40
was saying earlier using labeled data from one source to
34:44
supplement. The the other that really
34:48
highlighted I think for us the The speed that you can get, if
34:54
you, if you have the right technology, the right,
34:58
particularly the right ml platform in place, we will
35:01
already up and running with. We were leaning heavily on data.
35:04
Brooks ml flow, some of the Delta Lake stuff that comes out
35:11
of them as well. And we were able to do
35:15
something. I think from the initial sort of
35:19
proof of concept which was what one of them were our senior data
35:23
scientist is kind of cooked up in a notebook.
35:26
I say we basically one week got sort of the go ahead.
35:30
So let's get this in production. We were talking in terms of I
35:33
think, three months later. We had an entire new technology
35:37
that was up and running and we were able to Benchmark that
35:40
against some, some competitor, that's great coming in from
35:43
different angles, and we were We were the best in best in class.
35:49
So that that really that brought home to me.
35:52
Like when you've got the technology in place, when you've
35:57
got a team that and an organization is gelling together
36:02
and you've got some, you've got discipline in the processes, you
36:08
can move very quickly. Yes.
36:12
I mean, yeah processes. I'm it I remember starting.
36:16
Off with five years ago at qbm trying to introduce like stand
36:22
ups or any kind of any kind of agile process.
36:26
And there's always this intrinsic, pushback from data
36:30
scientists. Particularly those ones that are
36:33
incredibly creative, they they feel that this is like pulling
36:39
them back but as that sort of takes on, as you get more and
36:42
more into production as it becomes, A core part of business
36:47
operations. These these sort of these
36:50
rhythms help a lot and you can Yeah, you have an advantage of a
36:56
product owner of an agile coach in tweaking things and
37:00
particularly the interaction across the organization, you
37:04
understand, how these things benefit, the the whole machine,
37:10
you know, it's a, yeah, the little bit like I forgot to
37:14
mention it like, bro, I've heard the quote before, like process
37:17
is like traffic lights. It may slow down an individual
37:19
driver, but it makes the whole traffic across the city.
37:23
Flow a lot more smoothly. So, yeah, very nice as you if
37:28
you get away from the sort of this individual driver
37:32
perspective into this, sort of high-level overview of the
37:34
organization and what you can do to speed up the process to
37:37
really really help things. Yeah, I mean I just I love that
37:42
you were able to communicate to us the value and the power of
37:45
platform, right? Because I think that is
37:48
something that, you know, Demetrius and I have seen an
37:51
incredible amount of energy and discussion around.
37:53
And every company is really trying to invest in, not just ml
37:57
but also what does it ml platform?
37:59
Look like, how do we enable each professional at as the, you
38:03
know, get into 4 ml strategy in the right way?
38:06
Whether it's through technology, whether it's due process.
38:08
There's great presentations at, apply from door - you know, I
38:14
think lemonade had some interesting comments about this
38:16
as well, so the power of platforms concept is certainly
38:21
something that we've seen a lot of energy in our community about
38:23
My question from from that, really interesting story is.
38:28
How did you communicate, or did you two other?
38:31
Maybe other stakeholders in the business?
38:33
You know how in fact of this in from investment was like, how
38:37
did you how were you able to say to them?
38:40
We just got we just got our Roi guys.
38:43
Yeah. Yes, it's a good question
38:47
because that's that's particularly that stage we were
38:50
talking about earlier going from that sort of sort of sensor of
38:55
expertise into Center of Excellence into business
38:58
operations. As usual, we were very
39:02
fortunate. Sort of QB scale that the the
39:07
business the use cases, we were tackling.
39:09
First of all were direct product features and due to our
39:13
business. We had a monthly subscription,
39:16
business model and we were able to increase that monthly
39:21
subscription fee that we were judging to our to our end users.
39:26
Directly as we were bringing on these new services that were
39:29
powered by Machine learning. So we could see immediately the
39:34
money that was flowing in that wouldn't have been there
39:38
otherwise outstanding that certainly helped a lot in
39:44
justifying. The Investments that have been
39:46
done to date. You could demonstrate that
39:48
incremental progress. Certainly, if I contrast it with
39:52
organizations that are perhaps using data science to To make
39:56
better predictions about the future.
39:58
You can only know whether that was beneficial when you get to
40:02
the Future and they're able to look back on it.
40:03
So you're maybe talking to three years down the line, we were
40:07
very fortunate in that sense of being able to demonstrate that
40:12
we had with created Services. They had value for our end
40:16
users. They were willing to pay for
40:18
them. We will getting were we work
40:22
whizzing around users? Like do they like does it bring
40:25
Value. Does it does it really solve
40:28
some of your challenges? We're looking at things like
40:31
MPS, measurements such like and that allowed us to to move quite
40:38
quickly. I'm just before I joined QB five
40:41
years ago, we were fortunate enough to have a CEO at that
40:44
time who was really willing to make that investment in a data
40:48
science team and they sort of head of products the believed in
40:53
the power of data science. Being.
40:58
Allowing you to create things that seemed impossible at the
41:02
time, getting to those next levels of energy saving or
41:06
control of your heating system. So we have a lot of belief at
41:10
that time. That's really what kicks?
41:11
Things kick things off, but I'm pretty sure that if we hadn't
41:16
have been able to demonstrate that that end customer value
41:20
sooner the the question marks would have come up because I
41:24
mean like we don't need to go into the numbers but we were
41:26
spending a Amount of money on AWS cost for collecting and
41:33
hosting the data as well, as data bricks costs, for spinning,
41:38
up, clusters and processing. We're talking in terms of
41:42
petabytes of iot data bits, that cost money.
41:47
So definitely needed to have strong support.
41:52
I think it's important. Also to me, think, particularly,
41:54
from my perspective, it's that it's having that openness it's
41:59
sharing with particularly, with peers at the leadership level of
42:04
Of what you anticipate, the, the challenges are right now.
42:08
When you anticipate a sort of a further payback because you're
42:12
always going to your first machine learning use case is
42:16
always going to cost a lot of money you know?
42:18
It's because you need to build everything to get get up and
42:22
running that's where it's going to take a lot of time.
42:24
You're going to, you're going to perhaps struggle a little bit
42:27
with that, the enthusiasm you you're mentioning earlier, like
42:30
you start like the enthusiasm is like right at the peak.
42:35
What were you when you start off there I'm it reminds me quite a
42:38
lot of when I did my PhD. Like you know I was told by
42:42
colleagues. Then like you need to start at
42:44
the beginning of that period with such a high amount of
42:48
enthusiasm, the when you get like like three years in and you
42:51
hit that lowest of the lows then you want that's going to what's
42:54
going to drag you through. I think in the business context
42:58
is nice having that that openness that communication of
43:02
just Showing the value that automation can bring to the
43:06
business. And I think the fact that we
43:09
were able to demonstrate like the first use case took what 12
43:13
months. The second would probably
43:15
talking in terms of Yeah. Probably six months we would now
43:20
like by the time and getting through to like the tenth 215
43:24
fuse case we're talking in terms of smaller use cases.
43:28
We've been outer kind of chop it up into smaller chunks but
43:31
offering new value every every Sprint.
43:34
So every two weeks, yeah, offering something new.
43:37
So really showing that incremental value and that did
43:39
that really helps to get the buy it.
43:42
No, this is a, it's a remarkable success story for us to hear
43:45
about how these things, Can pay off and how they can do.
43:49
And how really showing that End user benefit is just so crucial
43:54
to getting that the whole buy-in.
43:56
And really, yeah, what I found also is that going in the
44:02
outside world and talking about it helps a lot, not only
44:05
forgetting the word out and it sort of explaining explaining
44:09
what you're doing and getting the buzz, but also getting your
44:12
colleagues within the organization to to understand it
44:16
to become proud of it, particularly to shift that You
44:21
want to join the organization? It was very hard.
44:23
We're focused on the company and my colleagues were very proud,
44:27
quite rightly. So of creating a very good
44:30
Hardware product. But then you could see that that
44:33
needed to evolve over time to them being proud of a
44:38
data-driven services to becoming proud of data-driven services
44:42
that were originating from data, not coming from that smart
44:46
thermostat product. You know, it's that, that helps
44:50
a lot of talking. Sharing sharing the lessons
44:53
learnt getting a feeling within the organization that yeah,
44:58
you're doing something that is breaking boundaries that you're
45:02
not every organization out there is being successful over AI.
45:06
This is this is a tricky problem to solve and as an organization
45:11
we are doing, we're doing really good.
45:13
We're in there were in the top 10, top five percent of
45:16
companies out there that that builds up the energy internally
45:19
and that's the Thing that you want your CEO to be telling the
45:25
investors and just you know a lot of these sort of elevated
45:30
chats or coffee machine conversations, that's the things
45:33
you want to be getting out. So I'm wondering about This
45:39
quote from Luigi patru, know who we had on a few weeks ago.
45:44
And he was talking about how You can work on problems where you
45:50
have these, huge benefits and huge gains, but these problems
45:54
translate to really relatively small benefits for the company
46:00
as a whole because really you're working on the wrong problems.
46:05
And so, you're focusing on this wrong problem where you're
46:10
getting these huge gains. But when you look at it from a
46:13
position that's higher up. Then you see that well, Actually
46:18
not the best use of our resources and energy.
46:22
And so from your standpoint being that you had more of a
46:26
bird's eye view. How were you able to direct and
46:30
how were you able to make sure that the problems that you were
46:33
working on? Were the ones that were the
46:38
biggest impact on the business? Yeah, that's always a huge
46:43
challenge. I mean, I think the approach
46:46
that we took the worked best is always keeping that sort of end
46:50
user in mind always being customer-centric.
46:55
And if you're always looking at, how can you add value to that
46:59
end user? So for us, it QB, that was
47:01
typically domestic homeowners that are wanting to stay in
47:09
control of their, their heating bills.
47:11
And Reduce their electricity consumption and to understand
47:14
that. So if we took it from the
47:15
context of trying to solve problems for them, that helps to
47:21
simplify things because a lot of the time the answer is, is not
47:26
machine learning. I mean, probably 95% of the time
47:31
the answer is not machine learning or is certainly
47:34
shouldn't be. The first thing you do.
47:35
I like official, I mean I've been guilty of this in the past
47:39
as well. You get so excited about the
47:41
potential of what you could do with something that you you want
47:44
to jump to show and do that. But a lot of the time just I
47:49
mean a very Pull model. Very simple data driven approach
47:55
can work very well or just a completely non a I approach.
48:01
So I think that helps a lot certainly having a strong
48:05
product management organization that are always thinking.
48:10
In terms of the end-user perspective, particularly the
48:14
interaction was sort of a ux teams that are so in tune with
48:19
Thinking in terms from that end user customer perspective.
48:25
Tying those together. But then being very clear of
48:30
like when do you need to take a strategic step away?
48:34
Because as you were mentioning I mean you can spend a lot of time
48:37
focusing on problems that look like the main thing you should
48:40
be solving. I mean, we could have to be for
48:43
example, focused on making a much better model for detecting
48:48
inefficient washing machines in a home.
48:50
But at some point you know it gets good enough you know that
48:54
That's what we were looking away to new data sources to to
48:58
partnering up. Not just with utility companies,
49:01
but with banks and insurance companies, and spreading out in
49:04
different Power routes there. But always trying to keep that
49:09
end customer sort of as a constant throughout.
49:14
It's tough to say, because obviously, you're you're
49:17
addressing different personas of end customers.
49:19
Well, you everyone is individual and you're having to try to make
49:25
something. That is there's enough value
49:28
that they're willing to pay for it.
49:31
Certainly you see I think this is a problem that the majority
49:36
of companies struggle with is, how do you prioritize the right
49:39
initiatives? Because everyone is everyone's
49:46
always looking at from a limited perspective.
49:50
No one truly has that helicopter view, even, even from my
49:54
experience. I mean, you do get a more wider
49:57
view, but you certainly am not seeing seeing everything.
50:00
Having to be very sensitive there to to the process.
50:05
When I say process you need to have a good way of prioritizing
50:11
initiatives across the organization.
50:14
Particularly when it comes to things, particularly involving
50:17
data science, where you've got a very limited pool of resource
50:21
that you that you want to be maximizing its use.
50:26
So you need to be flexible in how you, how you deploy that
50:30
across. Ization.
50:33
What it really comes down to is saying no to things.
50:37
Yeah, it's always about saying, no, there's at a tech company.
50:44
There's a hundred things you could do.
50:46
There's 10 things that would probably make sense for you to
50:49
do, but you need to focus down on that top one, two, three
50:54
things that are really going to make the difference and I don't
50:58
claim to have solved all of the all of the problems they're
51:01
certainly That's going to be something.
51:04
I imagine I will be continually learning on for the rest of my
51:08
career, but it's it's having that it's having that
51:13
confidence. It's learning from previous
51:15
mistakes, it's saying no when you when you need to and then
51:19
doubling down on something that you know is is heading in the
51:22
right direction. Excellent, Stephen.
51:25
This has been highly. Informational session.
51:30
I thank you so much for coming on here.
51:32
Here and talking to us about it time, just flew by.
51:34
I was really did. I mean we're both taking notes
51:37
and getting all this great information trying to download
51:40
as much as possible into our brains.
51:43
It's really cool to hear this MLM success story.
51:46
That you talked about the idea of you're always going to trip
51:52
and fall especially on those first times that you're trying
51:56
to use machine learning when you're in the business.
51:59
So really have that. That excitement up high.
52:04
So it can carry you through the low periods and how to focus on
52:10
the end user and how to get these metrics.
52:13
How to know what is going to make the most impact where to
52:19
say, know how to define the different personas needed for
52:23
your machine learning initiatives to succeed and and
52:27
the processes that you talked about.
52:29
So I really thank you for coming on here.
52:31
I I want to extend, I hope that if you're not in the slack, you
52:38
and everyone that's listening. If you're not in our community
52:41
slack, you jump in it because I'm sure a lot of people from
52:43
this conversation will have some questions.
52:45
If not, they can always reach out to you on LinkedIn, which
52:50
will leave the description in the description.
52:52
Your your LinkedIn the show via, please, reach out to me on on
52:57
slack. I'm on I'm involved.
52:58
I'm not an incredibly active member of the channel just yet,
53:01
but I imagine And that might change shortly.
53:04
If it does find me on LinkedIn or Twitter or Tennessee.
53:08
Well, thank you again and we here, yeah, we hope to see you
53:13
in slack soon. This has been great.
53:16
Thanks, thanks a lot. Thanks for the opportunities to
53:18
really, nice to give that perspective from from that
53:21
leadership perspective. And I hope we can, we'll be
53:24
seeing more and more. People are joining the community
53:27
in the, in the coming months. Yeah, and I hope like that's
53:31
really It's a great point that you say this and I'm sure
53:34
everybody stop listening right now but it is that leadership
53:37
perspective and hopefully more leaders can start to understand
53:44
and see things. And I think that is what this
53:47
session was really about being able to see the different layers
53:52
of the onion and help peel it back a little bit.
53:56
Totally. Yeah, I mean, how many liters is
53:58
still getting their heads around what devops means?
54:00
So yeah, the surprise Is that they have ml Ops is coming in.
54:04
Hopefully, hopefully, the, the mental shift required is, is
54:08
minimal that? Yeah, yeah.
54:10
All righty. See y'all later.
54:14
Great. Thank you so much, guys.