Latest / Agentic Conversations (formally mlops.community) / Re-Platforming Your Tech Stack // Michelle Marie Conway & Andrew Baker // #281

Transcript

Re-Platforming Your Tech Stack // Michelle Marie Conway & Andrew Baker // #281 · Agentic Conversations (formally mlops.community)

0:00
So hi guys, I'm Michelle Conway. I'm a lead data scientist for
0:04
Lloyds Banking Group. I work in the consumer lending
0:06
data science team and my coffee choice of the morning is always
0:10
a flat white. Hey there, I'm Andrew.
0:12
I'm a data science delivery lead in the consumer lending data
0:16
Science team, Lloyds Banking Group.
0:19
For my coffee of choice, it's always a cappuccino.
0:22
Whole milk and chocolate sprinkles on the top are a must.
0:26
Remember that time Michelle came on here and talked to us about
0:29
how Lloyds Bank was moving from on Prem to the cloud and it was
0:34
going to cost them something to the tune of $5 billion?
0:41
Well, they've made some progress and this conversation was all
0:47
about the sticky situations they got into, some lessons learned
0:52
and how that shift in migration is going.
0:56
She brought a friend along, which is also always fun.
1:00
Let's get right into it. And then it's always, if you
1:03
enjoy this episode, you probably can think about how you heard
1:08
about this podcast and I'm guessing it was because somebody
1:13
told you be that somebody for a friend and tell them about it.
1:20
Let's get into it. Michelle, your veteran on the
1:29
podcast, Andrew, this is your first time.
1:31
It's great to have you here. I was reading through some of
1:35
the fun facts about you all. And Andrew, you're a classical
1:40
guitarist. I am a fellow guitarist.
1:42
That's pretty cool. Classical.
1:43
Electric. Yeah, I just make noise.
1:47
Really. You're much of playing music.
1:51
Yeah. I drew Ning around on it.
1:52
Yeah. No.
1:53
It's to be honest, I don't get to play as much as as much as I
1:56
would like to anymore. It was it was something that
1:58
I've been doing since the age of eight years old.
2:01
Did it did it university as well alongside alongside some maths.
2:04
So I'm not, you know, I'm not a complete, you know, kind of, you
2:07
know, kind of artist, musician. If there is, there is some
2:10
technical stem STEM background and.
2:13
Then real life took a hold. Absolutely.
2:16
Yeah. Yeah.
2:16
Guitar playing doesn't pay the bills unfortunately, so here I
2:19
am. You did just say the guitar
2:22
stools last weekend, Drew. We had a Christmas party in
2:24
Chester for the department and there was a pink guitar in the
2:27
hotel and as it came and it was played it's.
2:31
Like riding a bike though, isn't it?
2:32
It's one of those things once you, once you've got it, never
2:35
lose it. So.
2:36
Yeah, and it makes sense. You you enjoy math because music
2:40
is so much math. They are intrinsically linked,
2:43
although there's not that many places that actually kind of
2:45
offer joyed courses. So I was, I was pretty lucky to
2:48
be able to, to study both of those things and enjoy it as
2:51
long as I could. And then like you say, reality
2:53
set in and you've got to find a job.
2:54
And as I say, I found myself in the in in the corporate world.
3:00
Yeah, happened to me too, so I feel for you.
3:04
So Michelle, last time you were on here, you said something in
3:09
passing that was something along the lines of, let me see if I
3:11
can remember it correctly. It was like, yeah, we're moving
3:14
to the cloud. We budgeted like five years and
3:18
$3 billion for it. And I was like 03 with AB and
3:22
you said yeah, yeah, B So Fact Check me on that is, is that
3:26
right? Five years and $3 billion later.
3:28
Yeah, that is right. That is out in the press
3:30
release. We did that.
3:32
I think we launched it two or three years ago, but it was
3:35
definitely 5 billion into cloud services to help us move.
3:40
I think the main strategic 1 was Google Cloud.
3:43
So we've been on a journey with that since we last spoke.
3:47
Wow. So the both of you, what do you
3:50
do at the company for the listeners who missed maybe our
3:53
first podcast and so they can Orient themselves?
3:58
Certainly. So.
3:59
We're both senior managers on the same MLF's team.
4:02
I operate as a technical lead on the team, so lead data
4:05
scientists and we have a portfolio of around 10 live
4:09
production models that do various different things for the
4:11
business. And so we've been on a journey
4:14
helping to move that onto the cloud platform and away from our
4:18
on Prem tech stack. And we've had a lot of
4:21
learnings, experiences and we're happy to say we've successfully
4:24
moved them. Yeah.
4:27
And I think it's fair to say Michelle, I mean Michelle is
4:28
definitely the more technical out of the two of us.
4:30
My background's primarily in, in business.
4:32
So I'm kind of the the sort of manager of part of the part of
4:37
the team. So I've got a team of, I think
4:39
it's about 10 now data scientists and machine learning
4:41
engineers. So we've kind of built that up
4:43
from from humble beginnings and yeah, into the team, that team
4:47
that we've got today who have, as Michelle has been on, been on
4:50
the journey with us to, to sort of migrate everything across to
4:52
the cloud. And working primarily on the
4:57
platform, correct? Or is it also youth bases and
5:02
and all that? So that's that's primarily what
5:05
we do. So I mean historically our team
5:07
was kind of set up to take on models that had been built
5:12
elsewhere. So by central, central team
5:14
centres of excellence and our skill set is primarily in
5:18
maintaining live machine learning models in production.
5:20
So there are, I would say there are very few teams, Michelle,
5:24
I'll keep you honest here, but very few teams in the
5:26
organisation that I think have got the breadth of experience
5:29
when it comes to the ML up side of things.
5:31
As as, as we do over the last sort of 12 months or so, we've
5:36
been sort of pushing further back in the in the development
5:39
journey, if you like. So a lot more focus now on the
5:42
end to end. So from inception of an idea
5:44
right through to developing that, getting your business
5:47
stakeholders and your risk stakeholders comfortable with
5:49
things and, and as I say, doing what we do best, which is
5:52
deploying things into into production.
5:55
And then of course, as everybody knows, you know, the journey
5:57
doesn't stop there. You've got to work with the
5:58
business to continue to improve things.
6:00
And actually, you know, you I look at it and say projects
6:04
never truly finished. There are always things that you
6:06
can be doing. So yeah, there you go.
6:09
Well explained. I was expecting something else
6:13
to come there rocking Michelle, but I I will keep it rocking and
6:19
rolling. So I think I understand you
6:23
right now. We're like the tail end of the
6:25
machine learning productionizing cycle.
6:28
And you're starting to move upstream and more and more
6:32
upstream. You.
6:33
And before what was happening was you had modellers that would
6:39
create models and then they would say, all right, this looks
6:41
good. And they'd send it to you in an
6:43
e-mail so I could send you an Excel sheet or what?
6:45
It's not not quite, not quite that bad, but there's there was
6:49
definitely, definitely a sense of, you know, we've, we've got
6:52
to find this model a home now we've built it and, and people
6:55
sit and say, oh, geez, we've actually got to run this thing
6:57
now, what are we going to do? And they said, well, we're not,
6:59
we're not really equipped to do this.
7:00
So that was, that was kind of where our team came from, if you
7:05
like. So it started very small from
7:06
humble beginnings. I think there was a, a senior
7:09
manager, a data scientist and a, and one laptop and, and kind of
7:13
off, off you go. And, you know, over time we will
7:17
take more and more of these models on and we've had to, had
7:20
to establish some pretty robust handover processes.
7:23
I think we, it's fair to say, Michelle, you'll know this.
7:25
We, we kind of learnt as learnt as we've gone along, Michelle
7:29
found herself on, on the other side.
7:31
So on the kind of the handover side to begin with.
7:33
So she's had the experience of, I'm going to say, tossing things
7:36
over the fence. That's probably a little bit
7:37
unfair, but you get the right, we just want to get rid of this.
7:40
Off you go and we'll sit there and say, hang on, what about,
7:43
what about the monitoring? What about this?
7:44
What about that? What about the risk angles to
7:46
some of this? So I think what I would say is
7:50
our team probably thinks very differently about machine
7:54
learning models and projects. And, and almost we, we will come
7:57
at things thinking about the longevity of them.
8:01
There is, there is always a danger.
8:02
I think that once you've built something, it's on to the next
8:05
thing and you can, you can leave things.
8:07
And if they get left, as we know, you know, they'll start to
8:10
drift whether in terms of performance, in terms of
8:12
stakeholder engagement and interest.
8:14
So actually the role that we play at the back end of that
8:17
cycle is super important in terms of continuing to realise
8:21
the value that that can be sort of gleaned from these models and
8:26
and sort of make the the upfront cost and investment all the more
8:29
worthwhile. No, 100% because like as well as
8:34
Andrew said, I've seen it from when you go from the proof of
8:36
concept to the MVP to the build to the deploy and like live and
8:40
production. Umm, there is definitely
8:43
something about being at the other end when you're in the
8:45
production end because like you see the engineering standards,
8:48
you see what's good and what's not.
8:50
And our portfolio, I have to say like the engineering and it is
8:53
pretty epic. Like even after re platforming
8:56
other areas of the bank were like, wait a minute, we we want
9:00
to stay close to your team because you have these ethic
9:03
code bases that are locked down and locked away their
9:06
production. And it's so you do see some
9:09
really good stuff when you are on that and a lot side of
9:11
things. What are some questions that you
9:15
feel like your team asks because they are at the tail end now
9:20
when they are engaging in different ML projects?
9:24
So, so I've got a, I've got a great example of this.
9:26
So over the course of this year, we have worked with a third
9:30
party consultancy firm on a new project and it was through that
9:35
build process. So they've got their solution
9:36
design that they were bringing in.
9:38
And we knew that the end, the end game was us taking ownership
9:42
of that, of that model. So we said, right, we need to be
9:44
involved right from the get go in the building, the design
9:47
choices that have been made. And I remember this because it
9:50
sticks in my head. They, the, the consultant had
9:53
said, right, we want to use this particular library, this
9:56
particular package. And we sat there and said, OK,
9:58
we've never heard of this, let's go and have a look and, and see
10:00
what it is. Turned out it was something that
10:03
a PhD student had built for a thesis and hadn't been touched
10:08
for about 18 months or two years.
10:10
And I sat there and I said, absolutely not because, you
10:13
know, you get into a position where the thing suddenly becomes
10:16
unsupported. It, you know, you get, I don't
10:20
know, I'll, I'll pick on Python because that's, that's kind of
10:23
the primary programming language that, that, that our team uses.
10:26
I don't know, you get a Python upgrade that comes along and all
10:28
of a sudden this package is no longer compatible.
10:31
What's the next nearest alternative?
10:33
Well, who knows? And actually, if that's, if
10:35
that's the package that is fundamental to your modelling
10:38
methodology, well, now you've got to go back to the drawing
10:40
board. So like actually we, I'd, I'd
10:44
consider that a small win for our team that we managed to, to
10:46
get them to change their mind and actually adopt something a
10:48
little bit different. It didn't quite have all the
10:51
bells and whistles that they were looking for.
10:52
But again, what we what we would look at as we say, we value the
10:56
longevity and the robustness of a solution over the, you know,
11:00
the perfection or the you know that that pursuit of accuracy,
11:04
which I think you know intrinsically every day to
11:06
scientists wants a perfect model.
11:08
And my experience is that in the real world when people are
11:12
trying to use these things, I would say robustness, explain
11:16
ability, simplicity, the ability of users to understand it
11:19
Trump's accuracy time and time again.
11:21
Now, when I say that I don't, I don't mean let's let the thing
11:25
be inaccurate and, and put out, you know, terrible performance
11:28
because at at some point the business users all come knocking
11:31
and say, well, that doesn't work for me.
11:33
But there is definitely a trade off there that I think we
11:35
appreciate more than perhaps some of our colleagues in in
11:39
kind of other areas of the bank. You're looking for a Toyota, not
11:44
a Porsche. I mean, I, I don't know about
11:47
Toyota. I'll probably I, I, I, I go for
11:49
Ford, I find my my Ford Focus pretty reliable.
11:52
So if that's what we're going for, I'd say reliability over
11:56
Yeah, yeah, yeah. Exactly.
11:58
For sure, yeah, Ferrari I'm getting exactly.
12:02
You don't need that Ferrari, the the Toyota or the Ford that can
12:06
go couple 100,000 miles and you don't need to replace the
12:11
engine. You see that there is the
12:13
longevity in it. You recognise at least in all of
12:17
the known risks. You're covering your backside,
12:21
right? And who knows, maybe something
12:23
comes up that you weren't really aware of, but you do everything
12:31
you can to make sure that you don't have those surprises.
12:35
Yeah. I think, I think that's, I think
12:37
that's totally fair. I mean, I, I don't think he can
12:40
ever be immune from the unexpected.
12:44
There's, there's always going to be, I mean, things evolve at
12:46
such a pace, right? That actually the, the most
12:50
important thing in my mind is having a team who is resilient,
12:54
who is adaptable, who can react quickly to some of these things
12:58
and, and effectively problem solve.
13:00
You know, it's less about the situation you find yourself in
13:03
and it's more about, OK, how do I get myself out of this into a
13:06
place where, you know, we can be happy with what's happening.
13:10
You know, our risk teams are happy with what we're doing.
13:12
And ultimately the business users who are, you know, the
13:15
people that we're serving when we're, when we're maintaining
13:17
these models, they're the people that have got to be happy with
13:20
this. So you know, where we find
13:22
ourselves. OK, let's let's just let's just
13:25
deal with it and and see where we can get to.
13:29
It makes a lot of sense that you're moving upstream because
13:32
I'm sure there's hard lessons learned that you've had over the
13:36
years of being so close to production that when you do go
13:41
upstream, you can nip something at the bud so much faster.
13:45
And just small things, I imagine with security and compliance
13:50
that you have to think about when you're trying to take
13:53
something to production. It's second nature to you.
13:56
And then the teams that are in that AI centre of excellence,
14:01
they're living on cloud 9 like, Oh yeah, this will get through
14:05
or our productionized team will figure out how to get it
14:08
through. So the I, I can see how it can
14:11
be more efficient in that regard.
14:14
But let's talk a little bit now about what the situation looked
14:20
like before as far as the infrastructure and everything.
14:25
And then how did the idea of going to cloud even spur?
14:31
Like it's a big project and I'm sure it was painful in many
14:36
different ways, so why put yourself through all that pain?
14:40
Paint the picture for me maybe? Yeah, we can definitely give
14:44
examples. So like we had loads of models
14:47
that are on our on Prem text track that bells and whistles
14:50
and we went through the journey of moving them to the clouds and
14:54
we noticed like one of our pipelines used should take five
14:57
hours to run. When we moved everything on
15:00
cloud as is, we didn't change anything in terms of the way
15:03
stuff was laid out. It was like a lift and shift as
15:05
much as possible for these models.
15:07
The pipeline went from 5 hours to 20 minutes.
15:10
So we unlocked a lot of capability just being on cloud
15:15
with the same portfolio. Yeah, it's it's crazy, but a
15:19
massive benefit and that's a massive win.
15:22
Obviously there was other things, but that was the one
15:23
that definitely stuck out. We were all kind of like, oh,
15:25
OK, this was a good thing. We.
15:27
Didn't Why? Do you know why it was so much
15:30
different? There was something to do with
15:32
the on Prem configurations and infrastructure was just so
15:36
restricted and locked down that you couldn't unconfigure it.
15:39
And it was so limited that when you went to cloud, you could
15:43
like optimise it with like more noise and computing power.
15:45
And it was like so much better. And then I think I spoke to an
15:48
engineering leader recently and he was just like, oh, the
15:50
signalling and the setup for like on Prem was just, it was
15:54
blocking everything. Like the only time we could
15:56
really use our old platform was in the evenings when no one was
15:59
on us. It unlocked really good
16:02
capability for us and we did move.
16:05
Yeah, So people would just click train model and then go home and
16:11
who knows what happens. Hopefully when you show up to
16:13
work the next day, it's all ready for you.
16:17
Yeah, it was a little bit like that was a little bit like
16:19
fingers crossed if I run it now in the evening, like everything
16:22
will be that no one will be using US.
16:24
And I've ever a few times done that.
16:27
But like, OK, if I hit it off after 6:00 PM, it's more likely
16:30
to survive. And a loads of people in Group
16:31
have done that. So now it's brilliant.
16:33
That stuff works during your working hours.
16:37
Yeah, I think that's really important actually the point
16:39
Michelle was making here the, the scalability is, is a massive
16:43
piece, right. So I mean as as Michelle's kind
16:45
of alluded to there, there was kind of a an infrastructure that
16:49
is predominantly on Prem. And you know, I think certainly
16:53
over the last few years and and more so now with the explosion
16:57
of of Gen AII think there is a real first for organisations to
17:01
do more with the data that they've got.
17:03
And you know, certainly in terms of the focus that that Lloyds
17:07
Banking groups puts on recruitment in the sort of the
17:10
data and tech space, I think they want to sort of use
17:14
certainly the the vast kind of amounts of data that we've got
17:17
at our disposal, together with the latest thinking, you know,
17:20
and the brightest young minds to sort of think about things that
17:23
we can do. Now, of course, on Prem
17:25
infrastructure has its limitations.
17:27
So I think, you know, the, the scalability that cloud offers in
17:30
in particular is, is vital in being able to just just unlock
17:34
the number of different use cases that that people can dream
17:38
up. You know, it's, you know, it's
17:39
that there's so many people I think have got great ideas and
17:42
things that can be done. And what you don't want is your
17:45
infrastructure to be the thing that holds you back.
17:48
And do you think it was just years of using on Prem so that
17:54
there was this very solidified process and structure around
17:59
things that you could potentially run into that same
18:02
problem in 5-10 years of being on the cloud and solidifying
18:08
things and having more and more restrictions every year?
18:11
Or is it more of an underlying infrastructure that's ideally
18:16
not going to happen now on cloud because of the way that the
18:21
design of the system is? So I think what I would say to
18:25
that is there is less in my mind, there's less chance of
18:29
that happening. And I think partly because
18:32
moving away from on Prem and towards a cloud based
18:35
infrastructure means that, you know, the organisation is
18:39
learning about how this works. And it works in a very different
18:43
way to the way that we might think about on on Prem.
18:46
Whether that's, you know, the technical aspects of it or
18:49
perhaps some of the risk aspects of it.
18:50
I think everybody, no matter what part of the organisation
18:53
they're in, when you talk about a move to cloud, they're having
18:57
to upscale and think slightly differently about things.
18:59
So all of those old press, old processes and established ways
19:03
of working I think are being thrown up in the air and people
19:06
are ripping up the rule book and saying, right, OK, we've we've
19:09
got this great opportunity at our fingertips, how do we need
19:13
to think differently? Because those old ways of
19:15
working just just don't make sense anymore.
19:18
So OK, you were having this scenario where you recognised
19:23
things were getting way too entrenched.
19:25
You're on Prem. You're probably not the one that
19:30
made the decision to go into the cloud or maybe you do have 3
19:35
billion in budget or 5 billion in budget that you can throw
19:38
around. You can have a much bigger.
19:39
Team if that were the case. How did the the transition
19:45
happen? As in, when was it decided or
19:47
how was the journey going from on Prem to clouds?
19:51
I want to know how was the? How did it come down to like
19:57
being sold on this idea of changing from on Prem to cloud?
20:03
How did the how did you all get this information?
20:08
Like this is very it's a very big decision, right?
20:13
And I know, like I said, you probably weren't the ones that
20:17
made that decision, but what was that like?
20:20
Like how did? How did you experience it?
20:24
You were just told, hey, we got to go to cloud now and you're
20:27
like, all right, well, let's go about it.
20:29
And how do you even like prepare for such a big project?
20:31
How all of these, these ideas of what did it look like from your
20:37
side of the fence of OK, now we're going to make this shift.
20:43
It was definitely, we had been trying to move to Google Cloud
20:48
over the last few years and then we got new leadership in the
20:52
last two to three years where the very clear strategic
20:55
direction was data's going to drive our organisation now like
21:00
we need the best cloud computing to go with that.
21:03
So there was very good messaging before all this started.
21:06
And then I'd say it would happen as a waterfall effect.
21:09
Like we would have our cloud services department who were
21:13
basically taking the Google Cloud platform, like going
21:17
through it with a fine tooth comb to make sure everything was
21:20
secure and able. And then that would come down
21:23
into our like analytics and AI platform.
21:26
The engineers would look at like what products they needed to do
21:29
their specific jobs. So we worked really closely with
21:33
those AI engineers because they were like building out the
21:35
infrastructure that we needed to set our models on top of.
21:39
And we kind of found that sometimes they're excellent
21:42
engineers. The engineers in our small
21:45
little ML apps team were just as good, if not better when it came
21:48
to the technicalities of things because they would see us as the
21:52
customers and they would build something for us as customers.
21:55
But what they would build, we would be like, no, send that
21:57
back. It's, it's not, it's not good
21:59
enough for what we needed to do. And it wasn't out of like egos
22:04
or anything like that. It was more out of like the
22:06
technical details we knew and we knew exactly what we needed.
22:10
So it ended up being we were more peers building together
22:13
rather than being a customer that was being served.
22:18
And I think. And did you?
22:20
Sorry, just to just to build on what Michelle's saying, I think
22:23
it links really nicely back to what we were talking about
22:25
earlier. I actually as a, as an ML OPS
22:27
team and as a team that plays at that sort of back end of this
22:30
life cycle, our views on what is needed for a machine learning
22:35
platform probably different to a view that you would get from a
22:39
team that is purely there to develop models, right?
22:42
I think there are a lot of things that we have to consider
22:46
from an OPS perspective that if, if your job is purely to build
22:49
something and get it signed off and, and live, there's, there's,
22:53
there's definitely a lot less that you would need to think
22:57
about if if you restrict yourself to that part of the
22:59
life cycle. What did you end up going with?
23:03
Is it off the shelf vertex AI type of thing?
23:07
Are you able to like grab pointed solutions when you need
23:12
them? I would say we definitely use
23:17
some aspects of Vertex because that's where machine learning
23:20
sets within Google Cloud. We did heavily use the Python AP
23:26
is a lot within our code bases and I'd say that's something
23:29
that we learned about. So when it comes to moving
23:32
models that are already live to make changes to them, you have
23:35
to go through a lot of governance and like work with
23:37
your risk department. So our goal was to move them
23:41
without making too many changes because otherwise you have to
23:43
explain your changes. You have to explain your
23:45
commits, your pull requests. The reasons for this, the
23:47
reasons for that, like our release notes are epic.
23:49
Like they're not A1 pager. They are like absolutely epic
23:53
when it comes to explaining everything.
23:55
So you don't want to be like ripping loads of stuff in and
23:59
out. So we were very clever with how
24:01
we used our models and changed the infrastructure points that
24:05
it touched. And so we used a lot of
24:09
different ways around that by using Python based APIs rather
24:14
than just the GUI system of products.
24:17
And that enabled us to move it and lift it nicer and easier.
24:21
I'm sure, I'm sure you remember all the lovely releases that we
24:25
had to do. Yes, yes, there were many of
24:27
them. I think just to just to answer
24:29
what Michelle said there as well.
24:30
I mean, like, I think we are probably one of the few
24:35
organisations that as a well as a financial services
24:38
organisation, as Michelle said, we, we kind of use aspects of
24:43
it. But there is an awful lot, as
24:44
we've kind of alluded to, that has to go through kind of cloud
24:48
risk partners. And they'll look at actually
24:51
from a, from a risk perspective, what are we, what are we
24:54
comfortable with as an organisation.
24:55
Now, different organisations will have different levels of,
24:58
of appetite for some of this stuff.
25:00
But we, we do, we do kind of have to consider the fact that
25:04
we're, we're in a highly regulated environment.
25:06
So I think some of those considerations, you know, we've
25:09
got huge teams of people that are looking at that and thinking
25:13
about what can we use and actually where do we perhaps
25:16
need to adapt things to remain compliant either with broader
25:20
regulatory kind of pieces or our own internal risk appetite.
25:24
So everything that you did behind the scenes of the model,
25:29
you didn't necessarily need to give a big description, Iliad
25:34
and the Odyssey type of description, you could just do
25:39
it. And it's only when that you
25:42
update that model that you need to go and write this compliance
25:47
report. Is that how it?
25:49
Works this kind of I guess there's two strands.
25:51
So there are other teams that that are responsible for
25:54
building out and implementing the platform as it is.
25:56
So that's that's not us. As Michelle said, we worked very
25:59
closely with them. So when they are designing
26:01
capabilities and making those available to us, that was that
26:05
was the process of kind of review and feedback to make sure
26:08
that it worked for us. And then when it comes to each
26:11
of the individual models, you know, a lot of the time
26:15
fortunately there were changes that were not unique to specific
26:19
models. So you're able to make a similar
26:20
sort of change across, across your portfolio.
26:23
And of course then you can sort of explain those changes once
26:27
and, and everybody gets comfortable, particularly your
26:29
risk partners with the, with the idea of what you're doing.
26:32
Where it gets more fun and more interesting is where you have
26:35
models that do something a little bit different.
26:36
And kind of as, as I've said earlier, our portfolio was
26:40
predominantly built by other teams of people.
26:43
And I love this analogy of if I asked five different people to
26:46
write chapters for a book, would I get a book that's coherent?
26:49
Probably not. So you end up with with all of
26:52
our models being a little bit different and you having to sort
26:55
of adapt your approach depending on, you know, specific patterns
26:59
that a team may have used. So it's, it all comes down to
27:03
kind of individual team preferences as to how they've
27:05
done things. One thing that we really push
27:08
for and actually I'm, I'm a really big advocate of is the
27:11
more consistency you can bring to your portfolio, particularly
27:15
from an engineering standpoint, the better.
27:17
You know, how much easier, easier is it for somebody new to
27:20
the team to upscale on ways of working and understand how you
27:24
go about maintaining code bases if you can keep components and
27:28
elements of that consistent right, right across the board.
27:31
So that's definitely something that we used the re platforming
27:35
journey as a as an opportunity to have a think about.
27:38
I think there are still still things that we can do, but it
27:41
was definitely an opportunity to take that step back and and just
27:44
look at the portfolio in the round.
27:47
Yeah. Did you do any type of I, I've
27:50
heard of like model cards being popular or do you have a
27:56
standard way of doing it now since you've gone and you had
28:00
the whole process and you saw like these five different people
28:06
or five different teams that wrote five different models.
28:09
Wow. We might be able to make this a
28:12
little bit more standardised across the team so that we're
28:16
not having to do all of this different like Game of Twister
28:23
to support these models. Yeah.
28:25
So I would say certainly in terms of in terms of our team
28:29
pushing earlier in that journey, that's part of the reason for
28:32
it, right. Is that actually now that we are
28:35
established and we've been around for a while and people
28:37
can see the capability, the the ability that we've got to, to
28:41
take an idea and turn it into something and and sort of
28:44
maintain it at scale. The more new business we can put
28:47
through our doors, the more control, creative control we
28:50
have over designs. And I know Michelle has been
28:53
involved in, in sort of trying to design templates and cookie
28:56
cutter type approaches for, for certain things.
28:58
And that's, that's something we take quite seriously.
29:02
I think there will still be circumstances where others have
29:04
built models and we have to, and we have to take them on.
29:07
And whilst we will, we will try and get in there as early as we
29:10
can in the process to influence their decision making.
29:13
I, I think it's fair to say we don't always catch everything.
29:16
So there are, there are things where we're, I suppose you just
29:19
take materiality based approach, right?
29:21
And say, can I, can I live with this even if it's for a sort of
29:25
short period of time until we, until we've got some capacity to
29:28
then go back and, and perhaps reengineer this to, in, in into
29:33
a format that suits us? No, we definitely do and we, we
29:37
do kind of when it comes to like patterns and standards, like
29:41
every business problem will be different.
29:43
So your data sources will also be different and then higher
29:46
your building that data science model will will be different,
29:49
but the infrastructure that it sits on, that's something you
29:53
can give consistent patterns and like the data sources that's
29:56
coming in and out of like you know, if you're going from like
29:59
a SQL Server database, like maybe mapping to like Cloud SQL
30:03
might be more optimised than going straight into big Query
30:06
because you have less changes in terms of your schemas and things
30:09
like that. So there is certain patterns
30:11
that you can align and make similar, but obviously machine
30:14
learning for different business problems will be different.
30:17
And are you? Oh man, there's so many, so many
30:20
ways that I want to go here. But because the standardisation
30:25
in all of the just optimization for these different teams has
30:30
got to be like a never ending puzzle that you can constantly
30:38
try to figure out better ways. And and I imagine it's in phases
30:42
also as you've, you see this might be better XYZ or done like
30:50
this or these capabilities could give us a bit more efficiency
30:57
gains. And I'm just thinking about that
30:59
like 5 hours to 20 minutes type of pipeline.
31:03
Were there other areas that you saw that kind of lift, maybe not
31:07
that drastic, but things that you recognised, whether or not
31:12
it was just for moving to the cloud or maybe it was because
31:15
you got the chance to move to the cloud and use some different
31:20
tool or have a new design pattern.
31:24
You got to see this works out so much better for us.
31:29
There is definitely I think two things come to mind.
31:31
One was the on Prem and we try to spin up workspace and stuff
31:36
like that. Sometimes the connectivity would
31:37
just drop and it would not be there and would not be
31:39
consistent. And then we moved to cloud.
31:42
I'm not saying we didn't have challenges with connectivity,
31:45
but the connectivity was much better when it was off and
31:47
things were smoother and there wasn't lags and like it was
31:51
better. And then when it came to like
31:54
doing patterns, like being able to massively influence the
31:59
platform engineers with like, hey, you're building this cool
32:02
thing, but actually this is how we're using it.
32:05
Can you align it this way, in this way?
32:08
Because when you're building something generic, that's fine,
32:11
but without actually the use case that it's being built for,
32:15
you might be missing a few functionalities.
32:17
We were able to like tap into that to be like, hey, here's the
32:20
bits we need to configure to make it better.
32:22
And I did find our engineers were great at highlighting those
32:27
bits and pieces and twigging it to be like, hey, change it.
32:30
And it makes it much more robust for everyone using it.
32:34
But it comes from the users using the toolkit rather than
32:36
the engineers of the toolkit. Oh, very cool.
32:41
And do you have some some different ways that you were
32:46
able to configure it? Like was there any any tricks
32:51
that you found or anything? Maybe it's that you were using a
32:54
database in in a different way, or maybe it was that you just
32:58
optimise one database towards a certain.
33:04
We definitely tried to tweak like for like because we were
33:07
mindful of if you move on to certain products that are
33:11
completely different to what you're using now, that's a lot
33:14
more changes that you need to be more mindful of, as well as you
33:19
need to be mindful like what libraries are in your ML models
33:22
to begin with? Like, do you need to strip out
33:25
those Python libraries in order for it to be compatible with
33:28
something new? Sometimes we didn't have to do
33:31
that because we were like, OK, we're no longer using this
33:33
product. Therefore, like these files that
33:35
relate to the config of that product must be removed.
33:39
But other times it's like, oh, actually, if we strip this out,
33:41
like if you're removing a lambic or C card commit, like that's
33:44
lots of Python code. You need to find ways of keeping
33:48
it compatible because it's not that you shouldn't upgrade these
33:50
things, but do it incrementally. Like don't do it as one Big Bang
33:54
because if you do a massive release with loads of changes,
33:57
you don't know what's gone wrong and those changes, whereas if
34:00
you do it incrementally, at least you'll be able to be like,
34:02
right, OK, cool. We changed just the product type
34:05
here, change the data source there or we change new
34:08
infrastructure thing over here. And if you do that into like 3
34:11
different releases, at least you know where you can pinpoint.
34:14
Whereas if you do it all in one bundle, it's like best of luck
34:16
buddy figuring that out. Yeah.
34:19
And I think actually Michelle says there's two things I was
34:22
going to say. So Michelle's touched on an
34:24
important point. I think just more generally when
34:26
you're maintaining a portfolio of of models, right, is the, the
34:29
little and often approach, I think on reflection is probably
34:34
a better way to go. There's there's, there's always
34:36
a danger. I find that because business
34:41
users are so focused on value that they can see you don't
34:45
spend the time on the oh, I'm going to go from this package
34:48
version to that package version. You know the things working
34:52
let's just leave it alone. Let's go and spend our time on
34:54
something that's going to deliver value for the business.
34:56
Well, the, the problem with that is you might find something
34:59
creep up on you where all of a sudden the model that was
35:01
working yesterday no longer works.
35:03
And I've now got to pull three people off what they were doing
35:06
to try and find out what the has but and spend, I don't know, a
35:10
week, two weeks trying to then figure out what we what we do.
35:14
And of course, it's then the knock on impact on, you know, on
35:18
other packages that may be dependent on the package that
35:21
you were using. So it can be a real minefield.
35:23
I wouldn't, I wouldn't say that's something that as a team
35:26
we've necessarily cracked in in the past.
35:29
And I'm not sure there is ever a A1 best way of of dealing with
35:32
that. But I do think you know, for
35:34
anybody who is aspiring to to have a team of of sort of data
35:38
scientists and engineers maintaining a portfolio of
35:41
models, thinking about your approach to ongoing maintenance
35:45
is, is super key. Yeah, it's that classic tech
35:50
that can come back and it can get you in the end, cute.
35:55
And it's never when you want it. You're never ready for it.
35:58
It's not like, oh, cool, we just happened to be doing a hackathon
36:03
and things broke. It's like when engineers are
36:06
out, they're out of office, it's summertime, nobody wants to
36:10
actually work on anything. And boom, what can go wrong will
36:14
go wrong. And Speaking of things going
36:16
wrong, tell me about what hit the fan when you were making
36:22
this shift. What are some huge challenges
36:24
that you had and how did you overcome them?
36:28
I think because we're so large as an organisation, like I think
36:31
we had to talk about maybe 16 or 18 live NL models in Group to
36:36
move and the logistics of figuring out who's moving Watson
36:40
with who. Because you need the platform
36:42
engineers who know the infrastructure and you also need
36:46
the SMEs that are like using the models on a daily basis.
36:49
And you need to form little squads of everyone.
36:52
Like we had scrum and scrums across all these little sister
36:56
teens and China organise this as like an ML apps team where we
37:00
owned 10 of the models and we had like a massive dev team to
37:03
distribute. And as well, you have to think
37:05
about all the models are different.
37:06
Like some might have 10,000 lines of code, some might have
37:10
40,000 lines of code. No model is the same.
37:13
And so some models might need like more months of work on them
37:17
to move them than others. So trying to like do the
37:21
planning and logistics around that was like, it was massive.
37:25
Like it ended up needing like 3 senior managers from our team
37:28
like me and June another like working on it.
37:31
I ended up getting pooled into the central engineers teams
37:33
because they were like, well, we've to move 18 models, your
37:36
team as like 5060% of those, you're coming in here.
37:40
So we want someone who can read the code bases with us.
37:43
And I was kind of like, what the hell am I doing here?
37:45
And being on calls or in going through the 40,000 lines of code
37:49
for 10 different models and explaining where the
37:51
infrastructure is hitting and upscaling the engineers and the
37:55
ML pipeline of each of the different ones.
37:58
It was a massive logistical operation.
38:01
And I'd say it went on for like Andrew, I can't remember, was it
38:03
12 or 18 months? I don't know, it feels like
38:06
forever to be honest. I think.
38:08
I think also just just as Michelle was talking, I was sort
38:12
of reflecting on a couple of things like it's kind of
38:16
implicit in what we've already said.
38:18
But I guess just to be really clear, we were, we were having
38:21
to give feedback on some of the tooling and the capability that
38:24
had been built or perhaps, as we've kind of said, adapted so
38:28
that it's compliant with our kind of security policies and
38:32
risk appetite. You only really find out what
38:35
works and what doesn't sometimes as you're in the process of
38:38
moving these models. So there were definitely moments
38:41
where you're trying to prepare the groundwork for the model to
38:45
move and at the same time saying, right, can you just go
38:47
and have a look at this capability that we don't think
38:49
is is quite hitting the mark in terms of what we need?
38:52
So, you know, you, you might think that you're going to have
38:55
a, you know, fully established, perfectly working, all bells and
38:59
whistles machine learning platform there.
39:01
And right, I can now I can just get on with the process of
39:03
moving the models. In reality, I think it's, it's
39:05
often you, you are, you would be very, very lucky, I would say if
39:10
you ever found that scenario in it in an organisation anywhere.
39:12
If anybody has seen that somewhere, I'd love to hear from
39:16
them. But I think there's definitely
39:17
something about that process of, I mean, I've sort of talked
39:21
about hitting and moving target as as sure kind of moving
39:24
things. Like Michelle's laughing because
39:26
she knows she knows exactly what I mean by that.
39:28
But it's yeah, you kind of have to roll with the punches a
39:32
little bit, right? It's an imperfect world.
39:35
And and actually by starting the process of moving some of these
39:39
things, you figure out what will work and and what won't.
39:43
What I would say is, I mean, it would, it would be great if you
39:46
know if anybody were doing this in the future, a Guinea pig
39:50
project that you can sort of throw it throw at your platform
39:54
to say what are the pitfalls? What are the things that need to
39:57
be adapted? What do we need to think about
39:58
when we get to the rest of the portfolio would be great.
40:01
Now it's you. You didn't get to a weird space
40:04
of do I pick the easiest one to move, which you could do, but
40:07
the easiest 1 isn't necessarily the one with all the different
40:10
capabilities that might be required by your portfolio.
40:14
And as Michelle rightly said earlier, those two models are
40:16
the same, so. In some ways I get to a space
40:18
that says you probably want to start with the most difficult
40:20
one first. But yeah, you might, you might
40:23
find that you sort of turn your developers off in the process by
40:27
saying, right, let's, let's start.
40:28
We don't. We don't really know what we're
40:29
doing. But let's start with this
40:30
complicated 1 here and and figure it out as we go.
40:34
All of a sudden you stop getting invited to the work birthday
40:39
events and you're wondering why nobody's I've got no work.
40:42
Friends anymore and I don't know why.
40:46
Yeah, we definitely got invited to the Christmas party last
40:49
year. Not this year, but no, even like
40:54
there was stuff we realised on the way, like obviously Google
40:57
Cloud are going to be really smart with how they do things
41:00
and they're going to like move with the times.
41:03
And we may have discovered halfway through that the minimum
41:06
version of Python that they support is Python 3.10.
41:11
And you weren't on that? Some things might not have been
41:14
on that, but they were swiftly moved on to that and we
41:17
discovered that on our journey, so there was lots of interesting
41:23
insight to be done. And that's the stuff you
41:26
discover in flight, I imagine, where you're like, why isn't
41:28
this working? And then it's one of those where
41:32
it's really fundamental and you go, oh, OK, we got to spend a
41:36
few more weeks on this. Yeah.
41:37
And, and as we said earlier, right, it's not just, it's not
41:40
just the Python package, it's everything else that depends on
41:43
that Python package. All of a sudden you've opened
41:46
the massive can of worms and what what sounds like a simple
41:49
change when you articulate it like that, you know, just move
41:51
from this Python version to that Python version becomes a right
41:54
now I need to go and check all of these libraries and, and see
41:57
what needs to change. And, you know, if you if you
41:59
move this here, OK, well, that breaks that thing over there.
42:02
And it's it because it can become a bit like whack a mole.
42:04
So it's but that's that's part of the fun of it, right?
42:06
It's probably problem solving is the name of the game.
42:09
Yeah. Yeah, 100% feels like Domino's.
42:13
Have you heard of this? There's a while back I had this
42:18
guy named Benjamin who's the CEO of Steady Bit.
42:21
Come on to the podcast. And he is more in the DevOps
42:26
realm and they do like chaos engineering and their whole
42:30
thing is we're going to stress test the systems by almost
42:37
turning things off. And it comes from the idea of
42:41
chaos monkey where I think at AWS or at Netflix, they had this
42:45
where they would let a monkey loose among the system and it
42:49
would just go and turn things off or increase this or decrease
42:52
that and see what happens, see how resilient the system is.
42:56
And chaos engineering is the next level of that where it's,
43:00
you are having a thesis in mind, Like if our traffic suddenly
43:06
spikes 10X, are we going to be able to handle it with our
43:09
current infrastructure? And how does that look?
43:12
Or if we lose signal from this database or that database, do we
43:18
have enough replicas in place to be able to get that database,
43:22
get that data back? And in that conversation, he
43:28
talked about how well I asked him.
43:32
Some things that I've heard from DevOps folks are, yeah, I don't
43:36
really want to know how brittle my system is.
43:38
I don't like your tool because I know it's brittle.
43:43
I just don't want to know how bad it is.
43:47
And it kind of feels a little bit like that, where you had to
43:51
figure out how brittle it was and then you had to go and fix
43:54
it and you almost didn't realise it.
43:58
Or maybe you did realise it to a certain extent, that what you
44:03
were building your foundations off of may have been false
44:11
security in a way, because you thought the foundation was
44:15
strong and then when you go to change it, you're like, actually
44:18
it's not as strong as we originally had the idea.
44:22
I think that's fair. I mean, I'd say it's ignorance
44:24
is definitely built it in that regard.
44:26
But like, like you say, I think I think the the journey that
44:29
we've been on certainly opened my eyes to to some different
44:35
challenges. But I think with that is, is a
44:37
massive opportunity, right? It's it's an opportunity to take
44:40
a step back and look at look at the infrastructure as you say
44:44
and if there are things that you thought were safe plus or not.
44:48
How do I think about how do I think about improving them?
44:52
I mean, it's really interesting actually.
44:53
You talking about chaos engineering.
44:54
I sort of think our, our friends in the risk teams probably
44:57
wouldn't be massive fans of any of that to be honest.
45:00
But there is definitely something in almost the stress
45:04
testing the system to sit there and say, you know if this were
45:07
to happen, what's going to happen to My Portfolio now?
45:11
I think we're probably fortunate in the, the vast majority of the
45:15
models that we look after are not time sensitive or time
45:19
critical, but there are definitely there will be use
45:21
cases across the bank I am sure where that is that is not the
45:24
case. So of course as you would expect
45:26
in a in a highly regulated financial services organisation,
45:30
anything that is considered critical in terms of a process
45:33
has you know very strict controls around it and there are
45:36
regular sort of stress tests. I think the increasingly as we
45:41
make more use of data science, machine learning, Gen AI and,
45:47
and some of those use cases become more entwined with
45:52
business processes. I think we will fundamentally
45:55
see, see a shift as you say to, to far more focus on stress
45:59
testing of some of these these tools, platforms, the
46:02
infrastructure and, and kind of think about some of these things
46:05
because it's all too easy to forget about it until it doesn't
46:08
work. Exactly.
46:11
Well, Michelle, I would love to know in those weeks or months, I
46:17
don't know how long it was when you were actually it was, yeah.
46:20
What, 912 months of you being on the other team with the platform
46:24
team? What are some things that you've
46:27
learned on how to best interact with these platform teams?
46:32
And the reason I asked this is because in the ML OPS community,
46:38
the question comes up constantly, how much ML do I
46:43
need to know if I'm trying to build an ML platform?
46:47
How? And on the other side, someone
46:50
that's coming from the data science aspect of it, it's how
46:53
much platform or how can I better relate with my DevOps
46:57
team? How can I better help them to
47:00
understand what's going on? So did you learn anything in
47:04
those 12 months of working very closely with the platform team?
47:09
I learned loads. I learned I was a terrible spy.
47:16
I'm a terrible liar in general. So when I devoted 50% of my time
47:21
to be part of the platform team engineer squad that was moving
47:25
everything and then also be a tech lead for our team, it was
47:30
like, I couldn't lie to either side and be stuff like, I can't
47:33
tell you this, but I'll show you anyway.
47:36
I did learn like obviously my data science is the strongest.
47:39
My software engineering, not so much.
47:41
I, I, I do think I'm massively upskilled in software
47:44
engineering because I was plugged into a core
47:47
infrastructure library where I'm like, right, OK, I need to
47:49
massively learn how to upscale here and how to do it.
47:53
And I'm learning bits and pieces and I've been thrown into scrum
47:56
and scrums and really technical meetings where I'm like, I am
47:59
out of my depth. 100% happy to hold his hands up to that.
48:03
But I do know like my verbal skills, my people skills
48:07
definitely helped because people are more likely to help you If
48:10
you're like, hey, you do that really cool thing over there
48:13
trying to show me for just 10 minutes and help Take Me Out
48:15
with this massive hole I have over here because I have no clue
48:17
what's going on. And like the I was so
48:22
uncomfortable during that time that I just got used to being
48:25
uncomfortable. Like everyone talks about this,
48:29
oh, imposter syndrome, like imposter couldn't exist during
48:32
that time. It was like it just wasn't
48:34
allowed. It was like we need to put on
48:36
the big girl boots and get on with us.
48:39
So I learned loads, but I definitely think I added value
48:42
to certain blind spots that the platform might have had as well
48:45
as they teach me so much in engineering that I have a
48:49
massive value for us. And I think it's enabled us
48:52
massively because I now understand the platform like the
48:54
back of my hand and we are looking to expand our portfolio
48:58
into more Jenny I and we're building that on top of our
49:02
existing platform. So the fact that I understand it
49:05
really, really well means I'm now dancing circles around
49:08
different proof of concepts that are coming out on Jenny I
49:10
because I'm like, no, no, that won't work.
49:12
No, that might work. And oh, I like what you're doing
49:14
here. Can I have some of that?
49:16
So it means now we're cherry picking and we have a lovely
49:19
working group with the right technical leads across the
49:22
group, which I think people don't like because we're
49:25
advanced users. But it's not our fault for
49:28
wanting to learn, grow and develop and having a really good
49:31
portfolio that we care about. Michelle's been quite modest
49:35
there as well. She was, she was so infused with
49:38
the platform that she, she also helped our risk partners on
49:42
their journey to kind of understanding, but understanding
49:45
how the platform works. So when we were trying to
49:47
describe some of the changes we've made to the models, in
49:49
fact, she made them so helpful that the challenges that I then
49:52
got back about why we were doing certain things made my life a
49:57
little bit painful, should we say for a few weeks.
49:59
So thanks for that, Michelle, but no, it is absolutely the
50:02
right thing to do, right? I think we're we're all going on
50:05
a journey and actually us being able to take what we know and
50:07
help, whether it's the platform team in terms of understanding
50:11
our requirements, whether it's the risk teams that are there to
50:13
keep us safe, helping to equip them with what we know so that
50:18
they can effectively challenge how we thought about things and
50:22
then genuinely be partner in working through things and
50:25
keeping us safe. I think that was that was super
50:28
important. So umm, yes, as I say, I made my
50:30
life hell for a couple of months.
50:31
So thanks Michelle, but we we got.
50:34
There yet I I was asked just to talk to some of the risk team
50:39
and I was like, oh, they weren't too sure how to use platforms.
50:41
So I did hours of pair programming and I made them a
50:44
little bit too good like they're very good now they could they
50:48
know how to choose that platform, how to pick apart.
50:51
They're an excellent. But I did make Andrew's life a
50:55
little bit difficult when it came to releases and I totally
50:58
forgot about. Incredible.