Latest / Agentic Conversations (formally mlops.community) / Monzo Bank - An MLOps Case Study // Neal Lathia // MLOps Coffee Sessions #20

Transcript

Monzo Bank - An MLOps Case Study // Neal Lathia // MLOps Coffee Sessions #20 · Agentic Conversations (formally mlops.community)

0:16
Everyone will want to welcome me back for another episode of
0:19
coffee sessions. Feels like it's been a little
0:21
while since we've been here but we're super happy to be back.
0:25
Demetrius this year with me as usual and we have a very special
0:29
guest. Today we have nearly a lot.
0:30
Not let me last year, forgive me, if I'm pronouncing that
0:33
wrong and the Mitre said, what do we have in store for today?
0:36
We have a man of the world, Neil is with us.
0:39
He threw me off, man. He was speaking like a true
0:42
American and then I'm like, oh, you're from the states, but what
0:45
are you doing in London? Nope, not from the states.
0:49
It just has the accident in London, grew up in Milan and
0:53
family from that was born in Belgium.
0:55
He's all over the world. All over, remember it.
0:58
Yeah man, forget it. It's like when people ask your
1:02
name and you forget, like immediately after it's like, oh
1:05
crap, you know, but I remembered because it was so it's good to
1:07
know, you know. Yeah, yeah, yeah.
1:09
It's very, it's not common. So anyway, Neil is the, I think
1:15
the official title you have is the doer of cool things at
1:19
Manzano with ML. I like I'll sell it internally.
1:24
Yeah, yeah my title at mon-sol is technically Senior data
1:30
science manager but officially I sort of lead the machine
1:35
learning team inside a month. There we go.
1:38
And we really wanted to break down today.
1:41
What manzo's been doing. I mean Neil's been an awesome
1:45
asset to the community and really sharing his wisdom, and
1:49
what you're doing at manzo's. Really cool nails.
1:52
So I wanted to talk about what your infrastructure looks like.
1:57
What machine learning looks like at Manzano.
2:00
And get into it like a little bit of a case study so that
2:04
others can see what you're doing and how you're doing it.
2:06
And learn from it, hopefully, learn from mistakes that you've
2:09
had along the way and and learn from some best practices that
2:12
you've come up with. And I think we should probably
2:15
start there because I'm going to be very transparent with you.
2:20
Not that it matters for you. I know you have like no horse in
2:23
this race but I'm a transferwise user.
2:26
I also use n 26 and I I'd to sign up for revolute but I they
2:32
denied me like they couldn't. This is the irony of it.
2:35
They couldn't the machine learning Vision.
2:40
What was it? It was like the where it looks
2:43
at your ID. And says, if you are the person
2:46
you say you are like the computer vision.
2:49
Algorithm denied me and said that they couldn't verify my
2:53
identity. So I don't have revolute but I
2:56
want to know like what is munzo doing.
2:58
How are they different than all these other That I just listed
3:01
off. Yeah, cool.
3:03
Great place to start. So Monza is the biggest fintech
3:09
bank in the UK. It's about five years old now
3:14
and we have probably close to 5 million customers.
3:19
Now, how mon-sol is different from these other companies that
3:23
you mentioned like transferwise and revolute is that we are a
3:27
bank and so revolutionary. It is as the last time I checked
3:33
isn't a licensed and regulated bank.
3:35
And so Monza is the so the story of monster that when it was
3:40
founded, they launched a prepaid card where you could load money
3:44
onto your card and then spend it and they use that time to
3:47
develop their app where you would get all of these really
3:52
cool features that have now been adopted across the fin tech
3:56
industry and and even in the bigger Banks.
3:59
So things Like instant notifications for your
4:01
transactions things like being able to freeze your card.
4:05
If you've lost it, things like savings pots and automatic
4:10
roundups. And then while they were working
4:15
on that app, they applied and went through the very difficult
4:18
challenging process of becoming an actual bank.
4:21
And I joined munzo in 2018 when that license had come through
4:28
and they were going through the challenging.
4:30
Part of like, moving everyone from the prepaid, you know,
4:35
account over to the actual retail account that we have now.
4:41
It's been a long journey when I joined mom's.
4:44
Oh, just coming on three years ago, we had about 500,000
4:48
customers. Actually, that was one of the
4:50
first things that I did for a little hack day, was an
4:52
animation of our customer growth, and I meant you and fast
4:57
forward, two and a half years and we We now have business
5:01
banking, we have two different tiers of paid accounts, that
5:05
offer extra things on top of our retail account.
5:09
And we just have this mad growth going on where thousands and
5:14
thousands of customers are joining us every single week.
5:18
That's awesome. So cool to hear.
5:20
And I'm wondering what kind of problems you're solving with
5:24
machine learning at Margaret. Great questions.
5:26
So I guess if I think about the broader industry and a lot of
5:33
these companies that we look towards in terms of machine
5:36
learning, a lot of them kind of have one killer application
5:42
broadly. So like for example if you look
5:45
at Spotify and I know this is a bit reductive But like Spotify
5:50
music recommendation, is there there thing?
5:52
That's where they're they're getting probably majority of
5:56
their gains from machine learning.
5:58
So if you look at a bank, it's not really like there's one
6:01
killer feature that is gonna, you know, be all of the machine
6:06
learning that we do. So instead, And this is my
6:10
experience of like being in a bank, over the last two and a
6:14
half years. Is that a bank under the hood is
6:17
actually like probably like 10 different companies?
6:22
So you have customer service, you have loans and overdrafts
6:28
and credit scoring. You have Financial crime
6:30
detection and then you have all of the cool stuff that you can
6:34
ship into the app to improve customer experience itself.
6:38
So, All of these areas have really interesting machine,
6:45
learning problems. The ones that we have been
6:47
focusing on a lot for the last few years is on the customer
6:52
service side, especially because with Mom's oh when you want to
6:57
get in touch with customer service you do so by the app and
7:01
you get in touch via chat. So that means we have a boatload
7:06
of conversations text. You know, are customer service
7:10
agents are replying via chat and so we have a lot of problems
7:15
that we want to solve to like mediate this interaction between
7:19
the agents to help them help you.
7:21
And also use the text data that we get to help customers help
7:27
themselves and point them in the right direction.
7:30
More recently this year specifically.
7:34
And this is a broader Trend in the industry when lockdown hit
7:37
with Coronavirus. There were, there was a huge
7:41
surge of different types of fraud, like, people calling you
7:46
up and pretending to be the police and saying, you need to
7:50
transfer all your money to, a safe account.
7:53
And this hit every bank. And so, we spent a lot of time
7:58
this year as well. Improving our controls there and
8:02
and shipping machine learning models to detect different types
8:05
of payment from. Yeah, and that is very useful.
8:10
I'm sure lots of people. Thank you for that but yeah, for
8:14
sure, yeah, we're good did meet you before you.
8:17
I just want to, I want to kind of dig a little deeper into some
8:20
of these cases. I'm really curious.
8:22
Why do you think that there's? It's very, it seems like it's
8:24
there's a lot of problems that you could frame as a machine
8:27
learning problem in finance. Why do you think that is?
8:29
Because I find that it's not always, so straightforward.
8:32
For a lot of companies to just, you know, you know, find I'm
8:36
sure. Or they could find a way, but
8:37
where it actually creates real value, where, you know,
8:40
automating that task actually makes a huge difference because
8:42
it just wouldn't work at at such a small scale of you're doing
8:46
manually. So, in your experience, why
8:49
maybe? I don't know, they don't know if
8:50
the, if I can frame the question better.
8:52
But, yeah, well y-you think it's fairly easy to frame a lot of
8:55
problems using machine learning. I think probably because
9:00
majority of the things that we need to do in finance start with
9:04
a human. And so this is actually a common
9:09
Journey for four different product features inside of moms.
9:12
Do that. Like let's take a simple example
9:16
that a customer is expecting a refund on a transaction and At
9:24
the very beginning, we didn't have any way is to report that.
9:29
So people will get in touch with us by the chat screen and then
9:33
maybe the agent would have to go and search for some information
9:37
about, like, what what should I tell this customer and so it
9:40
always begins that we discover customer Problems by them,
9:45
getting in touch with us, by them struggling, with different
9:47
things. And so, in my mind, anytime you
9:52
have a problem. We're like a human can kind of
9:55
make a decision and like one or two seconds about it and in
10:00
doing so they're generating data about the decision that they
10:03
made. There's a good opportunity for
10:06
machine learning. So in this case, as a direct
10:11
example, We want to make sure that all our agents are giving
10:17
customers like clear information.
10:20
And so, one of the things that we do, which is common practice,
10:23
I've seen, even other companies do this, as well, as we have a
10:27
knowledge base that has all these like prefilled responses
10:32
that they can give to customers. So, like, for example, the
10:36
process of like how to replace your card, or the process of
10:39
like, oh, you need to wait a few days if you're expecting a
10:42
refund and so we don't want these Agents to have to type all
10:45
of these words out. So what they end up doing is
10:49
they use a little shortcut that pre-fills that response and then
10:52
they personalize it, you know, maybe say happy birthday, it's
10:56
your birthday or something like that and then they send it off
10:59
to you and it makes it so fast for them.
11:01
But if you look at it from a machine-learning side, now we
11:03
have something that a customer said, we have an agent who's
11:08
looked at that and picked a shortcut to send back to you.
11:11
Yeah. And so now what we did is we
11:13
built A system that when a customer rights in, we recommend
11:17
these shortcuts to the agent, we say, hey, these ones look like
11:20
the relevant ones because we have now probably in the order
11:24
of like a thousand of these responses.
11:27
So that's not that system, I dying to improve it.
11:33
But but it's a good example of the type of systems where you're
11:39
basically looking at work that humans do.
11:42
And that the fact that as they do their work, They're making
11:44
decisions, they're logging data about that work, and then you
11:47
automate it later. Makes total sense.
11:51
Yeah, I'm wondering how you decide on, like, for that use
11:55
case. It seems pretty clear that you
11:58
would want to use some kind of recommender system, but in other
12:02
use cases, I imagine there are a bit of, it's like, not so clear
12:09
if you want to use just rule base or if you want to use
12:11
machine learning. And so how do you go about
12:13
deciding? Which one you're going to use.
12:16
Yeah, so this this plays into Like the fact that we're a very
12:23
small machine learning team right now.
12:28
And so generally whenever there's an opportunity to use
12:31
machine learning in a certain part of the business.
12:34
The, my actual first recommendation was like, let's
12:37
put a rule engine in place, right?
12:39
And made, and that's kind of, for two reasons.
12:43
One, because you don't need a machine learning person and
12:45
build a rule engine, and we definitely have way more Years
12:49
in the company. Then then machine learning
12:51
folks. And then the second bit is that
12:54
sometimes that rule engine is good enough.
12:56
And then, you know, we can focus on where machine learning will
13:00
make the difference sort of unlock a problem.
13:03
So the only things that I factored in there are like, is
13:07
it possible to write a rule engine?
13:10
So with text, it becomes really hard but to write a rule engine
13:17
but an Ample of a ruling in that we use today is when a customer
13:23
does get in touch with us and they come from a specific help
13:26
article in the app. Like, if you go and find a help
13:29
article that is about changing your legal name and then you
13:34
click chat with us. We will then use that signal to,
13:38
as a rule, to send your chat, to an account management specialist
13:44
So there's there's always effectively, we start with a
13:48
rule engine everywhere. And then the question becomes
13:52
like, will machine learning add value here as in?
13:56
Will it make a huge difference? Yeah, that's such a great way to
14:00
get it. I think it's a smart way to
14:02
approach it. You like you develop some sort
14:04
of Baseline. Something some sort of heuristic
14:06
something that you know, is like well-established, you know, some
14:11
excuse for in the People use case that I'm unfamiliar with,
14:14
it's like some feature about a disease and you could use it as
14:18
a baseline. And if you can't beat that, then
14:20
you know, it's probably not worth investing all this all
14:23
this time, you know, developing such a big model, if something
14:25
so simple can really solve the problem.
14:27
And that, that happens more often than I guess.
14:31
Like I would expect, that makes sense.
14:32
Because oftentimes, something really simple can have huge
14:35
impact, but the downside to that is that it doesn't go so far.
14:39
Like I've had a lot of challenges where you reach a
14:41
like, it's It does okay to solve the problem to automate some
14:46
task but it's not as performant as you would like it to be.
14:49
Yeah. But so then you start doing all
14:50
this extra stuff to try to make it super performance.
14:52
But then I don't know sometimes a good Baseline goes really far.
14:56
Yeah. And to be honest, the, the extra
15:00
value of that rule engine is that it often forces the teams
15:04
to start logging data about outcomes.
15:07
And so it creates your data set that you need for for machine
15:11
learning, right? Nice.
15:13
If you're starting from scratch, you know, there's the whole,
15:16
where's the data, do we even have it?
15:18
Like, are we labeling these things?
15:19
Do we have the outcomes and once we build a rule engine and an
15:24
outcome there is is to get that data that we need.
15:29
So one of the things that were curious about is, you know, you
15:32
mentioned that you guys have a small machine learning team.
15:34
So how does that work? How do you go about structuring
15:37
the way that you guys tackle projects and how does that
15:39
relate to? I guess, you know, ml Ops
15:41
because to meet Emily Ops is not just a technical thing.
15:44
It's also cultural it's it's how you organize your, you know,
15:47
your company. So I just can we talk a little
15:49
bit more about how you structure the team and how that affects
15:52
your day-to-day project? Yeah.
15:55
So right now, the Inside of mons.
16:00
Oh, we have data science as a discipline of the organization.
16:05
And there are probably around 40 people across all of data signs
16:10
but then data science breaks down into several different
16:14
areas. So, one of those areas is
16:18
product analytics, understanding our customers building
16:22
dashboards kpis, evaluating experiments, and that's a super
16:27
important part of the business. Second, Area is all of the like
16:31
domain specialisms of banking. So like people who understand
16:35
credit scoring, people who understand finance and they use
16:40
their their data skills to build up the numbers that we need
16:44
there. And then the final area is this
16:48
data, engineering, they build our platform and tooling but
16:52
primarily for our analysts and product data scientists, and
16:56
then machine learning. So machine learning There's
17:00
currently three of us. Soon to be more and yeah, you're
17:05
hiring, right? We've closed the ad because we
17:08
got so many. I've been, I've been on hiring
17:11
calls like, yeah, I think I've done about 20 hours in a lot of
17:14
way. Yeah, it's been crazy.
17:18
So the machine learning team is in terms of the spectrum of data
17:25
science work. We said more on the like
17:28
Building Systems side of of of Data science.
17:33
So what does that mean? In practice is, this is a lot of
17:39
my thinking here was shaped by my experience that other
17:44
companies and by Consulting with other companies and talking to
17:48
different different teams. And one of the most recurring
17:52
frustrations that I hear is, I've trained a machine learning
17:56
model and nobody has shifted for me, and I'm waiting like Months.
18:02
And I'm trying to convince people, please ship my model and
18:07
that is one of the frustrations I've heard so many times from
18:10
from machine learning folks. Yeah, that when when I was
18:14
setting up the team at Manzano, I was like that's the one thing.
18:19
I don't want to happen here. I don't want us to be training
18:22
models and then like chucking them over a wall and at some
18:25
other team to like put into production.
18:29
So that plays straight up into ml Ops it on those two fronts.
18:35
One is that my team needs to upscale?
18:36
They need to learn. What does it mean to put
18:39
something into production? They need to factor that in when
18:41
they're training their models like is this something that
18:43
could be shipped or am I just going crazy with all sorts of
18:47
data that we would never have in production?
18:51
And then the second side is the ml up specific stuff, like
18:55
building that tooling, that makes it easier for us and
18:58
faster for us and safer for us to get those.
19:02
Out. So if I rewind to when I joined
19:06
moans, oh, there was one one, big neural network that was in
19:13
production already and it was powering the article search in
19:18
the app and and it was great that that model had been
19:25
shipped, but getting that model into production, what I was a
19:29
nightmare and it was what was very Difficult.
19:33
Well, we're Summit, just out of curiosity.
19:34
What were some of the challenges there?
19:35
Was it like sighs, like latency things of that nature.
19:39
It was that. So the Monza back end is, is on
19:44
AWS and it's running on kubernetes primarily, it's go
19:49
microservices and our main data stores Cassandra.
19:55
This neural net was written in pie towards and we had no python
20:02
in the back end at all. So what they had done at the
20:05
time. Which to be honest, I don't
20:07
fault them at all because like, getting this thing working is
20:10
the most impactful thing they can do it.
20:13
They built a small flask microservice that was like
20:17
rapped with like a go layer so that I could talk to other
20:22
services and I get that. And and what it would do is, so
20:29
when it was starting up, it, this service needs to have
20:33
embeddings for all of our help articles and it would compute
20:37
that while I was starting up. So just starting up the service
20:40
would take like 20 minutes alone and and then yeah.
20:46
So it was kind of a black box. It was Black Box to other
20:50
engineers and this is a type of risk that I really wanted to
20:53
avoid. Because if that thing NG went
20:56
down and the two people, the two of us were like on holiday.
21:01
Like it would be trouble and, you know, we'd have to be called
21:05
up. And hey, how can we start this
21:07
thing back up again? So our journey has been starting
21:12
from there where, to be honest, like that's actually is the best
21:16
place to start which is get something working, get something
21:19
over the line. Yeah, and then work your way to
21:22
making it quick and easy and safe to do that again.
21:25
And again. Again, and that's a common
21:27
strategy, right? Like, I don't know, I'm thinking
21:28
about, like, writing algorithms, right?
21:30
Start with a Brute Force algorithm, get something
21:32
actually works, that's correct. And then try to, you know,
21:34
optimize it one more thing I want to I want to dig a little
21:37
deeper into. So I think it's important that
21:40
you were bringing up this fact that like you want your team to
21:42
learn how to pick up some of these skills.
21:44
Rice, you want to minimize that bus Factor, you want to share
21:46
that load shared knowledge. How have you gone about that?
21:49
Because I find it's a challenge, like it even in my day-to-day
21:51
work. That's not easy to keep everyone
21:53
up to date with all the things that you're doing.
21:55
Doing and sometimes it just doesn't make sense.
21:57
Like I don't need to worry about what they're doing so much but
22:00
it's helpful if something does go wrong someone leaves or you
22:04
know which is you know, that happens.
22:05
You have to now pick up that scale there, pick up that
22:07
project. So how have you address some of
22:09
those challenges? So from the team perspective,
22:15
again, this sort of anchoring on my previous experience, we're in
22:20
different Orcs. Sometimes you have, like, a one
22:22
to one mapping between data scientists and projects.
22:26
And and that's a huge risk because then you have that one
22:31
person who leaves or goes on holiday and like that project
22:34
stalls, all together. So that was one thing that we
22:39
try to avoid and what as an example of that.
22:44
Is collaboration is so hard in machine learning but because
22:48
most of the work is happening in like people's heads and like the
22:54
code that we write is not really, you know, it's not like
22:58
a traditional engineering thing. We're like two Engineers can
23:02
say. Alright, this bit of code and
23:03
another interesting. Alright, that bit of code.
23:05
I know in machine learning you're like going through
23:07
hypotheses, you're like trying things.
23:11
So the code is nearly like you can even throw it away.
23:13
Way. So one of the ways that we
23:16
tackle this is therefore to like take a problem family and then
23:22
we each work on one of those, right?
23:25
So as an example, last year we were shipping a bunch of text
23:28
classifiers. And we needed to detect all
23:31
these different topics that our customers were talking us to
23:34
about. And so one of the ways you could
23:37
treat, this is like a huge text multi class classifier, right?
23:43
One machine learning model and instead what we did was we broke
23:48
it into rather than let's train one model that can predict like
23:53
15 20 topics. Let's break it up into 20 models
23:57
that can each binary classify one.
24:00
Very nice. And so, that means each of the
24:03
folks, in my team can, like pick up one of these models, we're
24:05
all working on text Data. We're all like anything that one
24:09
person does to like denoise the data, or whatever can help
24:12
someone else and and and at that time was when we did some early
24:19
investigation into Transformers which was then called like pie
24:24
torch pre-trained Bert or whatever it was called
24:27
previously. And so we ended up with With two
24:31
pipelines, one pipeline, which was generating a text data set.
24:36
So you'd say this is my label and it spits out a dataset
24:40
another pipeline that given a data set would train a Bert
24:43
model fine to number of model, right?
24:45
And so it we managed to get collaboration because we're both
24:51
like all everyone in the team is like using the same products if
24:55
you want to call it but they're each using it to like train
24:58
different models. So that's one way.
25:00
Way that we do is like we break up big problems into like
25:03
related problems. And then the second thing that
25:06
we do which again, I guess plays more into the mL of space is is
25:13
Look at what is it that we're doing over and over and over
25:15
again. And that's painful right?
25:20
And so an example there is when way back in the day when we
25:25
would ship them all into production we would put it into
25:28
an S3 bucket would you know put the model artifact into an S3
25:31
bucket. And so we have to ensure our
25:34
like S3 credentials were set up properly and all that jazz.
25:39
And and it was it was a little painful, right?
25:42
So, So we built a model store service and a little CLI tool
25:48
that when a when a machine learning scientists are trying
25:51
to model, they do like python upload this model and they get
25:56
prompted for all the information that they need to add like
25:58
what's model for and all that. And then under the hood, it
26:04
takes care of all the hard stuff like put it into S3 in the right
26:07
place. Make sure that it's available to
26:11
other services when they start up and All that stuff and
26:15
critically also, this really helps me.
26:17
Is it logs data about the fact that we've uploaded a model into
26:21
production so that I can see it on our nice pretty dashboard and
26:26
so, yeah. Like that's the second side is,
26:29
what are we doing over and over again?
26:31
And can we replace it with a tool where do you have something
26:33
that? Like you just said on that?
26:35
Nice, pretty dashboard. That will tell you everything
26:39
about all of the different pieces of the puzzle and when
26:43
they're being updated or when they're being trained and who's
26:45
training them all that metadata there.
26:48
Yeah. So what it has is total number
26:53
of models we have in production and then right beside it for
26:57
each. So, basically, the way we
26:59
categorize, our models is by domain, right?
27:04
So, let's say, one domain is predicting a type of fraud,
27:08
right? Or another domain is like,
27:10
predicting, whether this text About requesting a bank
27:15
statement, and for each domain, then I can see when was the last
27:19
Model uploaded, and then you scroll down and, and it has some
27:26
more information about running training pipelines and all that
27:29
kind of stuff. Are you also having like the
27:32
monitoring of these models on that dashboard?
27:34
Or is that something else that they usually end up being
27:38
separate dashboards to step back a little bit there?
27:42
So Manzo's, analytic infrastructure, is that?
27:47
And there's like some blog posts about this as well.
27:51
But effectively we run all of our production infrastructure in
27:55
AWS. But then all of the data that
27:58
gets law to their gets stream, appended into bigquery Google
28:01
cloud and then we write a bunch of SQL using DBT and that gets
28:10
run a different intervals with air.
28:13
Airflow. Generating these different
28:17
tables effectively which then get visualized in looker.
28:22
And so looker is used across the entire company for everything
28:28
from like high-level kpis. How many customers do we have?
28:33
You know, that type of stuff all the way through to experiments
28:39
And so when it came to picking a place to surface information
28:43
about machine learning models and their performance what
28:47
better place than the place, where everyone else in the
28:50
company is looking, right? Yep.
28:52
I think that was really fascinating.
28:54
What you put in that blog post and we'll link to all your blog
28:57
post because you're pretty prolific on your GitHub.
29:00
So we'll make sure to put that all in the description but in
29:04
that blog post, you talked about how it's like, why am I going to
29:08
bring on another Other tool that the whole company isn't familiar
29:13
with when we are already using this.
29:16
So let's see if I can make what I'm trying to do work with what
29:20
we already have. So that there's no like oh these
29:24
guys need to relearn it or we need to translate it into this
29:26
other tool. And from the tool that I want
29:29
makes complete Sense on that aspect.
29:32
Yeah, there is not Reinventing the wheel to.
29:34
Yeah, I guess in this kind of I know it's different a different
29:40
companies, the thinking behind this, but in my mind if we can
29:48
even tweak an existing tool for a new use case it's easier than
29:54
bringing in a new tool because the more tools that we have the
30:00
harder, it becomes to manage them and to know all about them
30:04
and and effectively maintain them and And and that applies
30:10
also for a lot of our ml Ops tooling that we have under the
30:15
hood, all of the different things that we've built.
30:17
So like model training pipeline, automation model stores, feature
30:23
store under the hood. They're all using the base
30:27
technology that we use to ship stuff features an app as well,
30:32
right? Well, I was wondering about that
30:34
because when I talked I think it was like, two or three weeks
30:38
ago, I talked to. Tim Timothy chain, and he was
30:44
saying how, like what we're seeing in the maturity of the ml
30:49
Ops space is actually, it was Nathan.
30:52
So, I was talking to Tim and Nathan and Nathan.
30:55
I think said this, he was like, you'll see that people will the
31:00
structure of the ml Ops infrastructure or the machine
31:04
learning infrastructure is very much like what you have set up,
31:08
right? So it's you've got something
31:10
that works with what you have and you've made it, you brought
31:14
it on, you may be built some things yourself, you've got
31:17
things working, you've proved out a business case for it, but
31:21
what he felt would be the way forward.
31:24
As we start really advancing is you're going to see people that
31:29
say, okay now I want to stop spending so much time on the
31:34
infrastructure part and I'm just going to buy something.
31:37
And of course let's just Big disclaimer.
31:40
These guys are investors, right? So they hope that that is the
31:44
way forward so that these tools that they've invested in can
31:48
start making money. I imagine and if Tim or Nathan,
31:51
if you're listening out there, hopefully you have to make that
31:55
Roi. But I'm wondering what your take
31:58
on that is. Do you feel like there will be a
32:00
point in time where you say, okay, you know what, we just
32:04
need to go out and buy something because this now is taking up
32:08
too much. Much of our time to try and like
32:11
integrate it. Yeah, that's a tough one.
32:16
So there's some weird things on that front that I've seen so
32:19
far. Some of the companies that I've
32:24
spoken to who have bought these new MLS platforms, I feel that
32:35
they're they're buying them to kind of like leapfrog into being
32:39
able to be more productive with machine learning so it's not
32:43
that they've reached a point of saturation where they're like oh
32:46
this is too too much for us. They're like wait, we actually
32:49
can't do this at all so let's buy this platform to start doing
32:53
it. So that's one pattern that I've
32:56
seen in some companies then the second point that I've seen, is
33:03
that Most of the companies across the industry that we look
33:08
to as like role models for machine learning.
33:13
I don't know. Places like Spotify place, like
33:14
Airbnb Pinterest, Uber Google like, I don't see any of them
33:22
reaching that point, right? Whether I owe this is too much
33:24
for us, right? I actually see them doubling
33:26
down on investing into the tooling that they have for for
33:33
for shipping machine learning. and so, you know, there's there
33:40
I know there are a lot of X over AI folks for setting up ML
33:44
platform companies and you know, a lot of them are in our
33:48
community. But what I'm saying is, is that
33:51
the companies that where we, where we use them as role models
33:55
for ML there, the companies that have invested into that tooling
33:59
and into building it themselves. So because ultimately having
34:04
employees who are productive in your company is like you are You
34:10
right, and my tan, right? And so.
34:15
So what that means is that the company is today.
34:17
We're shipping machine learning and doing it for real.
34:20
And I mean, I by that, I mean, not doing it for research or
34:26
not, doing it for blog posts, right?
34:31
But like actually shipping and for real, like, they're the
34:33
companies that that are thinking about.
34:36
How can I make my employees more productive?
34:40
So, the similar mentality is exist at Mom's 04, our
34:45
engineering org. We used to have a team called
34:50
engineering Effectiveness and that was their whole rehmat was
34:54
make our Engineers more effective.
34:56
And so now we have several teams who look at different angles of
35:02
that problem, in terms of storage, in terms of build and
35:05
stuff like that. And I do feel that machine
35:08
learning is This is like a few years behind, like that's
35:12
basically the recurring discussion between devops and ml
35:15
ox, but, but, yeah, a lot of the companies I see who are doing
35:21
well, in this space are not buying.
35:25
That's an interesting point, it's very interesting it, I
35:28
think it makes sense to that. Some of these bigger companies,
35:31
don't really buy one, they have the, the, either, the money
35:35
right to pay people to build these things, maybe they just
35:39
You can also afford like exploring things that don't
35:41
really pan out and don't end up becoming like this big money
35:45
maker but serve some purpose, like, you know, I'm thinking
35:47
like Google. Right.
35:48
There's a lot of failed projects that they do and they can afford
35:51
to do that. They can try out things and then
35:53
I'll work. But a smaller companies on in a
35:54
position to experiment with maintaining a huge
35:58
infrastructure, that may be fairly complicated.
36:00
So in those situations I feel like it makes more sense to
36:03
start considering like you're saying what's like, how can I
36:06
enable the people that I'm asking it?
36:09
Do something to actually do good work and for, you know, data
36:11
scientists, maybe you know, the the the smart decision is to use
36:15
something that is, you know, Enterprise that is ready to go.
36:19
Yeah, I'm that way you don't have to do too much boiler play
36:21
or write too much boilerplate code and do a lot of these
36:23
things may be that they're not trained to do and maybe the
36:25
company is just that's not their focus, you know, if they're, you
36:28
know, if finance company, that's there, they have engineering
36:31
right to support a lot of the things that they're doing, but
36:34
that's not their primary domain. It makes sense not to, you know,
36:37
waste too much time. Trying to be.
36:39
Tonight, maybe you don't have the bandwidth, the, the
36:41
resources, the expertise to do. And that's I think that's fair.
36:45
But you're I think it's a good point.
36:46
That you said about the like this tournament, I'll go point
36:49
but like a good observation that a lot of these companies when
36:52
they don't have that expertise, they buy something to kind of
36:55
replace that. But yeah, I don't know how well
36:56
that works. I'm really curious to see like
36:58
does that actually end up making your engineers and data science
37:02
is more effective because some of these tools while they are,
37:05
maybe they, you know, I guess fairly mature there still Um,
37:09
you know, there's a lot of things that are changing in this
37:10
capsule. Yeah, sure how far that goes be,
37:13
especially this week, right? With all the AWS announcements
37:18
going out. Having said that, I'll caveat it
37:21
with one thing. So there have been a couple
37:23
times a month. So that we have started using
37:27
tools that we didn't build from scratch in house.
37:31
The two examples I would highlight one is DVT.
37:37
We're using their open source version of vit, and And it did
37:43
that though the reason why we started using it was because we
37:48
saw that it would have a huge gain on our productivity and and
37:53
collaboration across this growing team of data scientists
37:55
and then the second one is the Transformers library.
38:01
And so before we were training text models from scratch and so
38:08
you know, very long pipe lines and a lot of Of code and a lot
38:13
of difficult stuff to maintain and and tweak and know that it's
38:18
working well and and when all these pre-trained models came
38:24
popped up, it just made our lives easier, right?
38:29
So that's kind of one entry point for stuff that we have
38:33
adopted, is we go oh, shit. This thing is going to make us a
38:36
lot faster and more efficient and so it's less about giving us
38:40
the capability to do something. But more about Out giving us
38:43
like the productivity when doing that thing and do you guys have
38:47
a, I want to get into infrastructure, but I know we're
38:51
getting kind of low on time, and I want to touch on that.
38:54
But before we do that, maybe last question here, you talked
38:56
about these like engineering optimization teams.
39:00
Do you have that for the machine learning team also, or is that
39:03
kind of your job? Is that what you were saying?
39:05
How? It's like a little bit in the
39:07
back burner? Yeah, it's been our job.
39:11
Um, and, and I've seen this as a pattern in a lot of companies
39:16
that like machine learning, people are laying the track
39:22
ahead of the train. And so, like, and, and sometimes
39:27
it's a frustration for some companies where machine
39:30
learning, people are spending majority, their time like
39:32
enabling their own work to happen, right?
39:37
So but the key thing that I've seen is that right?
39:41
Now we are the best in terms of knowing what our problems are.
39:46
And so if I look back on this year, for example, we spent Q2
39:52
shipping a bunch of tabular models and they were working
39:56
really well. And then we looked at the
39:58
patterns that we had and we said oh wow, like a feature store
40:01
would really help us here. And so in Q3 we built feature
40:06
store. So we kind of have to do this
40:09
trade-off as a team of like platform in Spence versus, you
40:13
know, like other types of Investments I think, as the team
40:17
grows this sort of specialism will become part of the team.
40:22
Yeah, it makes sense. So let's talk about the
40:25
infrastructure and and I would love to hear about basically,
40:29
from inception, from an idea you touched on it a bit, but maybe
40:33
go a little deeper just like when you realize hey, there's a
40:37
problem here or there might be a problem.
40:40
Here you do this. A rules-based test.
40:44
You see like, hey if we put some machine learning on top of it it
40:48
could make things more interesting.
40:50
So how do you then start with that and take it in?
40:54
Get it across the line. So and this is where you'll hear
41:00
me being opinionated as if I haven't been already tried.
41:04
And I love that we don't mind that at all.
41:07
So often we'll start We say with the data and and prototyping
41:16
ideas in notebooks, and I know that notebooks are a point of
41:24
great debate in the community. Yeah, you said that in your blog
41:28
post to. Yeah, so so my view on that is
41:31
we use a notebook in the same way that Engineers use a
41:34
whiteboard, right? We're mapping out ideas, it's
41:39
code that will throw away. We Care about the results.
41:42
We don't care so much about like, the code itself.
41:46
We care about the insides from there.
41:50
Once we've sketched stuff out, we will typically write a
41:55
pipeline in Python that will say, train a model or create a
41:59
data set. I touched on those earlier and
42:02
those will get run in the Google API platform.
42:06
We're effectively, you're submitting Docker containers to
42:10
run the way. We've made that a little bit
42:13
easier is that we have a cookie cutter template that when a data
42:18
scientist wants to make a new pipeline they just run a
42:20
command, it gives them all the boilerplate.
42:23
They don't have to write Docker files or any of that nonsense.
42:28
Once we train these models, they will get stored in in TCP and
42:36
That's when we will often have to make a decision.
42:39
Is this good enough for product or not?
42:41
Maybe we'll have several candidates and these are the
42:46
type of things we will talk about in the team meetings, and
42:50
when we have something, that's good for prod, then there's that
42:55
one command that I mentioned earlier, which send something
42:59
into our production model store, Across our two environments
43:04
staging and production. And the same person who train
43:09
the model will then build a python micro service to serve
43:13
that model. And again, it's the same concept
43:17
of like one command, it clones a template and it fills it with
43:24
boilerplate. And so the job of the machine
43:28
learning person then is effectively to fill in the Take
43:32
function. What does that look like, you
43:36
know, and what does it have to do?
43:39
And we don't take any strong views on like, we only use this
43:43
type of framework. We try to keep that open.
43:45
So we've shipped pie, torch models scikit-learn Jensen, you
43:51
know, quite a wide range. But the key decision that we
43:56
made there is that our python microservices are, their job is
44:01
to do the predict And they don't do anything else.
44:06
There we go. A little function as a service,
44:08
right? Yes.
44:09
Since we just do one thing and one thing.
44:10
Well, just predict as a service basically, right?
44:13
But the main reason behind that is that python is not the like
44:19
primary like mother tongue of our production infrastructure.
44:22
So I mentioned that we write go services and so all of our
44:29
Engineers use go and write go and understand. go and can debug
44:34
and maintain go and so It's very easy for them to be to be given
44:43
a service where it says, look if you send this data to the
44:46
service you'll get a prediction back and and they don't need to
44:51
know that it's python, they don't need to know what anything
44:53
else that's happening under the hood and we can worry about
44:56
that. And so then that final step is
45:01
either some some of the more senior folks in my team.
45:05
No, go as well and they can just go and build stuff and integrate
45:09
it and And and go, yeah, they just go right?
45:13
And go go in Python are actually very similar languages, right?
45:18
I mean, you say, like, Spanish and Italian, but they look very
45:24
similar. If you know, one, you can learn
45:26
the other. And yeah, I love go side note,
45:29
right, and I didn't know it before moans.
45:33
Oh, I've learned it all in my time here.
45:37
So that's the final step of like, calling the It function
45:40
from the right place of our system.
45:43
The only last thing that I'll mention this is something that
45:46
I've I've also blogged about before is that there's, this
45:50
recurring problem, in machine learning of training model,
45:55
getting some numbers and then do those numbers look the same in
46:00
production. And so what are typical Journey
46:04
will look like there is if we have a brand-new model or even a
46:08
newly trained model, For an existing domain will put it into
46:11
production but we'll put it into this, what we call Shadow mode,
46:15
or I know it goes by other names as well.
46:18
Wear that predict function is getting called but it's not
46:20
getting acted on and so it was just logging data and then that
46:28
all of that data just feeds right back to what I described
46:31
earlier, which is our analytic stack, where will write some SQL
46:35
and surface it in looker. Two things go for it, sorry to
46:39
jump in. That come to mind.
46:42
Instantly is a, how do you deal with reproducibility?
46:46
Because I know banks are under a lot of scrutiny, right?
46:50
Like fintech is one thing but then once you're in a bank, you
46:54
have to there's a lot of regulations and especially when
46:58
you are doing machine learning. I don't think there's so much
47:02
regulation now, but in the EU they're talking about like
47:06
bringing down the hammer. Mer.
47:08
So how do you prepare for that? And how do you make sure that
47:10
you comply with all of that? Yeah, that's a great question
47:14
and it's also a space that's moving.
47:17
Yeah, as you mentioned quite fast in my mind, this breaks
47:21
down to like, what area of the business are we shipping models
47:24
form. So, for example, shipping a new
47:30
text classifier that helps our customer service agents get like
47:35
saved response recommendations. That's kind of a low risk area.
47:42
It's a recommender. Michael's to a human, whose
47:44
internal, they make the final call.
47:47
And so these flavors of machine, learning problems aren't usually
47:54
attended to in such detail by Regulators.
47:59
Whereas on the total opposite, end of the spectrum, is we have
48:04
a separate team that is exclusively focused on credit
48:07
scoring and, you know, shipping a credit score model.
48:12
Is kind of a like that's that's as hard as it gets, right?
48:16
Yeah, because if you ship a credit scoring model, you're not
48:19
going to get a signal back for like 12 months on how good it
48:22
was. And so they go through a really
48:28
like fine-grained, process to replicate validate the results
48:34
report on them and this place straight into the typical
48:38
structure of a bank where you have several Lines of Defense as
48:43
they call it. So, you'll have the front line
48:45
which is the people doing the work, you have the second line
48:48
which are the people who check the work of the Frontline.
48:52
And, you know, and you can, you can go back further there.
48:56
So with these models that are more Critical anything that's
49:02
making decisions about financial crime signups, whatever,
49:07
anything like that. Then we have a much stronger
49:10
focus on the data. The only caveat I because I know
49:16
that replicability is like a big Topic in our community is that
49:21
it's not so much about. Like can I definitely replicate
49:25
this right? Like, can I rerun the code it's
49:29
more about like, did I follow? A structured process that shows
49:32
that I was like doing this safely and in control and not
49:35
just like making ad hoc Cowboy. Yeah.
49:39
Cowboy decisions and say. I like that name I'll bring it
49:43
back in and and then being able to show that we monitor it and
49:52
and that we are aware of the customer impact if there is any
49:56
and that we have ways to find out when things are going wrong.
49:59
So one thing that I wonder about is like, do you have any ethics
50:05
people on the team too? Because you're talking about
50:08
these credit scores and this is what I would consider like
50:11
high-stakes ml you could potentially ruin somebody's
50:15
life, right? Like if you give them well,
50:18
maybe not ruin it. Yeah, maybe you could and also
50:23
like with a loan or something. If you're deciding who gets
50:26
alone and who doesn't, and this is just some Just a quick side,
50:30
note like another passion of mine and I've been doing another
50:33
podcast on AI ethics and this comes up a lot.
50:37
Like this is really a big thing. When you start getting into
50:42
machine learning, you put it perfectly like, if it's just a
50:45
text classifier. Yeah, so yeah, not that big of a
50:50
deal, but if it's something that is going to decide whether
50:54
someone gets to advance in society or if they have good
50:57
credit, bad credit. That's really playing with fire
51:01
in my mind. Yeah.
51:03
So we don't have anyone whose job it is solely to look at
51:10
this. This is like a collective
51:12
responsibility and maybe the the one example that jumps to mind
51:18
is what you mentioned earlier about trying to sign up for a
51:21
revolute account and, you know, for transparency for
51:25
transparency. I have no idea how their system
51:29
as work. But at moans, oh, we do not use
51:35
machine learning to like match your face with your face in the
51:40
document. And what happens instead is you
51:45
submit everything and we pass that on to a identity
51:52
verification provider. And they will manually review
51:57
this stuff. So I know it's one that stands
52:01
out to me because specifically image classifiers are one area
52:07
where you will see the most recurring headlines around bias
52:10
around, you know, race and gender disparities in these
52:16
classifiers. So, I'm not saying that revenue
52:19
does this in any way just to be 100% clear, but I'm what I am.
52:24
I'm saying is that we don't and that's an example of a problem
52:28
that we know machine learning could, you know, like we could
52:31
try it. But we've decided not to because
52:34
it's an area that we just, we don't feel comfortable.
52:39
We don't feel that it would be ethical to reduce someone's sign
52:42
up to a classifier of that. In that way instead what we've
52:48
been exploring is trying to make customers lives Easier in
52:55
signing up for these things. So I'll tell you like one
52:59
specific example. When customers sign up for a
53:02
monster account, they need to take a photo of the front and
53:04
the back of their document. And one of the common errors is
53:09
that people upload the photo of the front twice, right?
53:14
And so that's also an image classification problem and
53:18
actually by classifying that we can give feedback to a
53:22
prospective customer Within In seconds, rather than sending
53:25
that off and asking them to wait hours for a human to review it.
53:30
So that's the angle that we take is like, okay, this is a problem
53:34
that could help our customers could help us.
53:37
It could reduce the work that workload that we create in
53:40
inside of the company. And it doesn't really have this
53:44
potential of you know, going severely wrong.
53:50
So it's definitely a point that we keep in mind, but I see it as
53:54
a shared responsibility across the team, awesome?
53:58
So the last question that I was wondering about on your
54:02
infrastructure and this gets into, I mean, it seems like you
54:06
have so many different use cases.
54:07
Some are potentially streaming like you just said, if you are
54:11
looking at this image, Fire. And maybe something pops up that
54:15
says, this is the same front of the picture that you just took.
54:18
Are you sure this is the back or whatever?
54:22
Like maybe breaking down, how the differences between shriman
54:24
and batch, but before that, maybe you can explain why the
54:29
two clouds. Why are you using gcp 41 and AWS
54:33
for other things? Yep, so that's a decision that
54:36
predates me. But my, my hunch, There is that
54:44
it's because of if you look at the most senior people in the
54:50
company who were who were starting it up five years ago,
54:53
our VP of data was at Google before in and our VPS.
55:01
In engineering worked at companies that were working on
55:05
AWS. So my hunch is it's that but the
55:11
main benefit That we've seen from it particularly in the
55:15
early days and is that by having two clouds you create a blast
55:21
radius around one Cloud, right? So like effectively when data
55:27
scientist would on board, you know a year or two years ago, we
55:30
could say you know what, you can actually break everything there
55:33
in gcp because it won't affect our customers but it won't make
55:38
it won't stop the app from running and anything like that.
55:42
At and then we can even make our production infrastructure, much
55:46
more secure because like very few people have access to it and
55:51
we can we can really lock it down.
55:54
So that's one of the main benefits that we saw which is
55:57
that you know. Yeah.
55:59
Like delete that table right. Nothing bad is gonna happen or
56:05
the worst. That's going to happen.
56:06
Is someone is going to pop up in our slack town going.
56:09
Hey, what happened to this table?
56:10
And we'll go hooks will put it back.
56:13
So it's been good and and plus then I get to learn about both
56:16
of those clubs. And so, as far as the real-time,
56:21
like streaming first batch, what are these?
56:25
These different use cases and how do they differ if they do at
56:28
all? When you're doing things?
56:32
So usually for bat, now we can run it fully in gcp.
56:39
Because of the nature of batch problems where, you know, say it
56:44
will be looking at the last day of data and running predictions
56:50
on it or looking at all the customers who did X and running
56:54
predictions on them, it makes it much easier to access for for
57:01
data scientists, including those outside of my team for the
57:06
streaming, I guess. Under the hooded sort of to use
57:11
cases so we could maybe call it live predictions.
57:14
Where one is, it's a system that is consuming events from streams
57:20
and then running predictions or the second is it's a system that
57:25
is receiving live our PCS or HTTP calls to return a
57:32
prediction. And those are the ones that
57:34
require that second half of the journey that I mentioned of
57:37
like, creating a micro service. And Began production.
57:41
In general, we have broadly been biased towards that one because
57:46
that's where we tightly integrate with the product more.
57:49
So than the bad stuff, I even though I know that in a lot of
57:53
companies, the batch pattern is probably the most common one.
57:57
Yeah. And it makes a lot of things can
57:59
be start off that way, right? Like there's no, you know,
58:01
latency constraints and you could just do things off Klein
58:05
in bulk, but after a certain point like yeah.
58:07
Like so I'm like, you were mentioning With some of these
58:10
applications I guess that that just doesn't work.
58:12
And what do you how do you address those issues?
58:14
You guys? I think original question, we
58:17
were targeted to meet you. It was about like the different.
58:19
Like how do you you have bad, you have streaming, but do you,
58:22
how does that affect your infrastructure?
58:23
So that are you doing some things on AWS?
58:26
Some things on gcp? Like, how do you choose?
58:27
Which one? Because sometimes, they have
58:29
pretty much like the same service, right?
58:30
Like I manage instance of something.
58:32
So how do you go about, you know, picking those things up?
58:35
And that's just the last question, sorry?
58:36
Yeah. So the Summary would be trained
58:41
and batch in gcp, and then serve and live in AWS.
58:47
And and that's just because of the historical reasons of of our
58:51
infrastructure. That the one thing that I would
58:55
add to that, which happens on the live side is that if there
59:01
is a service that does some sort of inference that already
59:05
exists, right? Take these like very popular
59:08
areas like Comment analysis or these sort of things that like
59:14
AWS offers a service. Now like that saves three months
59:18
of my team's time. We don't need to train a model,
59:21
just call that API. And so that means that the
59:24
problems that were tackling are the ones that are specific to us
59:29
rather than generic problems that are now turning into apis,
59:34
that these big tech companies offer.
59:36
Yeah, awesome. Well, Neil, we A really great
59:40
time. We talked about a lot of
59:41
different things on a high level summary.
59:43
We've talked about a little bit about Manzano about the work
59:46
that you guys are doing the scale that some of the
59:49
challenges you guys are dealing with the different ways.
59:51
You frame machine, learning problems.
59:54
We talk about teams or how that, how that?
59:56
How that plays in time. Elapsed solving problems by
59:59
breaking them down. A lot of really great things too
1:00:01
much to summarize here. But to to close one thing I
1:00:04
would like to ask in, this is something we haven't done this
1:00:06
in a while, but if you were to give the ml Ops community, E one
1:00:09
piece of advice. It could be something that's
1:00:10
just in your mind right now. Something like your, you were
1:00:13
thinking about earlier. But what's that?
1:00:15
One thing that, I guess that one little piece of nugget of wisdom
1:00:18
to close this out. Yikes, putting me on the spot
1:00:22
here. Sorry, that we have done it.
1:00:26
Okay, I won. I won.
1:00:28
I won. And so As you've seen, I've been
1:00:36
following along with the community a lot.
1:00:39
And I really enjoy it to be honest.
1:00:41
Nice, what I'm seeing in a lot of the propositions for ML that
1:00:50
are coming out from new. MLS companies is like worried
1:00:54
about this problem. Worry about that problem, worry
1:00:56
about that problem, right? Like and and so, all of these,
1:01:00
I'm a large companies are like saying, like, oh, You're not
1:01:04
like monitoring your features, then you're doing it wrong.
1:01:07
If you're not doing this, you're doing it wrong.
1:01:08
If you're not doing that, if you don't have a feature story
1:01:10
you're doing it wrong and and to a certain extent they're right
1:01:15
but don't let that stop. You from actually just shipping
1:01:18
some machines, right? And and get something over the
1:01:22
line first, and and then worry about like, starting to monitor
1:01:27
stuff and starting to make it more replicable and iterate
1:01:31
easier to do so. Yeah, like I said, the first
1:01:35
thing that happened at Monza was we shipped a huge complicated
1:01:38
difficult to ship neural net but we monster.
1:01:43
Yeah but they did it right and that was the success of that
1:01:46
work and then we made it easier so so don't fall into like
1:01:51
analysis, paralysis, yeah, ml Ops tooling and yeah, that'd be
1:01:58
my little piece of advice, very, wise and exhausted.
1:02:02
I will also pretty And we've heard that same sort of like
1:02:06
that same advice before right now, iterate on things just to
1:02:09
start and there's like almost like this like like f foam,
1:02:13
alright like fear of missing out if I'm not doing the cool thing,
1:02:15
right? I'm not using the Cool Tools but
1:02:17
yeah I think it's a great point it's it's really not what's
1:02:19
important get something done focus on creating business value
1:02:22
and then take it from there. Yeah that it was really very
1:02:25
similar to what Satish said. I don't know if you remember
1:02:28
that David but it was like that's what he was talking about
1:02:32
to and it was like just because you can't do this thing.
1:02:36
Do like one part of it doesn't mean you shouldn't be trying to
1:02:39
do other parts of it, right? Little similar there and my
1:02:45
daughter yelling in the background.
1:02:46
I don't know if you can hear but I was also going to mention one
1:02:50
thing that stood out with me on what you said.
1:02:53
Was this idea of a blast radius. That's so good man.
1:02:57
So awesome. And I love that you set it up
1:03:00
that way so that you can insulate yourself from Mistakes
1:03:04
because we always make them, we tend to do them.
1:03:07
That is one thing that I'm going to take with me from this talk
1:03:11
for sure. And I really appreciate you
1:03:13
coming on here telling us all about what's going on with
1:03:16
Mom's. Oh and yeah yeah we'll see you
1:03:19
in the community man. Yeah this is an awesome having
1:03:21
me and I will be there. See you later.
1:03:24
Anyway so thank you everyone for the oh crap.
1:03:27
No I was just going to say it will link all the blog posts and
1:03:30
everything we referenced down below.
1:03:33
As we do, I did realize after we started, I forgot to turn on the
1:03:36
music. We like, that was just the
1:03:45
warmup. Yeah.
1:03:47
If anyone said us anyone, listen, if anyone's good with
1:03:49
that type of stuff to make our videos, awesome, please reach
1:03:52
out to us because I mean Demetrios we have we suck at
1:03:54
this we're just going to start talking and forget about what's
1:03:57
going on. And we only anyways is it can
1:03:59
impose, all right, that's it. We're conscious for.
1:04:02
Listening by guys. Listening by guys.