Latest / Agentic Conversations (formally mlops.community) / Scaling Machine Learning with Data Mesh // Shawn Kyzer // Coffee Sessions #116

Transcript

Scaling Machine Learning with Data Mesh // Shawn Kyzer // Coffee Sessions #116 · Agentic Conversations (formally mlops.community)

0:00
I drink a lot of coffee. In fact, I like coffee different
0:04
ways. Sometimes I do the French press
0:06
in the morning. Sometimes I go downstairs and
0:08
get a nice Italian espresso, and then every now and then I feel
0:11
like a cortado, which is like milk and coffees.
0:14
Alright? Today, we are graced by the one
0:18
and only Sean Kaiser. He is working at thoughtworks
0:22
and you probably have heard of thought Works before and more
0:27
recently, probably around. Around something called the data
0:31
mesh where Shawn Works closely or used to work closely with the
0:37
creator of this whole movement, the data mesh, shh, Mock.
0:42
And what I enjoyed about this conversation, I'll break it down
0:47
a little bit for you. We had Adam on as a co-host and
0:51
everyone knows that Adam is very into the data mesh.
0:56
So Sean and Adam got to go toe-to-toe and Adam.
1:00
Is actively trying to implement some of these data mesh
1:04
procedures into his company origami and he got to pick the
1:09
brains of Sean to see how the best, and the brightest who were
1:13
having success with the data mesh are actually achieving the
1:19
success. So the other pieces that I liked
1:22
was that Sean really broke down the cultural piece of data mesh.
1:27
And how, Many parallels, there are with the ml Ops movement
1:34
when it comes to the cultural side of ml Ops you can't just
1:38
get something done by using tools.
1:41
You need to go much deeper than that.
1:43
And it is very clear that this data mesh movement is the same
1:47
thing. You have to really educate the
1:51
organization towards the movement.
1:55
Next up like data. And the needs that data
2:02
scientists have in the eyes of a data mesh, like, through the
2:09
lens of the data mesh is very interesting because the data
2:12
scientist is a data consumer and therefore, if a data scientist
2:17
needs data in a certain way, or has certain specificities about
2:23
the data that needs to come through, they can push that up
2:29
the chain. Jane push it onto the engineers
2:32
and tell them like, hey this is how I need the data, this is my
2:37
data product needs to look like this.
2:39
So as a data scientist, you can make these requests.
2:43
You can go up the chain and put it onto the engineers.
2:47
And I thought, Adam had a great Insight around this because he
2:51
is almost seeing it at his company.
2:54
He feels a little bit scared to push so many requests Upstream.
3:00
And ask Engineers for all these special things on the way that
3:04
they want their data. And they want to be able to
3:06
access their data, but Shawn told him, straight up, look your
3:10
data consumer, you are a stakeholder, it is up to the
3:13
producers the data producers to service your needs.
3:18
So without further Ado, let's get into it.
3:20
And before we do, if, you know, anyone that would enjoy this
3:25
conversation, we would be honored if you share it.
3:29
With just one person. And if you have any ways that
3:34
you think this podcast could be better.
3:38
We would love to hear it. We've got thick skin, so give us
3:42
a whole lot of suggestions. We will take them and like,
3:48
honestly, I love hearing feedback from everyone out
3:51
there. That's all, let's jump into this
3:53
conversation with Adam and Sean and myself.
4:00
Yeah, we brought you out here to talk about many things
4:04
specifically what we were mentioning before we hit record,
4:07
which was all about how a data scientist /ml engineer's.
4:12
Last data, engineer can think about data mesh for their
4:18
machine learning projects, you gave a great talk at.
4:21
ODS. See, and we'll link to that in
4:24
the show notes if anybody wants to get really into the
4:26
nitty-gritty but I think it's probably worth just Getting a
4:31
can you like set the scene for us what's going on?
4:33
Why are we confused? And as you know, we've had Scott
4:37
on here before he tried to explain to us what the data mesh
4:40
was. And I remember saying like,
4:43
yeah, data mesh is kind of like World Peace.
4:46
Everybody wants it, nobody's against it.
4:48
But how are we going to implement it?
4:49
So hopefully we can, we can get some more specifics from you.
4:53
And I know we've also got Adam on here who is a huge data mesh
4:58
Advocate and he's trying. Figure out how he can use it at
5:01
his job and at his company, current company.
5:04
So we want to ask a few, kind of pointed questions about maybe
5:09
some implementation details, but let's start by setting the
5:12
scene, what's going on with this data mesh, right?
5:16
So data mesh is really starting to take off in the last couple
5:21
of years and most recently, a lot of what we've done like kind
5:25
of here at thoughtworks and Jamaica.
5:27
She's recently love, thought works but she She's basically,
5:31
you know, kind of been our kind of spirit guide through this
5:33
whole process and of course, she wrote the book on it.
5:35
So we've been able to have like a lot of like interaction with
5:38
her right? Which is really, which has been
5:40
really incredible and a good experience.
5:43
But initially, when we started off this project, it seemed to
5:46
be like okay, it was mostly created for like the analytics
5:50
plane. Right.
5:52
So you know, time series data Flows in, You know, we capture
5:55
it, we subdivided into its architectural quantum's.
5:59
It's all about Product. So every data is a product,
6:02
right? So you don't build what we call
6:05
a data product without having a valid use case and it doesn't,
6:08
you know, it has to add value. And so, in the data space, this
6:12
made a lot of sense, right? And we also said, you know, it
6:15
needs to be trustworthy discoverable.
6:18
And so each one of the individual data products would
6:21
have all of these aspects and it would be tied to tools like
6:25
discoverability would be tied to like a data catalog, right?
6:28
Trustworthiness would Be tied to like measuring things with like,
6:32
you know, measuring the data with Great Expectations, stuff
6:34
like that and that works, you know, orchestration was like
6:38
airflow, right. So that works really nicely for
6:41
our like batch flows and even for our streaming flows.
6:44
Totally fine. And so we got through this means
6:47
that. Okay, you know, this makes sense
6:50
in the data world, we understand this.
6:52
Like we can create output ports that data scientists can
6:55
leverage like well-curated sets of events, right?
6:59
And a data. Just can go in, they know where
7:01
to find it from the data catalog.
7:02
They know it's addressable where to go get it, but then that's
7:07
where it stopped, right? That's where it started to kind
7:08
of break down. Because then we were like, okay,
7:10
there's these weird little kind of ml use cases, right?
7:15
So and a lot of times what would happen is the machine learning
7:19
would start to cross into different planes.
7:23
So it would start in the analytics plane where you would
7:25
go fetch the data feet, you know, you would feature engineer
7:28
it and then you would Train the model test, the model, all that
7:31
good stuff. And then create maybe an API and
7:35
then suddenly like the operational plan needed to
7:39
leverage that API things like fraud detection or you know
7:43
order delivery times or like any number of like ml kind of
7:48
applications that you can think of.
7:50
And so then we were like, wait a minute.
7:51
I thought we said that this was just for the analytics plane.
7:54
Now, the operations plane is getting back involved all over
7:58
again. And then the second thing that
8:01
happened is we had this really nice set of tools that we were
8:05
using right like different things like the air flows or
8:09
like Luigi or whatever. We had these good orchestration
8:11
tools and we were kind of factory pattern creating these
8:14
data products, right in the data space in the ingestion space,
8:17
right? And we had Source, the line data
8:18
products that made a lot of sense and Aggregates where you
8:22
like, take two different sorts of lines and put them together
8:25
and create a data product from that.
8:27
So, like, in the analytic space, But then, when you move to the
8:30
ml side, you were like, oh, whoa, everything's different
8:33
now, like I'm not using air flow anymore.
8:36
I'm now using. Now I need to consider things
8:39
like Cube flow that are like hyper centralized, right?
8:42
Everything's internal 2q flow. For example, right?
8:44
You know, you can do your experimentation in the space,
8:47
you can build your pipelines in there, like everything and so
8:50
this is decentralized, right? So so now that some of the tools
8:56
kind of forced us into this centralized kind of approach
8:58
again Within Am L op space so we had to really kind of think
9:03
through this and I was like okay it's not that we can't create a
9:07
machine learning data product it's that we don't really have a
9:09
name for it. It's like a story that hasn't
9:11
been told and we need to now evolved the data mesh kind of
9:18
concept the framework so that we can accommodate for that kind of
9:21
thing and shamanic actually did a lot of this in her latest
9:24
version of the book. Right?
9:25
Because early on I was just we were reading the rough cuts and
9:27
it wasn't that we hadn't been included yet.
9:30
And so, there's this idea of like, a multi playing consumer
9:33
aligned data product, and I think that's where these machine
9:36
learning products kind of live. And then, I also think there's
9:41
an element of like feature extraction or feature
9:44
engineering, which probably comes into the space of maybe
9:48
like a sidecar of the ml data product.
9:51
So now we're inventing, or we're kind of coming up with figuring
9:55
out how the framework works with the machine learning models.
9:59
And, in fact, It's still the same principles, right?
10:02
You still want a machine learning model to be
10:04
trustworthy, right? You want to measure its drift?
10:07
And make sure you can still use it and it still make sense and
10:10
like, you still want it to be discoverable?
10:12
Like a lot of the clients that we work with.
10:14
The machine learning Engineers were like, hey, I'd like to
10:16
expose my models to the rest of the company so that they can
10:20
either transfer knowledge or they can reuse the same
10:23
algorithms that we use. We don't have a way to do that.
10:27
Yeah. And so then then we I just
10:30
started picking all the pieces together and putting all the
10:32
different tools and just applying the dot cyst.
10:35
Right? The discoverability
10:36
addressability, trustworthy security to building a data
10:41
product. So that's kind of I had to
10:43
really use an example like a real example to figure out how
10:47
to do it. Like, I just couldn't do it
10:48
Theory. Very interesting.
10:50
So I have them. Yes.
10:51
I was saying before. The course started.
10:54
I've taken a lot of interested. To mesh and it's something that
10:58
origami really keen on because Energy Service Company.
11:02
Sorry it energy. SAS platform.
11:04
Essentially, it's a big complicated space.
11:07
Loads of stuff going on the Rangers and Designs, the
11:09
solution, really to start making sense of it.
11:12
This something about data mesh, though.
11:14
So, but I was previously a data scientist, there's something
11:16
about it that I've struggled with, and I think it's kind of
11:20
wraps up and what you've just said.
11:22
But the big question for me was always that exploratory part of
11:27
a new modeling project. So that bit where you don't
11:31
actually know what daily you need it.
11:32
You don't know what's going to be useful.
11:34
Where's the signal, right? Where's the edge now?
11:38
The bit that kind of stumped me a little bit with betameche was
11:41
that if everything's in a nicely set up date a mesh, we've got
11:45
these lovely date products and things easy to use.
11:48
What if what I need is an in the data product or if it's been
11:52
Kind of abstracted away all the interfaces protecting me from it
11:56
is the like a loss there is that if you come across that as
12:00
anyone sort of talked about that before, is that me as an
12:03
unnecessary concern? Yeah, that's a common with
12:06
right. So, imagine that you have a
12:08
source of line data product. That's just, you know, it's
12:11
taking a bunch of events and it's, you know, streaming it
12:14
into like, a Delta Delta Lake, right?
12:17
Something like that. And it just so happens that the
12:21
way that they're curating, The set on the output Port is really
12:23
made for a consumer. That's like a business analyst.
12:27
And so then a bunch of information gets redacted like
12:30
from the event itself, right? And that information actually is
12:33
very useful for like a data scientist.
12:34
Who needs to see all of the the data and needs to explore the
12:37
data and needs to be like, okay, is this going to be a feature?
12:40
Does this make sense like what is this about?
12:43
And you intentionally exclude that for a business analyst
12:47
because they just confuses them, right?
12:50
You know, they're like I don't know why this date is You're
12:52
like, why is it 200? You know, 200 columns wide when
12:56
I just need like 15 of these. And so that's where like data as
13:00
a product comes in. And I would say that one of the
13:03
output ports for this source of line data product, that process
13:05
is events is made for the business analyst, right?
13:09
But there is a use case for a well-curated.
13:13
Let's say raw said that's that's catalogued nicely in the
13:18
catalog, right? It's still the quality still
13:20
measured on this set, so it's Not like a raw asset that's
13:24
made, especially for a data scientist.
13:27
And so, a lot of times the data products, we build will
13:30
bifurcate in two different ways, one for more of like, a business
13:33
user. And then another 4, which will
13:35
be like the more like, like all of the data write a more
13:38
complete set for the data scientist.
13:41
And so you just build that in as an output port for your source
13:44
of line data products. So okay, so yeah, that makes
13:46
sense that's kind of that sort of, that does feel a little bit
13:50
of cheating, but Yes, I get that.
13:54
That's kind of where my head went, it was like, yeah, you've
13:56
got nice curated data products, but then there's also a bit of a
13:58
back door into the raw, but that's kind of where my head win
14:02
the next thing was then. Okay, so that's fine.
14:05
We build a model was really valuable and it's got these
14:08
features. Now, there's talk in the book
14:10
about pushing ownership Upstream, do I like do you let
14:15
your data scientists push? Loads of feature requests on
14:19
that date of product after the source line domain, or, or do
14:23
you keep that centralization in the team that's doing the data
14:25
Sciences like bit? Like if I say, I need one of
14:28
these, these raw things processed actually and I need it
14:31
in the day of product and it can you just handle that for me,
14:34
that ownership bit kind of stumped me a little bit as well.
14:37
Like can I just put loads of requests on people Upstream from
14:40
me to kind of babysit my data science needs, right?
14:45
I mean you are a consumer, right?
14:47
I mean it End of the day, you're a stakeholder even though your
14:50
internal as a data scientist, you're still the stakeholder.
14:53
You're just as important as the business analyst.
14:55
So to me, the way it goes, is the there's the domain.
14:59
Now, maybe there's multiple teams working on multiple data
15:02
products that live inside the domain, but those teams
15:04
typically don't work in other domains, right?
15:07
So so to me, I would say that. Yeah, if you need the data in a
15:11
specific way, you can request a specialized output data port.
15:16
And the other thing I was also thinking inking that data
15:18
scientists like about the, the data mesh is that they can count
15:23
on the output ports all being in the same format.
15:26
Right? It always, you know, maybe if
15:28
it's you, uh, you know, you would always accessed through a
15:31
unified query engine. And underneath that is like a
15:33
layer of the Delta, like, right? Like through a presto or at Reno
15:37
or whatever, and you can always just be like, okay, it's
15:39
basically like accessing the table, but it's like distributed
15:42
but, um, yeah, it is okay to put request in and I've even taken
15:46
it one step further in, The in the talk and I was like the ml
15:52
Ops Team and the data science team.
15:54
They, you know, I kind of challenge this notion that they
15:57
are centralized. Why do they have to be
16:00
centralized as well, right? Like, I mean, with the data
16:02
teams, we broke them up into domains and different data
16:04
products. Like, we could also have
16:07
cross-functional teams where we have machine learning Engineers,
16:10
or data scientists that belong to like a domain or a specific
16:13
type of data product, right? So so I kind of also Challenge
16:18
the centralization in the ml off space as well.
16:20
I think that probably come from just the reality of skill sets
16:24
in the market, like I could get enough.
16:26
People might be a challenge. And can you tax these people
16:30
with being cross the main? But again, that's cheating a bit
16:33
like that. Isn't your kind of not doing it
16:35
right then by the book. Yeah.
16:39
Okay, that's really cool. The one thing I wanted to also
16:41
you mentioned Delta like Presto. So one thing that I think is
16:47
actually King in the literature. And a lot of the discussions out
16:50
there is technology and some of the implementations and like,
16:55
what? Okay, what, what, how do I do
16:56
it? What's the tech?
16:57
And there's the stuff from zalando, which is quite cool and
17:01
they've done all stuff some but like I really, I really want to
17:04
see behind the kimono like I want to know what's under there.
17:07
What you doing? Have you seen the blogs from?
17:12
So that's one question like interested to hear your thoughts
17:14
on Tech and then a follow-up around.
17:16
Have you seen the bogs? Bye.
17:17
Is it python string? Hold from Microsoft about dating
17:21
missions. Apologies specifically no I
17:23
haven't have to check those out. Okay.
17:25
Yeah it's one of the few articles out there that's got a
17:28
bit architectural and he actually kind of talked about
17:32
the six different topologies for datum, it like pure Federated
17:36
being kind of by the book and then this second one that feels
17:40
like a more natural fit where actually the infrastructure
17:43
there's that there is a bit of centralization to support it so
17:45
I wanted to get your thoughts on that if you've not read it maybe
17:47
that's Yeah, no, nothing, right, short summary techy stuff.
17:52
What how'd you do it? What's the kind of approach?
17:55
What, what would you recommend? Yeah.
17:56
So, the way I approach that is, I do or what I call like, the,
18:00
the techie stuff for the tools. It's the last thing that I do,
18:04
right? So, I always start with a bucket
18:07
of really high value. Use cases, right?
18:09
Rather. It's machine.
18:09
Learning, use, cases, or whatever.
18:11
Let's assume it is for this for our purposes.
18:14
I analyze those and say, okay, I extract The domains right domain
18:19
driven from those use cases. And you know, you do a lot of
18:22
there's a lot of different like kind of exercise.
18:24
We go through to kind of break up the use cases and how they're
18:26
out of analyze them. And then from there, I'm like,
18:29
okay let's break these up into individual data products that
18:32
are meaningful like meaningful chunks and then show how all of
18:35
these data products feed into each other, right?
18:37
To Source aligned systems into an aggregate, the aggregate,
18:40
maybe feeds an ml like right future store somewhere.
18:43
It's an output Port, it gets ingested, that kind of thing.
18:46
And then once I've done all That then I go through each of the
18:49
individual data products and start mapping capabilities,
18:52
right? So like the capability for like
18:55
schema management data Quality Storage, right?
18:59
And just abstractly and then finally I kind of take a very
19:04
practical approach to recommend and Technologies, right?
19:08
Because for a couple of reasons. The first reason is right now,
19:12
we're kind of in this, like, I don't know, Cambrian explosion,
19:16
if you will of data. Science and math ml Ops
19:19
products. Do you know I'm saying?
19:20
Like, there's just like there's so many and keep.
19:24
Uh, yeah. I'll keep up.
19:26
There's a new one every day and they're open source their
19:28
software as a service, you know, all sorts of stuff in like there
19:32
through the cloud providers. And, and so when I'm like from
19:36
the Consulting hat perspective, I say, okay, you know what?
19:40
Like, then, I map tools that map, the capabilities that I
19:43
need for my data products. Right?
19:46
And I say, okay he's are they already using this tool, right?
19:51
Whatever it maybe right maybe it's stage maker for example.
19:54
And and is it fit for purpose? Does it have all the
19:57
capabilities that I need for these?
19:59
Use cases that I've analyzed. If the answer is yes and they're
20:02
already using it. Stay with it, right?
20:06
You're right. So that's that's kind of 111
20:08
way. I'll kind of make those
20:09
decisions. The other one is When selecting
20:12
tools. I also look at like that Like a
20:16
lot of, like, what we used to kind of do is say, okay, this
20:19
tool is the, it has the most features that I could possibly
20:23
ever need. Therefore, this tool is going to
20:26
be the one that I'm going to pick, and it's also the most
20:28
popular one, and Gartner says that I have to use it.
20:31
I've now since kind of moved away from that and realize that
20:34
every company situation is just going to be a little bit
20:37
different. So some companies we go in and
20:39
we talk to and they maybe don't have the, the, the people to
20:45
manage an The structure, right? So which means that there's a
20:48
lot of cool custom stuff. Maybe they can't do, but at the
20:51
end of the day, if they want anything at all, that's that's
20:54
going to work. They have to go with a like an
20:56
off-the-shelf or software-as-a-service kind of
20:58
solution. So then you kind of have to be
21:00
like all right they're going to do they're going to be more of
21:02
like a by so you construct the architecture that way and you
21:05
can do that in Azure AWS and all of those things more or less
21:09
work, okay? You know they will be fit for
21:12
the purpose and then on the other side Sometimes some of
21:15
Other clients we work with will have, you know, ml Ops teams.
21:20
They'll have like a big team of data Engineers, they'll have
21:23
backing Engineers front end engineers, and they're more the
21:26
ones that are saying, you know, we're going to adopt this open
21:28
source software and we're going to start contributing, right?
21:31
Like, I'm going to start, contributing to this LinkedIn,
21:33
data LinkedIn data Hub, right? Right.
21:35
That kind of thing. And so, so I find those for me,
21:39
there's the most fun, right? But but at the same time like
21:43
that they, you know, a company. Ooh, that didn't have that skill
21:46
set, could not possibly, you know, maintain a kubernetes
21:51
cluster, and then on top of the kubernetes cluster, a deployment
21:54
of cube flow and then also like a modification to cube flow to
21:58
use ml flow instead of tensorflow, right?
22:00
So they can't do all that cool kind of custom stuff.
22:04
So I'm very like and I think the reason the book is like that
22:08
like it doesn't talk about the tech as much is because it is
22:11
intentionally attempting to be technology agnostic.
22:16
Right? Because we know that data mesh,
22:19
is the data mesh flavor is going to be different across every
22:22
different organization. And I think that it's, we're
22:25
trying to avoid, is more of like a framework and less of, like,
22:28
you know, adopt this tool, right?
22:30
Like, because there's been discussions around people
22:32
thinking like, oh, can we turn data mesh into a product and so,
22:37
and it's just, it doesn't quite it just doesn't the way it's
22:41
written and the way we have applied it.
22:43
I just I don't know if that's possible.
22:45
Honestly, I don't know if you could just be like a, let me buy
22:47
a daytime at. Yeah, it seems really messy and
22:51
it would be very hard to do just like all of a sudden this whole
22:57
Legacy system that you've got. We're just gonna feed it through
23:01
the shredder and pop out a data mesh on the other side.
23:05
Yeah, I don't know if that, yeah, it could work.
23:07
So I'm wondering as far as like what the machine learning
23:12
Architects are thinking and when Are interfacing with someone
23:18
that is trying to build this data.
23:22
Mesh culture in their company, what should they be asking?
23:26
What are the questions that they should be asking the others when
23:31
they're trying to figure out? Like, how are we going to
23:34
integrate with this self-serve data platform?
23:37
Yes. Oh so now we're kind of getting
23:39
in so there's there's data mesh, that's like socio which is this,
23:43
you know, social side of it and there's the technology side,
23:45
right? The Techno side right?
23:46
And now we're kind of getting into the social side, right?
23:49
And it's interesting because my advice to someone who's working
23:55
in the ml Ops. And if you're separate from like
23:57
the data's like the, what's happening in the this typical
23:59
data ingestion data leg space is to like intentionally get
24:04
involved in all the conversations about data mesh,
24:08
right? Because I think a lot of like I
24:10
think it's really important, especially right now because
24:12
there's not a lot of I mean, there's I'm president but
24:16
there's not a lot of precedent yet.
24:17
And I would say just try to be as much involved in those
24:22
conversations as they happen and be thinking about like how
24:26
because it's all about interoperability, right?
24:28
And at the end of the day you need to be thinking about like
24:31
how when I build, you know, whatever whatever my tech stack
24:35
is, how am I going to integrate into the output ports of all
24:40
these different data products they're building?
24:42
And you need to make sure like we talked about earlier with the
24:45
If acknowledge you for the data scientist that every data
24:48
product that you want or need, that has that right before they
24:52
build it, right? Just make sure it's built in
24:55
with the design and that's that's what we've done.
24:57
I'm like, okay, well let's ask data science.
24:59
Let's make sure they're in the room.
25:02
As we collaboratively build this data product, at least one
25:07
right? Make sure they're part of it.
25:10
And I would even say the same thing for the business folks, as
25:12
well. I think a lot of data products
25:14
get built without the - like the business folks involved.
25:18
And so, I think they need to be in the room to which I know is
25:23
like, oh no, another meeting, I have to go to.
25:25
But this is a really important one that could affect
25:28
Downstream, you know how you access the data.
25:31
Okay. So I've got a bunch of things I
25:32
dropped off here and my head off and go.
25:35
So for like just saying on the theme of envelopes in an
25:39
engineer's, the I'm a big fan of this idea of like, Building that
25:46
platform that then enables the kind of from the terminal, a
25:50
generalized Specialists these like, basically empowers like a
25:54
really talented software engineer, that's just never done
25:56
a male to be able to spin up an interlocked pipeline, right?
25:59
That's the dream. You so that's the dream.
26:01
So my domain is populated with software Engineers right there.
26:04
All Java guys, or girls, they, they realize they need a machine
26:08
learning model. There's an outlet box upon them,
26:10
do the job. They just need to stick it to
26:12
production and they don't want to have to go and learn in a
26:14
lots, right? So that that's Gold.
26:16
That is where we want to get. Where do you start with that
26:20
kind of platform team? Because that's a big undertaking
26:22
especially is you're trying to build a solid foundation on the
26:26
ml Ops wider Community. That's moving and shifting all
26:29
the time, right? We can't even decide what to
26:31
call the stages. Let alone build a consistent
26:34
framework. Yeah, thing advice on that.
26:36
How to get going there? Yeah, so I just literally said,
26:42
involve yourself in all the conversations that being said, I
26:47
have in a couple of places that we've had this, where we've had
26:50
the the data message implementation completed or
26:53
closed, right. Pretty good.
26:55
And then, they've started this journey down.
27:00
I don't like I say, Auto ml. But when I say that, I think
27:03
that means that in that term actually means a bunch of
27:05
different things I discovered, right?
27:07
Like, Google will tell you. It's one thing, it's like high
27:09
for parameter tuning, and then you might go over to like the
27:12
AWS website and it's like, oh no, it's just like this API
27:15
service that happens to be machine learning.
27:17
So, but away like that, I as a data engineer, can go to a place
27:25
and I can say, I want to use these sources tombs.
27:28
I want to leverage, maybe these algorithms, or this type of
27:33
Pipeline and the outputs, I need like an API and a table in the
27:38
database somewhere, right? And I click and then it
27:41
Provisions my infrastructure, and then I'm off to the races,
27:44
right? We're just now getting to kind
27:48
of building those kinds of tools and it first begins with a level
27:53
of maturity. In the data mesh like, like five
27:56
or six, maybe source of line systems that you can use in ml,
28:00
right? So once you kind of act finally
28:02
have that and it's self-service just like spin up those data
28:05
products. Then you can start thinking.
28:08
Like, okay let me build my very first machine learning
28:13
multi-plane data product, right? And you Build it once and you're
28:17
like, okay these are all the elements in the components and
28:20
then you say, okay so this is a, maybe you build a couple, right?
28:24
You see these are common blueprints or these are common
28:27
patterns that I see in ml and I want to like, you probably turn
28:32
them into a template and then comes like a command line
28:35
interface that you can spin up automatically provision this
28:38
infrastructure. And, I mean, like, literally
28:41
like, spin up the cluster if you need one like a template for
28:46
Pipeline. And then, that gets committed to
28:48
a repository and you have your repository and then you can
28:50
start, you know, kind of coding and committing.
28:54
And, and so just that basic framework of the infrastructure
28:57
and then once you have those, once you do that, once with a
29:00
command line, interface to spin up the template and provision
29:02
the infrastructure, then you can start building more and more and
29:07
more. And then once you're done with
29:08
the command line, interface, why stop there, what you could get
29:12
some front-end developers and some back-end developers
29:14
involved and maybe set up. Some backstage situation where
29:18
you create like a graphical user interface, right where the whole
29:22
company can access this. But the build-up to that has to
29:26
be slow and increment like the build-up, initially is slow, it
29:29
kind of works like this, right? Kind of like a like a hyperbolic
29:32
curve. We're like At first, it's very
29:35
slow when you do the first couple and then suddenly before,
29:37
you know, it you've identified all the common patterns and your
29:40
you just kind of like multiplies really fast, but but you've got
29:45
to start with that first. On we typically call that like
29:47
the thin slice where it's like I build everything from top to
29:51
bottom like, starting with my governance layer and then like
29:54
my data product layer and then like my platform layer and you
29:58
can do the same thing that you do in date.
30:00
You can do that in ml op space as well but small, this is
30:04
Skyler. I lead machine learning and
30:06
health rhythms if you want to stay on top of everything
30:09
happening in ml. Ops subscribe to this podcast.
30:12
Just leading into that land. So yeah.
30:14
Okay. That doesn't make sense.
30:15
I sense, I just think that for someone that's tackling this or
30:19
starting out, and you can give them the green light to go.
30:23
What's the where the quick wins? Like, how do I prove value?
30:27
The fastest, like the risk and gain confidence and Trust like
30:30
because this is a big cultural shift for folks.
30:33
So you need to kind of there's a bit of a heart from my testing
30:35
people like me, come on go. Yeah.
30:36
Well it's so the quick wins live in a couple of spaces.
30:39
Right. I mean the first is you know you
30:41
put together a list of like you know we kind of have like this
30:44
thing called we call a few. Perspective where we tell
30:47
people, you know, this is three months, six months, three years,
30:51
whatever, right, or twice? Three months, 12 months, three
30:53
years. And so, you get everybody very
30:56
excited about what they can someday do with the mesh, right?
31:00
Like all of the things that they want their hopes and dreams,
31:02
right? And and, and so, then you start
31:06
from there. And when you start to kind of
31:08
figure out, like, okay, well, what are all these use cases?
31:10
That like realistically, what are our use cases?
31:13
And I've worked with certain With certain companies and we've
31:17
actually ranked the use cases by a set of like waited criteria.
31:21
So this is really intense. You don't have to do it this
31:23
way. You might say, oh I have five
31:25
really good use cases. I'm going to start with but like
31:27
you want a way to identify High business value use cases and you
31:32
start with those but but and the reason we waited it because we
31:37
were like, okay how easy is it to solve?
31:40
For this use case, right? So if it's a super complex use
31:42
case that should not be the first one that you do, right?
31:45
Want to kind of figure out from all of your use cases.
31:48
How do I rank and sort these so that I'm hitting the low hanging
31:52
fruit? That's achievable within like a
31:54
three like a 1/2 2/4, three to the right three to six-month
31:57
time period. That way you can show value fast
32:02
and then because it's not complex you can build it right?
32:05
Like you don't have to cut Corners because sometimes when
32:07
it's too complex then you're going to have to.
32:10
Then you're going to be like, oh wow, this is taken me, six
32:12
seven, eight months and then I'm going to To have to cut Corners.
32:16
So so I think there's a little bit interesting of a an idea
32:22
there, which is like, how do you or what metrics are you
32:27
monitoring right? When you're looking at this,
32:31
right? Like, as you're bringing it
32:33
together and you don't know what the future, you're kind of like
32:36
carving out the future. What then do you start to look
32:40
at? What are the the needles that
32:41
you want to move? Yeah, that's a good one.
32:44
I think so, I think She wants actually she dresses this in her
32:47
book, I don't know, maybe like 10:30, but it's called Fitness
32:51
functions, right? So you want, you need to agree
32:54
ahead of time more or less. What is going to be your
32:58
measures of success from the data mesh in your organization?
33:03
And everybody has a different kind of Fitness function, right?
33:06
Like, some are like centered around the developer experience
33:11
so like very technical organizations will just be like
33:13
if you can make the developer experience Ian's easier and more
33:16
efficient than, you know, I'm happy, right?
33:19
Like if you can improve efficiency by 30 or 40 percent
33:23
so that we're not building snowflake architectures over and
33:25
over again, right? Because with a centralized
33:28
approach, you do kind of get into that like you're building
33:31
on top of this Central monolith and there's a bunch of system
33:34
bottlenecks, and you've got to kind of work around that or like
33:38
you have every you have all of your pipelines deployed to like
33:40
one instance of Q flow and then you have all you know that again
33:43
you kind of have the same problem.
33:46
So so you have to think about like what kind of you're
33:49
different. Like measurements of criteria on
33:51
the other hand, maybe you have organization that's not like is
33:55
developer focused or like ml focused and that that it might
33:59
be more about business value. So then you have to say, Okay,
34:02
like these data products that I delivered like what is the
34:07
utilization of the, the output ports of these data products by
34:10
the business, users, by the customers, right?
34:13
And, and so, you can measure that through survey You can
34:16
measure that through like actually just seeing you
34:18
accesses in how frequently. And so there's different
34:22
depending on the organization. There's different types of
34:24
Fitness functions. Maybe you do both right?
34:26
Yeah, and I guess each, of course, each instance is going
34:33
to be different, but as you're trying to set this up, I imagine
34:37
that you're not going to be able to set it all up in one day.
34:41
So how do you have? Do you like, how do you break up
34:46
different metrics to know that you're making progress on like
34:50
this bigger Vision, right? So if it is, for example, I
34:53
really like the one where it's the developer experience because
34:57
that I think a lot of people can relate to and A lot of people
34:59
have that dream but then if it's a if it's a 36 12-month project,
35:08
what are you looking at to be like, all right we're making
35:11
progress and it's better than the last iteration that we had.
35:16
Like you're saying it does that? Just mean that we're not
35:18
rebuilding these snowflake type architectures but before you're
35:23
at that end State, how do you know that?
35:25
That's where you're going to get to, right?
35:29
Yeah, this is like a, like, a philosophical question away,
35:32
right? Like I'm kind of thinking
35:32
through this. Like, how do I like, how do I,
35:35
how do I know what? I don't know.
35:37
So yeah. It's well, I guess yeah.
35:41
So I guess only the way that I'm thinking about it is like you
35:46
really gotta just be bought in on it, right?
35:49
And then know that there's going to be a payoff at some point.
35:55
But is there things that you can see beforehand, that can be like
35:58
good That there's going to be that passionately.
36:01
Definitely, yeah, I got you. Okay, I understand now.
36:03
Yeah, there are like when so at the at the end you kind of we
36:08
typically will do kind of a strategy piece first like to
36:11
kind of just like set up like this is the day to manage.
36:15
This is kind of how it will more or less be organized.
36:18
Like, you know, you're going to need a governance board, right?
36:20
This is what we'll do and then we kind of do like a road map,
36:23
right? Like we set up Milestones, right
36:25
3, 6, 9 12, whatever, right? Per quarter.
36:29
And so, so you at the same time, though, you have to acknowledge
36:35
that while you're building this data match, your probably
36:37
needing to keep business as usual going to write.
36:41
And so, you're slowly taking pieces.
36:44
You're slowly decomposing. Just like you would do with
36:47
them, like, a monolith into microservices.
36:49
Well, you actually probably have like a giant data monolith that
36:52
you're also decomposing as well. And so, as you do that, you are
36:57
going to have to slowly kind. Take pieces out of business as
37:00
usual and then convert them into various data products and and
37:07
then, you know, kind of like catalog them, that kind of
37:10
thing. I'm just thinking like what,
37:13
what is a good way to measure that?
37:15
I mean a lot of times, you know, people will send out surveys
37:18
that kind of thing. Another, another thing that
37:21
we've done kind of internally is you kind of you definitely you
37:24
evangelize the data mesh, right? So like every every milestone A
37:29
few Sprints. If you have a showcase you make
37:31
it public. Look, everyone.
37:33
This is what we're doing. Now.
37:34
We can now instead of hunting around in the redshift instance,
37:38
you can just go to the data catalog and search by keywords
37:41
for what you want and there's descriptions for every field and
37:44
there's summaries for tables, right?
37:46
Even just simple things like that.
37:49
Just will give you a high business value.
37:51
Yeah and one of the ways you know people are using it though
37:54
is You take this idea of discoverability, right?
37:58
And one of the central things is you have a data catalog.
38:01
And so you can hook in analytics like Google analytics for
38:05
example, or you know, I'm saying like different way like to
38:07
measure web traffic. You can actually kind of look at
38:10
analytics on like what people are searching for what data
38:14
sets, they're going to, how often are they using the tool
38:17
and you can use that to measure in a way of an element of
38:20
success because of people are using it and they're accessing
38:24
the tool? Are you know like it's being
38:27
highly utilized, you know that now, right?
38:30
So that's one way, from a central location, you can kind
38:33
of measure how the organization is adopting.
38:35
It Is by analyzing the web analytics from your catalog year
38:40
that that does come to me. That would be one of the major
38:43
major benefits of like looking at.
38:45
There's an article from Spotify and at all, they do called
38:49
Mexican, that gives them. Yeah.
38:52
And it's like, I think is internal only but it talks about
38:54
how they can Can see who using what they are sets, like how
38:57
they're getting used. And I think that is again, a
39:00
dream state that would love to get to.
39:03
That's yeah, yeah, that's it. Yeah, that's it.
39:06
Yeah, nodes of other questions. So we talked about starting
39:09
small quick wins. The a lot of the chat and like
39:15
discussion always leads me to think though, is this reserved
39:18
for big organizations? He's there because there's lots
39:23
of roles that need to be involved as well as Get
39:25
alongside BAU, we talked about boards and stuff.
39:28
Like, is there a lower limit where by it's not achievable, or
39:33
is there a lower limit by, which it's just not worth doing
39:36
actually? Yeah, there is I tell people
39:41
this doing when we got it will go in and kind of like people be
39:44
like, okay can you help me with my data space that kind of thing
39:46
we go in and we don't just come in and be like, okay, you're
39:49
going to use data mesh. I mean, sometimes like it's
39:52
pretty obvious and it makes sense you like yeah, of course.
39:55
But you have to go in and you have to kind of do this this
39:59
final cost benefit, or you have to think again, like, even when
40:02
we broke up the monolith into microservices, we knew that
40:05
there would be a sacrifice of the Simplicity of a All right,
40:10
it to single deploy, its Riley. Like I know where everything is.
40:13
It's one code repository, right? Like it's one a pra.
40:16
It's simple. And that's and that's fine.
40:19
But it's so. So until you need to scale,
40:24
right? Right.
40:26
If you tell, you need to scale out, I'm not sure that it makes
40:30
sense because remember that you are sacrificing kind of what is
40:34
pretty simple. Like a like, a monolithic
40:36
structure for what has now? A pretty complex network of like
40:41
nodes and edges and and you have to enforce like interoperability
40:46
between each of these and individual pipelines that build
40:49
and deploy. So there is definitely like if
40:53
you're if you're like you're like I like a classic example is
40:56
like a small start-up that needs to be first to Market really
40:59
fast and they have a pretty simple product, right?
41:03
There is no reason probably for them to use data mesh and till
41:08
They turn into a high-growth startup and we've seen several
41:12
clients like this, right? Where they'll be like, okay,
41:14
wait? Now my monolith is, is reaching
41:17
all these bottlenecks, right? Like I'm streaming data into my
41:21
data warehouse and it doesn't make sense and, you know, it's
41:23
like high transaction and I reason I rights and then it's
41:26
like, okay, now we need to step back and we actually do need to
41:29
look at data mesh, seriously and break up that monolith into
41:33
smaller more manageable chunks, and then then organically it
41:38
actually solves. As a lot of the system
41:39
bottlenecks which is interesting.
41:41
But but yet, no, I don't recommend database for every
41:46
single situation. By any means nothing.
41:48
I guess. So because my head was like,
41:50
initially, is there a scale would be the limiting factor?
41:53
You need a lot of people and like what's the benefit?
41:56
Almost like that Conway's law type stuff.
41:59
So other than scale and there are other like indicators for
42:03
actually that's not, it isn't the best approach.
42:06
There are other things that you would say that In the culturally
42:10
technology choices things that we would say.
42:13
I thought you probably shouldn't do this just yet or at all.
42:16
So I struggle with this one because I'm very like technical
42:19
in my Approach, right? And this the idea of data mesh
42:24
is that. It's also like a social aspect,
42:26
right? So like you said, it's Conway's
42:27
law, right? If you have a highly centralized
42:29
team, you're going to have a style these centralized system,
42:31
right? And then that comes with its
42:33
with its trade-offs. But then there's this weird
42:36
aspect also of just culture. So if you're trying to build a
42:41
data mesh without also, applying the operating model of a data
42:46
mesh type organization, It's not like you won't get the benefits.
42:52
So if you're just trying to do a technical and Technical non
42:55
socio implementation of data mesh.
42:58
Yeah, you probably get the benefits, right?
42:59
Because at the end of the day, it advocates for Best Practices,
43:03
it advocates, for measuring the quality of your models.
43:05
It advocates for, you know, versioning right?
43:08
Like DVC right. It advocates for all these good
43:11
things that you should be doing anyway.
43:13
So so you get the benefit of that but you don't get nearly as
43:18
benefit is if you also. Apply the operating model and in
43:22
order to do that you have to have the Mandate from the top
43:25
down. Otherwise it just won't work,
43:27
right? You can't change.
43:29
I mean, you can't change the operating model probably like
43:31
like as like like an engineering manager.
43:34
I like you're going to really struggle with that, but you can
43:36
control how you build your like your data pipelines and your
43:40
data products, and you can control like adding quality
43:43
metrics and you know, like Loosely coupling 11 data product
43:50
from Mother. So, there are some benefits that
43:52
you get, but at the end of the day, I'm there, they're
43:55
absolutely does need to be if you want to do the full data
43:57
mesh. You absolutely have to have the
43:59
cultural by in and the the desire I guess and the mandate
44:05
to make some organizational change.
44:07
In addition to changing the way you look at architecture.
44:11
And when you were talking earlier about how to like,
44:18
basically win to make the jump, What scale and you're small and
44:22
starts crappy start up and then you go to this gigantic Behemoth
44:28
and you want to make that jump at some point.
44:32
Are there things that we can do in the beginning of our life
44:35
cycle at that small and Scrappy startup to set the foundation so
44:40
that when we do change it's a much easier change.
44:44
Yeah. Yeah definitely for sure.
44:47
And I think it just goes back to kind of what I was talking
44:49
about, right? Right?
44:50
If you just do like, if you just use some like, we call them
44:54
sensible defaults on, like how you should handle data and and
44:59
that's not data measure specific, right?
45:01
I mean, data mesh advocates for all of those things for sure,
45:03
because it has the dad says, it's like, discoverable
45:05
addressable trustworthy, so on and so forth.
45:08
But if you're good, let's say like custodians of the data,
45:14
right? Like, I think a lot of times
45:15
when I and this is, I've myself included, when I first started
45:18
doing this, like, you know, the data, Dentist were developing
45:21
everything in like jupyter notebooks and then like they
45:25
would run ad hoc runs of the you know of the model training and
45:29
it wouldn't be automated or anything like that.
45:32
And so like you know in this case for this context, right?
45:37
If you're just practicing good CD for ML Even if you're a
45:42
highly centralized like environment, and you want to
45:46
make the transition to data mesh, it's just that much easier
45:48
because all of those things are still applicable in the data
45:51
mesh. And a lot of times people aren't
45:54
even doing that when they start their journey into the data mesh
45:57
and it's like, oh now not only that.
45:59
But you also need to do this right on.
46:01
Like we're layering in like, oh and now you need to add, you
46:05
know, some metric system by which you can measure Driftwood
46:08
models, right? And can omit those It's like a
46:11
Prometheus are Griffon or whatever, right?
46:14
And so so if you start doing things right from the start and
46:19
using good data practices and good ml Ops ml, like practices
46:25
from the beginning which includes like testing and all of
46:29
those things. If you start that from the
46:31
beginning, then your transition to data mesh from a central kind
46:35
of system will be a lot easier. Yeah because you can't do it
46:41
without that. I imagine it's just like not
46:44
going to happen you wouldn't. Yeah you wouldn't be able to
46:47
kind of like you wouldn't be able to do the the like
46:50
self-describing interoperable secure trustworthy, addressable
46:53
discoverable, right. All of those things even though
46:56
they're like kind of like descriptions or capabilities,
46:58
those all tied to tools that you have to learn how to use so far
47:02
food long. I've been it's been quite a lot
47:05
like because it's big and scary, right?
47:07
It's a big scary thing. It's like it.
47:09
Feels like a lot of work, it's still funny, its feet.
47:12
No one's like no one's written the blueprint of how to do it,
47:15
really? It's kind of hear some guiding
47:16
principles. Let's discover this together.
47:18
Even the book says, you've got to be an early adopter.
47:21
So to put a big positive spin, just from your experience.
47:25
I'd love to hear like, are they unforeseen benefits that
47:29
surprised like you or the team from a well implemented data
47:33
mesh. Have you all of you got like a
47:34
call sort of War Story. Use case about like all they did
47:38
it. And then For a data scientist
47:40
thinking more about analytics and data science professionals.
47:43
Were they like call outcomes that no one had seen before?
47:46
That actually like would be a good reason to get excited about
47:50
this stuff and put the effort in?
47:51
Um, I would say, I mean, some some, some of the various things
47:54
that I've kind of like, we've kind of encountered is that
47:59
participate like specifically. And I kind of touched on this a
48:02
little bit, but like as a data scientist, when I've worked with
48:05
them before in the past, like the way we used to work was just
48:08
like, hey, Can you give me a snapshot of this database?
48:12
Right? And so, then the data scientists
48:14
would be like, hey, can you sit down with me and explained to
48:16
you? What is this table?
48:18
How do I do these queries, like War?
48:20
Where do I get this data from you?
48:22
Well, you got to join it from over here, and right?
48:24
And so, so when you have these data products, at the very
48:28
least, you get a level of like almost like governance and
48:31
standardization. So this is what surprises
48:33
people, I think because there is this like Federated governance
48:36
piece to this, where you say, Hey, look, You've got to measure
48:40
data quality, but you only have to do it in these four areas and
48:43
how you measure it. That's up to you just make sure
48:45
you measure it right? Make sure it's there.
48:47
And so also, by the way, all of these things need to be
48:50
interoperable again. How the team decides, how they
48:54
want to make it interoperable completely up to them, but at a
48:56
governance level, at least you need to be interoperable.
49:01
And so the governance layer actually is really important.
49:05
And then when that like kind of like I call it governance light
49:09
Federated governance where you kind of right?
49:12
Will you just give meaningful guidelines when you apply that?
49:16
There are a lot of like hidden benefits of like not being to
49:21
standard, but being standard enough that as a, like someone
49:25
who's in the ml off in, like the ml like side or a data
49:29
scientist, they can almost always count on the data being
49:32
presented in a certain way and because the quality is being
49:36
measured of the data, you can actually trust the data.
49:39
Data. Because I remember a lot of
49:41
times when I've worked in these coasters teams with the ml
49:44
engineers and the data scientists and stuff, we would
49:47
build a model, right? And training test and that kind
49:51
of thing can be like okay yeah, this looks like it's working and
49:53
then we would productionize it and it the performance on, it
49:57
wasn't great and what we didn't know was like, was it the data
50:02
or was it how the model was trained or where we just never
50:06
going to get the performance on the model that we thought,
50:08
right? And At least now, you know that
50:11
the source data that you're getting, has some kind of
50:14
quality check on it, whereas, before you were like, you know,
50:19
I could make like, as a data scientist, you know, banging
50:22
your head against the wall being like, why isn't this working?
50:24
I don't understand. When in fact, like, you know,
50:28
maybe the data that you originally got wasn't very good
50:30
quality data. And so, you know, garbage in
50:33
garbage out, right? So I think that's a, that's an
50:36
added benefit because it there's the governance for the Quality
50:39
and in the quality, gets set up and an automated fashion and you
50:42
know that the data is more trustworthy.
50:44
And so then you also, I also have discovered that when we've
50:48
run data, quality, checks over the data which we thought was
50:51
trustworthy and we deployed the data product for the first time.
50:54
And it was like, oh I didn't realize that like 75% of this
51:00
field that we thought was really valuable is all null, right?
51:03
You know. And you go back to the operation
51:05
system you're like hey just noticed this for the first time
51:09
in a year. Classic classic null values
51:14
getting in the way, but that's I mean, that's when actually, he's
51:20
when you profile the data for the first time and you discover
51:23
act like just kind of, like, kind of how trash maybe that
51:25
column is where the field probabilities from someone.
51:28
Once said there was a quote around anyone that thinks their
51:30
data quality is good, hasn't looked properly like that.
51:40
Never trust a idea exactly but you got to use that feedback to
51:43
give, you know, to tell the operations folks or or the the
51:47
data Engineers who are processing it through the
51:49
pipeline a you know, not good, right?
51:52
We got to fix this completely. So Sean, man, this has been
51:57
eye-opening. I really appreciate the way that
51:59
you break it down and you sat through some of our qualms and
52:05
questions. I imagine you're getting this
52:08
day in and day out when you're talking.
52:09
Talking with people about the transformation and trying to go
52:12
through the data mesh. It seems like you have some War
52:16
Stories. You have a lot of experience
52:19
helping people make that transition and then seeing the
52:22
benefits I think this is where we're going to finish it.
52:27
I've got one last question for you and that is going to be a
52:32
deeper one. So brace yourself.
52:36
How do you want to be remembered?
52:39
Well, that was that is kind of a deep one.
52:45
How do I want to be remembered? Oh wow.
52:48
I like this one why. I'm I I'm a philosophy major.
52:51
So I'm really I'm really thinking on this one.
52:55
I don't know. I mean, you know, I think yeah,
53:02
I want to be remembered as you know, someone who really like I
53:07
love helping others. So I just wanted Remember to
53:09
someone that, you know, is just like a good person, that helps
53:13
others, right. Like a good, A Good Samaritan.
53:16
That's that's, you know, trying to kind of further along like
53:19
are like slow slog through the primordial slime of like
53:24
learning like ml modeling and artificial intelligence and
53:28
building Data Systems, right? Because I just feel like we're
53:32
kind of in the space where like software engineering was like
53:36
maybe five ten years ago and now we're just Kind of like
53:40
everything is new and like we're figuring out how to use it and I
53:43
just kind of want to be there to like, like help, you know, fight
53:47
the fight to fight of improving and be better.
53:51
So, that's an answer I can get behind.
53:54
Thank you so much John. Cool.
53:56
Thanks a lot.