Latest / Agentic Conversations (formally mlops.community) / The Only Constant is (Data) Change // Panel // DE4AI

Transcript

The Only Constant is (Data) Change // Panel // DE4AI · Agentic Conversations (formally mlops.community)

0:04
Let's see, we have Chad, Kristoff, Ben, Maggie up for the
0:08
panel. What might be like a good, like,
0:11
dramatic way to introduce you guys?
0:14
I'm not sure. How about this?
0:16
We should. OK, I'm going to bring one
0:18
person on 1st and then they can introduce the next person.
0:21
Do you all know each other? I'm not even sure you all know
0:23
each other. Let's see.
0:24
Maggie, are you here with me? I'm here.
0:27
Hey. Hi, Maggie.
0:28
How are you? Hey.
0:29
Adam. What are you guys going to be
0:31
talking about today? We're talking what it takes to
0:34
work in data engineering these days.
0:36
We've all been in the space for, I don't know, collectively half
0:39
a century, I think. So yeah, we're going to dig into
0:43
kind of dig into how the data, the data engineering role has
0:48
shifted and, and kind of how we evolved with it.
0:51
The only constant is data. Change data, change data, change
0:56
OK. I love it.
0:57
So next to the stage, maybe we'll introduce Chad.
1:03
Chad, are you with us? I'm with you.
1:05
Hi, Chad. We got Chad.
1:07
We got Kristoff. Hello, Kristoff.
1:10
Hey. Ben.
1:12
Hello, hello 2. 3-4 I think we're good to go.
1:15
We're good. OK, I'll be back in a bit.
1:20
Good luck guys, enjoy and I'll be eating popcorn and watching
1:25
it. All right, well, we do get
1:29
intros. Do we want to do a quick brown?
1:31
I guess? Yeah, let's do it.
1:33
Awesome, awesome. I guess I'll, I'll just start
1:35
the things first. Hey, everyone.
1:36
My name is Ben Rojan. Online I go as the Seattle data
1:41
guy before I fully became the Seattle data guy.
1:45
Also, is there a reverb? I feel like I'm hearing myself
1:47
somewhere. Yeah, but before I kind of went
1:49
full Seattle data guy, worked at Facebook for a few years as a
1:52
data engineer. Before that, worked a lot in
1:54
healthcare doing data engineering and data analytics
1:57
work so great to meet you all. All right, I will go next.
2:03
I'm Maggie. I'm part of the actual founding
2:05
team at Actual Data. I'm a community product manager
2:08
for Data Hub. Data Hub's an open source
2:10
metadata platform. I've been in the data space for
2:13
I think 14 years at this point, done everything from kind of
2:18
individual, you know, I see work as we weren't called a data
2:21
scientist, but it's basically data science back in the day.
2:24
Then ended up moving into kind of data engineering as a tech
2:28
lead for our data team and then made my way into product
2:31
management. So in the data space for this
2:34
whole the whole journey, it's been a journey to say the least
2:37
and excited to be here. Cool.
2:42
I can continue. So I'm Christophe.
2:45
I've been doing that engineering for the last 10 years and for
2:50
six years I've been like in permanent position and for the
2:53
last four years I've been doing freelancing on the side.
2:56
I also do like content creation on blef dot FA where I have a
3:01
newsletter where I do like content creation every week
3:04
about data AI and so on. And since this week I'm
3:08
launching a new company called now to help people doing data
3:13
transformation without any technical knowledge because like
3:18
AI gives super work to everyone. So maybe like the next thing?
3:24
Awesome. Well, I'll wrap it up.
3:26
I'm Chad. I'm the CEO of a data
3:28
infrastructure company called Gable AI.
3:31
Prior to that, I've done a little bit of everything in data
3:35
managed data engineering teams. I was an analyst, I was a data
3:37
scientist. I worked on infrastructure.
3:40
I led the data platform and artificial intelligence platform
3:43
team at a company called Convoy. Part of that I was a tech lead
3:48
on Microsoft's AI platform as well, so that's cool.
3:55
Cool and there is something cool to say that I live in Paris
3:58
slash building and you 3 of you like living in the states I
4:02
guess. Yes, yes, we're we're all boring
4:06
in stateside. I know, I know.
4:09
Well, you guys can speak to yourselves at at the moment I'm
4:11
actually in in London. So I'm in a, I'm in a London
4:15
hotel room in between presenting at other conferences.
4:19
So. Oh yeah.
4:21
Yeah. Hustling out there, man, Yeah.
4:26
Yeah, multitasking to the extreme.
4:27
Awesome, awesome. Well with the topic of this
4:31
panel, I kind of posed the first question to kind of just help
4:34
level set. So I just want to ask, you know,
4:37
when y'all first started in the data world, what, what did it
4:40
kind of look like both in terms of like tools that we you used
4:42
or maybe just terms that were kind of very standard, like
4:46
what, what was it like for y'all?
4:47
And maybe I can just direct that towards you, Maggie.
4:51
Yeah. So I, my first role was at Bank
4:53
of America. This is where I was, you know,
4:57
acting as a data scientist. I was a market information
4:59
consultant, which, what does that even mean?
5:02
But the very first language I ever learned was SAS.
5:06
I don't even, I think SAS is still around, but you know, kind
5:09
of doing all of our, our data modelling and analysis in there.
5:12
I stumbled my way through Sequel, took me a while to
5:17
figure it out and then really kind of hit my groove.
5:19
But I mean, this is back. We're we're working in SAS and
5:22
Teradata. There were days of the month,
5:24
like the first five days of the month, you couldn't query
5:26
against the database because they were like running all the
5:29
batch backfill. So things were just like
5:32
incredibly slow and managed by a central team of, you know,
5:38
database database engineers who we never really interfaced with,
5:42
right? Like the magic just magically or
5:44
the data just magically appeared and there was no conversation
5:47
around kind of like how it was created, how was it derived?
5:50
It was just like very pristine and, and ready for us.
5:53
And I remember there was one of my team mates was fighting the
5:55
good fight to get us to adopt Python.
5:58
And like no one knew what it was.
6:01
We were all looking at them like, why would we use something
6:03
else? SAS is everything we need.
6:06
So just seeing that the evolution of how quickly that
6:08
has evolved has been, has been tremendous.
6:12
Chad, what about you? Yeah.
6:16
So this is going to sound very basic, but I think my first
6:20
experience in the data world was was Microsoft Excel.
6:23
So there's a lot of Excel work being done.
6:26
So my first sort of real technical job was more on the
6:29
SSIS side, so working a lot more heavily with the relational
6:33
databases. And that, that's sort of what I
6:36
understood data to be was more of the transactional operational
6:39
sense. I, I didn't have that much
6:42
analytical experience. And then we ended up doing sort
6:46
of regular FTP based sort of data dumps.
6:49
And I, I was sort of managing that process and trying to think
6:54
about what, what, what quality look like when and quality was
6:59
was 0 automation. And it was more just making sure
7:01
the person responsible for the, for the data dump wasn't, wasn't
7:06
significantly changing the schema every single quarter
7:09
because it, it caused some pretty significant.
7:13
And and then from there started going into our from some of the
7:16
data science stuff and and then it just kind of exploded.
7:20
I don't know exactly when it was.
7:23
Maybe maybe there was a specific year that I could point to on a
7:26
calendar where it felt like the number of technologies just
7:29
basically jumped really substantially.
7:32
I I think it was probably post Hadoop era once we went to cloud
7:36
and then you had a lot of like vendors and sort of open source
7:38
companies all pile in at the same time.
7:40
But yeah, that's 1000. 16 I think.
7:43
Yeah. Yeah, that's that's probably
7:45
right. Yeah.
7:48
Yeah. And on my side, I don't know if
7:51
I came late to the party compared to you, but back in the
7:53
time, so when I started like in 2014, I was, so I started with
7:58
Hadoop in 2014 in France. And at that time it was like one
8:04
of the first projects in France, like to build data lakes and so
8:06
on. I was in a constitutional film.
8:08
And at that time I was like every time saying to the
8:10
clients, yeah, in France we are like three years later before
8:13
the States. So we are doing the same as they
8:16
do. And it exploded on, on their
8:18
side, but they started in 2011. And so when I was like building
8:23
the Forza lakes in 2014, I, I guess I was like a, a pioneer, a
8:27
pioneer in, in France. And yeah, it's how it got me
8:31
into data. I was just like graduated and I
8:35
was building data lakes and no one was knowing what it was
8:40
meaning at that time, to be honest.
8:44
But yeah, that's how I started. I knew urban.
8:50
Yeah, yeah. I mean, I I started like the
8:52
first company I worked for was a hospital and we were using a lot
8:55
of SQL Server. I remember there's always like
8:58
whispers of like a data lake and Hadoop being built off somewhere
9:02
else. But you know, like for most
9:04
everyone else, the the data warehouse was in SQL Server.
9:08
And I always say that I didn't know what a data warehouse was
9:10
at the point because like I knew what a database was.
9:12
I was literally taking a database course in school at the
9:15
time. I was like interning at this
9:16
hospital and then they're like here, here's, you know, SQL
9:18
Server. So I'm like, oh, yeah, this is
9:19
just a normal database. And I remember I had to like
9:22
eventually figure out in a month or two be like, oh, this is
9:25
something slightly different. This is, you know, they don't
9:27
teach this in school. I mean, now they do.
9:29
But back then it was just like, unless you took like some course
9:33
afterwards or some certificate afterwards for like ABI or data
9:36
warehousing programme, you wouldn't really realise it was
9:38
different unless someone like told you directly or you kind of
9:41
eventually started to piece together that, yes, you're using
9:44
sequel and you're, you know, things are called keys instead
9:48
of IDs. But eventually you start putting
9:50
things together. So that that's definitely where
9:52
I where I started. Yeah, it's actually, it's
9:56
actually pretty interesting now that you bring that up, what it
9:59
sort of feels like, you know, like back back in the back in
10:03
the 70s and, and, and in the 80s, there was sort of a very
10:08
defined split between sort of the, the, the DBA, the
10:14
relational operational database owner and you're, you're sort of
10:21
data steward. That was thinking a lot more
10:24
carefully around things like data modelling.
10:26
And, you know, they had a strong opinion on whether Ralph
10:30
Kimball's data marts are the right thing to do or, or did
10:32
they follow Ben and men and they were focused more on the data
10:35
warehouse. And it, it feels like over time,
10:38
those two sides of sort of the more production data modelling
10:45
software engineering skill set and the more like philosophical,
10:52
the data steward, data strategists have started to come
10:55
closer and closer together. And now on as like data
10:59
engineers, it feels like we're having these conversations a lot
11:02
more. Whereas I think when I was first
11:04
starting my career like that, at least on the on the operational
11:07
side, it wasn't something that teams are really talking about
11:10
all that much. It was a lot of like a lot of
11:13
what I started with was, you know, kind of setting the
11:15
business requirements upfront and just handing it over for
11:18
someone else to deploy, right. So it's very much kind of like
11:21
the business was driving, the business needs were driving how
11:24
that data was ultimately modelled.
11:27
And what I've seen it, it's been really interesting to kind of
11:29
see that relax that kind of requirement relax because it's
11:33
so much easier to produce or transform or store kind of get
11:37
data in and out to where you don't need to.
11:39
It's it's like lower risk or lower time to, to value there.
11:43
But then on the other end of that, when you have poorly
11:45
modelled data, it's, it's difficult to actually
11:48
manipulate, right? So it's, it's interesting to
11:49
see, you know, you still have people who are staunchly in the
11:52
Kimball camp, right? Like it doesn't matter what,
11:54
what your storage layer looks like.
11:55
It doesn't matter, you know, what tool you're using.
11:58
You really need to have the foundational data model.
12:00
But those things take time, right?
12:01
So it's, it's like the trade off of, of having those
12:05
conversations around really well modelled data for analytics or
12:09
for kind of data science use cases.
12:12
And I see that debate just still kind of we, we haven't settled
12:16
that debate on figuring out how we, we kind of sequence all
12:19
these things together. But but there is I think there
12:22
is an issue because like data teams are often sees as cost
12:25
centre, whereas like software engineering team or like proper
12:30
product team, I would say produce something that makes
12:33
money. Whereas like data team is often
12:37
like doing like reporting or stuff like this, but it doesn't
12:41
make money. It just show numbers.
12:43
And I think this is something I've said a lot of time that to
12:47
me like a lot of data teams are still in their existence of
12:50
crisis. They didn't find like their goal
12:53
in the company. So they say yes to everything
12:56
they say they are like doing shadow IT trying like to to find
13:02
like their their purpose within the company.
13:05
And so that leads to poorly designed poorly that wrong data
13:10
models, wrong, no contract whatever, because they just say
13:13
yes. They do it on the rush and they
13:15
they want to provide you actually and still the same.
13:19
Actually, it was the same 10 years ago.
13:20
It's still the same today. That's exactly where I was going
13:22
to go. It was like, so it feels like
13:24
what you're saying is not much has changed.
13:25
We've just changed maybe the platform and the technology, but
13:28
like the the people problems remain the same, right?
13:31
So I think that's like if you were to ask like what's some of
13:33
the biggest changes right over the last decade or so, It's like
13:36
just the adoption of cloud has been, you know, it just takes
13:39
time, right? Like when, when, when things
13:41
were launched and now, you know, I think things take time and
13:44
then you're just seeing. I think the other thing is that
13:46
a larger range of companies now have things like a data
13:49
warehouse, whereas before it was like, OK, do you have $5,000,000
13:53
to sign up for Teradata? OK, If not, you're just doing a
13:55
sequel server, you know, underneath someone's desk.
13:57
And, and that's if you're willing to pay someone for it.
14:00
And you can't access your data because you have like an ERP
14:03
that either you don't have access to the database or the
14:05
only way you can access it is through some sort of Excel plug
14:09
in that lets you export data. I actually had that for one of
14:13
the places that I worked where like it was limited to the row
14:16
count of I don't know how many. Yeah, exactly.
14:18
It was like. 56,000. And it was like, OK, so just
14:23
that, you know, and so you couldn't even export all the
14:26
data and it was just such such a hassle.
14:28
So, you know, I think that's kind of when the changes is like
14:31
the technology platform has changed.
14:32
But somehow we're still kind of we're still trying to somehow
14:35
figure it out. I think in in some spaces or
14:38
what I think sometimes happens is like the, you know, the
14:42
people doing the data work are, you know, still inexperienced
14:46
when they're coming in. So they're just learning.
14:48
And then they're kind of going through the same lessons that
14:50
someone went through that maybe has that 20 or 30 years of
14:53
experience that maybe he's working either for a larger
14:55
company or maybe just was like data is exhausting somebody.
14:59
Could do something. I, I think you're, I think
15:05
you're, I would agree and disagree.
15:07
I think that you're absolutely right that the, the platforms
15:11
have changed, the technologies have changed and the cultural
15:14
problems have. I, I, I think there are a lot of
15:18
the same cultural problems, but I think the nature of the
15:21
cultural problems have changed as well.
15:23
So, so for example, back in the back in the 1990s, like when,
15:28
when everything was on Prem and you know, you have sort of
15:30
you're, you're running Teradata or, or something like that,
15:34
there was a, a cost, like there was a, a cost barrier to who
15:40
could actually have a meaningful data warehouse, right?
15:44
Like you, you had to be willing to, you know, stand up servers
15:47
and buy these tools and hire DBA.
15:49
Well, hire like data architects and have like some way of doing
15:53
ETL, which is also a very cost intensive process.
15:57
And so it, it limited the number of companies that had sort of
16:00
real data warehouses and were doing analytics and, and sort of
16:04
early, early state, early days machine learning to your, your
16:08
like Fords and your Nikes. And, you know, these like
16:12
multibillion dollar firms. And in a lot of cases.
16:15
And in that world, not not only was it limiting the number of
16:20
companies, but it was also severely limiting the amount of
16:24
data that you actually had to work with because of that same
16:26
cost constraint. And that lack of data created a
16:30
very different organisational environment.
16:33
Like the data architect, I think had a much bigger role in in
16:38
1995 than they do at most companies today.
16:42
And they were effectively the bridge between what was
16:44
happening in the transactional systems and what was going on in
16:48
the, in the analytical systems. And they were sort of thinking
16:50
about both like what is our, what is our entity model?
16:53
What, what are the domains that we're creating?
16:55
How does that actually map to our, to our database structure?
16:59
How are we getting that data to our customers and and what does
17:01
the catalogue look like? You know, back back then you
17:04
could say something like you could say like, oh, I'm going to
17:07
join this company as a data steward and it wouldn't seem
17:09
weird. But if you said that today.
17:11
Yeah, everyone, everyone's a data steward at this point,
17:14
right? And some some flavour of it.
17:16
Yeah. Actually.
17:20
Yeah, exactly. So, So, yeah, that's just to say
17:23
like I I think that the the cloud has open like making
17:29
making storage and compute more or less a, a relatively solved
17:33
problem, not a totally solved problem.
17:34
Of course there's always going to be this like long tail of
17:36
issues and how exactly do we optimise it?
17:38
But solve to the point that your average business doesn't have to
17:42
it, it doesn't become an, an impossible wall to climb over.
17:45
Just to have a, a basic data warehouse has created a totally
17:49
different set of challenges, which is OK.
17:51
We're we're living in this new Federated world where we're all
17:54
kind of throwing data over the fence because it's super cheap
17:58
to do that. And, and now, and now we don't
18:01
data steward can't manage that anymore.
18:03
And, and, and I think you see this, you start to see the rise
18:05
of the data engineer in that world, whereas in the previous
18:08
world they may not have been. Yeah.
18:09
I think there are two 2 two piece of the puzzle.
18:12
There is like the hardware part with like storage and compute
18:15
that is like very at low cost today.
18:18
So you can do a lot of stuff. And there is as well like the
18:23
the pyramid of software I would say, because back in the 70s or
18:26
80s you could not do like a group buy on into like 3
18:32
terabytes of data with a single SQL query and do it like in the
18:38
in the Wink. It's like related to storage and
18:40
compute, but sorry, also related to the software and the the
18:45
stuff that people factorised and the stuff that we built on top,
18:49
on top, on top on top with all the software that we have today
18:52
and the frameworks and the distributed stuff and so on, I
18:55
guess. Yeah.
18:57
One thing I'm seeing a lot in the data or like in the open
18:59
source community for Data Hub is a lot of these challenges around
19:03
kind of like governing or kind of compliance practises to make
19:06
sure that all the data you're producing is, you know, well
19:09
documented or well, kind of like contained and well understood.
19:13
I think a lot of the practises that that were established in,
19:17
you know, in the early days of big data, we haven't really
19:20
evolved those as much, right? We haven't evolved, we haven't
19:23
kept up with the speed in which you can create net new data
19:26
resources, plug and play, you know, spin up a, a data, a data
19:30
warehouse, connect all of your tools and suddenly it's there.
19:33
And so one of the things that I'm, I'm really excited to see
19:35
is kind of how we how we start to really speed up up those
19:39
processes around or those workflows around governance and
19:42
compliance because we're just simply not keeping up.
19:45
And I don't know if that's like anything that you guys have seen
19:48
on your side. Like if there's any kind of like
19:50
tooling solutions. I mean, at the at the crux of
19:51
it, I think it is a very human centric problem of figuring out
19:56
how we kind of collectively govern these assets.
20:01
But yeah, curious if that's something that's popped up in
20:04
y'all's realms at all. Yeah, I mean, I, I, I think, you
20:09
know, I, I definitely see plenty of it from a consulting side.
20:12
You know, you, you all can come in and, and people kind of
20:14
either either through the fact that they kind of got what I
20:17
call like a key person dependency issue, or maybe they
20:19
had someone who built other pipelines using some sort of
20:23
like either loco tool or something of that nature.
20:26
That wasn't necessarily built with any form of governance,
20:29
which just kind of like built often or what I've heard it
20:32
called by, by Joe Reese's like query driven approach or just in
20:35
time kind of data modelling where there's no real data
20:38
model. You're just kind of building it
20:39
for one specific use case. I, I think that you do, you do
20:43
see that happen more often just because, you know, the tools are
20:47
easier. Everyone can kind of easily
20:49
ingest data. I, I think to some degree it's
20:53
not necessarily bad. It's just you have to figure out
20:55
how to control like once something becomes more
20:58
productive, like this is part of production or this is part of
21:00
something that's getting used a lot more often.
21:02
I think then then there's this often need to, to figure out,
21:05
OK, well, how do we now put this maybe more towards the data
21:08
engineering team versus having, you know, someone in like
21:10
marketing analytics try to manage it?
21:12
Or is this this something that we can model?
21:14
Because on one side you want people to be able to like, you
21:17
know, do their work and, and, and do it well and do it quickly
21:21
to deliver. But on the other side, you don't
21:23
want to end up a year down the line and have like 10,000 kind
21:27
of one off data pipelines that are now being then depended upon
21:31
for something else that had then being depended on pop for
21:33
something else. And then some small change like
21:37
Chad mentioned earlier, someone changes a schema file and all of
21:40
that breaks down. And that was all somehow
21:42
connected to some machine learning model that suddenly
21:46
your AD spends this, you know, crazy expensive for some weird
21:49
reason that you're going to have to go spend time figuring out.
21:53
Yeah, I, I think, I think there's, I sort of like to think
21:57
of, of data infrastructure in three big categories.
22:01
So it's sort of your bottom most foundational layer is storage
22:05
and compute. Then your second layer is data.
22:07
So it's basically like where, where do we put all of our data
22:10
and how do we run queries on top of it?
22:12
And then the second layer is, is data movement.
22:14
So like the value of data is you, you produce it sort of over
22:19
here. And, and generally the whoever
22:22
is the producer of that data does not get maximum utility out
22:25
of it. That utility comes from sharing
22:27
that data with other people in the organisation when they may
22:30
have different use cases for it. So we have to figure out how to
22:33
get the data from, from this person to A, to a consumer.
22:36
And, and, and that includes things like like, you know, the,
22:40
the, the actual transformation, the loading of the data,
22:43
something like a streaming would sort of fall into this category.
22:46
And then sitting on top of that is OK, now that people can
22:49
access data really what, whenever they need it, however
22:52
they need it, we, we need some way to manage it.
22:55
And that includes sort of the quality control, governance
22:59
compliance, these types of of systems.
23:03
And then I think of the layer on top of that as sort of like data
23:06
applications, right? Like how are you actually using
23:09
that well managed data in order to accomplish some business
23:12
tasks. And I think that that that set
23:16
of things like that, those set of categories was totally true
23:19
like it was, it was needed and necessary in the on Prem world
23:24
as well. And in the on Prem world,
23:27
because the data sets were so much smaller, companies built
23:31
their entire internal data ecosystem and they did all of
23:35
their hiring around those four things, sort of around the, the,
23:40
the premise that like storage is incredibly expensive.
23:43
Therefore, it's actually cheaper to do all this transformation in
23:47
one place by a group of specialists.
23:50
And it's cheaper to have a data steward who is thinking, who is
23:53
like manually adding all the information about the catalogue
23:57
and the, the, the, the PII, although that wasn't really a
24:00
big deal about then, but you know, like all the other sort of
24:02
access control policies and things like that.
24:04
And, and this is sort of how the modern data organisation was was
24:09
formed and then the cloud happened and basically every
24:14
single CTO at every company in the world said we need to make
24:17
our data team more like software engineers because software
24:20
engineers are actually the ones that are starting to make us
24:24
more money than these data people.
24:25
That used not to be the case. It used to be that that data
24:28
people made the company, you know, way more money on the
24:30
technology side than, than the software people.
24:32
But then apps became a big thing and like websites and e-commerce
24:36
became a, a really big thing. And so it totally shifted.
24:39
And they said, all right, how do we make sort of our, our, our
24:42
data team and, and our engineering team that works with
24:44
data more similar to software engineering?
24:48
And that means we need to go fast.
24:50
It has to be super fast. Everyone needs to be very
24:52
isolated and, and fragmented and, and they need to be able to
24:55
work independently because that's how software engineers
24:57
build features, right? Is you, you silo them and you
25:00
say you got you, you go work separately on your own and you
25:03
go really, really quick. And that, that's basically what
25:06
happened, right? So everyone started organising
25:08
their data teams that way. They started organising data
25:10
production that way, where every producer is kind of like
25:14
generating data from their database or their events and
25:17
they're pushing it into cloud storage and you don't really
25:20
know where it's coming from and you don't really know the
25:22
quality of it. And that makes governance a
25:24
very, very hard thing. So I think, I think that Maggie,
25:28
to go back to your question of, you know, what have, what have
25:31
we sort of seen as successful in governing data today?
25:34
I think what it's going to take is almost like a ground up
25:39
approach. And, and, and, and that is to
25:41
say what, what does it actually mean to govern data in a
25:47
Federated? That's a very different sort of
25:50
thought process. And I think that's why tools
25:53
like DBT and five Tran and Snowflake have been so
25:57
successful because they've rethought what storage and
26:00
compute means in a Federated ecosystem or what
26:03
transformations mean in a Federated ecosystem or what
26:06
orchestration means in a Federated ecosystem.
26:08
And I think now we're sort of arriving at that at, at, at
26:11
taking the same approach to to that data management layer.
26:18
Yeah, that makes sense. Actually there there is
26:20
something that changed as well, like in the compared to the 70s
26:24
or 80s is that the amount of data that we get right now is
26:29
kind of crazy like we, we, we, we generate way more data than.
26:34
Years before. And actually it created a
26:38
requirement that we didn't have before.
26:40
And so we have to be faster, more efficient.
26:43
And so it created like a requirement for like our tools
26:47
to be better. And it's going to continue
26:51
actually. And, and, and in data there,
26:54
there is something I love to say when I talk, for instance, about
26:58
DBTDBT is a great tool to govern data assets, But DBT is so easy
27:04
to use for analysts that creates more assets.
27:08
It creates a lot of entropy. And for instance, if you have
27:12
like a data team with 10 analysts, they create like, I
27:15
don't know, 5 models per week at the end of the year, it's going
27:18
to be already a mess. And so at some point you, you,
27:21
you need someone that say stop. You have to do a factorization,
27:26
you have to, you have to rethink and you have to, to, to, to
27:30
rethink, to refactorize, to redo something.
27:32
You you accept entropy for a few months and then at some point
27:36
someone say no stop now, do it better.
27:39
It is really interesting like if we think about in kind of like
27:42
the same software engineering or in the same software engineering
27:44
principles, it does empower analysts to move much more
27:47
quickly. If we treat a data set or a
27:48
model as a feature, right? There's there's one specific use
27:52
case. Cool.
27:52
It's super easy to go build that out and produce it.
27:55
I think the, the part that's so drastically different between,
28:00
you know, an app feature and a DBT model is actually measuring
28:04
the value or the, the interaction with that and like
28:08
deciding when to deprecate that feature, right?
28:11
If you roll, if you roll something out to your mobile
28:13
app, you could you have a way to monitor does anybody use it?
28:16
Is it driving, are we moving them through our funnel
28:18
effectively? Are we creating a user
28:21
experience that, you know, encourages people to come back
28:23
and engage, etcetera, etcetera. Whereas with the data model, it
28:28
could just sit there, right? And So what, what do you do?
28:30
You look at query activity, you look at, you look at maybe you
28:33
know, how often it's, it's referenced in Looker or Tableau
28:36
or whatever that is. But what I've seen again and
28:38
again and again, and this is true, when I was tech lead for
28:40
a, for a data team as well, there wasn't really a
28:43
consequence to maintaining or to, to keeping them live.
28:47
And, and it took more kind of cognitive overhead to say, well,
28:50
do we need this? Like, what's the strategy around
28:53
it? Right.
28:54
So like the incremental cost of, of deprecating it was actually
28:56
like a pretty large like, well, we might use that or the
28:59
stakeholder said it was really important.
29:01
So I don't want to deprecate it and make them mad, right?
29:04
Until you get to the point where you're, you have thousands of
29:07
interdependent models and it's just a nightmare to maintain.
29:11
So I think on, on kind of like the, the software engineering
29:13
principles of it. I would love for us to get to a
29:15
point where there's like true pattern, like true development
29:20
patterns where we say we build this and it's in, you know, you
29:24
have like your staging environment, your dev
29:25
environment, your prod environment, but then also like
29:27
your deprecation strategy because the the cost of
29:30
maintaining just one net new model week over week, not a big
29:34
deal, but years later, it's so much to untangle.
29:39
It's so much to untangle. Face the reality and look at the
29:43
usage. I'm like writing that down like
29:46
that's a good next like article or thought.
29:49
Hey, wait, I'm a co-author on that one, Ben.
29:53
The data deprecation strategy. No, it's huge though, right?
29:56
Because it's, it the, it's just, it's a, it's a, it's a different
30:00
type of tech debt that, that software engineers don't.
30:03
It's a different way of thinking about tech debt that that
30:06
doesn't neatly fit into the kind of software engineering
30:09
playbook. Yeah, I think one of the reasons
30:13
that is the case is because software engineers and data
30:17
teams sort of approach writing code very differently.
30:21
And I think we're we're finally starting to realise that that
30:23
this is this is these are actually like relatively
30:25
different disciplines in the software engineering world.
30:28
It's the, the things that the, the code that you write is
30:32
generally driven by a very clear set of requirements.
30:37
And those requirements map 11 typically with external facing
30:41
customer value. So someone says, hey, I, my, my
30:45
customers are asking me for a drop down list because they're
30:50
having trouble finding items in their cart or whatever.
30:53
And then you go out, you build that you, you, you come up with
30:56
a set of specifications, you deploy it.
30:58
Then you check to see if customers are indeed using it
31:01
the way that you want. And if they are great, you roll
31:04
it out. And if they're not using it the
31:05
right way, you go, crap, let's roll it back.
31:08
That clearly wasn't the right thing.
31:10
So it's very easy to sort of map software to, to value.
31:14
Whereas in, in Dataland, it's not really like that data almost
31:20
always starts with a question, right?
31:21
You have a question about the world, You say, well, I wonder
31:24
I, I, I want to understand X how customers are interacting with
31:31
this functionality or how marketing performance has been
31:35
over the last three or four years.
31:37
And the answer to that question could be valuable, like it could
31:40
ultimately lead to something that makes a company a lot of
31:43
money, or it could be totally non valuable where you're like,
31:46
oh, OK, that, that didn't really tell me anything.
31:48
So that that's a very different type of approach to building
31:51
software. To your point where it's more
31:53
about experimentation, then it is sort of following this very
31:57
linear trajectory towards deployment.
32:00
And there there is like another axis which is a question is very
32:03
short lived. Like you have a question and you
32:05
want the answer in 5 minutes and in 10 minutes it may be too
32:09
late. Actually you don't care about
32:11
the answer like in 10 minutes or in one day or one week.
32:13
It's like from now. I'm questioned.
32:18
So does anyone like in terms of timing for this, this panel?
32:22
Oh, there we go. He's.
32:23
The timer is up, so. You're going to have to cut us
32:26
off because we can keep going. But I.
32:28
Want you to keep going because I mean, I wrote down a bunch of
32:31
different you guys think you're co-authors, but I wrote down all
32:34
these new ideas for articles that I'm going to write after
32:37
this. Wait, hold on.
32:39
I'm we're we're co-authors. I'm just for ideas.
32:43
It was in the fine print that you didn't see.
32:45
Oh. No.
32:47
Is that what this is, Adam? This is just.
32:48
Yeah, right. So actually there's a question
32:52
somebody had here in the chat, or at least a thought, Brian put
32:57
this in the in the chat. You need to have your teammates
32:59
take turns being data model pruners.
33:02
Have you ever experimented with that?
33:04
What? And if so, do you have any
33:05
thoughts on that? We did something similar at
33:07
Braintree when I was a tech lead there, where we kind of cycle
33:10
through owning those data, like kind of owning the the data
33:14
modelling approach. And honestly, it was a really
33:17
valuable experience for, I mean, tenfold reasons.
33:21
But number one, it really it forces to continually refine our
33:26
kind of core data models that fed into hundreds and thousands
33:30
of resources. But two, it also was because
33:32
we're on a centralised team, it was so beneficial because it
33:36
exposed other team members to other parts of the business.
33:39
And so there was just like so much embedded context sharing in
33:41
that, which really removed it from being, well, this person
33:45
knows this part of the company, that person knows that part of
33:47
the company. And if one of them's out of
33:48
office, we're screwed, right? So it was, it was extremely
33:52
valuable to cycle through it, but it was something that it
33:54
took a lot of time to kind of build out that extras or that
33:57
practise and it, it led to us potentially moving slower on
34:02
stakeholder requests, right? So we had to really show value
34:07
or kind of like demonstrate why that was worth the investment.
34:10
But I highly, highly recommend taking a similar approach.
34:13
It was it was extremely valuable for us.
34:18
Thank you. Yeah, I have another.
34:19
This is like another thought that came to mind.
34:23
We're discussing the difference between data that is often used
34:28
to answer questions versus features that are much more
34:33
commensurate and they react to value that users can get more
34:37
immediately. And sort of like some of the
34:40
things that you've been talking about is kind of like the
34:42
consequences of this distinction, right?
34:44
It can be a little bit more messy and perhaps like in
34:47
tropically unstable to just continue to amass more and more
34:51
of these answers to potential potentially interesting
34:53
questions at the same time. So, OK, do you think it's more
34:57
likely to be the case that we will move towards a paradigm
35:01
where we parse what data actually provided value or right
35:07
so that the data kind of moves to see more like a feature or
35:10
the features that we build? And I'm mostly thinking about
35:13
like start-ups building features in like a pre like product
35:15
market fit world where in a sense you are also asking
35:18
questions. Oh.
35:23
Uh oh. Oh me, am I back?
35:26
Am I back? You're back.
35:28
Yes, you're back. You're back, Eric.
35:32
You're gone. Back and forth back.
35:39
And forth. Oh no, What's going on here?
35:40
You know what I was? I was saying something that
35:42
might be offensive to some of the data gods, I decided.
35:47
Yeah. OK, I'll summarise this quickly
35:50
as the question. Do you think it's more likely
35:54
the data will begin to function more like features in the sense
35:58
that it's very important that we attribute value to each data or
36:04
model or data set that we put or that data or that we will start
36:10
to see features as more experimental in conjectural and
36:14
answering particular questions, business questions.
36:16
And I'm mostly thinking like in the paradigm of, let's say like
36:19
startups and like pre product market fit, where almost
36:22
everything you're coming up with in terms of features is somewhat
36:24
conjectural. It's it is, or at least ideally
36:27
answering some kind of like hypothesis.
36:30
So do you have a sense of which way we're likely to move?
36:36
There's, oh, I'm sorry, there's, there's a little bit of a split
36:41
because like there are some aspects that like sometimes you
36:43
do have data that you might be using to build into a feature,
36:47
right? Like maybe it's for an ML?
36:48
Model. Maybe it's and.
36:49
Then there's the other side where it's like, OK, if it's
36:51
building a report or as you know, Christophe mentioned
36:53
earlier, it's like, we just need to know this question right now
36:56
and you know, the next two minutes.
36:57
And then after that, you know that the value of it kind of
37:00
kind of dissipates. So I, I think there's, I don't
37:02
think it's going to push towards anyway.
37:03
I just think there's there's different places that data adds
37:05
value and some of it is in the feature and some of it is just
37:08
this ethereal, like we just need to know so we can make a choice.
37:12
I I kind of think of that. It makes me think of kind of
37:15
like a metric layer or like a metric store semantic layer,
37:18
where I actually think we could start thinking about those as
37:21
features where it does go through many stages of
37:23
refinement iteration, but then it becomes a building block by
37:27
which everything else is kind of framed, right?
37:29
So I just, I wonder if we start thinking about kind of like
37:32
components of data modelling or components of data kind of like
37:37
like specific calculations or kind of standardised ways to
37:41
measure like customer lifetime value, right?
37:44
You go through, it takes you a while to figure out what does
37:46
that mean for the business? How do we measure it?
37:48
What are our inputs? The feature kind of that output
37:52
is maybe something that that starts to evolve more into a
37:54
feature where kind of the, the, the ultimate output remains the
37:59
same even. And it's more like interoperable
38:03
with the evolution of your of your data ecosystem over time.
38:08
Yeah. I, I think, I think this is a
38:10
really interesting topic. I, I would love it if the
38:15
business started to look at our data model, at their data model
38:18
more like a product. Unfortunately, I don't think
38:22
that will ever happen until you can trace a very clear line from
38:27
the data model to a dollar and a bank because that's exactly what
38:30
you can do with with a software feature.
38:33
And that's how CT OS and other engineering leaders make
38:37
resourcing decisions is they basically say, well, I know
38:39
exactly how many people I should hire because it's going to get
38:43
me X amount of ROI on this product that we're deploying by
38:47
by this date. So I think that that This is why
38:51
the data product phenomenon is becoming so exciting for data
38:56
teams is because we're moving away from just saying, hey,
38:58
we're, we're doing data modelling for the sake of it, or
39:00
we're doing data quality for the sake of it because it's the
39:03
right thing to do. And we're actually doing it
39:05
because data needs to be treated like a product because it can be
39:08
traced to that value. And my view is that in the same
39:12
way like a, an application has a front end and a back end, right?
39:15
You have a front end that is a customer interface and they go
39:19
in and they type some things and they click on buttons and they
39:21
get value out of that. And then you have this really
39:23
big, big, complex, gnarly back end system with like databases
39:27
and a whole bunch of other things.
39:28
I think that data products will effectively have that same sort
39:31
of relationship where you've got a front end and the front end
39:34
might be might be a dashboard, it might be the prediction of a
39:37
machine learning model. And the back end is your
39:40
pipeline, right? Like it is your data model.
39:42
It's all the stuff that ultimately produces the thing
39:44
that a customer has interacts with.
39:47
And once we start doing a better job sort of categorising what
39:51
are the data products in our eco system like what are the and
39:55
and, and again the data product is the front end and the back
39:57
end. So what are the pipelines all
39:59
together that are money and creating value for our customers
40:03
that we can tie to a dollar in a bank.
40:05
Then I think we're going to start to see this convergence
40:07
between the way that software engineers sort of take.
40:11
Accountability and responsibility and visibility
40:14
for their products and the way that that data teams to.
40:17
And then Adam, to your point, I agree.
40:19
I, I totally think that engineers will start getting
40:22
more experimental as well. So those there's probably going
40:25
to be a bit of a bi directional convergence here over time.
40:29
Guys, thank you very much for this.
40:31
I thank you. I will.
40:33
I will at least tag you or mention you.
40:37
Not a problem, excited to see it.
40:40
Thank you very much, this was fascinating.
40:42
I'm sure we could keep going for many more hours.