Latest / Agentic Conversations (formally mlops.community) / Data Engineering for ML // Chad Sanderson // Coffee Sessions #117

Transcript

Data Engineering for ML // Chad Sanderson // Coffee Sessions #117 · Agentic Conversations (formally mlops.community)

0:00
Hi, I'm Josh Wills. I am a software engineer at
0:03
we've grid. I'm a recovering manager.
0:06
I used to be the director of data engineering at slac.
0:08
Many, many lifetimes ago, director of data science and
0:10
Clara and I drink espresso, I'm in a particular bluebottles 17
0:16
foot ceiling is my just perfect like it's it is basically
0:19
perfect espresso as far as I'm concerned.
0:21
Just straight-up double espressos is that that's my
0:24
happy spot. My name is Chad Sanderson.
0:27
I am the Lead product manager for the data platform, team at
0:34
Convoy. And then a part of this, I was a
0:38
product management, Microsoft working on the big data team
0:42
called the AI platform over at Microsoft.
0:45
And I don't drink coffee, but I do drink green tea every
0:50
morning. I am stoked because we've got
0:56
Josh wheels and Chad Sanderson. This is like Christmas came
1:01
early for me. I'm so stoked to have the both
1:04
of you on here. I know this was kind of last
1:06
minute rearranging all of our schedules to make it work.
1:09
It worked. I mean, originally Chad, I
1:12
wanted to talk with you. We had our first encounter, you
1:15
came on the Meetup, you blew my mind.
1:18
I want to dive into more of the stuff since then you've become
1:21
somewhat of a link. Tin personality.
1:24
You might win, think tins, top voices for 2022.
1:28
And yeah, your personal brand has been growing.
1:31
You're killing it, you've got your sub stat going.
1:34
That is also killing it. It's been making a lot of waves
1:36
in the ml Ops Community. Anyone out there that is not
1:40
subscribed to Chad, stub, stack, go and do that right now.
1:43
Do yourself a favor, you will not regret it.
1:45
And last thing I will say Josh, we gotta plug this before.
1:48
We jump into it, Josh's going to be leading a course.
1:53
On data engineering for machine learning.
1:55
I believe. That's right.
1:56
That's right. And that's coming up in October.
1:58
So anyone in September is September to mid-september.
2:02
Yeah, sorry just had to get sober.
2:04
I do the slacker. Yeah, exact September it's
2:08
happening. Get after it.
2:09
Will leave a link to that in the show.
2:11
Notes, if you want to jump in to his course and we should
2:15
probably start Chad with what I feel like made a lot of waves in
2:20
the MLS Community. You wrote an incredible blog.
2:22
It's about data modeling. Can you just give us some
2:25
background on that? What the hell is is it that
2:27
you're talking about for those who do not subscribe to your sub
2:30
stack and then we'll start doing the good old fun of talking
2:35
about why it's so important? Yeah, so data modeling, if
2:42
you're not super familiar with it, like the sort of elevator
2:46
pitch is you're basically building relationships between
2:52
like core Within your data. And there's actually two places
2:56
that you can do this. So there's something called the
2:58
physical data model, which is like how the relationships
3:02
manifest in your data environment, you know, whether
3:05
it's like snowflake or whatever, whatever it happens to be.
3:09
So this is, you know, the process of using DBT and you're
3:12
joining tables together and all of that stuff.
3:16
But then there's something else which is like a semantic model.
3:20
This is the way that I hate to call it like, Legacy or old
3:25
school because I don't really think it's Legacy.
3:26
I think it's still pretty meaningful, and a lot of data.
3:29
Architect still do. This is like entity-relationship
3:31
design. And that's the like, you find an
3:35
inn. You think about the entities
3:36
that are meaningful at your company?
3:39
So, abstracted away from any data Centric, implementation
3:45
you, then diagram the relationships between those
3:49
entities. You think about the cardinality
3:51
like, does it have a one to many?
3:52
Any relationship or minute one. You create, this essentially
3:57
diagram of the world, all of the properties that are relevant and
4:01
sort of germane to each entity. And then that, that is what
4:05
informs your physical model. Like the thing that you actually
4:08
know build in your, in your in Snowflake or whatever it might
4:12
be. The, the point that I called out
4:15
is like that process of taking sort of a strategic very
4:19
thoughtful approach to modeling is not that But in a lot of
4:24
clouds Centric, like data based organizations Convoy was
4:30
certainly among them what has become increasingly more
4:33
requirement. Common.
4:34
I've noticed is the elt approach of like let's just collect
4:38
everything we possibly can from every Source.
4:40
We possibly can dump That Into You Know, snowflake or dump it
4:44
into Data bricks and then we'll kind of like do a little bit of
4:48
modeling just so that we can actually use the data to answer
4:51
some business question. But it's Not comprehensive.
4:54
It's not like the full, the full spectrum of relationships, which
5:00
makes our data like fundamentally less useful.
5:02
And then you see a lot of these. Basically, the same query is
5:05
being written over and over and over again.
5:07
One reason because that happens is because the data model is
5:10
actually like not comprehensive. And so people end up, you know,
5:14
the replicating things and duplicating things.
5:16
I think that's a big problem. I think there's a huge problem
5:19
there on like snowflakes spend. We're seeing this at Convoy.
5:23
Now, like if you do not have a great data model, you get a ton
5:26
of replication and your spin goes to the roof, the complexity
5:29
of the queries is is quite impactful.
5:33
And when you start achieving, you know, Big Data at scale,
5:37
that query complexity is like, very meaningful.
5:40
Maybe when you're a small company, it's not.
5:42
But once you start adding a lot of sort of data to the fire, it
5:45
is, it is actually pretty meaningful.
5:47
And then just from the perspective of like the data
5:51
being useful like being able to answer questions and not having
5:56
data engineers and data scientists and data analysts
5:59
like constantly getting pinged from your business partners like
6:02
product managers. Yeah, I'd like having a great
6:05
data model solves a lot of that. So that was sort of the core of
6:08
the of the piece. So good man and who would have
6:16
thought like data that is useful for people outside of the data
6:21
scientists? That's theory that we can all
6:24
rally around and I also think about like this idea of strategy
6:29
and needing to have the strategy when you're going into things
6:33
and I imagine that you can get away with it.
6:37
It's almost like the idea of technical debt, right?
6:41
You can get away with it for a certain amount of time until you
6:43
get to a certain level of scale and you see that.
6:47
Whoa, now once we've hit such a large scale, we're not able to
6:54
properly or quickly, take advantage and gain insights from
6:59
the data that we've got, is that right?
7:03
I think, I think that's certainly correct.
7:05
But, and I think it manifests in other problems like there are it
7:09
manifests in in problem? Surround ownership, you know, I
7:12
manifested and problems around like there's a bunch of
7:16
questions that are important for data teams to be able to answer
7:20
that are just either not possible or it takes a lot of
7:25
work. So a very common, use case that
7:27
I've seen happen is actually on like the customer service side
7:31
or like data science for a customer service or operations
7:34
where, you know, say you have some support ticket or you have
7:37
like an email thread between like an operations person and a
7:41
You're circling back and forth and you want to understand if
7:45
this particular thread or the service ticket or whatever it
7:48
might be, had some real world impact on that customer.
7:52
Like, did they buy the thing? Because their problem was
7:55
resolved, or did they not by the thing like what happened?
7:58
If there's not a great data model in place to allow you to
8:01
answer that question? It's really the difficult to do.
8:04
You end up having to write like you know a whole bunch of like
8:06
do a whole bunch of fuzzy matching and things like that
8:08
whole bunch of case statements? Well, if this thing happens,
8:11
Happened and this thing happened.
8:12
And, you know, we see that like this particular shipment that
8:15
that we're kind of guessing might be related to the thing
8:17
that we're talking about, like, you see, you see a lot of that.
8:20
And it's frankly, it's just, it's an accurate, it's wrong.
8:23
If you want to start automating, those processes is really hard
8:26
because you're going to be wrong like fifty fifty percent of the
8:28
time or more. It's really hard to build
8:31
machine learning on top of on top of that type of stuff and
8:34
it's everywhere and then the last piece I would say is that
8:38
there's just some there's fundamentally Some aspects of
8:42
the business that people are not even aware of that.
8:45
They need to be aware of. There are these the way I like
8:49
to think about a company is as a network, right?
8:52
You actually have a bunch of different entities.
8:53
They're interacting with each other all over the place and
8:57
through life cycles and what happens oftentimes is it product
9:01
teams, focus on like one particular piece of that life
9:03
cycle and if the modeling is not done pretty well, you lose out
9:08
on a lot of the benefit of like how does the entity that you
9:11
care about interact with all of these other entities in your
9:14
business. Like we have machine learning
9:16
teams that are focused on our pricing model, for example, and
9:20
there's a lot of things that impact a pricing model, but when
9:23
the modeling is not done, it becomes incredibly hard to use
9:27
that data for training and and so on and so forth.
9:30
So I think there's a lot of benefits to data science to
9:33
having modeling done well and some of them are not, they're
9:38
not immediate, they're not completely.
9:41
Parents at first glance 100%. Hey, I'm just gonna hop in here.
9:45
How you doing, guys? How's it going?
9:46
I was going y'all. You and I talked a long time
9:49
ago, I was a long time ago was like it was like one of our
9:52
mutual VC friends introduced us and even remember why we were
9:54
talking about, whatever, right. But but in that conversation you
9:58
said something to me that it's kind of stuck with me and I've
10:01
quoted it. Other people, you know, like,
10:03
you know, Chad Sanderson had a convoy you said something, which
10:07
was the, the idea that the modern, yeah.
10:11
The modern cloud data warehouses, right?
10:13
Have made it so cheap for me to just ask whatever question I
10:18
want relative to the old way of like, you know, teradata and
10:23
olap, cubes, and all that kind of stuff, right?
10:25
That it costs less for me to formulate and execute the query
10:29
that I want to run. Then it does for me to invest in
10:33
these sort of cognitive overhead required.
10:35
And understanding someone else's query, someone else's business
10:39
question and understanding okay. To the reusable components of
10:42
that that I can then extract into my, you know what I mean?
10:46
Just answer my question and, like, share that kind of stuff
10:49
and that was like deeply and profoundly insightful to me
10:52
because again, like I'm super old right.
10:54
I've been doing this stuff for like a long time like, you know,
10:58
reading like the data warehouse modeling toolkit like back when
11:01
I like started, like right out of college and like, 2001 was
11:04
like a formative, you know, formative experience in my life
11:07
and like, one of the true like, Nerdy.
11:11
I do this again. This is like, I don't know.
11:13
There's one day where I was when I was working at Google where I
11:16
met Dustin Hoffman and Jeff Dean on the same day.
11:19
And like, I was really excited to meet Jeff Dean While most
11:21
people would be excited to meet Dustin Hoffman, right?
11:26
But like beating meeting Ralph, Kimball who wrote the data
11:29
warehouse modeling toolkit was like.
11:30
I had him autograph, my copy and stuff, like I was just such an
11:33
aggressive, right? That like he had.
11:36
This is, this is sort of the core of the problem.
11:38
I think, at least as you framed it, I think this is the core of
11:40
the problem. It is.
11:41
I'll so cheap. I mean I had a tweet about this
11:43
once which was like, snowflakes genius is that it?
11:45
They have invented a business model where anyone can just
11:49
throw money at a problem, right? I want to answer my question.
11:53
I want it faster and I'm just gonna like throw a money.
11:56
This one's going to spin up the warehouse.
11:57
Make it bigger, answer my question, right?
11:59
And like, what a license to print money that is, and, and
12:03
that being like the core of our problem, right?
12:05
Is in that we move fast. We have infinite scalability,
12:09
infinite flexibility. And in, Getting so excited about
12:12
these things. We have thrown out all of the
12:15
things that were great about the old again to be fair heavily.
12:19
Resource-constrained, you know, single teradata box.
12:23
It's got 30 terabytes, that's all you have.
12:25
You know, you gotta, you gotta be careful with it.
12:27
You got to schedule it, you got a plan you know we've turned out
12:30
the baby with the bathwater here essentially.
12:32
Right. And I guess like my question for
12:34
you man is like what is the how are we going to put this Genie
12:36
back in the bottle? Like how do we start to like
12:39
unwind? Like, well, I think we're all
12:41
kind of coming down from, like, the Sugar Rush or, you know, the
12:44
cocaine, the cocaine, High whatever, whatever your
12:46
preferred analogy is here, right?
12:47
We're like really, okay, this is clearly gone too far, we've all
12:50
lost our minds, never your drug of choice has.
12:52
What every drug of choice is exactly precisely apple juice,
12:56
whatever, and teach their own. Right.
12:59
What do we do now? Like, how do we start, how do we
13:03
start to fix this? Yeah, yeah, yeah, I think, I
13:06
think that's a, I think that's exactly the right question and
13:09
the one of the I said I that I think about the sort of one of
13:14
the comparisons I think about it is is actually get Hub when
13:18
people talk about GitHub as soon as you know.
13:20
I mean, nowadays, it doesn't have a huge number of
13:24
competitors. But you know, there used to be a
13:25
lot of people doing Source control and get Hub, sort of
13:29
emerged because it did something else really well.
13:34
Which is it, it actually allowed people to be Be agile.
13:42
It allows someone to be agile and all the principles of agile
13:44
were like, you know, you've got this idea of like creating like
13:49
new branches and reviewing those branches and like you could do
13:52
like really great code reviews and make people like make sure
13:55
the right people are are sort of in the loop and you can like and
13:59
you can physically see what's changed and and like you have
14:03
alerts and monitors like you have all this basically have all
14:05
this like really great infrastructure on how you move
14:09
quickly safely. And it sort of it sort of
14:13
reminds me kind of of where we are in data right now.
14:16
Like when the cloud unlocked is our ability to to move really
14:21
fast to your point. A little brother just wants to
14:24
that right? Does the cost of that and it
14:28
seems like the next step is to kind of regress back a little
14:32
bit. Like we need to we need to put
14:33
the to say your phrasing to put the genie back in the bottle a
14:37
little bit but we still want to move fast, but we just need To
14:40
have the the safety guard rails in place that allow us not
14:47
necessarily to have like an extraordinarily well modeled
14:49
system from day one because I think it's just impossible given
14:52
how startups are formed these days like hired you know six
14:56
Engineers zero data people and and like maybe a product manager
15:00
and a designer so it's just it's just not feasible to actually
15:02
have a great data model from day one.
15:04
But what would be great is if it was very simple to like as these
15:10
Scalability problems start to emerge.
15:13
You can modify your data model easily and quickly and do it in
15:17
an iterative way and the more, the more your business scales
15:21
and the more senior data people come on instead of flooding them
15:25
with service tickets instead, they sort of focus on they take
15:29
this like GitHub like approach, they're like, hey there's some
15:32
re factors that need to happen. Here's what needs to happen, I
15:34
can do it quickly, I can do it in my sort of, you know
15:37
incremental part of the model over time.
15:40
We can figure Route. Like, how do these various
15:42
business entities, relate to each other and, and improves
15:46
incrementally. I think that that's the type of
15:48
thing that has to happen at Convoy.
15:51
We've been trying to do this in a pretty manual way, so far
15:56
through the idea of like contracts data contracts, which
16:00
we can. I mean, you wrote a post about
16:02
this? I'd love to talk about it more.
16:04
Yeah, I sure contracts with this sort of abstraction on top of it
16:09
of what I've been calling. Data design or like, you know,
16:12
or like data ux, which is literally like a design surface
16:16
for your data model that people can add to collaboratively like,
16:19
okay, I've got an entity called a shipment or I've got an entity
16:22
called a shipper. I've got these like 12
16:24
properties and I actually need a thirteenth foreign key, so that
16:28
I can answer like some question. And I should be able to add that
16:30
really fast. Someone should review it.
16:33
That should be enforced through a contract flows into the
16:35
warehouse, and then I can really easily and quickly answer my
16:38
business question and it's all safe.
16:40
And I think All the tools to do that stuff exists today.
16:43
Like it doesn't actually need any new, like net new advanced
16:46
technology or anything. All the stuff is there is just
16:49
is just applying it process workflow, like all, all that
16:52
kind of good stuff. Yeah, yeah, for sure, say it was
16:57
like, I had questions and thoughts and they all just got
16:59
away from me, like, all of a sudden as it happens.
17:01
That happens to me all the time. How about the oldest fish?
17:04
It's just story, a story being a human, you know, it's just it's
17:06
just water straight, so I guess, I guess Jack.
17:09
What is awake? Are you?
17:10
Saying is like, we don't need to throw the baby out with the
17:14
bathwater like we don't need to like, you know, obviously
17:16
obviously no one is going back to the old days of whatever,
17:19
right? I guess.
17:22
So what you're saying is, we can layer on contracts.
17:26
I think the contract contrary is a fun word for me, contracts,
17:28
that is a new term for me. Because when you say contract, I
17:31
hear in my head schema because I because I live in, like the land
17:35
of like protocol buffers and and apis and stuff like that, right?
17:39
But obviously, schema is an overload.
17:40
Term in the data warehouse world, if you say scheme of the
17:43
people are gonna be like, Oh, you mean like let's game.
17:44
I like a namespace in a database and I'm like, no, it's not
17:47
actually not what I mean, right? So contract.
17:49
So, I just, I love contract as a word for this just to be clear
17:52
and disambiguate the, we're not talking about like, a literal
17:54
database, schema, right? You're saying, this is a thing.
17:58
We can layer on to our existing data warehousing system.
18:02
It's not a thing. We need to let go.
18:03
Okay, we screwed up because I mean, there's just a very human
18:08
instinct and you see this with like the DBT people are
18:11
currently like completely rewriting their IDE from
18:13
scratch, right? To just like let's just throw
18:15
this out and start over. Yeah, I mean and I get that I
18:19
empathize with that but I just feel like in my career I have
18:22
learned that that is like 99% of the time the wrong thing to do.
18:27
You know? So, is that what you're saying?
18:28
Like this is something we can, you know, gently introduce
18:31
basically. Exactly.
18:32
Like I think, yeah, I think that pretty much every business.
18:37
I've talked to, about this and Convoy included, Usually has
18:41
some set of use cases where contracts are in sort of
18:47
enforced, schemas are incredibly meaningful in our case, it was
18:52
it was our pricing model, right? We had pricing model when the
18:56
most important models and convoys business is responsible
19:00
for like designing the the price of our auctions essentially.
19:04
Like when we when we, when we stopped an auction and The data
19:10
was not good, like we were, we were dropping, I think 10% of
19:14
the rows that were feeding into the our training set.
19:17
And that was because there was a tremendous amount of
19:20
Transformations, there was a lot of stuff that was just randomly
19:22
getting like columns and were being dropped, Upstream column
19:26
names that were getting changed. Like the engineers, like,
19:28
weren't really aware of what was going on Downstream, and in the
19:31
reason that was happening and because we took this approach
19:33
of, okay, like software engineer, is you guys don't
19:36
really need to think about the data that you're emitting.
19:38
Just going to build a pipeline that pulls everything that
19:41
you're capturing and dump it into the into our storage
19:43
environment. And we're going to build all of
19:44
our training sets on top of that.
19:46
Yeah. And turns out that's not a good
19:48
idea. That's actually not it.
19:51
Not a great idea. But, but if we can, if we can
19:54
say, hey, like, what? If that didn't happen?
19:57
What if we, what if we could make it so that this data, that
20:00
data was actually incredibly trustworthy, that there was a
20:03
clear owner to it, it solved a very discrete discrete problem
20:07
and and And the actual consumers of the data, the data
20:11
scientists, they understand, the data that they need that.
20:14
Like, I know what I need to power this model.
20:18
I just need someone to implement it for me in the way that I
20:22
want. And I need it to be trustworthy
20:24
and I need to arrive on time, and that to me is the purpose
20:26
that the contract serves. It does all it does all of that
20:29
through schema. And and you can, you can
20:33
introduce this in a total totally iterative way.
20:37
You'll one contract at a time, Solve a business problem, you
20:40
know, get the value, move on to the next one.
20:42
Totally totally. I love that.
20:44
This is an interesting. I also don't call like a
20:46
disconnect Chad. I think between us that I think
20:49
a lot. I think about a lot is for my
20:50
course, as well, right? Which is that I guess I learned,
20:56
you know what, back in my Google days, right?
20:58
So I was hard to Google like way back in, like 2008 as a
21:01
statistician, but as soon as I got there, all I did was write
21:05
code because basically no one stop me more or less, right?
21:09
Which would be like, I wanted to analyze some data about how an
21:13
ad auction worked. And so I went into the ad server
21:18
and rotate a coder vest and I added like the data that I
21:21
wanted in the Structure that I wanted it to do my Downstream
21:25
analysis and, you know, send it out for code review and like had
21:28
it reviewed the logs team. Because like the time, like, if
21:32
you added like a single input field to the logs, that was like
21:34
an extra terabyte of data every day, which was just got a great
21:38
right and got it approved and my log started flowing.
21:43
And I got my data and I have for better or worse like taking this
21:49
attitude of like it's sort of a you know it's a what is a game
21:53
of Own style. Like I will take what is mine
21:55
with Fire and Blood kind of thing to most of my career?
21:58
Like I If you're not going to give me the data I need.
22:01
Well, I'm going to go into your system and I'm just going to go
22:03
get it and while I'm there, I'm calling.
22:06
It affects a few bugs. I find because I'm to be
22:08
perfectly honest with you. Like, much better in engineering
22:10
than you are. And that's fine and like that
22:14
that's the way that it works and a small companies that's kind of
22:16
like that's the way because there is no you know, that's
22:20
just the way it's done, right? But it's coming to get larger.
22:24
Like I don't scale, you know, I mean like you can't and you know
22:28
God willing I shouldn't be allowed to scale, I guess.
22:30
Basically you need you need this, you need a process.
22:34
You need like unique contracts. You need workflows, you need
22:37
this too because again, to handle this kind of stuff and I
22:40
guess like, I have never succeeded.
22:42
Especially made that transition a my career like I a better, I
22:45
am very much myself. So I don't know how would
22:48
something like this work, you know in a big company where it's
22:52
like, there are too many people and you can't just go like
22:54
cowboy this kind of stuff. So I'm kind of like that's
22:57
that's the part that's Most Fascinating to me about like
22:58
what you're talking about. Yeah yeah.
23:01
Yeah I mean man I would love it. If you would love to get Convoy
23:08
I would be like I'm right. I could be like your ninja.
23:10
You would just like you, you're amazing.
23:12
Think you would be making a convoy?
23:14
That would be it's so funny that you say that because that, that
23:18
is traditionally been my experience.
23:21
That's been my experience as well.
23:22
It's just such a difference between like companies have
23:25
different scales and like and and it's something I didn't, I
23:30
didn't even realize until I joined Microsoft.
23:32
And then I joined Convoy after that of, you know, in some
23:37
places the the there's almost like a Stockholm syndrome type.
23:42
Thing going on with 80's where they did like I'm so like I'm
23:49
not allowed to go up and touch this database or whatever.
23:52
I'm and so I just need to accept everything and as I told the, I
23:57
called the Oliver Twist thing. It's like the please, sir.
23:59
May I have some more kind of thing.
24:00
Exactly, please, please data producer.
24:02
Please, please, let me have your data.
24:05
I'm pleased to meet you discuss. Where on here, I've been
24:09
swearing for a while, is that okay, okay, so I'm like, fuck.
24:12
That shit. Fuck that.
24:14
I am going to go in and get it. I don't care.
24:17
This is this is how I was taught.
24:19
This is how it this is, how it was done.
24:20
Like I don't like, I don't know, I want this to happen on my
24:24
timeline. Not now, your timeline like
24:26
anyway. And honestly I think I think
24:28
that like I think that is so the thing that I have felt sort of
24:33
as a data platform lead. Is that the what's really
24:36
missing is the interaction between the producers and the
24:40
consumers. And if the If the consumers just
24:43
going to go up into the where they're gonna go up into
24:45
production and make some change and the producer like doesn't
24:49
like that great like that's now a conversation that can happen
24:52
and we can talk about why we like this data is important to
24:55
me you aren't you? Like you aren't getting it to me
24:58
the way that I wanted. So I did it.
24:59
So if you don't like it we need to talk about our options here
25:02
because I need to run my model and you know whatever.
25:05
But but what I find is that, you know, I don't know if it's
25:09
almost like a meekness or sort of an acceptance of like Of, you
25:13
know, I can't get the stuff that I need.
25:14
I ask people for this, they don't give it to me.
25:17
And so instead, I'm going to spend a month like, you know,
25:19
writing SQL and snowflake or doing some crazy analysis
25:23
instead of just going and getting exactly what I need.
25:25
I'm gonna exactly, I'm gonna build a machine learning model
25:27
to impute. The missing features that I need
25:30
to train. Like it's just like this is
25:31
absurd. Just go get the goddamn data.
25:34
Like what is wrong with you, right?
25:36
But I think what I feel like we're saying here, and I think
25:38
this is important and this is the part I lack, right?
25:40
Is like, you basically need a carrot, and A stick model.
25:43
Yes right. Like I'm the stick and you are
25:46
the carrot because I'm just like if it's just all carrots, it's
25:50
not going to work, right? But if it's all sticks, it's not
25:52
going to work either. Because I mean again inevitably
25:54
I get fired because you know, someone takes me out like I you
25:57
know it's right every ninja goes down eventually no one.
26:00
No one survives forever, right? It so I mean that's that's the
26:04
thing is you need that sort of pairing.
26:06
Basically, you need to figure out like, what is your carrot
26:08
stick strategy for moving the stuff for to making this happen?
26:11
That's yeah, that's yeah. Yeah, you're right.
26:13
And I think I think that's so here's sort of an interesting
26:17
thought that I've been having, and I'm still trying to express
26:21
this in a way that's like not confusing and people don't yell
26:23
at me because I get yelled at a lot on LinkedIn and I love them
26:26
out there, so you do and it's, I thoroughly enjoy it.
26:29
Like you. I love your post with the best
26:31
at the comments are fantastic. Anyway.
26:32
Okay. Yes, please.
26:33
Yeah. But so, so, one of the things I
26:36
sort of, i-i've been thinking a lot about this, like GitHub
26:41
comparison, And I think at first, when I was approaching
26:45
this problem at Convoy, I was approaching at, basically, the
26:49
way that you were saying, which is a data scientist or someone
26:52
would come to me and say, Hey, you know, what's going on with
26:55
this quality or I don't understand why it's so bad.
26:58
I was like, well, you're a software engineer like you know,
27:00
in you care about the quality so you like it should be on you
27:04
like oh go do it like oh yeah. It done.
27:07
And yeah. But but there was like a real
27:10
hesitancy. This is there's a real hesitancy
27:14
to do that and I think a lot of it has just come from
27:19
organizational design of a lot of it has come from sort of
27:22
severe separation of concerns like mounting expectations of
27:27
like, you know, we don't we don't the business like doesn't
27:30
care if you put a lot of time into Data quality and sort of
27:34
data governance and things like that.
27:35
We just want the model and we just want, you know give us a
27:38
report and so I need to like, maximize my time as a data
27:41
person too. To deliver those things.
27:43
And now if I have to add sort of quality and ownership of some
27:47
Upstream, like production thing into my workload, that that's
27:50
going to be a net negative for me.
27:51
So I need to find someone else to do this, and the software
27:54
Engineers aren't listening. So a lot of the times like those
27:57
things get, redirected to the data engineering team like, hey,
27:59
the pipeline broke, go fix it, data Engineers, I can't yell at
28:02
software Engineers, I'm going to yell at you instead, thank God
28:04
because you have to listen to me.
28:06
Then that is so true. Okay good.
28:08
Oh yeah, yeah, but I all I was going to say, is like what?
28:11
That's kind of Led me to feel is that the sort of GitHub based
28:17
approach doesn't actually work for data.
28:24
If the consumer, you know, if they if they care about the
28:29
color, they care about the quality.
28:30
Then that's great. And then it does work.
28:32
But if you're living in a world where they don't care, and
28:34
they're basically just saying, I just want you to give me like
28:38
this particular data asset, like when I need it in this form at
28:41
this time. Then then then then code may
28:47
actually not be the right level of abstraction for that.
28:50
Okay. And and so we're sort of moving
28:53
to a world that Convoy where what we're trying to do is
28:57
abstract the code and instead focus on the semantics and like
29:03
the data itself where and we're in someone someone might be able
29:08
to say, for instance, there's like a real world thing that
29:11
happens. Let's say a shipment has been
29:14
canceled, and I want to know every time the shipment has been
29:18
canceled and I want to know that in real time.
29:20
And here's what a cancellation means to me.
29:23
And there's, like, six different places that there's like, the in
29:26
a life cycle. There are six different places
29:28
where that cancellation could occur.
29:29
I want to know all of them. And here's like, the, the
29:33
properties about those cancellations that are that are
29:35
meaningful to me, all of that can actually be described using
29:40
English and semantics. Of course, I know.
29:43
And if you if you're able to capture that as schema and then
29:47
either an engineer or data engineer, or whoever else, you
29:49
have some agreement and that's something we can talk about.
29:51
But if there's some agreement to actually implement this, as a
29:54
contract, then once that data flows into, you know, your leg
29:59
or your lake house or your Warehouse or whatever it is you,
30:02
you can essentially map the semantic meaning to like the
30:07
actual code in the warehouse, there by abstracting the code.
30:12
So if now, if I Make a change. I can make a semantic.
30:15
Change the engineer understands how to do that from like this,
30:18
the schema sort of code perspective and I can go and
30:21
train my model without even needing to think about like,
30:24
okay, I need to go into production and write something.
30:27
I'm Emmanuel. Mason machine learning engineer,
30:30
at stripe and author of building, machine learning,
30:32
powered applications. And if you don't want your
30:35
machine learning models to explode, well, you should
30:38
subscribe to this podcast. Did you?
30:41
You said so much and I you have to forgive me, I'm going to
30:44
comment on like three different things.
30:45
You said because they're each one's own words.
30:47
You said his greatness? Fantastic is a just it's
30:50
basically like if I could just take that your little like the
30:53
speech, you just gave there and just turn that it's like an ad
30:55
for my data. Engineering 4ml course.
30:57
Like that would be like a deal like this because we want to
31:00
make sure you hit on all the things which is that?
31:02
What's up? What's up bud?
31:03
Will make a clip for you? I appreciate that.
31:05
Thank you. Thank you, mr.
31:06
Speaker will be grateful. First thing you said was I think
31:10
which It's really like, the reason I want to teach a course
31:15
is because there is some things about the reality of doing data
31:19
that cannot be talked about in podcasts and cannot be written
31:23
about in blog posts, which is like the implied, social
31:28
dynamics, you know, like cultural hierarchy of various
31:31
engineering teams at various companies.
31:34
Like, this is a very real thing. Anyone who does this stuff for
31:37
real can, like, talk about this, right?
31:39
But we but we never talked about it.
31:41
We never talk because it's is very touchy.
31:43
It's very sensitive, right? It's not something that can be
31:45
like discussed out in the open, you need like, I mean, I don't
31:49
is that you need like a group therapy session or something and
31:51
that's to a certain extent like what my course is is like, it's
31:55
a place that's private, you know, we're like the entrance or
31:59
screen from practitioners like we're not letting you know,
32:01
like, where we can have these sort of conversations about
32:04
these things. And how do we deal with this,
32:05
right? The second one I want to say and
32:08
again this is just again, like advertisement for the Two topics
32:12
for the first session. My course are one data
32:14
collection, right? Obviously, because it's so
32:16
foundational. And the second one is data
32:18
quality, and I think I have like a lot of people, I have very
32:22
much have the challenge of, how do I sell data quality?
32:27
Like as a thing as a thing we invest in as the thing we like
32:30
to talk, you know like we put money into and I think what was
32:34
funny was I've been reading up on how different companies
32:37
handle this and I love, love, love Google's, right?
32:41
Wrote this wonderful paper about their data validation system.
32:44
I don't know if you've come across this is it's actually
32:45
like it's actually largely open source, it's in the tf-x the
32:49
tensorflow extended system it says data validation Library
32:52
they wrote and basically I'm so again talking to my friends to
32:55
Google it is like taken over a google-like data, validation
33:00
data quality with like clear defined schemas specifications
33:05
for like what values are allowed for different fields with what
33:08
tolerances is everywhere. Is everywhere.
33:12
They use it for everything, like all kinds of stuff.
33:14
It's completely taken over there because because the data quality
33:17
stuff, it does really like three different things.
33:19
One, it's documentation, its docket.
33:22
Like what does this model expect?
33:24
What inputs does it need to see for it to work correctly, right?
33:27
That's license documentation for new Engineers, exploring new
33:29
models to. It's actually like they use it
33:33
for testing, they will use the schema and the specification,
33:37
the contract, if you will to auto-generate training examples,
33:41
Is that are designed to push the limits of the schema.
33:45
Like let's intentionally throw some weird data into this model
33:48
and see what happens and see what breaks and it catches bugs.
33:51
So it's like fuzz testing effectively, right?
33:53
And the third thing, and this is the most important thing and
33:56
just this is like to me, this is separates like people who can do
33:58
ml from people who can't Google has learned through trial and
34:02
error. And lots of years of mistakes
34:04
that there is free money and doing a better job of
34:09
constructing your training data sets.
34:12
By using data quality checks to expansively test extreme values,
34:17
rebalance, your classes. There's free money in there.
34:20
You don't need a better architecture, you don't need new
34:22
features, you don't need anything.
34:24
You can do a better job of constructing your data set,
34:27
using these data quality metrics, as a guideline and get
34:30
like a three or four percent left, like you be insane.
34:34
Not to do that. It's like literally free money.
34:36
Anyway, I'm sorry, I shut up about this, but like this is
34:38
this is the stuff I want to talk about, because it's not getting
34:40
talked about And we and this is my opportunity to try to fix
34:43
that. Yeah.
34:44
Anyway I'm sorry there there is something that I wanted to talk.
34:47
I wanted to touch on which I think is a bit tangential to
34:53
this, but stay with me and I'll bring it back around and it is
34:58
going back to this point where you've got some data that maybe
35:03
has been. It's been tainted along the way
35:06
along its life cycle along its journey.
35:09
And that's because somebody Upstream Something with it.
35:12
How do you make sure? Or how can you get someone any
35:17
time that they touch data or they mess around with something
35:20
in the database? They know everyone that it's
35:23
going to affect like those second and third order effects?
35:27
Yeah, totally. Yeah, yeah.
35:30
So so so, so, okay, so there's a few things that I think are
35:34
really interesting to talk about.
35:36
I think the the core of your question, the core of the answer
35:39
is lineage The problem with most lineage systems today is that
35:43
it's just metadata a collected off of the warehouse like you
35:46
pull out so snowflake metadata and you can build a lineage
35:49
system based on that. And you can see how various
35:51
tables are connected. But to your point, there's
35:54
nothing that actually ties all the way back to the services
35:56
themselves. And if you had that even if you
35:59
went that one additional step, it actually becomes an order of
36:03
magnitude more valuable for like three different reasons.
36:05
Like 11 is sort of what you said which is if you are a data
36:10
producer today. Producers have literally no idea
36:14
what's going to happen if they do anything.
36:15
Like they drop a column there, like I don't know, like I don't,
36:20
who knows. Um, but it, but it seems like
36:22
you have some sort of lineage component and and that that, you
36:27
know, I could go quite deep at this actually.
36:29
But if you sort of combine this idea of contracts and lineage
36:32
together, then then the the data producer could essentially know.
36:37
Like if I am going to drop a A drop of property or change a
36:42
name I can go out. Check my lineage, see like who
36:47
are all the consumers of this who and go all the way
36:50
Downstream. And then be like listen if you
36:53
if you make this change and you do it in like a non version,
36:57
controlled way. You will break our pricing model
37:00
like you will break the training set for our pricing model and
37:02
just having that level. Like being told that you're
37:06
going to break something before you break it is, I think would
37:10
fundamentally Change how software Engineers interact with
37:14
data. It's not that like producers
37:18
don't care like they would just break the pricing model.
37:20
Even if they knew that it would happen they did they just they
37:22
just don't know what's going to happen and and and and how could
37:25
they like they're not. They're not really brought into
37:26
that that part of the data life cycle at all.
37:30
So yeah, I think that would be really important.
37:33
The other thing is like those not really there's not really
37:36
the idea of a like a backwards incompatible change when your,
37:40
when your life managing. Being database database changes.
37:43
Like there's a, there's the idea that chains like sort of within
37:45
the service, but not within, you know, like what's going to
37:48
happen to all the consumers of this, which is another area.
37:51
I think data contracts is really amazing because you can
37:54
essentially say like, hey, we understand what this schema is
37:57
supposed to be. There's a contract here and the
38:00
change that you're going to make is going to be backwards
38:02
incompatible. Like are you sure you want to do
38:03
that? Is this really the best?
38:05
The best thing for you to do? And if it is, if you are going
38:09
to have to make an incompatible changes, We should at least tell
38:12
people what's coming and and be able to communicate to all the
38:15
to all the consumers like Downstream.
38:17
So like that would be, I think massively beneficial.
38:21
Let's say a couple of things there like um is this sort of
38:24
like riffing on which I was saying?
38:26
Well one is that you know so it's slack.
38:29
You know, first of all most we rely primarily on our like are
38:32
structured logs. Like we have Thrift, you know,
38:34
Thrift schema for are structured logs.
38:36
That were designed to go in the data warehouse and that was
38:38
obviously integrated with code review, integrated with Eicd.
38:41
Checks such that if you introduce a backwards
38:44
incompatible change to the logs, the cic Jack would fail and
38:47
there goes my dog barking that's good.
38:49
When he goes, he probably wants to get up.
38:50
I'll go get her the cic D check would fail and exactly that you
38:56
could again it could be overridden but you had to have a
38:58
conversation with people before you did it right.
39:00
That kind of thing too is like It's certainly the case that
39:04
people like unintentionally, like, through ignorance or
39:07
whatever do these changes and stuff.
39:09
But in my experience, the bigger problem is really just like like
39:13
honestly bugs more or less like no one meant.
39:16
No one meant to break the thing, right?
39:18
They didn't mean to break the pricing model, but they
39:21
introduced a bug. And so, therefore, you know,
39:23
this RPC call like stopped, that was actually critical stopped
39:27
working silently and no one noticed and stuff like that,
39:29
right? So I just want to call out that
39:32
like, for me, Experience insofar as Lake schemas and contracts up
39:36
front. Remove just the huge class of
39:40
just the knucklehead stuff, right?
39:42
Like the unforced errors, which is creepy, just key.
39:45
That doesn't mean that you don't need data quality and data,
39:48
validation step, Don't Downstream because again, bugs
39:51
happen, nobody's perfect. This is, this is normal and Zoo,
39:55
but we, but again, we have to be defensive against it because
39:57
it's especially with ML. It's just way too easy.
39:59
Chad, as I'm sure, you know, for like this stuff to just slip
40:01
through and like, no one notices, right?
40:03
Until you end up with like a model that like is just, you
40:06
know, it's terrible anyway. So yeah, Good call.
40:11
I think that's a good point and, you know, to sort of continue on
40:17
that, I do find that in the data community and even the data
40:21
science community may I get theirs.
40:23
I use the term Stockholm syndrome where there is this
40:26
expectation. That like we need to put all of
40:30
the quality checks Downstream. Yeah.
40:32
And and and like you said, they're obviously does need to
40:36
be like, like you don't want to just leave your fate.
40:38
Up to the hands of the Gods and, and, of course, and hope for the
40:41
best. But at the same time, like
40:46
having like having the people who are producing the data also
40:50
having skin in the game and it comes to quality.
40:53
Yes it is incredibly is incredibly meaningful.
40:57
I think one of the questions and and Josh I'm actually interested
41:02
to hear. Maybe I was just about to ask
41:04
you a question. You can't do this to me.
41:06
Come on. I did hear how you sort of, how
41:10
do you handle this at slack? Yeah, it is, you know, the
41:15
biggest, the biggest sort of feedback I get from people when
41:18
I talk about these things and even earlier at Convoy, when I
41:22
was pushing a lot of these ideas was like, well, you know, the
41:26
engineers just don't care, right?
41:28
It's just not like the engineers don't understand that.
41:31
This is something that they need to be able to do.
41:33
It's like an additional like category of work that they need
41:36
to take on. It doesn't actually At their
41:38
service in any way to do all these things.
41:40
So like how is that something that you know?
41:43
Yeah, just adding these quality checks at slack.
41:46
Like how did you overcome that and sort of how do you think
41:47
about it more broadly? That's just that's just a
41:50
perfect question about. It's perfect question again and
41:52
it actually it's funny because it gets exactly to what I was
41:54
going to ask you which it my answer and I'm answer your
41:59
question that I'm going to ask visit my question like right
42:02
after that back back to you. Okay.
42:04
Right. So the answer is that we hired
42:06
Engineers who Heard about that data, and we basically know what
42:11
their title was their title, was machine learning engineer, that
42:15
was their title because the machine learning engineer.
42:18
Unlike literally every other engineer of the company is
42:22
basically the person who is going to both.
42:24
Generate the data Upstream in the production service and
42:28
consume it and process it. Downstream in the data
42:31
warehouse, they actually have to your point skin in the game and
42:36
a stake in the fact that the Data they get for them to put
42:40
their model to work and for their life to be good and happy
42:44
these things need to work right they're incentivized in this
42:47
way. Okay?
42:48
So ml engineering at least it's plaque was the first best
42:54
customer for all of this stuff they wanted the schemas they
42:59
insisted on the schemas right? They needed the schemas.
43:01
It's solved their problem because they again, we're the
43:04
only person who lived on both sides of the fence here.
43:06
Now once we had, The minimal tooling again, contracts, you
43:11
know, all the downstream data processing, all data quality,
43:13
checking, all that kind of stuff for those Engineers that it
43:17
started to grow. It started to expand and then
43:20
like the growth Engineers. So I mean, I would say to me
43:23
like, you know, broadly speaking in any company, there are only
43:25
like three, like there's like four use cases for data and
43:29
three of them are things and Engineers care about.
43:30
Right? One was obviously ml.
43:33
Obviously, second is growth, like growth product.
43:36
Like growth in particular, slack being Obviously, the canonical
43:39
product like growth, the original product, LED Growth
43:41
Company, right? So then the growth engineer
43:43
started counting on to this and, like, working with their data
43:45
scientists. And so, they started built, what
43:47
was great about the growth engines, like for the ml
43:48
Engineers, like again, love ML, and years.
43:50
We built like the most minimal sort of shitty tooling possible,
43:54
right? Just just the minimal
43:55
acceptable, tooling the growth Engineers come along and they've
43:57
got like their front end shops and their full stack, and they
43:59
just start making our tools. They start using our tools are
44:02
like, wow, this is garbage. Like we're going to make this
44:04
much better, right? So they start building the tools
44:07
and adjusting them for their needs.
44:08
And then the last one was really the performance engineers.
44:11
And this is my favorite one was that when we started getting
44:13
very serious about performance because, you know, performance
44:16
and like, cost Cloud, spend, all that kind of stuff.
44:19
The performance Engineers started coming in and then they
44:21
started using structured log to capture like weird.
44:24
Database query edge cases, and like teams were having cache
44:28
misses way too often and they started getting serious about
44:30
this stuff, right? So it all kind of spiraled out
44:33
from there, but I think the, the taxes back where I'm question to
44:36
you, if I don't have those Veneers who were designed to cut
44:40
across the data warehouse. I don't know how to get started.
44:43
My man. I don't I don't know how to
44:45
operate in a world where you're exactly right.
44:47
Because like the regular Engineers don't care.
44:50
They absolutely. It's it again.
44:52
I've been a regular engineer. I empathize they have to worry
44:54
about security. They have to worry about, you
44:57
know, obviously performance. I have to worry about
44:58
functionality, the product manager wants this, like they
45:00
have to worry about 800 million things and so, I feel terrible
45:04
piling on another thing they have to worry about, especially
45:06
if like the tooling is immature. Stuff like that.
45:08
Right? So dude, that is a problem.
45:10
I don't know how to solve to me that you solve the problem by
45:12
hiring ml engineers and making them the first best customer for
45:16
the stuff you're building because they do care.
45:18
It's in their interests. Anyway, I totally agree with you
45:23
so I can tell you. I can tell you one of the ways
45:28
that I've thought about this at Convoy and lead a bit of it, is
45:33
somewhat authoritarian, I guess. So, that's great.
45:37
I love my bed. I'm a big.
45:38
I'm the stick man. I'm with you.
45:40
Let's do it. Karen stick.
45:41
Let's bring some stick. What do you got?
45:44
Yeah, yeah. So so so, so I've basically
45:49
found success in into two, main ways.
45:51
Good way is, is actually with the underwater data engineering
45:58
team and where the data engineering team right now.
46:03
There are, you know, a lot of them are software Engineers.
46:07
They're building these Lines. The the stuff breaks for reasons
46:13
outside their control and they are essentially sort of caught
46:17
in between, you know, the pincer of like the software Engineers
46:22
who are owning these like, Upstream things and sort of the
46:24
downstream folks that are just causing them all sorts of pain.
46:27
Yep. And that was actually, where a
46:30
lot of these ideas originated. A convoy was the data engineer,
46:34
sort of, like, can we just go and solve this problem
46:36
ourselves? Like can we just go in and very
46:40
very easily like create like, add the data that we need, add
46:45
the data that this Downstream team needs, build it as a
46:48
contract and then hand it off to that software engineering team
46:51
and say hey if you make any additional changes to this, just
46:53
understand that like there is some process in place, like
46:55
there is some CI CD workflow. Where if you make a backwards
46:58
incompatible change, like you're not gonna be able to do that or
47:00
you have to alert people and and generally speaking like that
47:03
additional level of ownership which is basically just owning
47:06
an API Like not super painful for infertility things.
47:11
Exactly right. Yep.
47:12
And and so that was actually really helpful in the are the
47:15
data engineering team, love that and there was a lot of use cases
47:20
when I remember we were trying to get this rolled out to
47:23
certain to software, engineering groups, sort of in the organic
47:26
way that you were talking about. There was just sort of gradual
47:28
adoption. Is people realizing oh man these
47:30
contracts are awesome. And sometimes the software
47:32
Engineers when they were first starting to adopt was like, Hey
47:35
we're going to budget, you know, a week or maybe a Whole Sprint
47:38
for doing this work. Even though, it might just be
47:39
adding one or two properties to an existing thing.
47:42
And the data engineer said, yeah, I'm gonna go do it myself.
47:45
Like, I'm just gonna, I'm just gonna go in.
47:46
I can do this in, like, 10 minutes and then it's done and
47:50
that happened a lot. So, luckily exactly the same
47:53
time, you know, I see so many places where like, you know, the
47:56
data team and the prod team aren't even in the same AWS
48:00
account. Like, it's bizarre.
48:02
Like, literally physically, they're like p.m. again
48:04
physically in the before times, right?
48:06
In the before times, they like, Babysitting like 10 feet apart,
48:09
but like technology-wise, they would be living in like
48:12
completely and utterly different universes.
48:15
Well, they're galaxies away. Yeah, precisely that like, the
48:18
production system and the data system, you know, to your point
48:21
like, Chad live in completely different worlds.
48:23
So I'm curious for you, like the first time this happened with
48:28
the data Engineers. Like, how did that go?
48:30
Who do they talk to? How did that work?
48:32
Was there was there a meeting? Was it like, just a slack DM?
48:35
How does that work? Yeah, so So so there's so the
48:41
the type of tooling that we build is this probably similar
48:44
to maybe what existed at slack. But like, we essentially
48:47
created. So to there's actually two
48:49
things and this is going to sort of help that I can explain the
48:52
process that happens from as a result of this, as a result of
48:55
the tools, is we essentially essentially have sort of two
48:58
things we have one. One thing that's like an SDK
49:02
it's like a we don't we don't use like protobuf or Avro at
49:06
Convoy so we have like an Colonel sort of Ideal that we
49:09
built. It's like very similar to like
49:11
Json, schema. Like, what is it?
49:12
Yeah, yeah, Yucca nice. It's just one schema right on.
49:16
Yep. And and and so we basically
49:20
created a library put likes eicd checks around that made it like
49:26
really, really simple to event to like admit these things that
49:28
we called events but they could just be, you know, like a
49:31
database change could be an event.
49:33
Theoretically, we were actually mainly focused on this other
49:36
idea which I can I can get To, in a little bit, later called
49:38
semantic events, which are like, capturing, real, things that
49:41
happened in the world. And there's, there's a reason
49:43
that we focused on that, that is kind of an answer to the
49:46
question. You just asked the second ago.
49:47
And then the second piece that we built was actually a, a
49:53
abstraction of the, of the production table itself that,
50:00
that sort of lived in the string processing layer.
50:02
And so, the idea was, if you're an engineer, you could basically
50:06
wrap Are you know production table in this abstraction you
50:10
could say I only want to push these like seven or eight things
50:15
that are within a contract into the warehouse in sort of a
50:19
structured format. Everything else I can just
50:20
continue to dump and not think about and then and there's like
50:26
some really interesting things that go along with that.
50:28
So in that way, we are sort of capturing as events like as as
50:31
sort of clearly defined schema with a contract.
50:33
Every single update to sort of the important entities within
50:37
our database Basis. And then we also had sort of
50:39
these back and events that were being emitted from the services
50:42
by the software Engineers or the data Engineers, if it was those.
50:46
Some so the goal through like those two pieces of Technology
50:49
was like we want to make this as unbelievably simple as possible
50:54
for anybody to do that understands like you know like
50:57
you can write JavaScript or typescript or something like
51:00
that, it's just real easy. And then the pipeline is
51:03
actually auto-generated. So, we would say, okay your we
51:06
would check the contract. You create a new Kafka topic
51:09
push that event via Kafka into the warehouse, we'd
51:13
automatically parse it like once it landed in Snowflake.
51:15
And so you sort of then you can actually we did.
51:18
It built another thing that's like a catalog that sits on top
51:20
of that essentially links the the contracts to the schema
51:24
registry. And we're in very exactly the
51:27
underlying like snowflake table Tom and so this sort of a clear,
51:29
a clear line of sight there. And so what happened after we
51:33
sort of had these two pieces was that we had Add a data data
51:39
science team. Oh, and this is another piece of
51:41
context has data engineering. A convoy is actually like a
51:44
centralized organization. They're not embedded.
51:46
They're actually technically part of the software engineering
51:48
engineering organization. They're not a part of the data
51:50
organization and so what happens is that the a the data science
51:55
team would say we've got, we've got some data that's breaking,
51:59
they would sort of make their traditional complaint to date,
52:01
engineering like this is a problem.
52:03
Can you guys go fix it? And what we, what we did is we
52:06
initially work with them to say What is the data that you
52:08
actually care about and that you need?
52:11
Do you know what, the cert, what service that this data is coming
52:14
from and if not, we can go figure that out for you.
52:16
We went out and basically did that we said, okay, we figured
52:19
out that like the data you need is coming from this service and
52:22
the way that you the way that we can express the data set you're
52:26
asking for is actually through like maybe three or four
52:29
real-world semantic events. Like you want to know every time
52:33
that a new auction is completed every time like a new auction.
52:37
Ian has there's like a bid is made on that auction and every
52:42
time like some customer, I don't know like like makes a bid like
52:47
they press a button in the app to say, like lets you know, we
52:51
want to have this auction and then we need to add like three
52:53
or four new properties on the actual like auction entity and
52:58
would like that combination of data.
52:59
We can hand that off to like a business intelligence engineer,
53:02
and they're going to be able to derive, like, whatever data sets
53:04
that you need totally do data engineer went in, they
53:07
implemented, In like a day, the data flow flow through in real
53:11
time and then the bi engineer was able to like since it was
53:14
all properly modeled in advance, it took them like two or three
53:17
days to build the data set and boom, easy easier to build the
53:20
model cheaper to build the model.
53:22
So much cheaper to build the model, right?
53:24
Because everything is because I mean the cheapest time is like
53:27
one of my things I harp on like the cheapest time to do a join
53:30
is the moment. The data was created by the
53:33
point everything after that streaming snowflake way more
53:37
expensive. Of way more expensive than the
53:38
joint. Then do it when it's cheapest.
53:40
Oh my God. If you can, I love that man.
53:43
That's fantastic. I love that.
53:44
That mean that I mean that data engineer you're talking about
53:46
whoever was doing that. I mean that sounds like that
53:49
sounds like an ml engineer. I wouldn't I wouldn't under that
53:51
person's name lest you know, the recruiters like descend upon
53:54
them. Yeah, after hearing this, I just
53:56
keep that one to yourself. Yeah.
53:59
For sure. So there's one use case I want
54:02
to dive, I want to give you which which is really helpful in
54:05
thinking about this and we discovered That later is there
54:08
is actually a product use case here too.
54:12
And the product use case that we found what.
54:14
This this is sort of why we sort of been pushing for this.
54:17
I like this idea of semantic events and we found that a huge
54:22
number of the questions that like both data scientists and
54:25
product managers want to have answered is like I want to
54:31
understand in a in a sequence of faint.
54:34
Like what are all the things that happen to a specific
54:36
entity? The in some order.
54:38
Yep. And then and then, and then
54:41
like, there is some set of like real world funnels that exist,
54:44
and then I basically want it. So an example of this might be
54:46
shipment cancellations. It is really, it was really hard
54:49
at Convoy to figure out, you know, who, like, who is
54:54
responsible for a shipment? Cancellation, is commonly
54:56
responsible for that cancellation.
54:58
Do we like mess up some where did we get the wrong information
55:00
from a shipper like like why did this person cancel?
55:04
And the only way that you can know that is by like looking at
55:07
the law. Ugh.
55:08
Yeah of everything that happened in the sort of history of that
55:10
of that, of that entity over time.
55:13
So the and this was like, a huge product question.
55:15
Like this is something that, you know, if you can answer that
55:17
correctly, you know there's this is like millions of dollars that
55:20
we could potentially be saving and we didn't have a great way
55:23
of doing it. We're like well you can do it
55:25
through contracts. The way that you would do it is
55:28
you would say, let's look at the life cycle of this particular
55:32
like shipment event of this particular like shipment entity.
55:35
Let's like very, very clearly document like All the things
55:38
that would be germane to cancellation.
55:39
That would like, cause a cancellation.
55:41
Let's instrument that, like in the code itself through a
55:44
contract clearly defined schema well modeled in advance.
55:47
And then once it flows into the warehouse, we can basically
55:51
generate that sort of History table about particularly.
55:54
Yeah. And and then, and then, and
55:56
then, all you have to do is like it becomes trivial that point to
55:58
solve that. So we actually found that a
56:01
bunch of product teams, started asking their Engineers for that,
56:04
and then, and then contracts with a mechanism.
56:07
Of implementation. That's super smart, man.
56:09
You can get really. If you can call a product
56:11
manager, that they can have a funnel for something.
56:13
Oh my God. They'll do anything you want.
56:15
That's that's, that's dreamy. You want to call it?
56:17
Here is like again, like one of the themes of my course, I'm
56:20
sorry to be like a broken record about the stuff.
56:22
Right. Is really, is about really
56:23
relationships between engineering teams.
56:25
And the funny thing is like, is this this pattern of analysis of
56:29
like, the sequence of events, occurs all over the place, they
56:33
call it, like customer Journeys, over in marketing land, right?
56:36
And then, but it's Same time in like gland of observability.
56:39
This is this is traces and spans.
56:41
This is, this is like, what is the life cycle of a request?
56:44
When does a request die? What's low, right?
56:46
Yeah. And so it's all the same thing
56:49
and the joke to me is like, literally it's the same thing.
56:52
It's the same technology now for the your examples, perfect.
56:56
Because it's like for these custom things where it doesn't
56:59
fit naturally, like an e-commerce, you know, conversion
57:02
workflow or like a trace on a request.
57:06
The data teams are left with constructing these custom
57:09
systems basically to do these, you know, exactly this kind of
57:12
stuff, right? And the funny thing is like, the
57:14
price point of like constructing this funnel changes
57:17
dramatically, even though it's the same problem, you know,
57:19
like, the engineering challenge is essentially identical.
57:22
But like, yeah, for observability.
57:24
It's like not that much money and for marketing, it's like,
57:27
you know, extreme amounts of money.
57:28
Like that kind of thing. I'm sorry.
57:30
But yeah, it's, it's just funny to me and I love anyway.
57:33
I love these conversations because we get to talk about
57:35
this. Officer Van doodle with
57:37
Demetrius were. It's like 10, 20 and I got to
57:39
get going. I'm sorry.
57:40
I know, I wish we could talk. All I know, you know, I know you
57:44
got to go. I appreciate both of you coming
57:47
on here. I got to be a fly on the wall
57:49
and just watch you two chat about this stuff, which I was
57:52
hoping to do. I'm so thankful for this
57:55
happening and maybe we can do it again.
57:59
Sometime never know, right. Just ping me randomly like 10
58:02
minutes before you you want to do it and then I'll let you know
58:05
if I can Okay, if you're free. That is it?
58:09
No, this is awesome, guys. I appreciate this so much.