Latest / Agentic Conversations (formally mlops.community) / Airflow Sucks for MLOps // Stephen Bailey // MLOps Podcast #141

Transcript

Airflow Sucks for MLOps // Stephen Bailey // MLOps Podcast #141 · Agentic Conversations (formally mlops.community)

0:00
I'm Steve and Bailey. I'm a data engineer at whatnot.
0:02
And I drink a whole pot of black Folgers coffee in the morning.
0:08
Just the cheap stuff. But in volume.
0:11
What's going on everybody we are back with another ml Ops
0:14
Community podcast. I am Demetrios one of your hosts
0:18
and today I am joined by none other than mr.
0:22
Ternary data himself, mr. ODS see keynote speaker mr.
0:28
Wyoming Being a rock climber, aka the fundamentals of data
0:36
engineering, author, joão Reis stuff going on, dude, how you
0:40
doing? I am excited because usually
0:44
it's like I get someone like you that I get to interview but
0:48
today I'm Christmas came early man.
0:51
We got to interview Steven and you are here right by my side.
0:54
Making sure that this interview just It hit all of the right
0:59
notes. Yeah, it's great.
1:01
Great. Great Jam that we did.
1:03
So like little less. This was killer, I'm saying it.
1:08
I said it before but I'll say it again.
1:10
This may have been the best word.
1:12
This may have been my favorite one, just because of the
1:14
clarity. And I mean the thing I really
1:17
like about Stephen is he just has a I think a very innate
1:20
sense of practicality to him and it shows both of the discussion
1:24
and in the workflows that he's implemented at whatnot.
1:26
So I think that I always appreciate also being imitated
1:29
trying to be more pragmatic than not or whatnot, pun intended.
1:33
So, yeah, so, I mean, it was definitely a breath of fresh
1:36
air, and he went through how he got to build out the data
1:39
platform, what he does, where he sits, and I just thought one of
1:44
the key points that he brought up, was how it is so clear to
1:49
them what their priorities are like, it's got to be nice to be
1:55
able to have that clear? Fear of a vision at a company
1:59
and as a engineering team, and as the data platform team to
2:05
know what those priorities are and know what you need to keep
2:08
in mind when you're building and then just how he sits in the
2:12
middle and not him him only, but the whole data team and the data
2:16
platform team, they sit in the middle of so many different
2:19
stakeholders. And we mentioned this a lot.
2:22
When you're working with data, you have to have Loop so many
2:26
different people in But this is next level, I think the setup is
2:30
very simple to write this is it? There's moving Parts, but
2:34
there's not a lot of complexity, right?
2:36
And so, and it seems like as far as we can tell there, everyone
2:39
seems aligned. And those priorities and those
2:41
the goals. And the other thing I found cool
2:43
was the I just a sense of moving comfortably fast, I think good
2:47
Mantra doing what's 80%, it's good enough, right?
2:50
And I think all too often that in the data World, especially we
2:54
tend to have a lot of navel-gazing in the strive for
2:56
Perfection, which is which is great except that you do miss
2:59
out on the opportunities in front of you right in the
3:01
phone's could clearly lend the priorities and You all can move
3:05
fast together. I think it's a great setup to be
3:07
on your setup for success. That's awesome.
3:09
Yeah so true a lot of people probably know Stephen from his
3:13
are flog blog post or sub stack that he put out.
3:16
If you have not read that I highly recommend, it will leave
3:20
a link to it in the description and he's also got another great
3:23
one that we didn't even get into.
3:25
But Joe, I know you posted this in the notes on the document
3:28
that we were looking at. It's like what?
3:30
Exactly isn't DBT where he goes through and says what you should
3:33
and shouldn't be used. In DBT for in his ideas and why
3:38
DBT shouldn't be an orchestrator.
3:41
And so, it's so cool to see how deeply he's thought about the
3:45
orchestration layer and how it can be done differently.
3:49
How he would like to see it and the crew knows.
3:52
Maybe he'll end up. Starting a company around that
3:54
someday. I think it'd be really cool.
3:56
The thing I like about into is he's in Ohio, right?
3:58
So, he's landlocked, and he's not For Better or For Worse, not
4:00
polluted by all the noise in the data world, right?
4:03
So I think it having That, that kind of remoteness also gives
4:05
you a sense of clarity. That that's very pure.
4:07
Yep. It's that Purity like they do in
4:09
snow and a Columbus morning in the winter.
4:11
Yeah. Are you busting out poetry on
4:14
us? Joe.
4:15
I didn't know nobody. That would not inflict that on
4:17
people. That's terrible.
4:19
Oh well, for anybody that has not read those blog posts go and
4:24
read them for anybody that has not bought Joe's book yet go and
4:27
buy that the fundamentals of data engineering will change
4:30
your life. It'll make you famous just trust
4:33
us on that one. And if you are not already
4:37
joining us in the over 30 cities, where we have Mo Ops
4:42
Community meetups happening. I recommend you.
4:44
Do that. If you're not subscribed to our
4:47
three different newsletters that we put out, we got to weekly
4:50
newsletters and one monthly newsletter, go jump on that.
4:53
Will leave a link to that in the description and if you're not in
4:55
slack, join us because there's all kinds of great conversations
4:59
happening there. Last but not least, I want to
5:02
give a huge. Thank you and Shout out to our
5:05
sponsors of this episode. While the roux is a platform
5:09
designed to be a control room for production ml to facilitate
5:13
deployment management, observability monitoring and
5:17
optimization of models. In a production environment,
5:21
they cater to a i-team's larger, small working in projects,
5:26
organized in a way that works for them.
5:29
Via sdks, you eyes, or apis, get your hands on and grow.
5:34
Your skills with Wallaroo by downloading.
5:36
And installing their free Community Edition.
5:39
Will leave a link to that in the description at least go check it
5:43
out. That's the least you can do.
5:45
They sponsor the episode, they sponsor the community.
5:47
They are big supporters and we got to say thank you to him.
5:50
You can check out there free Community Edition right now.
5:54
So, without further Ado, let's jump into it with Stephen.
6:03
All right. Dude.
6:04
What is whatnot? Whatnot is a livestream auction
6:08
site. It's kind of like, at which
6:11
meets eBay. And so, we have sellers going
6:14
live all hours of the day and they are selling, whatever.
6:18
They're passionate about, a lot of times, it's Collectibles,
6:20
like, sports cards, Pokemon cards, trading card, other
6:23
trading card games, but we're branching out a lot and to other
6:26
other categories, like fashion and clothing.
6:29
Young and we even have some of the whatnot type categories
6:32
where it's just like whatever you bring to the show, you can
6:35
sell. And so we have people people
6:37
opening return pallets from stores and selling those
6:40
clothes. I bought like a I bought a, some
6:43
returned mail. The other day.
6:45
Gotta like a Harvard Medical School deployment Iran on these
6:49
Healthcare. That was great, interesting.
6:52
We've got to God. Like You See Me So Wayne's
6:54
World? Is that kind of the sounds like
6:57
my place? I need to hang out there.
6:59
Or bad. It's amazing.
7:00
It's it is really sneakily addictive.
7:03
So you kind of get on there and you start browsing find
7:07
something you're interested in. They just hang out for a while
7:08
and see what people are buy-in. And a lot of the sellers are
7:11
very, very engaging and and they can they kind of gamify the the
7:17
retail experience well where you buy one thing and then they'll
7:19
throw in another thing or they can kind of deal make on the
7:22
spot. I'm just watching the clip on it
7:25
right now I'm watching Tuesday value e with Cactus and I I'm
7:29
just going to spend the rest of the podcast just watching this
7:32
clip here, so you guys can carry orange juice thing.
7:34
Yeah, it's very interesting stuff about, dude, I'm done
7:37
with. Yeah, this is just we just
7:39
dropped. We just released a new feature
7:40
called drops, or you didn't like enter for a chance to win
7:44
something and sent the sold, someone a seat on the Voyager,
7:48
what is it called? The that the Virgin Galactic
7:51
flight know the yeah. So so we're sending someone to
7:53
space, just a minute. What wait, what?
7:58
I'm gonna die for whatnot. Turn it after I get your product
8:00
while they're flying into space or nope.
8:02
Nope. Just just be a part of a part of
8:05
the experience and just live. Yeah, just live.
8:08
Happily make it home. Yeah, that would be a PR
8:13
disaster. Actually, so wish them well, but
8:17
she said, anyway, do you have a Hall of Fame of like, the
8:20
wildest stuff that you've seen all there?
8:22
It's like every every month or something, but there's something
8:25
new. We've we had a sports car.
8:29
And seller crack open up the most valuable card in the world,
8:32
which is called the triple logo, man.
8:33
It's a LeBron James card that has a patch from Hit three of
8:37
his NBA championships. Jerseys, the logo man on the
8:41
shorts and it's all on one card in the card, sold 45 million
8:45
dollars. So, that was open.
8:46
But what not live stream earlier this year.
8:50
Whoa, it's it's, it's just, wow. I think one of the things
8:54
working at what, not one of the things you you learn, quickly
8:57
is, how passionate people are. These different niches of the
9:01
Collectibles world people will people will drop thousands of
9:05
dollars on a Pokemon stream. It just on a Wednesday night and
9:11
it really speaks to not just the like not just the thing, the
9:16
cards themselves but to the community around it.
9:19
So like really is like social experience to work.
9:23
Talk about what not because you work there and you're doing data
9:26
stuff there. Can you give us a little bit of
9:28
a break? Down on just your data,
9:30
thoughts, in general, because you've been putting out so much
9:33
awesome material, man. Like, when it comes to the data
9:36
layer in machine learning and in analytics data, engineering,
9:41
you've got some quote-unquote thought, leadership and hot
9:45
takes, as I would classify them. But let's just like, breakdown.
9:51
You're working at? What not, what are you doing
9:53
there? And what has your journey pin to
9:56
get there? One of the really great things.
9:59
About working at whatnot is we kind of built out the whole data
10:05
landscape quickly and and protoform.
10:07
So we had like up data platform that was distinct from the
10:10
analytics team that was distinct from the machine learning team.
10:13
And so we've been able to build like build with a bit more,
10:17
specialized mindset than just having like a single team that
10:19
was kind of doing everything a little poorly.
10:22
And we kind of grown into this distinct responsibilities is
10:25
that. So I'm on the data platform team
10:27
which is orchestration. It's the key pipelines really
10:31
getting data from especially our main application systems and
10:35
making it available for operational and I'll analytics
10:38
use cases and things like that. So it's I get to jump into the
10:43
individual domains in the company, trust and safety, and
10:46
the product experience occasionally, but it's very much
10:50
more about enabling the the rest of the company.
10:53
To do those more advanced use cases.
10:55
It has been great. I think I've been in places
10:58
where I was that the smartest data person in the room and that
11:03
is okay, like there's nothing wrong with that.
11:07
But being in a place where there's just so many talented
11:10
Engineers who are just like have a direction in mind, like,
11:13
they've seen it done at scale and they can kind of build from,
11:16
from the start towards those that future state, where we have
11:21
really mature event logging mature, analytics mature,
11:25
Station mature male models and like being able to be on the
11:27
ground floor and build up to that has just been an incredible
11:30
experience. Really, really great, dude.
11:33
So I have to ask you because I've been putting together a
11:37
bunch of different best of the year since we are kind of at the
11:41
end of the year and I wanted to create like the ml Ops Community
11:45
Awards. And so I was going through a
11:46
bunch of slack threads that were contenders for the best of the
11:51
Year, slack thread. And one of them that came up and
11:55
it Really makes me think about what you're talking about.
11:58
Was almost like a it was a bit of a discussion around.
12:05
How much should you be thinking about the future?
12:08
How much technical debt should you be putting or taking on just
12:12
to get something out there and do a dirty so that you can
12:16
validate something? And how much should you really
12:20
just try and like bulletproof it from the beginning even if it
12:23
makes you a little bit slower? And there's a certain few people
12:28
in the community that are very opinionated that you should
12:30
never just do something dirty because it's always going to
12:33
catch up to you. And then there's others who are
12:34
saying just get it out there and see you don't want to like
12:40
there's plenty of projects that have been absolutely incredible.
12:44
Underneath the hood that never saw the light of day because by
12:47
the time, they actually got out, there was nothing there and it
12:51
or the like goalposts had moved. So, what are your thoughts on
12:54
that? Yeah, that's a great.
12:57
That's a great question. I think that's like one of the
12:58
fundamental tensions and the data world is between business
13:05
value and building like, a platform and sort of like,
13:08
thinking holistically and systematically.
13:11
And I think it's, I think you almost can't do both at the same
13:13
time. If you have your, if you have
13:15
your one hat on, you're like get solve a problem hat on, it's
13:19
really hard to do. The do like the system level
13:23
stuff perfectly. You want to do it as well as you
13:25
can, but you really do want to move as fast as possible.
13:28
I think that whatnot. One of the things that's really
13:32
set us up for success is that we divided the analytic side and
13:37
the data platform side at the beginning, we hired two people,
13:40
they had two distinct job responsibilities, the data
13:42
platform side was much more focused on like getting a
13:45
machine learning product out in production and the analytics
13:49
side was 14, focus on like the board meetings and those two
13:52
concerns are pretty distinct. I think the machine learning
13:57
side you have to have like really high quality.
14:01
Systems and pipelines that are like rigorous, they're defined
14:03
us code, whereas the analytic side, none of that really
14:07
honestly doesn't really matter. As long as you're getting like
14:09
the right, you're able to answer the right questions and you have
14:12
good definitions and you have something that's fairly
14:14
maintainable. That could just be like a single
14:17
layer DBT models. So, I think my, my sense is you
14:24
want to get the value as quick as possible.
14:26
But if you want to build in processes where you can go in
14:30
and think, ethically, and the way we've done that, what not as
14:33
we have this data platform distinct from the lines of
14:36
business. Its kind of separated a little
14:37
bit and so we can think about how should we improve
14:40
orchestration. How should we create like a
14:43
improve or usability? How can we improve the developer
14:46
experience and those things? But it is, it is sort of a
14:48
secondary concern walk me through this.
14:52
I mean, when you, when you started at whatnot and it was
14:54
very early days of the data team, what were, what were some
14:57
of the initial questions that you wanted to answer in order to
15:01
I'm sure that you're on the right path.
15:03
Yeah I think the so why not as livestream retail.
15:06
So it's kind of got like all your typical retail metric like
15:09
number of orders and things like that.
15:10
But then we also have these product engagement metrics like
15:13
how what how many shows are people watching?
15:15
How long are they staying in shows how like what's their
15:17
product analytics Journey? So I'd say those those are the
15:19
two sets of metrics that are our most important for the customer
15:24
Journey at whatnot. And so getting those exposed to
15:27
the end users like, or to our internal, like, Like
15:31
stakeholders is concerned number one and making sure that those
15:34
are exposed reliably and consistently, but then it kind
15:37
of works from there. I think, I think, as soon as we
15:40
you have those we had those base metrics, we started wanting to
15:43
know how can we operationalize this for a recommendation
15:46
algorithm in the product? And so that became the ml, use
15:49
case recommendation, like, can we get people into a feed that
15:52
is being driven algorithmically, because that is your flywheel
15:56
for additional experimentation long-term.
15:59
So, the first version recordation, Like a huge, a huge
16:03
goal posts to get to. And then on the other side,
16:05
it's, I'd say, we picked a couple of the most highest value
16:08
use cases. Operationally the start drilling
16:11
into, and that was like my trust and safety.
16:13
For example, where you've got because it's live streams, you
16:17
have all these potential ways that people can kind of abuse or
16:20
ruin the experience, rather. So being able to expose data
16:24
quicker but also create kind of a language for us to understand.
16:27
Like all right, what does it mean?
16:29
What is a violet? What is a What is a problematic
16:32
activity? How do we Define that?
16:33
Then how do we get that in the hands of Agents?
16:35
I would say those were the three like top priorities for for data
16:40
and they all have very different flows.
16:41
Right? Because they analytics low is
16:43
fairly simple, like kind of like couple DBT models and you got to
16:46
have that consistent but and integrate it, but then the ml
16:49
world is its own world. Kind of once you have, but the
16:52
raw data and the operational world to is its own rule own
16:55
world, with its own stakeholders and that flows that stakeholders
16:58
use for. Getting at like trust and
17:02
safety. Data is very different from the
17:03
flow that a board. A board member would use.
17:06
We're looking at a dashboard, dude.
17:08
Awesome, that's so cool. And so key break down real fast.
17:12
I mean, I do want to get into some of the articles that you've
17:16
written and you said the magic words of orchestration earlier
17:19
which almost made me jump in and say let's talk about inflow
17:22
foot. I didn't I bit my tongue and I
17:25
wanted to go into the idea of you mentioned, the recommender
17:30
system. And recommender systems are
17:32
something. I've been digging into pretty
17:35
pretty deeply recently and I'm wondering like what's your
17:40
recommender system look? Like, what is this stack?
17:42
How do you guys? How do you set that up?
17:45
And how do you make sure that? Like I imagine it's real time
17:48
you want to be recommending things to people continuously
17:51
and making sure that it's the best recommendations and I mean
17:55
I want to start with just like what it looks like.
17:57
But I also want to like hear about these kind of Is and how
18:01
do you know that the recommender systems, actually recommending
18:04
things that are interesting? And it's not just recommending
18:06
the most popular stuff like, what is that, like, how do you
18:09
go through and all of that fun question just to?
18:13
Yeah, there's there's so much to unpack their first of all, and I
18:17
think the, the recommendation system is such an so interesting
18:21
to me. It's the first time I've worked
18:22
with a team, that's built a recommendation system.
18:24
And first time, I've really been exposed to it.
18:26
I think one of the fascinating things about data is how how
18:28
much you can do and also how little you can now about So
18:31
much, like I I feel a recommendation system.
18:36
It's like a clearly like a data application, but it has so much
18:40
Nuance in it because it's such a mature data application that it
18:44
almost has become almost as its own thing.
18:46
And it's I think it's I guess just a sidebar.
18:48
I think it's really interesting how data products their data
18:50
like you escapes. This tend to do that or whether
18:53
it's computer vision or they kind of as soon as they grow to
18:55
a certain point, it can bike Branch off as their own distinct
18:58
sudden problem. But anyway, the I think the big
19:00
thing So at what, not the the thing that distinguishes, our
19:04
recommendation system in a lot of our like product serving
19:08
infrastructure. Whether it's recommendation, we
19:10
now have like other use cases for serving data into the
19:13
product, is it has to be real time and it has to be low, very
19:16
low latency and asked to scale to very high workloads.
19:19
So the way we do that, for most of our real-time applications is
19:24
we use a tool called Roxette and so we'll have like are offline
19:29
data processing and then we'll dump data into a Roxette
19:34
real-time analytics layer, which basically has has extremely
19:38
fast, look ups and you can also write SQL queries.
19:40
So it makes it pretty easy to maintain and manipulate over
19:43
time and it exposes those the results of those queries to
19:46
other services through an API. So the main application can just
19:50
query at the red guy, get the result set and filter it down to
19:53
a certain users and that works really well.
19:56
It's the latency is very low. And I think most importantly,
19:59
though the speed of Raishin to get a new, a new endpoint up and
20:03
running. It's very low.
20:05
We can just as long as we can reliably dump the data into a
20:08
certain spot, we can get of add it into our repertoire of things
20:12
to serve. So a lot of our recommendations
20:14
are actually serve that way so will pre compute the
20:17
recommendations dump them in there and then they're ready for
20:20
the system. What goes on and like the
20:23
training aspect? I'm actually not that familiar
20:25
with and I think that's one of the one of the distinctions
20:28
between like a data platform engineer and an MLM.
20:30
Nero whatnot is did a platform we're responsible for a lot of
20:33
the infrastructure and what's going on like from from a data
20:37
movements and point. But what happens like within a
20:40
step a lot of times there's like so much more sophistication than
20:44
it's really necessary for me to dig into.
20:46
So I actually don't know what we're doing.
20:49
In many of the recommendation system cases.
20:52
Interesting to see, how do you coordinate with the the ml teams
20:56
to make sure that everything isn't flowing well, and yet,
20:59
what's the overlap that? Yeah, that's one of the projects
21:03
that I've been working on recently in the last six months.
21:05
We had to migrate off of a tool called spell which got bought
21:08
into Sage maker, the sage maker world and so Sage maker is
21:12
Amazon's like ecosystem of ml related model functions and it's
21:17
so the stage maker workflow determines kind of our
21:20
interfaces. What what we want to provide as
21:22
a data platform team is are the tools.
21:24
Basically, like copy-paste tools where people can configure a
21:29
Docker cats, Really like a training job and Sage maker they
21:32
provide the docker container, they provide a shelf script
21:35
configuration and the form Piper parameters and then they can
21:39
just copy paste it into our orchestrator.
21:40
Daxter, and then kick that off on a schedule, very simply and
21:44
so long, the work of the apply for game was in setting up.
21:49
What what are those different steps that data scientists can
21:52
use to launch these training jobs, create models, deploy the
21:56
models and then just making the very easy to reproduce for
22:00
Brett, variations of the task, you mentioned Daxter.
22:04
One of the things we want to talk about in this episode was
22:06
your article on your sub, stack called air flow problems.
22:09
So you switch sides. Now you you Dexter dude, now or
22:13
what happened? Yeah, we are, we are on stage
22:17
major pipelines. What do you got in under the
22:20
hood? You have everything.
22:21
I feel like, that's the, that's the state of the world.
22:23
Right now. You have a little bit of
22:24
everything. You got some TV Cloud, you got
22:26
some Sage make that's so true and that's why it's such a
22:30
headache. That's why I think.
22:31
Yeah, you're such a headache. Yeah.
22:33
Your your sub staggering. So true with so many people, but
22:37
hit us with what what it looks like you're using Daxter.
22:40
Are you also using any of these other ones?
22:42
So our philosophy is the orchestration orchestration
22:46
layer that scheduling where should be centrally managed.
22:48
So we don't want DVD Cloud to have a bunch of like, gooey
22:53
scheduled runs in it. And just have those running
22:56
independently and then having like Sage maker, it had their
22:59
own schedule. There.
23:01
We don't want our injunction pipelines to be running on like
23:03
their own schedule, that's built in their own tool.
23:06
We want a single like pane of glass for the for scheduling so
23:10
that we can kind of hook things together.
23:11
Make sure that Sage maker jobs run after DBT jobs and if the
23:15
DBT job takes an extra four hours for some reason, like all
23:19
of those training job shouldn't then be delayed and right and
23:23
half be rerouted things like that and those are very it.
23:26
Sounds like a very simple problem where I was when I wrote
23:29
that article air flows, Problem. I was setting up I would setting
23:32
up a lot of this infrastructure and we had just reached this
23:35
point as a team where we were starting to have lots of data
23:40
products and data applications and different places where we
23:43
had some some jobs that we're starting to run like trust and
23:47
safety related policy, violation sensors, where they like check
23:51
whether something it happens in a slack alert.
23:53
And then we had all these training jobs that were, that
23:56
were being created and they didn't look exactly the same.
23:58
So you had like, they One might have a training job and one
24:02
might have another like post-process job and then it has
24:05
to get put into another system. And then we had all.
24:08
Of course, a DVT in pipeline stuff and there was just, no, it
24:12
just felt like chaos and when I looked at the orchestrator
24:16
landscape, I looked at airflow and prefect and doctor.
24:19
And I had previously used Argo workflows which is activities.
24:22
Native orchestrator. So it's fairly familiar with the
24:25
space. I just felt like the
24:28
orchestration layer was the The right place to solve this chaos,
24:34
right? It's the only tool that so
24:36
tightly integrated with all of the different pieces of the
24:39
infrastructure and the only tool that knows about all the pieces
24:42
that could possibly provide like an interface to managing it.
24:48
But when I looked at air flow and really, you been what I
24:51
looked at Daxter and pre fact and like just orchestration in
24:54
general. It just didn't feel like we were
24:56
solving that problem of life, being able to see all of the
24:59
things that are out there. There in the world and then much
25:02
less being able to like help manage and and make them work
25:05
together. I do think that Daxter and the
25:09
elemental team are starting to push in this direction through
25:13
their use of like asset like asset declarations.
25:17
DBT does this a little bit? Because it has their references
25:21
so like if you add new model then they should fit into the
25:24
graph. And the problem with DDT of
25:26
course is that it's like it's just a subset of things.
25:29
It's not going to manager model definitions and stuff and then
25:33
actually airflow to like you certainly they've started to add
25:37
open lineage and really start getting trying to connect the
25:40
operations that are happening from a processing layer to sort
25:44
of looks like a map of what you have and what you're managing at
25:48
that. At the data level.
25:50
And I think that's, that's the right, that's the right
25:52
approach. And so, if I understand this
25:54
correctly, it is you saying because the orchestration layer
26:00
basically has its tentacles in everything, it should be doing
26:04
more not less yet. That's what I think if we're
26:07
going to solve the problem and like if we want to be able to
26:10
intervene and manage the system as a whole, then it's got to be
26:15
something that has much at least as much tentacles as that.
26:19
Sure does because you have something else like another
26:22
another tool that's kind of like this is catalogs.
26:25
So data catalogues they are the one tool that actually.
26:29
I think treats the data ecosystem in a way that is, as
26:33
broad as it is Data. Catalog will probably index like
26:36
Google Sheets for you if you wanted to.
26:38
So, they've got a Google Sheets though.
26:40
Pull in your machine, learning models.
26:41
A, look at your bi tool that look at your eye, your database
26:44
and they'll like, pull it all in one place.
26:46
But there, they can't do anything about managing any of
26:48
that stuff. Stuff right?
26:49
It's like it's just visibility, it's adding some metadata on top
26:53
of it, giving you kind of like a search engine on top of your
26:56
catalog. The orchestrator, on the other
26:58
hand, if you're pulling that Google sheet and the snowflake,
27:01
it knows about both of those things, right?
27:03
And it can do things to both of those things.
27:05
It can create that like, map between them and you're doing it
27:09
in a way that actually lets you intervene.
27:11
If you want to say, stop pulling on them or only pull them once a
27:16
month. Hi, this is mihail Eric.
27:22
I'm the founder of pun intended, Innovation, and confetti Ai.
27:26
And this is a quick Public Service Announcement.
27:29
If you're tired of having nightmares about machine
27:31
learning models, you have in production and I encourage you
27:33
to subscribe to the ml Ops Community podcast, it's where
27:37
all the cool ml Ops kids hang out and learn how to tame their
27:40
ml models. So, what was your walk me
27:47
through your decision criteria? I'm using air flow for a nice
27:51
article about it, then you're not using it.
27:53
So, but you also mentioned astronomer to in the article, if
27:56
I'm not mistaken to is this may be a viable option.
27:59
I mean, why did you decide to go the route that you did.
28:01
I, I did POC, like, personally did pocs, with prefect, and then
28:06
air flow itself. We were on air flow for a while
28:08
and then end activate, well, Daxter was ultimately what I
28:12
really like about Daxter was Their focus on the engineer.
28:15
I felt like of all of these tools that Dexter had the, the
28:20
best, the most clear, focus on the data engineer as a Persona,
28:28
and it was coming out like they did just release branch
28:31
deployments, which is that, like basically every time you create
28:34
a new PR, it'll create a sort of distinct environment for your
28:37
code, you do like Devin testing there.
28:40
They had see ICD actions out of the box and it's nothing.
28:43
Like any of any other tool? Like there's there's all there,
28:46
CI, CD for air flow. Of course like you have actions
28:50
all over the place but I think your vendors are a little bit
28:53
like a part of your team to some extent especially when you're
28:56
smaller. And I so I knew I was going to
28:58
be leaning on someone to help us get good stuff stood up.
29:01
And I just it definitely saw that energy from the Dexter
29:04
team. If you're going to go with a
29:05
vendor, you are making a partnership at more of the
29:08
platform level astronomer. For example, is their core value
29:12
is being able to make it really easy to Play airflow and to
29:15
deploy it multiple times. And then also the aggregate
29:17
metadata and likes observability metrics on top of those airflow
29:20
instances. And so it's not just like the
29:23
framework, the orchestration framework of like what's the
29:26
Syntax for creating a dag? It's really about what's the
29:30
like how are you deploying this thing and monitoring the
29:34
infrastructure and then getting metadata out outside of it?
29:38
How are you sending the slack alerts and making sure that like
29:40
they're going to the right people especially as the
29:42
complexity of olives. So it's almost like it's not a
29:45
decision in many cases. Are you going to use Daxter?
29:48
Or are you going to use airflow? It's like what are you going to
29:51
use at the level above that to help coordinate this thing?
29:54
If you're going to grow like we know we know our trajectory
29:58
we're planning for a lot of growth.
29:59
We're planning for other teens to be deploying, dagstar,
30:02
workflows, and plugging into this orchestrator.
30:04
And so that's that's where are we dude?
30:07
That was like the problem we wanted to solve.
30:09
I love the human design aspect of that.
30:12
How would they like, A bit of an abstraction above that and
30:16
thinking about this holistic view, not just like, oh this
30:20
tool gets the job done in the best way.
30:23
We'll what does that best way to find best for me because is it
30:26
the fastest? Well not quite and you're
30:28
looking at it in a much different way.
30:31
And it's funny you say that because I was at a meet up in
30:34
Berlin, one of the ml Ops Community meetups few months ago
30:38
and there was a data engineer there, and he was talking about
30:40
how he was so excited about Dexter.
30:44
Because of some I can't remember what the exact feature was, but
30:47
it was like it's data self-aware.
30:50
Some shit. I can't remember the yeah, they
30:53
make sense of it. Exactly.
30:55
And and so I'm just going to take this moment to say Daxter
30:58
does not sponsor this episode but if they want to pee, hit me
31:03
up man. We're here.
31:05
What you eat an interesting thing you brought up though is a
31:07
sort of centralizing orchestration layer which makes
31:10
a ton of sense. It's sort of the Hub that are
31:12
trafficked, I guess maybe the try air traffic, We control for
31:14
maybe a overused metaphor, but what you mentioned to you one
31:18
other teams to start deploying Daxter and have that level of
31:20
control. One thing I want to know more
31:23
about is, how do you train and share knowledge among different
31:28
stakeholders on these platforms. So we definitely see.
31:31
Make you want to at least have a sort of a common body and
31:34
knowledge or a competencies in order to be productive.
31:37
How do you do that? It's a challenge.
31:39
So I think there's two levels to it.
31:40
There's the first is just like education on the platform,
31:43
right? You write a doctor since we're
31:45
in the might plug in it. One of the things that I really
31:47
like about Daxter is it's it's created to for the jobs that you
31:52
run to be modular. So we have a, we have like four,
31:55
repositories that push jobs into the Dexter Central scheduler and
32:01
they're managing the family. So it's intended to be used that
32:04
way where you have like different different environments
32:07
and different Docker containers and different requirements
32:09
because we had data science, workflows machine learning
32:12
workflows. I knew we had to support.
32:13
Port, all sorts of insane python package requirements and stuff
32:18
like that. So there was never going to be
32:19
like a single Basin. It's right?
32:20
That that you just use this and plug it in and it works.
32:23
So we have this kind of heterogeneous set jobs to get
32:27
pushed to Dexter and that that is just a training issue, really
32:31
this? You just coming up with a
32:32
Playbook and a template for how you do it, how you create the
32:34
dockerfile like making the image creation process, as simple as
32:38
possible. The more challenging part is
32:41
making processes. We're each other and knowing
32:46
what can like, allowing other teams to know what can be
32:49
reliable. This is, this is like a lot of
32:53
what data match I think is intended to, to make clearer to
32:57
create processes. Whereby one group can publish
33:01
data products to other groups and then other groups can do
33:05
whatever they want with them, right?
33:07
And that can be very challenging because we have, we have our GT
33:10
project and that's really where all of the, the majority of our
33:13
Our data pipeline work is done but you can be kind of unclear
33:17
to end users like alright, which model should I select from?
33:21
And which ones are not subject to change?
33:24
So a lot of the that's it. We do been two things here that
33:28
have made our lives a little easier.
33:29
One is we've decomposed are one main DVD project into like marks
33:35
sub markets that have different classification.
33:39
So like we have like raw data Marts.
33:41
We have a core Set of core like Kimble modeled Mart's that we're
33:46
building out. Now we have a couple of other
33:48
like product like use case specific Mart's.
33:52
And then we also give people scratch spaces to build their
33:54
own leg kind of ad hoc stuff. And so there is a bit of a
33:58
choose-your-own-adventure opportunity available for end
34:02
users. They can choose from the raw
34:04
data, they can get something like our core dimensionally
34:07
model data that we like have Team guarantees around.
34:10
They can pick more highly vetted stuff that's been Published by
34:14
like our machine learning team for example, and there's an
34:17
implicit knowledge that like I'm getting data from this Source
34:21
rather than just selecting from a random snowflake table.
34:24
So that sort of classification of like what are each of these
34:27
things and who owns them is really important.
34:29
And then the second piece is Daxter.
34:31
We've moved all of our stuff over to daxter's assets
34:35
framework, which allows for the jobs themselves to be linked to
34:40
upstream dependencies. And so Someone is writing a
34:44
machine-learning job, they actually reference the DBT table
34:50
by name, that that job depends on and present.
34:53
Brilliant. It creates an explicit
34:55
dependency on that table. Now whether you like how you
34:59
triggering that machine learning job in very you can make it sort
35:02
of like Center driven. So like at the Upstream cable
35:04
changes, that's when the machine learning job runs.
35:06
And that's very nice. But at the at least, by having
35:10
that asset framework, you can go and look at that machine
35:12
learning pipeline. And then see exactly what jobs
35:16
it's referencing and it allows me as a data.
35:19
Platform person who doesn't really have perfect insight into
35:21
things more quickly jump in and troubleshoot things if they go
35:25
wrong and also to give them some sort of advice.
35:28
So it kind of creates an interface for us to support
35:31
other people simply by having them declare like what
35:34
specifically they're relying on so many times.
35:37
I've tried to ask like how people solve that problem
35:40
because that is one of the cruxes.
35:43
That you'll get between, maybe it's data scientists or data
35:48
analyst, and platform teams and it's such a headache to know,
35:52
all right? If I'm changing something, or if
35:54
I'm iterating on some kind of data, how is that going to
35:58
affect everything Downstream? And to know that, all right,
36:02
this has been claimed by these projects, were this model, or
36:08
whatever it may be, then you have an idea of like, all right,
36:12
great. Now, Now if something is broken
36:15
or if somebody comes to me and says hey why isn't this working
36:17
then you can just troubleshoot it and you can go into it and
36:21
say oh maybe this is why so it helps you get to that and state
36:26
much faster and I really love hearing about how you
36:30
implemented that. It's so cool.
36:32
So I just want to know like, why do you think it struck such a
36:35
nerve with people when you wrote this airflow blog?
36:39
Like what is it that? Well, I mean it's a sub stack.
36:41
I keep calling it a Blog but same same, Right and it created
36:45
a lot of Buzz. First of all, a lot of people
36:47
really liked it and and then it was on like Hacker News for a
36:52
few days. It was on the front page of that
36:54
and there was a Batson controls just away.
36:59
Yeah, it's yeah. It's it's so funny.
37:01
I mean, I think what what really struck me personally was like as
37:07
a writer to get on the top of Hacker News.
37:09
Like I didn't even realize it happened the first like for like
37:12
20 like Maybe 20 hours, right? So I like, I just published this
37:16
in my pajamas on at 9 a.m. on a Friday and then like doesn't
37:20
feel any way tempted to whatnot stream and by myself.
37:26
But yeah, I think, I think it it struck a nerve because I think I
37:32
caught the like, the feeling that a lot of people have with
37:34
their flow where like you feel like they're a little frustrated
37:39
but it's fine. Like airflow is like good ever
37:43
ever those. Great.
37:43
And like it is it solves the problem of orchestration and it
37:48
solves it pretty well and everybody knows about it.
37:51
But there's so many people that are like a little bit frustrated
37:56
or just enough frustrated that they want to like they really
37:58
want to talk about it. So I think just critiquing
38:00
airflow. I think is is is going to strike
38:02
a nerve, that's what I found out.
38:04
But also there's this sense that like, Airflow is so inevitable.
38:09
In the data industry. I think this is this is really
38:11
what like the what I came down to in that post was airflow is
38:15
so inevitable that stifles the conversation of what that what
38:21
that tool there could be because it has defined the space.
38:24
So so completely for the last six or seven years.
38:28
Write it. I think the way we the tools
38:31
that we use Inform the way we think about problems.
38:36
And the way that tool that airflow is created as its
38:39
created as like pipelines individual bags.
38:42
And so you tend to think of your world as like a set of discrete
38:46
processes that run independently of each other on a schedule in
38:49
their item potent than they like ballot run on the the backfill
38:53
and things like that. And that's the, we engineer
38:55
start to think about the problem.
38:57
But if you look at something like DBT, DBT, Basically creates
39:02
this graph and there's all like the the he's a developer stuff.
39:05
But like to me, what especially by DT is it puts you in this
39:08
mindset of like managing this graph of data assets?
39:12
And some of them are managed by DBT and some of them are like
39:15
exposures. But, like, ultimately what
39:17
you're thinking about it, you're thinking about how data is
39:19
moving from the point A to point B.
39:23
And that just, it makes you build in different ways and so I
39:26
think that's really That like that sort of.
39:34
Completeness of air flows. Like the way airflow is modeled
39:37
to the problems were solving. As I think what struck such a
39:40
nerve, it's good for asking me to know it.
39:42
So I mean the fact that other like you can hire an engineer
39:44
and they know how to use air flows because you can get
39:46
someone up and running like that and they don't have to learn a
39:48
new tool. I think like you look at what
39:51
astronomers doing with it. They bought open lineage and are
39:55
integrating like lineage into the product and they're starting
39:58
to build some of that monitoring like the higher level monitoring
40:01
and into the product and maybe maybe they'll come In and really
40:04
improve the way things. The the speed at which you can
40:07
like deploy a data platform and it's kind of the scope that it
40:11
can address. But I also think air flow
40:15
therefore, like air flow is not going to.
40:17
They're not going to like change very much in the next seven
40:20
years. I would almost like, get rid of
40:21
the value. The Big Value.
40:23
Those smooth out the corners of improve the UI, but I mean, all
40:27
they need to do is just pay attention to what prefect index
40:29
for doing. And just make it criminal
40:31
changes, just said they got there, the incumbent base, so
40:33
it's Moving do just a concept just to stay somewhere.
40:36
Thank you. You brushed over something that
40:39
is so important and it kind of opened my eyes in a way, which
40:45
was saying, like, it's been this way for so long that we haven't
40:49
been creative and thinking about the solution, it's like we've
40:52
gotten complacent and said these are dags.
40:55
These are how we run dags and this is what it is.
40:57
And then I just think about all of the subsequent tooling that
41:02
is based on that theory. URI.
41:04
And that's how you do things. And really if you're thinking
41:08
about it in a different way and you come with something, totally
41:12
out of the box and you say, like you mentioned okay DBT has a
41:17
graph way of looking at it. Now that brings a whole other
41:21
perspective into what could be the next iteration of this.
41:25
If we were to just scrap everything and start over today
41:31
and we never had heard of Dags What would that actually look
41:36
like right? Yeah, I think one area where it
41:41
drives me crazy right now. Is that ml that ml world and the
41:45
data world? The like I would say airflow is
41:47
kind of like traditional data data processing world and then
41:50
the machine learning world is not necessarily like the speed
41:54
is not any any greater? The just have all these offline
41:58
training jobs, that's what most most things happened.
41:59
But what's different there is that they have this on the
42:03
machine. The models have this sort of
42:05
ontology of what is happening at each step and what's getting
42:10
deployed. So it's not just this.
42:13
This processing job runs daily, it's this model is getting re
42:18
trained on a certain basis and then we're going to take action
42:22
based on what happened in that last free training.
42:24
And so you end in when I as a user want to look at it I want
42:27
to look at that model and that model is history.
42:30
I don't want to look at like the last couple dates Etc.
42:33
It's like Well, I'm trying what I'm thinking about is I'm
42:35
thinking about the model and everything, like all of my
42:38
language is geared around that one of the reasons, the machine
42:41
learning world is has like gotten, so chopped off,
42:44
especially from an orchestrator standpoint is like, is because
42:46
of that difference, in looking at things, it's not just like a
42:50
dag, that's continuously loading data.
42:51
It's like, you want to think about the model and you want to
42:53
think about whether the models been deployed, Etc.
42:57
And so, what ends up happening is, you have to build a new
42:59
orchestrator, just, for machine learning model and I would love
43:03
one orchestrator. Or less like just love it to
43:05
easier shit on this way. There's been too much VC money,
43:11
that's poured into the space for that to happen.
43:13
Anytime soon. It's very next cycle.
43:18
Yeah. Give me ask you this.
43:21
How do you interact with the software engineering team?
43:24
We are, we are implementing an event bus like Kafka Kafka
43:27
streaming solution and end the data platform team.
43:30
Not made that someone else in the team has set that up and has
43:33
been the point has been Point person, setting that up.
43:35
And so, we really see the data platform team as an extension of
43:38
the engineer, like the core. The core stakeholder than the
43:42
people. We report to is the engineering
43:44
organization. In some way.
43:46
We are kind of like an interface that queryable, high volume data
43:51
interface into the engineering organization, that's our primary
43:54
affiliation. And so especially with the event
43:58
streaming project, we are taking the lead in setting up the
44:01
schema and data contract and did. a quality infrastructure
44:05
for for not just making like not necessarily enabling what events
44:10
are going to be going through, but tracking them across
44:12
projects across the different clients and and I think that
44:17
When I think about data, what I really think the purpose is
44:20
often times is to take data from one domain and give it to
44:25
others. And so events are an interesting
44:27
example, where within the organization we already have
44:30
these discrete domains of like client-side different client
44:34
operating systems. We have the backend Services, we
44:37
have a couple of different back-end services and so our
44:39
role there is just to make sure that we are building A set of
44:43
events that anybody can use, it's like anybody any of the
44:46
engineering teams can use in a reliable way.
44:50
It's kind of like that public infrastructure of the team to
44:54
dude. So many different pieces that
44:57
you're servicing and thinking about it and just how you're in
45:01
the middle of all of this. You're right where the action
45:04
is. Yeah, it's crazy.
45:06
It's a lot to keep track of and I think one of the tensions that
45:10
we see the data platform team experiencing is just trying to
45:15
keep up with with everything that's going on.
45:17
Like this. We are really pushing To
45:21
allocate all of our effort towards the highest impact, use
45:23
cases, and then letting the other things just work
45:26
themselves out. And that that was not true for
45:30
my experiences of previous employers, where the data team
45:33
had high visibility but like, it wasn't always clear?
45:36
What was the highest value and highest impact thing.
45:38
So, we pursued a lot of things that were not that didn't end up
45:41
being that useful at what, not? Because so much is happening.
45:45
If we don't get to something quick and solve it well enough
45:49
then I think either just takes on a life of its own and it's
45:52
like gone it dies or it becomes something that like becomes a
45:57
bit of a mess that we then go and have to clean up.
45:59
So the priorities come to us. Yeah, a lot more themselves.
46:02
Uh yeah. Yeah.
46:04
That's I mean it's just it's just one of those things where I
46:08
mean, it's pretty cool that you're close to the line of
46:10
value though. I think there's a direct
46:11
reflection between like what you do from a platform perspective
46:14
and the output of it. I mean that that's it because
46:16
your tightly coupled to the engineering aspect of it.
46:18
Right? It right.
46:19
Yeah. The common Trope.
46:20
I've Ben, I've been hearing in the for last bit and data is
46:23
like all we need to provide value and Roi and monetize.
46:27
Our data May notice the conversations for disappeared
46:29
from data teams for a bit, too, or that was delicious.
46:32
Yeah, it's like oh well, just so provide like things and stuff
46:35
and it will experiment a lot. And I think that's some very
46:37
quickly coming to a halt, over everything, it's actually in a
46:40
lot of ways. I can't blame my people on these
46:42
teams, maybe their managers or maybe the people that hired him,
46:44
but I feel like a lot of teams are hired and this is, there's a
46:47
lot of money sloshing around. So you get something off of what
46:49
not you'll buy a day or two. You're at it and it's just it's
46:52
pretty fun. So if you can buy data teams on
46:55
what not now, this is crazy. No rules against that sort of
47:02
teams and what I want to ask one.
47:06
Oh, go ahead go. Yeah, yeah, I was just going to
47:08
add whatever. Like I think I've been that
47:10
person who was hired into like the first day, the science role
47:13
and there's so much. It's such a poorly understood
47:16
role. And, and it can go so many
47:18
different ways, I think. A lot of times, it's just people
47:21
don't understand how to get value out of out of the data
47:25
team, right there. Not being pushed, the people are
47:28
not being pushed to create things that can be put in
47:31
directly in the products. Or they're not part of the
47:34
engineering team. They're not part of the Sprint
47:35
cycles, and it's, I really think I could see a world in the
47:39
future, where there is no data team.
47:40
That's just your hiring didn't professionals.
47:42
And yeah, better understood. What kind of value because they
47:45
just do it. So true, that's so true and
47:51
that's an interesting world to think about.
47:52
Now, to finish, I know you mentioned before that you have
47:57
philosophies of What Not and you're building with certain
48:01
philosophies in mind. Do can you tell us what these
48:05
philosophies are? What are some things that you
48:08
try to keep in mind as you're building?
48:11
I know, when we interviewed the doordash folks, they were like
48:14
very big on making everything one.
48:17
Said better all the time that was their thing.
48:19
Like, just getting one percent better and really chipping away
48:23
at that the big ice sculpture or they also talked about like the
48:28
velocity of machine learning. So how quickly can you take an
48:32
idea and put it into production? Do you all have like, certain
48:37
philosophies? I remember, I can't remember
48:39
what philosophy you mentioned earlier.
48:41
I think it was building with scale and mind knowing that
48:45
things are going to be big. So, That's, that's one.
48:48
Right? What else did you have on there?
48:51
We one of our one of our more distinctive philosophies, as a
48:55
company, is to move uncomfortably fast and to, to
48:59
get things in front of like customers for the most part, but
49:04
when it comes to data platform team to get things in front of
49:06
our internal stakeholders and also to not not overthink things
49:09
too much. Can be really tempting to To try
49:14
and build the best data model out of possible.
49:16
And like really get Feedback from everybody and things like
49:21
that, but we do try to minimize figure out the 80% best solution
49:26
and then go with it as fast as possible and just iterate over
49:29
time, that does as a data platform team.
49:33
Sometimes we have to think about that and, like, really be Like
49:37
make make space to to have more more thoughtful decisions
49:43
especially when it comes to data modeling because in data in the
49:45
data world, like, as soon as you publish something, it's part of
49:49
the infrastructure. It's like laying down pipes and
49:51
then a city gets both on top of it, right?
49:52
You're like her, you can't go back and it's a very expensive
49:56
thing to go back and redo redo, some of those decisions.
49:59
So we try very, very much to identify what is a type one
50:04
decision like where we can just walk it back if we need to.
50:06
What to type to decision and treat them accordingly.
50:09
Look up really fast. Is is one the other piece, I
50:11
think that is pervasive as own, everything and nothing in the
50:16
data platform world. We see a little bit of
50:19
everything, like from in the client side and back end, code
50:23
of our application, we have events, getting admitted, and we
50:26
want people on our team to be able to go back there.
50:28
And even if they don't, I'm not making commits and changing
50:31
stuff. We want them to be able to know.
50:33
Why is this event sending light in error like Not being
50:37
validated correctly, they need to under we need to understand
50:40
like the protobuf like part of of schemas and how that gets
50:43
validated during development cycle and we need to know like
50:47
what is our trust and safety team using from a data
50:50
perspective and so we need to have that visibility across the
50:53
whole system even if we're not going to manage it tightly we
50:56
want to be operating with like an understanding that we can
50:59
step in and assist any of our stakeholders across the whole
51:03
platform and that is a constant learning learning.
51:06
Learning effort for everybody that transparency man.
51:11
I love that. That is, that's incredible.
51:13
Well like honestly I don't want this one to end.
51:17
I feel like I'm just getting so much wisdom downloaded here and
51:20
this has been so good and but I know you got coral reefs to go
51:25
and live stream and maybe some guitar playing or what not.
51:29
So, I will let you get back to that and get back to your life.
51:33
But thank you so much for coming on here and teaching us.
51:36
Yeah, this is great. Thanks guys, a lot of fun
51:40
course. Thank you.