Latest / Agentic Conversations (formally mlops.community) / MLOps + BI? // Maxime Beauchemin // MLOps Coffee Sessions #104
Transcript
- 0:00Bishnu. You see all the kids toys behind
- 0:02me. Oh, yeah, where you at on
- 0:06officially in Toronto, for the ml ops world event?
- 0:10It is going to be in person. You are threatening to come.
- 0:15I'm too threatening. It's a firm threatened and you
- 0:17gotta send me the times. Yeah, we're going to try and
- 0:19record some live stuff at the event.
- 0:22So, should be cool. But today, we're talking with
- 0:26Maxine. Buscemi, Buscemi, you know how I
- 0:29am with the names and I cannot get them right for the life of
- 0:32me. But this guy You have not heard
- 0:35of them, you probably have used a tool that he has created,
- 0:39whether that was air flow or preset.
- 0:42The dude is all over the place. And sorry.
- 0:46Did I say preset? Is it preset?
- 0:48You since the managed company that he runs?
- 0:50But superset is the original tool, which is now run by the
- 0:53Apache Foundation. Yeah, there we go.
- 0:56That's what it was. I knew there was some and so I
- 0:58loved some of this hot takes around like why he would go and
- 1:02start a company around. Round.
- 1:05Like preset well, no. What is it?
- 1:10God, I can't remember the name of it.
- 1:12Now I just started getting started super set, which is
- 1:14opens, open source and then started preset.
- 1:18And the hot takes were about. Here we go, get a business
- 1:19intelligence. There we go.
- 1:21That's what it was. So I have a total mind blank
- 1:24right now. And thank you for walking me
- 1:26through that the other part, which we just drove right into.
- 1:30And I really enjoyed that we got right into to it was this
- 1:34entity. Eccentric type of thing.
- 1:37Maybe you, what did you take away from that?
- 1:39Like, because I know that you were loving it.
- 1:43Yeah, for sure. I mean my big takeaway and I'll
- 1:46go through Max Max's like by on a second.
- 1:48Everybody should should get to know who this who this man is
- 1:51and what he's done for all of us.
- 1:53But what we talked about was entity Centric data modeling.
- 1:57And what why that's important is increasingly organizations are
- 2:02are generating tons of data that we want stored.
- 2:05Putting that into the data warehouse and using it to drive
- 2:07analytics and machine learning outcomes.
- 2:10But how we make that repeatable, reliable, scalable and
- 2:14maintainable is through data modeling which not enough
- 2:17companies do well, that was Max's point and he gave us sort
- 2:21of a understanding of how to think about doing data modeling,
- 2:25in a modern context that isn't just hey, go read Kimble's books
- 2:29about dimensional modeling or like that stuff was written
- 2:33before we had like the MLS Ops and machine learning stuff we
- 2:36have today and he gave us an intro into like how to think
- 2:39about it in 2022 is feature stores and all the other sort of
- 2:42systems that are around now. This is cool.
- 2:45Yeah, let's get into it man. That was really good.
- 2:48Let me take you through. Oh yeah, 95i real quick.
- 2:52Yeah, for sure Max, you should follow Max pusherman because
- 2:56he's currently the founder and CEO of preset but more relevant
- 3:00to this podcast. He's actually the original
- 3:02creator of both Apache superset, a business.
- 3:05Halogens open source tool and Apache airflow, which is a data
- 3:09orchestration probably. All of you have interacted with
- 3:12and a he has had an exciting career shaped at places like
- 3:15Facebook and 2012 Airbnb in 2014 and lift and so many cool blog
- 3:20post, check it out and do not forget, we started a pretty
- 3:24awesome newsletter that you will want to also subscribe to if you
- 3:29have not as the best of slack or just join us in slack and get
- 3:34all the information. Formation drink from the fire
- 3:36hose, let's talk with Max. So max, you just have something
- 3:43pretty interesting and you need to talk about this idea of
- 3:46entities entities eccentric, data modeling and why that's an
- 3:50interest to you. And let me start by saying like
- 3:54I feel like a lot of machine learning is starting to meet the
- 3:58modern data stack and like people are trying to do machine
- 4:04learning in the day. The warehouse and are trying to
- 4:08apply Concepts like that before we get to like actually talking
- 4:12about data modeling. I wanted to get your thoughts on
- 4:15that Paradigm of just throwing a shit ton of data into the data
- 4:21warehouse and then doing machine learning on top of it right
- 4:23there. Does that make sense?
- 4:24Or are we just like setting ourselves up for a future
- 4:26problem? Yeah, I don't know.
- 4:28I think one thing that makes sense, it's the data as kind of
- 4:31gravity to do it, right? It's going to have the very
- 4:34heavy. Thing.
- 4:35And they don't want us to be together in one place and
- 4:38ultimately all data wants to be joined and uni, and and kind of
- 4:44brought together. So I think the bid to bring it
- 4:46all in one place and call it data warehouse, her legs or data
- 4:50Shack and you know whatever you want to call it make sense to
- 4:54take all your data and bring it one place.
- 4:56Where you can make sense of it then, does it make sense to use
- 5:00to compute resources inside the data warehouse to do amount?
- 5:05It's like bring the computer where the data is, bring the
- 5:07data where the computer is. I don't know.
- 5:10It seems like it does make sense if it is, you know, on a cluster
- 5:13on still flake or cluster and on bigquery cluster to bring the
- 5:18compute right there. It's easier to move to compute
- 5:20than moving the data then, you know, a sequel, the right set of
- 5:23semantics to express and now transformation or you started
- 5:27having this stuff where you have.
- 5:28Like I remember doing some that stuff at Facebook, where we add
- 5:31them, you'd ETFs and udf's and So you guys like and I already
- 5:36did user-defined, you know, functions inside your Sequel.
- 5:40And I think it's not that crazy to think you would have a
- 5:44program running in line in between like, SQL resources,
- 5:48very like, kind of Select star from you DTF.
- 5:51And that you DTF going to be, you know, whatever language I
- 5:56think. Like I was talking withdrew from
- 5:58DBT recently and he was saying like, oh I want like they're
- 6:01really interested in people running python code that you
- 6:03don't want to be running. You know, I think about in DBT
- 6:06itself, they want for other things to run the python for
- 6:09them. It's like, I don't want to
- 6:10python interpreter running random stuff in my, in my own
- 6:15kind of Stack, but I think we're getting there.
- 6:17Snowflakes, got some stuff I think, where you can run kind of
- 6:21python in line and things like that.
- 6:23So it doesn't seem that crazy to me, you know, who's been
- 6:25crazier, things done in the past.
- 6:29Yeah, you're definitely right that there's been crazier,
- 6:30things done in the past especially when we think about
- 6:33all the hacks that people have applied to data over the years,
- 6:37over the generations. And I think where that question
- 6:40came from. For me, as you know, I've been
- 6:42reading a lot of Chad Sanderson stuff.
- 6:44He's the director of data platform at Convoy and he writes
- 6:47a lot about how data modeling is more crucial than ever.
- 6:53We had him on gonna meet up recently, check it out to all
- 6:55our listeners, but he talks about how data modeling is more
- 6:58crucial. The never with the proliferation
- 7:00of data sets and the fact that they're all ending up in one
- 7:04place where sometimes too little structures applied.
- 7:06So that's where that came from. But I guess my question to you
- 7:10now is you mentioned entities Centric, data model data
- 7:14modeling, what does that actually mean?
- 7:16And how does that apply to ml this?
- 7:18It's an idea that I've been pushing forward, not talking a
- 7:21whole but bunch about or not even writing a blog post on
- 7:26basis. I was looking at ml
- 7:27practitioners and what they do and how they work and just like
- 7:31another training and what it requires.
- 7:33And really often, I think what you need is a really clear
- 7:36entity. And then a shit-ton of, you
- 7:39know, features and attributes and things.
- 7:42And some of these things are not just like an attribute of the
- 7:46data. So if you're, you know, doing
- 7:48some Trend predictions around a user, you might have a lot of
- 7:50demographics on that user. And that's interesting.
- 7:53But you also have a lot of facts and metrics and you know 7 David
- 7:58It's and 28-day photo upload on something like Facebook, right?
- 8:04And the number of like 364 day, bookings at Airbnb.
- 8:09And you want to take all of these features and then
- 8:12associate them with an entity and it ethic like, like stepping
- 8:16back and talk about data modeling.
- 8:17So it's a little bit of a lost art, right?
- 8:19Let's say we have a weird democratizing analytics process,
- 8:22and everyone is invited and everyone knows Sequel and, you
- 8:25know, airflow and DBT and but like no.
- 8:28No one really knows that lost our data modeling.
- 8:33Very, very well. So that's kind of an interesting
- 8:36thought and then getting into dimensional modeling that it's
- 8:40just for people. There's some books, you know
- 8:42written by Ralph Kimball from probably like you know, 30, 2530
- 8:47years, old books around like methodology on how to organize
- 8:51and structure your data, how to create the right data set?
- 8:55So that it's best organized for retrieval.
- 8:58Consumption rights. And dimensional modeling is one
- 9:00of these things where you have your typically you know you have
- 9:03what you call this star schemas or snowflake schema is and
- 9:06sounds very fancy. But but generally, you can think
- 9:09of these schemas adding facts, you know, like something like
- 9:13bookings at Airbnb. And in Dimensions, that are
- 9:16collections of entities things, like, you know, hosts and, you
- 9:22know, you have like, who is doing the booking?
- 9:25Whoo. What is the list thing for the
- 9:26bookings? For those Earth?
- 9:28Mentions and those dimensions are entity Centric in some ways.
- 9:32Are there that dimensions a dimension like user or listing
- 9:37is in a lot of ways, a an entity then dimensional modeling the
- 9:43available. What is a fact the city of fact
- 9:45tables and dimension tables. Well, we see in ml is like you
- 9:49know people want to have these that more feature for store.
- 9:52So you take an entity like a user can input all of the
- 9:54attributes including the facts as attributes.
- 9:59Of the entity and I really often very common practice is having
- 10:04like these time Windows. Like how many people?
- 10:07How many times did someone do this over a certain period of
- 10:10time? Which is usually down to now,
- 10:13right? Because I'll many many times you
- 10:16booked on Airbnb, last year's a great predictor.
- 10:19Perhaps of like, how many times you will book next year?
- 10:21So this very useful and, you know, machine learning models,
- 10:25may know how to make sense of these metrics and weigh them.
- 10:29The point being like, okay, so that's a thing now, call it
- 10:33feature store, but then I think beyond the idea of a feature
- 10:36store and Beyond the idea of making prediction and feeding ml
- 10:40model. There's a bunch of other reason
- 10:42why it's really cool to have and City Centric metrics, right?
- 10:46Like the future store is greater and it's use than it is to the
- 10:54to just like train models, right?
- 10:56So if you can have like a million attributes about a you,
- 10:59They're at Facebook precomputed when it comes time still
- 11:04retrieving. Say, I don't know scorecard for
- 11:06a user to be like a what's happening with this user?
- 11:08So that's it, there's there's there's a lot of value and
- 11:11knowing these things and there's a there's a lot of value in
- 11:15having the data organized and that way.
- 11:17And that's what I call. You don't have to T Centric
- 11:18modeling, where you have free, really clear entity, and then,
- 11:22you know, an Infinity of not only attributes that user about
- 11:26features in the sense that time window.
- 11:28Action and Mike, you know, metrics at that level 2.
- 11:33So it's a little funny because I remember visiting someone's
- 11:39office when I was in London and they were talking about how they
- 11:43have so much metadata now that they need metadata for their
- 11:47metadata. And it kind of feels like you're
- 11:51touching on that a little bit. Like he amount of stuff that is
- 11:55going on but this entity Centric.
- 11:59View will be a way around that. It's like you have the main
- 12:04protagonist and then everything else that goes on there.
- 12:09Could possibly be a remedy for that.
- 12:14Yes, it really interesting thought and, you know, the I
- 12:17think like that, the idea that metadata is growing is
- 12:21definitely super clear to me to write like this idea to of that,
- 12:26we need a meta data warehouse the same way.
- 12:27We have a data warehouse, we need to take all the metadata
- 12:30from the company, like information about, then image
- 12:34and data provenance and data Ops information, like, operational
- 12:38metadata. Like, you know, what was
- 12:40computed and one and by which grip, and who's the owner?
- 12:44Of that stuff. And where is it coming from?
- 12:47So business metadata to of like labels and definitions, we still
- 12:53only like have more and more of that, nothing like the stuff
- 12:57like Munson and data Hub. If you're familiar with these
- 13:00like open source project is trying to treat all that
- 13:03metadata like bring it on a central place when you feel kind
- 13:07of a graph of all your metadata and your organization or all
- 13:12your data assets and bring it all together.
- 13:13Together so you can see kind of neighborhood.
- 13:17So you look at a table, you know, where it's coming from
- 13:19what's upstream and downstream of it was the owner that kind of
- 13:22stuff that's of is like the more I think the more your data
- 13:27platform gets complex and ages and has history, the more
- 13:31important that is to like make sense to stuff the ways when
- 13:34you're stuck with like you know, tribal knowledge and asking your
- 13:37peers, like can I trust this table?
- 13:39You know, is that reliable? Yeah.
- 13:41Who owns this stuff? So that's, that's an interesting
- 13:46thought, you know, that metadata as data experiment and talks
- 13:51about like how do you know they exhibit B?
- 13:53You know, they were, there's like tons of data everywhere and
- 13:56it's like growing, it's potentially, I think it's, you
- 13:59know, metadata is similar. It was a challenge to.
- 14:02Yeah, it's your point about the data platform and its age and
- 14:07it's the relative knowledge and then, you know how it kind of
- 14:11becomes like hey you know, what's going?
- 14:13Hear what you think about this table?
- 14:14What is this feature? How does that actually computed?
- 14:16How are we actually Computing? Like, whether someone's
- 14:18defining, someone with this moment active or not, right?
- 14:20That's where all this like, tooling for tooling starts to
- 14:22evolve and, you know, the question I have for you, which
- 14:27is actually very much like my job.
- 14:29So I'm gonna ask you to do my job.
- 14:30Here is, let's say you're in a rapidly growing business and
- 14:36said rapidly growing business has mountains and mountains of
- 14:38data and very, very pressing business questions to answer Oh
- 14:43hey, we have a business model to figure out here.
- 14:45And you need to tell me whether or not my tab is what I think it
- 14:48is today. So you can answer.
- 14:50All these analytical questions doesn't feel like you have a ton
- 14:52of time to model the data and it doesn't feel like you have a ton
- 14:54of time to think about. Well what are the entities that
- 14:57map are rapidly to a map to a rapidly changing business model
- 15:01and say, said data, professional was in a team of two people with
- 15:05the team of two people. How would you go about wrapping
- 15:09your arms around this problem like data Organization for a
- 15:12rapidly scaling. All right.
- 15:15What's the first as I tell them is mostly, you know, an external
- 15:20problems that beset? Maybe that's not the best
- 15:22example, the Tammy just like, sure.
- 15:23Sure. Here I need specialized in like
- 15:25that us already done the analysis for the time of your
- 15:28Market. Well, let's say, what if it was
- 15:30a new market, that no one really knows about maybe an Emerging
- 15:33Market. Yeah, I think that becomes it's
- 15:36less of an internal and of the warehouse type question and
- 15:39maybe more of a research problem for the case of town, but let's
- 15:43Take. If if the question is more like
- 15:45York actu, LTV ratio customer acquisition, acquisition last /,
- 15:50lifetime value of customers. And pretty intricate complex
- 15:54like entry looking question inside your organization and you
- 15:59have to say, what is the vat, light, lifetime, value of a
- 16:01customer? You need to figure out, you
- 16:03know, look at your history and your customers and do some
- 16:06projections and Cactus much. Does it cost me to get a
- 16:09customer as pretty introspective of a question too.
- 16:13How do you do this? I think for me, I think
- 16:17historically or ones at the pattern that I've seen in the
- 16:20past is if you're in a rich, Jewish organization is you're
- 16:23going to do whatever you need to do to get an answer that
- 16:26question this week and that will typically will be, you know, a
- 16:29data analyst or I guess nowadays, maybe it's more an
- 16:32analyst engineer who's like, going to go and hit some raw
- 16:35tables and write a shit on a sequel and then maybe get a, you
- 16:40know, half these that answer to that question.
- 16:44And in fact, the last week. Yeah.
- 16:47Yeah, I didn't. Yeah, so that might be like a
- 16:49little bit better, put a finger in the air and then the best
- 16:52answer, I could get you this week, is this?
- 16:54And then and then hopefully this person, you know, and maybe is
- 16:58the same person, maybe it's a different person.
- 16:59But someone else is working on no closely with this person and
- 17:04they would be thinking about like, how do we do this, right?
- 17:06So we can get an answer to this question every every week for
- 17:11the foreseeable future and we can actually measure the And
- 17:14that's a little bit more of the data engineer Persona, who that
- 17:18tends to work at a much slower Pace, but does things right?
- 17:21Like builds things that are there, that will be there in a
- 17:25year or two from now and maybe five years from now, you know,
- 17:28we will still this person treats, you know, their art,
- 17:33like something their craft, or what they do, as a little bit
- 17:36more infrastructure that will have to be maintained in the
- 17:39future. And it may be is it the same
- 17:41person or not? Probably not, right?
- 17:44So one is probably vertically aligned and like, working on a
- 17:48weekly timeframe and the other one thinks more in terms of like
- 17:51maybe quarters and you know, and then use the source control and
- 17:56that's code review and thinks about data modeling and costs,
- 18:00and make sure that's everything that they've built so far, will
- 18:03lend every morning before 9:00 a.m. are, you know, has some
- 18:06essays and that sort of thing. So, the answer is like, at the
- 18:09end data, as in many, many things in fast, moving business
- 18:12has yet to do both. The short term and the long term
- 18:15solution in parallel. At the same time, the reality is
- 18:18like, do you always add have the resources to do both the often
- 18:22dry? So it's likely to give an answer
- 18:24this week and you put in your pilot project and maybe you
- 18:26don't get to it until something's on fire or you
- 18:30really do, you know, Seth or an executive's.
- 18:32Like I need my cocktail TV every every you know, every Monday
- 18:35morning, I'm kind of processing that answer because there's a
- 18:39lot there's a lot to dig into their and but you know what
- 18:42sticks with me? Is this idea?
- 18:43idea that like, yeah, you're right, somebody's got to be
- 18:45working on doing on fighting fires and somebody's got to be
- 18:48working on, sort of building the You know, building the
- 18:52foundation. So to speak, write and splitting
- 18:55that difference and putting two different roles there and having
- 18:58two different mandates to saying like we're going to be doing
- 19:00both in parallel is really important for I think data teams
- 19:03to both you know build and maintain trust and also like
- 19:08support the business for the long term, right?
- 19:10Because it's the twin mandate that you kind of have when your
- 19:13data professional. One thing I wanted to ask you
- 19:16about coming off of that is, you know, this conversation has kind
- 19:20of started off on a very In general sort of a data team data
- 19:26team focused, right? Like data Engineers analysts
- 19:28like what's what's generally happening in the modern data
- 19:31world and you know it sometimes feels like the machine learning
- 19:35world is like sometimes a subset of that and other times like
- 19:38whole world of its don't, you know?
- 19:40And you know they're kind of like slightly different forums.
- 19:44There's slightly different experts and I'm kind of
- 19:46wondering what your reaction is to that.
- 19:47Like do you feel like the way that we should think about
- 19:50machine learning tooling and stuff?
- 19:51It's just A subset or is it a little bit of its own?
- 19:55Yeah, it's there's, there's an interesting parallel to be drawn
- 20:00here. And if you saw that, you could
- 20:03say the future of data centers in our wits, came up with the
- 20:05modern data stack diagram, so that blog post and there, there
- 20:09was going to version 1.0 of it, which brought ML and did it like
- 20:13the whole data platform with MLS part of it in a single diagram.
- 20:19And then, for their 2.0 version of it, they can Did they decided
- 20:23okay, this is too complicated, it looks like at the spaghetti
- 20:25plate, you know? And then let's take the ml stuff
- 20:30and make it in its own diagram which is kind of interesting
- 20:35that I'm not sure if the fact that they did that, you know, as
- 20:37informative or you know, it'd be interesting to look at the
- 20:41differences between the two side-by-side.
- 20:43The, you know, I don't know if we could pull it here and and
- 20:45look at like what's the Venn diagram of that?
- 20:48Like what is on boats on both diagrams?
- 20:50And what is very unique to one? Or the other to get those relics
- 20:54of the data warehouse question, you asked before I like, so are
- 20:57we kind of converging or diverging?
- 20:59Here it is. Like clearly.
- 21:03It is different people with different concerns and different
- 21:07backgrounds operating on different time frames.
- 21:10I think on the ml side like notebooks are just really great
- 21:14which is what is a notebook, really.
- 21:16It's a program that you execute, you know and not Is the
- 21:23sequential order, statistic the repple diet, so the rebel cells
- 21:27like really works, I think it's important.
- 21:29And then for NFL practitioners and in the open-ended you know
- 21:35power of just like an interpreter as opposed to you
- 21:38know something like SQL is not sufficient, that's for sure.
- 21:43So yeah I think you know unfortunately I think it's it's
- 21:45at the answer is like it's an intricately shared resources
- 21:50across the two and then Some areas were there, like it's
- 21:54intricately, you know, diverging in some ways and if you think
- 21:58the funkier stuff, if you do, take the fun to your stuff on,
- 22:01the MLS that on the MLS side of the house, or that is like,
- 22:05let's say, like, you know, notebooks notes are also useful
- 22:09for data Engineers to write, like so it, maybe they would
- 22:11spend more time in SQL in general, and less time in
- 22:14notebooks. Back notebooks are still like it
- 22:16pretty great paradigms. If anyone you're trying to do
- 22:18there might not be an ml use case before more.
- 22:22Could things and did engineering, you might want to
- 22:24use no choice to. So, let's, let's pull on that
- 22:30thread a little bit more because it feels to me like, like
- 22:33airflow is very much a data engineering thing, but then a
- 22:39ton of data scientists and machine learning Engineers, love
- 22:43it. It's like some people have given
- 22:45talks on the community meet up where it's like air flow.
- 22:50The unsung hero of the The data scientists even.
- 22:54And so I wonder about the. Did you have that in mind was
- 22:59like when when you're looking at creating such a tool like that
- 23:03or when you're going through it, or you solely focused on a data
- 23:08engineer and then it just so happens that the data scientists
- 23:12also found a lot of value on it or were you doing research?
- 23:16Looking at what the data scientists use cases.
- 23:18Where to? Yeah.
- 23:20So there's this fair amount of History.
- 23:22Be there to select first, the inspiration for air flow.
- 23:26So, no air flow as I started at Airbnb 2014.
- 23:29Actually, I started in between jobs and between my job like
- 23:33Facebook, and I taken the job, you know, Airbnb.
- 23:36I talked with the team and it was really clear that I was
- 23:41going to work on something like the airflow because they didn't
- 23:44really have anything like that. And then I was coming out of
- 23:47Facebook, where Facebook ad had this Cambrian, explosion of,
- 23:51like, Inventing the whole data platform from the from scratch
- 23:55because the data the scale of the data were just too big to
- 24:00fit and Oracle or teradata database, right?
- 24:03So then they had invented them things like higher than Presto
- 24:07or had been in the making while I was there.
- 24:10But I build their own visualization tools.
- 24:13On top of that. I built their own kind of data
- 24:16portal at the metadata repository.
- 24:18I was talking about we were we had built one of those, we had
- 24:21had problems. Really like I would imagine a 5
- 24:24to 10 different schedulers and you know are like that's
- 24:28processing come that you generate computation type things
- 24:32like air flow and then when there was there were 22 of those
- 24:37had been emerging internally at Facebook, one called this one
- 24:40arm and one called day to be. And they're both, you know, a
- 24:43way to run that's jobs like bags of bad of that dags of batch
- 24:49jobs, which were expressed as new.
- 24:52No Sequel and or python. So so at that point you know and
- 24:57it's warm Facebook had been used for a general comment like
- 25:00General competition was used by their scientist was used by all
- 25:04sorts of people wearing all sorts of different hats just to
- 25:07do like data pipelines, which included bunch of emelda prepuce
- 25:12cases and probably is improper, you know, training as well.
- 25:16So I think like I knew so what do you think about what airflow
- 25:20is is just Either Express dags of batch jobs, with that are
- 25:26kind of time bound and you know, airflow will just like allow you
- 25:31to express sets of jobs that depend on each other and provide
- 25:36guarantee about the order of those jobs and one are going to
- 25:39run. So that's generally when you
- 25:41think about, it's not even like a data pipeline, specific thing,
- 25:46where it's very generic. Like it's a scheduler, you know,
- 25:50and an orchestrator. Stir.
- 25:52And then what you're going to do in there, you know, as kind of
- 25:55free to like limited only by you know what you can trigger and
- 25:59python which is like you know, pretty much anything.
- 26:02So that I knew it was going to be used in all sorts of ways but
- 26:06you know we were talking earlier about all the funky stuff that
- 26:11people do you know in the history of data and I the
- 26:15airflow certainly enabled people to do all sorts of things and
- 26:19you know at whatever ways. Now there's a real That article,
- 26:22if you saw it, there's a article called that unbundling of
- 26:25airflow people. I found it.
- 26:29Well, yeah, it's actually pretty, it's a cool article
- 26:32because it shows like a all this stuff that you've been like,
- 26:36doing an air flow which is like generate computation generally
- 26:39around data, you know, all these things that you've been doing
- 26:44right now are getting like cater to and in a more directed way,
- 26:48right? So if you're doing a maybe you'd
- 26:50better do that and give you T if you're doing Data sync and bring
- 26:53data from her staff services. Then you should just, you know,
- 26:55five trainer are bite. It's got an issue, interesting,
- 26:59take on it against things specialized, there's like
- 27:03services that emerged at do certain things very, very well
- 27:06and an airflow is pushed back to become much more of an
- 27:08orchestrator, and less of a compute engine, like, generic
- 27:13batch compute engine. I'm Emanuele.
- 27:16Mason machine learning engineer at stripe and author of building
- 27:20machine learning powered What if you don't want your machine
- 27:24learning models to explode? Well, you should subscribe to
- 27:27this podcast. So I am, I have to ask before
- 27:32Vishnu jump in was and you've probably been asked this a
- 27:37million times. So forgive me for just following
- 27:40the crowd as as you see astronomer doing what they're
- 27:45doing like that was never a thought in your mind.
- 27:48Or did you analyze trying to do like a managed airflow and you
- 27:52decided not to how did that play out?
- 27:54Oh yeah. So, so for context to for people
- 27:57who don't necessarily know me and the history said.
- 28:00So I started airflow, I started another project called superset,
- 28:04so Apache airflow Apache. Super said I started both of
- 28:07those at Airbnb in 2014-15. My passion really wanted to work
- 28:12super sad because it's visual it's interactive.
- 28:15It's color full. I'm kind of you know I wanted
- 28:21this rub business. Halogens and I want to change
- 28:23the way that you know, Mortals work with the data every day and
- 28:28that's just like a much more or at the time and now start things
- 28:32more interesting to me than data pipelines and orchestration.
- 28:36So after like a maybe two years of airflow is like okay I build
- 28:40this thing. You know I'm going to let it fly
- 28:42and let it grow and my passion really went to Super set and you
- 28:45know ended up starting a about three years ago.
- 28:48I started a commercial open source company around soup.
- 28:52For said called preset. And that's really just going to
- 28:54following my passion and interest and the stuff I wanted
- 28:58to work on. And I also like the risk-reward
- 29:00I think is more interesting or its higher risk higher reward on
- 29:07the dative is you know data consumption layers.
- 29:10How about like 4:00? Let's yeah let's talk about it
- 29:14but first like four people don't know what superset is urge you
- 29:18to check it out, right? So superset is an open-source
- 29:21pretty Say data consumption, dashboarding data, exploration
- 29:25data visualization platform. It's very much a competitor to
- 29:29things like, you know, Tableau and liqueur so that all the
- 29:31things that you can do with Tableau in look here you can do
- 29:34with on the open source front with super said we've been
- 29:38working super hard on it so it has become really awesome and
- 29:42really competitive the risk-reward ratio is higher to
- 29:47me because it's such a very like red ocean.
- 29:52Market. And there's just so many vendors
- 29:54and business intelligence harder to get open source, has to come
- 29:58in and disrupt, because there's like, very, very established
- 30:03vendors where orchestration was kind of this little bit more of
- 30:06a new segment. So I did think, you know, about
- 30:09starting a, you know, to go the more the direction of airflow
- 30:14and you know, investors kind of came to me and I hate we don't
- 30:17you start an airflow company and my heart and passion was very
- 30:21much a gun a superset. And then I've been really
- 30:24friendly and following closely the folks at astronomer so they
- 30:28they tried to get me to join too early on.
- 30:33I was talking with them up super early on and you know I'm very
- 30:37friendly with them and love it invested to some and an advisor.
- 30:41What them too. So I still have passionate talk
- 30:44about this stuff and being close to that world.
- 30:47But my really focus has been on the data consumption.
- 30:50Layer did a Ation dashboarding and how people like make sense
- 30:56of data, as you know what I've been working on.
- 30:59So you mention it is a red ocean that you're getting into like in
- 31:06a very blunt wait like what makes you think that you can
- 31:12disrupt this field? Well, so death takes, there's
- 31:16like two big angles and I think like through Innovation, you
- 31:20know, I think we can disrupt and Startups that have been
- 31:23disruptive but I would say there's two things that that we
- 31:26do that are very disruptive one is open source that like open
- 31:29source is just like this tidal wave of disruption, right?
- 31:34It comes it comes in and what it's, you know, it's free as in
- 31:41Freedom and free as in beer to so that means that you can just
- 31:44pick it up. Use it and get value off of it.
- 31:47I'll disruptive is that, you know, that's like a version of
- 31:49Freedom. You can also like extended you
- 31:52You can you can you can even build you know, integration
- 31:55points and we can extend things, you can make it your own.
- 32:00That's extremely powerful. I think people are pretty fed up
- 32:04with vendor lock-in to, and proprietary languages and tools.
- 32:09And, you know, just being at the mercy of companies, now, like
- 32:14Google cloud and on a side of looker and sales force on the
- 32:18side of Tableau. So you don't I think You know,
- 32:22people have a new way to discover validate and pick and
- 32:28ramp up on software and that new way is, you know, closer to what
- 32:33open sources or SASS. And that's the other, the other
- 32:36angle of like, how we can disrupt in this market, believe
- 32:39it or not is just like some stuff that comes with SAS like
- 32:42freemium. So, you know, preset offers free
- 32:46version of super set up to five users, for instance, and you can
- 32:49just like prove value and use it ran.
- 32:52And up to five users and make sure that it works for you
- 32:56before you even talk to a sales person.
- 32:57You might not even have to talk to a sales person ever, which is
- 33:00fantastic. So I think like, you know, all
- 33:03the things that come from SAS to things like time to value in our
- 33:08case like time into first dashboards time to dashboard,
- 33:10he's connecting to things that are really focusing on that
- 33:14early Journey user experience. And then using their mechanisms,
- 33:19like re-engagement emails. And things like being able to
- 33:24send people emails saying, here's some dashboards.
- 33:27You might like in your workspace or in your, you know, that
- 33:30people in your team, I've created.
- 33:31So the things that the Macan isms of, you know, your typical
- 33:36SAS company as applied to data visualization VI make total
- 33:40sense to and we'll make it more more virata.
- 33:43More variety to know? Yeah, it's, you know, this
- 33:47conversation went exactly the direction.
- 33:49I was hoping despite my complete lack of involvement, so, Oh,
- 33:52it's outstanding to see that but, you know, I guess what I
- 33:57want to share here is I work at a company where we struggle a
- 34:01lot with how to do business intelligence, right?
- 34:03And what that means is not necessarily that we can't figure
- 34:07out whether to buy Tableau or power bi but really that we are
- 34:10like we want business intelligence to work in a
- 34:12fundamentally different way than it has been, which is like, put
- 34:17up a dashboard of kpis and check it every day.
- 34:19And over time, people stopped paying attention.
- 34:22Six weeks and people are like is that even around?
- 34:24And you know we've had that experience was like quick site
- 34:28and you know the other sort of standard stale offerings.
- 34:31You could say that are out there and I think for us what we find
- 34:34is that you know, there are really two use cases, right?
- 34:37One. It's just like just monitoring
- 34:39to make sure that certain business things that need to
- 34:42happen are happening. And then the other thing which
- 34:45is like more insight driven which is, you know, what are
- 34:49some things we should be doing or what are Something we're
- 34:52aware of our need to explore and get ahead of before it's
- 34:56happening, right? And I would really say like we
- 34:58want to empower that ladder piece a lot more but really
- 35:01struggle with it and I was wondering you know from your
- 35:03standpoint from your Vantage Point like you've been at like
- 35:06your Facebook Airbnb, your now, helping no commercial as
- 35:09superset and bring it to the world.
- 35:10Like what are some of the solutions that were the ways
- 35:14that companies should think about that ladder problem of
- 35:16empowering, that sort of exploratory work and getting
- 35:18Mortals superpowers with data? Yeah.
- 35:22You know, there's your you were talking about like one angle,
- 35:26which is like, how does data inform even then, you know, the
- 35:29intuitive part of like, what we should be working on or what we
- 35:33should not be working on Based on data.
- 35:35But the way, one way to think about it is that things like
- 35:38data is kind of a dimension of all the work that every worker
- 35:43does every day. So if you're starting your work
- 35:47out, a new product within your company, like a news, the new
- 35:50SKU or new feature. ER, if you're or essentially for
- 35:56any, whatever role inside a company, whatever workflow,
- 36:00whatever process that data is a dimension of that.
- 36:03And really often, we don't use data to power like much of the
- 36:07processes and much of the role. So you might work in customer
- 36:10success and at a company, right? And not be data-driven.
- 36:15I'm just pick up your tickets and get on the phone with people
- 36:18and help them. But there is a huge data
- 36:21dimension of This yes, work, right.
- 36:23And you could ever get really intricate set of dashboard
- 36:27around. Not only what?
- 36:29This? Yes.
- 36:30Team a customer success team does every day, but also like
- 36:34the new things they might want to try our that they might want
- 36:36to do. So there's like, you know, so
- 36:39essentially the data Powers every role and every processes
- 36:44and there's a question like how do we make sure that, you know,
- 36:48all of the data that is needed by most of the people?
- 36:52EBay and organization is available to to create the
- 36:56validation. And the opportunity is validate
- 37:00the Intuition or invalidate some intuition.
- 37:03Really often that did is just not there, right?
- 37:05We don't have the data is just like, it's engineering, like, we
- 37:07don't really have the right now. What to do.
- 37:11So yeah, I don't know if this is like insert answering that
- 37:15question that I was like how do we become more data-driven?
- 37:17I think like it's there's a hunch of like how do you show up
- 37:20to a meeting with like the State question of like a.
- 37:23I wonder what our percentage of you know our success rate for X
- 37:28is then it's like well, instead of just wondering about it went
- 37:32on someone, I can go and get that data and bring it to the
- 37:35next meeting. Or maybe it's the data is
- 37:38already there, so we can validate validate hypothesis
- 37:41instead of it as just like getting my intuition versus your
- 37:44intuition. Is this not super Concrete in
- 37:46useful? So these are these are
- 37:48organization that I was at are like lift Airbnb.
- 37:52Facebook Yahoo that. I think you'd have a lot of
- 37:54that, culture of like, I wonder, you know, about X.
- 37:58And let's get the answer to that question and live.
- 38:00Let's get that data and bring it to the table.
- 38:03It's not accepting, you know, raw intuition without data.
- 38:09Yeah, it makes sense and it's the cliche, right?
- 38:13Like the data-driven organization and you hope that
- 38:16most organizations are doing that.
- 38:19And so changing gears a little bit.
- 38:22It. You spoke about how you play in
- 38:26the data engineering field quite a lot, but you also are building
- 38:30for data scientists and machine learning Engineers because a lot
- 38:35of the stuff that you do inevitably touches on their
- 38:39space to are there things that you found over time as a builder
- 38:46and the data engineering space that you need to watch out for
- 38:50when it comes to building. For data scientists in terms of
- 38:55tooling, or did a pipelines or both?
- 38:58Yeah, I would. Let's start with data pipelines,
- 39:01but if there's something that's calling at you, for sure, jump
- 39:05on that. Well so I think like, you know,
- 39:09big part of the science is you know making models and big part
- 39:13of making models is training models and big part of like
- 39:16trading model is about like prepping the data to train your
- 39:19models and being rigorous food. To be a rigorous around that
- 39:24which you should be, especially given, you know, the impact that
- 39:28we see in that, you know, ml can have on and what in the world
- 39:32like you look at the Facebook like, you know, news fit
- 39:35Newsfeed ranking algorithm and the impact that has had on like
- 39:40parallelization and our society. You know, you're like okay the
- 39:44way that we train this model or this machine learning Beast
- 39:51seems Is really, really important and, you know,
- 39:54provenance and reproducibility for something like that.
- 39:59Seems really, really important. So that means like a lot of the
- 40:02best practices. I think to make sure that data
- 40:07engineering is done a right does apply to data prep for training
- 40:11models. So in concerns around reproduce
- 40:14the bowl of reproducibility Province.
- 40:18What was this mod? What was this machine learning?
- 40:22All trained with and can we reproduce that?
- 40:24And then becomes very important. So I would say the work that I
- 40:31did around sites, I did some, I gave some talks and wrote a blog
- 40:35post on something called functional data engineering,
- 40:37which is applying some of the functional programming Paradigm
- 40:42to date and real. It's like trying to draw some
- 40:44parallels between the two and seeing where these parallel
- 40:47makes sense, then there where their virtuous and And and then
- 40:53encouraging people to go in that direction to say like a or jobs
- 40:57should be pure functions, your tasks, you know, when you do
- 41:01pipelines to be liked or functions, you should use
- 41:05immutable data blocks and create other immutable data.
- 41:08Bluffs, and you should have like deterministic results.
- 41:11Like when you run this function, with this immutable data block,
- 41:14you should always get this result so that you can get to
- 41:17some level guarantees around reproducibility or at least be.
- 41:22Able to explain their results that you got.
- 41:25And it's like, that's important. So, the rigor and data,
- 41:28Engineering Process does apply to everything related to like
- 41:31preparing data, sets to train models.
- 41:34Yeah, I actually I love that point.
- 41:36The last point you made you for those of you who haven't read
- 41:39them, go check out Max's blog posts.
- 41:42Especially you know, that one about functional programming in
- 41:44the data context. Like there's so much of it work
- 41:47that we do as data professionals and is machine learning
- 41:49professionals is just applying the principles of software.
- 41:51Nearing to data, right? Like that's what it's about a
- 41:55lot of times it's like figuring out where it works and where it
- 41:57breaks right? And it's a really cool blog post
- 42:00that I would check suggest checking out and with that I
- 42:04want to get into some rapid fire.
- 42:05Some lightning round stuff. We got some we got some stuff
- 42:10lined up for you. I want to go first what was the
- 42:13craziest crazy man? Have the right programs you use
- 42:16it, craziest work environment for you to walk into Facebook
- 42:20and 2012 Airbnb in Or lift in 2017.
- 42:23So Define crazy. But I would say like the most
- 42:26surprising and changing was Facebook into in 2012, huge
- 42:31environment and reinvented everything from the ground up,
- 42:35super Dynamic, no rules. And by the same time like
- 42:40tremendous and like amount of amazing work taking place so
- 42:45that that was before we knew that Facebook or before a lot of
- 42:48the issues that came up with the facebooker to is like I was
- 42:51Before there was all that, this doubt in the air.
- 42:54But, um, you know, moving fast and breaking things and really
- 42:58living by. It was really impressive and,
- 43:01and power and power. See like I'd never seen anything
- 43:05like it, and never imagined that it work.
- 43:08Replaced could work with like, loose rules.
- 43:11Like that was pretty and amazed to be part of it and really
- 43:14empowered by it to drink Kool-Aid.
- 43:17For sure. I gotta say like those three
- 43:20things is on your on your resume.
- 43:22Times that it's at. It's kind of like playing for
- 43:23like Barcelona then Real Madrid. Then like you know, Manchester
- 43:28City, like one. Actually the next question that
- 43:31I have rapid fire is which of your blog posts.
- 43:35Were you most surprised by the reception to goodness?
- 43:39I think I like the rise of the day engineer was like super well
- 43:43receive, I'm not really surprised by it because I think
- 43:46I had to read maybe five years before the rise of the data
- 43:50scientist was like I'm gonna do this.
- 43:52For the data engineer, you know, and I think ideas, like likely
- 43:55to be fairly big because I knew that the role was going to be a
- 43:58big role. He starts to get her to data and
- 44:02interacting together as just such a powerful.
- 44:04You know, emerge of concept that I knew was going to be big but
- 44:09but it's still quoted many. What's surprising is just like
- 44:11the along and and how like a the life cycles it seems pretty only
- 44:18I would have thought like, well people read it.
- 44:20You get some comments in the first.
- 44:21Like few months and then it fades away, never talk.
- 44:24No, nobody talks about it again now.
- 44:26It's still like people still read this blog post.
- 44:29Yeah. Okay.
- 44:31So what is a piece of technology that would surprise people that
- 44:36you're bullish on? Well, that's a that's a tough
- 44:41one. I'm not sure if I feel like I
- 44:43need to really scan through my head, all the things that are
- 44:47controversial, you know, maybe I'll bring something, that's
- 44:50controversial. And bring up the angle that I
- 44:54like. But I don't know if you've heard
- 44:55about data mesh and it's been controversial on like data
- 44:59better and now aware because I think like, it's not very
- 45:04prescriptive in terms of like Solutions and what it means for
- 45:08data modeling or for today two products, you know what really
- 45:15it means. But I think it does a really
- 45:16good job at summarizing the issues that we have around the
- 45:21day. Governance and you know how
- 45:24people should work with data and modern organization.
- 45:28There's new to YouTube point, you know?
- 45:30It's like people don't really know what are the what are the
- 45:33roles than what are they approach is and how do we do
- 45:36data, right? The, you know and large are fast
- 45:38growing companies so I think they Identify some of the
- 45:41challenges very well. I don't think they offer like
- 45:43really clear solutions to it. But they do a good job at summer
- 45:49I think on the issues of our time.
- 45:50So you know, I do Data that a team culture, it's hilarious.
- 45:55You say that because we when we've had people on here that
- 45:58talk about data mesh, I liken it to World Peace.
- 46:03It's like yeah everybody's for World Peace like don't know,
- 46:06Buddy's not saying we don't want world.
- 46:08Peace is just like, how do we go about that and how do we get it
- 46:12like in actually execute on it? That's another story.
- 46:15Like it's kind of in that I throw it in that field of my
- 46:20brain. So All right, I got more on
- 46:23that. It's like when you read the
- 46:24paper you like, oh my God, this is so good.
- 46:27Like I relate to everything they say.
- 46:29And I can't wait to do to get to the chapter, where they offered
- 46:32the solution, and solve it all, and then he read it, it's like
- 46:35treating data like products and like, okay, that's kind of what
- 46:38I've been doing forever, right. And now and it's like all they
- 46:42just rename this concept of different name and that still
- 46:45does its kind of Falls flat a little bit on.
- 46:47Yeah verified. It calls out the problems and
- 46:51the pains Really well. And speaking we were talking
- 46:54about Chad Sanders earlier, and he's very much drink that
- 46:58Kool-Aid. And I think he's one of the ones
- 47:01in my eyes, that is exploring how to do it and what the
- 47:06remedies could look like at his own job he's trying to like push
- 47:10really move the needle and so that's that's a cool one.
- 47:15Alright. Last one for you and I'm torn
- 47:19between whether I want to ask you what the last book you read
- 47:22Or what the most ridiculous piece of marketing around data.
- 47:25You've seen has been, so I'll leave it open for you.
- 47:30Oh, goodness, that I mean, I think that the message might be
- 47:34like, it's a little bit, like every kind of a Consulting, like
- 47:37marketing play or, you know, there's a little bit of like,
- 47:41you know, naming things kind of land grab there too.
- 47:44So I think that we go there, I picked up a book that I just
- 47:49started, and now I've got to go look up the exact site.
- 47:51Tell, but it's about the the the life of Frank Zappa As Told by
- 47:56is like life partner and I do know that because I have to go
- 48:01and dig out the exact name. I read like the first chapter
- 48:04and not like take my life with Zappa.
- 48:07It's something like that. I'm just really interested in
- 48:09Zappa is like, you know, a musical genius kind of very
- 48:15interesting character to see the documentary.
- 48:18The recent one that came out. What was it called?
- 48:22Yeah, what was it called? Know like yeah, maybe but I'm
- 48:28gonna go look at my Amazon purchase list.
- 48:31There you go. But we can put in the show notes
- 48:34like let's just thrown on the documentary was great.
- 48:37There's something amazing about is that he's been kind of a
- 48:41hoarding as on material so you've been recording
- 48:46everything. So it's got like a is like he's
- 48:50got like a Like somewhere and is basement and one of his houses,
- 48:54it's got all the tapes and all the movies and all of the
- 48:58everything we ever recorded. So we can expect a lot more of
- 49:02the treasures to emerge out of that.
- 49:05Those are guys, which is amazing.
- 49:07I'm sure you've seen Zappa Plays Zappa Yeah, there's no result is
- 49:12a delete. Yeah, I haven't seen it.
- 49:14I know, I know it exists unless big andand Wizzle, you know,
- 49:17really? I'm sure it's a, yeah, I just
- 49:20don't know. I'm don't know him as much.
- 49:22It's crazy. How much so Dweezil, like looks
- 49:25like him and like acts like her and everything is so crazy.
- 49:29How similar it is. I like yeah dude his guitar
- 49:33playing skills. Are ya off the chart?
- 49:35Like like Zappa, like Frank. But I went to a few and they get
- 49:41the old legends that used to play with Frank to come on and
- 49:45just like tour with weasel or make some guest appearances.
- 49:50So I saw Return to Forever and and then Dweezil play with like
- 49:56Zappa Plays Zappa. Yeah, it's definitely worth it.
- 50:01Yeah, yeah. Yeah.
- 50:10Is. But American genius.
- 50:12Yeah. And check it out and maybe it is
- 50:14for you, if you haven't really tried, you know, what's a good
- 50:17Gateway album for people to start Joe's Garage, is my
- 50:20favorite Joe's Garage 80s, more like fun on the funny side.
- 50:26Yeah, by so many good album for the good jazz people are a
- 50:29little bit more coming from jerez.
- 50:31There's a Grand, Wazoo and wife it junghwa.
- 50:33Yeah, I really like hmm freak out and start with these
- 50:38recommendations. Well, And then it started.
- 50:41Stop a podcast. You don't speak out on that.
- 50:45Well, I mean, I could right now that's a so funny.
- 50:48You're saying this because one of my favorite things in that
- 50:52documentary that they talk about is how they asked him like, why
- 50:56are you doing this? You're spending all this money,
- 50:58you're going overseas. You're like Contracting
- 51:00orchestras, to make your music and he's just sits and they're
- 51:04like, and you're not going to make any of this money back.
- 51:07And he tells them straight up. He's like if it's music.
- 51:10That I want to listen to I've won.
- 51:14It's just like that's how I want to live my life.
- 51:16Like if it's worth doing because I will enjoy it, then I want to
- 51:21do it and I want to make it happen principal delivered by.
- 51:24Everybody says their new science.
- 51:26You want to consume or I don't know if there we go.
- 51:30We can extrapolate that into a moab's world.
- 51:33Maxine has been amazing. Man.
- 51:36Thank you so much for coming on here and sharing your wisdom and
- 51:39advice. To us all awesome.
- 51:41Once I give us pleasure to be on the podcast here.
- 51:44Thanks for coming.