Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah
Transcript
- 0:00SPEAKER_01: We build those models and we have them running in Airflow, and this powers like downstream systems and dashboards and everything.
- 0:07SPEAKER_01: But the insight we got from that allowed us to be able to see that professionals who were using some of these marketing tools that we provided were able to 3x their growth, right?
- 0:18SPEAKER_01: So this was very, very important because then we could see that, oh, if you made use of XYZ number of tools, it means that you could potentially see your revenue grow by 3x.
- 0:34SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering in AI.
- 0:45SPEAKER_02: Hey Airflow community, exciting news.
- 0:48SPEAKER_02: This fall, we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
- 0:55SPEAKER_02: Eastern.
- 0:56SPEAKER_02: You'll hear from data leaders on how modern data ops and orchestration are powering AI, machine learning, and production grade data products.
- 1:03SPEAKER_02: And here's the best part.
- 1:04SPEAKER_02: You can join Mark Lumberti for a fast track Airflow 3 certification prep session and get a discount code for a free certification, a$150 value.
- 1:13SPEAKER_02: The conference is virtual, it's free, and it's made just for you.
- 1:17SPEAKER_02: You can register now at astronomer.io slash beyond slash data flowcast.
- 1:26SPEAKER_02: Hi everyone, welcome to the Data Flowcast.
- 1:29SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Pascal Onora, who is a senior data engineer at Style Seat.
- 1:36SPEAKER_02: Pascal, how are you?
- 1:38SPEAKER_01: I'm fine, thank you.
- 1:40SPEAKER_01: How are you too?
- 1:41SPEAKER_02: Yeah, doing good, enjoying the summer.
- 1:43SPEAKER_01: Yeah, I had wanted to actually I think I still have plans for that, but I would wanted to travel because I'm currently in Nigeria, but I wanted to travel to like Europe to get a bit of like that experience.
- 1:58SPEAKER_03: Yeah.
- 1:58SPEAKER_01: I've had like some delays with a few things, but maybe by the time this podcast is out, I probably would have traveled.
- 2:04SPEAKER_02: Yeah, definitely.
- 2:05SPEAKER_02: There will be a bit of a delay.
- 2:06SPEAKER_02: So I hope that by the time this comes out, you're like somewhere in Europe doing a summer vacation.
- 2:12SPEAKER_02: That sounds amazing.
- 2:14SPEAKER_02: Okay, well, let's go ahead and dive in.
- 2:16SPEAKER_02: So for today's podcast, we're going to discuss how Style Seat uses Airflow and DBT, a very popular topic.
- 2:23SPEAKER_02: But let's back up a bit and start with some background.
- 2:26SPEAKER_02: Can you tell us what Style Seat does?
- 2:29SPEAKER_01: Yep.
- 2:30SPEAKER_01: Sure.
- 2:30SPEAKER_01: So Style Seat is a beauty and wellness marketplace that basically helps beauty professionals across the US grow and manage their businesses.
- 2:41SPEAKER_01: And we do this using a sort of like business and marketing tools that we provide.
- 2:46SPEAKER_01: Essentially, the goal of what we try to do is we try to help the pros focus on what they know how to do best, which is like the their craft, and not have to worry about some of the nuances of running their business or even just dealing with like appointment schedules with clients and all of all those things.
- 3:03SPEAKER_01: And so to date, we have a network of tens of thousands of professionals across over 16,000 US cities, and we've helped these professionals generate like billions of dollars in revenue.
- 3:15SPEAKER_01: And most of the tools that we build are sort of across three different areas, right?
- 3:20SPEAKER_01: So we have what we call business and marketing tools, and that's things around the scheduling, appointment scheduling, things around client management, things around payment processing.
- 3:31SPEAKER_01: So we have integration with like several partners to make it very seamless to allow these sort of professionals accept payments using things either card readers or tap to pay, or even Clana or whatever it is that works best for them.
- 3:45SPEAKER_01: And around the marketing area, we also have built a collection of tools, one of which is something we call smart pricing.
- 3:53SPEAKER_01: And smart pricing is essentially we're able to look at the pros calendar and be able to figure out like how to optimize the calendar and have to help them to be able to generate the most revenue that they can.
- 4:06SPEAKER_01: And then we also have new client connection where we sort of are able to connect them with clients that they probably hadn't connected with before and just help them grow their businesses generally.
- 4:17SPEAKER_01: And then on the other side, we also provide pros with like financial tools as well, especially providing them with loans to allow them to grow their businesses as well.
- 4:27SPEAKER_01: What we've done at Style City is just created this flywheel effect where pros come to our platform because they want to be able to get access to these tools that we provide.
- 4:38SPEAKER_01: And then on the other hand, these pros that come to our platform and engage with these tools come with their own clients.
- 4:44SPEAKER_01: And then using our marketing tools, we're also able to get them clients.
- 4:48SPEAKER_01: And then these clients are able to find other pros that provide other services that they may be interested in, thus creating like a network effect where like you have pros and then clients benefiting, and just everyone is able to make the most out of the platform.
- 5:01SPEAKER_01: So that's just a little bit about what Starship does.
- 5:04SPEAKER_02: Yeah, very cool.
- 5:05SPEAKER_02: I love hearing about companies like this that maybe aren't super well-known household names, but yet probably everybody has interacted with in some way.
- 5:14SPEAKER_02: Because I assume if you're going to get your haircut or whatever, that it's actually powering a lot of that on the back end.
- 5:20SPEAKER_02: Yeah, that's really cool.
- 5:21SPEAKER_02: And I think you touched on, you know, a few different pieces in that description that I can see already where a lot of data orchestration is going to come into play to make those services happen.
- 5:33SPEAKER_02: But tell me a little bit more about that.
- 5:35SPEAKER_02: So, what does the data engineering team do?
- 5:37SPEAKER_02: And then how does Airflow fit in?
- 5:40SPEAKER_01: Sure.
- 5:40SPEAKER_01: I work with the data engineering team.
- 5:42SPEAKER_01: And then what the team does primarily is we focus on building out the tooling and the infrastructure to help Style State become a data-driven organization.
- 5:51SPEAKER_01: Actually, one of the things we try to optimize for is to make it possible that we can become one of the most data-driven marketplaces within the beauty and wellness industry.
- 6:02SPEAKER_01: And there are four key areas where we try to do this, right?
- 6:06SPEAKER_01: One is, of course, reliable data infrastructure.
- 6:09SPEAKER_01: And this is significantly powered by a tool like Airflow, right?
- 6:13SPEAKER_01: So we're able to build out tooling that makes it possible for like other teams, the analysts, the data scientists to be able to sort of confidently do all of the stuff that they need to do to, you know, allow us to deliver on our value proposition.
- 6:28SPEAKER_01: And then the other thing that we've also doubled down on is self-service and democratization.
- 6:34SPEAKER_01: Also leveraging the tool like Airflow, we've been able to, you know, make it possible that teams across the organization are now able to self-serve and like data is now generally available to everyone, regardless of what part of the organization you're part of.
- 6:50SPEAKER_01: You're able to like make data-driven decisions.
- 6:53SPEAKER_01: And then in addition to that, we're able to then provide sort of strategy thought partnership, right?
- 6:59SPEAKER_01: Because we sit with it with the data, we see what's happening on the data.
- 7:03SPEAKER_01: So we're able to, in a sense, advise the business and help people across different places make more decisions or just generally understand how they can get the most value out of whatever it is that they're trying to do.
- 7:15SPEAKER_01: And then the fourth area is innovations around AI and data science, right?
- 7:20SPEAKER_01: So we've done a few things there.
- 7:22SPEAKER_01: One of the things, for example, is personalization, right?
- 7:25SPEAKER_01: So where we've personalized the experiences for clients, right?
- 7:30SPEAKER_01: When you come on the platform, you're able to sort of the entire thing is kind of tailored in the way that it fits your specific use case, right?
- 7:38SPEAKER_01: Again, this is something that's significantly empowered by some of the value we get out of Airflow.
- 7:44SPEAKER_01: Personally, this is like a lot of what my focus is on on the day-to-day basis.
- 7:48SPEAKER_01: Very involved in collaborating with the analytics team and the data science team to make sure that they have all of what they need, at the same time, also be very involved in building out some of this core components of our infrastructure.
- 8:02SPEAKER_02: Very cool.
- 8:03SPEAKER_02: So you've got, I would say, even kind of a cutting-edge use case there with the MLOps.
- 8:08SPEAKER_02: And I don't know if there's any Gen AI involved in that.
- 8:10SPEAKER_02: But still, even that level of, I think MLOps implementation is something we don't see at a lot of companies.
- 8:16SPEAKER_02: So that's really cool.
- 8:17SPEAKER_02: I'd love to dig in a little bit more about your usage of DBT and Airflow together, which I mentioned at the beginning, since this is a really common pattern.
- 8:26SPEAKER_02: So, can you tell me a little bit more about the use case there and why you chose to use DBT and Airflow?
- 8:32SPEAKER_01: I'm gonna focus on three different areas, right?
- 8:35SPEAKER_01: So one is the I've talked about like the real-time thing that we try to do, basically real-time marketplace intelligence, right?
- 8:42SPEAKER_01: So Airflow really helps us.
- 8:44SPEAKER_01: What we've done with Airflow, right, and with DBT is we have this sort of models that are built around like understanding users and understanding sort of like some of the things they do.
- 8:56SPEAKER_01: So we take like events from multiple sources, including like events that are generated from the ways that users are interacting with the platform, and then we're able to take those things and then build them into like models.
- 9:06SPEAKER_01: And then that's like where the magic that you get with Airflow and DBT using customers comes in, because then we can then have these things be executed and then they feed into some other downstream processes.
- 9:16SPEAKER_01: But at the core, like the sort of the model that powers all of this sort of use case is built with dbt.
- 9:24SPEAKER_01: And then we also have again, so we have a marketplace, right?
- 9:27SPEAKER_01: And as you would get with a marketplace, you would have like search and things around search and booking, right?
- 9:33SPEAKER_01: And so we have this sort of search and booking funnel that we've built.
- 9:37SPEAKER_01: And again, this is something we build out through DBT, where we're able to like use that to generate insights and generate analytics to allow the business to be able to further improve or invest in improving that experience as a whole, right?
- 9:52SPEAKER_01: So that's that sort of unlocks value for the clients who come to the platform to connect with other professionals and you're able to get the best experience because of this particular investment.
- 10:02SPEAKER_01: And then the other use case around that is something I'm going to talk about with respect to marketing, right?
- 10:08SPEAKER_01: So a good amount of the tools that we build for professionals are around marketing, right?
- 10:14SPEAKER_01: Helping them to connect with new clients, making it easier for them to be discovered, even when they probably wouldn't have been previously discovered.
- 10:22SPEAKER_01: A lot of sort of the insight that we use to then know what kind of marketing channels one invest in, what kind of platforms one invest on locking value when it comes to marketing.
- 10:31SPEAKER_01: That again is also powered by Airflow and DBT.
- 10:34SPEAKER_01: Particularly have DBT models that we've built to be able to serve this specific use case, right?
- 10:40SPEAKER_01: So that's on the marketplace intelligence side.
- 10:42SPEAKER_01: On the business intelligence for professionals.
- 10:45SPEAKER_01: So, like I said, these are like tools that make it possible for them to be able to understand their business performance as a whole.
- 10:52SPEAKER_01: Again, this is something that we do through Airflow and DBT, right?
- 10:56SPEAKER_01: Because we have pipelines and data models that aggregate all of the data from the professionals' transactions and all of all that stuff.
- 11:04SPEAKER_01: And then we use that to generate insights that these professionals are able to then look at and then understand how well they are trending towards whatever goals that they may have set for the month or for the year, and just be able to understand what kind of levels they need to pull to be able to then accomplish what all of the things they set out to do.
- 11:22SPEAKER_01: And then the other area which I'm just gonna sort of wrap it up on is like the network effect thing that I talked about.
- 11:29SPEAKER_01: What we do with DBT models there is we're able to look at like client behavior and look at like the way they engage with the different professionals, and then also use that to then say sort of things like, for example, surface what kind of professionals you want to look at, what kind of professionals you want to like help to like optimize things like, for example, their visibility, right?
- 11:52SPEAKER_01: So that these clients are then able to easily find them, right?
- 11:55SPEAKER_01: So those are some of the ways.
- 11:57SPEAKER_01: One sort of maybe real use case or real life use case I'm gonna maybe talk about is so we have this like appointments analytics pipeline and sort of job or process that we invested in.
- 12:11SPEAKER_01: And the insights we got, so we build those models, we have them running in Airflow, and this powers like downstream systems and that spots and everything.
- 12:21SPEAKER_01: But the insight we got from that allowed us to be able to see that professionals who were using some of these marketing tools that we provided were able to 3x their growth, right?
- 12:32SPEAKER_01: So this was very important because then we could see that, oh, if you made use of XYZ number of tools, it means that you could potentially see your revenue grow by 3x.
- 12:43SPEAKER_01: And we took this insight to one, invest in like the experience around those kinds of tools, but more importantly, also like invest in education, helping the professionals to understand that hey, this is something that you should be doing, and this is the exact outcome that you will get if you focus on this kind of thing.
- 12:58SPEAKER_01: So those are some of the ways that we use DBT and AirFloop across the organization.
- 13:05SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
- 13:14SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
- 13:26SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
- 13:35SPEAKER_02: Tell me a little bit about the implementation of this from the technical side.
- 13:39SPEAKER_02: So are you using Cosmos to run your DBT models from Airflow, or what does that look like?
- 13:45SPEAKER_01: You know what's interesting is when we started this DBT journey, we actually weren't using Cosmos, right?
- 13:50SPEAKER_01: So we had this complicated setup, if I would say so, because I was very informed in building that.
- 13:56SPEAKER_01: So we would basically like take DBT models and then deploy these DBT models using like ECS tasks and like then have this ECS task be triggered by Airflow.
- 14:06SPEAKER_01: But the issue with this is all of this infrastructure, these ECS tasks are deployed using Terraform and all that stuff.
- 14:12SPEAKER_01: As in UNAS, probably like easy to like work with those kind of things.
- 14:15SPEAKER_01: But with God that it was complicated for like other people to be able to get into that kind of system.
- 14:21SPEAKER_01: And even though we built this and we sort of were able to like grow that, but we couldn't quite get the usability or get people to use it the way what it wanted them to use it.
- 14:32SPEAKER_01: And so, in a sense, we almost gave up on DBT, right?
- 14:35SPEAKER_01: We're like, okay, you know what?
- 14:36SPEAKER_01: This is not working, let's put it off to some other time until we discovered Cosmos, right?
- 14:42SPEAKER_01: And that was quite the game changer for us because one is that you could sort of have almost like the DBT and the sort of DAG systems like sitting in the same repository, which is what we did.
- 14:55SPEAKER_01: And then it made it possible that it didn't feel like the DBT process was isolated from the ever, like you were scaling both of them together, right?
- 15:05SPEAKER_01: You'd create the model and then you could instantly like write the DAG that's going to execute that model and then have everything be deployed together at once.
- 15:13SPEAKER_01: So that quite unlocked a lot of value for us.
- 15:16SPEAKER_01: But then more so some of the things that was even exciting was before when we had the ECS task, it was almost like you had a black box, right?
- 15:25SPEAKER_01: Everything DBT was running somewhere else, but like the DAG was running somewhere else, right?
- 15:31SPEAKER_01: So when things went wrong, you didn't have that sort of integration to be able to see, first of all, what's the relationship between all of the models and the test and what is failing and why is it failing?
- 15:42SPEAKER_01: Like you had to then go into CloudWatch and try to start searching through the lot.
- 15:46SPEAKER_01: It was tedious, right?
- 15:47SPEAKER_01: So just that value that has unlocked for us, where you know you can see everything.
- 15:53SPEAKER_01: More importantly, when you deploy like a dbt DAG, you have like all of the different models and the different tests.
- 16:00SPEAKER_01: You have visibility into that.
- 16:01SPEAKER_01: You can see the relationship between each of those models, you can see the relationship between those tests, you can easily debug.
- 16:09SPEAKER_01: Like it unlocked a lot of value for us as an organization and as a team, and that's something that we were excited for.
- 16:15SPEAKER_01: And then, of course, one of the things that I was personally excited for was I've worked with DBT across different organizations, and I've worked with Airflow across different organizations.
- 16:23SPEAKER_01: And DBT Docs was something that I thought was like incredibly useful, especially with like knowledge sharing, right?
- 16:31SPEAKER_01: But it was always a concern with like how do we get this when we're using DBT core, how do we integrate this?
- 16:38SPEAKER_01: And that was a huge thing that you know, Cosmos with DBT Core and Airflow, that was like a huge win of something we got out there.
- 16:46SPEAKER_01: Those are like some of the specific use that I'm happy to dive into anything more specific, but like those are like some of the things that come to mind for me.
- 16:53SPEAKER_02: I think that's actually a really great overview and consistent with what I've heard a lot of teams say about just the benefit of Cosmos, helping to simplify everything.
- 17:03SPEAKER_02: And my team has been doing some work recently on creating some more resources for helping folks implement Cosmos in different ways for Airflow and DBT.
- 17:11SPEAKER_02: You know, it turns out DBT is quite complicated.
- 17:13SPEAKER_02: And of course, Airflow is relatively complicated.
- 17:16SPEAKER_02: So using them together, I think, can be a challenge, just as you've described.
- 17:20SPEAKER_02: And so it's cool to hear about where Cosmos is kind of helping bring that all together in a way that just makes the lift a lot lower for teams implementing it.
- 17:28SPEAKER_01: And then maybe something I can touch on a little bit there is another thing that was incredibly helpful was now.
- 17:36SPEAKER_01: So you you have like this thing with Cosmos where you can specify what kind of environment you what environment you want to run.
- 17:44SPEAKER_01: So if you have like dev stage in broad, you can like by basically allowing you to like change whatever, maybe the data warehouse parameters or whatever, changing the configurations, you can run your models across different environments.
- 17:57SPEAKER_01: Now, this allows you to integrate that seamlessly with like your airflow experience, where like if you wanted to run Airflow with DBT in dev, you could easily do that.
- 18:07SPEAKER_01: And then in stage or in prod, it just simplifies the developer experience as well.
- 18:12SPEAKER_01: Because now, like, you can think about how do I want to build in confidence into like my DBT models and make sure that I test everything either in stage or either in dev or just to make sure that everything is working fine before I then go to prod.
- 18:26SPEAKER_01: So that's also like one huge benefit there as well.
- 18:29SPEAKER_02: Absolutely.
- 18:30SPEAKER_02: That's hugely helpful.
- 18:31SPEAKER_02: Well, I think we could spend a long time talking about this, but there was one other thing that I wanted to get to while we have a couple of minutes left.
- 18:39SPEAKER_02: And that is that I understand outside of your role at Style Seat, you also teach Airflow as part of a data engineering course.
- 18:45SPEAKER_02: And I would love to talk about that just a little bit because obviously it's very close to my heart and a big part of what my team does is helping to teach Airflow.
- 18:52SPEAKER_02: So I'm curious, based on your experience there, do you have any high-level guidance for folks who are looking to get more proficient with Airflow?
- 19:00SPEAKER_01: I would say start with the fundamentals, right?
- 19:02SPEAKER_01: So you want to first start by understanding what a DAG is.
- 19:06SPEAKER_01: What's the concept of what a DAG is, what a task is, how does the scheduler work, what are execution dates and task instances.
- 19:13SPEAKER_01: Like those concepts, you want to first start with understanding them.
- 19:17SPEAKER_01: Because I think that one of the things I've seen is that if you do not have like a clear understanding of all those fundamentals, sort of diving deep into it might be a little challenging because you could get to points where like understanding those fundamentals would be critical to like anything you need to do, and then you sort of can't handle that.
- 19:35SPEAKER_01: So you want to start with now.
- 19:36SPEAKER_01: Then the next thing I would say you want to learn like the best practices around like local development, what kind of understanding, for example, the different things, like maybe it's the salary executor or the local, like understanding some of those concepts that allow you to be able to like do things, develop your DAGs locally, and then before you then go to prod, once you understand those basics and you've sort of started getting your hands deep into building out airflow DAGs, then you want to understand things around like common pitfalls, right?
- 20:05SPEAKER_01: So one of the things that I see that's like a common pitfall is that you could have like maybe computations that are a little heavy inside your DAG, right?
- 20:14SPEAKER_01: Because maybe go into why that you shouldn't do that, but like understanding why you shouldn't do that, or understanding like some of those with using Airflow are things that I would say are important.
- 20:24SPEAKER_02: Best practices implementation is so important, especially with something like Airflow that is so flexible.
- 20:30SPEAKER_02: That's really cool that you do that.
- 20:31SPEAKER_02: Can people sign up for your course?
- 20:33SPEAKER_01: It's in partnership with an organization called Old School.
- 20:36SPEAKER_01: So, yeah, you can go check out Old School and check out the data engineering course, and yeah, you would probably be able to find me there.
- 20:44SPEAKER_02: Yeah, that's great.
- 20:45SPEAKER_02: We can throw a link to that in the show notes.
- 20:47SPEAKER_02: Okay, well, while we have uh just a couple of minutes left, I always like to end with a standard final question, and that is what would you most like to see from the Airflow project in the near future?
- 20:56SPEAKER_01: So, well, I'm just gonna maybe talk about something that I'm working on that's really exciting.
- 21:00SPEAKER_01: It probably will be out before this is so I would like to see more native integration with like maybe MCP.
- 21:07SPEAKER_01: I don't know if there's something out there, but like I'm working on an on-call data engineering agent, right?
- 21:12SPEAKER_01: So something that's able to like sort of know when things go wrong and like connect to like each of these different systems and be able to maybe start with generating like direct insight that helps the team know what to do so that the time between finding out what to do and like eventually doing it is shortened and then potentially even help with like making maybe smaller changes or smaller modifications.
- 21:35SPEAKER_01: But like that's something that I'm working on right now.
- 21:38SPEAKER_01: I already have it on GitHub, but like not yet fully functional, but like maybe an initial world before this podcast is out.
- 21:45SPEAKER_01: But like I would love to see more sort of integrations with like things like some of these things within Airflow.
- 21:52SPEAKER_01: And then more importantly, I think something I would like to see is like better visual data lineage visualization, especially like with the DAGs.
- 22:03SPEAKER_01: I guess the goal of something I would like to see is like where I can understand like how all of these DAGs are in Airflow are interacting and be able to easily like find like where something is wrong and then isolate and fix.
- 22:15SPEAKER_01: Like those are some things I would like to see.
- 22:17SPEAKER_02: Those are both great answers, and I love hearing about the contributions that you're making yourself to the community.
- 22:22SPEAKER_02: That's really cool.
- 22:24SPEAKER_02: Well, Pascal, it was really great having you on today.
- 22:26SPEAKER_02: Thanks so much for joining me.
- 22:28SPEAKER_02: What is the best way for folks to get in touch with you?
- 22:30SPEAKER_01: On LinkedIn.
- 22:31SPEAKER_01: You can just search for me, Pascal On Mora, on LinkedIn.
- 22:34SPEAKER_01: I can also send my LinkedIn handle or URL if you need.
- 22:38SPEAKER_02: We'll put it in the show notes for sure.
- 22:41SPEAKER_02: Okay, well, thank you again.
- 22:42SPEAKER_02: It's been really great speaking with you.
- 22:44SPEAKER_01: Yeah, thank you.
- 22:47SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
- 22:54SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
- 22:58SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.