Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Managing a Customer Analytics Platform with Airflow at Skimlinks
Transcript
- 0:00SPEAKER_00: Well what what I'm seeing in the industry is the the rise of data contracts.
- 0:04SPEAKER_00: I think everyone is really talking about it, but not many perhaps are implementing it.
- 0:09SPEAKER_01: You're listening to the Data Flowcast, the podcast about the pet DI flow and the world of data and AI around it.
- 0:17SPEAKER_01: Let's get into it.
- 0:18SPEAKER_02: Okay, hi everyone.
- 0:20SPEAKER_02: Welcome to the Data Flowcast.
- 0:22SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Julian Lerald, who is Director of Data Engineering at Skimlinks.
- 0:29SPEAKER_02: Julian, how are you?
- 0:31SPEAKER_00: Hey Kenton.
- 0:32SPEAKER_00: It's great to meet you.
- 0:33SPEAKER_00: And it's a pleasure to be here in the podcast today.
- 0:35SPEAKER_00: So thanks for the invite.
- 0:37SPEAKER_02: Yeah, of course.
- 0:38SPEAKER_02: It's great to have you on.
- 0:39SPEAKER_02: Well, I'm excited to get into it.
- 0:41SPEAKER_02: For our listeners for today's podcast, we're going to talk about how SkimLinks uses Airflow to power external analytics for their clients.
- 0:48SPEAKER_02: But before we get into all of the details, I want to start with a little bit of background.
- 0:53SPEAKER_02: So first, maybe we'll focus on your company.
- 0:56SPEAKER_02: Can you tell me more about what SkimLinks does?
- 0:59SPEAKER_00: Of course.
- 0:59SPEAKER_00: Yes.
- 1:00SPEAKER_00: Well, Skim Links basically operates in the affiliate marketing industry.
- 1:05SPEAKER_00: What we do is basically we help publishers, websites, uh content creators, anyone who has an audience to monetize their content.
- 1:13SPEAKER_00: They're writing about basic products in the marketplaces.
- 1:18SPEAKER_00: So yeah, you probably this week you might have already clicked on a Skim Link link without even noticing.
- 1:23SPEAKER_00: But yeah, we we help you set up everything with just like two clicks to basically monetize your content.
- 1:31SPEAKER_00: Why do I say that we are a network aggregator?
- 1:34SPEAKER_00: Because publishers, they have their own needs and goals.
- 1:38SPEAKER_00: So there are multiple affiliate networks out there, more than a hundred percent perhaps.
- 1:44SPEAKER_00: But uh with Gimlings, you you take that effort all away, and then with just one contract and then one portal, you're able to go and get the best rates across the entire industry in a very easy way.
- 1:58SPEAKER_02: Okay, yeah, that makes sense.
- 2:00SPEAKER_02: I think there's so much that goes on behind the hood of ads and marketing online that people just don't even realize because it all works pretty seamlessly.
- 2:09SPEAKER_02: But yeah, so that's very cool.
- 2:13SPEAKER_02: And then uh tell me a little bit about your team and what you're responsible for.
- 2:19SPEAKER_00: Mm-hmm.
- 2:20SPEAKER_00: Well, I'm a director of the data engineering here at Skinlinks, and I manage a team of seven data engineers.
- 2:26SPEAKER_00: We basically own what we call the data platform, and what we are responsible is basically we provide with reporting portal to our clients, the external clients, which are our publishers.
- 2:38SPEAKER_00: And this reporting platform basically is like a BI tool that we have developed, right?
- 2:44SPEAKER_00: So we give them metrics around the performance, on the content that they're writing about, but also the content they should be writing about to uplift, increase that revenue.
- 2:54SPEAKER_02: Okay, interesting.
- 2:55SPEAKER_02: So that that sounds like a big part of your business.
- 3:00SPEAKER_02: And I understand that a part of your focus is you know powering the analytics platform that your publisher clients use.
- 3:07SPEAKER_02: Can you tell me a little bit more about how that works?
- 3:10SPEAKER_00: Mm-hmm.
- 3:11SPEAKER_00: Sure.
- 3:11SPEAKER_00: So this platform then that we have built serves around 2,000 weekly publisher users that they access it through our Schimless Hub, what we call the data platform.
- 3:22SPEAKER_00: Perhaps we can split it into like a few different phases.
- 3:25SPEAKER_00: Uh starts with the ingestion of the data.
- 3:27SPEAKER_00: We have real-time sources, you know, the performance marketing metrics such as page impressions, clicks, and in this case, because we operate in the affiliate marketing, it will be commissions.
- 3:37SPEAKER_00: So every time someone's clicks on a link and ends up buying that product, we pass that commission to the publisher, after all.
- 3:46SPEAKER_00: So those are basically the metrics that we evaluate, and then we made certain calculations.
- 3:51SPEAKER_00: But all that data we have it in transactional database and then come from different applications, and we sync everything to the query.
- 3:59SPEAKER_00: Once the data is in BigQuery, then everything is orchestrated by Airflow, and this is where it comes.
- 4:07SPEAKER_00: Airflow is the our main single tool that we use to orchestrate all our data processes, all our data needs.
- 4:15SPEAKER_00: And well, within BigQuery, we have like the classic few layers, uh raw, silver, and gold.
- 4:22SPEAKER_00: And we generate these reports, we have different data marts, and then we end up exposing it into a separate database called Apache Truage.
- 4:30SPEAKER_00: This database is the one that the publishers would eventually in the need query in the reporting hub.
- 4:36SPEAKER_00: And it gives us the sub-second latency that they are after when it comes to a BI tool up to like today's standards.
- 4:46SPEAKER_00: With that, because we have so much for our external customers, the reports are mostly the same.
- 4:52SPEAKER_00: So we reutilize most of it to create internal analytics as well.
- 4:56SPEAKER_00: So we count with a hundred internal customers from the departments of marketing, finance, operations, managers, anyone from our internal business that wants to dig in into the data.
- 5:09SPEAKER_00: So yeah, we we we provide them with BQRY access, or or we have Lucre as our internal BI tool as well.
- 5:15SPEAKER_02: Okay.
- 5:16SPEAKER_02: Yeah, very interesting.
- 5:17SPEAKER_02: So it sounds like Airflow is very central to both the product that your company provides and of course for internal analytics as well.
- 5:24SPEAKER_02: I think a hundred internal customers sounds very like a lot to me.
- 5:29SPEAKER_02: Well, I would love to dig in a little bit deeper into kind of your use of Airflow and how it's helping you power your analytics platform.
- 5:40SPEAKER_02: I imagine that there's a big component of that that's you know, sort of on demand or needs to be updated based on when particular customer data is being is coming in.
- 5:52SPEAKER_02: Tell me about how your scheduling DAGs are you using sensors or yeah, something like that.
- 6:00SPEAKER_00: As I mentioned, Airflow is the core of our platform.
- 6:03SPEAKER_00: Everything that we orchestrate goes through Airflow.
- 6:06SPEAKER_00: And we as data platform provide perhaps like the best practices on how to use it, we utilize it and we make sure that it's it's up and running and tasks don't get delayed, they don't get scheduled, or they don't get uh outside the defined SLAs, let's say.
- 6:24SPEAKER_00: But yeah, in a nutshell, we have a few different products, but we basically use Airflow as a as an orchestrator, but as I mentioned, all our data sits in BigQuery, and therefore all our business logic sits in SQL, right?
- 6:39SPEAKER_00: So with Airflow, we what we mostly use are like a BigQuery operators to kind of like define that lineage from one task after the other.
- 6:48SPEAKER_00: We we do some of the complex stuff, like for example, there is uh a service that we provide to certain publishers, the largest ones actually, to that we provide them with CSV, Avro, Parkhead, just raw file exports with the raw data in case they need to ingest it to the warehouse.
- 7:05SPEAKER_00: And and Airflow plays a critical part in that because that that is quite a complex orchestration that we need to do across a few dozens of time.
- 7:16SPEAKER_02: Okay, yeah, interesting.
- 7:18SPEAKER_02: And then so I understand that with that you've also shifted from the sensor-based approach to using Airflow data sets and which are now Airflow assets and Airflow 3.
- 7:30SPEAKER_02: What what does that look like?
- 7:32SPEAKER_00: Yes.
- 7:33SPEAKER_00: So we've been running Airflow for many, many years, since Airflow 1, right?
- 7:38SPEAKER_00: So I'm always like quite excited when a new version comes out and and there's that need for doing the uh the migration and and tackle deprecations, but these new features like dataset or or assets, now they're called, has been like an integral part uh over the last few years.
- 7:57SPEAKER_00: We've been uh historically using sensors as a way of like orchestrating dependencies from different DAGs, particularly task sensors.
- 8:05SPEAKER_00: So making sure that only a particular DAG, a particular DAG airflow process would only run when the data is available, right?
- 8:14SPEAKER_00: So uh external task sensor would normally be marked as finished, and then only then the second DAG will kick in as long as the schedule uh uh has met already.
- 8:27SPEAKER_00: But we've seen across the industry that within data things were moving from a cron-based, from a schedule to more like an event-driven approach, right?
- 8:37SPEAKER_00: And we knew that we wanted to be part of that because nobody wants to have like um a dead time in between tasks, or you don't want to like constantly be waiting, poking for uh for data to see has it arrived yet, has it arrived yet until the final death.
- 8:50SPEAKER_00: Like all that created like a worker overload, right?
- 8:55SPEAKER_00: From a cost perspective, you'll be like utilizing workers for a certain amount of time that were not actually doing much labor.
- 9:02SPEAKER_00: But then from another perspective is the amount of uh tasks that the Airflow scheduler will have to organize quite often.
- 9:10SPEAKER_00: Like you know, we have the different types of poke and reschedule to release the worker slot usage, but but still, it still needs to go to the scheduler.
- 9:20SPEAKER_00: So that migration to the dataset event-driven approach was was critical, you know, to to help like streamline actually what is like a smooth transition from one process to the other as soon as this the data becomes available.
- 9:34SPEAKER_00: But also, I think the the added bonus was this lineage that Airflow provides in in the UI.
- 9:41SPEAKER_00: And we were able after that migration to be to see within like a single panel how everything connects together, right?
- 9:50SPEAKER_02: Yeah, absolutely.
- 9:51SPEAKER_02: That's such a huge benefit of the data sets or assets feature is being able to get that visibility into yeah, into all of the connections between your DAGs.
- 10:01SPEAKER_02: So yeah, that's really cool.
- 10:04SPEAKER_01: This epicot is brought to you by Epcot.com.io and focused on I guess looking ahead, you have mentioned, you know, on a different topic, plans to kind of open up airflow usage to other engineers across the company so that they can, you know, contribute in some way.
- 10:42SPEAKER_02: What is your vision there and kind of how how are you going about implementing that?
- 10:48SPEAKER_00: That that's a project that I'm quite excited about.
- 10:50SPEAKER_00: Yeah, my vision, yeah, and it I think it's starting to become perhaps uh a bit of a standard across the industry when you have like a central hub of data platform uh providing services to help other engineers achieve their tasks.
- 11:02SPEAKER_00: The idea is to break down the different silos of other teams across the world, running either their own Airflow instances or or run the jobs or cron jobs.
- 11:12SPEAKER_00: You don't want to dig in much, but yeah, the running locally in the laptop.
- 11:17SPEAKER_00: So effectively we want to become like the center of excellence, and when it comes to orchestrating something that is reliable, it can run on time, and you will have visibility on like alerting notification when when something is not quite right.
- 11:31SPEAKER_00: By us like being the experts in airflow and providing the best practice in a safe environment, because security is quite important nowadays with with the with the rise of uh AI agents and so on.
- 11:44SPEAKER_00: Yeah, I'm hoping to be there within this year and provide all my fellow engineers the best experience they can possibly get.
- 11:53SPEAKER_02: Okay.
- 11:54SPEAKER_02: Yeah, very cool.
- 11:55SPEAKER_02: This is definitely something that we hear about a lot.
- 11:57SPEAKER_02: And Astronomer has kind of our own take within our platform on how we can open up, you know, DAG or pipeline creation to folks who might not know Airflow very well or might not know Python.
- 12:09SPEAKER_02: How are you handling that aspect of it?
- 12:12SPEAKER_02: So you want more people to be able to contribute, but as you mentioned, of course, you have to provide some guardrails uh so that best practices are followed.
- 12:19SPEAKER_02: What does that look like in practice for the implementation?
- 12:23SPEAKER_00: Exactly.
- 12:23SPEAKER_00: Well, Astronomer Airflow is very tempting, you know, because Airflow has been uh everyone has been asking for multi-tenancy for years, and astronomer came one day and said, look, guys, we did it.
- 12:33SPEAKER_00: Here it is, start using it.
- 12:34SPEAKER_00: So it's it's really tempting, and and it's something that you don't have to reinvent as well, right?
- 12:40SPEAKER_00: It's it's off the shelf available, and of course, providing these security guard delays.
- 12:45SPEAKER_00: In terms of what does it look like?
- 12:48SPEAKER_00: I guess it's what what I was mentioning first, a common standard uh across like dark testing, providing SLA or alerts when those are missed, monitoring to make sure that things are running properly.
- 13:00SPEAKER_00: Airflow gives you the possibility to create your own plugins, um provide like a little dashboard if you want it that is custom to your business, making sure that each team has a green light when it comes to checking that the data has run, but not only has run, but it has actually the data that they're expecting.
- 13:19SPEAKER_00: Providing like baseline for monitoring in terms of standard deviations when it comes to broad counts or sums over the days or like um a particular data assets.
- 13:34SPEAKER_00: I think all of that contributes to them being able to focus on what it matters, right?
- 13:40SPEAKER_00: On delivering uh business value for the projects without worrying that much about the data infrastructure or the cumbersome uh that that Airflow might entail.
- 13:49SPEAKER_02: Are there any main challenges that you're anticipating there, other than we've kind of talked about the best practices aspect already?
- 13:56SPEAKER_02: But as you look forward to implementing this.
- 14:00SPEAKER_00: Yeah, um I think one of the challenges that I've been having over the years is trying to convince the business that we need to upgrade Airflow.
- 14:08SPEAKER_00: It's it's it's always such a not a tedious task, but it requires effort.
- 14:13SPEAKER_00: Uh and the business might not always uh see the benefits of doing so.
- 14:17SPEAKER_00: Of course, like us engineers, we always get excited about new features, making things in a more elegant way.
- 14:23SPEAKER_00: Uh, but for the business, it might be a bit hard to say, yes, let's prioritize doing this and yeah, yeah, and help you achieve what you're after.
- 14:32SPEAKER_02: Yeah, sure.
- 14:32SPEAKER_02: Totally.
- 14:33SPEAKER_02: That's always a challenge for data teams.
- 14:36SPEAKER_02: Well, I guess then on a slightly different topic.
- 14:38SPEAKER_02: So you mentioned the DAG factory.
- 14:40SPEAKER_02: I had you know incorrectly assumed that was part of this platform you were talking about.
- 14:43SPEAKER_02: But is that a different project that you're working on that you could share some details about?
- 14:48SPEAKER_00: Yeah, yeah, of course.
- 14:49SPEAKER_00: Yeah, quite quite recently, actually, over the last six months, uh, we were able to migrate a quite a long-standing project of us.
- 14:56SPEAKER_00: So we have a data product called the Tata Pipe, uh, which effectively we export raw data to our largest publishers to ingest the data into the warehouse if they need to.
- 15:06SPEAKER_00: Uh, rather than giving them the insights on a reporting layer, we give them the both data so then they can analyze it any way they they they want.
- 15:14SPEAKER_00: Now, this project coming from Airflow One has been longstanding and it serves that particular purpose, right, many, many years ago.
- 15:22SPEAKER_00: But over time, business is uh started to rise and requirements changed, and we moreover, we we started to utilize this way of exporting data to share data within our business as well.
- 15:35SPEAKER_00: So not only it kind of like out it has outgrown itself to like different needs.
- 15:40SPEAKER_00: So that led us to perhaps not developing it in the most elegant way because it was always in a hurry and we didn't need it by by yesterday.
- 15:50SPEAKER_00: And over the years we just like kept adding layers that weren't ideal and it wasn't like a robust of a system as um we would have liked in terms of what we did.
- 16:01SPEAKER_00: So yeah, but basically it was uh a single class, Python class, doing absolutely everything, uh, generating the DAG.
- 16:08SPEAKER_00: But if we were to split it by by phases, what we did was break it down, break that class down into what we call the single responsibility.
- 16:17SPEAKER_00: Uh so we create one class uh for each responsibility, doing first uh uh the configuration of it, who is this intending to be sent to, what file format they want.
- 16:28SPEAKER_00: The second one would be actually building the query that they wanted, the query in our BigQuery database.
- 16:34SPEAKER_00: The third stage would be actually extracting that data out to the file format that they were after, and finally would be sending that data to the cloud provider that they are compatible, they're working with.
- 16:47SPEAKER_00: So we broke that down into multiple classes that helped us make it more reliable in terms of um testing, but also making sure that every section was uh was kind of independent and building more of a robot system.
- 17:01SPEAKER_00: On top of this, because we were used to changing Python files for configuration every time we had to onboard a new client, it was quite risky to do so.
- 17:10SPEAKER_00: So we decided to create our own DAC factory and provide it with a YAML format for the configuration.
- 17:17SPEAKER_00: So now when we need to onboard a new publisher, we basically go to the YAML and we just add the requirements that we have there, then the name, then the export format, the and the particular query format that we want.
- 17:31SPEAKER_00: Yeah, it's super easy to integrate.
- 17:33SPEAKER_00: Not only do we have the YAML, but on top of that we added a Pydentic layer.
- 17:37SPEAKER_00: So that helps us create something way more stable.
- 17:41SPEAKER_00: Because the YAML might be valid, for example, but the key might be mistaken and or or the value might be incorrect.
- 17:48SPEAKER_00: And you would only notice that on execution time.
- 17:51SPEAKER_00: So one of the main benefits that that we see with with Pydantic is that the DAG would not be able to render otherwise.
- 17:58SPEAKER_00: It it would fail.
- 18:00SPEAKER_00: So um that I think that that that was a critical uh perhaps improvement that was done for for adding like robust mass.
- 18:07SPEAKER_02: Interesting.
- 18:08SPEAKER_02: And so so the Pydenc layer is kind of like a test that the DAG is going through before you deploy it so that you're catching any failures before it actually runs.
- 18:18SPEAKER_02: Exactly.
- 18:18SPEAKER_00: Yes.
- 18:19SPEAKER_02: Interesting.
- 18:19SPEAKER_02: Yeah, that's very cool.
- 18:20SPEAKER_02: I haven't heard of PyDantic being used in that way before.
- 18:24SPEAKER_02: Yeah, that's great.
- 18:25SPEAKER_02: And so, and overall, this project, it sounds like was meant to kind of help standardize a lot of this repeatable work as you're like onboarding clients into the platform.
- 18:36SPEAKER_02: So I imagine it allows you to move much faster.
- 18:40SPEAKER_00: Exactly, yes.
- 18:41SPEAKER_00: And it's something that we've been trying to do for years.
- 18:44SPEAKER_00: You know, we we had the design in our mind, but we were never able to prioritize it from a business need.
- 18:51SPEAKER_00: But yeah, over the last six months, you know, the world has changed recently with the uh AI adoption.
- 18:56SPEAKER_00: So I'm not saying that the AI did it, but it really helped us uh guide us through making sure that the decisions that we were taking from a design perspective were sound, uh, but also helping us like writing tests, comprehensive tests across each class, uh making sure that the decisions we were taking along the development were correct.
- 19:15SPEAKER_00: Um is a great help to brainstorm like different ways of doing things that we you were not thinking before, like.
- 19:23SPEAKER_02: Yeah, absolutely.
- 19:24SPEAKER_02: That sounds like a great AI success story.
- 19:27SPEAKER_02: And I'm I'm curious, did that work pretty well out of the box for you for like an Airflow-specific context?
- 19:34SPEAKER_00: I guess with AI, we are all reinventing ourselves, you know, in our roles.
- 19:39SPEAKER_00: Then there are multiple ways, multiple tools, and multiple ways of doing it.
- 19:42SPEAKER_00: You have like uh then the UI, you have the client, you have uh running in your in your own IDE.
- 19:49SPEAKER_00: So it was a project that was mostly like a self-discovery project as well, to to learn the ways of AI, evaluating different models, uh, starting with uh definitions on the on the project rules, adding skills or certain elements.
- 20:05SPEAKER_00: So yeah, I I guess it it worked well both ways.
- 20:09SPEAKER_02: Okay.
- 20:09SPEAKER_02: Yeah.
- 20:10SPEAKER_02: Awesome.
- 20:10SPEAKER_02: Well, that's uh really cool to hear about.
- 20:12SPEAKER_02: And thank you for going through those couple of different use cases.
- 20:16SPEAKER_02: I always love talking to folks where Airflow is really powering a huge part of their product.
- 20:20SPEAKER_02: It's so cool to see Airflow being used in that way.
- 20:23SPEAKER_02: And yeah, you've got some really creative and interesting projects going on with that.
- 20:28SPEAKER_02: I guess just to close us out, I always like to end with a standard final question for our guests, and that's what would you most like to see from the Airflow project in the near future?
- 20:38SPEAKER_00: Well, I guess we we talk about data sets, right?
- 20:40SPEAKER_00: The sand source data sets, uh, asset watchers.
- 20:44SPEAKER_00: I I think, well, asset watchers is basically the devolution of data sets, right?
- 20:48SPEAKER_00: But for external data sources, I think what what I'm seeing in the industry is the rise of data contracts.
- 20:56SPEAKER_00: I think everyone is really talking about it, but uh not many perhaps are implementing it.
- 21:01SPEAKER_00: So perhaps helping validate that asset watcher assigning it with a data contract, making sure that the schema uh fits what we were expecting.
- 21:13SPEAKER_00: That kind of thing, if we were able to validate at the time the asset watcher is triggered before we start processing the data, that would be amazing.
- 21:22SPEAKER_02: Yeah, yeah, definitely.
- 21:24SPEAKER_02: I can see where that would be super helpful.
- 21:26SPEAKER_02: And I I think if I remember correctly, the data contracts was included in the previous Airflow survey as one of the features of the folks were requesting.
- 21:36SPEAKER_02: I think it was pretty high up there.
- 21:37SPEAKER_02: So I think I think you're definitely not alone, as you said.
- 21:40SPEAKER_02: It's definitely a big focus for the industry right now.
- 21:43SPEAKER_02: So absolutely.
- 21:45SPEAKER_00: I mean, the software engineers uh have it, you know, when when when they do uh API development, we engineers as well, right?
- 21:52SPEAKER_00: In data.
- 21:52SPEAKER_00: So why why why shouldn't we be investing in uh data contracts?
- 21:56SPEAKER_02: Yeah, absolutely, totally agree with you there.
- 21:59SPEAKER_02: Well, Julian, thank you so much again for coming on.
- 22:01SPEAKER_02: It's been really great speaking with you.
- 22:03SPEAKER_02: Just to close us out, what is the best way for folks to get in touch with you?
- 22:07SPEAKER_00: LinkedIn is the best way, Julian Rald.
- 22:09SPEAKER_00: I'm I'm responsive there.
- 22:11SPEAKER_00: So yeah, reach out if you just want to talk about Airflow.
- 22:14SPEAKER_00: I'd be glad to.
- 22:15SPEAKER_02: Great.
- 22:15SPEAKER_02: Well, we will put a link to that in the show notes.
- 22:18SPEAKER_02: Thank you again.
- 22:18SPEAKER_02: As I said, this has been a really great conversation.
- 22:21SPEAKER_02: I have enjoyed speaking with you.
- 22:23SPEAKER_00: It was my pleasure.
- 22:24SPEAKER_00: Thanks, Content.
- 22:25SPEAKER_00: See you soon.
- 22:26SPEAKER_01: Thank you for listening to the Data Flowcast.
- 22:29SPEAKER_01: Check the donuts for links to everything mentioned today.
- 22:33SPEAKER_01: If you're enjoying the podcast, subscribe to Never Miss Episode and Five Star Review.