Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov

Transcript

Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: We use Airflow to ingest this data and put it on a snowflake, it's our database.
• 0:05
  SPEAKER_03: And this made by our team all this ingestion work because it's very unique work for each source.
• 0:12
  SPEAKER_03: We need to build some extraction stuff.
• 0:15
  SPEAKER_03: And second scenario is data orchestration.
• 0:18
  SPEAKER_03: When we have a lot of projects, data products, and we need to trigger one product based on another product.
• 0:26
  SPEAKER_03: And this is done by business teams, by data teams.
• 0:45
  SPEAKER_01: Hey Airflow community, exciting news.
• 0:48
  SPEAKER_01: This fall we're launching Beyond Analytics, a virtual data ops conference happening September 16th from 12 to 3 p.m.
• 0:55
  SPEAKER_01: Eastern.
• 0:55
  SPEAKER_01: You'll hear from data leaders on how modern data ops and orchestration are powering AI, machine learning, and production grade data products.
• 1:02
  SPEAKER_01: And here's the best part.
• 1:04
  SPEAKER_01: You can join Mark Lumberti for a fast track Air Force 3 certification prep session and get a discount code for a free certification, a$150 value.
• 1:13
  SPEAKER_01: The conference is virtual, it's free, and it's made just for you.
• 1:17
  SPEAKER_01: You can register now at astronomer.io slash beyond slash data flowcast.
• 1:29
  SPEAKER_01: Hi everyone, welcome to the Data Flowcast.
• 1:32
  SPEAKER_01: I'm your host, Kendon Danis, and today I'm joined by Evgeny Prusov, who is a senior data platform engineer at Daishi Sanko Europe.
• 1:41
  SPEAKER_01: Evgeny, how are you?
• 1:43
  SPEAKER_03: Thank you.
• 1:43
  SPEAKER_03: Great.
• 1:44
  SPEAKER_03: Thank you for having me there.
• 1:46
  SPEAKER_01: Yeah, it's great to have you on.
• 1:48
  SPEAKER_01: Thanks for joining in your evening as we're recording this.
• 1:52
  SPEAKER_01: All right, well, let's go ahead and dive in.
• 1:54
  SPEAKER_01: So for our listeners for today's podcast, we're going to discuss why Daiichisanko implemented Airflow as an orchestrator.
• 2:02
  SPEAKER_01: This is a great story, but let's start at the beginning.
• 2:05
  SPEAKER_01: And can you tell everybody a little bit about what Daishi Sankyo does?
• 2:09
  SPEAKER_03: So Daichis Sankyo is basically a Japanese global pharmaceutical company that develops medicines.
• 2:18
  SPEAKER_03: And in Europe, we have in Munich our office.
• 2:21
  SPEAKER_03: We covered 15 countries in Europe and we focus on two areas.
• 2:26
  SPEAKER_03: It's cardiovascular and it's oncology.
• 2:29
  SPEAKER_03: So our team serves this all countries in Europe.
• 2:33
  SPEAKER_03: We are building data platforms for them.
• 2:35
  SPEAKER_01: Okay, that's great.
• 2:36
  SPEAKER_01: So those are two very important categories of drugs.
• 2:40
  SPEAKER_01: And so you mentioned so your team manages the data platform for all of that.
• 2:45
  SPEAKER_01: Within that, can you tell me a little bit more about what your team does?
• 2:49
  SPEAKER_03: So basically, my team was established two and a half years ago.
• 2:53
  SPEAKER_03: And before that, in Europe, we didn't have any centralized platform.
• 2:58
  SPEAKER_03: Had some tools which business used for some analytics, but it was different tools.
• 3:03
  SPEAKER_03: There were no standards, there were no processes and automation.
• 3:08
  SPEAKER_03: And what we did is we chose some data modern stack and started building the centralized platform.
• 3:15
  SPEAKER_03: And the operational model here is decentralized.
• 3:19
  SPEAKER_03: So basically we build a platform, but the analytics built by regions, by countries, by people who understand the data and the demand.
• 3:27
  SPEAKER_03: And we just provide the capabilities, standards, and all the automation.
• 3:32
  SPEAKER_01: That makes sense.
• 3:33
  SPEAKER_01: I always think that this sort of trajectory is an interesting process.
• 3:38
  SPEAKER_01: I've been in this position myself to build a platform like that from scratch.
• 3:42
  SPEAKER_01: It's both fun in a way because you get to choose everything where there wasn't a platform before.
• 3:48
  SPEAKER_01: It's also very challenging to implement a platform where you didn't have one.
• 3:53
  SPEAKER_01: And so tell me, you said you went through this process about two and a half years ago of building out the data platform.
• 4:00
  SPEAKER_01: Was Airflow the initial orchestrator that you chose at that time?
• 4:05
  SPEAKER_03: No, we started using the Azure Data Factory as a tool to create pipelines.
• 4:11
  SPEAKER_03: But very soon we understood that it's very hard to scale with that tool because it's not GitOps-based, so you can't read the code.
• 4:20
  SPEAKER_03: There is no code, you can't review the code, so it's really hard to create a modern process to deploy this stuff and to control the quality.
• 4:29
  SPEAKER_03: And then we start to think about what tool we need, what tool to choose.
• 4:33
  SPEAKER_03: And we look for some different orchestrators, and astronomer was one of them.
• 4:39
  SPEAKER_03: And we decided that Airflow and Astronomer is the way to go because it's an amazing community around Airflow, and it's very easy to find some solutions and best practices.
• 4:50
  SPEAKER_03: And yeah, that's why we choose Airflow basically.
• 4:53
  SPEAKER_01: It makes sense.
• 5:05
  SPEAKER_01: That's great.
• 5:06
  SPEAKER_01: And so you mentioned you're running on Astro, so you're using Astronomer.
• 5:11
  SPEAKER_01: Our product is called Astro.
• 5:12
  SPEAKER_01: Maybe now that you're a couple of years into it, tell me a little bit about just what airflow use looks like for your team now.
• 5:19
  SPEAKER_03: Our team, it's very slim.
• 5:21
  SPEAKER_03: So we have only a few platform engineers.
• 5:24
  SPEAKER_03: So we decided from day one that we will not manage airflow by ourselves.
• 5:29
  SPEAKER_03: So we need some SARS solution.
• 5:32
  SPEAKER_03: And that's why we choose SMMR.
• 5:35
  SPEAKER_03: And the first problem was that nobody in our team didn't understand the Airflow, didn't have experience with Airflow.
• 5:43
  SPEAKER_03: So we started to learn it two and a half years ago and understand how to set up the development environment, how to set up the whole continuous integration pipelines, and how to scale it through the whole regions, develop standards and so on.
• 5:59
  SPEAKER_03: So it was a long journey, but it was really fun and very interesting for me personally.
• 6:04
  SPEAKER_03: Yeah, to go very deep into this topic and to Airflow.
• 6:09
  SPEAKER_01: That's great.
• 6:09
  SPEAKER_01: That sounds like a very interesting journey.
• 6:13
  SPEAKER_01: And so now, if I understand right, so you manage this Airflow platform for multiple teams that are working with their own data, the analysts, is that correct?
• 6:26
  SPEAKER_03: Currently we have different deployment environments in SNOMR.
• 6:30
  SPEAKER_03: First, we have different scenarios how we use it.
• 6:33
  SPEAKER_03: So first is we use it for ingestion.
• 6:36
  SPEAKER_03: So we have a lot of sources, our internal sources in the HSU, external data providers, some files, APIs, and we use Airflow to ingest this data and put it on a Snowflake.
• 6:50
  SPEAKER_03: It's our database.
• 6:51
  SPEAKER_03: And this made by our team, all this ingestion work because it's very unique work for each source.
• 6:58
  SPEAKER_03: We need to build some extraction stuff.
• 7:01
  SPEAKER_03: And second scenario is data orchestration when we have a lot of projects, data products, and we need to trigger one product based on another product.
• 7:10
  SPEAKER_03: And this is done by business teams, by data teams.
• 7:14
  SPEAKER_03: This is the second scenario, and that's distributed across all regions.
• 7:18
  SPEAKER_03: We just make some standards.
• 7:20
  SPEAKER_03: And third scenario is to automate some platform stuff like deleting some old objects or build some reports on usage, and we use it internally also for orchestration our own stuff, engineering stuff.
• 7:34
  SPEAKER_03: And we build CI pipeline.
• 7:36
  SPEAKER_03: We're not only deploying Airflow decks, we also check the quality of the code by linting the code, by running some PyTests, other checks.
• 7:46
  SPEAKER_03: So we have a very strict pipeline, and also we review in all the PRs manually and with these tools.
• 7:54
  SPEAKER_01: Got it.
• 7:54
  SPEAKER_01: So you're instituting the best practices and making sure that everybody follows them.
• 8:00
  SPEAKER_01: Yeah, that's great.
• 8:01
  SPEAKER_01: So through this experience over the last couple of years and managing this platform for a couple teams and using a managed service like Astro, what have been the biggest challenges that you've faced?
• 8:12
  SPEAKER_01: Anything in particular that you've had to tackle?
• 8:15
  SPEAKER_03: I think the biggest challenge is that people who are common platform users, they are not Python developers and they are not aware of this Python ecosystem of good quality tools, how to debug stuff in Python, how to run how to read, write tests and run them.
• 8:36
  SPEAKER_03: So the challenge is to teach them and to build guidelines and some good examples and spread this knowledge across all the data teams.
• 8:48
  SPEAKER_03: Because we don't have so much experience in airflow, and all the time that something happens, we need to go deep, investigate, then create some standard how to tackle this problem and share this knowledge to everybody.
• 9:16
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 9:28
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 9:36
  SPEAKER_01: One of the most powerful things about Airflow, and also one of its biggest drawbacks, I think, is that it's just very open-ended.
• 9:44
  SPEAKER_01: You can do most anything, but it does require, I think, pretty good Python knowledge to direct you and keep you within the walls of those best practices.
• 9:54
  SPEAKER_03: And also when you're using this Python ecosystem tools, for example, for check the quality of the code, they are not aware of the airflow specifics.
• 10:05
  SPEAKER_03: So you have to disable some rules, some commands, some stuff.
• 10:10
  SPEAKER_03: And if you are not an expert in that, it's really hard to understand what is the right way.
• 10:16
  SPEAKER_03: So that's where we need this expertise centralized, and then we can share with all the teams how to better use it.
• 10:25
  SPEAKER_01: Definitely.
• 10:25
  SPEAKER_01: I think that challenge, my mind, has existed for the lifetime of Airflow, of both in the as you're describing, with the debugging and testing aspect of it, just this idea that you have to both develop like the logic for your pipelines as well as develop the logic for airflow, both of those together in that context.
• 10:47
  SPEAKER_01: I think it's even just the local development experience is really difficult for a lot of people who aren't as familiar with Airflow, trying to write again, just the logic for managing your data, but within the context of Airflow and like running a DAG and seeing it fail, that sort of debugging process can be quite challenging.
• 11:05
  SPEAKER_03: Yes.
• 11:06
  SPEAKER_03: Especially when you have some limitations.
• 11:08
  SPEAKER_03: For example, we have limitations on our uh laptops, corporate laptops.
• 11:13
  SPEAKER_03: We're not allowed to use Docker, and we can't use Astro CLI directly from Windows.
• 11:18
  SPEAKER_03: We need some pre-work, let's say, uh so we need to install VSL, then set up the Ubuntu inside Windows, and there we can use some Astro CLI automation stuff.
• 11:31
  SPEAKER_03: Yeah.
• 11:31
  SPEAKER_03: So it's another layer of complexity for people who are not aware of how to work in the terminal, how to set up all this stuff.
• 11:39
  SPEAKER_01: That definitely makes it a lot harder, especially I think the Astro CLI is so great for getting started with Airflow.
• 11:45
  SPEAKER_01: But if it's hard to install it, it can be an extra set of hurdles that you have to go through that can make things pretty challenging.
• 11:53
  SPEAKER_01: Well, I'm curious to hear a little bit more about, you know, I understand that some of the work that you've done at your company, in addition to providing this data platform to, in my mind helped evangelize Airflow, is share how you are using it and build some community around it.
• 12:11
  SPEAKER_01: I'm curious if we can talk a little bit more about that.
• 12:14
  SPEAKER_01: It's something that my team, the DevRel team at Astronaut, does a lot of work on building community around Airflow.
• 12:20
  SPEAKER_01: So it's a topic of particular interest for me.
• 12:24
  SPEAKER_03: That's a great question.
• 12:25
  SPEAKER_03: Uh actually, this year our company actively started to globalize the IT functions.
• 12:32
  SPEAKER_03: And before that, we have different teams in the US, in Europe, and in Japan who build their own platforms basically on their own tools.
• 12:42
  SPEAKER_03: But currently we are trying to centralize all the stuff and sharing knowledge, and that's led us to the point that we need to understand what other regions do, what tools they are using.
• 12:54
  SPEAKER_03: And we decided to build this global community across all the regions, and we bring all data engineers, analysts, ML engineers into one place, and then at that place we share the knowledge.
• 13:10
  SPEAKER_03: So we explain that what tools we have, how we use these tools, and we already know that in the US they already have astronomy instance, but now we can share how we use it so to bring our expertise to the US.
• 13:26
  SPEAKER_03: Maybe in Japan we will also install some instances of astronomy in the future, I don't know.
• 13:34
  SPEAKER_03: But this community, it's a great way how you can understand what other regions do and synchronize what the best way to do some stuff, especially if we already have the same tools but you know in different teams.
• 13:47
  SPEAKER_01: Yeah, I'd imagine that both helps with just building kind of camaraderie and community internally amongst the teams and with avoiding reinventing the wheel in terms of how people are using Airflow.
• 14:00
  SPEAKER_03: And when we share some experiences after that meeting, so we have monthly calls with the community.
• 14:07
  SPEAKER_03: Some people just write to me, oh, and we also want to use this technology.
• 14:11
  SPEAKER_03: Can you share more details?
• 14:13
  SPEAKER_03: So it's really useful.
• 14:15
  SPEAKER_01: I love that.
• 14:15
  SPEAKER_01: Have there been any surprising, maybe or interesting use cases that you've heard of other people using Airflow for kind of from these meetings?
• 14:24
  SPEAKER_01: Anything that surprised you?
• 14:26
  SPEAKER_03: Not how they use it because they can't use it.
• 14:29
  SPEAKER_03: We have centralized infrastructure, but they share how they want to use it because they already have some other tools that they uh don't want to use or that it's hard to use.
• 14:40
  SPEAKER_03: For example, this ADFs or uh something in other ecosystems, and they trying to understand and ask us how we can leverage Astronomer because it looks very good and scalable.
• 14:54
  SPEAKER_03: So explain please how we can get that tools.
• 14:58
  SPEAKER_01: Okay, got it.
• 14:59
  SPEAKER_01: So people are just excited in general about Airflow and Astro.
• 15:03
  SPEAKER_01: That's cool.
• 15:03
  SPEAKER_01: And they're mostly, you said these are often people who are using Azure Data Factory like as a default and potentially moving over to Airflow.
• 15:12
  SPEAKER_03: In Europe, we have a lot of decks and some in US, but for example, in US they also have uh AWS and some other stuff, and in Japan they have Google Cloud and completely different tools.
• 15:28
  SPEAKER_03: So there is really a zoo, and we're currently trying to simplify this ecosystem, choose some tools that we can use in every region, decrease the complexity of the infrastructure.
• 15:41
  SPEAKER_01: That makes a ton of sense from an efficiency standpoint.
• 15:44
  SPEAKER_01: Well, that's really cool that you're doing that.
• 15:46
  SPEAKER_01: We do a lot of work with the global airflow community to try and do similar things, just provide inspiration on how you can use Airflow.
• 15:53
  SPEAKER_01: So I love hearing about just within a company that effort to try and do the same thing.
• 15:58
  SPEAKER_03: Recently I started to contribute to the airflow, and I want to somehow engage our community.
• 16:05
  SPEAKER_03: Maybe someone else will also do that because it's really a very good way how you can go deeper and understand airflow better.
• 16:14
  SPEAKER_03: So how to better use it, how to better customize it, how to debug it.
• 16:17
  SPEAKER_03: Because there's a lot of information that's in the community, but when you're just using the airflow, it's not visible.
• 16:26
  SPEAKER_03: When you go deeper, you can find a lot of very interesting information.
• 16:30
  SPEAKER_01: Always love a good plug for more contributors to the project.
• 16:34
  SPEAKER_01: And I think that's actually a good tie-in.
• 16:37
  SPEAKER_01: Speaking of contributing to open source airflow, a good lead-in to my standard final question, which is what would you most like to see from the Airflow project in the near future?
• 16:47
  SPEAKER_03: I think one thing is to see some more guidelines and maybe tools how to control the quality of code because or maybe customizations of Python ecosystem tools for airflow.
• 17:02
  SPEAKER_03: The second thing is more tutorials how to debug and how to write high-quality tests for airflow.
• 17:12
  SPEAKER_03: And the last one is that we want to see is some AI coding assistant for airflow.
• 17:18
  SPEAKER_03: Maybe adjusted for airflow by astronomer or by some community experts.
• 17:24
  SPEAKER_03: Because when you're trying to use the common assistant, they are not very good understand how airflow works.
• 17:32
  SPEAKER_03: They generate Python code, but it's unsecure, it's not scalable, it's not good.
• 17:39
  SPEAKER_01: I'm very familiar with that challenge.
• 17:41
  SPEAKER_01: Definitely for the AI assistant.
• 17:43
  SPEAKER_01: I don't think that's planned for open source airflow, but it's certainly something that Astronomer is working on that will be available to you.
• 17:51
  SPEAKER_01: The debugging and testing one is a good thing as well.
• 17:54
  SPEAKER_01: Actually, today, at least as of the day that we're recording this, probably not the day it was released.
• 17:59
  SPEAKER_01: We just did a webinar this morning on debugging and testing in Airflow.
• 18:03
  SPEAKER_01: But yeah, that's something I always love to hear feedback on how we can help folks go deeper.
• 18:08
  SPEAKER_01: It's such a big topic.
• 18:09
  SPEAKER_01: And it's hard for us to address it in a deep level because everybody's experience varies.
• 18:15
  SPEAKER_01: But there's certainly a lot of interest and need there as it's I made a comment this morning that I think debugging is something every Airflow developer deals with.
• 18:24
  SPEAKER_01: I can't imagine writing a DAG and not having to debug it.
• 18:27
  SPEAKER_01: That's probably impossible.
• 18:29
  SPEAKER_03: Different people uh understand debugging differently.
• 18:32
  SPEAKER_03: Yeah, some people printing something, it's also debugging.
• 18:36
  SPEAKER_01: Right.
• 18:37
  SPEAKER_01: Yeah, exactly.
• 18:38
  SPEAKER_01: Exactly.
• 18:39
  SPEAKER_01: But yeah, those are both good community requests, I think.
• 18:42
  SPEAKER_01: Okay, Kenny, I know we're right about at the end of our time.
• 18:45
  SPEAKER_01: So thank you so much for coming on.
• 18:48
  SPEAKER_01: What is the best way for folks to get in touch with you?
• 18:50
  SPEAKER_02: I think they can find me on LinkedIn.
• 18:53
  SPEAKER_01: Okay.
• 18:54
  SPEAKER_02: Yeah.
• 18:54
  SPEAKER_01: Great.
• 18:55
  SPEAKER_02: And thank you for having me.
• 18:57
  SPEAKER_01: Yeah, well, we'll throw a link to your LinkedIn in the show notes for sure.
• 19:01
  SPEAKER_01: And yeah, thanks again for coming on.
• 19:03
  SPEAKER_01: It's been great speaking with you.
• 19:05
  SPEAKER_02: Thanks.
• 19:05
  SPEAKER_02: Bye-bye.
• 19:08
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 19:15
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 19:19
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.