Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin

Transcript

Building Secure Financial Data Platforms at AgileEngine with Valentyn Druzhynin · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: We have a limited release windows because we have a rebalance for our ETF every three months, and we have a code freeze for one month before every rebalance.
• 0:12
  SPEAKER_01: So we have like two months of releases, then one month freeze, and then cycle repeats.
• 0:17
  SPEAKER_01: I think it somehow affects our development.
• 0:20
  SPEAKER_01: We're trying to put some heavy impact tasks at the beginning of the release cycles and just some little tasks at the end of the release cycles.
• 0:35
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering Airflow for data engineering in AI.
• 0:45
  SPEAKER_02: Here, Airflow community, we've got exciting news.
• 0:48
  SPEAKER_02: The annual Apache Airflow survey is live.
• 0:51
  SPEAKER_02: In just seven minutes, you can help shape the future of Airflow and grab a free Airflow 3 fundamentals or DAG authoring certification worth$150 each.
• 1:01
  SPEAKER_02: Plus, you can enter to win an exclusive workshop with Mark Lumberti.
• 1:06
  SPEAKER_02: Don't miss out.
• 1:07
  SPEAKER_02: The survey link is in the show notes and closes November 21st.
• 1:13
  SPEAKER_02: Hi everyone.
• 1:14
  SPEAKER_02: Welcome to the Data Flowcast.
• 1:16
  SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Valentine Druzinin, who is a senior data engineer at Agile Engine.
• 1:24
  SPEAKER_02: Valentine, how are you?
• 1:25
  SPEAKER_02: Good, good.
• 1:26
  SPEAKER_02: It's great to have you on the podcast.
• 1:28
  SPEAKER_02: I'm excited for this conversation.
• 1:30
  SPEAKER_02: For our listeners, for today's podcast, we're going to discuss how Airflow can be used for financial service use cases.
• 1:37
  SPEAKER_02: Before we dive into that, which I think is a very popular topic right now, let's start with some background.
• 1:44
  SPEAKER_02: And can you tell me what Agile Engine does and what sort of projects you're currently working on with them?
• 1:50
  SPEAKER_01: Sure.
• 1:50
  SPEAKER_01: Yeah, Agile Engine is a medium-sized outsourced software company.
• 1:55
  SPEAKER_01: We build products and data solutions for clients in fintech, e-commerce, media, and many other domains.
• 2:04
  SPEAKER_01: For my current project, we run a financial data platform.
• 2:09
  SPEAKER_01: This platform calculates different ETFs daily.
• 2:13
  SPEAKER_01: So my daily workflow includes like uploading market data from files, APIs, and some remote databases.
• 2:23
  SPEAKER_01: After that, we do some data validation.
• 2:27
  SPEAKER_01: After that, we do some calculation of weights and other factors which will impact the list of stocks with what will be included in ETF.
• 2:38
  SPEAKER_01: And the final step is just gathering the report to provide to researchers and managers.
• 2:44
  SPEAKER_01: So they will just do some actions or ETFs.
• 2:47
  SPEAKER_01: So my current role, I'm just responsible for running, maintaining current DACs without failures, creating new DACs for some new calculations, building some different reports for managers, and developing new integrations for different data sources.
• 3:08
  SPEAKER_02: Okay, great.
• 3:09
  SPEAKER_02: And you mentioned managing the DAGs that help you provide all of this data.
• 3:15
  SPEAKER_02: So tell me, I assume those are Airflow DAGs.
• 3:17
  SPEAKER_02: Tell me how Airflow fits in this helping with data platform management for financial services.
• 3:24
  SPEAKER_01: We mainly use Airflow as orchestrator.
• 3:28
  SPEAKER_01: We use a managed airflow on AWS.
• 3:31
  SPEAKER_01: So it connects like many uh secure sources, and we run a lot of quality checks and given a lot of clean reports for it.
• 3:41
  SPEAKER_01: I can probably group some best practices.
• 3:44
  SPEAKER_01: What we use for our financial platform.
• 3:48
  SPEAKER_01: The first group is security because security is the most important part.
• 3:53
  SPEAKER_01: We do not store secrets in logs, so secrets and keys never appear in our logs.
• 3:59
  SPEAKER_01: We also limit the login of data and data what we process and data what we receive from sources because a lot of them are very sensitive, and some critical calculation results also go to some securely limited data storage where just a limited number of people have access.
• 4:19
  SPEAKER_01: So sometimes this leads to some complicated debugs, but security is a main priority.
• 4:25
  SPEAKER_01: So we're trying to keep our system more secure.
• 4:29
  SPEAKER_01: The main rule that we try to apply, we're trying to keep our tasks as small as possible so that can help us to find some errors much faster.
• 4:39
  SPEAKER_01: So we also do some least privileged connections, so we just minimize access for every connection.
• 4:46
  SPEAKER_01: For example, just to read some S3 bucket or just to read some database table.
• 4:51
  SPEAKER_01: And for running some tasks, we apply some additional approvals.
• 4:55
  SPEAKER_01: Only some managers or some developers have access to run it.
• 5:00
  SPEAKER_01: Also, we do some data quality, we do some schema and sanity checks.
• 5:06
  SPEAKER_01: For example, checking if prices are not negative, but that's some just simple checks.
• 5:12
  SPEAKER_01: What comes to my mind.
• 5:14
  SPEAKER_01: And we also often do some after-calculation validations.
• 5:20
  SPEAKER_01: So almost every calculation leads to an additional validation step.
• 5:25
  SPEAKER_01: Because we need to be sure that the current calculation we do not proceed with the future efforts to prevent some spending additional time and spending additional money for it.
• 5:40
  SPEAKER_01: I would say that's reliability and retries.
• 5:44
  SPEAKER_01: We do some smart retries.
• 5:46
  SPEAKER_01: So we do a lot of exponential backoffs for 500 errors, but we do not do retries for some data logic.
• 5:53
  SPEAKER_01: So we prefer to do alerting instead.
• 5:57
  SPEAKER_01: And we prefer to do some important rides.
• 5:59
  SPEAKER_01: So every rerun Fodak just recalculates all the states.
• 6:04
  SPEAKER_01: And the operations and observability, that's also the separate group.
• 6:08
  SPEAKER_01: For every task, we track what data came from, who ran the task, when it was run, for what it was executed.
• 6:17
  SPEAKER_01: So a lot of things in a logs.
• 6:19
  SPEAKER_01: So Airflow is not just like a scheduler sometimes.
• 6:25
  SPEAKER_01: So it's very like secure, auditable orchestration layer for us.
• 6:30
  SPEAKER_02: Yeah, got it.
• 6:31
  SPEAKER_02: That makes sense.
• 6:32
  SPEAKER_02: And so if I understand you correctly, the orchestration functionality you're using is what's baked into Airflow, but you really leverage the security and kind of the flexibility to implement things like data quality checks, because of course, I would imagine with financial services, the data quality is really important.
• 6:50
  SPEAKER_02: And as you mentioned, obviously a highly regulated industry.
• 6:54
  SPEAKER_02: Yeah, you have to leverage those security features.
• 6:57
  SPEAKER_02: Maybe relevant to that, but I'm curious for this specific industry, how you think Airflow fits in compared to maybe other airflow or other industries that you've worked in.
• 7:12
  SPEAKER_02: As you mentioned, with your job, it sounds like you get to work on projects with companies in a couple of different industries.
• 7:18
  SPEAKER_02: So, how do you think Airflow is helpful for financial services maybe compared to other types of projects that you've worked on?
• 7:27
  SPEAKER_01: How to compare it if it's useful?
• 7:29
  SPEAKER_01: I mean, we just use it a bit differently in comparison to other domains.
• 7:34
  SPEAKER_01: As I said, if the main priority is security and compliance, so we keep the secrets and logs.
• 7:41
  SPEAKER_01: We do not store the secrets and logs.
• 7:44
  SPEAKER_01: We have some limited storages, as I said before.
• 7:47
  SPEAKER_01: We also have some timing and market windows.
• 7:53
  SPEAKER_01: So we work with market calendars and strict cutoffs.
• 7:57
  SPEAKER_01: And sometimes, I think maybe in 30 and 40%, vendors are late with our files, and we have to do some controlled backfills.
• 8:07
  SPEAKER_01: Data quality checks.
• 8:09
  SPEAKER_01: Also, I've said that we have some data quality checks here, more deeper and more detailed.
• 8:14
  SPEAKER_01: I've just provided an example that we have like just simple priceless checks, but here they are very complicated.
• 8:21
  SPEAKER_01: Reliability and smart retries.
• 8:23
  SPEAKER_01: And I think the other interesting part, I think it's maybe not connected to airflow itself, but we have uh limited release windows because we have a rebalance for our ETF every three months, and we have a code freeze for one month before every rebalance.
• 8:42
  SPEAKER_01: So hell, two months of releases, then one month's cont freeze, and then the cycle repeats.
• 8:48
  SPEAKER_01: I think it somehow affects our development.
• 8:50
  SPEAKER_01: We're trying to put some heavy impact tasks at the beginning of the release cycles and just some little task at the end of the release cycles.
• 9:01
  SPEAKER_02: I'm curious when you've come in on these projects and you're helping companies implement a data platform.
• 9:08
  SPEAKER_02: Are you considering other orchestrators or Airflow competitors, or what makes you choose Airflow?
• 9:15
  SPEAKER_01: We tried a couple of them, but I think Team likes the community, how it's big and how it's I think friendly.
• 9:23
  SPEAKER_01: The documentation itself, it's very verbose.
• 9:26
  SPEAKER_01: You can find a lot of different implementations, a lot of different approaches.
• 9:33
  SPEAKER_01: And we agree that a lot of complicated things can be done very easily.
• 9:40
  SPEAKER_01: So that's I think the main point what we uh focused on when we decided to use Airflow.
• 9:50
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:00
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:11
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
• 10:20
  SPEAKER_02: I love a good plug for the Airflow community.
• 10:22
  SPEAKER_02: It is my favorite part as well.
• 10:24
  SPEAKER_02: It's definitely very strong.
• 10:26
  SPEAKER_02: I'm just coming off of Airflow Summit probably weeks ago by the time this comes out, but it was great to see everybody there and lots of contributors and folks getting in one place.
• 10:36
  SPEAKER_02: And the virtual version is happening this week.
• 10:39
  SPEAKER_02: Yeah, it's everybody showing up and lots of stuff going on in the community, which is really cool.
• 10:44
  SPEAKER_02: I'm curious for your current project.
• 10:47
  SPEAKER_02: In my experience, financial services or other highly regulated industries that have really strict security and compliance related requirements.
• 10:55
  SPEAKER_02: Sometimes upgrading in those cases can be more challenging.
• 11:00
  SPEAKER_02: And we spoke earlier about Airflow 3.
• 11:02
  SPEAKER_02: I'm curious if you're already running Airflow 3 or if not, what are you considering it?
• 11:07
  SPEAKER_02: And then what is the process for you to migrate to a new version again, given the constraints that you have?
• 11:14
  SPEAKER_01: As I said before, we run our Airflow on a managed Airflow service on AWS.
• 11:20
  SPEAKER_01: And unfortunately, we don't see a plans for migrating to Airflow 3 from AWS side.
• 11:28
  SPEAKER_01: We have plans to switch to the new version.
• 11:32
  SPEAKER_01: I think it's probably one of the major steps.
• 11:35
  SPEAKER_01: And we already started it.
• 11:37
  SPEAKER_01: So we just finished the migrating of all our services and components to a newer Python version.
• 11:45
  SPEAKER_01: So we switched to the 312 version.
• 11:48
  SPEAKER_01: And the next step, I think, will be migrating to the Airflow itself, but we're just thinking of what to choose.
• 11:56
  SPEAKER_01: There are just dilemma.
• 11:57
  SPEAKER_01: We're thinking of switching to GCP and using managed GCP because GCP managed airflow on GCP because GCP already supports version 3, or doing some custom deployment on AWS.
• 12:12
  SPEAKER_02: Got it.
• 12:12
  SPEAKER_02: Interesting.
• 12:13
  SPEAKER_02: Yeah, it can definitely be a challenge with managed services.
• 12:16
  SPEAKER_02: I think I would be remiss without adding a plug for Astronomer's Managed Service Astra.
• 12:21
  SPEAKER_02: This is part of why we offer new release support on day zero, because it takes folks a long time to upgrade.
• 12:28
  SPEAKER_02: We want them to be able to try it out.
• 12:30
  SPEAKER_02: But in any case, no matter where you're running Airflow, always a consideration and a process.
• 12:36
  SPEAKER_02: Once you migrate in whatever managed service, whether you move to Google or stay on Amazon or kind of roll your own, what features are you most excited about?
• 12:45
  SPEAKER_02: Do you think anything will be particularly helpful for your use case?
• 12:49
  SPEAKER_01: I think the new version of Airflow gives a lot of cool features, a lot of very useful features.
• 12:55
  SPEAKER_01: And I think that's why we decided maybe to move to the new version.
• 12:59
  SPEAKER_01: I can maybe elaborate three top of them, what I think is very useful.
• 13:04
  SPEAKER_01: One is DAG versioning.
• 13:06
  SPEAKER_01: I think that features is what really we need right now, because right now we're manually doing versioning for every DAC.
• 13:16
  SPEAKER_01: So every time we have some changes, we just deploy almost the same or just a changed DAC with just a different name just to keep track of the versions.
• 13:27
  SPEAKER_01: And I think this feature will help us a lot.
• 13:29
  SPEAKER_01: I think we will use a scheduled managed big fills because it's easy to plan and easy to monitor and perfect for our use cases for late vendor files and event-driven scheduling and assets.
• 13:45
  SPEAKER_01: We don't have that functionality on Airflow 2.
• 13:49
  SPEAKER_01: And we have a lot of things which can be triggered, like events, because all DAGs are just scheduled.
• 13:57
  SPEAKER_01: Yeah, I think it's three top features what we would like to use.
• 14:02
  SPEAKER_02: This is consistent with what we've heard a lot at Astronomer, particularly on DAG versioning, is that for those in again highly regulated industries, if your DAGs are changing over time, just for compliance purposes, being able to see exactly what code was run for every run is really helpful.
• 14:19
  SPEAKER_02: I love this feature, especially because you don't really have to do anything for the most basic version anyway to implement it.
• 14:25
  SPEAKER_02: So it's the, I think one of the most impactful and also the easiest to use from the user perspective.
• 14:32
  SPEAKER_02: Backfills and event-driven scheduling are also really powerful.
• 14:35
  SPEAKER_02: So that's cool to hear that you might implement those.
• 14:38
  SPEAKER_02: Yeah.
• 14:39
  SPEAKER_02: And I understand that speaking of Airflow 3, that you also contributed to that release.
• 14:44
  SPEAKER_02: So I would love to talk a little bit about that and your experience contributing to the project.
• 14:50
  SPEAKER_01: Yeah, that was very fun and very interesting.
• 14:54
  SPEAKER_01: I decided to learn more about Airflow itself and just started looking at the repository itself and noticed that you can easily contribute and bring something useful to the community.
• 15:08
  SPEAKER_01: Yeah, I made some several useful contributions.
• 15:12
  SPEAKER_01: They're not so big, but I think they're useful.
• 15:16
  SPEAKER_01: I've improved UI on Airflow 3, did two interesting things on Doug Run's page.
• 15:23
  SPEAKER_01: So I made a run duration field sortable.
• 15:26
  SPEAKER_01: I think it will be even useful for us.
• 15:29
  SPEAKER_01: You can easily uh spot long runs or find some defects when some runs are very long.
• 15:36
  SPEAKER_01: And I also added a conf column on uh Dougrand's page.
• 15:41
  SPEAKER_01: It was on the Airflow 2 version and just aligns the same behavior for Airflow 3.
• 15:47
  SPEAKER_01: This field I think is very useful for us because we do a lot of backfields, and you don't need to just select every Dog run to see what configuration there was used for running some specific runs.
• 16:01
  SPEAKER_01: You just can take a look on the list of runs and see what configuration was used.
• 16:07
  SPEAKER_01: Yeah, and to addition to that, I also added some features and did some bug fixes for providers.
• 16:16
  SPEAKER_01: In my experience, the community is great, and reviewers are very quick and helpful, and makes your contributions feel easy.
• 16:26
  SPEAKER_01: Everyone is trying to help you, no bullying.
• 16:29
  SPEAKER_01: I've even learned some useful tools like Breeze, very useful tool for running Airflow locally.
• 16:36
  SPEAKER_01: You could just run it in one single command.
• 16:38
  SPEAKER_01: That's really great.
• 16:40
  SPEAKER_01: Yeah.
• 16:40
  SPEAKER_02: Yeah, that's great to hear.
• 16:42
  SPEAKER_02: I think from my perspective, the breeze setup can be a little tricky the first time.
• 16:46
  SPEAKER_02: And then once you have it running, everything is great.
• 16:49
  SPEAKER_02: It works really well.
• 16:51
  SPEAKER_02: So that's really cool.
• 16:52
  SPEAKER_02: I didn't realize you had contributed the configuration column in the UI.
• 16:56
  SPEAKER_02: I've used that myself.
• 16:57
  SPEAKER_02: I think it's super helpful.
• 16:59
  SPEAKER_02: So that's really cool that you were able to jump in and just add the things.
• 17:02
  SPEAKER_02: I think to your point, lots of other people are benefiting from those as well.
• 17:07
  SPEAKER_02: During this process, I'm curious, you spoke about the community being really welcoming and really helpful with getting you started.
• 17:14
  SPEAKER_02: Was there anything you found more sticky in the process or difficult?
• 17:20
  SPEAKER_01: No, I don't think I found something sticky and hard in the process.
• 17:25
  SPEAKER_01: I just can repeat that I really like the community, really how they do the reviews.
• 17:30
  SPEAKER_01: I really love that you can use some popular frameworks because the new Airflow version, Airflow switch to the fast API and start using React for UI, I think that's very good for your resume if you would like to improve your technical skills.
• 17:49
  SPEAKER_02: Absolutely.
• 17:50
  SPEAKER_02: And just to close out that conversation, if others are listening who have maybe been on the fence or a little intimidated about jumping in and contributing to the project, but would like to, do you have any tips for them for getting started?
• 18:04
  SPEAKER_01: Yeah, sure.
• 18:05
  SPEAKER_01: From first perspective, it maybe sounds very scary, but I can give like advice to starting contributors, to new contributors.
• 18:13
  SPEAKER_01: Just start small.
• 18:15
  SPEAKER_01: Even a tiny UI or documentation change is very valuable.
• 18:20
  SPEAKER_01: And try to keep the pull request focused and don't be afraid to ask questions.
• 18:26
  SPEAKER_01: The very cool thing for Airflow community, what I like, is that Airflow has its own Slack chat.
• 18:34
  SPEAKER_01: So you can ask people for help.
• 18:36
  SPEAKER_01: You can tag them, you can ask for help in the channels.
• 18:39
  SPEAKER_01: That's very useful.
• 18:41
  SPEAKER_01: Also, I would recommend for beginners just search for issues with a good first issue label.
• 18:49
  SPEAKER_01: This will help you to jump into the project development and start learning new about airflow.
• 18:57
  SPEAKER_02: Those are all really great tips.
• 18:59
  SPEAKER_02: And we have some PMC members on astronomer staff, and they always talk about how actually smaller PRs are much, much easier for them because they're a quick review and you can just get them reviewed and emerge really easily.
• 19:13
  SPEAKER_02: So they always say there's no such thing as too small of a PR.
• 19:17
  SPEAKER_02: So that's yeah, a great thing to keep in mind.
• 19:20
  SPEAKER_02: Okay.
• 19:20
  SPEAKER_02: In the interest of time, Lentine, this has been really helpful.
• 19:23
  SPEAKER_02: Thank you for going through that.
• 19:25
  SPEAKER_02: I always like to end with a standard final question, and that is what would you most like to see from the Airflow project in the near future?
• 19:31
  SPEAKER_01: Yeah, I would like probably to emphasize two things, and most of them are connected to the assets and vouchers.
• 19:38
  SPEAKER_01: I think we expect some SFTP arrival watcher, so we maybe can track all the arrival of our vendor files.
• 19:48
  SPEAKER_01: And also that I think that would be useful for us if we have some voucher for AWS Event Breach that can help us maybe unify all AWS events and all AWS services, and we can connect, not just using by uh AWS SQS, but event breach, I think will be very useful for us.
• 20:13
  SPEAKER_02: Yeah, that's a great one.
• 20:14
  SPEAKER_02: Definitely looking forward to more asset watchers being added.
• 20:18
  SPEAKER_02: I know there's been some that folks have implemented themselves and are intending to contribute back to the community.
• 20:25
  SPEAKER_02: So hopefully one of yours is on that list or somebody will, or maybe you can contribute it since you're already somebody who has gone through that process.
• 20:33
  SPEAKER_02: Well, Valentine, it's been really great speaking with you.
• 20:36
  SPEAKER_02: Thanks for joining us.
• 20:37
  SPEAKER_02: And what's the best way for folks to get in touch with you?
• 20:40
  SPEAKER_01: Yeah, you can talk to me in LinkedIn or GitHub.
• 20:44
  SPEAKER_01: I'll just leave it on the video.
• 20:46
  SPEAKER_01: Yeah.
• 20:47
  SPEAKER_02: Okay.
• 20:48
  SPEAKER_01: For sure, connect and talk.
• 20:49
  SPEAKER_01: Thank you.
• 20:50
  SPEAKER_02: Great.
• 20:50
  SPEAKER_02: We can put those in the show notes.
• 20:52
  SPEAKER_02: Thank you again for joining me.
• 20:53
  SPEAKER_02: It's been really great speaking.
• 20:55
  SPEAKER_01: Thank you.
• 20:58
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 21:05
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 21:08
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.