Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Airflow Strategies for Business Efficiency at Campbell with Larry Komenda

Transcript

Airflow Strategies for Business Efficiency at Campbell with Larry Komenda · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: I've made it a big point that we should be very clear whether our DAGs are idempotent or not idempotent, because that's something that many years ago would cause major issues when we were managing these things in a production environment.
• 0:11
  SPEAKER_01: And so, you know, having that all built in the airflow UI ends up making it extremely clear and easy to deal with when you know that this process is not going to be idempotent and you have to be mindful of that versus, you know, I can run this thing 50 times and it's always going to look the same as long as it's working correctly.
• 0:28
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:37
  SPEAKER_02: All right.
• 0:38
  SPEAKER_02: Hello everyone.
• 0:39
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:41
  SPEAKER_02: I'm your host, Mark Lamatli, and today I'm super excited because Larry Commander, Chief Technology Officer at Campbell, is joining us for this episode.
• 0:52
  SPEAKER_02: First question, how are you doing?
• 0:54
  SPEAKER_01: I'm good.
• 0:55
  SPEAKER_01: I'm good.
• 0:55
  SPEAKER_01: I didn't have my coffee, so I'm ready to rock.
• 0:59
  SPEAKER_02: Awesome, awesome.
• 1:00
  SPEAKER_02: So before diving into the episode, I would like to give just the primary takeaway that people will get from this episode.
• 1:07
  SPEAKER_02: So we will talk about testing DAGs and testing environments.
• 1:11
  SPEAKER_02: Also a little bit about self-serving airflow and some additional tips that we may talk about during this episode.
• 1:20
  SPEAKER_02: But let's begin with what is your role at Cumbul today?
• 1:26
  SPEAKER_01: Yeah, sure.
• 1:27
  SPEAKER_01: Yeah, thanks for having me, Mark.
• 1:28
  SPEAKER_01: Very excited to talk about Airflow and everything we do.
• 1:30
  SPEAKER_01: So I think you said the CTO at Campbell.
• 1:33
  SPEAKER_01: And so my main role is I'm responsible for setting and leading the technology vision and strategy at the firm.
• 1:38
  SPEAKER_01: I also oversee our technology and engineering teams, both from a personnel as well as a project management perspective.
• 1:44
  SPEAKER_01: And then I'm responsible for communicating progress along that technology roadmap.
• 1:48
  SPEAKER_01: For those not familiar with Campbell, we're a quantitative hedge fund based out of Baltimore, Maryland.
• 1:53
  SPEAKER_01: And because our investment process is fully systematic, it's really important that we have robust technology and data infrastructure.
• 2:01
  SPEAKER_01: It's very critical to our success.
• 2:03
  SPEAKER_01: And that's a lot of the driver behind why we've done things like implement Airflow.
• 2:07
  SPEAKER_01: Ultimately, I like to think about my role as just ensuring that the company has the right technology to do their jobs as effectively as possible, right?
• 2:14
  SPEAKER_01: I mean, that's the job of a CTO, especially as it applies to that those core competencies.
• 2:19
  SPEAKER_01: And so that means just not just today, but you know, going forward in the future, as technology is evolving as our business needs are changing.
• 2:26
  SPEAKER_01: And so a few years ago when we implemented Airflow or initially, that was in that same vein, right?
• 2:30
  SPEAKER_01: We we saw a need.
• 2:31
  SPEAKER_01: And I was the person who led the team that adopted Airflow a few years back in order to solve business needs and business challenges.
• 2:37
  SPEAKER_02: Okay, so that's very interesting because that means you are using Airflow with critical workflows today.
• 2:43
  SPEAKER_01: We are, yes.
• 2:44
  SPEAKER_02: Okay.
• 2:45
  SPEAKER_02: And out of curiosity, what were you using before Airflow?
• 2:50
  SPEAKER_02: Did you have any other orchestrator at that time?
• 2:52
  SPEAKER_02: Or it was maybe before your time?
• 2:54
  SPEAKER_01: Yeah, so a little before my time when I took over the market data team originally, but it was just an in-house built scheduling platform.
• 3:00
  SPEAKER_01: Okay.
• 3:00
  SPEAKER_02: I see.
• 3:01
  SPEAKER_02: Yeah.
• 3:01
  SPEAKER_02: And actually, can you tell us more about how do you use Airflow within your organization?
• 3:06
  SPEAKER_02: Not diving maybe too much into details about your use cases, but just to have a broad idea.
• 3:11
  SPEAKER_01: Yeah, sure.
• 3:12
  SPEAKER_01: And feel free to ask if you if you want more details on anything in particular.
• 3:15
  SPEAKER_01: Okay.
• 3:16
  SPEAKER_01: But I think it's interesting is we use Airflow in a variety of ways and different patterns within the company.
• 3:21
  SPEAKER_01: So, first, our that initial use case, that original reason for onboarding Airflow, and so our probably our primary platform for using Airflow is for managing data ingestion pipelines.
• 3:31
  SPEAKER_01: So this is within our market data team.
• 3:33
  SPEAKER_01: That team is responsible for the support of research and other business teams within the organization for bringing in data, ensuring that those data pipelines are running in production, they're timely, the data is correct, able to provide production support and those kinds of things.
• 3:49
  SPEAKER_01: So that was that initial original use case.
• 3:51
  SPEAKER_01: But we also use it as an automation tool within things like our trading and operations department.
• 3:55
  SPEAKER_01: It supports a number of different teams there.
• 3:58
  SPEAKER_01: It's used as the external scheduler system for investment models.
• 4:01
  SPEAKER_01: And then we also use it on the side as for some specialized use cases, things like some ML processes.
• 4:06
  SPEAKER_01: And as a lot of firms are continue to explore generative AI processes, I would expect that as we do the same, Airflow is very likely going to be right in the middle of a lot of what we do there.
• 4:16
  SPEAKER_02: Well, so if I understood correctly, you are already using for machine learning workflows.
• 4:21
  SPEAKER_01: Yeah, yes.
• 4:22
  SPEAKER_01: We have a dedicated Airflow instance that is running a machine learning model to provide data to a dashboard that our investment committee uses to review things.
• 4:31
  SPEAKER_02: That's awesome because I know that many people wonder if they can use Airflow for machine learning models.
• 4:36
  SPEAKER_02: And they totally can.
• 4:37
  SPEAKER_02: Airflow is able to do that.
• 4:38
  SPEAKER_02: And that's also why Airflow is great, in my opinion.
• 4:41
  SPEAKER_02: It's the flexibility that it brings.
• 4:43
  SPEAKER_02: Speaking of the use cases, do you have any near real-time workflows to run, or is it more like DAX that run that run daily, or do you have any DAX that need to run quickly, like maybe every five minutes or something like that?
• 4:58
  SPEAKER_01: Within the market data team, generally nothing that that's running on that type of cycle.
• 5:03
  SPEAKER_01: We have intraday processes that are running throughout every 30 minutes or so to bring in intraday data.
• 5:09
  SPEAKER_01: We have a lot of daily processes that are pulling in end-to-day pricing and other things like that.
• 5:13
  SPEAKER_01: We have some processes on the operation side that are running in a continual mode, just to support more event-driven processes.
• 5:20
  SPEAKER_01: We have, you know, that's one of the, again, as you said, one of the great things about Airflow is it's so flexible and customizable for kind of any need.
• 5:26
  SPEAKER_01: That's what we've we found has been really valuable for us.
• 5:28
  SPEAKER_01: And for instance, we're able to use it to assist with processes that are, say, our compliance team follows where they need to be able to upload data to a third party on a as quickly as possible.
• 5:39
  SPEAKER_01: And so we have event-driven processing that's happening there.
• 5:41
  SPEAKER_01: It's waiting for file drops and those kinds of things.
• 5:44
  SPEAKER_02: Okay.
• 5:44
  SPEAKER_02: And so as you said, like those workflows are pretty critical.
• 5:47
  SPEAKER_02: How do you make sure that they don't fall apart?
• 5:49
  SPEAKER_02: Like, how do we make sure that they run, they keep running?
• 5:52
  SPEAKER_02: Do you have anything in place?
• 5:53
  SPEAKER_02: Do you test your tasks, DAGs, do you have a testing environment?
• 5:57
  SPEAKER_02: What is your strategy around those workflows to make sure that they are reliable?
• 6:01
  SPEAKER_01: Yeah, across everything, we have a robust software development lifecycle.
• 6:05
  SPEAKER_01: All of our development, whether you're talking airflow DAGs to anything else that we are building, goes through a very in-depth testing process, all of those kinds of things, deployment through test environments before anything ever makes it to production.
• 6:19
  SPEAKER_01: We ensure that appropriate testing is in place.
• 6:21
  SPEAKER_01: We ensure that uh appropriate approvals are in place that make sure that we fleshed out all the edge cases and those kinds of things before anything that goes to production.
• 6:29
  SPEAKER_01: So ultimately, what we're putting in production, we have a high degree of comfort that it's doing what it's supposed to do.
• 6:36
  SPEAKER_01: Once we're in production, we have a lot of good infrastructure in place for ensuring that the environments themselves are up and running.
• 6:43
  SPEAKER_01: So we have monitoring systems to ensure that the Airflow schedulers, Airflow web server, things like that are operational and that things aren't going down without us being aware of it.
• 6:52
  SPEAKER_01: Obviously, where we utilize a lot of the error handling and the notification uh functionality built into Airflow itself.
• 6:57
  SPEAKER_01: One of the things that was really appealing to us was being able to utilize that type of functionality without having to write a lot of with our plate customized notification, notification framework outside of it, the stuff that we used to have to maintain and manage ourselves.
• 7:11
  SPEAKER_01: Now that's all built right into the platform.
• 7:12
  SPEAKER_01: And so we can focus on just building the kind of the meat of these DAGs and these business processes and testing those specifically.
• 7:20
  SPEAKER_01: Just it helps speed things up and it, like you said, it ensures that we know that everything's running as expected.
• 7:26
  SPEAKER_02: Yeah.
• 7:26
  SPEAKER_02: That's interesting because actually, so you said that you have different environments to make sure that everything runs smoothly in production, which is great.
• 7:33
  SPEAKER_02: But also, can you maybe tell us more about how do you test your DAGs or your tasks, or maybe like in the what are the kind of checks that you run on your DAGs to make sure that they are reliable, or maybe that they follow some prerequisites before going to production?
• 7:49
  SPEAKER_02: I know that, for example, some people, like some companies actually check that their DAGs have some tags, they have the owner or something like that.
• 7:57
  SPEAKER_02: Do you test that as well?
• 7:59
  SPEAKER_01: Yeah, we don't have any automated testing around things like tags.
• 8:02
  SPEAKER_01: We generally will catch a lot of that during the code review process.
• 8:04
  SPEAKER_01: So, one of the things that's interesting about the way that we, especially on that market, the market within the market data team, which I'm sure was is one of our main kind of drivers and one of our main users of the of Airflow in the company, the way that we went ahead and the way we build our Airflow DAGs is maybe a little different than what you might see in other places.
• 8:22
  SPEAKER_01: And so rather than build everything right into an Airflow DAG itself, what we ended up doing was we decided to organize the meat, the complexity and the actual processes themselves, the things that these market data DAGs are doing into separate what we call quote unquote business logic packages, which are really just Python packages that conform to a standard template that we've built internally.
• 8:45
  SPEAKER_01: We're able to version and test these business logic packages externally from the Airflow system before we ever go ahead and wrap and orchestrate this inside of a DAG.
• 8:54
  SPEAKER_01: And so that makes it very easy for an engineer to go through the steps, build and do developer testing, refactoring, those kinds of things right within their own development environment before they decide to add that additional layer of abstraction by wrapping the code itself inside of a DAG and orchestrating it through Airflow.
• 9:14
  SPEAKER_02: Wow.
• 9:14
  SPEAKER_02: So just to make sure, so those packages contain like business logic functions.
• 9:19
  SPEAKER_02: So nothing related to Airflow.
• 9:21
  SPEAKER_02: They are just like functions, things that you want to execute, right?
• 9:24
  SPEAKER_02: So you are able to test those functions like you would usually do without Airflow because it's something else.
• 9:31
  SPEAKER_02: And then you add those tasks, those functions to your uh Airflow DAGs.
• 9:36
  SPEAKER_02: Is that correct?
• 9:37
  SPEAKER_01: Exactly.
• 9:37
  SPEAKER_01: And so our rules have always been that every process that we build should be able to run within an Airflow DAG, of course, but it should also be able to run outside of Airflow itself.
• 9:47
  SPEAKER_01: The reason we did this, you know, a few years ago, we were in the process of learning Airflow at the same time that we were beginning to migrate to this new platform and build this more robust, reliable data framework.
• 9:58
  SPEAKER_01: And engineering, you always know there's in technology in general, there's always unknowns.
• 10:02
  SPEAKER_01: And so we expected they were going to be learned along the way.
• 10:05
  SPEAKER_01: And so we wanted to make sure that we always had because ultimately the this team and anything that we do is to support the business and that production environment.
• 10:12
  SPEAKER_01: It's critical that it's operational and that it's running as expected, and that it's not preventing anything in the business from functioning.
• 10:19
  SPEAKER_01: And so we wanted to always make sure that we always had a plan, a backup plan if we were to encounter any issues with Airflow itself.
• 10:25
  SPEAKER_01: Now, knock on wood, the Airflow environment has been incredibly stable and we never really needed to go that route.
• 10:30
  SPEAKER_01: But we found that there's a lot of other benefits to separating these things out.
• 10:34
  SPEAKER_01: It makes it a lot easier for engineers to do that development, right?
• 10:38
  SPEAKER_01: They can go through and make these lay out these business logic processes in a very functional way.
• 10:46
  SPEAKER_01: They're able to script things out, they're able to test it right down on the workstation, like I said.
• 10:49
  SPEAKER_01: But the other thing too is we're able to build a nice framework of unit tests and other tests around these business logic packages.
• 10:56
  SPEAKER_01: And because we're able to version them and deploy them inside of our own internal package repo, it's easy to see and understand changes between environments.
• 11:04
  SPEAKER_01: You had asked about testing environments.
• 11:06
  SPEAKER_01: So we have a dedicated testing environment, you know, multiple dedicated testing environments, as well as obviously our production environment.
• 11:12
  SPEAKER_01: And then versioning these packages means that we can look at environment A and say, oh, we're running version 1.0.
• 11:18
  SPEAKER_01: And environment B, we're running version 1.1, which is where we added some additional data validations and notification, you know, error handling and those kinds of things.
• 11:27
  SPEAKER_01: And we can very easily tell the difference between the two environments.
• 11:31
  SPEAKER_01: And because we have a robust set of unit tests that that are one of the standards that we enforce whenever we're building these business logic packages, it also gains that any change to the package automatically kicks off these tests in our build tool.
• 11:44
  SPEAKER_01: So we can easily find issues.
• 11:47
  SPEAKER_01: It means that if we're also doing things like testing a new version of NumPy by two that just came out and broke a lot of people's processes, it's very easy for us to validate all the kind of that core functionality without having to run it in Airflow and then step through and debug those Airflow logs directly.
• 12:03
  SPEAKER_01: We can just run it into our build process very quickly and identify potential issues.
• 12:07
  SPEAKER_01: Things like upgrading Python versions, which generally go fairly smoothly, but there's always some additional complication.
• 12:13
  SPEAKER_01: And so we're able to do the same thing.
• 12:15
  SPEAKER_01: And so by the time we put these processes in Airflow, we are very confident that the underlying process, the code itself, is sound.
• 12:22
  SPEAKER_01: We know that it's been tested either locally or and we know that it's been put through a code review and reviewed by other members of the team.
• 12:30
  SPEAKER_01: We know we've written unit tests, we know we all these things are in place, and then we orchestrate the process within a day.
• 12:37
  SPEAKER_01: And then as you're debugging, the only thing you need to worry about is ensuring that communication between tasks is working as you expected, that your timings are right, that your dependency graph is connected appropriately, and things like that.
• 12:50
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:59
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 13:11
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 13:18
  SPEAKER_02: That's pretty amazing.
• 13:19
  SPEAKER_02: Like I'm sure that people will take notes because that's many insights about testing your DAGs and your tasks.
• 13:25
  SPEAKER_02: That's actually a really nice way to test your tasks.
• 13:28
  SPEAKER_02: Like separating the logic from the airflow logic is much easier way to test that everything works as expected.
• 13:36
  SPEAKER_02: Thank you so much for those insights.
• 13:38
  SPEAKER_02: Also, I think you mentioned that many people, or actually many teams, interact with your Airflow environment today.
• 13:45
  SPEAKER_02: How do you make sure that it works smoothly for everybody?
• 13:49
  SPEAKER_02: Did you put anything in place, or are you thinking of self-serving Airflow in some ways?
• 13:55
  SPEAKER_02: Like how do you manage it?
• 13:56
  SPEAKER_01: Yeah, so one of the things that we follow is we have multiple teams within the company, obviously, and also even within there's a technology umbrella that I oversee.
• 14:04
  SPEAKER_01: And I've read some articles about this.
• 14:05
  SPEAKER_01: I think it was a viraj.
• 14:07
  SPEAKER_01: Somebody wrote an article about the concept of do you do a monolith airflow?
• 14:10
  SPEAKER_01: Do you go independent micro kind of airflow environments?
• 14:12
  SPEAKER_01: And we definitely lean more in that direction.
• 14:14
  SPEAKER_01: We found that separating airflow environments by kind of business area made a lot more sense because then it was a lot easier to customize things like permissions and ensure that the right teams had the right level of access, ensure that our market data team is doing a lot of hefty data processing.
• 14:29
  SPEAKER_01: Their environment needs more resources versus maybe our operations and trading airflow environment that is doing a lot more simple type things and maybe doesn't need quite the same level of CPU and RAM and whatnot.
• 14:39
  SPEAKER_01: And what we discovered very early on, as we onboarded Airflow, we started utilizing the Airflow UI to support production, to test our DAGs and whatnot, we realized how just useful and easy it was to interact with these workflows.
• 14:53
  SPEAKER_01: And so kind of realized, hey, it would be really useful in order to give other teams, people who are maybe even non-technical users at the firm access to Airflow DAGs that kind of drive things that they are dependent on for doing their work.
• 15:08
  SPEAKER_01: So while we have a very strong technology segment and our research team is incredibly technical, we also have a lot of other teams that are not so technical.
• 15:17
  SPEAKER_01: Having said that, everybody at the company is incredibly interested in learning new things.
• 15:22
  SPEAKER_01: They're incredibly interested in learning new processes.
• 15:24
  SPEAKER_01: No one is really, everyone is eager to learn how to use a new system, especially if it's able to support their work more effectively.
• 15:32
  SPEAKER_01: And so what we're able to do is use things like DAG level permissions to give specific teams different levels of access to DAGs that are that they care about.
• 15:38
  SPEAKER_01: For instance, we give our trading team U-level access to a lot of DAGs on that market data environment that are ingesting data in the morning for our equities program, right?
• 15:49
  SPEAKER_01: Because our trading team needs to be aware of the status and the timeliness and when this data is available because they're responsible for some other dependent processes.
• 15:57
  SPEAKER_01: And so giving them view access means they no longer have to reach out to the market data team, the chat or a phone call.
• 16:03
  SPEAKER_01: If they see data's late, they can just look at the Airflow UI and they understand exactly what's going on and allows them to stay focused on the main things that they're supposed to be doing.
• 16:11
  SPEAKER_01: Similarly, we have teams like our accounting team that is responsible for looking a lot of reports and other things that we get from third parties.
• 16:18
  SPEAKER_01: And rather than we have a lot of processes in that Airflow environment that are just literally just pulling files for them and putting them in a location for them to be able to review and look at.
• 16:26
  SPEAKER_01: But anytime you're dependent on an external resource, things are bound to go wrong eventually.
• 16:31
  SPEAKER_01: The other company is delayed or the file's delayed.
• 16:34
  SPEAKER_01: And so we were able to give this accounting team access, not just view, but also the ability to rerun or uh run a DAG to download these files.
• 16:42
  SPEAKER_01: And it gives them this very like self-empowering opportunity for them to solve their own issues and problems without having to reach out to one of the engineering teams to say, hey, why can you rerun this thing for me?
• 16:53
  SPEAKER_01: They can just handle it themselves.
• 16:54
  SPEAKER_01: And so we're continuing to embrace that type of framework.
• 16:57
  SPEAKER_01: We want to make give as many teams as possible the opportunity to interact with the system, be able to solve their own problems or trigger their own workflows.
• 17:06
  SPEAKER_01: There are a few things that I have in mind for in the near future that will allow us to utilize things like the built-in Airflow manual DAG UI forms for inputting values, trigger specific workflows.
• 17:17
  SPEAKER_01: There's a lot of opportunity beyond everything what we've already implemented today.
• 17:21
  SPEAKER_02: So are you thinking of customizing the Airflow UI?
• 17:24
  SPEAKER_02: So different people will have, for example, a form to uh fill in order to do some work.
• 17:29
  SPEAKER_02: Is that an option for you, like customizing the UI or maybe creating uh custom operators as well?
• 17:35
  SPEAKER_02: Is that something that you are thinking about?
• 17:37
  SPEAKER_01: Yeah, we thought about doing both of those things.
• 17:39
  SPEAKER_01: We haven't gone down either route yet.
• 17:40
  SPEAKER_01: We try to use as much of the built-in airflow functionality as possible.
• 17:45
  SPEAKER_01: For instance, with the built-in custom UI or custom forms, I think we could probably get a lot of what we're looking for just by utilizing the DAG parames and binding those to the JSON data.
• 17:56
  SPEAKER_01: But if needs be, you know, we we 100% would build out customized U forms and UIs.
• 18:01
  SPEAKER_01: We definitely have the capabilities for doing so.
• 18:03
  SPEAKER_01: Same thing for custom operators.
• 18:04
  SPEAKER_01: There's a couple of use cases that we have that we've been considering writing some custom operators for doing different data validations and things like that.
• 18:11
  SPEAKER_02: Okay.
• 18:11
  SPEAKER_02: Sorry, you are mentioning data validation.
• 18:13
  SPEAKER_02: For data validation and data quality, are you using a framework today or do you run your own data quality checks?
• 18:20
  SPEAKER_02: Like how does it work if you can say a little bit more about that?
• 18:24
  SPEAKER_01: Yeah, and without getting into a ton of detail, but we do run, we don't use any external framework.
• 18:30
  SPEAKER_01: We have our own kind of internal validation tools.
• 18:33
  SPEAKER_01: And so that's one of those things where, so for instance, the market data team has worked really closely with our research department to build out internal packages and other things.
• 18:41
  SPEAKER_01: And that's another benefit of the internal package system, is that we're able to get a lot of code reuse.
• 18:46
  SPEAKER_01: And so the data team has built some very good general purpose data validation packages that we're able to then deploy and utilize within other business logic packages.
• 18:58
  SPEAKER_01: And so we're able to do different validations along price histories and those kinds of things to ensure that the data quality is there before we ever say persist it to any of our data stores.
• 19:07
  SPEAKER_01: And then we're able to generate customized alerts and other reports that out of the Airflow environment to the particular teams that may be interested in that data.
• 19:15
  SPEAKER_01: And because these are internal tools, but they're built in a general purpose way.
• 19:20
  SPEAKER_01: Like I said, we're able to use this across a lot of a lot of different DAGs, a lot of different data process workflows, and then we're able to slowly enhance and revise and deploy data versions of these things as appropriate and as we discover new things that we want to be able to do.
• 19:33
  SPEAKER_02: Yeah, I was asking because many engineers and many companies struggle today with data quality.
• 19:38
  SPEAKER_02: It's a hot topic.
• 19:39
  SPEAKER_02: I'm sure that that, especially with Gen AI, your data must be definitely uh good, otherwise your model won't be good.
• 19:46
  SPEAKER_02: Also, just maybe one last question.
• 19:49
  SPEAKER_02: You mentioned different ways of serving airflow for different teams.
• 19:54
  SPEAKER_02: But is it also something that you do when you write DAGs in your company?
• 19:59
  SPEAKER_02: For example, if I'm a data scientist and I want to write a DAG, do I have to follow a template for that?
• 20:05
  SPEAKER_02: Or can I just write my DAG right away?
• 20:07
  SPEAKER_02: Or did you put in place anything to make sure that, again, DAGs follow some requirements and so on?
• 20:12
  SPEAKER_01: Yeah, when I said that we have different flavors and whatnot, one of the things I was thinking about, this is slightly off of what you're asking, but then I can get back to that, is the mantra that we try to follow is that you try to keep things as simple as possible.
• 20:24
  SPEAKER_01: And in the case of the market data processes, generally these are things that are handling hefty data processing or a lot more customized business logic.
• 20:31
  SPEAKER_01: And so building these business logic packages is critical for enhancing speed of delivery and testability and all those kinds of things.
• 20:39
  SPEAKER_01: When we're talking something like our trading and operations Airflow instance, where it's just a matter of pulling a file and placing it on a specific network share for the accounting team, building something like a business logic package would be massive overkill.
• 20:52
  SPEAKER_01: And so we just build DAGs, we just got we generally build the code right into the DAG itself and maybe reuse some shared packages or other general purpose code that we've built internally, where we try to keep things kind of as simple and kind of top level as possible where applicable.
• 21:06
  SPEAKER_01: And then things like with our research infrastructure, Airflow is acting purely as that external scheduling tool.
• 21:11
  SPEAKER_01: So it's really just kicking off things that are running on a separate environment in our data center.
• 21:15
  SPEAKER_01: So going back to what you were asking, one of the things that's really been a side effect of business logic package process is that as a, so for instance, with a research team, they come in and say, hey, we have this new data request.
• 21:27
  SPEAKER_01: We need data to feed this project that we're working on.
• 21:30
  SPEAKER_01: Oftentimes we will have there's a lot of domain knowledge that's embedded within that research team.
• 21:34
  SPEAKER_01: They understand the data because they're building research code around that data.
• 21:39
  SPEAKER_01: And so we've actually had opportunities where a research engineer that's embedded in a research team has built the business logic package.
• 21:45
  SPEAKER_01: He's actually conformed to our business logic template and done it all in the end.
• 21:49
  SPEAKER_01: The majority of the work, since he's since that team's already working with the data, and then just turned it over to the market data team.
• 21:55
  SPEAKER_01: And then all they have to do is orchestrate it within a DAG, right?
• 21:58
  SPEAKER_01: So it adds a lot of parallelization.
• 22:00
  SPEAKER_01: To the work that's being done.
• 22:02
  SPEAKER_01: But when it comes down to actually building a DAG, yes, we would expect things to conform to a specific template.
• 22:06
  SPEAKER_01: All somebody with airflow expertise at the company would always be on a code review for a DAG summed by anyone to ensure that it's following and conforming to certain expectations, the appropriate use of tags, appropriate types of triggers, appropriate scheduling intervals, those kinds of things.
• 22:23
  SPEAKER_01: For instance, we ensure I've made it a big point that we should be very clear whether our DAGs are idempotent or not idempotent, because that's something that many years ago would cause major issues when we were managing these things in a production environment.
• 22:36
  SPEAKER_01: And so, you know, having that all built in the airflow lines of making it extremely clear and easy to deal with when you know that this process is not going to be idempotent and you have to be mindful of that versus, you know, I can run this thing 50 times and it's always going to look the same as long as it's working correctly.
• 22:51
  SPEAKER_02: That's a very good one.
• 22:53
  SPEAKER_02: That's a very good one.
• 22:54
  SPEAKER_02: You absolutely need to make sure that your data pipelines are identity.
• 22:57
  SPEAKER_02: And if they are not, you better know that because otherwise you might have a lot of struggle.
• 23:02
  SPEAKER_02: Yeah, I wish they all work with what you got.
• 23:06
  SPEAKER_02: True.
• 23:06
  SPEAKER_02: Thank you so much for showing that.
• 23:08
  SPEAKER_02: Okay, last question.
• 23:09
  SPEAKER_02: What would you like to see in the next Airflow version?
• 23:12
  SPEAKER_02: Like if there is any feature that you would like to see coming or any improvements, anything that will make your life easier with Airflow?
• 23:21
  SPEAKER_01: Sure.
• 23:22
  SPEAKER_01: I mean, this is going to go back to a lot of what we've been talking about, but just continuing to enhance the ability to test Airflow DAGs would make it close that gap.
• 23:31
  SPEAKER_01: We have a very reliable method for testing these things, testing our processes today, up until the point where, you know, once they're in Airflow, it's just a matter of running the DAGs and validating and looking at logs and results and those kinds of things.
• 23:43
  SPEAKER_01: Being able to test that entire system in a very automated, a repeatable way would be really useful.
• 23:49
  SPEAKER_01: I was thinking about a little bit, and we obviously when our unit tests, we heavily utilize things like Python mocks to control external dependencies.
• 23:56
  SPEAKER_01: Being able to do something like that in Airflow could be really powerful, I think, to be able to reliably run an Airflow DAG from start to finish and be able to validate the results.
• 24:05
  SPEAKER_01: Then for you know, our use case, we would potentially make a change to the business logic package.
• 24:09
  SPEAKER_01: We could run a mocked Airflow DAG and validate that the business logic change had no works as expected, or vice versa.
• 24:15
  SPEAKER_01: If we need to make a change to the DAG, we can ensure that it still interacts and works with our business logic package appropriately.
• 24:21
  SPEAKER_01: So with that, I'm gonna add one more, it's just general.
• 24:23
  SPEAKER_01: I think continuing to focus on UI enhancements, I think is is gonna be really for us as we continue to give more users access to Airflow environments, especially non-technical people, making it easier for them to see what they want to see, filter, search, access the UI in a way that is non-intimidating will be really important.
• 24:42
  SPEAKER_02: I fully agree with you.
• 24:44
  SPEAKER_02: I think those features will be amazing.
• 24:45
  SPEAKER_02: Testing DAGs and tasks has been a pretty difficult and it is still a difficult topic in Airflow because of all the things that you need to manage in your DAGs and external services is one thing, as you said.
• 24:57
  SPEAKER_02: That's why you need to mock those.
• 24:59
  SPEAKER_02: And also, as you said, the UI announcements, improvements for sure.
• 25:03
  SPEAKER_02: Like I think at the like people that were using Airflow were mostly data engineers, but now it is growing and you have different profiles, different backgrounds that use Airflow today, and definitely some improvements on the UI to make it uh easier to use, would be a great thing to have.
• 25:18
  SPEAKER_02: Awesome.
• 25:18
  SPEAKER_02: Thank you again, like Larry.
• 25:20
  SPEAKER_02: Thank you for joining truly.
• 25:22
  SPEAKER_02: And what is the best way for uh people to get in touch with you?
• 25:27
  SPEAKER_01: Yep.
• 25:27
  SPEAKER_01: So if anybody wants to get in touch with me, they can just email me directly.
• 25:31
  SPEAKER_01: My email is Larry.comenda at Campbell.com.
• 25:34
  SPEAKER_01: And then if anyone listening wants to learn more about Campbell, just I would suggest just go to www.campbell.com.
• 25:40
  SPEAKER_01: We actually have that domain.
• 25:41
  SPEAKER_01: It's very easy to find us.
• 25:42
  SPEAKER_01: And so there's a lot of good information on the site about who we are and what we do.
• 25:45
  SPEAKER_02: That's truly awesome.
• 25:46
  SPEAKER_02: Thank you so much, Larry.
• 25:47
  SPEAKER_02: Have a great day, and I hope to see you very soon.
• 25:50
  SPEAKER_00: Thanks, Mark.
• 25:50
  SPEAKER_02: Thank you.
• 25:52
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 25:58
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 26:02
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.