Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Exploring the Power of Airflow 3 at Astronomer with Amogh Desai

Transcript

Exploring the Power of Airflow 3 at Astronomer with Amogh Desai · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: To understand some mechanics of Airflow, we should be aware that Airflow doesn't do Jira.
• 0:05
  SPEAKER_01: Like some of the other Apache projects, we don't do Jira.
• 0:08
  SPEAKER_01: All of our issues, problems, bug reports, everything lies on the GitHub project itself under the issues tab.
• 0:14
  SPEAKER_01: So this is the one and only mechanics that we need to know with respect to Airflow.
• 0:19
  SPEAKER_01: As I said, we follow GitHub issues, right?
• 0:21
  SPEAKER_01: When we do that, we usually, as maintainers, we go through the list on every now and then and we add a good first issue label to the issues that can be solved by beginners.
• 0:37
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:46
  SPEAKER_03: Hello everyone, welcome to the Data Flowcast.
• 0:50
  SPEAKER_03: I'm your host, Mark Lometti, and today I'm joined by Amog Desai, software engineer ad astronomer.
• 0:56
  SPEAKER_03: I'm very excited.
• 0:57
  SPEAKER_03: How are you doing?
• 0:58
  SPEAKER_03: Yeah, I'm doing good.
• 1:00
  SPEAKER_03: How's your day going?
• 1:01
  SPEAKER_03: Doing great, doing great.
• 1:03
  SPEAKER_03: I'm really excited to have you in this in this episode because today we are going to discover what is it to begin with Airflow?
• 1:11
  SPEAKER_03: What was your journey?
• 1:12
  SPEAKER_03: And more importantly, how did you start contributing to Airflow?
• 1:17
  SPEAKER_03: What kind of challenges did you have along the way?
• 1:19
  SPEAKER_03: So I'm very excited because that will be helpful for anybody that wants to uh contribute to Airflow too.
• 1:26
  SPEAKER_03: Um so let's start with some background.
• 1:29
  SPEAKER_03: What did you do actually before joining Astronomer?
• 1:32
  SPEAKER_03: Because right now I know that you are a senior software engineer at Astronomer, but what did you do before?
• 1:38
  SPEAKER_01: So hi everyone again.
• 1:39
  SPEAKER_01: I'm very happy to be here.
• 1:41
  SPEAKER_01: Thanks for having me, Mark.
• 1:42
  SPEAKER_01: So I'm Amok Desai, I'm a committer at Apache Airflow, and I'm also working as a senior software engineer at Astronomer.
• 1:50
  SPEAKER_01: Uh so currently I'm working on uh different parts and AIPs of Airflow 3 so that we are looking at a very good Airflow 3 release.
• 1:58
  SPEAKER_01: So that's what I'm doing right now.
• 2:00
  SPEAKER_01: But before joining Astronomer, I was still in corporate.
• 2:04
  SPEAKER_01: I was uh working as a senior software engineer in Cloudera for about four years.
• 2:09
  SPEAKER_01: Yeah, and mainly uh at Cloudera, I was working on building a data engineering platform which was on Kubernetes, and my area of focus was mainly towards the private cloud.
• 2:20
  SPEAKER_01: So I was looking at somewhere around Red Hat, OpenShift, and related distributions like Rancher, etc.
• 2:26
  SPEAKER_01: So yeah, my day-to-day that time involved building and building, deploying, and testing features for this product known as Cloudula Data Engineering.
• 2:33
  SPEAKER_01: And we had a very dynamic team uh in Bangalore and uh India.
• 2:38
  SPEAKER_01: So uh we used to yeah build build the product uh from there.
• 2:42
  SPEAKER_01: And yeah, we offered managed airflow as well as Spark in our product, and uh, due to that I got a very brief familiarity with Airflow before actually diving in into Airflow.
• 2:55
  SPEAKER_03: Okay, so this is interesting because I didn't know that Claude had a managed airflow service.
• 3:02
  SPEAKER_01: Uh we do have it.
• 3:04
  SPEAKER_01: There's a uh we have a line of platform known as Cloudera Data Engineering, but we offer Airflow as a uh service within the same data service or data uh experience, if you will.
• 3:16
  SPEAKER_03: Okay.
• 3:16
  SPEAKER_03: So for some reason I I thought that it was only uh Spark, but okay.
• 3:21
  SPEAKER_03: That's good.
• 3:22
  SPEAKER_03: That's cool.
• 3:23
  SPEAKER_03: So can you tell us more about your journey with Airflow, when and how everything started for you?
• 3:28
  SPEAKER_01: Yeah, that's a very it's a very good question and it's a very random answer.
• 3:32
  SPEAKER_01: So yeah, this was back in January of uh 2023, uh which is last year, when I reached out to one of my then directors in Cloudera regarding my interest to contribute to open source, not particularly Airflow back then, but I wanted to contribute to open source software in general.
• 3:50
  SPEAKER_01: And the driving factor for me that time was to set a brand value for myself outside my uh day-to-day uh working job, right?
• 3:59
  SPEAKER_01: So yeah, I reached out to the director.
• 4:01
  SPEAKER_01: We did a little bit of brainstorming, we had some chats, and yeah, once we uh discussed things around this, we realized that two projects are standing out Apache Spark and Apache Airflow.
• 4:11
  SPEAKER_01: And we were actually only looking at the Apache ecosystem in general, not looking at other ecosystems like CNCF, etc.
• 4:18
  SPEAKER_01: Reason being we had people in the company that time who were very familiar with Apache, we would have a better guidance.
• 4:24
  SPEAKER_01: So we looked at these two projects and on comparing them, I will come to that later.
• 4:29
  SPEAKER_01: We compared a bit and we ended up picking Airflow.
• 4:32
  SPEAKER_01: So once we picked Airflow, we started having these weekly sync ups where we were just discussing progress in general.
• 4:39
  SPEAKER_01: Uh, and we were just doing it on a almost a weekly basis where we used to uh chat about how is it going, right?
• 4:46
  SPEAKER_01: In general, are you blogged at something?
• 4:48
  SPEAKER_01: Are you not able to proceed?
• 4:49
  SPEAKER_01: And uh things like that.
• 4:50
  SPEAKER_01: And uh this was a very interesting phase for me because uh this was something that uh kept me motivated because every week I have to show up to that meeting and I have to be like, oh, I don't have many problems, I'm doing okay.
• 5:01
  SPEAKER_01: So that kind of so that's where we started.
• 5:04
  SPEAKER_01: Then slowly I started uh getting involved in uh different parts of Airflow over this course of I would say six to eight months.
• 5:12
  SPEAKER_01: So I started with uh doc-related fixes, fixing links on the website, hanging on to the Airflow site repo, fixing some back references, STMLs, things like that.
• 5:22
  SPEAKER_01: And slowly I moved on to Helm charts because my work experience involved a bit of Helm charts, so I was already comfortable with it.
• 5:29
  SPEAKER_01: So I thought, let me try and help with Airflow Helm charts.
• 5:32
  SPEAKER_01: So started getting into that, and that opened a path for me into the CNCF provider that we have.
• 5:38
  SPEAKER_01: So I started contributing and reviewing PRs and helping with the CNCF provider as well.
• 5:44
  SPEAKER_01: And eventually I thought I am using Breeze on a day-to-day basis.
• 5:48
  SPEAKER_01: It's a development tool which is helping me get PRs in into the Airflow repo.
• 5:53
  SPEAKER_01: So why don't I contribute to Breeze?
• 5:55
  SPEAKER_01: That was my thought.
• 5:56
  SPEAKER_01: And then when I looked into Breeze, it seemed like a very interesting project.
• 6:00
  SPEAKER_01: So I started contributing to Breeze, and I do not know many people who are active contributors to Breeze as of today.
• 6:06
  SPEAKER_01: So I got into Breeze as well, and I've been fixing a lot of things around Breeze, both bugs, features, and different asks from release manager and uh things like that for Breeze.
• 6:19
  SPEAKER_01: So the general theme was I was keeping myself active and visible in the community.
• 6:24
  SPEAKER_01: I was doing this by testing all the RC bits that came out on the emails.
• 6:28
  SPEAKER_01: Then I was involving in lazy consensus, discussions, any kind of voting, any kind of AIPs that were out there.
• 6:35
  SPEAKER_01: I was helping to review it.
• 6:36
  SPEAKER_01: And these things happened.
• 6:38
  SPEAKER_01: And during this period, I came to know this something known as the Airflow Summit that happens uh every year.
• 6:43
  SPEAKER_01: So I tried my go at it.
• 6:45
  SPEAKER_01: I submitted a I submitted a topic as a as one of the talks.
• 6:49
  SPEAKER_01: And fortunately, I was selected as well, so it was even more exciting now.
• 6:54
  SPEAKER_01: So as time passed, came September 2023, and that is just before the Airflow summit, I was invited to become a committer.
• 7:02
  SPEAKER_01: So that was a very happy moment for me, and I accepted it.
• 7:06
  SPEAKER_01: I'm here today, a committer for Airflow.
• 7:08
  SPEAKER_01: So this was my journey, and yeah, a small, short but fun packed one.
• 7:13
  SPEAKER_03: Yeah, I mean that that is incredible.
• 7:15
  SPEAKER_03: So actually, we need to step back a little bit.
• 7:18
  SPEAKER_03: I would like to start by asking you did you find it hard to getting started with airflow?
• 7:28
  SPEAKER_03: Did you feel that I don't know, like the learning curve of of starting airflow was steep or not?
• 7:35
  SPEAKER_01: Not really, it was not very steep.
• 7:37
  SPEAKER_01: Uh I had a generally uh there are two aspects to this, right?
• 7:41
  SPEAKER_01: One is understanding airflow in itself, one is contributing to airflow in its airflow.
• 7:45
  SPEAKER_01: So two parts to it.
• 7:46
  SPEAKER_01: Uh airflow in itself, I had a I had a decent level understanding because we offered it as uh part of our product, right?
• 7:53
  SPEAKER_01: So I had a decent understanding of what airflow is and things like that.
• 7:57
  SPEAKER_01: Concepts were good, obviously your tutorials also.
• 8:00
  SPEAKER_01: So that that way I was fine, but contributing uh is usually the harder part where the steep learning curve comes in.
• 8:07
  SPEAKER_01: So yeah, that is not what I faced with airflow.
• 8:09
  SPEAKER_01: It was seamless, I would say.
• 8:12
  SPEAKER_01: Um the initial roadblocks were very minimum, but it was it was there obviously, it has improved much more now, but generally there were very few roadblocks than what I expected in the early days.
• 8:24
  SPEAKER_03: Okay.
• 8:25
  SPEAKER_03: Now that's that's good to know because because like some people believe that airflow is hard to get started with, where I think it is not.
• 8:34
  SPEAKER_03: It is like a pretty it can be complex because you have many features, but if you actually focus on what you need at first, I think it is pretty easy to start with Airflow.
• 8:44
  SPEAKER_03: But focusing on the contributing on the contributing part, can you tell us more about what was your first contribution to Airflow?
• 8:52
  SPEAKER_03: And also, you actually mentioned that you you picked Airflow over Spark.
• 8:56
  SPEAKER_03: So then it would be interesting to know more about that.
• 8:59
  SPEAKER_03: Like why did you decide to use uh yeah, first contribution, right?
• 9:04
  SPEAKER_01: That's a very interesting story.
• 9:06
  SPEAKER_01: So yeah, so I want I as I decided to pick Airflow, I'll uh tell you why.
• 9:11
  SPEAKER_01: But the story is very interesting.
• 9:13
  SPEAKER_01: What happened when I tried to contribute first?
• 9:16
  SPEAKER_01: So I was trying to set up Airflow because I wanted to contribute to it.
• 9:19
  SPEAKER_01: And when I was setting it up locally, I noticed that on the website a few links were uh mismatching with the content it serves versus its use usage was not matching very well.
• 9:31
  SPEAKER_01: So this looked like something I could fix, and I was really excited that oh, I found an issue, let me fix it.
• 9:37
  SPEAKER_01: I'll probably get my first commit in.
• 9:38
  SPEAKER_01: And I was very pumped about it.
• 9:40
  SPEAKER_01: So out of excitement to do this, I went ahead, I did all the things like cloning and creating a branch and finally writing code and pushing it because it it was barely like two lines or three lines of a PR.
• 9:53
  SPEAKER_01: So I thought I fixed a link, let me push it, somebody will approve it, merge it.
• 9:58
  SPEAKER_01: So I thought it's that easy.
• 9:59
  SPEAKER_01: Uh but it was funny here because I erased the pull request and one of the maintainers, Jarek, he requested changes on it.
• 10:08
  SPEAKER_01: I'm like, what?
• 10:09
  SPEAKER_01: How is it possible?
• 10:10
  SPEAKER_01: It's a link.
• 10:11
  SPEAKER_01: So this happened, and he mentioned he had a very valid reason.
• 10:15
  SPEAKER_01: I had just replaced with the actual website link, but he told me to replace it with something known as the Spinx directive.
• 10:23
  SPEAKER_01: That is a tool we use to uh build docs for Airflow.
• 10:26
  SPEAKER_01: But I little did I know that time I did not know what Spinx was.
• 10:30
  SPEAKER_01: So I asked him back on the I asked, asked him back what is Spinx, how do I use it, things like that.
• 10:37
  SPEAKER_01: Very basic questions on the pull request comment itself.
• 10:39
  SPEAKER_01: He was very patient.
• 10:40
  SPEAKER_01: He pasted a couple of links explaining to me what it is, how we use it, and things like that.
• 10:46
  SPEAKER_01: So I followed that.
• 10:47
  SPEAKER_01: I eventually made the fix and I pushed it again.
• 10:50
  SPEAKER_01: And he told me there's something called a build that runs on your uh pull request.
• 10:55
  SPEAKER_01: So that'll so he has he told me that he has approved it, and he approved it for the very very first contributor.
• 11:02
  SPEAKER_01: Somebody else has to approve it because otherwise you can it can lead to security issues where you can create trash PRs and it can consume resources, things like that.
• 11:11
  SPEAKER_01: So he told me he approved it, but he told me meanwhile, can you please try testing it locally?
• 11:16
  SPEAKER_01: And I told him I don't know how to do that either.
• 11:19
  SPEAKER_01: So he pasted, he he told me what breeze is, he told me how to set it up, and he gave me a couple more informative medium articles and a few more helpful links.
• 11:29
  SPEAKER_01: I followed that and I told him I've tested it, it works fine.
• 11:33
  SPEAKER_01: So yeah, this is uh what I expect, and this is exactly what it's uh doing.
• 11:37
  SPEAKER_01: Once I did that, I went, I closed the laptop for the day and it was late for me, so I just logged off.
• 11:43
  SPEAKER_01: And then I came back the next morning and I saw that Jarek has merged it.
• 11:47
  SPEAKER_01: I'm like, wow, he merged a pull request, and just like that, I was like, wow, I should do this more often because this whole process took just one day.
• 11:55
  SPEAKER_01: And I got my first ever contribution for Airflow.
• 11:58
  SPEAKER_01: I was really happy about it.
• 12:00
  SPEAKER_01: This was the story behind my first PR.
• 12:03
  SPEAKER_01: Now, coming to why I picked Airflow, right?
• 12:06
  SPEAKER_03: Uh so I I just I just want to add one thing because this is a great story that also that that shows that contributing to Airflow and actually contributing to any project, it doesn't have to be big.
• 12:18
  SPEAKER_03: Like it doesn't have to be uh like a huge thing to fix.
• 12:22
  SPEAKER_03: No, it can be as simple as a link, and that will still help the community.
• 12:26
  SPEAKER_03: So that that is actually a great example because just by fixing a link, you learn a lot along the way.
• 12:32
  SPEAKER_03: And also, I I want to give a big shout out to uh to Yarek because we know that he does a wonderful job to uh bring more and more people to the Airflow community.
• 12:42
  SPEAKER_01: So tell me more about the like why did you pick Airflow or Spark?
• 12:45
  SPEAKER_01: Yes, I'm curious.
• 12:46
  SPEAKER_01: Yes, so why did I pick Airflow in the first place?
• 12:49
  SPEAKER_01: So uh as I mentioned earlier, I had two choices.
• 12:51
  SPEAKER_01: I had the liberty to choose between uh Spark and Airflow mainly stood out for me because of two reasons.
• 12:58
  SPEAKER_01: One was it was not as mature and old as Spark.
• 13:01
  SPEAKER_01: That meant that there is a little more room to contribute compared to Spark because it's evolving, and that was one of the reasons.
• 13:08
  SPEAKER_01: And second reason was the community.
• 13:10
  SPEAKER_01: I had just subscribed to the dev email, and I could see a lot of emails coming in and a lot of activity and uh conversations happening.
• 13:18
  SPEAKER_01: So it seemed like a very vibrant and welcoming community.
• 13:21
  SPEAKER_01: The other reason was again Yarek.
• 13:23
  SPEAKER_01: So uh the cup a couple of months before I started uh doing any of this, we had a uh mini meetup at our office in Bangalore, and Yarek was he had given a guest speech, which was again about how to contribute to airflow.
• 13:37
  SPEAKER_01: So once I I was interested that time, but I didn't really uh do anything for three months after that.
• 13:43
  SPEAKER_01: But I got the interest then, and then once I started doing, I realized okay, there's someone who can help me actually get into this path easier or in a much more invited manner.
• 13:53
  SPEAKER_01: So that also uh really motivated me to pick Airflow.
• 13:58
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 14:07
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 14:19
  SPEAKER_00: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
• 14:28
  SPEAKER_03: Airflow has been around for a while now, but there is definitely still a lot of things to do, which is great.
• 14:34
  SPEAKER_03: It's it's a really fast-growing project.
• 14:37
  SPEAKER_03: And yeah, you can feel that Spark is actually it is even older than Airflow.
• 14:41
  SPEAKER_03: And to that sense, you can feel that Spark is more mature than Airflow.
• 14:45
  SPEAKER_03: But um, but yeah, to your point, yes, I agree.
• 14:48
  SPEAKER_03: Like the Airflow community uh is great.
• 14:51
  SPEAKER_03: Okay, so I guess let's imagine that I actually want to contribute to Airflow.
• 14:58
  SPEAKER_03: So, what should I do first?
• 14:59
  SPEAKER_03: Should I start with an issue?
• 15:01
  SPEAKER_03: Because I know that there are like some issues.
• 15:03
  SPEAKER_03: Should I can you work me through the typical process, like step by step of what it means to be a contributor?
• 15:10
  SPEAKER_03: Starting with I guess like an issue.
• 15:14
  SPEAKER_01: Yeah, why not?
• 15:15
  SPEAKER_01: So yeah, I'll be really happy if I I motivate you to come over to Airflows.
• 15:19
  SPEAKER_01: So yeah, you should, yeah.
• 15:22
  SPEAKER_01: So the first coming to the very initial part of the story, how do you uh really start?
• 15:27
  SPEAKER_01: The initial part is you need to find something to fix, obviously.
• 15:31
  SPEAKER_01: So the first part of the story will obviously be finding an issue to work on.
• 15:34
  SPEAKER_01: So this can be really challenging for beginners and maybe overwhelming too, but to understand uh some mechanics of Airflow, we should be aware that Airflow doesn't do Jira.
• 15:44
  SPEAKER_01: Like some of the other Apache projects, we don't do Jira.
• 15:47
  SPEAKER_01: All of our issues, problems, bug reports, everything lies on the GitHub project itself under the issues tab.
• 15:54
  SPEAKER_01: So this is the one and only mechanics that we need to know with respect to Airflow.
• 16:00
  SPEAKER_01: Yeah, as I said, we follow GitHub issues, right?
• 16:02
  SPEAKER_01: When we do that, we usually, as maintainers, we go through the list on every now and then and we add a good first issue label to the issues that can be solved by beginners.
• 16:13
  SPEAKER_01: By that, these issues do not require a very much in-depth knowledge of airflow, and this is probably something worth fixing, and this is worth fixing by a beginner as well.
• 16:23
  SPEAKER_01: So these are some really good issues to start with.
• 16:25
  SPEAKER_01: So when you see such issue or such an issue, you can just go to GitHub and you can filter out by that particular label.
• 16:31
  SPEAKER_01: Uh, it's called good hyphen first hyphen issue.
• 16:34
  SPEAKER_01: So once you do that, you'll get a uh big list of them.
• 16:38
  SPEAKER_01: So just to make sure that you're not stealing someone's issue or your multiple people working on the same thing, it's always a good idea to have something assigned to you.
• 16:47
  SPEAKER_01: And how do you do that?
• 16:48
  SPEAKER_01: Just go to the issue and comment.
• 16:50
  SPEAKER_01: I would like to work on this or I'm interested.
• 16:52
  SPEAKER_01: Can you please let me work on this?
• 16:54
  SPEAKER_01: Things like that.
• 16:54
  SPEAKER_01: Basically, the idea is to get that issue assigned to you.
• 16:58
  SPEAKER_01: Now, how does that happen?
• 16:59
  SPEAKER_01: You comment and then one of the maintenance will acknowledge it.
• 17:02
  SPEAKER_01: He or she will either assign it to you or they will tell, okay, go for it.
• 17:06
  SPEAKER_01: Things like that.
• 17:07
  SPEAKER_01: So once you have the green signal from them, what you can do is you can obviously start working on it.
• 17:13
  SPEAKER_01: But you might have a few questions, right?
• 17:16
  SPEAKER_01: How do I how do I do this?
• 17:19
  SPEAKER_01: Let's say I got an issue assigned to me.
• 17:21
  SPEAKER_01: Now, how do I uh basically fix this?
• 17:23
  SPEAKER_01: Or how do I have a doubt related to it?
• 17:25
  SPEAKER_01: What do I do about it?
• 17:27
  SPEAKER_01: So the idea is you can feel free to ask about that issue, uh, any random questions about that issue on the issue itself as a comment, or you can feel free to uh reach out to the maintainers on uh Slack as well.
• 17:39
  SPEAKER_01: That's the messaging platform we use for Airflow, right?
• 17:42
  SPEAKER_01: So feel free to do that.
• 17:44
  SPEAKER_01: And yeah, it's part of the maintenance and it's part of the committer's PMC duty to also help new contributors.
• 17:50
  SPEAKER_01: So they'll be very happy to guide you.
• 17:52
  SPEAKER_01: So yeah, first part done.
• 17:54
  SPEAKER_01: You have a we you have an issue now.
• 17:56
  SPEAKER_01: Now the next step will obviously be how do I uh work on it.
• 17:59
  SPEAKER_01: So, first for that, you need a development environment setup.
• 18:02
  SPEAKER_01: So to have a dev and environment setup, you can uh use one of the few methods that Airflow can be developed in.
• 18:10
  SPEAKER_01: So we have either you can use a local virtual environment by Python to install Airflow in it and develop on that, or we have something known as Breeze.
• 18:18
  SPEAKER_01: So Breeze is basically a development tool made to develop Airflow.
• 18:23
  SPEAKER_01: And this is based on a Docker Compose config.
• 18:27
  SPEAKER_01: And this the Docker files that we use here are the exact same ones we use in the CI.
• 18:32
  SPEAKER_01: And the reason for that is if there is any failure in the CI, it's very easy to uh reproduce it locally.
• 18:38
  SPEAKER_01: And you can once you reproduce locally, you can obviously fix it.
• 18:41
  SPEAKER_01: So it avoids the it works for me uh syndrome, right?
• 18:44
  SPEAKER_01: So two choices you can pick any of that.
• 18:49
  SPEAKER_01: And how do you set up any of this?
• 18:51
  SPEAKER_01: We do have a very short document called Contributor's Quick Guide.
• 18:55
  SPEAKER_01: It's in the repository itself.
• 18:57
  SPEAKER_01: So that document is it's good enough for to be able to provide you a very minimal, stable, and a very effective development environment which you can start developing airflow on without uh diving into too much of internal details, which you don't need at that point in time.
• 19:14
  SPEAKER_01: So, yeah, once you're there, it's very simple.
• 19:16
  SPEAKER_01: After that, you fix your issue, you write the code for it, or you fix the link or whatever needs to be done and make the fix, test it either using Breeze or something like that, and then create a pull request.
• 19:28
  SPEAKER_01: So once you have a pull request, so ideally you should wait for uh reviews on it.
• 19:34
  SPEAKER_01: One of the maintainers will jump in and they will uh help you with the review.
• 19:37
  SPEAKER_01: But obviously, you can ping them on Slack also, but uh be mindful about one thing though that open source software is generally it's done by people in their free time and they're unpaid volunteers for it.
• 19:49
  SPEAKER_01: Be a little respectful of their time and keeping that in mind accordingly, nudge them or just give them a send them a message that okay, I have a PR, can you please help me with a review?
• 19:58
  SPEAKER_01: Something of that sort.
• 20:00
  SPEAKER_01: And one last step here would be for code changes.
• 20:02
  SPEAKER_01: Usually we have to write unit tests.
• 20:05
  SPEAKER_01: So unit tests are expected to be written.
• 20:08
  SPEAKER_01: So one tip I will give you is that become good friends with your debugger, with your IDE debugger or whatever debugger you use.
• 20:14
  SPEAKER_01: It's going to be very helpful for you, not just for writing unit tests, but also for learning airflow better.
• 20:20
  SPEAKER_01: Because when you add breakpoints and you step over things, step out, you get to know the paths of code that your uh uh your code is touching or your test is touching, which gives you much more awareness of what's actually happening under the hood.
• 20:35
  SPEAKER_01: So I think that's yeah, that's all.
• 20:37
  SPEAKER_01: These are the steps.
• 20:38
  SPEAKER_01: That's a good point.
• 20:39
  SPEAKER_03: Okay, so I think like now I'm I I think I'm ready now for contributing to Airflow.
• 20:44
  SPEAKER_03: One last question.
• 20:44
  SPEAKER_03: So when you said make a PR, so I guess should I commit and push on a specific branch or yeah, that that part, right?
• 20:51
  SPEAKER_01: Yeah, so usually fork it, the repository Apache slash Airflow, just fork it into your your account, uh, whatever your GitHub account is.
• 20:59
  SPEAKER_01: So then basically the idea is to push to your own fork.
• 21:03
  SPEAKER_01: And you can create a branch out of you can clone the repo locally, create a branch, and then push the branch to your own fork.
• 21:11
  SPEAKER_03: Okay.
• 21:12
  SPEAKER_01: And once you do that, you anyway, you wouldn't have access to push to the master anyway of Apache.
• 21:17
  SPEAKER_01: So you push to your fork, and then you'll get a notification to create a pull request against the main of Apache slash Airflow.
• 21:25
  SPEAKER_01: So all the pull requests are created against main always.
• 21:28
  SPEAKER_03: Okay, here we go.
• 21:29
  SPEAKER_03: Okay, awesome, awesome.
• 21:31
  SPEAKER_03: That's very I think that is the actually best explanations that I got so far on contributing to Airflow.
• 21:37
  SPEAKER_03: So thank you so much, yeah.
• 21:38
  SPEAKER_03: I mean it, I mean it.
• 21:40
  SPEAKER_03: Um okay, so I guess like I have to ask, because actually I know that many people do that, like they contribute to open source projects because they know that to your point, I think you said that at the beginning, it adds something on your resume, uh, for sure.
• 21:56
  SPEAKER_03: Do you feel that I think this one is obvious, but do you feel that contributing to Airflow helped you getting a job at astronomer.
• 22:03
  SPEAKER_01: Yes, of course.
• 22:04
  SPEAKER_01: Yes.
• 22:04
  SPEAKER_01: The reason being, as I mentioned earlier, also, contributing to open source is a resume in itself.
• 22:10
  SPEAKER_01: Because when you are contributing to open source, all your work, be it your code, your pull requests, your comments, your discussions on forums, your emails, everything is online.
• 22:22
  SPEAKER_01: So the people who want to see can see everything about you.
• 22:26
  SPEAKER_01: So that basically that those kind of interactions with these people whom you're targeting to eventually join, or even if you don't want to join someone, you just want to proceed in a certain track.
• 22:36
  SPEAKER_01: It's really good because when you have these interactions, uh these slowly build into a meaningful relationship with different stakeholders and they will open it'll open up opportunities such as this one.
• 22:47
  SPEAKER_01: So it can always be the case.
• 22:49
  SPEAKER_01: And yeah, the other thing here, the other aspect of it, when you're interviewing, let's say you get some kind of opportunity.
• 22:56
  SPEAKER_01: So in my case, contributing to Airflow also made my interviews with the hiring team much more valuable for both myself as well as for them.
• 23:04
  SPEAKER_01: Because we were now relating at the project level.
• 23:07
  SPEAKER_01: We were talking about Airflow, we were talking about Apache Airflow in general and not just general science fundamentals like most interviews are.
• 23:15
  SPEAKER_01: So we were relating much more uh project-oriented interview.
• 23:20
  SPEAKER_01: And this is very important from your career point of view as well, because when you're interviewing for such a role, right?
• 23:26
  SPEAKER_01: You will be you're getting a North Star of what you'll be working on.
• 23:30
  SPEAKER_01: So it'll also make your day-to-day very much more it'll make it much more exciting, and you can always be in a very uh forward-looking motion because of what's there for you in store.
• 23:40
  SPEAKER_03: That's a very good point.
• 23:42
  SPEAKER_03: That's a very good advice.
• 23:43
  SPEAKER_03: Yeah, I hope that at some point like companies will tend to hire people based on what they do on open source projects rather than on what they can do with uh lead code, uh like all of those kind of brainstorm algorithm uh exercises.
• 24:03
  SPEAKER_03: Because to your point, I think contributing to an open source project is so more meaningful, and you actually see how you as a person interact interacts with others, right?
• 24:23
  SPEAKER_03: Great.
• 24:23
  SPEAKER_03: I guess is there any recommendations you would like to share for people that want to contribute?
• 24:28
  SPEAKER_03: Like maybe some last recommendations.
• 24:30
  SPEAKER_01: Oh, yes, I do have a couple of them that I can think of.
• 24:34
  SPEAKER_01: Yeah, from my experience and with interacting with uh different people in the community, I do have a couple of things that I want to share for anyone who's looking to contribute to Airflow.
• 24:43
  SPEAKER_01: So yeah, the first one, very first one would be I've been in a place where I always want to get big things done.
• 24:50
  SPEAKER_01: Uh like uh big PRs and big features and everything that looks big, right?
• 24:57
  SPEAKER_01: In general when it comes to contributions.
• 25:00
  SPEAKER_01: But that's the thing.
• 25:01
  SPEAKER_01: If you want to work on uh bigger items, you obviously want to start small.
• 25:05
  SPEAKER_01: So it will not work out otherwise with that attitude because you will eventually get overwhelmed and you will give up.
• 25:11
  SPEAKER_01: So always my advice would be to start small.
• 25:14
  SPEAKER_01: So the idea is to start small and keep going small, keep making smaller contributions until you're very comfortable with the project, its processes, its nuances, and different things around the project.
• 25:26
  SPEAKER_01: And once that is done, you don't really have to look at big things because it'll eventually start coming to you.
• 25:31
  SPEAKER_01: The things you will be doing for the project will be much bigger than you uh doing something, let's say a couple of months back when you were uh ramping up.
• 25:38
  SPEAKER_01: So ideally, you should I would say start with documentation or dev environment issues, the initial few PRs, right?
• 25:46
  SPEAKER_01: I would suggest to be in one of these areas because if you're doing documentation, that means obviously you're reading documentation.
• 25:53
  SPEAKER_01: You're getting some knowledge out of that.
• 25:55
  SPEAKER_01: And the second thing is development environment is something that you're going to use on a day-to-day basis.
• 26:00
  SPEAKER_01: So getting very familiar with it is something that will uh help you in a long run.
• 26:05
  SPEAKER_01: And the more you experiment and experience it, the more you will benefit when it comes to contributing in a in a longer uh run.
• 26:13
  SPEAKER_01: So it's going to be your so your developer environment is your best friend and debugger is your second best friend.
• 26:18
  SPEAKER_01: So stay with them.
• 26:20
  SPEAKER_01: That's the thing.
• 26:21
  SPEAKER_01: Uh then yeah, if you ever find yourself in a situation where things are not working for you, but it's working for someone else when it comes to dev environment or something similar.
• 26:30
  SPEAKER_01: Ask questions.
• 26:32
  SPEAKER_01: But before that, experiment and experience it so that you can tell, you can ask that I have tried these many things, it's not working out.
• 26:38
  SPEAKER_01: Can you please tell me what I have not tried?
• 26:40
  SPEAKER_01: So I said ask, but that's the point.
• 26:43
  SPEAKER_01: Where do you ask now?
• 26:45
  SPEAKER_01: So Airflow has a couple of uh different discussion forums.
• 26:49
  SPEAKER_01: We do Gmail, we do Slack, we do uh GitHub PR and uh GitHub discussions as well.
• 26:55
  SPEAKER_01: But if you have a question, the most common, informal, or uh a place where you will get the quickest response potentially will be Slack.
• 27:04
  SPEAKER_01: So once you have you've added yourself to the Airflow Slack workspace, you're automatically part of different Slack channels.
• 27:11
  SPEAKER_01: And just to give you examples, we have things like troubleshooting, we have Airflow 3 development, we have a random channel, maybe we have a channel for UI development, and maybe a channel for first time, first timers where they want to create a PR and then two problems.
• 27:25
  SPEAKER_01: So we do have these different channels.
• 27:27
  SPEAKER_01: So you're already a part of them once you join the Airflow Slack workspace.
• 27:31
  SPEAKER_01: So yeah, go to these channels, just jump in and ask your question, ask away.
• 27:35
  SPEAKER_01: So that's the other thing.
• 27:37
  SPEAKER_01: Yeah, I think the other, yeah, maybe a last one last two would be to subscribe yourself to both the dev and the user email lists of Airflow.
• 27:48
  SPEAKER_01: So the reason being once you subscribe to these, you will be getting regular Airflow-related emails.
• 27:54
  SPEAKER_01: And if you don't do anything and you only read those emails also, regularly seeing these emails in your inbox will already it'll start to increase your airflow knowledge automatically just by reading it for a couple of days.
• 28:06
  SPEAKER_01: Even if you don't understand everything in detail, just by uh catching up with it, you know what's happening around the project, the direction it is headed in.
• 28:14
  SPEAKER_01: So things like that.
• 28:16
  SPEAKER_01: So all always subscribe to it.
• 28:18
  SPEAKER_01: And the other yeah, the other last point, maybe this is slightly an uncommon one, but if your city hosts any of these airflow meetups, make it a point to go and attend these meetups because uh you'll meet some great people there.
• 28:33
  SPEAKER_01: The people who have done great things with airflow, the people who want to do great things with airflow.
• 28:37
  SPEAKER_01: And this knowledge is obviously going to uh benefit you.
• 28:41
  SPEAKER_01: And the last thing, consistency, very important.
• 28:45
  SPEAKER_01: In my case, how it helped in my case, what I did was I used to block a chunk of my calendar daily for just for doing airflow.
• 28:54
  SPEAKER_01: And in this time, I did nothing but airflow.
• 28:57
  SPEAKER_01: And when I say I did airflow, it can be anything.
• 29:00
  SPEAKER_01: You can read emails, you can reply to emails, you can help Slack, you can create issues, you can create PRs, review PRs, any of these things, but make it a point to do only airflow during this time.
• 29:13
  SPEAKER_01: And once you keep doing this for a while, you're going to be very comfortable and you're going to get so used to things that you would not need the calendar event anymore.
• 29:22
  SPEAKER_01: So that's the idea, consistency.
• 29:26
  SPEAKER_03: That's a lot of great recommendations.
• 29:29
  SPEAKER_03: Thank you so much, Amok, for everything.
• 29:31
  SPEAKER_03: Actually, thank you for joining.
• 29:32
  SPEAKER_03: That that was a great episode.
• 29:34
  SPEAKER_03: And now I think I'm all set up for contributing to Airflow.
• 29:38
  SPEAKER_03: So thank you so much.
• 29:39
  SPEAKER_03: I guess one last question what is the best way for folks to get in touch with you?
• 29:44
  SPEAKER_01: Oh, yes.
• 29:45
  SPEAKER_01: So if you're already part of the community, then you can reach out to me on Slack.
• 29:49
  SPEAKER_01: That'll be a place where you'll get the quickest response.
• 29:52
  SPEAKER_01: Otherwise, feel free to connect with me on LinkedIn.
• 29:55
  SPEAKER_01: I'll share it with you later.
• 29:57
  SPEAKER_01: So, yeah, these two things I think will be good to connect with me.
• 30:01
  SPEAKER_01: Awesome.
• 30:02
  SPEAKER_01: Thank you again.
• 30:05
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 30:12
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 30:16
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.