Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer

Transcript

Getting Into Data Engineering with Shrividya Hegde, Data and AI Engineer · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: AI is only good based on what data it gets fed on.
• 0:03
  SPEAKER_01: Otherwise, it'll just be confidently wrong.
• 0:05
  SPEAKER_01: And confidently wrong is more scary than obviously wrong because you can't even catch it.
• 0:13
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:25
  SPEAKER_02: Okay, hi everyone.
• 0:27
  SPEAKER_02: Welcome to the Data Flowcast.
• 0:28
  SPEAKER_02: I'm your host, Kenton Dannis, and today I'm joined by Sri Higdi, who is a data and AI engineer and an airflow champion in Astronomers Champions Program.
• 0:38
  SPEAKER_02: Shree, it's great to see you.
• 0:39
  SPEAKER_02: It's good to see you too, Kenton.
• 0:41
  SPEAKER_02: Yeah, how are you doing?
• 0:42
  SPEAKER_02: I'm good.
• 0:43
  SPEAKER_02: How are you?
• 0:43
  SPEAKER_02: I'm good.
• 0:44
  SPEAKER_02: I'm good.
• 0:44
  SPEAKER_02: Thanks.
• 0:45
  SPEAKER_02: It's great to have you on.
• 0:46
  SPEAKER_02: I'm excited to dig into it.
• 0:48
  SPEAKER_02: Uh today we're going to have a little bit of a zoomed-out conversation, which I'm excited about because it's one I see people asking about online all the time.
• 0:56
  SPEAKER_02: Uh, we're going to talk about what it actually takes to get into data engineering, um, as well as things like how to contribute to open source and how AI is reshaping the role.
• 1:06
  SPEAKER_02: Very popular topic right now.
• 1:07
  SPEAKER_02: So I think we'll have lots of interested listeners.
• 1:11
  SPEAKER_02: Uh, before we get into all of that, let's start with a little bit of background on you.
• 1:16
  SPEAKER_02: Uh, I guess first tell me how did you get involved with Airflow?
• 1:20
  SPEAKER_01: Uh, my entry point to this role was basically by as an estate, actually, to my previous organization.
• 1:26
  SPEAKER_01: I wasn't hired as a data engineer.
• 1:28
  SPEAKER_01: So, but uh that role required me to work a lot on airflow and understand the DAGs and understand pipelines to basically understand the pain points of data engineers.
• 1:37
  SPEAKER_01: So that was what what the role involved because it was a lot of testing and things like that.
• 1:42
  SPEAKER_01: But while I did that, I kind of realized that I want to do this more often and probably transition into the role.
• 1:48
  SPEAKER_01: And when I joined an observability team, that's where things got a lot more technical.
• 1:52
  SPEAKER_01: And I eventually ended up debugging and checking a lot of things, and now I'm uh like accidentally transformed into a data engineer and I'm loving it.
• 2:00
  SPEAKER_02: So okay, that's great.
• 2:02
  SPEAKER_02: I I feel like this is a similar to my own story.
• 2:05
  SPEAKER_02: I my educational background is in data science, that was what my master's was in.
• 2:09
  SPEAKER_02: And then uh one of the first projects I worked on after grad school was sort of intended to eventually be data science-esque, but before they just didn't have the data in a good place to do any like modeling or anything on it.
• 2:23
  SPEAKER_02: And so it was a lot of data engineering up front.
• 2:25
  SPEAKER_02: And then so I was doing that and just found that I actually really liked it um even better than some of the data science work.
• 2:32
  SPEAKER_02: So uh yeah, I think that happens to probably a lot of people in the industry.
• 2:36
  SPEAKER_01: That's true.
• 2:37
  SPEAKER_01: I've met a lot of other people as well who had very interesting stories and completely different career backgrounds, transitioning to data and excelling at it.
• 2:45
  SPEAKER_01: So yeah, yeah.
• 2:46
  SPEAKER_02: Definitely lots of accidental data engineers out there.
• 2:50
  SPEAKER_02: Um lots of accidental airflow users who are now loving it.
• 2:54
  SPEAKER_02: That's what we hope for anyway.
• 2:56
  SPEAKER_02: Yeah, cool.
• 2:56
  SPEAKER_02: And so you, I think, have described yourself previously as someone who wears you know a lot of different hats.
• 3:01
  SPEAKER_02: You mentioned you work in data engineering.
• 3:03
  SPEAKER_02: I know you've done some AI work and now do some community leadership as well.
• 3:07
  SPEAKER_02: Right now, what does your day-to-day typically look like?
• 3:11
  SPEAKER_01: It's honestly an organized chaos.
• 3:14
  SPEAKER_01: So at any given day, you might be seeing me writing uh something about data engineering concept or trying to fix a data pipeline problem or maybe contributing to Airflow, helping somebody in the women in data community to whoever is trying to break through the field.
• 3:28
  SPEAKER_01: So the point is that all of it connects, right?
• 3:32
  SPEAKER_01: So nothing is too different from each other.
• 3:34
  SPEAKER_01: Uh, writing makes me a better engineer because in order to write something, you need to completely actually understand it.
• 3:40
  SPEAKER_01: And that gets me a lot of depth in the concept that I'm working on, as well as trying to learn or upskill.
• 3:45
  SPEAKER_01: And women in data community helps me keep myself grounded and get I it I get to see the struggle that I had once gone through, and I know exactly what to say or do to make their struggle a little easier if possible.
• 3:57
  SPEAKER_01: So at the end of the day, it's very satisfying and very intentional.
• 4:01
  SPEAKER_02: Yeah, that's great.
• 4:02
  SPEAKER_02: I love that.
• 4:02
  SPEAKER_02: And I definitely want to dig in a little more to a couple of different pieces of that that you mentioned.
• 4:07
  SPEAKER_02: Before we get there, I think just taking a high-level look at like getting into data engineering or this industry in general, this is something that I hear, I certainly personally get a lot of questions about from folks who are newer to their careers.
• 4:23
  SPEAKER_02: I see a lot of people talking on like Reddit, the data engineering subreddit, things like that about getting into data engineering now coming from either school or like a different background and transitioning into it.
• 4:35
  SPEAKER_02: Uh, so in your experience, I guess what advice would you give to someone who is like a new data engineer or looking to get into that field or a related field?
• 4:46
  SPEAKER_01: So fundamentals are always the key.
• 4:48
  SPEAKER_01: It's better to start boring than you know, catch up with something that is looks looks trendy and then uh try to call like it gets you collapsed.
• 4:57
  SPEAKER_01: So it's always to start better to start boring.
• 5:00
  SPEAKER_01: I don't mean to say in a wrong way, it's the best way possible.
• 5:03
  SPEAKER_01: So before you touch a tool, you need to understand what the tool does, why does it even exist for what problem is it trying to solve?
• 5:10
  SPEAKER_01: Uh things like SQL, data modeling, how does the data move through pipelines and all of these are your fundamentals, and those have to be very strong before you try and catch any tool that is trending on LinkedIn or any other like outside you so you see a lot of trends, right?
• 5:24
  SPEAKER_01: So instead of catching the trends on, it's otherwise like a beautiful house that is having a lot of fixtures and interiors, but then the foundation is shaky, so it won't hold together for a long time.
• 5:34
  SPEAKER_01: I personally have seen many uh amazing data engineers who have faced burnout because of all this.
• 5:39
  SPEAKER_01: And it's not because of a talent problem, it's just sequencing, uh, the wrong sequencing, basically, fundamentals being skipped and then jumped to a higher aim.
• 5:48
  SPEAKER_02: Yeah, definitely.
• 5:49
  SPEAKER_02: I think it would be so easy to look at.
• 5:52
  SPEAKER_02: I I'm gonna forget like the actual name of, but the diagram that has all of the different tools in it that are like data related.
• 5:59
  SPEAKER_02: Um and like airflow is one.
• 6:01
  SPEAKER_02: I think usually astronomer is one that's in there.
• 6:04
  SPEAKER_02: Um, but like it's just so overwhelming.
• 6:06
  SPEAKER_02: I think it would be really easy to look at that and be like, no.
• 6:09
  SPEAKER_02: And just there is not, you know, it's not really clear like where you should start always.
• 6:14
  SPEAKER_02: So I love that advice of working with the fundamentals first.
• 6:18
  SPEAKER_01: Yeah, and also knowing the alternatives to the tools, right?
• 6:21
  SPEAKER_01: So if you know just one tool about it, if you know your fundamentals, you'll probably go and figure out what problems is this tool not solving.
• 6:28
  SPEAKER_01: And does there is there any other tool in the market that exists so that we can weigh options between uh both of them and decide which is best for a given scenario?
• 6:36
  SPEAKER_01: But if it is this fundamental skipped and we don't know the concept that is underneath that, we just stick to the tool and like try to bang our head against it.
• 6:44
  SPEAKER_01: Like, why is this not working?
• 6:45
  SPEAKER_01: What do we do now?
• 6:46
  SPEAKER_01: So it's purely that.
• 6:48
  SPEAKER_02: Yeah, definitely.
• 6:49
  SPEAKER_02: As you said, easier to build like on top of the foundation, build a good tech stack if you have like good fundamentals in place.
• 6:57
  SPEAKER_02: I guess to dig in on that a little bit more, what do you think that means in practice, like in terms of where somebody should get started in terms of tools?
• 7:06
  SPEAKER_02: What would you consider fundamental for data engineering?
• 7:10
  SPEAKER_01: So understanding the structure of data, SQL for sure.
• 7:14
  SPEAKER_01: And it's not just about writing queries, it's about understanding what's happening underneath.
• 7:18
  SPEAKER_01: When you say joins aggregations, how does the query performance matter and all of these little things, right?
• 7:24
  SPEAKER_01: And also learning how the data moves between the pipelines.
• 7:28
  SPEAKER_01: What does a DAG do?
• 7:30
  SPEAKER_01: What does it mean for a job to fail and recover?
• 7:32
  SPEAKER_01: And how does the results change when things are failed and recovered, idem potency and such topics and all of these smaller nuances that is very important in order to make the data flow proper and tools come and go.
• 7:45
  SPEAKER_01: It's just these things that stay with us.
• 7:47
  SPEAKER_01: Yeah.
• 7:48
  SPEAKER_02: To uh ask a bit of a lead-in question, I assume that you would think that airflow is also part of that fundamentals stack.
• 7:56
  SPEAKER_01: Definitely, definitely.
• 7:57
  SPEAKER_01: Because for data orchestration, airflow is the tool that exists to make sure that the data is cleaner and reliable wherever it has to go.
• 8:04
  SPEAKER_01: Definitely airflow.
• 8:05
  SPEAKER_01: But yeah, we we need to know the structure of data if we understand basically it it gets more cleaner later on.
• 8:11
  SPEAKER_02: Yeah, yeah, absolutely.
• 8:13
  SPEAKER_02: That makes sense.
• 8:14
  SPEAKER_02: And I guess uh continuing on with sort of high-level advice for people getting into data engineering or new to the field, what is the most common mistake that you think you see people make as they're trying to transition into a role like that?
• 8:28
  SPEAKER_01: Clear goals should be first defined and then uh maybe a small POC that they try to work on it and start something smaller and try to keep getting better at it.
• 8:39
  SPEAKER_01: Again, I would like to stress the point that we have seen people chasing around whatever is trending in the market, and then because it looks good, like there is agentic orchestration, agentic AI, and then AI orchestration and so many other new things.
• 8:52
  SPEAKER_01: Tomorrow there'll be another new framework and it trends on LinkedIn.
• 8:55
  SPEAKER_01: So when something breaks it and it will, right?
• 8:58
  SPEAKER_01: So when you work on something, it definitely will, and they'll not have an idea how to fix it because they have skipped everything else that is required to get there and they have started aiming higher, which is good, but it can come later once you stack on top of the fundamentals.
• 9:12
  SPEAKER_01: So the path and the goal has to be very well defined rather than just trying to catch up with the trends.
• 9:18
  SPEAKER_02: Yeah, absolutely.
• 9:19
  SPEAKER_02: And maybe a follow-up question.
• 9:23
  SPEAKER_02: So uh you mentioned having clear goals up front.
• 9:26
  SPEAKER_02: Do you mean, you know, when someone's taking on a new role, making sure that the organization or the team has clear goals that they're working towards, or like the person themselves has clear goals for what they want in the role, or is it both?
• 9:40
  SPEAKER_01: Um, it's a mix of both.
• 9:42
  SPEAKER_01: At least like if you know, if you're very clear on what your day in that organization looks like, or if you have a clear idea of what would you do once you wake up in the morning and try to log in, that I think is a success.
• 9:53
  SPEAKER_01: At least going through case studies will case study uh analysis, in my opinion, a best way to understand how things work in an organization so that um the you get to see the solution implemented and it kind of invokes a curiosity in you as to what other ways could have been done to solve this problem.
• 10:10
  SPEAKER_01: And now that you cannot do it because it is already you get a lot of case studies on YouTube, right?
• 10:15
  SPEAKER_01: Rather than checking tutorials, I would feel this is the better way to do it.
• 10:18
  SPEAKER_01: You can also try to do a hands-on in your local dev environment and see how it goes.
• 10:22
  SPEAKER_01: If it works, you've learned something.
• 10:24
  SPEAKER_01: If it doesn't, you know why it doesn't work.
• 10:26
  SPEAKER_01: So it's a kind of analysis you do for yourself.
• 10:29
  SPEAKER_01: And it doesn't have to be on day one.
• 10:31
  SPEAKER_01: You can slowly figure it out, but at least getting a lot of hands-on rather than mugging up everything that is possibly available everywhere is uh what is important in my opinion.
• 10:42
  SPEAKER_02: Yeah, I totally agree with that.
• 10:43
  SPEAKER_02: I think it also helps for anybody who is searching for a job in the role.
• 10:48
  SPEAKER_02: I think it's very common, and I say this as someone who has interviewed and hired data engineers myself in my current role, and we do ask people to do this, but to have somebody do like a little take-home, you know, POC with for us it's Airflow.
• 11:01
  SPEAKER_02: I think for others it might be Airflow or SQL or um other tools, but just to sort of show basic proficiency.
• 11:07
  SPEAKER_02: So if you've already played around with that yourself, that can be super helpful when you do get those, you know, assignments as part of interviews.
• 11:14
  SPEAKER_01: Yeah.
• 11:15
  SPEAKER_01: And even the organizations, I think would be a lot flexible in terms of your innovative innovation and creativity, because they'll define a tech stack which is as per their enterprise standards, because the talent that they have is more aligned with that tech stack and they have been working with that tech stack and it has been very giving to them.
• 11:32
  SPEAKER_01: But if you come in with the newer idea and newer tool, I don't think the organizations, any organizations, will have a problem because you're getting innovative.
• 11:39
  SPEAKER_01: That will never happen.
• 11:40
  SPEAKER_01: So if you have an analysis of what works where, and there could be different factors on to why you have to consider a particular tool.
• 11:47
  SPEAKER_01: And to make that kind of judgment, you need to understand again the problem that the tool solves or what is your problem that you're trying to solve.
• 11:54
  SPEAKER_01: Will it match to the existing tools, or is there a better coverage of solutions that will be given from a different tool, which is probably not in the said tech stack, but you kind of know it.
• 12:04
  SPEAKER_01: It's okay to not know it completely, but if you can figure it out, or you just know that there exists some other tool which can be used to solve this problem rather than sticking to the given tech stack just because organization endorses it.
• 12:16
  SPEAKER_01: So yeah.
• 12:18
  SPEAKER_02: Yeah, yeah, absolutely.
• 12:19
  SPEAKER_02: Well, you spoke about kind of the innovation aspect of it and also a little bit earlier, kind of uh lots of you know new trends happening all the time.
• 12:29
  SPEAKER_02: Um, it seems like this is a particularly fast-paced time in data engineering and other tech roles.
• 12:35
  SPEAKER_02: So, related to that, I think we have of course seen that AI is, you know, a big part of that trendiness and what's moving really quickly right now.
• 12:44
  SPEAKER_02: And I think we're already seeing that it is changing, you know, what data engineers do in their day-to-day quite substantially.
• 12:52
  SPEAKER_02: So, from your perspective, for somebody who is newer to the field, what should they be thinking about in terms of using AI in their role?
• 13:01
  SPEAKER_02: Or actually, maybe I'll rephrase that and say, not even necessarily new to the field.
• 13:05
  SPEAKER_02: Maybe just how do you think that data engineers, period, should be considering this new tech?
• 13:11
  SPEAKER_01: So AI is actually, I feel that AI has got a lot of has created a lot of scope for data engineers rather, because it's exciting and there is real opportunity for data engineers.
• 13:23
  SPEAKER_01: Because I believe that AI is only good based on what data it's get it gets fed on.
• 13:28
  SPEAKER_01: Otherwise, it'll just be confidently wrong.
• 13:30
  SPEAKER_01: And confidently wrong is more scary than obviously wrong because you can't even catch it.
• 13:36
  SPEAKER_01: So it's not about do not use AI or use less AI, it's about understand what makes the AI work.
• 13:43
  SPEAKER_01: And for even for us to learn something or ramp up today, it is a lot more easier than what it was before AI came in.
• 13:51
  SPEAKER_01: To learn anything possible, any concept or an example.
• 13:54
  SPEAKER_01: We get a lot of AI samples these days.
• 13:57
  SPEAKER_01: We get to the pace of learning something is more faster because of AI.
• 14:02
  SPEAKER_01: So I think it's all about knowing how to work along with AI.
• 14:06
  SPEAKER_01: And also uh one small step is to test whatever the AI gives us output and not blindly believing it because it's all just a game of data that goes into the AI pipelines as well.
• 14:17
  SPEAKER_01: At the end of the day, it's a good data engineering that has resulted in a good AI engineering as well.
• 14:24
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 14:33
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 14:45
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 14:54
  SPEAKER_02: Yeah, absolutely.
• 14:55
  SPEAKER_02: I love that summary.
• 14:56
  SPEAKER_02: And to draw from an airflow feature, I think having the human in the loop via airflow with the feature or whatever in whatever way it makes sense for your workflow is still really important, of course.
• 15:09
  SPEAKER_01: Yeah.
• 15:09
  SPEAKER_01: Orchestration tools like Airflow exist for the same thing, right?
• 15:12
  SPEAKER_01: So they try to make this uh data reliable and cleaner, and we have all the supporting operators that can actually help us get there.
• 15:20
  SPEAKER_01: So when used along with these kinds of tools, is when the AI becomes a multiplier.
• 15:24
  SPEAKER_01: Without it, it's just like building on sand.
• 15:27
  SPEAKER_02: Yeah, definitely.
• 15:29
  SPEAKER_02: My with my team, we talked about it kind of in two different ways within the context of Airflow.
• 15:35
  SPEAKER_02: We say there's both like airflow for AI, which is a lot of like managing the data and the context that you just talked about, that's so important because otherwise, yeah, just using a generic model with no context, you're probably not going to get the results that you want.
• 15:50
  SPEAKER_02: Or to your point, they'll seem like what you want, but they're not actually what you want, which is even worse.
• 15:55
  SPEAKER_02: Um, and then there's the other aspect of it, which is the AI for airflow.
• 15:59
  SPEAKER_02: So, like using AI to help you write your DAGs or manage your airflow instance.
• 16:04
  SPEAKER_02: Of course, astronomer is very heavily invested in that area and has provided some open source skills and things like that to help everybody along.
• 16:12
  SPEAKER_02: But I think both of those aspects for data engineers are are really important to be, you know, keeping up with and kind of integrating into their day-to-day workflows.
• 16:20
  SPEAKER_01: Yeah, that's very clearly put uh in terms of the contribution of yeah, it's uh two different lenses to see and both are equally, you know, uh relevant today.
• 16:29
  SPEAKER_01: So yeah, I I completely agree with that.
• 16:32
  SPEAKER_02: Yeah, definitely.
• 16:33
  SPEAKER_02: It's a I guess exciting time within the field.
• 16:37
  SPEAKER_02: Certainly.
• 16:37
  SPEAKER_02: It feels like things are changing every single day, which is yeah, a lot to keep up with.
• 16:43
  SPEAKER_02: But I do, I do love the framing of just having data engineers, their role has sort of gotten up-leveled a little bit, where you can move a lot faster.
• 16:53
  SPEAKER_02: And then, but that also means you get to spend hopefully more of your time kind of architecting the systems and figuring out what that should look like, um, how you should be managing your data at a higher level and hopefully less time like figuring out how to use like window functions or whatever the hard things are in SQL.
• 17:12
  SPEAKER_02: I'm thinking back to grad school when I learned SQL and was like, oh, some some of it's kind of tricky, and I could never quite remember how to use it every time.
• 17:20
  SPEAKER_02: I don't really have to do that anymore, which I guess is nice.
• 17:23
  SPEAKER_01: Yeah, some little things like that you kind of can always cross-check.
• 17:27
  SPEAKER_01: So earlier I remember uh there there used to be two modes of interviews.
• 17:31
  SPEAKER_01: One is you can either use Google or you cannot use Google and you still have to do the coding rounds.
• 17:35
  SPEAKER_01: Now it is either you can use AI or you cannot use AI.
• 17:39
  SPEAKER_01: But when we uh in a day-to-day task, when we log in and start working, we most probably will use AI for cross-checking the syntaxes or you know, anything technical if you want to just double check, it's okay to use AI.
• 17:51
  SPEAKER_01: And most probably you will also get a better optimized answer sometimes if you have not thought it out through because you're focusing on something else like architecture designing and things like that.
• 18:00
  SPEAKER_01: Technically, it can also give you a better answer.
• 18:02
  SPEAKER_01: Uh, I've seen the way AI generates DAGs, and it's very, I mean, it it's impossible for somebody who is starting to have so much knowledge to put a DAG together the way AI does it.
• 18:13
  SPEAKER_01: Generating the DACs and going through it and understanding where these concepts come from and helping the person who is checking the DAG understand the concepts better is also the good part of AI, right?
• 18:24
  SPEAKER_01: So yeah.
• 18:25
  SPEAKER_02: Yeah, yeah, absolutely.
• 18:26
  SPEAKER_02: It can be very, very helpful.
• 18:28
  SPEAKER_02: Um, so yeah, cool.
• 18:31
  SPEAKER_02: Well, changing uh tracks a little bit, you mentioned a little bit earlier that you do some work with women in data and are involved in that community.
• 18:41
  SPEAKER_02: Um, tell me a little bit more about that work.
• 18:44
  SPEAKER_01: So, women in data community is a space provided for anybody who is trying to navigate their career into data or already is in data and want to contribute to the people or uh just a community to talk more about data and feel get that relevance uh in the way you do or the tasks that you do in your day-to-day life.
• 19:02
  SPEAKER_01: And there's it's a very friendly space where there is no barriers for beginners to ask simple questions, or if you want to just showcase whatever you worked on in an advanced topic, you can also show that.
• 19:15
  SPEAKER_01: So it's a very accepting space for anybody in data.
• 19:19
  SPEAKER_01: And uh the goal is to reduce the barriers to ask questions and discuss and things like that.
• 19:25
  SPEAKER_01: It's a very welcoming community.
• 19:28
  SPEAKER_01: Yeah.
• 19:29
  SPEAKER_02: Yeah, that's great.
• 19:30
  SPEAKER_02: I think it's uh it's especially helpful, in my opinion, to have those communities, both just for a local community where you can interact more with people who are doing the same types of work without, you know, so much of our work is remote.
• 19:45
  SPEAKER_02: So having local chapters, I think is helpful for that connection.
• 19:48
  SPEAKER_02: And certainly, you know, my uh from my own experience, like having chapters specifically related to folks who are like in a minority group within a particular field can just add that.
• 19:59
  SPEAKER_02: What am I trying to say?
• 20:00
  SPEAKER_02: I I think it can be it can be very intimidating to start in a field when you're a minority, especially a technical one.
• 20:08
  SPEAKER_02: Um, so I think having that community is really helpful.
• 20:11
  SPEAKER_01: Yeah, and I've seen women who uh join the community sometimes are from a completely different background, but they're really good at data and they are just actually intimidated to see where to start from or whom to contact if they want to learn more or understand how it is in the real life when they once get into the roles and they get an idea of all of that here in Women in Data.
• 20:32
  SPEAKER_01: So such kind of support is immense and it's very welcoming.
• 20:36
  SPEAKER_01: We say that it's a local chapter, but we have virtual events as well, right?
• 20:39
  SPEAKER_01: So anybody who doesn't stay in that local area is also as much welcome to join uh those chapters as well.
• 20:46
  SPEAKER_01: I myself am part of almost every chapter over there in Women in Data.
• 20:50
  SPEAKER_01: I was heavily relying on the NYC chapter because it is close to New Jersey.
• 20:54
  SPEAKER_01: We did not have a chapter for New Jersey, which is why I took the initiative and I said that I can lead the chapter.
• 21:00
  SPEAKER_01: But we have people in New Jersey who work in data and are relying on New Jersey uh New York events.
• 21:06
  SPEAKER_01: It's okay for virtual, but in-person events, they have to go all the way there.
• 21:09
  SPEAKER_01: And if they can't make it due to some commitments and other work-related commitments or personal commitments, they're going to miss out those events, right?
• 21:16
  SPEAKER_01: So if we can conduct it here in NJ, so it'll be helpful.
• 21:20
  SPEAKER_01: So they were uh very much kind enough to say, yeah, I can lead the team.
• 21:24
  SPEAKER_01: So now yeah, we are.
• 21:26
  SPEAKER_02: Yeah, that's great.
• 21:27
  SPEAKER_02: Uh, love to hear that you're taking that initiative.
• 21:30
  SPEAKER_02: And also, I think you kind of answered my next question, which was going to be for anybody, um, any women in data are looking to get into data who might be listening to this if they're interested.
• 21:38
  SPEAKER_02: I assume there's a website that will list all of the chapters, and folks can find if there is a local one or maybe virtual events if there's not.
• 21:46
  SPEAKER_01: Yeah, there is a uh you can go to www.womenindata.org and that itself is a website where you can just sign in as a community member.
• 21:54
  SPEAKER_01: So there are two types either you can join as a pro member, where it's a uh paid course for like you.
• 22:00
  SPEAKER_01: Get a paid membership for$120 or something, and that's for a year.
• 22:03
  SPEAKER_01: Or if you just want to explore and just be a part of the events in the community, you can join as a community member, which is absolutely free.
• 22:10
  SPEAKER_01: It's a nonprofit organization, so anything that comes as part of the membership uh payments goes to sponsor somebody else who applies for say scholarship or anything like that, or if they uh need assistance joining for a pro member and but they can't do it right now, so it goes as an assistance.
• 22:27
  SPEAKER_01: It runs on uh fundraising campaigns as well, so all of that money goes back to the community.
• 22:31
  SPEAKER_01: It's all about a friendly data environment.
• 22:35
  SPEAKER_02: Yeah, very cool.
• 22:36
  SPEAKER_02: Well, we can definitely throw a link to that website in the show notes here.
• 22:40
  SPEAKER_02: And then speaking of other kind of community-focused things, I know that you are also very involved in the Airflow community and obviously with our Astronomers Champions program, but also with contributing to Airflow.
• 22:54
  SPEAKER_02: Uh, what was your experience like getting started, just getting involved with the open source project?
• 22:59
  SPEAKER_01: Oh, the support I got from community cannot be understated.
• 23:03
  SPEAKER_01: It I did not expect a community with such a huge project to be that welcoming and addressing anything that we ask in the channels.
• 23:11
  SPEAKER_01: So there is a champions channel.
• 23:13
  SPEAKER_01: There are other lot of channels in Airflow Slack as well.
• 23:16
  SPEAKER_01: So if you are working on something, you get any questions or not able to work further, you can just type it as a message and there are people answering them very actively.
• 23:24
  SPEAKER_01: So that itself is a real-time advice to you when you're working in a local dev environment.
• 23:30
  SPEAKER_01: That kind of support, if given, I don't think it's difficult for anybody to start contributing to open source.
• 23:36
  SPEAKER_01: I had contributed to the open source uh repository as uh my my role as a during my role as a tester, where I used to raise issues if I see something in Airflow and uh log them.
• 23:48
  SPEAKER_01: And that was an indirect sort of contribution, but I never committed PRs as such.
• 23:52
  SPEAKER_01: But when I once I joined Airflow Championship and the channels and the discussions that went through it, kind of sparks the curiosity to see what problems they are trying to solve or if there is a PR, just out of curiosity, going there and checking what changes were made to solve what issues and if I know something about it, or if I have a better solution, or if I think differently, just an attempt to, you know, raise a PR.
• 24:15
  SPEAKER_01: I didn't feel that intimidating as I thought it would be.
• 24:18
  SPEAKER_01: And there are very friendly review comments.
• 24:20
  SPEAKER_01: People are really wanting to advise you to get better in whatever you do.
• 24:24
  SPEAKER_01: Uh, they're very kind actually.
• 24:26
  SPEAKER_01: The community is a lot kinder than what I expected for such a huge project.
• 24:30
  SPEAKER_01: It makes it look human-sized because of that support.
• 24:33
  SPEAKER_02: Yeah, that's great.
• 24:34
  SPEAKER_02: I know it does feel very intimidating, I think, to a lot of folks getting started uh just because of the complexity of the process and the project.
• 24:43
  SPEAKER_02: So that's really great to hear.
• 24:45
  SPEAKER_02: Is there anything right now that you're contributing on that you're particularly excited about?
• 24:50
  SPEAKER_01: Uh, I'm I'm just trying to check any issues that I feel relevant and if I can contribute to it, but it's more of an impromptu thing that I do with Airflow with the other commitments given.
• 25:00
  SPEAKER_01: I on a given day, I just try to see if there is anything that I can solve, or if not, if somebody else has solved it, it's a good learning for me to see, okay, that's how they have done it.
• 25:09
  SPEAKER_01: Well, I couldn't do it.
• 25:10
  SPEAKER_01: So it's just that way, but not anything planned at the moment.
• 25:14
  SPEAKER_02: Yeah, yeah, totally.
• 25:16
  SPEAKER_02: That makes sense.
• 25:16
  SPEAKER_02: Well, I think that's a good tie-in to my standard final question that I ask all of our guests, and that is what would you most like to see from development of the Airflow project in the near future?
• 25:27
  SPEAKER_01: Development-wise, I think we have most of it covered, at least after Airflow 3 versions.
• 25:32
  SPEAKER_01: There was everything that the community has asked for has been catered, be that the React UI or uh the backfield triggering or HITL operator, asset partitioning, everything I think has been already catered.
• 25:45
  SPEAKER_01: Now the ask is to move from maybe uh what can the community provide to how to make it more adaptable to the ecosystem.
• 25:53
  SPEAKER_01: Uh, probably more support to the managed services or maybe uh more documentation and examples for the already developed aspects and contribution guidelines, I think was very helpful for me, at least when I joined.
• 26:07
  SPEAKER_01: So I don't see any scope of improvements there.
• 26:09
  SPEAKER_01: But I think more examples and documentation will be much better for somebody to, you know, say, I'm trying to check airflow to I actually understand what's going on in Airflow repository.
• 26:20
  SPEAKER_01: So that kind of shift is a win for us, is what I feel.
• 26:24
  SPEAKER_02: Yeah, definitely.
• 26:25
  SPEAKER_02: Well, I love hearing that feedback because that's a big focus of my team is providing examples and guidance on using Airflow.
• 26:32
  SPEAKER_02: So yeah, always great to hear that there's an appetite for that.
• 26:36
  SPEAKER_02: Okay.
• 26:36
  SPEAKER_02: Well, Sri, thank you so much for going through all of that.
• 26:39
  SPEAKER_02: I think this has been really insightful and hopefully helpful for folks, you know, newer in their data engineering or other data-related journeys.
• 26:48
  SPEAKER_02: Just to close us out, what is the best way for folks to get in touch with you?
• 26:52
  SPEAKER_01: Uh, LinkedIn is a place where I'm mostly active.
• 26:55
  SPEAKER_01: I write on Medium and Substack, so that that's also a way to follow along.
• 26:59
  SPEAKER_01: And yeah, Airflow community is always open for discussions.
• 27:02
  SPEAKER_02: So Yeah, yeah, definitely.
• 27:04
  SPEAKER_02: Well, we will throw links to those things in the show notes as well.
• 27:07
  SPEAKER_02: Uh okay, well, Tree, uh, again, it's been really great speaking with you.
• 27:11
  SPEAKER_02: I appreciate you coming on.
• 27:12
  SPEAKER_02: Thank you for the opportunity, Kentuck.
• 27:15
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 27:22
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 27:26
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.