Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Automating Threat Intelligence Using Airflow with Karan Alang

Transcript

Automating Threat Intelligence Using Airflow with Karan Alang · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Because of so much data flowing in and the innovative ways in which the hackers are actually hacking today's systems, right?
• 0:06
  SPEAKER_02: There's a need to automate the way in which organizations deal with these uh different ways of being hacked, basically.
• 0:17
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering in AI.
• 0:30
  SPEAKER_01: Hi everyone.
• 0:31
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:33
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Karan Along, who is a principal software engineer at Versa Networks.
• 0:40
  SPEAKER_01: Karan, how are you doing today?
• 0:41
  SPEAKER_02: I'm doing good, Kendon.
• 0:43
  SPEAKER_02: Thanks for having me.
• 0:44
  SPEAKER_02: By the way, a quick disclaimer the views expressed in this podcast are my own and are independent of my employer.
• 0:50
  SPEAKER_01: Good.
• 0:51
  SPEAKER_01: Yeah.
• 0:52
  SPEAKER_01: It's great to have you on.
• 0:53
  SPEAKER_01: We really appreciate you taking the time.
• 0:56
  SPEAKER_01: For today's podcast, we're going to discuss uh using Airflow for automating threat intelligence, which I think will be a really interesting topic.
• 1:04
  SPEAKER_01: But before we get into that, I want to start with some background upon you and the company that you work for.
• 1:09
  SPEAKER_01: So maybe start by telling me a little bit about where you work and what Versa Networks does.
• 1:14
  SPEAKER_02: Worsa as a networking and security company.
• 1:17
  SPEAKER_02: It has softwares which help companies securely connect remote workers, offices, data centers, and cloud applications.
• 1:24
  SPEAKER_02: So everything works safely and reliably.
• 1:27
  SPEAKER_01: Okay, got it.
• 1:28
  SPEAKER_01: And at a high level, how does the company use Airflow?
• 1:32
  SPEAKER_02: So Airflow has been uh in the group that I am in, AIML, right?
• 1:36
  SPEAKER_02: It's been a fairly important part of the entire architecture.
• 1:39
  SPEAKER_02: So we're using it for orchestration.
• 1:41
  SPEAKER_02: It's been quite helpful.
• 1:43
  SPEAKER_02: Uh I think I've been using it for the last three, four years now.
• 1:46
  SPEAKER_02: And uh it's been a lifesaver in many ways, right?
• 1:49
  SPEAKER_02: Uh the UI is amazing, and being able to debug issues, being able to launch different types of workloads like machine learning and Apache Spark workflows has been quite helpful.
• 2:00
  SPEAKER_02: So Airflow has been great in that regard.
• 2:03
  SPEAKER_01: Okay, that's great.
• 2:04
  SPEAKER_01: And I'd also like to hear a little bit more about your own background because I understand you have quite a lot of experience in the data engineering and AI ML fields.
• 2:13
  SPEAKER_02: Yeah.
• 2:14
  SPEAKER_01: So, what has been your general experience using Airflow, maybe beyond your current company?
• 2:19
  SPEAKER_02: Oh, well, uh, Airflow actually I started using when I started working with Varsa.
• 2:23
  SPEAKER_02: Right.
• 2:24
  SPEAKER_02: In terms of my experience, I think I've got like it's a fair amount of experience, 25 plus years of experience in the software industry.
• 2:31
  SPEAKER_02: I've uh currently working as a principal software engineer at Warsaw Network, like you mentioned earlier.
• 2:37
  SPEAKER_02: Prior to that, I've been involved with companies like Apple Link, uh, Lenovo Computers, US Steel, iTunes Technologies, and a bunch of others.
• 2:45
  SPEAKER_02: Right.
• 2:46
  SPEAKER_02: Uh in the last 10, 12 years, uh, especially in my tenure with Apple and uh Wursa, a lot of it has been uh you know, uh data engineering, uh, big data, cloud computing, uh, AIML, and cybersecurity.
• 3:01
  SPEAKER_02: So good amount of experience with data, good amount of experience with cloud computing, big data, right?
• 3:06
  SPEAKER_02: The different solutions within uh the big data stack.
• 3:09
  SPEAKER_02: And in the last few years, cybersecurity and AIML has been the focus of my career.
• 3:15
  unknown: Yeah.
• 3:16
  SPEAKER_02: So it's a very varied experience.
• 3:18
  SPEAKER_02: I mean, actually, I started off my career as a in the sales uh uh space.
• 3:23
  SPEAKER_02: Yeah, I was doing a bit of sales and then I moved into software engineering, and after that, has been uh a long tenure in the software engineering world.
• 3:30
  SPEAKER_01: Yeah, interesting.
• 3:31
  SPEAKER_01: Yeah, okay.
• 3:32
  SPEAKER_01: Yeah, very cool.
• 3:33
  unknown: Yeah.
• 3:34
  SPEAKER_01: Uh and to dig in a little bit more on uh the particular use case that we mentioned at the front, I know you've spoken publicly uh in the past, providing some thought leadership on using Airflow to automate threat intelligence, uh, which I think is really interesting.
• 3:50
  SPEAKER_01: So uh tell me first about just what interested you in this use case and why you wanted to speak publicly about it.
• 3:57
  SPEAKER_02: Yeah.
• 3:58
  SPEAKER_02: Yeah, I think uh this is actually a very important topic in today's world.
• 4:01
  SPEAKER_02: Uh so as you are aware, you know, in the last few years, uh, the number of devices that people have started using, right?
• 4:09
  SPEAKER_02: In terms of uh the mobile devices and the laptops and the tablets, as well as companies started using cloud computing.
• 4:17
  SPEAKER_02: Cloud computing has become a big part of the entire uh usage of devices per se, right?
• 4:22
  SPEAKER_02: So uh and of course the fact that people have been sharing data on social media for one thing, right?
• 4:27
  SPEAKER_02: Companies have become a lot more comfortable uh you know storing data in the cloud.
• 4:32
  SPEAKER_02: So the cloud computing usage has become quite a lot.
• 4:34
  SPEAKER_02: Now, what what has happened in this entire uh uh scenario is that hacking has actually become quite profitable now, right?
• 4:42
  SPEAKER_02: So much of data is actually flowing into the different systems, and hackers are having a field day, right?
• 4:48
  SPEAKER_02: They're finding innovative ways of hacking systems in the corporate world, especially, right?
• 4:54
  SPEAKER_02: Uh, we've had so many instances of you know data being stolen and uh you know PII is being stolen and you know ransomware and stuff like that, right?
• 5:04
  SPEAKER_02: So that is the background of this particular presentation because of so much data flowing in and not enough uh and the innovative ways in which the hackers are actually hacking today's systems, right?
• 5:16
  SPEAKER_02: Uh there's a need to automate uh the way in which organizations deal with these uh different ways of being hacked, basically, right?
• 5:25
  SPEAKER_02: And the fact is that there are not too many, there are not enough people or not enough SOC analysts.
• 5:31
  SPEAKER_02: SOC is security operations center.
• 5:33
  SPEAKER_02: Okay, these are the guys who looked at, look at the different, look at the data that is flowing into the systems, right?
• 5:38
  SPEAKER_02: And uh they're not enough people to be able to manage these threats, so to speak, right, uh, in an effective manner.
• 5:46
  SPEAKER_02: And that is where the automation actually comes into picture.
• 5:49
  SPEAKER_02: There have been efforts to uh you know to be able to automate all this, but a lot of it is lot lot more needs to be done.
• 5:57
  SPEAKER_02: Okay, and that is where this actually present this presentation is actually, I think, in some ways, a novel way of trying to address this particular problem.
• 6:06
  SPEAKER_01: Okay, interesting.
• 6:07
  SPEAKER_01: Yeah, that totally makes sense.
• 6:09
  SPEAKER_01: Uh certainly sounds like a very important use case.
• 6:12
  SPEAKER_01: And then uh give me a little bit of background on you know what particular solution you're solving with airflow and you know, maybe some of the background kind of concepts that folks might not be aware of.
• 6:25
  SPEAKER_02: Sure.
• 6:26
  SPEAKER_02: So uh like I mentioned, the topic that I was that I presented at the Airflow Summit 2025, right, was essentially automating threat intelligence using a bunch of software like Airflow, right?
• 6:37
  SPEAKER_02: XDR, which stands for extended detection and reaction.
• 6:41
  SPEAKER_02: Then you have, and I'll go into some details, a little bit of details on on that.
• 6:45
  SPEAKER_02: Then you have uh uh LLMs, which is uh probably one of the most important pieces of the entire architecture.
• 6:51
  SPEAKER_02: Then you have the GraphDB like Neo4J.
• 6:54
  SPEAKER_02: That's that's also super important.
• 6:55
  SPEAKER_02: And of course you have miter attack, right?
• 6:57
  SPEAKER_02: So these are the four different, four or five different pieces of the entire architecture which I've highlighted, which I've used in their particular presentation.
• 7:05
  SPEAKER_02: Right now, yeah, so so so that's the those are different solutions.
• 7:10
  SPEAKER_02: And Airflow, of course, is a big part of it because that is what I've used for the entire orchestration, leveraging the experience that I've had at Warsaw.
• 7:17
  SPEAKER_02: Yeah.
• 7:18
  SPEAKER_01: Got it.
• 7:19
  SPEAKER_01: That makes sense.
• 7:19
  SPEAKER_01: And explain to me, um, and those who might not be familiar, what the MITRE attack framework is.
• 7:26
  SPEAKER_02: Yeah, MITRE attack actually is uh the standard for so essentially it's a globally available database, right?
• 7:33
  SPEAKER_02: What it does is what it should what it has is something called TTP, uh, which stands for tactics, techniques, and procedures.
• 7:41
  SPEAKER_02: So in Laban's term, it is essentially a database which specifies how uh or which lays out the different ways uh in which a hacker can actually attack a particular system.
• 7:51
  SPEAKER_02: So tactics is nothing but the how of the attack.
• 7:55
  SPEAKER_02: Essentially it says that, okay, this is what is happening in the system.
• 7:58
  SPEAKER_02: Uh why is it happening or how is that actually it's the why, not the how.
• 8:02
  SPEAKER_02: Uh, this is this is a system, and what why is the attacker trying to do what it is trying to do.
• 8:06
  SPEAKER_02: So when you look at the logs, okay, that is the first step.
• 8:09
  SPEAKER_02: And each of these whys have the techniques and subtechniques, which essentially specify uh how the system is actually being attacked.
• 8:18
  SPEAKER_02: For example, you might get a phishing email, okay.
• 8:20
  SPEAKER_02: That could be a uh technique.
• 8:22
  SPEAKER_02: Okay, you could you might get uh ransomware kind of uh attack, okay.
• 8:25
  SPEAKER_02: That could be one technique, right?
• 8:27
  SPEAKER_02: So those are techniques and subtechniques, okay?
• 8:30
  SPEAKER_02: And again, the the interesting thing is, you know, many of these tactics can be uh many of the tactics can be part of one single attack.
• 8:38
  SPEAKER_02: Okay, it's not just one tactics and one technique, you know, uh which is equivalent to one attack.
• 8:43
  SPEAKER_02: It could be a series of uh tactics and techniques which are employed by this or deployed by these hackers to attack a particular system, right?
• 8:52
  SPEAKER_02: And finally, you look at the procedures, these are essentially implementations of these specific uh techniques, right?
• 8:58
  SPEAKER_02: So mitre attack essentially, to summarize, it is a global database which most of the organizations use to essentially identify uh that you know this is how an attacker can actually attack a system, and therefore it helps them prepare or mitigate these uh attacks, essentially.
• 9:16
  SPEAKER_02: Right.
• 9:17
  SPEAKER_02: So that is what a mitre attack is.
• 9:19
  SPEAKER_02: It is a globally recognized database for cybersecurity, right?
• 9:23
  SPEAKER_02: And most of the organizations essentially leverage that to be able to uh you know deal with these uh ever-increasing, you know, uh efforts at hacking, right?
• 9:33
  SPEAKER_02: The interesting thing, of course, I mean, while it has so much of information, as I mentioned earlier, in today's world, you know, the hackers have become quite uh innovative.
• 9:43
  SPEAKER_02: There are new ways that are being uh that they are actually utilizing, and those also get added to these murder attack databases.
• 9:55
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 10:01
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 10:05
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five-star review.
• 10:12
  SPEAKER_01: And then, so with that context and what you said earlier about kind of the framework for which you're implementing this solution.
• 10:21
  SPEAKER_01: Tell me a little bit more about why you've chosen the tools that you do for this solution that you're advocating, particularly Airflow.
• 10:28
  SPEAKER_02: Yeah.
• 10:28
  SPEAKER_02: So Flogin is the backbone of the entire thing, right?
• 10:31
  SPEAKER_02: It is what I used for orchestration.
• 10:33
  SPEAKER_02: Uh, but the other important pieces are uh, for one thing, XTR.
• 10:37
  SPEAKER_02: XTR, like I said, uh stands for extended detection and reaction, right?
• 10:42
  SPEAKER_02: Uh so it has a cybersecurity tools uh tool, uh which is uh used by many organizations, right?
• 10:49
  SPEAKER_02: It is used in conjunction with other tools like SAM and uh uh SOAR and uh EDR, right?
• 10:56
  SPEAKER_02: I'm not getting details of these because that would be a separate podcast in itself.
• 11:00
  SPEAKER_02: But uh what it does is it basically you know gives an overall view of uh the different logs that are coming in, right?
• 11:07
  SPEAKER_02: So a cybersecurity personal, a SOCX, a SOC analyst, does not need to go into these different other tools like SIM and uh you know EDR and SOAR, and essentially can focus on looking at one different tool, which is the XTR.
• 11:21
  SPEAKER_02: And in in the uh experiment or in the in the presentation that I have that I put it put out, right?
• 11:28
  SPEAKER_02: I've used uh Microsoft Defender X XTR.
• 11:31
  SPEAKER_02: So that is the one that is a system that provides provides a log, okay, the network logs that uh we essentially uh use for uh this analysis, right?
• 11:42
  SPEAKER_02: So that is the first piece.
• 11:43
  SPEAKER_02: The second piece is the Graph DB, which is Neophoria.
• 11:47
  SPEAKER_02: Now the logs do come in, the network logs, right?
• 11:49
  SPEAKER_02: Uh that'll that'll essentially those so those logs need to be looked at to figure out what is happening in the network, right?
• 11:57
  SPEAKER_02: Whether an attack is actually happening and what kind of attack is.
• 12:00
  SPEAKER_02: More importantly, what kind of attack is actually happening, right?
• 12:04
  SPEAKER_02: So now uh one option is to put it into a SQL database, right?
• 12:07
  SPEAKER_02: So you could have uh different entities, for example, a person trying to attack, right?
• 12:12
  SPEAKER_02: Into one database, a person database, for example, and the you know, the type of attack into a separate database.
• 12:19
  SPEAKER_02: But what I've chosen to do is, and which is probably the right way, is to put this into a Graph DB, like near like Neo4j, primarily because it gives you the entire picture, right?
• 12:29
  SPEAKER_02: So you have different entities.
• 12:30
  SPEAKER_02: So you have the person, you have a particular uh resource, for example, and then there's a relationship between the two.
• 12:36
  SPEAKER_02: That's what the graph database actually shows you, right?
• 12:38
  SPEAKER_02: So it gives you the entire picture, and you don't have to, you know, uh and the sock analyst, for example, doesn't need to look at different tables in a SQL database to figure out what the issue is, right?
• 12:48
  SPEAKER_02: He looks at the entire picture and you can use graph traversals or techniques like BFS and DFS to essentially look at uh the entire data.
• 12:56
  SPEAKER_02: So that's one critical piece, right?
• 12:58
  SPEAKER_02: This entire data gets into a Graph DB, right?
• 13:01
  SPEAKER_02: And beyond that, the key part is uh uh the LLMs now.
• 13:06
  SPEAKER_02: That is the novel, uh that that is the key part in this entire architecture.
• 13:11
  SPEAKER_02: Now, as you're aware, LLMs are being used quite a lot now.
• 13:16
  SPEAKER_02: And in this particular architecture, what it does is it uh looks at the Graph DV, looks at the data, and maps it to the different miter attack tactics and techniques.
• 13:26
  SPEAKER_02: And that is the key, right?
• 13:27
  SPEAKER_02: So you are trying to you already have that information.
• 13:30
  SPEAKER_02: Now you want to be able to uh make sure that you know whatever is happening in the network, the attack that is happening, you want to be able to map it to that particular technique so you know what uh the solution for that is or how how how you're gonna actually deal with that particular thing.
• 13:44
  SPEAKER_02: Okay, so LLMs do that.
• 13:46
  SPEAKER_02: So without the LLMs, the sock analyst will have to do a manual look at the uh you know data and map it to the different uh tactics and techniques, and that is very difficult because the volume of logs is huge, like I mentioned earlier, right?
• 14:01
  SPEAKER_02: And they're not enough people to be able to do all that effectively, right?
• 14:05
  SPEAKER_02: So LLMs should do that, it also assign a score to the different users, right?
• 14:10
  SPEAKER_02: And that essentially indicates that you know a particular uh person, for example, if I'm logging into a network and I'm doing things which are actually bad, right?
• 14:19
  SPEAKER_02: The LLM will actually flag that out and say, okay, you know, I have a low score and therefore I'm a bad actor.
• 14:26
  SPEAKER_02: And when that happens, the human in the loop, which is again a very common, very important concept in uh the world of Gen AI and LLMs, that comes to the picture.
• 14:35
  SPEAKER_02: He comes to the picture and he looks at it and you know tries to uh uh understand if uh you know I'm actually a bad actor or actually a good actor, right?
• 14:43
  SPEAKER_02: So that is one critical piece in the entire thing, right?
• 14:46
  SPEAKER_02: So LLMs essentially are automating, you know, mapping the uh different indicators of the IOCs and the IOS into the mitro attack database, right?
• 14:57
  SPEAKER_02: And assigning a score, and finally also explaining, giving an explainable uh reason uh as to what has happened and what we need to do about that, right?
• 15:07
  SPEAKER_02: So LLMs are very good at that.
• 15:09
  SPEAKER_02: So that is one very important part of the entire architecture, and that is what I wanted to highlight, okay.
• 15:15
  SPEAKER_02: Uh so yeah, so so beyond that, then you have the airflow, like I said, right?
• 15:20
  SPEAKER_02: Uh so that is a key piece again.
• 15:22
  SPEAKER_02: So it is used for orchestrating the inflow of data.
• 15:26
  SPEAKER_02: So we're trying to make sure that you know data is actually coming in, and that is where airflow is helping in orchestration, it is helping in you know, also uh converting the data into the graph format, right?
• 15:39
  SPEAKER_02: It is helping in uh essentially uh orchestring the LLM uh work, right?
• 15:44
  SPEAKER_02: That as well.
• 15:45
  SPEAKER_02: So the entire orchestration is done by LLMs.
• 15:48
  SPEAKER_02: So these are the four or five P uh parts of the entire architecture, which uh is helping this architecture or this particular solution automate the entire uh threat that the networks are actually facing.
• 16:00
  SPEAKER_01: Yeah.
• 16:01
  SPEAKER_02: Okay.
• 16:01
  unknown: Yeah.
• 16:02
  SPEAKER_01: Yeah, very cool.
• 16:03
  SPEAKER_01: That's uh such a unique, from my experience, way of using Airflow, and I think it sounds uh yeah, very effective there.
• 16:10
  SPEAKER_01: Yeah.
• 16:11
  SPEAKER_01: Uh just on uh one last question on this particular use case, and that's what advice would you give to teams who are looking to improve their threat detection and might be interested in this solution that you've presented?
• 16:22
  SPEAKER_02: Yeah, so I think the the key problems that most organizations face when they're trying to deal with uh threat detection is essentially the fact that uh they are unable to or they're not able to deal with the volume of logs that that is actually flowing in.
• 16:36
  SPEAKER_02: Uh they are short staff for one thing, right?
• 16:39
  SPEAKER_02: And even if they are if even they're they're using the traditional methods which are you know like signature-based uh detection of threats, right, and uh stuff like that, right?
• 16:49
  SPEAKER_02: What would happen is the amount of time it takes for them to actually understand what is happening.
• 16:54
  SPEAKER_02: Uh by that time, this threat uh the threat would have actually materialized, right?
• 16:58
  SPEAKER_02: So you would have so some of the you know uh things that a hacker would typically do is you know, he logs into one particular system, and then because uh because there's not enough security built into the system, he would move on to a different system within the different uh system within the same uh same environment, right?
• 17:17
  SPEAKER_02: And that is called lateral movement, right?
• 17:19
  SPEAKER_02: Then the higher level of that would be something like exfiltration.
• 17:23
  SPEAKER_02: So you have the entire data, right?
• 17:25
  SPEAKER_02: And that is what happens in many of the cases where you know organizations say, okay, there's been a uh leak, you know, in terms of uh your data has actually got leaked, right?
• 17:34
  SPEAKER_02: So what happens is uh because they are able to get it get access to the system, they are actually uh the system the data actually going out, and that's called exfiltration.
• 17:43
  SPEAKER_02: So these are things, these are the issues that many of the organizations face, and the reason for that being that they're not able to automate uh the entire process of threat detection, right?
• 17:52
  SPEAKER_02: And this this architecture that I've defined essentially helps in doing that, right?
• 17:57
  SPEAKER_02: So any any if people who are actually facing this these kind of issues, and then many of them, right?
• 18:02
  SPEAKER_02: Uh, from my knowledge, uh, you know, leveraging you know the Graph DBs and the LLMs and the and Airflow, of course, actually helps uh mitigate this very serious problem that we have at this point of time.
• 18:16
  SPEAKER_01: Okay, great.
• 18:17
  SPEAKER_01: Uh well, thank you for going through that super super useful information there.
• 18:21
  SPEAKER_01: Uh I always like to end with a standard final question for all of our guests, and that is what would you most like to see from the Airflow project in the near future?
• 18:29
  SPEAKER_02: Yeah, but good question.
• 18:31
  SPEAKER_02: Uh like I said, uh within my uh in the last uh three, four years, you know, I've had a fairly good experience with Airflow.
• 18:38
  SPEAKER_02: We have been able to launch uh and monitor you know some key and complex uh DAGs, you know.
• 18:46
  SPEAKER_02: We've been able to launch uh Spark workflows, we've been able to uh do machine learning training and stuff like that, right?
• 18:52
  SPEAKER_02: We've also been able to you know host Airflow on our environment and uh you know use different techniques to be able to uh make this work very smoothly.
• 19:02
  SPEAKER_02: But I think one thing that again, uh we we are on Airflow 2.x.
• 19:06
  SPEAKER_02: I've not moved to 3.x and I'm only partially familiar with 3.x.
• 19:11
  SPEAKER_02: I know it was a big release, and uh, but we haven't really moved there yet.
• 19:14
  SPEAKER_02: So one thing that uh that uh I was thinking was you know, given that I have a background and I'm I'm very interested uh on ML AI and cybersecurity, you know, I think and the fact that uh LLMs have become so important in today's world.
• 19:30
  SPEAKER_02: I think LLMs probably should be a first-class citizen and Airflow as well, maybe, right?
• 19:35
  SPEAKER_02: So you have different operators.
• 19:37
  SPEAKER_02: For example, you have the Spark operators, data proxy spark operators, but I don't, I'm not sure if they are you have operators for LLMs and for uh, you know, essentially enabling these cybersecurity uh use cases, right?
• 19:51
  SPEAKER_02: So, for example, if I'm doing something like anomaly detection or trying to uh do something like user uh behavior, uh user entity behavior analysis, right?
• 20:01
  SPEAKER_02: So that right now is dependent on my knowledge of you know of the entire process of the entire industry, right?
• 20:09
  SPEAKER_02: Uh and and essentially from uh Airflow perspective, I think it would be great to have LLMs actually actually as a first-class citizen, and so you could have some operators maybe, you know.
• 20:18
  SPEAKER_02: It's a thought.
• 20:19
  SPEAKER_02: I'm not sure if that is possible or if if you already have that, right?
• 20:23
  SPEAKER_02: But that would be great actually.
• 20:25
  unknown: Yeah.
• 20:26
  SPEAKER_01: Yeah, I think that's a great answer.
• 20:27
  SPEAKER_01: I uh do you think that that's something that the open source project in general is focused on is adding more support for LLM workflows and uh Astronomer has the R Airflow AI SDK.
• 20:39
  SPEAKER_01: Um so that's uh provide some decorators that make it a little easier to interact with LLMs.
• 20:44
  SPEAKER_01: But uh yeah, I think there's definitely a lot of room for for further contributions there.
• 20:49
  SPEAKER_02: Yeah, perfect.
• 20:49
  SPEAKER_02: And apart from LLMs, I I think you could probably also look at the cybersecurity space and say that okay, this is what, for example, this is the kind of model that you would probably want to use for, let's say, anomaly detection or what some of the other use cases.
• 21:04
  SPEAKER_02: I mean, these are use cases which are being used by many organizations, most of the organizations, right?
• 21:09
  SPEAKER_02: So it could be a way forward for Airflow as well.
• 21:12
  SPEAKER_01: Absolutely.
• 21:13
  SPEAKER_01: I love that.
• 21:14
  SPEAKER_01: Okay, well, Karan, it's been really great speaking with you.
• 21:17
  SPEAKER_01: Really appreciate you sharing uh all of your knowledge on this topic.
• 21:21
  SPEAKER_01: Uh, just to close us out, what is the best way for folks to get in touch with you?
• 21:24
  SPEAKER_02: Yeah, I'm I'm very active on LinkedIn.
• 21:26
  SPEAKER_02: Uh uh, so LinkedIn is probably the best way.
• 21:29
  SPEAKER_02: All right, anybody who has questions or who needs you know advice on uh AIM or data engineering for that matter.
• 21:36
  SPEAKER_02: I'm always open to uh connecting with them on LinkedIn and responding to questions.
• 21:42
  SPEAKER_01: Okay, great.
• 21:43
  SPEAKER_01: Well, we'll include that in the show notes.
• 21:45
  SPEAKER_01: Uh Karan, thank you again.
• 21:47
  SPEAKER_01: It's been a pleasure speaking with you.
• 21:48
  SPEAKER_02: Yeah, thank you so much, Kenton.
• 21:50
  SPEAKER_02: It's been a pleasure uh being part of this podcast.
• 21:53
  SPEAKER_00: Thank you.
• 21:56
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 22:02
  SPEAKER_00: You'll find links in the show notes to any resources mentioned in today's show.
• 22:06
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star reveal.