Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri

Transcript

The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: We don't want to put hands on so many things.
• 0:01
  SPEAKER_00: And today, our Airflow calls a data factory job, a Databricks job, a Snowflake or BigQuery or Amazon, you know, API.
• 0:08
  SPEAKER_00: There's no limitation what we build within the pipeline.
• 0:11
  SPEAKER_00: Within the organization, it is more like anybody wants to do something complex.
• 0:17
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:26
  SPEAKER_02: Everyone, welcome to the Data Flowcast.
• 0:29
  SPEAKER_02: I'm your host, Mark Lomati, and today we have a fantastic guest, Shiva, Cloud Data Architect at Win Las Vegas.
• 0:35
  SPEAKER_02: How are you doing?
• 0:36
  SPEAKER_00: Hey, I'm doing great.
• 0:37
  SPEAKER_00: I'm super excited for this data flowcast.
• 0:40
  SPEAKER_00: I'm excited and then waiting to learn from community and shared mindsights.
• 0:45
  SPEAKER_02: Me too.
• 0:45
  SPEAKER_02: I'm actually very excited, honestly, to have you, Shiva, because you know I remember when I saw you on stage at Las Vegas, you were explaining all the great stuff that you do with Airflow at Wins.
• 0:55
  SPEAKER_02: So I have so many questions for you.
• 0:56
  SPEAKER_02: I hope you are ready.
• 0:57
  SPEAKER_02: I know it's pretty early for you, so hope you are ready for the questions.
• 1:02
  SPEAKER_00: Go for it.
• 1:02
  SPEAKER_02: So just before getting started, before jumping into the discussion, I just would like to give some primary takeaway that people would get from this episode.
• 1:10
  SPEAKER_02: So at the end of this episode, you will have a better understanding of how Airflow is used in the real-world context at WIN Las Vegas.
• 1:16
  SPEAKER_02: Some insights into the learning curve for Airflow.
• 1:19
  SPEAKER_02: I know, Shiva, that you have a great perspective on that.
• 1:22
  SPEAKER_02: Learning about the practical application of custom operators.
• 1:25
  SPEAKER_02: I know that you use custom operators and they are widely used in the Airflow world.
• 1:30
  SPEAKER_02: And last but not least, maybe we will touch on some data quality strategies if we have time on that.
• 1:35
  SPEAKER_02: I know it's a hot and difficult topic, but it's always interesting to learn more about that.
• 1:40
  SPEAKER_02: But I have a question that I like to ask everyone is how did you get started?
• 1:44
  SPEAKER_02: How did you get being cloud data architect at Win?
• 1:48
  SPEAKER_02: And could you tell me what Win is for those who may not know?
• 1:52
  SPEAKER_00: To start with, I'm a cloud data architect at Win Las Vegas.
• 1:56
  SPEAKER_00: I am Sivakrishna Etekuri, and my background is in data engineering, ETL, ELT, and all sorts of low-level programming getting the data pipelines within the industry.
• 2:08
  SPEAKER_00: So, what I do at my current organization is I design and build cutting-edge data pipelines and architectures.
• 2:16
  SPEAKER_00: I enable my team to deliver high-impact data solutions that drive business success.
• 2:23
  SPEAKER_00: So that's a high level about what I do here.
• 2:26
  SPEAKER_00: And we are a luxury casino resort.
• 2:29
  SPEAKER_00: Being a luxury casino resort, we itself is a hub of so many industries.
• 2:34
  SPEAKER_00: We are gaming, we are entertainment, we are retail, we do marketing and sales, we have nightlife.
• 2:40
  SPEAKER_00: It's a hub of so many things.
• 2:41
  SPEAKER_00: It's not a single industry being a casino resort.
• 2:44
  SPEAKER_00: So at Win, knowing that there are a lot of industries, we get so much of data from these different places.
• 2:50
  SPEAKER_00: There is overlapping data and all.
• 2:53
  SPEAKER_00: What we are trying to do at Win, or what we are currently doing, is building a data platform that helps drive service, marketing strategy based on the data, based on the insights out of our customers.
• 3:05
  SPEAKER_00: It's no more a typical reporting platform.
• 3:08
  SPEAKER_00: You know, it's no more a platform where you do your typical tableau dashboards and so on.
• 3:13
  SPEAKER_00: So the goal being right offer, right customer, right channel, right time, and right place.
• 3:19
  SPEAKER_02: Now just tell us more, actually, you were talking about all the great and many domains that you are working on, right?
• 3:25
  SPEAKER_02: In a casino.
• 3:26
  SPEAKER_02: I would love to learn more about how do you use Airflow within your organization.
• 3:30
  SPEAKER_02: Like what specific use cases or projects have you implemented using Airflow?
• 3:34
  SPEAKER_02: Because I imagine that with the nightlife, as you said, with the, I don't know, maybe the slot machines.
• 3:40
  SPEAKER_02: I imagine that you have a lot of different use cases.
• 3:42
  SPEAKER_02: So can you tell us more about that?
• 3:44
  SPEAKER_00: Yeah.
• 3:44
  SPEAKER_00: Like I was saying, we get a lot of data in.
• 3:47
  SPEAKER_00: The goal being, the goal of the platform is to drive the service and the marketing strategy and improve the customer experience, right?
• 3:54
  SPEAKER_00: We get data from different places in within the industry.
• 3:58
  SPEAKER_00: We have so many source systems.
• 4:00
  SPEAKER_00: So what we are doing is we are getting the data in, getting the insights out of a customer.
• 4:06
  SPEAKER_00: And the goal being to track what our customers are doing as they are going through the customer journey, identify who they are, what they have done in the past, and eventually, hopefully not too far in the future, put some machine learning models to understand what they want to do, knowing that what they want to do, we can drive towards a certain set of objectives.
• 4:28
  SPEAKER_00: You know, that's a goal.
• 4:30
  SPEAKER_00: But as a base, we are building a platform where we can get all this data into the platform.
• 4:35
  SPEAKER_00: Airflow is our backbone of our data ingestion, curation, and then running different processes through different systems.
• 4:42
  SPEAKER_00: It does that integration across different systems.
• 4:45
  SPEAKER_00: It's our core component within the platform.
• 4:47
  SPEAKER_00: So using Airflow, we do all of this stuff.
• 4:50
  SPEAKER_00: We get our data in, we do the curation, we integrate different services so that the objectives that we are trying to do, the marketing strategies or giving the better customer experience, is what we are trying to achieve using Airflow within the platform.
• 5:04
  SPEAKER_02: Do you have like maybe an example of that?
• 5:06
  SPEAKER_02: Like an example of a pipeline maybe that actually increases the customer experience?
• 5:11
  SPEAKER_02: Do you have an example in mind?
• 5:13
  SPEAKER_02: I'm curious.
• 5:14
  SPEAKER_02: I don't know if I can go more into the detail.
• 5:16
  SPEAKER_02: Oh, okay, okay, okay.
• 5:17
  SPEAKER_02: That's fine, that's fine.
• 5:19
  SPEAKER_02: I'm very curious.
• 5:20
  SPEAKER_02: I'm very curious.
• 5:20
  SPEAKER_00: Yeah, but I can talk the technical side of it, you know, what we do within our pipelines.
• 5:25
  SPEAKER_00: Today, like I was saying, Airflow is our core component for all our data movement or processing or powering our workflow management for different data pipelines.
• 5:34
  SPEAKER_00: Snowflake is our data warehouse within the platform.
• 5:37
  SPEAKER_00: And the source systems are very different.
• 5:39
  SPEAKER_00: We get from PigQuery or we get from Redshift, we get from plat files, APIs, you know, so many sources, but the target is being Snowflake and Data Lake.
• 5:48
  SPEAKER_00: So what we do within Airflow, we have operators within Airflow that everybody knows that Airflow is built with tag tasks, and every task has a certain operator.
• 5:57
  SPEAKER_00: Within the operator, we do not use the regular operator that community is giving.
• 6:03
  SPEAKER_00: We always build something, you know, where the operator is pretty extensive in what it does.
• 6:08
  SPEAKER_00: We wrap around some custom logic so that it enhances that what we are doing.
• 6:13
  SPEAKER_00: For instance, we have a Snowflake custom operator.
• 6:15
  SPEAKER_00: The moment you trigger any Snowflake job, be it getting some moving some data from one table to other or writing some business logic, it logs some metrics.
• 6:24
  SPEAKER_00: You know, it says, hey, for so and so gaming data or for an entertainment data, I started a process and this is the time I started, and these are the road counts I'm seeing from source, you know.
• 6:36
  SPEAKER_00: And then if it fails, it logs zero.
• 6:38
  SPEAKER_00: If it is success, it logs zero with all those metrics.
• 6:40
  SPEAKER_00: Having all those metrics, we know exactly what process started when.
• 6:45
  SPEAKER_00: We know that Airflow even gives this metrics, but we want more than that what Airflow is giving.
• 6:50
  SPEAKER_00: For instance, you can always backfill with data processing, start data interval, start and end times, data and all.
• 6:56
  SPEAKER_00: But sometimes you want to go back in time, process some custom data because some business logic did not go as you expected, and you want to re-execute your data pipelines with a certain data processing start and end times, you need some custom framework.
• 7:10
  SPEAKER_00: You cannot always go with what you are given.
• 7:12
  SPEAKER_00: And that's the case with everyone.
• 7:13
  SPEAKER_00: So we have built a metadata database other than what Airflow has, where it captures everything and it drives the Airflow workflows.
• 7:21
  SPEAKER_00: You know, it exactly says so-and-so workflow execute on a so-and-so execution date with so-and-so processing start and end times and with a given input metadata.
• 7:31
  SPEAKER_00: And we use this metadata with chinja templating, passing to the tasks and operators, and use the task instance and also use the metadata within the task instance to get into our metadata database.
• 7:42
  SPEAKER_00: Thus a kind of level we use, and and we use data sets and we are pretty excited to use Airflow 2.9.
• 7:48
  SPEAKER_00: We are in the process of migrating to the Astro runtime 11.0 so that we get 2.9.
• 7:54
  SPEAKER_00: We are pretty excited, the combination of the schedule and then the data sets together.
• 8:00
  SPEAKER_00: Yeah.
• 8:01
  SPEAKER_00: Previously we were using sensors or dataset dependencies and all that stuff to achieve the schedule plus dataset dependencies.
• 8:08
  SPEAKER_00: Now we are pretty excited to use get away from sensor and then use a combination.
• 8:11
  SPEAKER_02: Yeah, I mean, there are actually so many exciting new features in 2.9.
• 8:14
  SPEAKER_02: I think you're gonna love it.
• 8:16
  SPEAKER_02: But what you are saying about custom operators is very interesting because, you know, for example, I know that many companies use custom operators and as a way to abstract the complexity of some of them.
• 8:25
  SPEAKER_02: For example, as a data scientist, I don't care about you know setting the number of CPUs or memory and so on.
• 8:31
  SPEAKER_02: And so for that, you might have a custom operator on top of the Kubernetes pod operator.
• 8:36
  SPEAKER_02: But it looks like for you it's a little bit different.
• 8:38
  SPEAKER_02: I feel it's more regarding the reporting and metrics side, almost like the monitoring side as well.
• 8:45
  SPEAKER_02: The reason why you build those custom operators.
• 8:48
  SPEAKER_02: Actually, can you tell us more about the experience that you have so far with monitoring and troubleshooting Airflow pipelines and how your custom operators help in that domain?
• 8:57
  SPEAKER_00: The monitoring part is pretty straightforward in Airflow.
• 9:01
  SPEAKER_00: Whenever you fail there's a failure in your task or in your DAG, it's pretty straightforward.
• 9:06
  SPEAKER_00: You clear your DAG or you clear your task downstream recurse.
• 9:09
  SPEAKER_00: There's so many options for you to fix your Airflow DAG whenever there is an issue.
• 9:13
  SPEAKER_00: The monitoring is pretty straightforward, but apart from what Airflow gives as part of monitoring, we use Airflow REST API to get the data out of the Airflow system, like your DAG status, task status.
• 9:24
  SPEAKER_00: Being a very big platform, it's not just Airflow we want to monitor, right?
• 9:29
  SPEAKER_00: We want to monitor our other systems.
• 9:31
  SPEAKER_00: So, what we do, we get all kinds of metrics into our Snowflake, be it our Databricks job statuses, our Snowflake tasks.
• 9:39
  SPEAKER_00: You know, there are different processes that we want to get alerted.
• 9:43
  SPEAKER_00: Since, like I was saying before, we are not just a reporting platform.
• 9:46
  SPEAKER_00: A lot of our Airflow workflows are directly doing the operational processes like sending an offer to a guest, you know, it triggers very complex process which directly hits the revenue.
• 9:57
  SPEAKER_00: So we are pretty careful in what we do.
• 9:59
  SPEAKER_00: We have ICE on the platform all the time.
• 10:01
  SPEAKER_00: So we get all the metrics of data from Airflow, like your DAC status, task status, your dataset events.
• 10:08
  SPEAKER_00: We get everything and then report on top of it.
• 10:10
  SPEAKER_00: And we also have a custom metadata database, we get other metrics out of there.
• 10:14
  SPEAKER_00: So the monitoring is pretty good today.
• 10:17
  SPEAKER_00: We have an internal product where we manage all the monitoring part of it in one place.
• 10:23
  SPEAKER_00: And that's it on the monitoring part.
• 10:25
  SPEAKER_00: Like I said, fixing any daggone task is as simple as clear it, you're good to go.
• 10:32
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:41
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:53
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
• 11:01
  SPEAKER_02: Okay, that's cool.
• 11:02
  SPEAKER_02: That's really cool.
• 11:02
  SPEAKER_02: I remember that last time we spoke, I think it was like a few days ago, I was actually really surprised by what you said about the learning curve of Airflow.
• 11:11
  SPEAKER_02: And I remember that you took a very nice example.
• 11:14
  SPEAKER_02: So can you tell us more about that?
• 11:15
  SPEAKER_02: What do you think about the learning curve of Airflow?
• 11:18
  SPEAKER_02: And I know if I'm not wrong, but I think you come from a legacy scheduler like Control-M, I believe.
• 11:25
  SPEAKER_01: Yeah, yeah.
• 11:26
  SPEAKER_02: So can you tell us more about how was the shift and why the learning curve wasn't that hard for you moving from Control M to Airflow?
• 11:34
  SPEAKER_00: Sure.
• 11:34
  SPEAKER_00: Everyone knows that Airflow is a workflow management system that powers your data workflows.
• 11:40
  SPEAKER_00: But when you look at Airflow in a way that you understand Airflow, that Airflow has DAGs, tasks, operators, it seems pretty complex to someone who is pretty new to Airflow.
• 11:51
  SPEAKER_00: Hey, what is this DAG, task, operators?
• 11:54
  SPEAKER_00: I have never heard of it.
• 11:55
  SPEAKER_00: And then they get into the architecture, you know, there is a web server, there is a scheduler, there is a trigger, there is a metadata database.
• 12:02
  SPEAKER_00: People learn architecture and then get into DAG.
• 12:05
  SPEAKER_00: But if you lay out the fundamentals of Airflow in such a way in a limited terms and correlate with something in your daily lives, it becomes much simpler.
• 12:17
  SPEAKER_00: If you think of processing something is a data pipeline, you can correlate with a DAG.
• 12:22
  SPEAKER_00: And every data pipeline will have an activity.
• 12:24
  SPEAKER_00: It is more of moving a data from one place to other or triggering an API or doing something.
• 12:28
  SPEAKER_00: Everything is an activity within a data pipeline.
• 12:31
  SPEAKER_00: So that activity you can call it a task.
• 12:33
  SPEAKER_00: And within a task or within an activity, you're doing something with beat if you're running a Snowflake script, then you're doing something in Snowflake.
• 12:41
  SPEAKER_00: So you need an operator.
• 12:42
  SPEAKER_00: Think of what you're doing within an activity is an operator.
• 12:46
  SPEAKER_00: And every operator needs some connection, some script.
• 12:49
  SPEAKER_00: These are all parameters that you pass to the task.
• 12:51
  SPEAKER_00: You know, if you think of it that way, it is very simpler in understanding the airflow.
• 12:57
  SPEAKER_00: But if you look at in a very technical way of writing a DAG, you start with DAG and you provide schedule, you know, all of that stuff.
• 13:05
  SPEAKER_00: People look more Python way of it.
• 13:07
  SPEAKER_00: So I always look at this way, right?
• 13:09
  SPEAKER_00: You first decide what you want and then pick your operators, pick your DAG name, put it in very lame terms in what you want, and then start writing it becomes very simpler.
• 13:18
  SPEAKER_00: Today, a lot of our data engineers that are working on Airflow today, they have no experience with Airflow before.
• 13:24
  SPEAKER_00: And they are pretty comfortable in writing DAGs because the way we have set up the platform, it is incredibly simple to write a DAG.
• 13:33
  SPEAKER_00: And we also have some programs that automatically generate the DAG code given you when you provide the right metadata.
• 13:38
  SPEAKER_00: Like I said, we have a metadata database that plays a key role.
• 13:41
  SPEAKER_00: You provide the right metadata in write tables in that, it will generate 98% of your code.
• 13:46
  SPEAKER_00: All you have to do is change little bits of things here and there, your dependencies and all that stuff.
• 13:50
  SPEAKER_00: We built a skeleton, and within the skeleton, there are placeholders for your tasks and dependencies.
• 13:56
  SPEAKER_00: And even for the dependencies, within our metadata table, you provide the order of sequence and all that stuff.
• 14:03
  SPEAKER_00: In the metadata database, it automatically builds that DAG.
• 14:06
  SPEAKER_00: And our first 60, 70 DAGs that we put in the platform, they are generated DAGs.
• 14:10
  SPEAKER_00: Nobody wrote it.
• 14:11
  SPEAKER_02: That's awesome.
• 14:12
  SPEAKER_02: Do you have like a UE or something?
• 14:14
  SPEAKER_02: So you can just generate those DAGs out of the box?
• 14:17
  SPEAKER_00: We use a Python script, basically.
• 14:19
  SPEAKER_02: Okay, okay, okay.
• 14:20
  SPEAKER_00: We use a Python script, we use ODB C driver to connect to our SQL server database, get the metadata, and then we have a skeleton template.
• 14:28
  SPEAKER_00: We use the template to generate the DAG and store the DAG files back into the ADLS and somebody downloads a file and commits into the repo.
• 14:35
  SPEAKER_00: It is as simple as that.
• 14:37
  SPEAKER_02: That's amazing.
• 14:37
  SPEAKER_02: I love it.
• 14:38
  SPEAKER_02: I love it.
• 14:38
  SPEAKER_02: And I'm curious, maybe you can tell us more about why did you decide to shift from Control-M to Airflow?
• 14:44
  SPEAKER_02: Do you have some insights regarding that?
• 14:46
  SPEAKER_00: Control-M was a long time ago.
• 14:48
  SPEAKER_00: Yeah, it was a long time ago.
• 14:50
  SPEAKER_00: After Control-M, I was using Databricks and Data Factory.
• 14:53
  SPEAKER_00: Data Factory for the orchestration and then Databricks for doing our data transformations and all other analytical work.
• 15:00
  SPEAKER_00: In my experience, when I was working in Data Factory, for me to build a multi-cloud job where I have to talk to Amazon Redshift and then write something into Azure, maybe take that into write it into BigQuery and all that stuff, building that in Data Factory is not an easy job.
• 15:19
  SPEAKER_00: At least not an easy at that time.
• 15:21
  SPEAKER_00: Maybe things change a little bit now.
• 15:23
  SPEAKER_00: But building that within Data Factory, you need a really good expert who knows a lot of Data Factory.
• 15:31
  SPEAKER_00: But doing the same thing in Airflow, you don't need an expert.
• 15:34
  SPEAKER_00: You just pick your operators and then do the job.
• 15:36
  SPEAKER_00: Building such kind of complex workflows where you call something in on-prem, then cloud, and then call an API and all other stuff.
• 15:43
  SPEAKER_00: You know, having that programmatic dependencies, passing the metadata between the tasks and all that stuff is very easy for us within Airflow.
• 15:52
  SPEAKER_00: Today we have Data Factory, but we don't use Data Factory for orchestration.
• 15:55
  SPEAKER_00: We trigger pipelines within Data Factory from Airflow.
• 15:59
  SPEAKER_00: So if we want one central workflow orchestration tool, we don't want to put hands on so many things.
• 16:04
  SPEAKER_00: And today, our Airflow calls a Data Factory job, a Databricks job, a Snowflake or BigQuery or Amazon, you know, API.
• 16:11
  SPEAKER_00: There's no limitation what we build within the pipeline.
• 16:14
  SPEAKER_00: Within the organization, it is more like anybody who wants to do something complex, hey, reach out to the data team.
• 16:20
  SPEAKER_00: They have a way.
• 16:22
  SPEAKER_00: That's right.
• 16:24
  SPEAKER_02: People wonder like how scalable Airflow is.
• 16:26
  SPEAKER_02: How many tasks can I run with Airflow?
• 16:29
  SPEAKER_02: How many DAGs can I run with Airflow?
• 16:31
  SPEAKER_02: Could you just tell us maybe some numbers here, like how many DAGs maybe you run in average at Win, just to give a better idea for some people?
• 16:40
  SPEAKER_00: We have within our organization, I think we have at least 200 plus DAX today that run in production, and each DAG, some of the DACs has more than 100 and plus tasks within that DAG.
• 16:52
  SPEAKER_00: Okay.
• 16:53
  SPEAKER_00: And we have never seen any instance of any DAG running slow or any task not running faster because of this so many DACs or so many tasks.
• 17:03
  SPEAKER_00: Take an average, 200 DACs.
• 17:05
  SPEAKER_00: Even if you take 40, there are 40 tasks within every DAG as an average.
• 17:10
  SPEAKER_00: You see how many tasks we are running a day.
• 17:12
  SPEAKER_00: And these numbers, if you run just once a day, but there are DACs that are running very frequently.
• 17:18
  SPEAKER_00: It's very scalable.
• 17:19
  SPEAKER_00: And we are running with a very low configuration.
• 17:21
  SPEAKER_00: We are not powered with so many Kubernetes clusters and all that stuff.
• 17:25
  SPEAKER_00: We are running on very low configuration.
• 17:27
  SPEAKER_00: We have never seen that issue because we offload so much work to the operators, not to the airflow.
• 17:32
  SPEAKER_02: Okay.
• 17:32
  SPEAKER_02: That's actually a good point.
• 17:33
  SPEAKER_02: That's a really good point.
• 17:34
  SPEAKER_02: You should definitely offload using the operators and not do all the working in Airflow.
• 17:39
  SPEAKER_02: And something else maybe we can touch on is, you know, local development.
• 17:44
  SPEAKER_02: How does it work?
• 17:45
  SPEAKER_02: Do you have some insights regarding that?
• 17:47
  SPEAKER_02: How does it work at Win?
• 17:48
  SPEAKER_02: If I want to develop my DAG, do you have like a development environment for that?
• 17:52
  SPEAKER_02: Do you have a local development for every data engineer?
• 17:55
  SPEAKER_02: Can you tell us more a little bit about that?
• 17:58
  SPEAKER_00: At Win, we develop DAGs in two ways.
• 18:01
  SPEAKER_00: Most of our DAGs are auto-generated for some of them.
• 18:04
  SPEAKER_00: We have set foundation in such a way, whenever there is a new DAG, if there is an already existing pattern, people just take it, change the tasks or task names and then metadata, and then just come into the DAG in the repo and test it.
• 18:18
  SPEAKER_00: It is as simple as that to introduce a new DAG.
• 18:21
  SPEAKER_00: But for someone to start fresh on a new requirement where such kind of DAG never exists, all of our data engineers have Docker installed on the machine.
• 18:30
  SPEAKER_00: They have Astro CLI Docker installed.
• 18:32
  SPEAKER_00: They start the project and everybody has access to the dev version of the key vault.
• 18:38
  SPEAKER_00: We store all our connections in the key vault.
• 18:40
  SPEAKER_00: Nobody has to set up any environment variables or anything for the connections.
• 18:44
  SPEAKER_00: All of our connections come from Key Vault.
• 18:47
  SPEAKER_00: So they have that service principle within Azure, which talks to the Key Vault and get them all the connections.
• 18:53
  SPEAKER_00: The secret backend for us is the Azure Key Vault and they have the requirement file with all the dependencies.
• 19:00
  SPEAKER_00: They test the code locally and then push the code to the Azure DevOps report.
• 19:05
  SPEAKER_00: From there, we have CICD pipelines which deploy to different deployments within Astro.
• 19:10
  SPEAKER_02: I mean there are so many different ways of doing that.
• 19:12
  SPEAKER_02: And I know that a lot of different companies also implement their way with some really custom solutions sometimes.
• 19:19
  SPEAKER_02: It's not an easy topic.
• 19:21
  SPEAKER_02: Solving local development environments is something really hard.
• 19:25
  SPEAKER_00: We have our difficulties, but it's going smooth.
• 19:27
  SPEAKER_00: There's no such complications.
• 19:29
  SPEAKER_00: Having Astro CLI is much simpler for you to start the environment and then start creating your DAX.
• 19:37
  SPEAKER_02: That's actually really true.
• 19:38
  SPEAKER_02: This is something that I keep pushing forward.
• 19:40
  SPEAKER_02: Honestly, the Astro CLI is amazing.
• 19:43
  SPEAKER_02: Like people think that it's not open source, but it is free.
• 19:45
  SPEAKER_02: Everyone can use it.
• 19:46
  SPEAKER_02: And it helps a lot.
• 19:48
  SPEAKER_02: It definitely helps a lot.
• 19:50
  SPEAKER_02: And can we talk a little bit about data quality?
• 19:53
  SPEAKER_02: It's a really hard, difficult topic.
• 19:56
  SPEAKER_02: But is it something that you have already implemented at Win?
• 20:00
  SPEAKER_02: Is it something that you are thinking about implementing?
• 20:02
  SPEAKER_02: Can you tell us more about data quality at Win?
• 20:05
  SPEAKER_02: How do you manage that?
• 20:06
  SPEAKER_00: For sure it's a hard topic, especially data quality.
• 20:10
  SPEAKER_00: At Win, we are still in the process of implementing it.
• 20:12
  SPEAKER_00: A lot of our data comes from source systems that are pretty relational.
• 20:18
  SPEAKER_00: We only have 20% of source systems that comes from flat files and other manual spreadsheets.
• 20:24
  SPEAKER_00: But most of our data comes from a relational system.
• 20:28
  SPEAKER_00: So where the constraints are pretty enforced, you know, we have keys, any database or table constraints already enforced.
• 20:34
  SPEAKER_00: The initial part of the platform, we did not spend uh freshness anomaly, uh, which predicting uh, you know, kind of like a future ends.
• 20:43
  SPEAKER_00: Your table is supposed to load at 8 a.m., 10 a.m.
• 20:45
  SPEAKER_00: You don't send within on your table.
• 20:47
  SPEAKER_00: There's no insert, no update, there's no delete.
• 20:50
  SPEAKER_00: The table, it's already 10 a.m., loads at 9 a.m.
• 20:52
  SPEAKER_00: every day, uh, Amazon DQ and then other custom data quality checks.
• 20:57
  SPEAKER_00: But from the data quality side, we are still going towards it.
• 21:00
  SPEAKER_00: Our custom metadatabase and the Snowflake load history tables already have some metrics towards row counts and when each table is getting loaded.
• 21:08
  SPEAKER_00: We are working towards two ML models in coming months.
• 21:11
  SPEAKER_00: One model is about the volume anomaly where you have a table where it updates every day with 100 records, and today there's no update.
• 21:22
  SPEAKER_00: It's a volume anomaly.
• 21:24
  SPEAKER_00: Right?
• 21:24
  SPEAKER_00: And then suddenly there are 1 million records populated to the table, which never happened, it's an anomaly.
• 21:30
  SPEAKER_00: We are working towards that ML model using isolation for us, and then we are working on a different model called freshness anomaly, which is predicting future events.
• 21:39
  SPEAKER_00: Your table is supposed to load at 8 a.m.
• 21:41
  SPEAKER_00: It's already 10 a.m.
• 21:42
  SPEAKER_00: You don't see any data or any operation within on your table.
• 21:45
  SPEAKER_00: There's no insert, there is no update, there's no delete, no operation on your table.
• 21:49
  SPEAKER_00: It's already 10 a.m.
• 21:50
  SPEAKER_00: It is on an average, it loads at 9 a.m.
• 21:53
  SPEAKER_00: every day.
• 21:53
  SPEAKER_00: It's a freshness anomaly.
• 21:55
  SPEAKER_00: So we are working on that model.
• 21:57
  SPEAKER_00: These are the two models as a priority we are working on, other than data quality.
• 22:02
  SPEAKER_00: So data quality is next in the pipeline.
• 22:04
  SPEAKER_00: But it's always a challenge to even put what data quality check you want.
• 22:08
  SPEAKER_02: Oh, yeah.
• 22:08
  SPEAKER_00: The hardest part is not implementing the data quality, the hardest part is knowing what data quality check you want.
• 22:15
  SPEAKER_00: It's a balance between managing what your business needs versus focusing on something so technical, data quality check when nobody wants to.
• 22:26
  SPEAKER_02: And out of curiosity, you just said that you are about to implement those machine learning models.
• 22:30
  SPEAKER_02: Are you thinking of doing them in Airflow?
• 22:32
  SPEAKER_00: Airflow will be orchestrating the data work, the machine learning models.
• 22:37
  SPEAKER_00: The actual machine learning models will be on Snowflake or Dataplic.
• 22:41
  SPEAKER_02: Awesome.
• 22:42
  SPEAKER_02: Well, I think honestly, I mean, we went through a lot of great topics, but I I do have one last question for you.
• 22:49
  SPEAKER_02: How Airflow has been so far for you?
• 22:51
  SPEAKER_00: It it is great.
• 22:53
  SPEAKER_00: No complaints on the platform itself.
• 22:55
  SPEAKER_00: It is super great.
• 22:56
  SPEAKER_00: Like I was saying, it's super incredible simple to introduce any workflow.
• 23:01
  SPEAKER_00: And our team loves it.
• 23:03
  SPEAKER_00: And they are very proud of what they are building every day.
• 23:06
  SPEAKER_00: And it is super easy to learn.
• 23:08
  SPEAKER_00: I was telling in the other topic.
• 23:09
  SPEAKER_00: And today our platform is 100% running on Airflow with no issues.
• 23:15
  SPEAKER_00: We are happy with it.
• 23:16
  SPEAKER_02: Here we go.
• 23:17
  SPEAKER_02: If you are happy, I'm happy.
• 23:20
  SPEAKER_02: That's awesome.
• 23:21
  SPEAKER_00: And thanks to the community with providing so many operators, introducing new Airflow versions with different features.
• 23:28
  SPEAKER_00: We are super happy about what we are using today here.
• 23:31
  SPEAKER_02: I think that's what I love as well on Airflow is the community.
• 23:34
  SPEAKER_02: As you said, like it keeps growing and all of those new features coming is really great.
• 23:38
  SPEAKER_02: Thank you so much for joining.
• 23:40
  SPEAKER_02: Shiva, it was great.
• 23:42
  SPEAKER_02: Honestly, it was really great.
• 23:43
  SPEAKER_02: I had a lot of pleasure talking with you.
• 23:45
  SPEAKER_02: What's the best way for folks to get in touch with you?
• 23:49
  SPEAKER_00: You can find me on Astronomer's Champion page.
• 23:52
  SPEAKER_00: You can also search search me on LinkedIn.
• 23:55
  SPEAKER_00: My name is Sivakrishna Eti Kodi.
• 23:57
  SPEAKER_00: You can find me on LinkedIn as a cloud data architect to win last week.
• 24:01
  SPEAKER_00: Thank you again for this opportunity.
• 24:02
  SPEAKER_00: I'm super excited and want to see how it goes.
• 24:07
  SPEAKER_02: Thank you so much.
• 24:08
  SPEAKER_02: Thank you so much, thank you.
• 24:10
  SPEAKER_02: A good one.
• 24:10
  SPEAKER_02: Take care.
• 24:12
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:19
  SPEAKER_01: You'll find links in the show notes to any researches mentioned in today's show.
• 24:23
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star reveal.