Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Transforming Data Pipelines at XENA Intelligence with Naseem Shah

Transcript

Transforming Data Pipelines at XENA Intelligence with Naseem Shah · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_03: I think curiosity should lead the learning, right?
• 0:04
  SPEAKER_03: So some since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
• 0:12
  SPEAKER_03: So resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
• 0:19
  SPEAKER_03: So when I learn something new, my equation or framework is that like watch a crash course, find what it's used for, and understand what's happening here.
• 0:34
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:44
  SPEAKER_00: Here, Airflow community, we've got exciting news.
• 0:48
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:51
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow and the free Airflow 3 Fundamentals or Demethrine certification.
• 1:05
  SPEAKER_00: Don't miss out.
• 1:06
  SPEAKER_00: The survey link is in the show notes and closes November 21st.
• 1:12
  SPEAKER_02: Hello everyone.
• 1:13
  SPEAKER_02: Welcome to the Dela Flowcast.
• 1:15
  SPEAKER_02: I'm your host, Mark Lumberti, and I'm super excited because today I'm joined by Nassim Shah, head of engineering at Xena Intelligence.
• 1:24
  SPEAKER_02: I mean, the first question is how are you doing?
• 1:26
  SPEAKER_03: Hi, it's good.
• 1:27
  SPEAKER_03: I'm really excited for this.
• 1:29
  SPEAKER_03: It was a really great day.
• 1:30
  SPEAKER_03: It was raining and all, and it was such a beautiful day to start on.
• 1:35
  SPEAKER_03: And you know how things go and had a great day at work, did a lot of problem solving and all.
• 1:41
  SPEAKER_03: It's exciting stuff.
• 1:42
  SPEAKER_03: So yeah, never bothered.
• 1:45
  SPEAKER_03: That's awesome.
• 1:45
  SPEAKER_02: Well, I'm really glad to be here with you today because we're actually gonna talk about a lot of exciting stuff, more specifically, how you and obviously your team also transformed from Chrome jobs to sophisticated LLM-powered review analysis using obviously AI.
• 2:02
  SPEAKER_02: And so you will discover also how Nassim and the team is able to process millions of Amazon reviews.
• 2:10
  SPEAKER_02: So this is super exciting, and obviously, everything is orchestrated using Airflow.
• 2:16
  SPEAKER_02: So I would like to start with your background.
• 2:20
  SPEAKER_02: Okay, so can you tell us about your journey into data engineering and what led you to discover Airflow?
• 2:26
  SPEAKER_02: And also what were some of the data pipeline challenges that you were facing before adopting Airflow?
• 2:33
  SPEAKER_03: So I come from a software engineering background, not really a data engineer as per se, when situations and things got into data engineering related works also.
• 2:44
  SPEAKER_03: So I actually come from more of like a software engineering background.
• 2:47
  SPEAKER_03: I didn't know much about the world of data engineering at the start.
• 2:50
  SPEAKER_03: I joined intelligence straight out of college as a first engineering hire.
• 2:54
  SPEAKER_03: So basically, as a founding engineer, so you have to gather things around and learn new things and the requirements from the stakeholders, from the founders, the CEO, director, reporting to them, directors.
• 3:06
  SPEAKER_03: Like you have to find things on your own.
• 3:09
  SPEAKER_03: And that gave me a lot of exploring and gave me a lot of freedom to explore and find things on my own and find it my own way.
• 3:18
  SPEAKER_03: So that was really exciting for me and straight off a college.
• 3:22
  SPEAKER_03: It's something that you don't get when you join an MNC.
• 3:25
  SPEAKER_03: The first that's how I come into this picture.
• 3:28
  SPEAKER_03: So my role earlier was to build MVPs and get product out of the ground, which ended up helping us raising funds and join accelerators like Techstars.
• 3:39
  SPEAKER_03: A little bit about the company.
• 3:41
  SPEAKER_03: So Xena is an AI-powered e-commerce ground platform.
• 3:44
  SPEAKER_03: So everything we do is really data heavy.
• 3:47
  SPEAKER_03: So we would pull in data from marketplaces like Amazon, Walmart, Shopify to generate insights for things like ad optimization, inventory planning, inventory focused, team market research, customer behavior, and all those stuff, right?
• 4:02
  SPEAKER_03: So in the beginning, our pipelines were just like ground scripts, right?
• 4:06
  SPEAKER_03: So honestly, it worked fine for a few customers, especially if we were like pre-revenue beta testing and all those stuff, right?
• 4:13
  SPEAKER_03: So initially, my motto is start simple, time update.
• 4:18
  SPEAKER_03: So you don't need to complicate into the first and waste our time and productivity on that, right?
• 4:24
  SPEAKER_03: So my first approach was like start out simple.
• 4:28
  SPEAKER_03: So it's few customers, handful of API calls and downloading report documents and processing it in Python.
• 4:35
  SPEAKER_03: And it was very simple and it was easy, right?
• 4:38
  SPEAKER_03: And we didn't waste much time on that.
• 4:41
  SPEAKER_03: That's why we could ship more features when so when we started to grow and move to product market fit, those crones started to become like a nightmare.
• 4:51
  SPEAKER_03: Scaling out database was easy where backends are easy, you can just containerize and it's fairly easy to scale back end APIs and stuff like that.
• 5:01
  SPEAKER_03: But when it comes to the data pipelines, it was really not that easy and was a very big challenge as a fresher when I started out.
• 5:09
  SPEAKER_03: We had cron jobs failing randomly, no clear visibility, which into let most data, no clear visibility.
• 5:17
  SPEAKER_03: I often find myself logging in through heaps of logs at later, uh trying to find the needle in the haystack kind of situation here.
• 5:27
  SPEAKER_03: So it was very hard to debug and stuff like that.
• 5:30
  SPEAKER_03: So we then started to try a small upgrade.
• 5:34
  SPEAKER_03: We tried small upgrade.
• 5:36
  SPEAKER_03: So that's how I re-architected with again baby steps, right?
• 5:41
  SPEAKER_03: SQS and easy to set up.
• 5:43
  SPEAKER_03: I didn't want to complicate it very fast, but slowly.
• 5:47
  SPEAKER_03: So it was interesting because we had SQS queues, so it's like Kafka queues and stuff like that.
• 5:53
  SPEAKER_03: Just the message comes in and consumer gets it and process the data.
• 5:58
  SPEAKER_03: And it was fairly simple.
• 5:59
  SPEAKER_03: So we had one queue for one report type.
• 6:03
  SPEAKER_03: And for customer, it was like five queues, five consumers.
• 6:07
  SPEAKER_03: It was okay for that time.
• 6:08
  SPEAKER_03: And when we had like sub-20-30 customers, it was okay for us like to go through that.
• 6:15
  SPEAKER_03: But when we started grow and customers flooding in and stuff like that, it was very hard again.
• 6:21
  SPEAKER_03: Like which customer data pipeline go down?
• 6:24
  SPEAKER_03: Why this isn't it?
• 6:26
  SPEAKER_03: Customers start comes reach out, support, and things were not uh looking good.
• 6:32
  SPEAKER_03: And AWS SQS and going to the dead letter queue, myself and our team, and trying to understand.
• 6:40
  SPEAKER_03: And sometimes the code works to customers, it doesn't work for others.
• 6:46
  SPEAKER_03: So the lack of visibility was slowing down the scalability and our health of the platform itself.
• 6:53
  SPEAKER_03: So again, it's not just the features, but we have to deliver it right.
• 6:57
  SPEAKER_03: So that lack of visibility, lack of scalability, and it was not developer-friendly, right?
• 7:05
  SPEAKER_03: So it cost us productivity time and all those different aspects of what we do.
• 7:10
  SPEAKER_03: And we were like a very small team, like we are three engineers just managing all these data pipelines, all these features, and we have to make the stakeholders happy, and we just have to grind it.
• 7:22
  SPEAKER_03: So that's started to research about data engineering.
• 7:25
  SPEAKER_03: That's how we get into data engineering, and that's how I understood all there's a world like this, and there is lots of tools, and that's how I ended up in the airflow, right?
• 7:36
  SPEAKER_03: Okay, it was perfect for us when you started.
• 7:39
  SPEAKER_02: Yeah, wow.
• 7:40
  SPEAKER_02: So I think like many teams will listen to that podcast and will realize that they are getting the same issues that you got in the past before Airflow.
• 7:49
  SPEAKER_02: And some people always wonder, hey, should I use Airflow?
• 7:52
  SPEAKER_02: If you listen, NASIM, you truly understand that if every day you are struggling with logs, with pipelines that you cannot properly monitor, that you cannot properly debug, and so on, obviously, Airflow will be a huge improvement for you.
• 8:06
  SPEAKER_02: And honestly, at some point, when you feel that you are overwhelmed with so many issues and so on, and that your data pipelines are not reliable as they should be, I mean it's time to go with an orchestrator.
• 8:16
  SPEAKER_02: And you know, Nassim, it's interesting because you said that you are a small team, right?
• 8:23
  SPEAKER_02: And many people actually, I wouldn't say many, but some people believe that you need to be a huge team just to set up and run Airflow in production, which is not true.
• 8:33
  SPEAKER_02: But tell me more about your experience with Airflow and more specifically, what was your experience learning Airflow?
• 8:40
  SPEAKER_02: And do you have any particular resources, approaches, or things like that truly helped you in your journey with Airflow and as a newcomer to Airflow?
• 8:48
  SPEAKER_03: Yeah, learning Airflow definitely felt intimidating at first.
• 8:52
  SPEAKER_03: It helped a lot when I came through the software engineering lens because I understood what Docker is.
• 8:58
  SPEAKER_03: I knew all these things, right?
• 9:00
  SPEAKER_03: What are the containers?
• 9:01
  SPEAKER_03: Why Airflow has different containers, and so it clicked me easily since I'm looking through an engineer's thing.
• 9:09
  SPEAKER_03: But again, for a starter, it might be a little hard, even though I knew that it was a little intimidating at first, because what is a lot of things at first?
• 9:20
  SPEAKER_03: As time moved, it was easier.
• 9:22
  SPEAKER_03: So, yeah, definitely.
• 9:23
  SPEAKER_03: I think I went through a lot of engineers go through what people call tutorial health, right?
• 9:28
  SPEAKER_03: So I was watching crash courses, jumping into dogs, trying to piece things together, and it wasn't just clicking right away, but slowly it was the progression was very slow, right?
• 9:39
  SPEAKER_03: What really helped me was stepping back and making sure if I understood the core concepts, right?
• 9:45
  SPEAKER_03: What a DAG actually is, how tasks work, what are the operators, how operator works, how scheduling and dependencies tie all together, what's upstream, what's downstream.
• 9:58
  SPEAKER_03: And once those fundamentals clicked, it was very easy.
• 10:02
  SPEAKER_03: Like when you go through the docs, it was very easy.
• 10:05
  SPEAKER_03: So after that, I was focused less on watching tutorials and practicing with real-world problems.
• 10:12
  SPEAKER_03: I started practicing with real-world small things, let's say small ETL workload, make some work APIs, and just make a DAG for it and try to understand how things are.
• 10:24
  SPEAKER_03: Then you discover things like XCOMs exist, like map tasks, and how do you find concurrency and all those different things?
• 10:32
  SPEAKER_03: So I think curiosity should lead the learning.
• 10:38
  SPEAKER_03: Since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
• 10:47
  SPEAKER_03: So, resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
• 10:55
  SPEAKER_03: So, when I learn something new, my equation or framework is that watch a crash course, find what's its use for, and understand what's happening here.
• 11:05
  SPEAKER_03: Crash course can be like 20 minutes, one hour, but you'll get a capsule uh information of what I think it actually is.
• 11:12
  SPEAKER_03: So you'll discover, for example, what Airflow has to offer, and you just know a mind map of things.
• 11:18
  SPEAKER_03: Okay.
• 11:19
  SPEAKER_03: Then you can start digging it a little bit later.
• 11:23
  SPEAKER_03: Understand core concept directly from docs.
• 11:25
  SPEAKER_03: So that's the best way to start.
• 11:27
  SPEAKER_03: Then sometimes we need some little different perspective of thinking to understand things.
• 11:32
  SPEAKER_03: That's how blogs help us.
• 11:35
  SPEAKER_03: Then, best way to learn is by practicing.
• 11:38
  SPEAKER_03: And for that, I really like the airflow in Docker containers.
• 11:42
  SPEAKER_03: It was very easy.
• 11:43
  SPEAKER_03: You could run a scalable Docker, a mini-scalable Docker inside your local machine and experiment.
• 11:49
  SPEAKER_03: You didn't have to waste compute or resources and all those different things.
• 11:53
  SPEAKER_03: So it was really easy to learn airflow moving on like that.
• 11:58
  SPEAKER_03: And if you broke the Docker, you can just delete the Docker and start new.
• 12:03
  SPEAKER_03: I digged into and broke Airflow a lot of times, and that's how I learned.
• 12:08
  SPEAKER_03: And again, yeah, Chat GPT, Google search, and all everything helped.
• 12:12
  SPEAKER_03: So a big aha moment for us, me like realizing Airflow is less about memorizing syntax and stuff like that, and more about thinking in terms of orchestration, how data moves, when it should move, and what to do when it fails.
• 12:28
  SPEAKER_03: So that's you know, basic things.
• 12:33
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 12:42
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 12:54
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
• 13:20
  SPEAKER_02: So just try to focus on what you actually need and to know that the best way, as you mentioned, is either to go on a crash course, but also just to go on the documentation.
• 13:31
  SPEAKER_02: I think there is like a basic example that goes through how you write your first data pipeline, creating your first task, and so on.
• 13:37
  SPEAKER_02: And that will be more than enough at least for the beginning.
• 13:40
  SPEAKER_02: But this is super cool.
• 13:41
  SPEAKER_02: I would love actually to know more about how did you set up your Airflow deployment?
• 13:46
  SPEAKER_02: Because again, like you are a small team.
• 13:48
  SPEAKER_02: What were the considerations when you decided to set up Airflow in production?
• 13:52
  SPEAKER_02: What decisions did you take?
• 13:54
  SPEAKER_02: Tell me more about your current infrastructure with Airflow.
• 13:57
  SPEAKER_03: So, again, small team is the big point here.
• 14:00
  SPEAKER_03: So, startup, right?
• 14:01
  SPEAKER_03: So, startup mindset is like limited resources, and every infra choice you make is like a trade-off, right?
• 14:08
  SPEAKER_03: You can have really fast DAG but high build, or you can have very slow DAG but convenient building and stuff like that.
• 14:17
  SPEAKER_03: And at a startup infrastructure decisions really come down to being practical.
• 14:23
  SPEAKER_03: You don't have unlimited resources, right?
• 14:25
  SPEAKER_03: So every use has a trade-off.
• 14:27
  SPEAKER_03: So with Airflow, the main thing for us was to find the balance between running things concurrently versus batching them, right?
• 14:34
  SPEAKER_03: You can do just pull the data, and let's say for each row, you can run a map task.
• 14:39
  SPEAKER_03: It's easy, right?
• 14:40
  SPEAKER_03: You can orchestrate it.
• 14:42
  SPEAKER_03: It's also maybe a good standard for some data pipelines and stuff like that.
• 14:47
  SPEAKER_03: But again, more tasks means more complexity, more resources, and more things, right?
• 14:52
  SPEAKER_03: So finding the balance between running things concurrently and batching them was really critical things I thought of when we made DACs and stuff like that.
• 15:04
  SPEAKER_03: Because sure you can make everything super parallel and concurrent, but more tasks and bigger cloud width.
• 15:12
  SPEAKER_03: So on the other hand, batching too much slows things down, but that's a use user experience.
• 15:17
  SPEAKER_03: So we thought of strategies like okay, what comes in the user interfaces and user experience will do is more concurrently and stuff like that.
• 15:28
  SPEAKER_03: But maybe for inventory forecasting and all those different problems, we'll batch it for more customers at once.
• 15:36
  SPEAKER_03: So you can save a lot of resources running that it's not about everything should be in concurrent, or it's like when you when developers start, okay, everything should be fast, super optimized, super scalable, and stuff like that.
• 15:49
  SPEAKER_03: But the latter is resources, build efficiency, are so my thing was like instead of over-engineering from the start, build simplest deployments that work.
• 16:00
  SPEAKER_03: We are treated as we learn.
• 16:02
  SPEAKER_03: So it's like questionnaire do we really need this pipeline to run every five minutes, or should we run this every week?
• 16:10
  SPEAKER_03: So is our league enough or is monthly enough?
• 16:13
  SPEAKER_03: So those kind of questions ask yourself, ask your team, and make trade-off tables and you and the decisions come like that.
• 16:23
  SPEAKER_03: So we didn't have like fanciest deployment setup.
• 16:26
  SPEAKER_03: So we had local Docker setup in everyone's machine, engineer's machine, so they can test it out on local first.
• 16:34
  SPEAKER_03: Then we have this EC2 server where we have again Docker deployed, but again, it works with staging APIs and other it works with real-world data, right?
• 16:45
  SPEAKER_03: Test that and concurrency and benchmark that.
• 16:48
  SPEAKER_03: Then again, for production, we use Google Cloud Composer because we don't have time to every day and it takes a lot of productive time and stuff like that.
• 16:59
  SPEAKER_03: In that way, we thought, okay, managed service helps in those kinds of things.
• 17:05
  SPEAKER_03: And also manage service doesn't mean that your bill will be very big.
• 17:10
  SPEAKER_03: Manage service means like you have to optimize for your data pipelines so that you can.
• 17:15
  SPEAKER_03: So the main fear of managed pipelines is that managed services that bill might get big and stuff like that.
• 17:23
  SPEAKER_03: But it's how you optimize and simplify your DAGs, that's where it matters, right?
• 17:28
  SPEAKER_02: Thank you so much for your sharing.
• 17:30
  SPEAKER_02: How did you come up with that infrastructure as a small team?
• 17:33
  SPEAKER_02: I think that will be super useful for many listeners.
• 17:35
  SPEAKER_02: But I would love to dive into one of your most interesting use cases.
• 17:39
  SPEAKER_02: I don't know if it is the most interesting one, but it's definitely the one that everybody is talking about today in the engineering world, which is anything AI related.
• 17:48
  SPEAKER_02: So, can you tell us about this project and how you architected the solution about orchestrating this pipeline for processing Amazon reviews using your LLM?
• 17:59
  SPEAKER_03: The approach to this problem was very unique, right?
• 18:02
  SPEAKER_03: So it's normally when startups I trade, we need AI, we need that, we need this, like that, and we just rest.
• 18:10
  SPEAKER_03: But this use case was very unique.
• 18:12
  SPEAKER_03: We had a lot of customers, right?
• 18:14
  SPEAKER_03: Some are very small CPG brands, some are medium-sized brands, some are like very large companies.
• 18:19
  SPEAKER_03: So we work with different kinds of startups and different kinds of businesses, and again, it's customers startup, right?
• 18:28
  SPEAKER_03: So we are running on heavy data and stuff like that, right?
• 18:32
  SPEAKER_03: So we had few enterprise customers.
• 18:34
  SPEAKER_03: So we have these products, so that's dashboard products and stuff like that, and we also have enterprise side of things, right?
• 18:42
  SPEAKER_03: So most of our DAGs are for like the actual product, so that's Xena 360 or different e-commerce.
• 18:50
  SPEAKER_03: So you can connect Walmart, Amazon into our dashboard, and we pull in data insights and stuff like that, right?
• 18:56
  SPEAKER_03: So that's our core providing, but we have this enterprise customers with custom and unique use cases, right?
• 19:04
  SPEAKER_03: So again, one of our core ideas is like we solve problems what the brands face, right?
• 19:12
  SPEAKER_03: So when one of our enterprise customers came to us and said, You're launching a new product in this market.
• 19:18
  SPEAKER_03: We are not sure if we want to launch that product into that market, but we would love to know what the market looks like and should we go forward or not.
• 19:27
  SPEAKER_03: So it was very simple use case and stuff like that when we started out.
• 19:32
  SPEAKER_03: So we have a team of data analysts there.
• 19:34
  SPEAKER_03: So we have you a couple of analysts there.
• 19:36
  SPEAKER_03: So these analysts work with customers to find problems and fix them.
• 19:41
  SPEAKER_03: So one of the core things in this providing was that understand the user behavior, understand the user market shopper preference, right?
• 19:53
  SPEAKER_03: So immediately our first choice was uh Amazon reviews, Walmart reviews, and all those.
• 20:00
  SPEAKER_03: We are working with the e-commerce company as well, right?
• 20:02
  SPEAKER_03: So if a company was going to this category, what people think on the best-selling products, right?
• 20:09
  SPEAKER_03: What do people think about specific competitors?
• 20:12
  SPEAKER_03: So all these questions were to be answered, right?
• 20:15
  SPEAKER_03: So we started pulling reviews manually.
• 20:18
  SPEAKER_03: These data analysts are pulling manually, uh copy-pasted each and every review, and again it's text mining, right?
• 20:26
  SPEAKER_03: If it was JSON, we could write some scripts and structure it out and run some plotly or plot some graphs and stuff.
• 20:34
  SPEAKER_03: So you can understand the data.
• 20:36
  SPEAKER_03: Text it was very unique thing.
• 20:39
  SPEAKER_03: Like it to understand, you just need human to read and understand.
• 20:44
  SPEAKER_03: But it's around the same time when Chat GPT came and the LLM boom came, and thought, okay, this is really interesting use case.
• 20:52
  SPEAKER_03: Okay, people would love, and it's for this report, shopper preference report, market research.
• 20:57
  SPEAKER_03: There is also demand for multiple enterprise customers ask in a time to for this, and that's how it drove us to build this solution.
• 21:08
  SPEAKER_03: It was totally manual process, so we started to pull in public reviews through the web crawling with ethical crawlers, proxies, and stuff like that.
• 21:18
  SPEAKER_03: And we had like a large amount of reviews per product, and we had to go through every review and find each of things, and it was interesting.
• 21:29
  SPEAKER_03: Initially, we thought, okay, like post-reviews are four stars and five stars, negative than two stars, but you see a lot of negative stuff in five star reviews.
• 21:40
  SPEAKER_03: I wish I could have less amount of sugar in this, right?
• 21:43
  SPEAKER_03: I wish uh it had this feature, that feature.
• 21:46
  SPEAKER_03: When we started out, okay, prompting with uh against a review and extracting stuff like that.
• 21:52
  SPEAKER_03: So we extracted topics, we extracted themes, negative sentiments, positive sentiments, and we also extracted uh Lot of different why this review deserves four star, five star.
• 22:04
  SPEAKER_03: So we invented new two-star rating.
• 22:08
  SPEAKER_03: So let's say if someone started four three-star, but the review itself has much more good sentiments than we say that okay, this reserve to be 4.5 stars.
• 22:19
  SPEAKER_03: And when you analyze reviews at large, we started to see patterns, right?
• 22:27
  SPEAKER_03: Most of these people said this, and most of this didn't say this, and some people said this, and the importance of and related to this product was this.
• 22:37
  SPEAKER_03: So LLMs was good for this kind of use case.
• 22:41
  SPEAKER_03: It again text, it's context, and it can relate across memories and stuff like that.
• 22:48
  SPEAKER_03: So that got us into like where this pipeline where we pull in each review and run against a standard prompt and get a JSON output of this analysis.
• 23:00
  SPEAKER_03: It included positive sentiments, negative sentiments.
• 23:03
  SPEAKER_03: Again, it's what it was not text, again, it was nested JSON.
• 23:06
  SPEAKER_03: So why it was interesting was like again, that's how engineering comes into this type.
• 23:11
  SPEAKER_03: We just don't write pipelines, we just also engineer it, right?
• 23:15
  SPEAKER_03: Then that JSON structure, we directly put it into a local Postgres instance and we started to write queries.
• 23:22
  SPEAKER_03: And then it made sense.
• 23:24
  SPEAKER_03: So this totally unstructured data, we could structure it in such a way that we could pull out four post-each sentiments, we could group positive sentiments into one.
• 23:34
  SPEAKER_03: And as we worked on this, it was for like initially we started out as like a project for data analysts to make their life easier.
• 23:43
  SPEAKER_03: But through this exploration, we understood that there is a demand for on-demand review analysis and exploration and find positive sentiments, negative sentiments, and explore through the review data set and send recommendations to the customers.
• 23:59
  SPEAKER_03: So we thought of to make it into a product.
• 24:01
  SPEAKER_03: Why not as simple as putting in a link and we'll pull the reviews, we'll analyze it for you, and you can explore positive sentiments, negative sentiments.
• 24:10
  SPEAKER_03: We'll also even suggest improvements.
• 24:13
  SPEAKER_03: So one of our customers had this monitor, gaming monitor.
• 24:17
  SPEAKER_03: So this gaming monitor, a lot of people were saying that, okay, I wish it would have HDR feature in this gaming monitor, right?
• 24:26
  SPEAKER_03: So that was like a most requested feature.
• 24:30
  SPEAKER_03: But if you just look at the Amazon page and try to read the reviews, you might not see at first glance.
• 24:37
  SPEAKER_03: But when we put it in our pipeline, it was on top of the requests, right?
• 24:43
  SPEAKER_03: People are saying this, people are saying that.
• 24:50
  SPEAKER_03: So it drove decisions in companies with large reviews.
• 24:55
  SPEAKER_03: So it also helped next iteration of the product, what to include, what not include.
• 25:00
  SPEAKER_03: And this is again without interacting too directly to customer, it's there, the data is there.
• 25:06
  SPEAKER_03: We just help them to get, and this was very successful at this level in the enterprise level.
• 25:12
  SPEAKER_03: So we we thought out okay, we'll give it to like medium customers who have a little bit of knowledge to how to use dashboards and stuff like that.
• 25:21
  SPEAKER_03: And so if they connect their Amazon account, we can pull the inventory, and this DAG will go through each product and it runs automatically.
• 25:31
  SPEAKER_03: And that Airflow Rust API is helped to show ITAs and stuff like that.
• 25:38
  SPEAKER_03: And essentially, how we did this DAG was pull in reviews, run the analysis on top of LLMs, then we made it into a structured format into JSON, and this JSON was converted into more structured table, and we stored it in again.
• 25:54
  SPEAKER_03: We also took some decisions like store the actual responses into a parquet format so that we can save question storage and run that DB on that and run analysis on that.
• 26:05
  SPEAKER_03: And it was a very interesting to work on this, right?
• 26:08
  SPEAKER_03: So, in the end, airflow wasn't just scheduling, right?
• 26:12
  SPEAKER_03: More of the control plane that lets us scale from manual analyst for flow to an automated system that could process millions of reviews reliably.
• 26:21
  SPEAKER_03: It was really a unique set of challenges and stuff like that.
• 26:26
  SPEAKER_02: That's awesome.
• 26:27
  SPEAKER_02: I guess like what I love with your use case is that you prove that you can definitely use Airflow as a way to orchestrate LLMs.
• 26:34
  SPEAKER_02: And like sometimes you hear that Airflow is legacy, blah blah blah, but that's not true.
• 26:39
  SPEAKER_02: Like the reality is that today you can still use Airflow for many use cases, and LLM is definitely one of them, and that you just show that you can do it, even as a small team, you can definitely orchestrate LLMs using Airflow.
• 26:52
  SPEAKER_02: Thank you so much for sharing that with us, Nassim.
• 26:55
  SPEAKER_02: I would like to ask you one last question.
• 26:57
  SPEAKER_02: What is the best way for folks to get in touch with you if they want to learn more about your use case?
• 27:02
  SPEAKER_02: Because we just touched it.
• 27:03
  SPEAKER_02: Honestly, there are so much things to talk about.
• 27:06
  SPEAKER_03: Uh, best way is email and LinkedIn is very easy to reach me there.
• 27:11
  SPEAKER_03: And yeah, I'm very uh eager to help everyone in this case, and I loved Airflow so much.
• 27:18
  SPEAKER_03: I started to contribute, right?
• 27:20
  SPEAKER_03: So get getting active into community is also really good.
• 27:23
  SPEAKER_03: So I'm there in the Airflow Slack, in the dev channels and stuff like that.
• 27:28
  SPEAKER_03: So I'm available there too.
• 27:30
  SPEAKER_03: But yeah, I'm very happy to work through this.
• 27:32
  SPEAKER_03: Hopefully, I can learn more.
• 27:34
  SPEAKER_03: So it's knowledge sharing, knowledge transfer.
• 27:37
  SPEAKER_03: When that exists, this community grows, and that's how we as a community we need this in this airflow.
• 27:44
  SPEAKER_03: Okay, we can have this feature in Airflow.
• 27:45
  SPEAKER_03: That's how you grow.
• 27:46
  SPEAKER_03: So it's really interesting.
• 27:48
  SPEAKER_03: Any small contributions to Airflow will be really great.
• 27:52
  SPEAKER_03: So I would love some of the reason I start there, the contributions Airflow.
• 27:57
  SPEAKER_03: And my code contributions are not a really big, very small thing, but it makes me happy that I can all contribute back to the project.
• 28:05
  SPEAKER_02: That's awesome.
• 28:06
  SPEAKER_02: Thank you so much, Nassim.
• 28:07
  SPEAKER_02: Again, thank you so much for joining, and uh pleasure to have you.
• 28:10
  SPEAKER_02: See you soon.
• 28:11
  SPEAKER_02: Take care.
• 28:14
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 28:21
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 28:24
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.