Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / The Intersection of AI and Data Management at Dosu with Devin Stein

Transcript

The Intersection of AI and Data Management at Dosu with Devin Stein · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: You can't replace a maintainer with an LLM or an AI agent.
• 0:05
  SPEAKER_02: But at the same time, LLMs and AI can help communities, both maintainers be more effective and set help them save times and automate work that is otherwise like very tiring and leads to burnout.
• 0:17
  SPEAKER_02: And similarly on the community side, everyone's had the experience of waiting hours, days, or weeks to get a response on an issue that is blocking you.
• 0:25
  SPEAKER_02: And so having ways to unblock yourself without always having to depend on maintainers.
• 0:35
  SPEAKER_01: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
• 0:45
  SPEAKER_00: Hi everyone.
• 0:46
  SPEAKER_00: Welcome to the Data Flowcast.
• 0:48
  SPEAKER_00: I'm your host, Kenton Danis, and today I'm joined by Devin Stein, who's the founder and CEO at DOSU.
• 0:54
  SPEAKER_00: Devin, how are you doing today?
• 0:56
  SPEAKER_02: I'm doing well, Kenton.
• 0:58
  SPEAKER_00: Awesome.
• 0:59
  SPEAKER_00: We were just speaking before we started recording about the Airflow Summit, which just happened probably will have been a couple of weeks ago by the time that this recording comes out.
• 1:08
  SPEAKER_00: Devin, were you at the summit?
• 1:10
  SPEAKER_02: I was not fortunately.
• 1:13
  SPEAKER_00: I wasn't sure if you are in the Bay Area, right?
• 1:15
  SPEAKER_02: Yeah, in the Bay Area.
• 1:17
  SPEAKER_02: I wanted to go in busy.
• 1:19
  SPEAKER_00: Close by, but yeah, timing doesn't always work.
• 1:22
  SPEAKER_00: But yeah, I think it was great because I also had to um miss it this year, which I was very sad about.
• 1:27
  SPEAKER_00: But um, we'll definitely try and make it next year.
• 1:30
  SPEAKER_00: It's last year was really fun.
• 1:31
  SPEAKER_00: I've heard great things about this year as well to catch it next time.
• 1:35
  SPEAKER_00: Awesome.
• 1:36
  SPEAKER_00: Let's go ahead and dive in.
• 1:37
  SPEAKER_00: So for today's podcast for our listeners, we're going to discuss how DOSU both uses Airflow for their critical data orchestration and also supports the Airflow project, which we always love to hear about.
• 1:48
  SPEAKER_00: So, Devin, just diving into what DOSU does, your website states that Dosu lets engineers focus on value add work by answering questions, triaging issues, and maintaining documentation for them.
• 2:01
  SPEAKER_00: Tell me a little bit more about what that means.
• 2:03
  SPEAKER_02: Yeah, absolutely.
• 2:04
  SPEAKER_02: At a high level, our goal with DOSU is to make it easy for engineering teams to generate, maintain, and disseminate knowledge.
• 2:12
  SPEAKER_02: And when you think about your work as an engineer, especially as an organization grows, you don't actually spend that much time coding, but you do a lot of what we call engineering knowledge work, which is using your understanding of code, code changes and like all the conversations going on within the engineering org, whether it's in Slack and JIRA on PRs to answer questions, triage issues, write and update documentation, or review PRs.
• 2:40
  SPEAKER_02: And these are all the types of tasks that we're trying to help engineers with.
• 2:44
  SPEAKER_02: And also for some background, I started DOSU in Judah 2023.
• 2:50
  SPEAKER_02: Um before that, I was an early engineer at a few different startups, and I'm also an active open source maintainer.
• 2:57
  SPEAKER_02: Um, and as a as an open source maintainer, a lot of what you're doing is falls in this engineering knowledge work category, where you are the expert about your project.
• 3:08
  SPEAKER_02: And if it's you're lucky and your project is successful, then you have hundreds or thousands or tens of thousands of engineers that depend on you, um, often asking questions, reporting bugs, or trying to figure out how to contribute to a code base, or making a PR and trying to understand if they're doing things correctly.
• 3:25
  SPEAKER_02: And DOSUE work heavily with open source, with maintainers with projects like Airflow to help them do that engineering knowledge work that goes into open source maintenance.
• 3:36
  SPEAKER_00: Got it.
• 3:36
  SPEAKER_00: That makes a lot of sense.
• 3:37
  SPEAKER_00: And so you have a product that gets installed within a software base that helps automate some of that work.
• 3:44
  SPEAKER_00: Is that how it works?
• 3:46
  SPEAKER_02: Exactly.
• 3:46
  SPEAKER_02: Very concretely within open source, it is GitHub to install on an open source project.
• 3:51
  SPEAKER_02: You configure it within our app, and then Dosu can, depending on the project settings, auto-label issues, deduplicate issues and discussions, and then actually respond to issues trying to help answer questions, triage bugs, show people how to contribute to the code base.
• 4:07
  SPEAKER_02: And then we also have kind of a human in the loop mode, which allows basically teams can use TOSU as like that triage assistant where it's doing a first pass and gathering related resources to answer any new issue, showing you a preview of what it would have responded, and then you can use that to either publish it, edit it, um, or create your own response.
• 4:28
  SPEAKER_00: Got it.
• 4:29
  SPEAKER_00: Makes sense.
• 4:30
  SPEAKER_00: That makes me think back to my first project when I was a data engineer working in consulting in my past life.
• 4:37
  SPEAKER_00: I inherited this massive repo with all of this very specific business code that was used for analytics.
• 4:44
  SPEAKER_00: And the person who had created all of it had left already by the time I got there.
• 4:48
  SPEAKER_00: And I felt like I had absolutely no idea what all of it was.
• 4:51
  SPEAKER_00: It took forever to just sift through all of it and learn what was going on.
• 4:56
  SPEAKER_00: I imagine that this tool would have helped me out a lot.
• 4:59
  SPEAKER_02: Yeah.
• 5:00
  SPEAKER_02: And like the idea or hope and to build with Dos was whoever that engineer was, like as they were building that code base, they're generating knowledge.
• 5:07
  SPEAKER_02: And can Dosu help you structure, organize that, and then maintain that over time for other people in the future?
• 5:14
  SPEAKER_00: Yeah, totally.
• 5:15
  SPEAKER_00: Uh so your product has some AI in the background to help it out and understand.
• 5:22
  SPEAKER_00: And though those applications are typically built on a lot of data to make them most useful.
• 5:27
  SPEAKER_00: So, can you talk a little bit about the importance of having a reliable, up-to-date data that feeds your product?
• 5:33
  SPEAKER_02: Yeah, absolutely.
• 5:34
  SPEAKER_02: So I guess DOSU is an AI native company.
• 5:39
  SPEAKER_02: I think the unlock that allows us, I guess, to build DOSU is that LLMs help structure unstructured engineering knowledge and information and then make decisions about how to use that to automate workflows.
• 5:51
  SPEAKER_02: But at the end of the day, a LLM product is a data or ML product as well.
• 5:57
  SPEAKER_02: Then, like kind of the data that we are ingesting and curating and structuring is integral to the product working.
• 6:06
  SPEAKER_02: If the data that DOSU is reading from is stale or out of date or incorrect or hasn't been updated, then its responses or its abilities is limited.
• 6:16
  SPEAKER_02: So data is really at the center of what we do, both in terms of like in the background and then also at inference time, what people end up seeing as a result of data.
• 6:26
  SPEAKER_00: Yeah, totally makes sense.
• 6:27
  SPEAKER_00: And so where does Airflow fit into that?
• 6:30
  SPEAKER_02: Yeah, so that was the I guess insight that led us to using Airflow was at the end of the day, like I was saying, any LLM company is just a data company fundamentally.
• 6:42
  SPEAKER_02: The whether you're training your own models or using off-the-shelf models, it's only as good as the information you curate.
• 6:48
  SPEAKER_02: Um, and in my previous roles, and actually part of the reason I got involved in open source was at my last role, it was a more traditional NML company where we ran a lot of open source projects and used Airflow as our scheduling for our data platform for all of our scheduled jobs.
• 7:05
  SPEAKER_02: And I realized as we were building out DOSU, when we are um ingesting data from all these different sources, whether it's GitHub, Slack, Jira, et cetera, we need to do that at some interval.
• 7:18
  SPEAKER_02: And what you end up having is a something that looks exactly like any data pipeline, a step for loading data into our environment, a step for transforming that either into embeddings or using LLMs to augment that information or extract important facts from it, and then actually writing that to a vector database and storing it.
• 7:40
  SPEAKER_02: So we use Airflow for the kind of traditional scheduling piece of what we do, the backbone of the product, as well as other types of scheduled activity that we want to do.
• 7:49
  SPEAKER_02: So we do PR periodic reviews of threads to determine if they were resolved and if what if so, what was the resolution and like what information we can extract from that, or running evaluations on some fixed schedule.
• 8:06
  SPEAKER_00: Yeah, interesting.
• 8:07
  SPEAKER_00: That makes sense.
• 8:08
  SPEAKER_00: And so you you said you just founded DOSU in spring of 2023, so not terribly long ago.
• 8:15
  SPEAKER_00: So you had, I assume, your choice of kind of any orchestrator you wanted to use when building out your platform.
• 8:21
  SPEAKER_00: What made you choose Airflow?
• 8:23
  SPEAKER_02: I think when starting a company or trying to move as fast as possible, I am an engineer, so I'm always alert.
• 8:30
  SPEAKER_02: I have a fascination with new and shiny tools.
• 8:34
  SPEAKER_02: But going back to data reliability being really important for what we do, I had used Airflow heavily in the past.
• 8:40
  SPEAKER_02: Other engineers at DOSU had used Airflow at scale.
• 8:44
  SPEAKER_02: So we knew scale wasn't going to be an issue.
• 8:47
  SPEAKER_02: We knew how to operate it, and we knew it would solve our use case very well.
• 8:51
  SPEAKER_02: So just a very good fit for kind of what we're looking for.
• 8:54
  SPEAKER_02: And we knew it would be both easy to get going with, but also that we could keep it run for the long term.
• 9:00
  SPEAKER_00: Yeah.
• 9:00
  SPEAKER_00: And imagine talking about institutional knowledge is something that your product helps out with a lot, and that's really important for a lot of engineering teams.
• 9:07
  SPEAKER_00: That is definitely a benefit of Airflow, is so many people know how to use it.
• 9:11
  SPEAKER_00: It's pretty standard within at least the data engineering space, that you're more likely to get people up and running quickly than choosing a more obscure tool, I would assume.
• 9:21
  SPEAKER_02: Yeah, exactly.
• 9:25
  SPEAKER_01: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 9:34
  SPEAKER_01: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 9:46
  SPEAKER_01: To see how you can maximize Airflow's potential within your data ecosystem, visit www.astronomer.io.
• 9:55
  SPEAKER_00: Any sort of interesting stories that you've come across implementing Airflow for your use case over the last year or so?
• 10:04
  SPEAKER_02: I would say that I don't know if there's any particularly like novel things other than re-emphasizing the point that a lot of things look similar to what we've done before in terms of just more like our data pipelines are doing instead of like using Spark to do transformation or something like that, like I did in my previous company, we're using LLMs to augment the data.
• 10:29
  SPEAKER_02: So nothing too novel.
• 10:31
  SPEAKER_02: I think something interesting, maybe that we're looking to explore, is using OpenAI's kind of batch endpoints and scheduling jobs that we want to do in the background that are like we want to compute some statistics over tens of thousands of threads or something like that.
• 10:47
  SPEAKER_02: Um, it's not time sensitive.
• 10:49
  SPEAKER_02: And using Airflow as like the scheduling mechanism and like for launching all of those jobs on a platform like OpenAI.
• 10:55
  SPEAKER_02: So I guess that's something that's interesting that we've been looking at.
• 10:58
  SPEAKER_00: Yeah, cool.
• 11:00
  SPEAKER_00: I do think it's interesting, just overall, to hear you speak about for an AI native company.
• 11:05
  SPEAKER_00: We hear about this a lot at Astronomer, where not everybody thinks of Airflow as kind of their first choice for a tool for MLOps or kind of AI-driven workflows, because it's this traditional bread and butter like people think of it as an ETL scheduler.
• 11:22
  SPEAKER_00: But as you've described, all of that AI native workflows are just data workflows under the hood.
• 11:29
  SPEAKER_00: And so there's not actually a lot of difference in terms of what you're doing with Airflow versus if you had a different end product.
• 11:35
  SPEAKER_02: Yeah, exactly.
• 11:36
  SPEAKER_02: And any, I would say most LLM companies today are doing RAG retrieval augmented generation.
• 11:42
  SPEAKER_02: Um, and when you're doing RAG, you need a data corpus that you have to index and store somewhere, which is ETL.
• 11:49
  SPEAKER_02: You're like taking data from somewhere, you're doing some transformations on it, you're loading into a vector database, and then you're using it at inference time.
• 11:58
  SPEAKER_00: Yeah, cool.
• 12:00
  SPEAKER_00: I understand that in addition to using Airflow yourselves, you mentioned that you install Dosu and support a lot of open source projects.
• 12:08
  SPEAKER_00: And you recently began supporting the open source Airflow project.
• 12:12
  SPEAKER_00: Can you talk a little bit about what you're doing for the Airflow project and how that partnership came about?
• 12:17
  SPEAKER_02: Yeah, absolutely.
• 12:18
  SPEAKER_02: So actually, so when I there's a fun backstory, when I was first starting Dosu and I was thinking about starting with open source maintenance as like our initial use case, I actually reached out to the Airflow maintainers because Airflow is just known to be one of the best run open source projects.
• 12:38
  SPEAKER_02: Um it's been going for a long time, and I thought there was like a lot of learnings there.
• 12:43
  SPEAKER_02: So I actually wanted to talk to them more on the user research side, trying to understand, okay, where could LLMs fit in?
• 12:49
  SPEAKER_02: And then it took a while to get uh approval by the Apache Software Foundation to actually get the app installed.
• 12:54
  SPEAKER_02: But then, like you said, a few months ago, we started, we're actually deployed, had got those to deploy it on Airflow open source.
• 13:01
  SPEAKER_02: And what we've been helping them with is auto-labeling by Air For the last thousands, tens of thousands of issues.
• 13:07
  SPEAKER_02: They've changed labeling schemes over time.
• 13:10
  SPEAKER_02: And so we've helped one, just auto-label all incoming issues, but then I've also been helping them backfill and standardized historic issues, which helps a lot for just discoverability, analytics.
• 13:23
  SPEAKER_02: There's a lot of downstream niceties that come when you have consistent high-quality labels.
• 13:27
  SPEAKER_02: And then what we've been also partnering them with is this accelerated mode, what I was describing at the beginning of the call around using DOSU as basically and helping accelerate their triage workflow.
• 13:40
  SPEAKER_02: So when they're going to meet and review new issues, they can go to DOSU to our UI, have an inbox of issues to review, and then you get to see all the research that DOSU did when trying to generate a response and the generating response.
• 13:55
  SPEAKER_02: So you can see similar issues or discussions, related PRs, uh relevant code files, documentation.
• 14:02
  SPEAKER_02: And then from the UI, you can publish DOSU's response, edit it, or create your own.
• 14:07
  SPEAKER_02: But yeah, we've been working closely with them on that kind of like accelerated triage mode to gently introduce DOSU to the community.
• 14:15
  SPEAKER_00: Okay, that's awesome.
• 14:17
  SPEAKER_00: Someone who has spent quite a lot of time looking at airflow issues and PRs, I think that this would be very helpful.
• 14:23
  SPEAKER_00: How do you see that partnership with the community evolving?
• 14:26
  SPEAKER_00: Have you has your experience been positive so far?
• 14:30
  SPEAKER_02: Yeah, it's been super positive.
• 14:32
  SPEAKER_02: I think it's uh a long discussion, but I think there is this feeling or tension within open source around using AI, whether you don't want to have what I always describe as like the stalebot effect, which you have something where Airflow has been successful because of the vibrancy of its community, and you don't want AI is not going to replace that.
• 14:54
  SPEAKER_02: You can't replace a maintainer with an LLM or an AI agent.
• 14:59
  SPEAKER_02: But at the same time, LLMs and AI can help communities, both maintainers be more effective and help them save times and automate work that is otherwise like very tiring and leads to burnout.
• 15:11
  SPEAKER_02: And similarly, on the community side, everyone's had the experience of waiting hours, days, or weeks to get a response on an issue that is blocking you.
• 15:19
  SPEAKER_02: Um and so having ways to unblock yourself without always having to depend on maintainers is really important.
• 15:25
  SPEAKER_02: Um so yeah, we've been trying to be very thoughtful about how we introduce DOSU to communities because we want to make sure that they don't feel like the maintainers are abandoning them for a bot, but actually, like DOSU is part of that community and it's there for a purpose and everyone benefits from it.
• 15:43
  SPEAKER_02: So we've been working really closely with the maintainers of Airflow to on things like this kind of accelerated mode as ways to gently introduce DOSU and show that it has like maintainer approval with its responses.
• 15:57
  SPEAKER_00: Yeah, I like that framing that it's a tool, not a replacement.
• 16:02
  SPEAKER_00: Uh, I think again, coming back to that concept of like institutional knowledge, I think that's even more relevant with an open source project because a lot of the people who are contributing are that's not their job.
• 16:14
  SPEAKER_00: There are a few people who it is, like we have some folks at Astronomer who it is their job to work on open source airflow, which is great, but it's certainly not the majority of the 3,000 committers, or certainly even all of like the PMC member members who are leading the project.
• 16:28
  SPEAKER_00: If those folks aren't available because they're doing something else or a different job or anything, then it's helpful to have that extra tool in there to fill in the gaps and keep the continuity.
• 16:40
  SPEAKER_02: Exactly.
• 16:41
  SPEAKER_00: Yeah.
• 16:41
  SPEAKER_00: Well, that's really cool.
• 16:42
  SPEAKER_00: I'm very interested to hear where that goes over time and to get the feedback from the community.
• 16:49
  SPEAKER_00: Uh so I always like to end with a standard final question.
• 16:54
  SPEAKER_00: And that's what would you most like to see from development of the Airflow project in the near future?
• 17:00
  SPEAKER_02: That's a good question.
• 17:03
  SPEAKER_02: So I haven't kept up to date with the Airflow 3.
• 17:06
  SPEAKER_02: I know there's a lot that was announced last week at Airflow Summit.
• 17:10
  SPEAKER_02: So some of it may already be in the works.
• 17:13
  SPEAKER_02: I think the thing that, especially within our space, we do a lot of, we use Airflow, like the traditional, like the ETL schedule workloads, but we also have workloads that are a bit more dynamic and I think um, but are fundamentally like mapped to workflows or like DAGs and tasks in Airflow world.
• 17:33
  SPEAKER_02: And so I think over time I would like to see more flexibility on Airflow to manage more dynamic schedule workflows in yeah, the easier and more first class way.
• 17:46
  SPEAKER_02: I know that we leverage dynamic DAGs and like existing concepts, but I think pushing towards better uh scheduling and management of like more ad hoc dynamic workloads, I think will be super, super useful.
• 18:00
  SPEAKER_00: Interesting.
• 18:01
  SPEAKER_00: And so when you say dynamic workflows, can you tell me a little bit more about what you mean about that?
• 18:06
  SPEAKER_00: Which part of it do you want to be dynamic that you feel isn't well supported in Airflow?
• 18:10
  SPEAKER_02: Yeah.
• 18:11
  SPEAKER_02: So maybe, so I guess a very like top of mind thing because we were just talking about is I we have this, we want to have a like like an exponential back off type of task where it is scheduled and then it pulls the information if it hasn't changed, then it updates its schedule to back off basically, and then it checks again, and then that counter then gets reset once it finds that content has changed and then repeats that process.
• 18:37
  SPEAKER_02: And so right now, I'm not sure the best way to do that with Airflow or if it's even possible, but it would be like it would be nice to have some sort of dynamic scheduling of a task like that.
• 18:48
  SPEAKER_00: Yeah, that's interesting.
• 18:50
  SPEAKER_00: There is an exponential back off on the retry function, which obviously less than ideal to set up a task to purposely fail and then like wait until it tries again.
• 19:01
  SPEAKER_00: It's not so much waiting, the task itself isn't waiting, it's just waiting to retry.
• 19:07
  SPEAKER_00: Yeah, that was the first thing that came to mind.
• 19:08
  SPEAKER_00: That is interesting.
• 19:09
  SPEAKER_00: I'm sure you could write a custom function to do that in one DAG and then use a data set just based on that DAG when it finished.
• 19:16
  SPEAKER_00: But yeah, okay, I see what you're getting at.
• 19:18
  SPEAKER_00: That that is a bit of a custom solution to implement right now within Airflow.
• 19:23
  SPEAKER_00: Yeah, that's interesting feedback.
• 19:25
  SPEAKER_00: I actually haven't heard that one specifically before.
• 19:27
  SPEAKER_00: Yeah, that's great.
• 19:28
  SPEAKER_00: Okay, cool.
• 19:29
  SPEAKER_00: Devin, thanks so much for joining us today.
• 19:32
  SPEAKER_00: It was great chatting with you.
• 19:33
  SPEAKER_00: To close out, what's the best way for folks to get in touch with you?
• 19:37
  SPEAKER_02: Go to our website.dev dosu.dev or reach out at hi at dosu.dev or follow us on Twitter.
• 19:46
  SPEAKER_00: Okay, that's great.
• 19:47
  SPEAKER_00: Well, again, we'll be looking out for your impact on the airflow community and look forward to hearing more about it.
• 19:53
  SPEAKER_00: So thanks again for joining.
• 19:54
  SPEAKER_02: Yeah, thanks so much, Kenton.
• 19:56
  SPEAKER_02: Thanks for having me.
• 19:59
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 20:06
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 20:10
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.