Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira

Transcript

Uphold’s Approach to Orchestrating Modern Data Workflows with Jaime Oliveira · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: The data engineering who build the pipelines, run the infrastructure, and so on.
• 0:04
  SPEAKER_02: And the analytics engineering who build the data models on the data warehouse.
• 0:09
  SPEAKER_02: So the use cases we have ingesting pipelines, we ingest data from different data sources, we ingest from our internal databases.
• 0:18
  SPEAKER_02: We have a microservice architecture, we ingest data from more than 20 Aurora databases.
• 0:25
  SPEAKER_02: We also ingest data from external parties, APIs, file systems, and so on.
• 0:36
  SPEAKER_00: You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering in AI.
• 0:48
  SPEAKER_01: Hi everyone.
• 0:49
  SPEAKER_01: Welcome to the Data Flowcast.
• 0:51
  SPEAKER_01: I'm your host, Kenton Dannis, and today I'm joined by Jamie Oliveira, who is a lead data engineer at Uphold.
• 0:57
  SPEAKER_01: Jamie, how are you doing?
• 0:58
  SPEAKER_02: I'm doing well.
• 0:59
  SPEAKER_02: And you I'm good.
• 1:00
  SPEAKER_01: I'm good.
• 1:01
  SPEAKER_01: It's great to have you on.
• 1:02
  SPEAKER_01: Thanks for joining.
• 1:04
  SPEAKER_01: Cool.
• 1:04
  SPEAKER_01: So for today's podcast, we're going to discuss how Uphold uses Airflow and DBT to manage data to support their fintech business.
• 1:12
  SPEAKER_01: This has been a really popular topic talking to fintech companies about how they use Airflow.
• 1:17
  SPEAKER_01: So I'm excited to dive into this.
• 1:19
  SPEAKER_01: Maybe you can start by telling me just a little bit of background about what Uphold does.
• 1:23
  SPEAKER_02: Yeah, sure.
• 1:23
  SPEAKER_02: So Uphold is a FinTech company that serves more than 16 million users.
• 1:30
  SPEAKER_02: People normally use our platform to trade crypto and also traditional fiat currencies.
• 1:39
  SPEAKER_02: And we are not only a consumer platform, we also work business-to-business.
• 1:45
  SPEAKER_02: We help other fintechs and traditional banks add crypto services to their own products.
• 1:53
  SPEAKER_02: So basically, they build on our platform to offer crypto services.
• 2:07
  SPEAKER_02: And for us, having a data-driven mindset is really important.
• 2:12
  SPEAKER_02: And yeah, data plays a really important role here.
• 2:17
  SPEAKER_01: Okay, great.
• 2:18
  SPEAKER_01: And I imagine with a data-driven mindset that your data engineering team is pretty important to your company.
• 2:24
  SPEAKER_01: So tell me more about what your team does and where Airflow fits in.
• 2:28
  SPEAKER_02: Data engineering team, we run all the data stack.
• 2:32
  SPEAKER_02: We are mainly splitting too many roles, the data engineers who build the pipelines, run the infrastructure, and so on.
• 2:42
  SPEAKER_02: And analytics engineers who build the data models on the data warehouse.
• 2:48
  SPEAKER_02: So the use cases we have like ingesting pipelines, we ingest data from different data sources, we ingest from our internal databases.
• 3:00
  SPEAKER_02: We have a microservice architecture, we ingest data from more than 20 Aurora databases, let's say.
• 3:08
  SPEAKER_02: We also ingest data from external parties, APIs, file systems, and so on.
• 3:16
  SPEAKER_02: We also have the transformation layer where we build our warehouse, involve create new tables, and give to our teams because data teams support almost all of poll teams.
• 3:35
  SPEAKER_02: And we have a lot of requests from product team, compliance team, Spinux team, design team.
• 3:43
  SPEAKER_02: It's a lot of requests that we have, and we also support our data science team with supporting the infrastructure to run Jupyter Notebooks, some data science pipelines.
• 3:58
  SPEAKER_02: Yeah, basically that, yeah, I would say.
• 4:02
  SPEAKER_01: Okay, great.
• 4:03
  SPEAKER_01: And how does Airflow fit into all of that?
• 4:07
  SPEAKER_02: We use Earthflow mainly as our orchestrator for dbt pipelines.
• 4:13
  SPEAKER_02: We have dbt pipelines, and yeah, basically is our main orchestrator.
• 4:19
  SPEAKER_02: We also use other orchestrators, but not so much broadly.
• 4:25
  SPEAKER_02: We have some snowflake tasks for some easily use cases, but mainly Airflow is our orchestrator.
• 4:34
  SPEAKER_02: Yeah.
• 4:34
  SPEAKER_01: Okay, great.
• 4:35
  SPEAKER_01: And so you're using that orchestration then to provide data to all of the different teams within your business.
• 4:41
  SPEAKER_01: That sounds really interesting.
• 4:43
  SPEAKER_01: And I understand that you personally have quite a long history with the Airflow project.
• 4:48
  SPEAKER_01: Tell me about that and how you first started using Airflow.
• 4:51
  SPEAKER_02: My first experience was a few years ago in a banking project where you're creating an enterprise data warehouse on top of Hadoop, and we decide to use Airflow to orchestrate our hive pipelines.
• 5:09
  SPEAKER_02: Yeah, it's funny that because nowadays no one seems working with that type of stack, so it's more like but yeah.
• 5:21
  SPEAKER_02: And around that time I also started following Maxim, the Airflow creator, and uh his post about functional data engineering was really important for me and helped me to learn and improve my skills about orchestration pipelines, best practice, and for more to learn more complex concepts about Snowflake, even deploying in Kubernetes, more features.
• 5:53
  SPEAKER_02: I don't remember yet, but I have some idea about some resources in New Damien and even in a blog that he had, yeah, really, really fine.
• 6:04
  SPEAKER_02: Good times.
• 6:05
  SPEAKER_01: Yeah, that's great.
• 6:06
  SPEAKER_01: Mark is, of course, another host on our podcast, and on my team at Astronomer, I work with him quite closely.
• 6:12
  SPEAKER_01: So love to hear the shout out there.
• 6:14
  SPEAKER_01: And that's cool that you were an early follower of Maxime as well.
• 6:18
  SPEAKER_01: Has your background with the Airflow project, now that you've used it for a while, influenced how you use it in your current role?
• 6:25
  SPEAKER_02: Yeah, for sure.
• 6:27
  SPEAKER_02: And then when we start using Airflow, we start labeling some of the features, and it's really nice to see how the tool has been emboldened and grow.
• 6:43
  SPEAKER_02: And even now that there is more tools, and I think the community is doing a great job.
• 6:50
  SPEAKER_02: So yeah, I think the beginning was impact for sure in my career.
• 6:55
  SPEAKER_01: That's great.
• 6:56
  SPEAKER_01: I always love to hear stories about folks who have taken Airflow with them from job to job and continue to use it.
• 7:03
  SPEAKER_01: One of the best parts about open source projects, in my opinion.
• 7:06
  SPEAKER_01: Cool.
• 7:07
  SPEAKER_01: Well, I'd love to dig in a little more to how your team uses Airflow and DBT together.
• 7:12
  SPEAKER_01: You mentioned you use Airflow primarily to orchestrate DBT models.
• 7:16
  SPEAKER_01: I think this is a really common pattern using these two together.
• 7:19
  SPEAKER_01: In fact, we were just looking at some results from the Airflow survey from 2025, and DBT was the most frequently cited other tool used with Airflow.
• 7:30
  SPEAKER_01: So I think you're in good company there.
• 7:32
  SPEAKER_01: Uh, but what made you choose to use these two tools together at your current role?
• 7:37
  SPEAKER_02: So Airflow was already part of all pulp stack when I joined.
• 7:43
  SPEAKER_02: And when I joined, the project was to migrate our analytics pipelines to Snowflake.
• 7:50
  SPEAKER_02: So I proposed to the team to use DBT to orchestrate those pipelines, and using Ashflow to do that felt natural.
• 8:00
  SPEAKER_02: So what was mainly that, and today we run different pipelines based on scope, data points, freshness of and other requirements.
• 8:14
  SPEAKER_02: And at this moment we run them with Kubernetes pod operator, which basically means our first starts a container with our dbt project in an isolated way, and it's basically that.
• 8:29
  SPEAKER_01: Okay, interesting.
• 8:31
  SPEAKER_01: I haven't heard as frequently about folks running uh dbt models with the Kubernetes pod operator.
• 8:38
  SPEAKER_01: What has been your experience using that methodology?
• 8:41
  SPEAKER_01: And have you ever considered any other ways of running dbt and airflow together?
• 8:46
  SPEAKER_02: So when we adopted DBT in 2021, we wanted something simple and reliable.
• 8:54
  SPEAKER_02: And installing DBT inside Flow felt too heavy for us.
• 9:00
  SPEAKER_02: We will need to manage packages and versions, and at that time we were still in SFO 110.
• 9:08
  SPEAKER_02: So the Kubernetes pole operator was a good solution.
• 9:12
  SPEAKER_02: We could have a Dockery image, one clean environment, and no impacting net flow itself.
• 9:20
  SPEAKER_02: But it also has downsides because uh local testing is harder, Kubernetes has extra overhead, and also other limitations regarding DVT, the lack of visibility, things like uh processing a single model or checking the logs is hard.
• 9:43
  SPEAKER_02: And during this year, we start looking for other solutions like Cosmos, and basically we did a summer internship where we uh did uh a POC with Cosmos and the first impression was good, and we are thinking on to move to Cosmos because using Cosmos, the way that Cosmos turned a DVT project or DVT pipeline and airflow tech felt very smooth.
• 10:16
  SPEAKER_02: And yeah, this give us a lot of other features like reprocessing, better logging, and airflow is released some new features regarding like feel and so on, is something that we are very eager to test and use now.
• 10:38
  SPEAKER_00: This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
• 10:48
  SPEAKER_00: Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
• 10:59
  SPEAKER_00: To see how you can maximize airflow's potential within your data ecosystem, visit www.astronomer.io.
• 11:08
  SPEAKER_01: At this stage, it sounds like you've concluded the proof of concept of using Cosmos and you like it, but you haven't yet migrated to it.
• 11:16
  SPEAKER_01: Is that right?
• 11:17
  SPEAKER_02: Yeah, right.
• 11:19
  SPEAKER_02: Yeah, a lot of projects going on, and yeah, it's in our to-do list.
• 11:28
  SPEAKER_01: Got it.
• 11:28
  SPEAKER_01: Got it.
• 11:29
  SPEAKER_01: Is the lift of migration the only thing keeping you from moving to it right away?
• 11:35
  SPEAKER_01: Or I'm curious to hear a little bit more detail about what your experience was like during that Cosmos POC.
• 11:41
  SPEAKER_02: From what I remember, we have some issues testing the cobber meets mode.
• 11:48
  SPEAKER_02: And for us, it's very important to have the full picture how we could test locally and the process to deploy our data.
• 11:58
  SPEAKER_02: We want to improve our development experience, the way that we test the devs and deploy them, and we want to leverage the cover needs execution mode that that flow allows.
• 12:12
  SPEAKER_02: So yeah, and the PLC was during the summer internship.
• 12:18
  SPEAKER_02: So basically we did that and we stopped, and now we need to pick it again and allocate some resources on that project.
• 12:29
  SPEAKER_02: Got it, that makes sense.
• 12:31
  SPEAKER_02: And also at that point, I know that we uh also have some problems with the documentation, but we noticed that I think was astronomer who leased a new we book about the DVT deployments and DVT different ways of running inside of our be orchestrated inside that inside that flow, and we want to check it because that could have nice tips and ideas to improve all our exercise, I would say.
• 13:06
  SPEAKER_01: Yeah, absolutely.
• 13:07
  SPEAKER_01: I love that you call that out.
• 13:08
  SPEAKER_01: That ebook was written by someone on my team.
• 13:10
  SPEAKER_01: So requests for more Cosmos documentation are really frequent in the community because it is a really powerful tool and I think one that lots of people like using for DBT and Airflow together, but it's quite complex to set up, especially figuring out how to run it effectively and the execution mode you choose and the things that you've touched on.
• 13:31
  SPEAKER_01: So yeah, our hope with that was to create a more comprehensive resource that would help people get started with it.
• 13:37
  SPEAKER_02: For sure.
• 13:38
  SPEAKER_01: Yeah, we'll link that in the show notes definitely in case anybody else is interested.
• 13:42
  SPEAKER_02: I read some pages and seems really good the content and how descriptive was the materials.
• 13:51
  SPEAKER_02: Yeah, I think it is really nice to check it for those that are hearing us.
• 13:58
  SPEAKER_01: We appreciate hearing that.
• 14:00
  SPEAKER_01: I'm also curious, you mentioned right now you're using Kubernetes pod operator to run dbt, and that pattern originated when you run an older version of Airflow.
• 14:12
  SPEAKER_01: Is that something that you do in your other Airflow DAGs as well?
• 14:15
  SPEAKER_01: So ones that might not necessarily be using DBT?
• 14:18
  SPEAKER_01: Are you one of those companies that you know lean very heavily towards Kubernetes pod operator for everything?
• 14:25
  SPEAKER_02: No, we only use Kubernetes operating for dbt pipelines.
• 14:29
  SPEAKER_02: We also have some pipelines running using Snowflake operator, Postgres operator, and others with sensors and so on.
• 14:40
  SPEAKER_02: But we only use the Kubernetes pod operator for DBT pipelines at this point.
• 14:45
  SPEAKER_02: We had a pipeline of a data science pipeline that we also use at Kubernetes Pulse operator, but we duplicated that.
• 14:54
  SPEAKER_01: Okay, got it.
• 14:55
  SPEAKER_01: So it sounds like only in cases where you're dealing with some pretty heavy package management and things like that.
• 15:00
  SPEAKER_02: But also because at that time the version that we are using when we start our project with DVT doesn't have much compatibility, and we don't have a DVT operator, so uh one that we could leverage, and we decided to use Kubernetes Poloperate and control everything, the secrets that we ingest, the image, the pipelines that we run, and yeah, but that don't backs for sure.
• 15:33
  SPEAKER_02: As I mean, now that our dbt project is growing, we are feeling some dumb size on that, and yeah, we need to change that.
• 15:43
  SPEAKER_01: Yeah, I imagine it gets quite unwieldy to manage.
• 15:46
  SPEAKER_01: Kubernetes pod operator is not a straightforward implementation a lot of times.
• 15:50
  SPEAKER_01: Well, thank you for talking through all of that.
• 15:52
  SPEAKER_01: That was really interesting to hear.
• 15:54
  SPEAKER_01: I'm excited to check in with you at some point down the road and see how your migration to Cosmos has gone once you get around to it.
• 16:01
  SPEAKER_01: Jamie, I always like to end these episodes with a standard final question for folks, and that is what would you most like to see from the Airflow project in the near future?
• 16:10
  SPEAKER_02: So now that we have DAG versioning, I think I would like to see a more interactive UI.
• 16:18
  SPEAKER_02: Something more dynamic, almost like a canvas, where we can design or build and adjust our DAGs directly in the interface.
• 16:31
  SPEAKER_02: Maybe even something like Figma.
• 16:35
  SPEAKER_02: I don't know if everyone knows Figma, where teams collaborate, comment, and build together.
• 16:42
  SPEAKER_02: Yeah, I think we'll be really nice to have something like that for us.
• 16:48
  SPEAKER_01: Interesting.
• 16:48
  SPEAKER_01: So more of a GUI for building DAGs instead of writing the code or just an interface to write the code in the UI and collaborate with people more.
• 16:57
  SPEAKER_02: Yeah, something like that.
• 16:59
  SPEAKER_02: Okay.
• 17:00
  SPEAKER_02: Both I would be that it's not easy and depends on during the development phase and promote to production.
• 17:10
  SPEAKER_02: But I think nowadays that we want to bring the analytics engineering to be more autonomous, so they have the capability to build pipelines with DVT and other stuff.
• 17:23
  SPEAKER_02: So turning the process to walk straight to those pipelines more easily, I think, will be nice.
• 17:32
  SPEAKER_01: Yeah, interesting.
• 17:33
  SPEAKER_01: I think that totally makes sense.
• 17:35
  SPEAKER_01: I know the open source project has always been very against any method for creating DAGs other than writing code, which I understand why it's part of the ethos.
• 17:44
  SPEAKER_01: It's also, I think, something Astronomer has spent some time on.
• 17:48
  SPEAKER_01: We have an IDE, at least for writing DAGs in the UI.
• 17:51
  SPEAKER_01: At this point, you're still writing code or using the AI to help you write code, but are definitely looking into ways for just to make it a little easier for people who aren't quite as familiar with Airflow to do more of a plug and play for like those common patterns that you might have where it is more analytical or you're just running a DBT model or a SQL query or something like that.
• 18:12
  SPEAKER_01: So I think you're not alone and wanting something like that.
• 18:14
  SPEAKER_01: Jamie, this has been really great to have you on.
• 18:17
  SPEAKER_01: Thanks so much for joining us.
• 18:18
  SPEAKER_01: Just to close out, what is the best way for folks to get in touch with you?
• 18:23
  SPEAKER_01: Okay.
• 18:24
  SPEAKER_01: Great.
• 18:24
  SPEAKER_01: Well, we will put links to that in the show notes.
• 18:27
  SPEAKER_01: And thank you again for coming.
• 18:29
  SPEAKER_01: It's been really great speaking with you.
• 18:31
  SPEAKER_02: Thank you for inviting.
• 18:32
  SPEAKER_02: Really nice to be here.
• 18:36
  SPEAKER_00: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 18:43
  SPEAKER_00: You'll find links in the show notes to any researches mentioned in today's show.
• 18:47
  SPEAKER_00: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.