Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado

Transcript

Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_02: Organizationally, data is the lifeblood of Ramp and it touches every single vertical in the company.
• 0:04
  SPEAKER_02: For instance, data enables our product teams to measure usage of our features and maybe even build new features that delight our customers.
• 0:11
  SPEAKER_02: It enables our risk team to measure and manage credit and fraud risk.
• 0:14
  SPEAKER_02: It enables our operations team to identify opportunities to optimize our customer experience team.
• 0:20
  SPEAKER_02: Within growth, we have a few use cases, so outbound automation, lead scoring, and also churn reduction.
• 0:25
  SPEAKER_02: And then we also enable our strategic finance team to measure progress on our financial goals and set our corporate strategy.
• 0:46
  SPEAKER_00: Airflow users, we have exciting news.
• 0:49
  SPEAKER_00: The annual Apache Airflow survey is live.
• 0:52
  SPEAKER_00: In just seven minutes, you can help shape the future of Airflow.
• 0:56
  SPEAKER_00: And as a thank you for filling it out, you'll get a free Airflow Fundamentals or DAG Authoring certification.
• 1:01
  SPEAKER_00: Those are worth$150 each.
• 1:04
  SPEAKER_00: Plus, you'll also be entered to win an exclusive workshop with Mark Lumberti.
• 1:08
  SPEAKER_00: So make sure you don't miss out.
• 1:10
  SPEAKER_00: The survey link is in the show notes and it closes December 20th.
• 1:16
  SPEAKER_00: Hi everyone.
• 1:17
  SPEAKER_00: Welcome to the Data Flowcast.
• 1:19
  SPEAKER_00: I'm your host, Kenton Dannis, and today I'm joined by Ryan Delgado, who's a director of engineering at Ramp.
• 1:25
  SPEAKER_00: Ryan, how are you today?
• 1:28
  SPEAKER_02: I'm great, Kenton.
• 1:29
  SPEAKER_02: Uh, it's great to be here.
• 1:30
  SPEAKER_02: How are you?
• 1:31
  SPEAKER_00: Yeah, I'm good.
• 1:32
  SPEAKER_00: I'm good.
• 1:33
  SPEAKER_00: So we'll go ahead and dive in for our listeners.
• 1:35
  SPEAKER_00: For today's podcast, we're going to discuss how Ramp approaches data and building platforms and the role that Airflow plays in that.
• 1:43
  SPEAKER_00: So, Ryan, first before we dive into Airflow and platform building, maybe can you tell our listeners a little bit about Ramp?
• 1:50
  SPEAKER_00: What does the company do?
• 1:52
  SPEAKER_02: Yeah, for sure.
• 1:53
  SPEAKER_02: Ramp is a financial operations platform.
• 1:56
  SPEAKER_02: And so we have a few different products that automate and streamline different aspects of back office finance operations.
• 2:01
  SPEAKER_02: And probably our best known product is Ramp Card.
• 2:05
  SPEAKER_02: And so that's our corporate card and expense management solution.
• 2:08
  SPEAKER_02: And what it does is it automates away much of the tedium associated with corporate cards like collecting receipts, filling out expense reports, categorizing transactions, while also giving finance teams more control over how cards are used to spend money with customized limits and also other restrictions.
• 2:24
  SPEAKER_02: And so the result is a product that helps our customers save their most precious resources.
• 2:29
  SPEAKER_02: And so that's their time and that's their money.
• 2:32
  SPEAKER_00: Awesome.
• 2:32
  SPEAKER_00: Sounds like go in.
• 2:33
  SPEAKER_00: I don't think anybody would be sad to never fill out an expense report again.
• 2:37
  SPEAKER_00: Myself included.
• 2:39
  SPEAKER_00: So I'm about to go on a trip next week to reinvent.
• 2:42
  SPEAKER_00: And yeah, we'll definitely not want to have to deal with any of that hassle.
• 2:47
  SPEAKER_00: So that's a, I'm sure, a very widely used product.
• 2:50
  SPEAKER_00: That's great.
• 2:51
  SPEAKER_00: And within that, so what is your role and what does your team do?
• 2:55
  SPEAKER_02: Yeah, for sure.
• 2:56
  SPEAKER_02: So I lead a data platform team at Ramp.
• 2:58
  SPEAKER_02: And we're a team of 12 engineers, and we're focused on building infrastructure and tools that enable RAMP to realize business value from data.
• 3:06
  SPEAKER_02: And so we partner closely with stakeholder teams to build these platforms that enable us to build and scale reliably and at high velocity, as well as the applications on top of those platforms.
• 3:18
  SPEAKER_02: And so our main stakeholders are, of course, the data team, which is an organization of analytics engineers and applied scientists.
• 3:25
  SPEAKER_02: But we also work with product engineering teams on applications, as well as other non-data team members, but still data professionals.
• 3:32
  SPEAKER_02: And so these are risk operators, these are growth analysts and the like.
• 3:35
  SPEAKER_02: And we operate with broad surface areas.
• 3:38
  SPEAKER_02: The bedrock of any data platform is its offline analytics platform.
• 3:41
  SPEAKER_02: And so that's going to be an orchestration layer, which we use Airflow, of course, but it's also a data warehouse for data storage.
• 3:47
  SPEAKER_02: And then also we use DBT for data transformation.
• 3:50
  SPEAKER_02: The second surface area is what I will call the data science platform.
• 3:53
  SPEAKER_02: And so that's a research environment, let's say a managed notebooking environment.
• 3:57
  SPEAKER_02: It's also infrastructure for model training and model serving, either synchronously through a ref service and maybe asynchronously through a key value store and some model inference process, as well as platforms for feature engineering and maybe AI infrastructure such as vector search or GPU-backed model serving.
• 4:14
  SPEAKER_02: And so you can think of us as a team of engineers with skills in data engineering, DevOps and infra, machine learning engineering, and then also a little bit of back-end software engineering.
• 4:23
  SPEAKER_02: And I think where we likely stand out among data platform teams is how customer-facing we are.
• 4:28
  SPEAKER_02: Over the years, we've become more deeply embedded in product engineering initiatives.
• 4:31
  SPEAKER_02: And this year we've actually built two customer-facing features.
• 4:34
  SPEAKER_02: So we built RAM for Excel zero to one and went live with it back in August.
• 4:38
  SPEAKER_02: And so that's a Microsoft Excel added for pulling in ramp data.
• 4:41
  SPEAKER_02: And since finance professionals work primarily in Excel, it's quickly become the fastest growing ramp integration of the year.
• 4:47
  SPEAKER_02: And that's not classical data engineering.
• 4:49
  SPEAKER_02: It's actually mostly front-end engineering with a little bit of back end engineering sprinkled in.
• 4:53
  SPEAKER_02: So it's been a lot of fun.
• 4:54
  SPEAKER_02: Stepping outside of our comfort zone there.
• 4:56
  SPEAKER_02: Another great example, it's actually a use case for Airflow, and that's our new enterprise data warehouse integration.
• 5:01
  SPEAKER_02: And so that's where we directly integrate with a customer's data warehouse to serve them their ramp data directly inside of it for more advanced analytics and reporting on the ramp data.
• 5:10
  SPEAKER_02: And so we'll use data sharing functionality in, let's say, Snowflake or BigQuery, as well as an Airflow DAG to pipe this data from, let's say, our BigQuery instance, or excuse me, from our internal data warehouses into our BigQuery instance.
• 5:22
  SPEAKER_02: Yeah, we do a lot of fun stuff here.
• 5:23
  SPEAKER_02: I love it.
• 5:24
  SPEAKER_00: Interesting.
• 5:24
  SPEAKER_00: That's really cool.
• 5:25
  SPEAKER_00: That sounds like a huge breadth of stuff that you're covering for what you said, a 12-person team.
• 5:31
  SPEAKER_02: Yeah.
• 5:32
  SPEAKER_00: Yeah, that's awesome.
• 5:33
  SPEAKER_02: Yeah, we keep busy.
• 5:34
  SPEAKER_00: So as most data teams, I'm sure, do.
• 5:38
  SPEAKER_00: But yeah, that's really cool that you're involved both in the kind of back end analytics and also in in the product development as well.
• 5:45
  SPEAKER_00: So it sounds like from all angles, data is obviously very core to your business.
• 5:51
  SPEAKER_00: So maybe you can talk a little bit about that high-level approach that Ramp takes towards managing and leveraging data.
• 5:58
  SPEAKER_02: Yeah, for sure.
• 5:59
  SPEAKER_02: Maybe I can answer this question from an organizational perspective and then also from a tactical perspective.
• 6:04
  SPEAKER_02: And so organizationally, data, it's the lifeblood of RAMP and it touches every single vertical in the company.
• 6:10
  SPEAKER_02: For instance, data enables our product teams to measure usage of our features and maybe even build new features at the light our customers.
• 6:16
  SPEAKER_02: It enables our risk team to measure and manage credit and fraud risk.
• 6:20
  SPEAKER_02: It enables our operations team to identify opportunities to optimize our customer experience team.
• 6:25
  SPEAKER_02: Within growth, we have a few use cases.
• 6:27
  SPEAKER_02: So outbound automation, lead scoring, and also churn reduction.
• 6:31
  SPEAKER_02: And then we also enable our strategic finance team to measure progress on our financial goals and set our corporate strategy.
• 6:36
  SPEAKER_02: And so that's a lot of stuff.
• 6:37
  SPEAKER_02: But what's in common is how the data team integrates into these different subteams.
• 6:42
  SPEAKER_02: And so a common pattern within startups is EPD pods.
• 6:44
  SPEAKER_02: So that's engineering product and design pods.
• 6:47
  SPEAKER_02: And we do that, but we actually take it a step further with PEDD pods.
• 6:51
  SPEAKER_02: That's product engineering design and data team members.
• 6:54
  SPEAKER_02: And so what that means is that data is at the forethought of product development rather than it being an afterthought.
• 6:59
  SPEAKER_02: Data team members are expected to partner with product managers on identifying the metrics that measure the success of features, and also with engineering teams on designing the data models that not just enable the feature to be developed, but also the analytics on top of that feature so that we can measure it.
• 7:14
  SPEAKER_02: And so another common aspect, of course, is going into tactical details, the infrastructure that they use.
• 7:19
  SPEAKER_02: And so, as I mentioned before, the bedrock of any data platform is its infrastructure for offline analytics.
• 7:24
  SPEAKER_02: And that's data collection, data storage, and transformation.
• 7:27
  SPEAKER_02: And it's also its orchestration infrastructure.
• 7:29
  SPEAKER_02: And so we centralize data storage in our data warehouse.
• 7:32
  SPEAKER_02: Snowflake, if there is any data at ramped, it lives within Snowflake.
• 7:35
  SPEAKER_02: We use DBT for transforming data within the data warehouse and building those pipelines.
• 7:40
  SPEAKER_02: And we also orchestrate all of this with Airflow.
• 7:43
  SPEAKER_02: And so Airflow is the central nervous system of our data platform and enables us to integrate different data from disparate data sources into one cohesive platform.
• 7:52
  SPEAKER_02: We've been using Airflow for a little over two and a half years now.
• 7:56
  SPEAKER_02: We have more than 100 active DAGs written by a little over 30 engineers spread across 10 different teams.
• 8:02
  SPEAKER_02: And so it's a broadly used and critical component of our platform.
• 8:07
  SPEAKER_00: Yeah, interesting.
• 8:08
  SPEAKER_00: And sorry, how many DAGs did you say that you had?
• 8:11
  SPEAKER_02: I think the number right now is 111.
• 8:13
  SPEAKER_02: So a little over Okay.
• 8:15
  SPEAKER_00: Okay.
• 8:15
  SPEAKER_02: So not a lot, but you know.
• 8:17
  SPEAKER_00: Yeah, not a super high number of DAGs, but like a very big breadth for what they're doing and spread quite broadly across your entire team or multiple teams internally, right?
• 8:27
  SPEAKER_02: Yeah.
• 8:28
  SPEAKER_00: Yeah, that's cool.
• 8:29
  SPEAKER_00: So tell me a little bit more about that airflow usage.
• 8:32
  SPEAKER_00: So you mentioned it's the nervous system tying everything together.
• 8:36
  SPEAKER_00: So how have you gone about integrating that with other data sources?
• 8:41
  SPEAKER_00: How does it help you kind of realize that that approach towards leveraging data that you just talked through?
• 8:47
  SPEAKER_02: I think Airflow strengths are its maturity and its integrations with existing systems.
• 8:54
  SPEAKER_02: And then also the established patterns of using it within data teams.
• 8:58
  SPEAKER_02: And so what that means is that if there's some third-party data source or maybe some third-party ETL tool for syncing data into our data warehouse, there's almost certainly an Airflow provider for that.
• 9:07
  SPEAKER_02: There's well-established patterns for running DBT within Airflow.
• 9:11
  SPEAKER_02: And so I think the mature ecosystem and community, as well as the existing integrations, that enables us to move really quickly because we're not thinking about the small details.
• 9:22
  SPEAKER_02: We can just pick up something off the shelf and move.
• 9:24
  SPEAKER_02: And so that I think combined with just the knowledge base around Airflow, it's enabled me to develop a top quality platform for building data pipelines and enables us to really easily scale this to different teams.
• 9:38
  SPEAKER_00: Yeah, that makes sense.
• 9:40
  SPEAKER_00: So those kind of features that you talked about of maturity and having a strong community and integrations with other tools, would you say those are things that are critical for your company in terms of how you build platforms in general?
• 9:56
  SPEAKER_00: Or is that more unique to your data orchestration?
• 9:58
  SPEAKER_00: Or how do you think about that?
• 10:00
  SPEAKER_02: Yeah, that's a good question.
• 10:01
  SPEAKER_02: I think it depends.
• 10:02
  SPEAKER_02: You have this maturity versus newcomer arguments.
• 10:05
  SPEAKER_02: There are certain parts of your infrastructure that you want a fairly mature player because you just need it to work.
• 10:12
  SPEAKER_02: But there are other parts of my platform, or our platform rather, that they're based on maybe more cutting-edge technologies.
• 10:19
  SPEAKER_02: Maybe they're solving a more niche problem.
• 10:21
  SPEAKER_02: And that's something where I would explore, let's say, a Series A startup or even younger to solve that for it.
• 10:26
  SPEAKER_02: So it really depends on a case-by-case basis.
• 10:30
  SPEAKER_00: Okay.
• 10:30
  SPEAKER_00: Interesting.
• 10:31
  SPEAKER_00: So you mentioned, you know, Airflow's maturity as one of its benefits.
• 10:36
  SPEAKER_00: So how have you guys changed your Airflow usage over time or how long have you been using it?
• 10:43
  SPEAKER_00: Has it changed?
• 10:44
  SPEAKER_00: What are some of the use cases that you've implemented over your tenure with it?
• 10:48
  SPEAKER_02: Yeah, I think our evolution of Airflow usage is probably familiar to other companies.
• 10:52
  SPEAKER_02: And so when we were initially using Airflow, so we started using Airflow in early 2022.
• 10:58
  SPEAKER_02: I think we went to production with Airflow.
• 11:00
  SPEAKER_02: Yeah, it was January 7th, 2022.
• 11:02
  SPEAKER_02: And our initial use case was actually just our DBT DAG.
• 11:05
  SPEAKER_02: And so we were running into step function.
• 11:06
  SPEAKER_02: Step functions is okay, but it has a lot of shortcomings that Airflow addresses really well.
• 11:11
  SPEAKER_02: And so the first task that I worked on was just building the Airflow platform just for replacing that step function state machine with an Airflow DAG.
• 11:18
  SPEAKER_02: So we were just supporting DBT and offline analytics at that point.
• 11:22
  SPEAKER_02: We realized that there were other use cases within capital markets engineering.
• 11:26
  SPEAKER_02: And so for automating some of their processes where maybe they're waiting for a response.
• 11:30
  SPEAKER_02: That's something that, let's say, a sensor would do really well with an Airflow.
• 11:33
  SPEAKER_02: We're waiting for files to show up with an S3, but we also want to execute just plain Python code.
• 11:38
  SPEAKER_02: Yeah, Airflow excels at that.
• 11:39
  SPEAKER_02: And so we work with the capital markets engineering team on building several DACs for their use cases.
• 11:44
  SPEAKER_02: And we rinsed and repeated that with a few more teams.
• 11:47
  SPEAKER_02: At a certain point late in 2022, we started looking at our first machine learning use cases.
• 11:52
  SPEAKER_02: And so we were just focused on mainly offline machine learning.
• 11:55
  SPEAKER_02: So that's daily batch model inference and maybe monthly model training.
• 11:59
  SPEAKER_02: And so we productionized our first machine learning model with that pattern with a combination of Airflow and then also a managed service provider for machine learning.
• 12:07
  SPEAKER_02: And we established we hardened patterns around that.
• 12:11
  SPEAKER_02: And I think where we're at now is we're using Airflow for its bread and butter use cases, data transformation, data or integrating data from third-party systems, but also applying it to machine learning use cases, mainly operational machine learning use cases.
• 12:25
  SPEAKER_00: Interesting.
• 12:25
  SPEAKER_00: Yeah, again, like a ton of breadth, right?
• 12:28
  SPEAKER_00: And a lot of the use cases that we see Airflow use both most frequently for and also like growing.
• 12:33
  SPEAKER_00: I think that made me think of the Airflow survey, which is currently live for the version this year.
• 12:39
  SPEAKER_00: We've seen the past couple of years that on the question of like has your Airflow usage changed over time or is it changing?
• 12:45
  SPEAKER_00: The majority of people are always saying that like their use cases are growing.
• 12:49
  SPEAKER_00: So definitely a typical pattern, like you said.
• 12:52
  SPEAKER_00: But yeah, that's really interesting.
• 12:54
  SPEAKER_00: You mentioned use of sensors for that more data-driven scheduling within Airflow.
• 13:00
  SPEAKER_00: Is that still something you do?
• 13:01
  SPEAKER_00: Do you also use data sets now that those are a little more mature since you first started?
• 13:05
  SPEAKER_00: I don't remember exactly what version we were on in early 2022, maybe like 2.3, something like that.
• 13:12
  SPEAKER_02: 2.3 had just come out, actually.
• 13:14
  SPEAKER_02: Yeah.
• 13:14
  SPEAKER_00: Okay.
• 13:14
  SPEAKER_00: Wow.
• 13:15
  SPEAKER_00: I'm actually amazed that I had that good of a guess.
• 13:18
  SPEAKER_00: But yeah, tell tell me about kind of your feature usage within Airflow.
• 13:21
  SPEAKER_00: What does that look like?
• 13:22
  SPEAKER_02: Yeah, I remember data sets.
• 13:24
  SPEAKER_02: It was released in Airflow 2.4, and we started using it almost immediately because you had some DAGs that would produce outputs and stuff like tables that other DAGs would depend on.
• 13:34
  SPEAKER_02: Let's say we had our DBT DAG that was deriving a table that contained features for some downstream predictive model that was going to run a daily batch inference job with.
• 13:44
  SPEAKER_02: Before datasets, cross-DAG dependencies was just not elegant.
• 13:48
  SPEAKER_02: Maybe you had to use an external task sensor or you had to trigger another DAG from an Airflow DAG.
• 13:53
  SPEAKER_02: But data sets gave us a more elegant approach.
• 13:55
  SPEAKER_02: The pattern that we use is that whenever something important, let's say a particular task within a DAG updates a data set, what we'll do is we'll put an empty operator after that task and we'll set the outlet of that empty operator to be, let's say, the analytics data set.
• 14:09
  SPEAKER_02: And then we'll have dependent DAGs depend on the changes in an upstream task that does the thing.
• 14:13
  SPEAKER_02: Those DAGs will schedule on that data set, the analytics data set.
• 14:16
  SPEAKER_02: And so those will trigger as soon as it reaches this empty operator step.
• 14:20
  SPEAKER_02: And so for us, that enables us to write more elegant airflow code.
• 14:23
  SPEAKER_02: We can also resolve just weird race conditions that we might come across with creative scheduling within Airflow or maybe multiple schedulers.
• 14:30
  SPEAKER_02: And that gives us a much more elegant experience for orchestrating data movements within our platform.
• 14:34
  SPEAKER_02: Yeah.
• 14:35
  SPEAKER_00: Yeah, cool.
• 14:36
  SPEAKER_00: And you mentioned DBT use again.
• 14:38
  SPEAKER_00: I think the DBT and Airflow are really common pairing.
• 14:41
  SPEAKER_00: What has your experience been like working with those two tools together?
• 14:45
  SPEAKER_02: It's been good.
• 14:46
  SPEAKER_02: We do it the old-fashioned way.
• 14:48
  SPEAKER_02: We have uh separate repos for Airflow and for DBT.
• 14:51
  SPEAKER_02: And our CSCD process for DBT builds a Docker image that we push to ECR.
• 14:55
  SPEAKER_02: And so with an Airflow, we just use a Kubernetes pod operator.
• 14:58
  SPEAKER_02: We run a DBT build bash command to orchestrate that.
• 15:01
  SPEAKER_02: And then we have alerting around uh, let's say, downstream failures in DBT models or DBT tests, and a an on-call rotation for analytics engineers to address those failures as they come up and then maybe rerun the or clear that specific Airflow task.
• 15:13
  SPEAKER_02: It works pretty well for us.
• 15:14
  SPEAKER_02: We're not doing anything super fancy, but works fine.
• 15:17
  SPEAKER_00: Yeah, cool.
• 15:18
  SPEAKER_00: Kind of the OG implementation for DBT and Airflow with the Kubernetes pod operator there.
• 15:24
  SPEAKER_00: A little more work to set up front, but I feel like once teams get really comfortable with Kubernetes pod operator, they tend to use it a lot because it's super flexible and it works well.
• 15:34
  SPEAKER_02: Definitely.
• 15:35
  SPEAKER_02: Yeah.
• 15:35
  SPEAKER_02: So it's coming handy for us, mainly for that use case.
• 15:37
  SPEAKER_00: Yeah.
• 15:38
  SPEAKER_00: Yeah, that's awesome.
• 15:39
  SPEAKER_00: Cool.
• 15:40
  SPEAKER_00: Well, anything else you would want to touch on, circling back to you know, your methodology towards building platforms, how you've talked a little bit about Airflow and DBT and the features that you use.
• 15:51
  SPEAKER_00: And do you see that kind of staying the same in the future?
• 15:54
  SPEAKER_00: You guys expanding a lot in terms of your platform usage internally?
• 15:58
  SPEAKER_00: What are you looking forward to the most?
• 16:00
  SPEAKER_02: What am I looking forward to the most?
• 16:02
  SPEAKER_02: Yeah, if you asked me in 2022, what would our platform look like?
• 16:05
  SPEAKER_02: I thought it would just be Airflow itself-like and DBT.
• 16:08
  SPEAKER_02: I thought that was it.
• 16:09
  SPEAKER_02: I didn't think we're going to do anything related to data science or anything related to, let's say, AI infrastructure or online analytics.
• 16:16
  SPEAKER_02: And something that's been a constant with us is change in new platforms.
• 16:19
  SPEAKER_02: And so we're constantly building.
• 16:21
  SPEAKER_02: And the general approach that we take to building out new platforms is we take a problem-centric approach rather than a platform-centric approach.
• 16:29
  SPEAKER_02: And platforms at ramp don't start as platforms.
• 16:32
  SPEAKER_02: They actually start as well, a novel application of some pattern or technology that solves a very specific business problem.
• 16:39
  SPEAKER_02: The platform engineer will be embedded in a specific engineering pod, will operate as if they're a member of that team, but their superpower is with data infrastructure and general data stuff.
• 16:48
  SPEAKER_02: With that, they'll roughly follow the same life cycle for developing platforms.
• 16:52
  SPEAKER_02: They first build this novel pattern or technology with the other engineers on that engineering pod to narrowly solve that business problem.
• 16:59
  SPEAKER_02: Then they do that again, two or three more times.
• 17:02
  SPEAKER_02: They're not really thinking about scalability or developer experience of the platform at that stage, nor are they thinking about building things the right way.
• 17:10
  SPEAKER_02: They're only focused on building applications and they're taking every shortcut possible in order to ship as quickly as possible within reason.
• 17:18
  SPEAKER_02: They're not, let's say, bringing down prod regularly.
• 17:20
  SPEAKER_02: But anyway, after they finish these cycles, the platform engineer will review these applications and determine what's in common among them that can be modulized.
• 17:27
  SPEAKER_02: They'll think about common bugs or production incidents that have happened, but they'll also think about friction points in the development lifecycle.
• 17:34
  SPEAKER_02: And so with these insights, they'll then platformise this novel pattern by building the tooling to support observability, automation, documentation, and then also development requirements to support many applications built by many developers.
• 17:47
  SPEAKER_02: And then they'll go back to building applications.
• 17:48
  SPEAKER_02: Maybe they'll partner with a second engineering team on their applications.
• 17:51
  SPEAKER_02: And at this point, they've organizationally scaled the platform.
• 17:54
  SPEAKER_02: They'll continue alternating between building the applications on top of the platform and then hardening the platform so that we can continue scaling it reliably until the end of the platform's life.
• 18:04
  SPEAKER_02: And so let me give an example of a fairly new part of our platform that we built last year.
• 18:08
  SPEAKER_02: And so late last year was actually it was during the Airflow summit last year.
• 18:12
  SPEAKER_02: I we realized that we needed to introduce an OLAP database into our infrastructure.
• 18:17
  SPEAKER_02: And so this is a component of our application stack.
• 18:20
  SPEAKER_02: And I led the investigation on that and I realized that click house is the best option.
• 18:25
  SPEAKER_02: And the problem that we were trying to address is that there were certain parts of our application that just didn't work.
• 18:30
  SPEAKER_02: Our internal financing, they also complained about it.
• 18:32
  SPEAKER_02: They didn't really voice a lot of pain points because they don't want a bad RAM.
• 18:37
  SPEAKER_02: But yeah, they effectively weren't using uh certain parts of the product because it wasn't usable.
• 18:41
  SPEAKER_02: So that was a huge problem.
• 18:42
  SPEAKER_02: The underlying issue is that we were running analytical queries against Postgres, a transactional database.
• 18:47
  SPEAKER_02: And then also we were hitting data models that required us to perform, let's say, 20 or 30 joins.
• 18:51
  SPEAKER_02: And so that's why the performance was so poor.
• 18:54
  SPEAKER_02: And so the approach that we realized that we need to take was first of all, use a different database.
• 18:58
  SPEAKER_02: We wanted to use an analytics database, Clickhouse in this case, for these analytical style queries.
• 19:03
  SPEAKER_02: But not just that, but we we needed a different, more denormalized data model so that we were performing less joins.
• 19:08
  SPEAKER_02: Because analytical databases, they don't handle joins incredibly well, especially in these online use cases.
• 19:13
  SPEAKER_02: And so in late September, we set out to build uh what we're now calling our OLAP platform, but it really just started as we're using ClickHouse to make things faster in our application.
• 19:22
  SPEAKER_02: We built uh a single monolith data model for spend events uh at ramp.
• 19:27
  SPEAKER_02: We built a process to sync data from Postgres into ClickHouse, but also denormalize it in flight.
• 19:34
  SPEAKER_02: Uh, and then we modified the routes in our back end application uh to point at ClickHouse rather than Postgres.
• 19:39
  SPEAKER_02: And then we flipped it on for our large enterprise customers.
• 19:42
  SPEAKER_02: They noticed immediately the performance improvements, let's say 100x speed up in these routes.
• 19:46
  SPEAKER_02: It was actually usable and not just usable, but fast.
• 19:48
  SPEAKER_02: And then we did it again, three or four more times on several different other applications.
• 19:53
  SPEAKER_02: And around January is when we started thinking about platformizing the OLAP platform, establishing patterns around the development lifecycle, improving reliability and observability, and then also developer tooling.
• 20:05
  SPEAKER_02: And today we have tens of use cases built on top of Clickhouse and many developers across our product engineering team using it.
• 20:11
  SPEAKER_02: So we've organizationally scaled it.
• 20:12
  SPEAKER_02: Yeah.
• 20:12
  SPEAKER_02: And that's how all platforms are built at Ramp.
• 20:14
  SPEAKER_02: For instance, our playmates platform started as an enhancement that we made to support ramp build pay, but now it supports many use cases for as sending payments within RAMP.
• 20:22
  SPEAKER_00: Yeah, interesting.
• 20:23
  SPEAKER_00: Common story, right?
• 20:24
  SPEAKER_00: I think things arise out of issues.
• 20:26
  SPEAKER_00: But that's cool that you've taken the mentality towards it of let's figure out a way to fix this problem that creates a better platform for us to move forward and then build off of so that you're not just in the firefight all the time, right?
• 20:38
  SPEAKER_00: But still working towards building something better for your product and for your developers to use.
• 20:44
  SPEAKER_00: Yeah, exactly.
• 20:45
  SPEAKER_00: Yeah, that's awesome.
• 20:46
  SPEAKER_00: I know we're getting close to the end of our time.
• 20:49
  SPEAKER_00: So I'm gonna ask you another looking forward question.
• 20:52
  SPEAKER_00: This is the standard question that I ask everybody at the end of our episodes.
• 20:56
  SPEAKER_00: And that is, what is one thing that you would like to see from the Airflow project in the near future?
• 21:01
  SPEAKER_02: Yeah, I think the main thing for me is that I would love to see Airflow become lighter weight.
• 21:08
  SPEAKER_02: And then I would also love to see, let's say, the DAG authoring experience to be more concise, maybe more idiomatic Python.
• 21:15
  SPEAKER_02: I guess like what I mean is by lighter weight is if I want to run Airflow locally, I have to spin up a scheduler, I have to spin up a triggerer and a Postgres database.
• 21:23
  SPEAKER_02: I think there's yeah, a web server too, in order to run DAGs.
• 21:26
  SPEAKER_02: I would love to just be able to just run my DAG as just straight Python code.
• 21:29
  SPEAKER_02: I think you might be able to do this with DAG.test.
• 21:32
  SPEAKER_02: I haven't tried that out.
• 21:33
  SPEAKER_02: But another aspect, I think the task flow API, I think it's a step in the right direction, but I would love to be able to just say write straight Python code and then also use the existing operators off the shelf from different providers in, let's say, a task flow-like API.
• 21:47
  SPEAKER_02: But I'm just writing Python code, I'm just writing straight Python functions and invoking functions and other functions.
• 21:53
  SPEAKER_02: And the Airflow diagram experience, it's it's intuitive, but it's not native Python.
• 22:02
  SPEAKER_02: It it it's it feels very different.
• 22:03
  SPEAKER_02: And so I I I I wish it felt more like native Python.
• 22:07
  SPEAKER_00: Okay, yeah, that makes sense.
• 22:08
  SPEAKER_00: Have you used the object storage feature at all?
• 22:11
  SPEAKER_00: It's came out in 2.8.
• 22:14
  SPEAKER_00: I think it's experimental.
• 22:16
  SPEAKER_00: It's also not the same as Taskflow, but similar to what you're describing, where it is a little bit more Python forward, but designed for interacting with obviously object storage.
• 22:28
  SPEAKER_00: So you're moving data in between file systems.
• 22:31
  SPEAKER_00: So it simplifies that a lot from having writing your functions.
• 22:34
  SPEAKER_00: But I think it came out of an idea very similar to what you're describing that was actually a talk at not this year's summit, but last year's summit, basically saying that task flow should be sort of the paradigm and everything else should fit into it as opposed to these operators that are chunked in terms of like a building block in terms of how you would write the DAG.
• 22:54
  SPEAKER_00: But so certainly something we've heard before from the community.
• 22:58
  SPEAKER_00: I don't know of any particular developments on that specifically.
• 23:01
  SPEAKER_00: I know Airflow 3 will have the ability to run in other languages and potentially some updates to the data sets feature in terms of tracking data assets that will be a little bit more, but I don't know about big changes to the way that DAGs are defined themselves.
• 23:16
  unknown: Cool.
• 23:17
  SPEAKER_02: Yeah, I remember that.
• 23:18
  SPEAKER_02: That was actually at last year's Airflow summit when I can't remember who gave that talk, but it was a really interesting talk.
• 23:22
  SPEAKER_02: And I liked as a parallel as like, yeah, you have an S3 to GCS operator and GCS to S3 operator.
• 23:27
  SPEAKER_02: And why do we have two of these things?
• 23:29
  SPEAKER_02: We have this weird Cartesian products of different object storage.
• 23:31
  SPEAKER_02: So we should just have one different system for that.
• 23:33
  SPEAKER_02: So it was a really insightful talk.
• 23:35
  SPEAKER_00: Yeah, Vulcade.
• 23:37
  SPEAKER_00: So yeah, he was one of the one of the Airflow PMC members.
• 23:40
  SPEAKER_00: Yeah, like I said, I don't know if there's any plans on expanding that object storage feature.
• 23:44
  SPEAKER_00: It's still experimental.
• 23:45
  SPEAKER_00: I do think it's pretty cool.
• 23:47
  SPEAKER_00: My team actually uses it quite a lot in the DAGs that we do and reference architectures and things like that.
• 23:52
  SPEAKER_00: It's nifty.
• 23:53
  SPEAKER_00: Yeah, maybe give it a look.
• 23:54
  SPEAKER_00: But cool.
• 23:56
  SPEAKER_00: I'll check it out.
• 23:57
  SPEAKER_00: Cool.
• 23:57
  SPEAKER_00: Well, I know we're at time, Ryan.
• 23:59
  SPEAKER_00: It's been a pleasure talking to you.
• 24:00
  SPEAKER_00: Thanks so much for coming on.
• 24:02
  SPEAKER_00: What is the best way for folks to get in touch with you?
• 24:05
  SPEAKER_02: Oh, LinkedIn.
• 24:06
  SPEAKER_02: Just shoot me a message.
• 24:07
  SPEAKER_02: Uh, I'm normally pretty responsive.
• 24:08
  SPEAKER_00: Okay, great.
• 24:09
  SPEAKER_00: Well, thanks again, Ryan.
• 24:10
  SPEAKER_00: This was a great conversation.
• 24:12
  SPEAKER_02: Yeah, it was a lot of fun.
• 24:13
  SPEAKER_02: Thanks so much.
• 24:16
  SPEAKER_01: Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
• 24:23
  SPEAKER_01: You'll find links in the show notes to any resources mentioned in today's show.
• 24:26
  SPEAKER_01: If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.