Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / Expanding the Data Engineering Toolkit at Reddit

Transcript

Expanding the Data Engineering Toolkit at Reddit · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_00: Airflow with the Reddit team as it's grown has been a gateway for that.
• 0:04
  SPEAKER_00: I've seen data scientists transition to be ML engineers, transition to be more backend engineers, go to become a PM.
• 0:11
  SPEAKER_00: And in many ways, I totally see that trend that you described happening because of those access patterns where they're, you know, they're able to get more data at their fingertips to build more powerful models.
• 0:21
  SPEAKER_00: Oh, they lean a little bit more that way.
• 0:22
  SPEAKER_00: They're able to build more ETLs and report and visualize more trends or more behaviors.
• 0:27
  SPEAKER_00: They lean a little bit more towards the PM side.
• 0:29
  SPEAKER_00: So you're totally on the money in there, where it's you you can sort of change even people's career trajectories by giving them more visibility and access.
• 0:38
  SPEAKER_01: Hi there, and welcome to this week's episode of the Airflow Podcast, brought to you by Astronomer.
• 0:43
  SPEAKER_01: This week, we met up with Ben Weisgarver, a staff data scientist at Reddit who runs their data warehousing and data engineering functions.
• 0:51
  SPEAKER_01: Our conversation with Ben touches on everything from Airflow as a tool for career mobility across the data stack to scaling out a self-service data architecture across many teams.
• 1:02
  SPEAKER_01: Before we get started, I wanted to mention that our team at Astronomer is growing rapidly, and we're on the hunt for new folks to join in a variety of different roles.
• 1:10
  SPEAKER_01: If you're passionate about Airflow and interested in building the future of data engineering, please get in touch.
• 1:15
  SPEAKER_01: You can check our job postings at careers.astronomer.io.
• 1:19
  SPEAKER_01: But we're constantly updating our listings to accommodate our quickly growing company and needs, so if you don't see a perfect job fit but are passionate about what we're doing, please feel free to email me directly at pet at astronomer.io.
• 1:31
  SPEAKER_01: Without further ado, here's Ben.
• 1:38
  SPEAKER_01: Ben, welcome to the podcast.
• 1:40
  SPEAKER_01: We're very excited to have you and hear about what you're working on over at Reddit.
• 1:45
  SPEAKER_01: For starters, can you tell us a little bit about yourself, your background, and what you're currently working on?
• 1:50
  SPEAKER_00: My name's Ben Weiscarver right now.
• 1:51
  SPEAKER_00: I manage the data engineering and data warehousing teams at Reddit.
• 1:55
  SPEAKER_00: So this is what we call collectively data platforms.
• 1:58
  SPEAKER_00: At a high level, these teams are responsible for owning the entire event collection and ingress systems for all of our consumer data.
• 2:06
  SPEAKER_00: And we additionally own all the aggregation and transformation for canonical data sets and delivery so that we can power our data science and our product teams to make data-informed decisions and to have all the access to not only raw, but also aggregate and canonical data sets.
• 2:22
  SPEAKER_00: I've been at Reddit for a little over three years now.
• 2:25
  SPEAKER_00: I came from a data science background, specifically in gaming.
• 2:29
  SPEAKER_00: So, you know, coming to Reddit with a much larger scale and much more user-centric focus was super exciting for me.
• 2:36
  SPEAKER_00: A lot of the projects over the past couple of years that I've worked on at Reddit range from standing up our use of Airflow and that as an aggregation tool and sort of reporting infrastructure.
• 2:45
  SPEAKER_00: I helped a lot with our experimentation platform and getting that launched across all of our engineering pods and product pods.
• 2:51
  SPEAKER_00: And now I'm helping out, as I mentioned, with engineering and warehouse teams to really make that repeatable and defined service for any team at Reddit using data or getting access to data.
• 3:01
  SPEAKER_01: What team specifically do you work on at Reddit?
• 3:03
  SPEAKER_01: And what is that team's purpose in the broader context of organization?
• 3:06
  SPEAKER_00: The two teams I work on right now, data engineering, I basically ingest all client events for we call them consumer events.
• 3:14
  SPEAKER_00: And what that means is for our native apps kit teams, for our product teams such as growth and SEO, and we have moderation teams, all of them have their own engineers that generate raw client events that they use to inform decisions, to make reporting decisions, and also to understand new user behaviors and trends that are occurring on the site.
• 3:34
  SPEAKER_00: The data engineering team is responsible for owning and building the event pipeline that collects all of those client events.
• 3:41
  SPEAKER_00: So as you imagine, there's a ton of volume coming in from all of our different clients from AS and Burhouse and AMP and whatnot.
• 3:47
  SPEAKER_00: And part of the big challenge there is making that endpoint accessible and scalable for all these teams to use successfully.
• 3:55
  SPEAKER_00: And another big part of the deliverable there is making those events accessible to downstream partners.
• 4:01
  SPEAKER_00: One of the big downstream partners for data engineering is the second team I work with data warehouse.
• 4:05
  SPEAKER_00: So they're going to be responsible for transforming, aggregating, and really understanding a lot of that raw data to make it more usable, more discoverable, and more documented for any number of our data users and data customers.
• 4:20
  SPEAKER_00: Beyond that, it goes down to serve our data science pods, which would be embedded in all those product teams I mentioned, our SEO and growth and moderation teams.
• 4:28
  SPEAKER_00: So the whole pipeline of raw events to aggregate data or reporting data that's used of the company.
• 4:36
  SPEAKER_00: The first two big stages are earned by the teams that I work with.
• 4:39
  SPEAKER_00: So it really begins at the raw event collection down through the aggregation and delivery of data.
• 4:44
  SPEAKER_00: And then it opens up to all the product teams in a variety of ways that they use that from model creation, reporting, predictive analyses, user segmentation or cohorting, and also just basic counting, making sure we have no anomalous behavior in terms of votes or comments or users.
• 5:00
  SPEAKER_02: Never underestimate the complexity of basic counting.
• 5:03
  SPEAKER_00: Yeah.
• 5:03
  SPEAKER_00: One of the two hardest things in engineering are counting and naming services, right?
• 5:07
  SPEAKER_00: Totally true, right?
• 5:09
  SPEAKER_00: Totally true.
• 5:09
  SPEAKER_02: It sounds like a website operating at Reddit scale with all the different things that you have to support from your users, your backend services, and so on.
• 5:17
  SPEAKER_02: You almost use like a ton of tools through your data stack.
• 5:20
  SPEAKER_02: Before we're diving into Airflow in particular, can you just paint a picture of some of the tools that you're using for different layers of NGS, compute, database, et cetera?
• 5:28
  SPEAKER_00: At a high level for the data platform teams, there's a few standouts.
• 5:32
  SPEAKER_00: First is we do have a lot of Humbream tools, our entire event collector, and some of the processes that are transforming or removing PII data from those raw events.
• 5:42
  SPEAKER_00: Our all Humbrew solutions, so we build them in house, and those are the endpoints that we use to receive all those raw client events.
• 5:50
  SPEAKER_00: From there, we treat right now BigQuery as our main warehouse.
• 5:54
  SPEAKER_00: So we stream all of those events into BigQuery, and that's where we access it from those data science and data warehouse teams.
• 6:00
  SPEAKER_00: I guess technically it streams into GCP and then we write it into BigQuery.
• 6:04
  SPEAKER_00: From there, we go into many tools.
• 6:06
  SPEAKER_00: It sort of leaves the data platform space and enters data science and product areas.
• 6:11
  SPEAKER_00: A lot of the usual suspects were a Python-based shop.
• 6:14
  SPEAKER_00: So you know, Jupyter is a big thing.
• 6:16
  SPEAKER_00: We use mode for basic reporting.
• 6:18
  SPEAKER_00: We've used other BI tools in the past, Colab, Pater Miller are some of the ways that we use and make Jupyter more powerful or notebooks more powerful from a data science perspective.
• 6:28
  SPEAKER_00: If you look back to the event collector pipeline and how we process world events, another big tech we use is Kafka.
• 6:36
  SPEAKER_00: We stream all these events in real time with what we call our Kafka pipeline.
• 6:40
  SPEAKER_00: And that actually gets split out into a pretty wide decision tree of streaming services, depending on how customers need access to real-time events.
• 6:51
  SPEAKER_00: In many cases, BigQuery and our data warehouse is a perfect solution for getting access and further aggregating data and transforming it into a useful service.
• 7:01
  SPEAKER_00: Other times, certain teams need more immediate access to those raw events, and it'll fall into a decision tree of coming out of a Kafka topic is a CaseSQL or a Flink consumer or Python consumer built on an ARP stack a more appropriate solution.
• 7:15
  SPEAKER_00: From a consumer side of things, those are some of the big techs, some of the big technologies that we use to not only deliver those events in real time, aggregate them, and then ship them off to data science.
• 7:26
  SPEAKER_00: And I know you said we put a pause on Airflow for a minute, but that would be the next big technology we use for anything that lands in BigQuery.
• 7:33
  SPEAKER_00: We use Airflow to actually govern the egress of data for out of air out of BigQuery into any of the other product services, whether that's uh reports, their own databases, or some of the downstream services that are using that data to modify the user's experience.
• 7:49
  SPEAKER_02: I think one thing that you said is pretty fascinating to me there.
• 7:51
  SPEAKER_02: You know, you use a bunch of homegrown tools along with a bunch of standard open source solutions.
• 7:56
  SPEAKER_02: For a company facing such scale challenges as yourselves, how do you really walk the line between trying to adopt something that's maybe a little more standard, but isn't exactly a perfect fit versus something that's a little more like LaFit, but might not be the most uh standard thing in the world?
• 8:13
  SPEAKER_00: That's a great question.
• 8:15
  SPEAKER_00: And to be honest, Reddit has to make that decision, not with every service that we launch uh from a data perspective, but with many of them.
• 8:23
  SPEAKER_00: And that's not unique to the data platform teams, but also to our machine learning teams, our search teams, our growth and our push notification services.
• 8:31
  SPEAKER_00: They all have had to make this decision.
• 8:33
  SPEAKER_00: I would say the most frequent path that we hit is for a little bit of background, you know, Reddit has an absurdly large user base.
• 8:42
  SPEAKER_00: We're ingesting petabytes of information every day.
• 8:45
  SPEAKER_00: And the company is still not that big.
• 8:47
  SPEAKER_00: So we're growing, but we're under a thousand employees.
• 8:50
  SPEAKER_00: And the ratio of you know engineer to data that that engineer is responsible for is massive.
• 8:56
  SPEAKER_00: So one of the biggest decision points in what service we use, when do we do something in-house, when do we pick open source versus a third-party solution often comes down to almost a budget and a time management decision, where we can look at the team structure, the bandwidth we have with our current staff.
• 9:15
  SPEAKER_00: And sometimes something is a high priority enough where we can invest our existing resources and say, we are gonna build this from the ground up, we're gonna make it perfect for Reddit and our use cases because it's gonna be a three to five to 10 year solution.
• 9:30
  SPEAKER_00: So that's an investment that's easy to fight for internally.
• 9:33
  SPEAKER_00: Other times, as you alluded to, you know, working in this area or this tech space, things change really quickly.
• 9:40
  SPEAKER_00: And there's a number of services that we use open source or third-party tools, where the decision was made only because we couldn't guarantee that this was going to be a five-year bet.
• 9:50
  SPEAKER_00: So, from our bandwidth perspective, it was a way better decision to get something sped up really quickly with the help of another company's existing use case or another company that is providing a service so that we can start to see the benefits, get these services out the door for our customers, and not have to worry about the long-term maintenance or signing ourselves up for this investment for the next six or eight quarters.
• 10:14
  SPEAKER_00: Does that give you a little bit of context though?
• 10:16
  SPEAKER_00: It's I know that's a little bit of a high-level answer.
• 10:17
  SPEAKER_01: It's an L on the head for me.
• 10:19
  SPEAKER_01: I think that's a really helpful context to have.
• 10:21
  SPEAKER_01: You don't want to spend necessary bandwidth on tools that you don't know are going to be the long-term commitment.
• 10:27
  SPEAKER_01: So going with the lowest hanging fruit, so to speak, makes sense in a lot of those contexts.
• 10:32
  SPEAKER_01: I want to step back briefly and just try to sequence some of the overall architecture that we were discussing prior.
• 10:38
  SPEAKER_01: It seems to me that all of the many, many events that are coming from Reddit's clients are being picked up by Kafka hues and streamed to BigQuery.
• 10:49
  SPEAKER_01: And then you probably have a lot of BigQuery, which then is egressed by Airflow into other systems.
• 10:56
  SPEAKER_01: Is that an accurate statement?
• 10:57
  SPEAKER_01: Probably transformed as well.
• 10:59
  SPEAKER_00: Yeah, that's exactly right.
• 11:00
  SPEAKER_00: And the branch that is sort of not covered by the Airflow section when events are landing in Kafka and getting distributed to all the topics and eventually landing in our warehouses or on the airflow.
• 11:12
  SPEAKER_00: We build in branches so that teams can access those raw events in real time outside of the warehouse so they can build their own consumers and listen for changes if they need access to things as they're occurring rather than the batch or aggregated data services.
• 11:25
  SPEAKER_00: But everything else you said is totally on point.
• 11:28
  SPEAKER_01: Sure, that definitely makes sense.
• 11:29
  SPEAKER_01: I'm kind of curious.
• 11:30
  SPEAKER_01: I've never actually thought about this before.
• 11:32
  SPEAKER_01: And I don't think this is something we've asked on the podcast.
• 11:34
  SPEAKER_01: Was Kafka part of that stack before Airflow or after Airflow?
• 11:38
  SPEAKER_01: Or were they brought in at the same time?
• 11:41
  SPEAKER_00: It was slightly after we brought on Airflow.
• 11:45
  SPEAKER_00: Uh so we've had a number of data ingestion solutions over my tenure at Reddit.
• 11:50
  SPEAKER_00: Kafka's been around for a couple of years in terms of us using it to stream our massive raw events.
• 11:56
  SPEAKER_00: Uh, it's only been in the past probably year and a half that we've transformed that into a core of our streaming solution for not only getting that data into BigQuery, but also getting that data into our specific consumers for our growth teams or or uh native apps teams or whatnot.
• 12:12
  SPEAKER_00: Airflow existed a little bit before then, where we had some other services to do more batch operations, slightly smaller scale, but we were also a younger company and smaller team.
• 12:20
  SPEAKER_01: When did Airflow come into the picture and what problem was it designed to solve in the Yeah, Airflow came to the picture right before I joined the company?
• 12:29
  SPEAKER_00: So this would be mid or you know spring of 2017.
• 12:34
  SPEAKER_00: It's a pretty interesting story.
• 12:35
  SPEAKER_00: Our first Airflow instance was actually the result of a hackathon where we have a quarterly hack week where engineers can pick projects that they're interested in and we'll move the company forward.
• 12:46
  SPEAKER_00: It can be weak spots, it can be blind spots or whatnot.
• 12:49
  SPEAKER_00: The original goal was to give our very small and growing data science team the ability to automate and schedule a lot of their most frequent queries, uh, most frequent reporting tasks.
• 13:01
  SPEAKER_00: At the time, we were using another company that was doing some of that aggregation for us, but it was, you know, it came to like a hive where you have to schedule a bunch of these parallel jobs and they have to output somewhere.
• 13:11
  SPEAKER_00: It was a pretty messy to maintain and manage with such a small team.
• 13:15
  SPEAKER_00: So the result of the hackathon was getting an airflow instance up, and almost immediately we had more access to our raw events.
• 13:23
  SPEAKER_00: And it sounds crazy, but at the time we were doing some things for the very first time, like measuring uh you know accurate session time and getting a really robust way to measure retention, getting more access into these user trends that you think are like P0 things.
• 13:37
  SPEAKER_00: As soon as you have data, you want to measure your install inflection points, you want to measure retention and whatnot.
• 13:41
  SPEAKER_00: And it was, that's why it's interesting.
• 13:43
  SPEAKER_00: It wasn't really until we stood up Airflow and gave the data science teams access into that scheduling and automation aspects of the tool that we started to see some of those things get defined at the company level.
• 13:55
  SPEAKER_02: One thing that we've seen a lot is once you give users an easy or standard way to do something, they tend to take a lot more ownership over it and flex their creative muscles a lot more.
• 14:04
  SPEAKER_02: Data scientists might be more willing to experiment with a certain model or change the inputs for a model that's already running if they have an easy way to go about doing that.
• 14:12
  SPEAKER_02: You know, it almost like giving an artist different colors of paint allows them just to do a whole bunch of other things.
• 14:17
  SPEAKER_02: So, can you talk about a little bit in the context of how Airflow use has grown through Reddit over that time?
• 14:22
  SPEAKER_02: What we find a lot of times is data engineering is never complete.
• 14:25
  SPEAKER_02: You know, you do your first set of workloads or something, and then what that actually ends up making is more work for the data engineers because now there's kind of deeper insights to be had, and it's a snowball effect, given all the different things you're dealing with.
• 14:36
  SPEAKER_02: How organic was that process at Reddit?
• 14:38
  SPEAKER_00: I'll walk through that.
• 14:39
  SPEAKER_00: I wanted to address the one thing you said real quick around giving artists more tools or giving more access to data.
• 14:46
  SPEAKER_00: I love that point.
• 14:47
  SPEAKER_00: I just wanted to share specifically Airflow with the Reddit team as it's grown, has been a gateway for that.
• 14:52
  SPEAKER_00: I've seen data scientists transition to be ML engineers, transition to be more back-end engineers, go to become a PM.
• 15:00
  SPEAKER_00: And in many ways, I totally see that trend that you described happening because of those access patterns where they're, you know, they're able to get more data at their fingertips to build more powerful models.
• 15:10
  SPEAKER_00: Oh, they lean a little bit more that way.
• 15:11
  SPEAKER_00: They're able to build more ETLs and you know, report and visualize more trends or more behaviors.
• 15:16
  SPEAKER_00: They lean a little bit more towards the PM side.
• 15:18
  SPEAKER_00: So you're totally on the money in there, where it's you, you can sort of change uh even people's career trajectories by giving them more visibility and access.
• 15:26
  SPEAKER_00: I love that point.
• 15:27
  SPEAKER_00: It's uh it's a really good thing to brag about for what it looks like when you introduce a great tool for a company.
• 15:32
  SPEAKER_00: As far as adoption, getting back to that question, I would describe the airflow adoption of the company tied to two main buckets of changes, I guess.
• 15:44
  SPEAKER_00: One would be in the in the beginning, you know, I mentioned it came out of a hackathon where it was, you know, one person spinning up an instance and connecting it to all of our data stores and getting things operational for the first time.
• 15:56
  SPEAKER_00: And the first bin of that adoption was really people like that, a few data scientists, uh, a few engineers, myself for a long time, just got really invested in promoting Airflow as an ETL construction tool, Airflow as a modeling feature building tool, Airflow as a way to check for anomalies and get those reports in PMs and directors' hands.
• 16:18
  SPEAKER_00: And it was those efforts that were ad hoc and almost individual that powered adoption for quite some time, where you know, we'd learn the new ability, we'd learn a new pattern that worked for the team, we'd see things that stick, we'd see things that wouldn't.
• 16:32
  SPEAKER_00: And it was super self-driven, but also at the same time really helpful because we had our customers and the wins, like the spaghetti that did stick to the wall.
• 16:40
  SPEAKER_00: It was like people could react to that really quickly because they were driving the progress themselves.
• 16:44
  SPEAKER_00: The second change that I think was the biggest driver of adoption, seeing teams hook up the exhaust of Airflow, of like the output of these ETLs and new data services to other downstream services that were visible on the product.
• 17:01
  SPEAKER_00: A specific example would be our push navigation strategy and our content recommendation features.
• 17:06
  SPEAKER_00: Right now we have examples in both of those bins that are powered by Airflow, where we can use raw client events landing in BigQuery, we can transform and migrate that data anywhere within the Reddit ecosystem.
• 17:18
  SPEAKER_00: And because it's loading in Airflow and is super lightweight, it's pretty scalable, it's really iterable, and we can move quickly and fastly.
• 17:27
  SPEAKER_00: The first services like that that were stood up were almost proof points to teams, and it showed them that, oh my gosh, like that's like a light bulb moment where we can take this data that was previously used only for reporting and almost inaccessible from a consumption perspective, and we can transform it into valuable data that we can shape a user's session with.
• 17:46
  SPEAKER_00: We can make better content recommendations, we can make a user's sessions safer, we can identify threats or harmful behavior, even without consuming data from a certain real-time data stream or whatnot.
• 17:58
  SPEAKER_00: The combination of those two things, those those individuals that were really driving the conversation and those services that were consuming the exhaust of whatever you build up in Airflow were the two biggest drivers.
• 18:10
  SPEAKER_00: Once we got to that point where we had more than one person driving the conversations and more than one service showing the impact or results, that's when we saw that flywheel effect that you always year and after is like, oh, more teams are starting to use this.
• 18:24
  SPEAKER_00: We have more process and more rules around the safe ways and the effective ways to protect users and to have a good impact on the on the Reddit.
• 18:32
  SPEAKER_00: And that eventually led us to where we are now, where we have Airflow is just one of the tools that we use to allow access to data, but it's still the one that we use that's um fastest to pivot and fastest to iterate on.
• 18:45
  SPEAKER_01: Yeah, I feel like that entire narrative is almost like the poster trial for the modern data-driven organization.
• 18:52
  SPEAKER_01: You create new data services that drive additional business value that then create additional data services that can be leveraged for additional business value.
• 19:01
  SPEAKER_01: You end up getting this internal flatwheel effect wherein you're just constantly using the data at your disposal to drive value.
• 19:10
  SPEAKER_01: Obviously, in Reddit's case, that effect is just so significant because of the sheer volume.
• 19:15
  SPEAKER_01: I just love that story and I think it's such a beautiful narrative.
• 19:18
  SPEAKER_01: I'm curious to know what some of the most interesting use cases that Airflow is driving are today, maybe at one layer lower than we just discussed.
• 19:28
  SPEAKER_01: You know, after Airflow was brought on and introduced all of these new downstream data services and then expanded to other teams and other use cases, what come to mind as some of the top use cases?
• 19:40
  SPEAKER_00: In my mind, there's a couple that stand out.
• 19:42
  SPEAKER_00: They stand out mostly because they replace services that would be vastly more expensive to build and scale than if you were to do them in Airflow, which we do precisely for those reasons.
• 19:54
  SPEAKER_00: Uh, the first one is we have a number of teams that use Airflow to do some mild aggregation or transformations on those role events from BigQuery and ship that data out to places that are easier to access from a client perspective.
• 20:07
  SPEAKER_00: So we you know write data to S3 or whatnot.
• 20:10
  SPEAKER_00: That use case we call it the unified feature store.
• 20:12
  SPEAKER_00: So it's a way to have a consistent approach to access data, regardless if you're on the growth team trying to recommend content for push notifications, or if you're standing up a new model and you want to ship some data for an experiment to consume to see how it impacts for better or for worse a user's engagement.
• 20:29
  SPEAKER_00: Those are very, very different use cases.
• 20:32
  SPEAKER_00: Both of them can use this, you know, the unified feature store, as we as we call it internally, to get really, really easy access data or accessible data from BigQuery, quickly write it out to S3, and then the engineering teams.
• 20:44
  SPEAKER_00: They don't have to build a huge consumer.
• 20:47
  SPEAKER_00: They don't have to worry about data latency or data quality because it's already coming from the warehouse.
• 20:52
  SPEAKER_00: All they have to do is read some stuff from S3.
• 20:55
  SPEAKER_00: And that makes it so fast and honestly so cheap to build, as opposed to like a streamy consumer or you know, another armada of AWS instances that a team needs to maintain and deploy.
• 21:07
  SPEAKER_00: And we've seen dozens of features from big to small from experiments where we shift, you know, slightly tune a feature, slightly change the UX a little bit, uh, onto content recommendation systems that are driven by this approach.
• 21:20
  SPEAKER_00: That's the number one example is using Airflow as a transformation plus migration tool to a more engineering friendly surface.
• 21:28
  SPEAKER_00: So you don't have to read straight from BigQuery, you don't have to listen to raw events or streaming events.
• 21:34
  SPEAKER_00: You can just write your aggregate files to wherever your team currently reads from and get access to really, really powerful client data.
• 21:41
  SPEAKER_00: If you extrapolate that to everything else that you can imagine, the data we're collecting from clients is when the user votes or comments or posts.
• 21:49
  SPEAKER_00: A lot of that is publicly accessible when you just browse Reddit.
• 21:52
  SPEAKER_00: But there's a whole other collection of signals that are super, super impactful or important than you can imagine or you know a User spends a certain amount of time on a page, a user reacts to another comment, a user sees something for the first time or joins a subreddit for the first time.
• 22:08
  SPEAKER_00: Those are all hugely valuable signals.
• 22:10
  SPEAKER_00: And giving teams access to that data for their own feature engineering and sort of development in a way that doesn't require them to communicate with the back end to request that information from like an API or something makes it possible for them to move really, really quickly.
• 22:25
  SPEAKER_00: The other example I have is Reddit has a couple of different data stores that complement the raw event stream we use.
• 22:32
  SPEAKER_00: So we've talked a lot about these raw events, these streaming events coming from our clients.
• 22:36
  SPEAKER_00: It's you know when a post comes in, when a comment occurs, when a user subscribes to a subreddit or whatnot.
• 22:42
  SPEAKER_00: Really, really valuable and important type of data that we collect.
• 22:46
  SPEAKER_00: The other is more of like a state or a static-based information.
• 22:50
  SPEAKER_00: As you imagine, you know, Reddit is driven by Postgres and Cassandra and other data stores that are actually used to power the site.
• 22:57
  SPEAKER_00: We use Airflow to actually transform and bring that data into our unified BigQuery store so that data science and ML and search teams don't have to rely on our production databases to get some of this information.
• 23:09
  SPEAKER_00: For a lot of companies, you'll probably see the same trend.
• 23:12
  SPEAKER_00: For Reddit, it's a really, really important function of how the company works, mostly because of the fact that Reddit's been around for 15 plus years.
• 23:20
  SPEAKER_00: We don't have the like the event collector and the ingestion proper that we have now didn't exist in 2010 or 2012 or whatnot.
• 23:28
  SPEAKER_00: So having access from a data science perspective to some of these state or static attributes brings in a whole other realm of possibilities for model creation, for user understanding, for opportunity sourcing, things like that.
• 23:40
  SPEAKER_00: So that's uh probably my second favorite use of Airflow.
• 23:42
  SPEAKER_00: Is we can use that as, again, a really lightweight and for lack of a better word, easy way to migrate that large database state data into a more accessible service, which is BigQuery, where you can then gain the efficiency and the power and the scalability that GCP services provide.
• 23:59
  SPEAKER_00: So it becomes a non-issue for data scientists to parse petabytes of information at a time to understand a new emerging trend or look up the all-time history of a user's interests or whatnot.
• 24:11
  SPEAKER_00: And related to the first outsourcing or shipping that data to our customer's own services, that can be paired with the client streaming data to make some really, really powerful tools to really make Reddit a fun place and a safe place for any user, regardless if you're going to get news or if you're checking sports cores or just looking at cats or whatever you want to do.
• 24:30
  SPEAKER_02: One thing you mentioned there that was really interesting was this notion of accessing your data from the surface of your choice.
• 24:35
  SPEAKER_02: If you think about like ETL and how it works into greater data workflows, right?
• 24:40
  SPEAKER_02: It's usually ETL, TL, TL, and three more keys and six more L's.
• 24:45
  SPEAKER_02: And then after all of that, it's something where it ends up getting used.
• 24:49
  SPEAKER_02: Sounds like what you're saying is Airflow lets users pick their surface a little more, such that when they go to read something for an actual use case, it's just a transform or it's just one or two layers of ETL.
• 25:01
  SPEAKER_02: It's not this long giant thing that has to read from a production database or something.
• 25:07
  SPEAKER_00: That's totally on point.
• 25:08
  SPEAKER_00: There's uh, as you might imagine, tons of benefits come from making teams or enabling teams to work in the ecosystem that they already do, or making that an easier way to work with.
• 25:19
  SPEAKER_00: The model or the path to production that we try to promote, or in some cases, even enforce that pairs with Airflow really nicely is, yeah, to your point, building that ETL, TL, TL, TL is most cases way easier than standing up a new API or getting access to a production service.
• 25:39
  SPEAKER_00: And giving teams the safety and ability to do that in Airflow, even if it's a temporary solution or a gamble, makes them able to pivot and work really, really quickly on a hunch or on you know one experiment result where they can say, these things look really interesting.
• 25:55
  SPEAKER_00: We want to dig in here, but we don't want to spend a whole quarter investing in infrastructure to make this possible.
• 26:00
  SPEAKER_00: Can we just do this in airflow and get access to a sample or a subset of this data to really prove its efficacy?
• 26:07
  SPEAKER_00: And we've seen that pattern a number of times where things will start in that flexible, easy airflow solution.
• 26:13
  SPEAKER_00: Teams really see the value in some cases almost immediately.
• 26:17
  SPEAKER_00: And then being uh smaller or maybe like uh understaffed in proportion to our user base company, it becomes then way easier for those organizations to go to their directors and their VPs and say, like, oh, we proved the value here.
• 26:32
  SPEAKER_00: Now I can 100% tell you that it's worth investing or it's worth making this a fully fledged production system.
• 26:39
  SPEAKER_00: From a data perspective, it makes the conversations more powerful and more accessible.
• 26:43
  SPEAKER_00: From a strict engineering perspective, it often opens doors that would be really, really heavy to move without strict finance or tech approval.
• 26:52
  SPEAKER_00: That's a side benefit that, from my perspective, I see a lot.
• 26:56
  SPEAKER_01: We've talked a lot about signals and how you're able to actually drive signal with Airflow.
• 27:01
  SPEAKER_01: I'm curious to know, both for myself and for folks who maybe are less familiar with the Reddit model, what type of things do you use those signals to drive aside from just further user engagement?
• 27:12
  SPEAKER_01: And what is that case you can make when you say this is driving X signal?
• 27:17
  SPEAKER_01: And that is why you should care about it.
• 27:19
  SPEAKER_01: Like, what does that conversation look like when it comes down to it?
• 27:22
  SPEAKER_00: From the Reddit perspective, to give you more context on the trade-offs we're making, the decisions we're we're leaning into, when you look at Reddit, there's several ways that we approach making the site what we want to make it, you know, fun, engaging, safe, all those things.
• 27:38
  SPEAKER_00: There's the strict users, people coming to Reddit for the first time, our hardcore subscribers and users that are on Reddit every day and making it great, the generating content, they're having conversations.
• 27:49
  SPEAKER_00: There's another audience of the moderators that are protecting users, they're creating new communities, they're enforcing rules and creating rules to make it possible to have subreddits dedicated only to baseball or only to cats surfing or whatever niche you want to make.
• 28:05
  SPEAKER_00: And then there's the background conversations around how we make our feeds relevant, how we make our feeds and communities safe, and how we really keep Reddit in the zone that is scalable.
• 28:20
  SPEAKER_00: So if we were to just remove a lot of the boundaries between how users engage in the site, you could hurt a lot of communities, especially those that are a little bit more sensitive.
• 28:29
  SPEAKER_00: Part of what makes Reddit Create, in my opinion, is that being somewhat anonymous, it's really possible to find support communities, communities that talk about more sensitive or difficult conversations.
• 28:39
  SPEAKER_00: And there are countless examples where the Reddit communities and Reddit users have helped other Reddit users get tough answers, get through difficult times, especially during this year.
• 28:50
  SPEAKER_00: 2020's had a ton of unique issues.
• 28:52
  SPEAKER_00: And that third bin sort of fits into that section of we want to make data that's useful for those safety questions, those tough questions, those difficult conversations accessible to teams to make sure that those conversations stay tough, stay difficult, stay relevant.
• 29:07
  SPEAKER_00: If we just dig into those three bins a little bit, the first one, just the user perspective.
• 29:12
  SPEAKER_00: That one, as you might imagine, is really similar to what other platforms are doing, where we want to understand user behavior, we want to surface the signals that are relevant so that we can recommend the right content, we can identify behavior that's threatening to make Reddit fun or to make Reddit safe for that user.
• 29:29
  SPEAKER_00: And also potentially to find them, you know, the community that they're looking for, the friends or the other Reddit users that they might have good conversations with.
• 29:35
  SPEAKER_00: And that's exactly what you'd see.
• 29:36
  SPEAKER_00: The output of that is when you go to the popular feed and you just go to the Reddit homepage, you're seeing the output of our learnings there.
• 29:42
  SPEAKER_00: The experiments that are data driven, where we're looking to find the signals that make popular great for everyone, and in some cases, the average Redditor.
• 29:51
  SPEAKER_00: The second bin is, you know, for our moderators and the people that are really powering the site in many ways.
• 29:58
  SPEAKER_00: We always make an effort to invest in tooling and in services that make it easy and possible to moderate Reddit in ways that are more scalable, more safe, and more engaging.
• 30:08
  SPEAKER_00: If we didn't have moderation or if we didn't have moderator tools, you know, Reddit wouldn't be what it is today.
• 30:12
  SPEAKER_00: So, in the same way that we want to tune our feeds and our features and the apps to a new Redditor just as much as a longtime core user, we want to do the same thing for our moderator populations and make sure that they have the tools necessary to keep the communities active and safe.
• 30:28
  SPEAKER_00: We want to use those data to drive signals and informed decisions to identify blind spots of like what's causing our moderators pain or friction.
• 30:36
  SPEAKER_00: And we also want to give them the proper aggregates or signals that make it possible for them to do this analysis and understanding on their own.
• 30:44
  SPEAKER_00: One signal that we use Airflow to generate is traffic stats and moderation aggregate stats that we ship out to communities and moderators so they can understand how is my community growing?
• 30:54
  SPEAKER_00: How is my user base changing or evolving over time?
• 30:57
  SPEAKER_00: And those are things that we weren't doing several years ago.
• 31:00
  SPEAKER_00: And they've been extremely helpful to our moderators because they just give more visibility into what's happening on the Reddit, specifically to their loved communities and the communities they support, which makes it possible for them to not only moderate better, but enjoy it more.
• 31:16
  SPEAKER_00: The last bin of the difficult conversation and the safety topics, that one is probably the most important from the data and the sanctity of data perspective.
• 31:26
  SPEAKER_00: It's also the one that we invest mostly in what we talked about earlier, in getting that prototype to production process going, where it's we want to be ahead of any threat that we see on the Reddit, any bad behavior or any risky behavior.
• 31:40
  SPEAKER_00: And in many cases, we use airflow and the signals that we generate from a data science perspective and a products perspective to help teams understand what do we need to focus on next to avoid bad behavior from thriving on Reddit, or what do we need to understand in terms of signal generation and model generation or content recommendation to make sure we avoid those potentially risky behaviors from ever occurring?
• 32:02
  SPEAKER_02: The way all those pieces play together, everything from how you ingest that through kind of how that gets computed and ultimately shipped out to users.
• 32:10
  SPEAKER_02: I feel like that's a never-ending conversation around just making sure that uh each piece of that is optimized because I'm sure you could change your full-time job to optimizing any specific piece of that a few times over.
• 32:20
  SPEAKER_02: Shifting gears a little bit, I'd love to kind of talk a little bit more on the infra side.
• 32:24
  SPEAKER_02: Like how are you all running Airflow in a way that you are supporting all these use cases and all these business critical and community critical processes that Airflow is kind of the general orchestrator around?
• 32:35
  SPEAKER_00: We have a number of teams running Airflow right now.
• 32:38
  SPEAKER_00: So it started as a one person doing one instance.
• 32:41
  SPEAKER_00: As you might expect, it's snowballed into consumer teams are using it.
• 32:45
  SPEAKER_00: All of Data Platform is owning a number of different services.
• 32:48
  SPEAKER_00: Our ads engineering team has the Airflow instances to run some of their tools.
• 32:51
  SPEAKER_00: As you might expect, there's a little bit of deviation in terms of how teams are using it.
• 32:55
  SPEAKER_00: From the data platform perspective, which is the teams that I work here support, there's a pretty standard way of working.
• 33:01
  SPEAKER_00: And in the beginning, we had manually curated instances, and people would have to SSH into their AWS box and stand things up and check on the schedule or whatnot.
• 33:11
  SPEAKER_00: Nowadays we have a significantly better setup where we essentially have the instances provisioned in Terraform.
• 33:18
  SPEAKER_00: So anybody on DIT Engineering can request or can spin up a new cloud box to run an Airflow instance on.
• 33:24
  SPEAKER_00: And that way we can provision whatever memory or security protocol we need to make sure that we can create them almost instantly, whereas if we need to clone a box, if there's an issue with one and we need to spin up a new instance to replicate what's currently on that, we can do that with just some Terraform changes, which is awesome.
• 33:39
  SPEAKER_00: On top of that, which gets us the VMs provision, uh, we use uh Vault and Puppet to control safety permissions and access where and the Puppet controls the packages and libraries that are actually served on the box.
• 33:51
  SPEAKER_00: So uh depending on the instance, some of the more data engineering or infra-e-boxes might have only a few packages, where they're doing some airflow task sensors, they're doing some Python checks, and then they're doing a whole bunch of big query scans or sort of big query operators, and that's about it.
• 34:08
  SPEAKER_00: Looking at the data science side of the thing, they might have some bash operators, a lot more Python, some NumPy or Jupyter doing some processing in RAM or whatnot.
• 34:16
  SPEAKER_00: Those often take a little bit more setup, which we do in Puppet.
• 34:20
  SPEAKER_00: Along from there, we still live in a world where the staging versus the prod is a little bit of a burden, a little bit of a friction point for a couple of reasons.
• 34:29
  SPEAKER_00: You know, we can provision, we can deploy boxes very quickly, we can get them set up and get Airflow running.
• 34:35
  SPEAKER_00: That includes actually updating Airflow, you know, same process there, which is awesome.
• 34:39
  SPEAKER_00: And the part that I think will be the next chapter of our Airflow development will be making it so that teams can develop and deploy new DAGs, new tasks, custom operators or whatnot, without the need to point to our production BigQuery environment.
• 34:55
  SPEAKER_00: One difficult thing we struggle with is the way that airflow typically goes.
• 34:59
  SPEAKER_00: And we sort of alluded to this before with the ETL, TL, TLTLs, is you start with your raw events and some output, and almost overnight it seems like that transforms into you have tables that are dependent on tables that are shipping out to another warehouse that are pulling in from another team's database.
• 35:14
  SPEAKER_00: And that's part of the power of airflow, but it's also creates a little bit of a tricky situation in terms of a staging conversation.
• 35:21
  SPEAKER_00: After that, the setup of the instance, we basically work in a place where we have a lot of drone checks, we have a ton of pre-commit hooks and safety measures to help teams avoid issues or avoid them from shipping bad code.
• 35:34
  SPEAKER_00: But we don't truly have a staging environment where there's uh an instance of Airflow reading from sampled data or nulled out tables where you can confirm your DAG construction before deploying.
• 35:44
  SPEAKER_00: The phase right now with one of our teams is that they have a staging environment set up through Kubernetes.
• 35:49
  SPEAKER_00: What that means is when you merge anything in Airflow, it'll go through all those drone and CL checks that I mentioned.
• 35:55
  SPEAKER_00: It'll actually deploy to that Kubernetes staging environment that you can access locally.
• 36:00
  SPEAKER_00: And that'll be one way to make sure your DAG is structured right, to make sure everything compiles, to make sure all your library dependencies are installed, which covers a ton of bases.
• 36:08
  SPEAKER_00: Still doesn't quite cover that use case of I've got a new DAG with 10 tiers of aggregation, and I have a customer in every tier that's going to siphon off data from mildly aggregated data to half-baked data to full and aggregated daily reporting to a single row of value.
• 36:28
  SPEAKER_00: Like there's real use cases in that realm.
• 36:30
  SPEAKER_00: We're still working on ways to deploy like a true staging solution.
• 36:35
  SPEAKER_00: But right now, that setup I described covers a lot of the most likely to break issues.
• 36:39
  SPEAKER_02: It's easy to replicate infrastructure.
• 36:41
  SPEAKER_02: It's not so easy to replicate data, right?
• 36:43
  SPEAKER_00: Yeah, it's it's a super interesting and but but also painful conversation because it's the sad truth is that at Reddit scale, cost becomes a part of that conversation where it's even just replicating a sample of our data and aggregate data is something that you have to consider with, you know, how long are we going to keep this around?
• 37:00
  SPEAKER_00: Is it worth that in terms of monetary value to make staging a little bit easier or just closing the hole on eight out of 10 of those issues around deploy steps and CI checks?
• 37:09
  SPEAKER_00: Is that all we need for now?
• 37:11
  SPEAKER_01: We've talked to plenty of folks who are actively thinking through what that true end-to-end staging process should look like.
• 37:18
  SPEAKER_01: And a couple of other companies that are working on that problem, but it's a very hard problem to solve, right?
• 37:23
  SPEAKER_01: Because as Raj mentioned, it's not easy to replicate data.
• 37:26
  SPEAKER_01: Even though it's easy to replicate infrastructure, it's not cheap.
• 37:28
  SPEAKER_01: It's going to be very expensive to run entire end-to-end staging pipelines that spin up massive clusters to do some processing, right?
• 37:35
  SPEAKER_01: Like you can never really have that properly mirror production.
• 37:38
  SPEAKER_01: Or maybe you can, but it'll cost you your entire budget.
• 37:42
  SPEAKER_00: Yeah.
• 37:42
  SPEAKER_00: To bring the conversation all the way back, it's counting.
• 37:45
  SPEAKER_00: As soon as you start to sample and duplicate data simply for a staging purpose, you open the door for that QA conversation.
• 37:52
  SPEAKER_00: And like, now we have to have somebody on staff to make sure that everything in staging is always accurate enough to inform a production decision or to inform our go-no-go decisions.
• 38:02
  SPEAKER_00: So you're totally right.
• 38:03
  SPEAKER_00: There's cost implications, there's maintenance implications, and then there's data quality implications.
• 38:08
  SPEAKER_00: It's a huge, huge problem to solve, but it does branch off into a number of pretty interesting conversations or projects.
• 38:15
  SPEAKER_01: Absolutely.
• 38:15
  SPEAKER_01: I feel like we can do an entire episode just about that.
• 38:18
  SPEAKER_01: Drilling into the infrastructure layer a little bit further, how do folks go about getting DAGs deployed to those Airflow environments?
• 38:23
  SPEAKER_01: We talked a little bit about what it takes to provision an Airflow environment, right?
• 38:27
  SPEAKER_01: For a data engineer, data office engineer, whoever needs one running for a team.
• 38:31
  SPEAKER_01: How does an end user actually go about getting uh DAG that they've written into production?
• 38:37
  SPEAKER_00: So there's a couple of different processes for different teams.
• 38:40
  SPEAKER_00: The teams that I support in the data platform world, typically the deploy process is pretty simple where we have every piece of code that's running on the boxes in Git so we can have easy conversations or communication between our data scientists and data engineers working locally.
• 38:57
  SPEAKER_00: Once it's on a master branch, we then have a company-wide deploy service that helps sync the our GitHub enterprise service to whatever VM instances are set up and running.
• 39:08
  SPEAKER_00: That process is basically just uh, you know, you've got a CI commands where you can tell our deploy service to look for this commit hash, send it to this box, and it'll take care of the rest for you in terms of getting it there.
• 39:19
  SPEAKER_00: Once that assumes that you know everything has been approved and the drone and CI chex have have gone through.
• 39:25
  SPEAKER_00: But in terms of that, it's pretty quick where you can sync the GitHub to your production box.
• 39:31
  SPEAKER_00: And after that, it's there immediately.
• 39:32
  SPEAKER_00: So new DAGs, new variables, or new args or whatnot will appear immediately.
• 39:37
  SPEAKER_00: We've removed almost all the use cases where data scientists or engineers are accessing the actual airflow box, you know, via SSH or whatnot.
• 39:46
  SPEAKER_00: And we now live in a in a space where almost all the work from deploy to running and rescheduling or rerunning Airflow tasks is just with GitHub and the Airflow UI.
• 39:56
  SPEAKER_02: You know, as you're kind of doing all this out and it sounds like you're managing these multiple Airflow environments, are you leveraging any sort of tools in the back end to actually manage everything using something like Stats D or Datadog around the monitoring piece of it?
• 40:09
  SPEAKER_02: What does that piece of the pie look like?
• 40:10
  SPEAKER_00: Right now, we don't monitor too much across the Airflow instances outside of the Terraform setup stages where we can make the deployments consistent and to make the instances look identical if we need them to, which is super helpful rather than manually repeating steps or whatnot.
• 40:25
  SPEAKER_00: For all of the stats and monitoring and checking on how airflow is running, the pattern that we've fallen into with most things up until now is to actually check on the exhaustive airflow.
• 40:36
  SPEAKER_00: So we point back to the data warehouse and use anomaly detection services or alerting services to make sure that the results are as expected rather than checking that on the box as they occur.
• 40:48
  SPEAKER_00: So it's a slightly different approach that a little bit more flexible in that we can, just like everything else at Reddit, we can surface those anomaly and alerting tools to any teams that need or ask for access there without tying into the Airflow conversation.
• 41:02
  SPEAKER_00: But it does mean that we have to be a little cautious in the Airflow development cycle to make sure that if it's a new engineer or a new team or a new organization, they get the full picture of what happens to data from beginning to end, because Airflow represents just a piece of the chain as opposed to a full spectrum solution.
• 41:22
  SPEAKER_01: That segues nicely into our final question of the day.
• 41:25
  SPEAKER_01: What you'd like to see from Airflow that would help Reddit's use case and implementation.
• 41:30
  SPEAKER_00: The thing that comes up most frequently is a side effect of the diversity of roles that are using Airflow.
• 41:37
  SPEAKER_00: So we've got PMs, we've got engineers, we've got data scientists, we've got managers, we've got a whole bunch of people accessing the Airflow UI for a number of things, checking on tasks, seeing what was deployed and what's running, backfilling, uh looking at the logs to understand where we have issues or where we might have issues in the short term.
• 41:55
  SPEAKER_00: Out of that, a lot of the dead ends or in that context, feature requests are really for just more robust UI capabilities, where you know the CLI and Airflow has a bunch of options for changing your scheduler logic, backfilling, setting priorities, doing all sorts of things that make it better for data science to use that without having to go into the DAG code and change and deploy things.
• 42:19
  SPEAKER_00: Getting more power in the UI to tune those functions of Airflow, like in what order things run, what are the rules for provisioning slots and whatnot, those are the things that teams ask for most often.
• 42:31
  SPEAKER_00: Because as you'd expect, getting the data or getting the DAG set up is really the first step.
• 42:36
  SPEAKER_00: And building on that iteration conversation we touched on earlier, it's teams that want to move fast and want to iterate quickly or frequently, going back to the deploy process and doing that in code is sometimes a little too slow or a little too painful.
• 42:49
  SPEAKER_00: In my mind, being able to tune the DAG bars and DAG priorities scheduled behavior more in the UI.
• 42:54
  SPEAKER_00: So teams have easier access to that would be amazing.
• 42:58
  SPEAKER_00: That'd be my top ask.
• 42:59
  SPEAKER_00: The second piece that is frequently comes up, especially with data science teams, making Airflow more friendly or scalable for distributed processing or tasks.
• 43:10
  SPEAKER_00: We have a lot of Spark, we've got a lot of Python, and as I mentioned, NumPy or Pandos and whatnot.
• 43:16
  SPEAKER_00: And just like with some of the engineering solutions we described, the ability to get access to data and use it for an experiment or use it for a new feature to see what sticks and what doesn't is awesome.
• 43:27
  SPEAKER_00: From a data science perspective, building new models, running new cohorts or some building new segments is one thing that Airflow can help do very quickly.
• 43:35
  SPEAKER_00: So you can transform that data and run a bunch of new model generators all at once.
• 43:39
  SPEAKER_00: The problem is that right now, most of the boxes we use are going to run all that in memory of the Airflow box.
• 43:45
  SPEAKER_00: So if you're trying to do process 10 terabytes of user data to make some K-means clusters or whatnot, you either got to go through a whole bunch of careful aggregation and cleanup or go through a whole bunch to make Airflow work in a distributed way, to not try to bring out all that into like one data frame in RAM or something.
• 44:03
  SPEAKER_02: There's not enough memory in the world for pandas, right?
• 44:05
  SPEAKER_00: So you know it's it's true when you're doing a notebook locally.
• 44:08
  SPEAKER_00: It's true when you're running Airflow code and it's a constant request that we get from data science pods.
• 44:12
  SPEAKER_00: Can we just be lazy and like throw a whole bunch of RAM at this problem?
• 44:15
  SPEAKER_00: And sometimes the answer is yes, but in the Airflow conversation for scaling, like most of the time you don't want it to be yes.
• 44:21
  SPEAKER_02: It's funny because it's just one layer removed from the data that the data user doesn't want to think about.
• 44:26
  SPEAKER_02: They just want, like, hey, model run.
• 44:28
  SPEAKER_02: They don't want to say model run with this or that or X or Y or have to track that.
• 44:33
  SPEAKER_02: It's like a whole other axis of problems that's like the data science headache and the infrastructure headache.
• 44:38
  SPEAKER_02: So no one wants to think about that at all.
• 44:40
  SPEAKER_02: This has been a great conversation.
• 44:42
  SPEAKER_02: I loved hearing about kind of the whole data lifecycle that y'all are doing at Reddit, especially how RFL plays into that.
• 44:47
  SPEAKER_02: Is there any kind of last shout-outs that you want to give in terms of if you're hiring or anything along those lines?
• 44:52
  SPEAKER_00: We are hiring.
• 44:53
  SPEAKER_00: Everything I spelled out today with the data pipeline and the ecosystem we're trying to build.
• 44:58
  SPEAKER_00: We've seen all of this develop really organically.
• 45:01
  SPEAKER_00: And it's been awesome to see certain individuals just stand up and drive these open source solutions and new tech for a company.
• 45:08
  SPEAKER_00: We're now in a phase where, from a data perspective, we're really investing in the future of data science, machine learning, data infra, and engineering to make all of the stuff that we touched on today, from role event collection to the distribution of that data and access to that data to the discovery and documentation core powerful attributes of how Reddit operates.
• 45:31
  SPEAKER_00: Everyone I just mentioned is going to be growing a whole bunch next year.
• 45:34
  SPEAKER_01: Ben, this has been such a pleasure.
• 45:36
  SPEAKER_01: Thank you so much for the time and for coming on.
• 45:38
  SPEAKER_01: This is one of the most interesting conversations that we've had in quite some time, especially since like everyone uses Reddit.
• 45:43
  SPEAKER_01: Like I don't know a single person that doesn't use it.
• 45:46
  SPEAKER_01: So getting a peek under the hood is just so interesting.