Latest / The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI / GDPR, Self-Service Data, and Infrastructure Automation with Typeform

Transcript

GDPR, Self-Service Data, and Infrastructure Automation with Typeform · The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

• 0:00
  SPEAKER_01: That way your team can act as a platform team where you're not actually manage the pipelines, you're just giving them the infrastructure to run their pipelines.
• 0:15
  SPEAKER_00: Hi there, and welcome to this week's episode of the Airflow podcast, brought to you by Astronomer.
• 0:21
  SPEAKER_00: This week we met up with Albert Franzi and Carlos Escuta, who work on the data platform team at Typeform.
• 0:27
  SPEAKER_00: Typeform is a tool that allows you to build beautiful interactive forms that you can use for a whole bunch of things, including customer surveys, employee engagement, product feedback, and market research, to name a few.
• 0:39
  SPEAKER_00: As you can imagine, Typeform generates a ton of data from their web clients that then need to be processed and stored in a wide variety of back-end services.
• 0:47
  SPEAKER_00: In our conversation, we talked to Carlos and Albert about how to transition from a data operations to a data platform team by empowering data teams to self-service Airflow.
• 0:58
  SPEAKER_00: Before we get started, I wanted to mention that our team at Astronomer is growing rapidly and we're on the hunt for new folks to join in a variety of different roles.
• 1:05
  SPEAKER_00: If you're passionate about Airflow and interested in building the future of data engineering, please get in touch.
• 1:11
  SPEAKER_00: You can check our current job postings at careers.astronomer.io.
• 1:15
  SPEAKER_00: But we're constantly updating our listings to accommodate our quickly growing company, so if you don't see a perfect job posting but are passionate about what we're doing, please feel free to email me directly at pet at astronomer.io to have a chat.
• 1:28
  SPEAKER_00: Without further ado, here are Albert and Carlos.
• 1:34
  SPEAKER_00: Albert and Carlos, welcome to the Airflow podcast.
• 1:37
  SPEAKER_00: Do you guys mind kicking us off by just telling us a little bit about yourselves and what you do at Typeform?
• 1:41
  SPEAKER_00: Albert, maybe you can start and Carlos you can follow.
• 1:44
  SPEAKER_02: Sure.
• 1:44
  SPEAKER_02: Thank you for having us here.
• 1:46
  SPEAKER_02: It's a pleasure for us.
• 1:47
  SPEAKER_02: I'm Albert Franzi.
• 1:48
  SPEAKER_02: I'm the data engineer lead at the data platform uh team at Typeform.
• 1:53
  SPEAKER_02: We are in charge of all the data in the company, all the ingestion, uh processing, storage, and also delivering data to everyone.
• 2:04
  SPEAKER_02: So we are the core, no we could say.
• 2:09
  SPEAKER_03: And Carlos.
• 2:10
  SPEAKER_03: And from my side, Carlos Doscura, DevOps on the data platform team, and I'm responsible of the infrastructure and all the operational part of the what Albert said all the ingestion, processing, and delivery of the data.
• 2:26
  SPEAKER_00: So in the greater context of type form, what type of projects do you guys work on?
• 2:31
  SPEAKER_00: What type of data are you working with?
• 2:32
  SPEAKER_00: And what general context does that serve?
• 2:37
  SPEAKER_02: Regarding the what kind of data now we are working for, we have, as you know, typeform is like trying to create the main goal is to create uh meaningful conversations through forms and surveys.
• 2:47
  SPEAKER_02: So the data we have is from forms creation, uh responses, a lot of responses, and because at the end the company is a global company, so we have users around the globe.
• 2:58
  SPEAKER_02: We also have a lot of data from tracking, no, um not just the user that is creating, but also how people interact with the type forms so we can make a better product and take data-driven decisions.
• 3:14
  SPEAKER_02: And also we are trying to bring machine learning to another level, no, so uh we can, as I say, no, build uh better products and be um one step forward than the competition.
• 3:28
  SPEAKER_02: We have a lot of models right now, like where we want to build and deliver.
• 3:34
  SPEAKER_02: Uh quick response.
• 3:35
  SPEAKER_00: Yeah, no, that definitely makes sense.
• 3:36
  SPEAKER_00: And we're big fans of typeform over here at Astronomer.
• 3:39
  SPEAKER_00: We use uh Yel surveys for quite a few things.
• 3:42
  SPEAKER_00: I know Viraj has built quite a few out.
• 3:44
  SPEAKER_00: Can you talk us through what the data generation process looks like from an end-to-end lifecycle perspective?
• 3:51
  SPEAKER_00: Like, if I'm a typeform user, I build a survey, I send it to my customer to gauge MPS score, for example, right?
• 3:58
  SPEAKER_00: Like, what happens when the customer submits that survey?
• 4:01
  SPEAKER_02: Okay, from the customer survey, so meeting a survey, we have a like different kind of events.
• 4:08
  SPEAKER_02: No, we could have like the back-end events and the front end is more how the user interacts with the each block.
• 4:15
  SPEAKER_02: The typical thing, no, uh time invest in each block, no, in each question.
• 4:21
  SPEAKER_02: Also, if the user is doing out of scrolls or the behavior of him regarding the back end, that at the end is what is going also to be more interesting for you as a user, is all the content.
• 4:35
  SPEAKER_02: No, all the content then uh we receive some events in real time.
• 4:40
  SPEAKER_02: So we have uh a data collector that is um managed by a Kafka cluster.
• 4:46
  SPEAKER_02: So we are connected there.
• 4:48
  SPEAKER_02: Also, we have different pipelines with lambdas storing data to S3 for more analysis, processing, and uh the end for you, the data comes to you after different ways.
• 5:02
  SPEAKER_02: No, the real time is like it goes directly to your uh user panel, then more analysis, more analytics dashboards as a user or for the product of for product managers at typeform, it comes in a batch way.
• 5:18
  SPEAKER_02: That's where airflow comes into the scene.
• 5:22
  SPEAKER_00: Yeah, that's a great segue.
• 5:24
  SPEAKER_00: I'd love to hear about when and why Airflow was brought into the picture at Typeform.
• 5:27
  SPEAKER_00: I know you all have been users for quite some time, but would like to rewind the clocks a little bit and talk about when uh you first adopted it and what the driving uh incentivizers for that were.
• 5:37
  SPEAKER_02: Yeah, I think that here maybe uh there is uh the need for some context where we before before I joined Typeform one year ago, no, everything was not well, not everything, almost everything, 90%, was developed using Pentajo.
• 5:55
  SPEAKER_02: No, it was like uh not the best tool to use, to be fair.
• 5:59
  SPEAKER_02: Everything is a black box, uh there is no review, there is no way to share what is going inside this.
• 6:07
  SPEAKER_02: And as an engineer, it's more like uh complicated to add new models and operators.
• 6:13
  SPEAKER_02: No, as Airflow, being an open source community, it allows you to code yourself, make contributions, and also it's more open-minded at the end.
• 6:24
  SPEAKER_02: So I was lucky that uh the two guys before I joined it, they were already triggering the initiative of moving to Airflow.
• 6:33
  SPEAKER_02: There were some new tags, but it was just two or three.
• 6:37
  SPEAKER_02: So we start uh migrating from there.
• 6:40
  SPEAKER_02: So that's where we come from, like moving away from uh legacy or old school ATL tools.
• 6:49
  SPEAKER_00: What were some immediate benefits you got from making that change?
• 6:54
  SPEAKER_00: From one side, is that you get away from Windows.
• 6:57
  SPEAKER_03: That's and that also means that you're gaining a lot of reliability on the system because you end up having just one single machine in Windows that runs Pentahu.
• 7:09
  SPEAKER_03: And if that machine fails, everything fails, and moving to Airflow already means that you can have high availability and number of workers, or not, depending on how you implement it or not, but you have that option.
• 7:21
  SPEAKER_02: Yeah, and also the uh it was really easy to to deploy between uh the development environment and the production.
• 7:29
  SPEAKER_02: The code was available for everyone in GitHub.
• 7:33
  SPEAKER_02: No, one we used to have is that all jobs were full of queries, the SQL queries and some uh API interactions, everything was there, and uh all the data team, all the data scientists, even that they are the users of the data, they also care about uh the flow or the logic that is behind uh these final tables.
• 7:55
  SPEAKER_02: No, they want to know and understand the process of filling these tables.
• 7:58
  SPEAKER_02: So having everything in Pentajo was not possible because then everyone had to access uh the Pentajo machine to review this, no, or to access this SQL code.
• 8:09
  SPEAKER_02: So that wasn't decided.
• 8:10
  SPEAKER_02: No, do you at the end Airflow moving to Airflow was like enabling transparency in the team across the company about how the data was being processed.
• 8:22
  SPEAKER_00: Totally.
• 8:23
  SPEAKER_00: That definitely makes sense.
• 8:24
  SPEAKER_00: And I imagine the migration wasn't particularly easy, or was it how long did that process take?
• 8:32
  SPEAKER_02: To be fair, we still have one process there that we still need to migrate, but it has a lot of dark magic.
• 8:38
  SPEAKER_02: But everything else was really easy because we developed like something that uh it's quite close to the uh DAC factory, no, an open consource library.
• 8:50
  SPEAKER_02: So we developed uh we call it Chester, it's uh like don't start video game.
• 8:56
  SPEAKER_02: But given that it was really easy to define some all the DACs with uh YAML as a DAC factory allows, but the good part was that we were able to like replicate some functionalities that we had in Pentajo with operators, and given that we were able to also automatize and temp and make them as templates, it was really easy to reuse for different use cases.
• 9:20
  SPEAKER_02: So I believe that the most difficult part it was more to navigate Pentajo and recreating these in uh in Netflix.
• 9:32
  SPEAKER_00: Sure.
• 9:32
  SPEAKER_00: Oh, that's really interesting.
• 9:34
  SPEAKER_00: I had heard about Chester, the DAG Factory style tool, and we've actually interviewed Adam Boscarino, the creator of DAG Factory, before.
• 9:42
  SPEAKER_00: I think that interview is living somewhere on my laptop and needs to be published.
• 9:47
  SPEAKER_00: It's just waiting.
• 9:48
  SPEAKER_00: So maybe we'll find a way to tie it into this episode or something.
• 9:51
  SPEAKER_00: But I didn't realize, did you use that as a migration tool primarily?
• 9:54
  SPEAKER_00: Or like how for what purpose was that initially built?
• 9:58
  SPEAKER_02: I believe that one of the trigger reasons of building Chester, it was moving away from Pentajo.
• 10:04
  SPEAKER_02: There was some use cases because our team, no, and right now it's called Data Platform, no, but we come from a team that was called Data Operations.
• 10:11
  SPEAKER_02: It was more like a slave uh mindset.
• 10:15
  SPEAKER_02: And one thing that we wanted it was to be more enabler, no, not just uh someone came to you and says, Hey, I have this script of 1,000 lines, put it into production in an orchestrate it.
• 10:26
  SPEAKER_02: So we wanted to move away from this to having more like some tool or some way that any data scientist could make their own pipelines, and that was one of the triggers of Chester, but it was in the same time when also the pentajo migration was uh breeding.
• 10:42
  SPEAKER_02: So both cases were were the triggers of this one.
• 10:47
  SPEAKER_01: Jumping in a little bit here.
• 10:48
  SPEAKER_01: What I really heard from that is from some perspective, you wanted to make it so that your team went from less of a bottleneck and more to a team that empowered other teams to do what they needed to with their data.
• 10:59
  SPEAKER_01: Did that accelerate data science, data analytics for other teams within the organization?
• 11:03
  SPEAKER_01: Like what was the impact that had on other teams that were previously your customers, but now they're almost your partners?
• 11:11
  SPEAKER_01: Yes.
• 11:11
  SPEAKER_02: So yeah, it uh accelerated completely.
• 11:14
  SPEAKER_02: No, it changed the play, no, it changed the game here at the Typeform.
• 11:19
  SPEAKER_02: The the first thing that we start like seeing, no, like uh changes, it was like the ownership, the feeling of ownership of the pipelines.
• 11:28
  SPEAKER_02: No, before it was here is my SQL, you are in charge of putting it to production, and you are in charge of this thing working.
• 11:36
  SPEAKER_02: Once it were the once the data scientists start making their own pipelines using Chester, and the same attack factory, we start seeing some changes on their way to think, no, to approach.
• 11:50
  SPEAKER_02: They were in charge of their own pipelines, and if there were some issues, they were the ones uh fixing it.
• 11:57
  SPEAKER_01: No, and then that way your team can act as a platform team where you're not actually manage the pipelines, you're just giving them the infrastructure to run their pipelines.
• 12:05
  SPEAKER_02: That's when you move from a data operations to a data platform.
• 12:09
  SPEAKER_02: No, you are an enabler, you are not a bottleneck.
• 12:11
  SPEAKER_03: Yeah, you're just empowering them to build their own stuff and maintain their own stuff.
• 12:16
  SPEAKER_03: And we are just only in partially charge of the infrastructure that is running and make sure that everything's operational and they can build and run their own code.
• 12:28
  SPEAKER_03: Yes.
• 12:28
  SPEAKER_02: And also, and also the good part is that as you say, no, the collaboration is more now, it's more from from scientists when they need some custom operator.
• 12:38
  SPEAKER_02: Then we jump into action.
• 12:40
  SPEAKER_02: No, we provide the custom operator, we analyze it, and we are happy to build the operators since it's quite easy at the end to that.
• 12:47
  SPEAKER_01: That was my next question, where it seems like you've provided this great abstraction layer for your users to use, but you don't want to get too abstract, you don't want to be limiting what they can do.
• 12:56
  SPEAKER_01: There's always this back and forth we see of like, hey, here's this thing you can just fill in and do.
• 13:00
  SPEAKER_01: But then maybe data science is like, oh, I can't use my machine learning library here or something.
• 13:05
  SPEAKER_01: So, how do you walk this line of making abstraction layers but also giving your users flexibility to get their job done?
• 13:12
  SPEAKER_02: Yeah, I think that this is, I don't know when you're going to publish no the DAC factory uh podcast, no, but it's really like uh following the same, no, it's like if you have the right operators in place, no, and uh like connected to the YAML, no, to the Chester, to the DAC uh factory tool, no, in that case or West Chester, then if you also provide the proper documentation and uh the onboarding to the tool, at the end it's they can use anything.
• 13:40
  SPEAKER_02: No, one of the greatest uh things that we built, it was a custom layer on top of the Docker, well, the Kubernetes spot operator, no, we were moving also from an EC2 to a Kubernetes approach.
• 13:53
  SPEAKER_02: No, so right now uh Airflow is running in a Kubernetes.
• 13:58
  SPEAKER_02: So by moving to that, because in the beginning it was H2, no, it was when Carlos joined us that we were able to move to the next level and have everything in Kubernetes.
• 14:08
  SPEAKER_02: When we started having this uh Kubernetes pot operator extra layer that was handling some magic, no, at the end it's more secrets, some different things, then data scientists were able to just build their own machine learning dockers or their own uh models, and that was their they just need don't need anything.
• 14:28
  SPEAKER_02: No, we just need to configure some pot sizes for the resources, but after that it was really easy.
• 14:35
  SPEAKER_03: Yeah, I was about to say that.
• 14:36
  SPEAKER_03: I mean, uh now the the only fight that we have between two teams is trying to leverage them to do whatever they want to do, but also having in mind that from time to time they need our help in terms of performance, resource consumption, and things like that.
• 14:51
  SPEAKER_03: Because as you all know, machine learning is memory and CPU ingester, it's like a black hole, and you always need to take care of what they're doing, but they are able to do whatever they want to do with their own Docker images.
• 15:06
  SPEAKER_00: Sure.
• 15:06
  SPEAKER_00: So that's actually a good segue.
• 15:08
  SPEAKER_00: I'm curious to know how you're currently operating that airflow infrastructure.
• 15:11
  SPEAKER_00: What does the overall setup look like?
• 15:14
  SPEAKER_03: Okay, the overall setup is pretty simple and pretty straightforward.
• 15:19
  SPEAKER_03: We have our master source of true, which in fact is Terraform, which deploys our clusters.
• 15:25
  SPEAKER_03: And on top of that, we also are using the Helm operator from Terraform, which allows us to use that installs Airflow.
• 15:34
  SPEAKER_03: That means that in an eventual case of a software recovery or a migration need or whatever we need to do, we can replicate our whole uh infrastructure, and and that means the infrastructure per se, and also the software layer that is running on top of that infrastructure.
• 15:52
  SPEAKER_03: So it's a combination of on top of that airflow running.
• 15:56
  SPEAKER_02: Atlantic to deploy it.
• 15:57
  SPEAKER_03: And Atlantic to deploy it in case that it's needed also to automatize the whole lifecycle.
• 16:02
  SPEAKER_00: Sure.
• 16:03
  SPEAKER_00: So what when does that Terraform module get run?
• 16:06
  SPEAKER_00: Is it set up as part of some complex CI pipeline that it's integrated with your source control?
• 16:12
  SPEAKER_00: What does that end-to-end lifecycle look like?
• 16:15
  SPEAKER_03: Yeah, the end-to-end is not that hard to understand because we're not deploying Airflow that often, and we are not deploying new clusters that often.
• 16:23
  SPEAKER_03: We have like one single instance of Airflow running on one namespace, and our Terraform module, it's really simple that only needs uh the namespace that you want to run that Airflow instance in.
• 16:35
  SPEAKER_03: A couple of things like the repository which contains the DAX and the brands that you want to clone from that repository, and a couple of things that are really simple, like OAuth secrets for OAuth authentication for the front end and things like that.
• 16:50
  SPEAKER_03: But they are minimal configuration, also PostgreSQL, database, host, and all this stuff.
• 16:55
  SPEAKER_03: But it's really simple the setup.
• 16:57
  SPEAKER_03: So in case that eventually something really weird happens and we need to redeploy the entire infrastructure or the entire airflow, rerunning everything is just a command, it's a telephone apply or creating a PR and letting Atlantis to deploy everything.
• 17:12
  SPEAKER_02: I believe that at the end, the most time or the when we were like changing, no, doing more deploys in the in with this flow, it was or it's more when we update the flow version.
• 17:26
  SPEAKER_02: No, and that's also a good part because we can deploy faster, no, we can like get advantages of the new Airflow updates faster, no, because then we apply it in a really easy to development, we test it, and once is everything working as expected, we just can um replicate it in production.
• 17:44
  SPEAKER_03: Like yeah, nothing.
• 17:46
  SPEAKER_03: And and by fast, we mean that advertisole, it's like subscribe to the events of new releases on Airflow, and wherever they change, next day we are already deploying that on development and testing it.
• 17:58
  SPEAKER_03: And if all tests are successful, we are promoting that new release to production in a matter of days.
• 18:04
  SPEAKER_03: So we are really like early adopters for new versions of releases of Airflow.
• 18:10
  SPEAKER_02: Just waiting for Airflow 2 right now.
• 18:14
  SPEAKER_00: What does that testing process look like when you deploy one of the new releases?
• 18:17
  SPEAKER_00: How do you validate that it all looks good and that things will run smoothly when you get it running in production?
• 18:23
  SPEAKER_03: Let's be honest, the best way to test it, because we don't think yeah, it's on Friday.
• 18:28
  SPEAKER_03: No, let's be honest.
• 18:29
  SPEAKER_03: The best way to test it is uh to update it and run all the DAX that can be run on a dev environment because depending on the law of each country, here we are under really strict control of GDPR, so we cannot migrate data from production to development and test airflow on that environment.
• 18:46
  SPEAKER_03: So that means that we test whatever we can test, and if all DAX run successfully and everything looks okay, we try to release that into production.
• 18:57
  SPEAKER_03: And if something gets wrong, we can roll back really easy just with a new teleform apply.
• 19:03
  SPEAKER_01: There was something I wanted to jump in on here, and I don't want to open Pandora's box too much, but uh, you mentioned something around how some of the ways you run your infrastructure and your application have to be GDPR compliant.
• 19:15
  SPEAKER_01: Is Airflow something that helps your GDPR story, or is it something that's more of a liability because of how flexible it is?
• 19:21
  SPEAKER_03: Good question.
• 19:22
  SPEAKER_01: Uh GDPR is GDPR, no?
• 19:24
  SPEAKER_01: Yes.
• 19:24
  SPEAKER_02: The way that we handle is that since we are in Amazon and we are using secrets managers to handle all the secrets for third-party tools or for databases.
• 19:38
  SPEAKER_02: One thing that we do is that uh depends on where you're deploying Airflow, no, you have access to one secret manager or another one, but at the end, the secret manager is shared across the same Amazon account.
• 19:50
  SPEAKER_02: No, it's not just that you have one different uh secret managers in different account in the same account, you have just one.
• 19:56
  SPEAKER_02: What we do is we replicate, we are using the same name uh between accounts, but each account has their own um secrets.
• 20:05
  SPEAKER_02: No, so we deploy Airflow in development, the secrets manager contain only secrets to access uh development DBs, no, or um the development and sandbox on third-party rules.
• 20:18
  SPEAKER_02: That guarantees at least that even with uh anyone deploying something into development by accidental, is not accessing uh production.
• 20:30
  SPEAKER_02: No, so the way that we handle GDPR in some way is also like guaranteeing that the secrets managers are not uh populated with uh production credentials.
• 20:40
  SPEAKER_03: Yeah, and and that answered your question that does Airflow help us to be like GDPR compliant and things like that?
• 20:46
  SPEAKER_03: Let's say yes, potentially yes.
• 20:49
  SPEAKER_02: Also, here maybe we can could talk about the ARCA, no, the uh EM role service account mapping that we are using in Airflow.
• 20:59
  SPEAKER_02: No, the the good part here is that we are mapping, no, we are using uh mapping with between roles and service accounts.
• 21:07
  SPEAKER_02: So it's we instead of using Amazon credentials, no Amazon keys and secrets, or assuming roles from each uh Airflow operator or each task, we deliver this to the to the pod itself.
• 21:23
  SPEAKER_02: So each pod is being executed using a service account, and it's that what allows the pod to access different resources in Amazon.
• 21:32
  SPEAKER_02: So by using this, we can also protect some S3 buckets, Athena, Spectrum, uh, Dynamo, and other things because the service account doesn't allow them to access this data.
• 21:44
  SPEAKER_03: Yeah, we can have several different service accounts on that premier is namespace that all the pods will execute.
• 21:51
  SPEAKER_03: And depending on that service account, each service account has an associated role, and that associated role at the same time has some kind of boundaries that.
• 22:00
  SPEAKER_03: That any pod will uh will not be able to pass that.
• 22:05
  SPEAKER_01: It's pretty awesome seeing just a lot of the native AWS abstractions that come together with the Kubernetes abstractions because then you can set up this really cool thing where your application is super secure, but it's also very elastic in the sense that you can launch as many pods as you need and they'll have the same service account.
• 22:20
  SPEAKER_01: I also saw that they launched you can specify pod security groups on EKS now.
• 22:24
  SPEAKER_01: I haven't really gotten a chance to play around with that feature yet, but it's just really cool seeing all those moving parts that come together, especially for use cases like yours.
• 22:32
  SPEAKER_01: Yeah.
• 22:33
  SPEAKER_02: But it's also just between teams, also, no.
• 22:35
  SPEAKER_02: Uh for example, we have a service account for machine learning.
• 22:39
  SPEAKER_02: No, so they can use it that and we know that they will only access in will we only access S3 machine learning buckets.
• 22:47
  SPEAKER_02: No, or that we can maybe in the future if uh there is another team that needs to build more DAX, we can provide them another service account.
• 22:55
  SPEAKER_02: So we are guaranteeing that they are not accessing or writing data that they shouldn't.
• 23:00
  SPEAKER_03: Yeah, because at the end, if you use an an Airflow, an Airflow secret or an Airflow connection, you cannot like constrain which connection they can use or which not.
• 23:10
  SPEAKER_03: So it's like, okay, they can eventually get the credentials for accessing that database that you don't want them to access to.
• 23:17
  SPEAKER_03: The way to constrain that and the way to be again GDPR compliant, EPA compliant in case that you want to be is having really short-lived tokens, which IRC provides that.
• 23:28
  SPEAKER_03: We are getting away from having long uh lived, long-lived secrets or credentials containing AWS access keys or secret access keys, and and that guarantees the need of being really methodical on what resources can each pod access to.
• 23:47
  SPEAKER_00: Yeah, that plays nicely into my next question, which would be what does it look like for a user to get a DAG running at Typeform?
• 23:54
  SPEAKER_00: I know we talked a little bit about your DAG generation tool, which I believe is YAML-based.
• 23:59
  SPEAKER_00: What other considerations are taking into account into that process?
• 24:02
  SPEAKER_00: What does a developer need to do to make sure that the DAGs they're writing are deployed safely, securely, and within the guidelines of what you all are imposing as the data platform team?
• 24:12
  SPEAKER_02: Sure.
• 24:12
  SPEAKER_02: As I said before, no, and we provided this gesture tool.
• 24:16
  SPEAKER_02: So all the user needs to start like having no to deploy their own pipeline is to make a pull request.
• 24:22
  SPEAKER_02: No, they provide the YAML with the different steps, the dependencies, also the team.
• 24:28
  SPEAKER_02: No, we're right now we are not still like using this property, the team, but we could start like uh using this to define the service account or the namespace where we want to run their pipeline.
• 24:40
  SPEAKER_02: No, we have uh two right now for but more for machine learning training models.
• 24:45
  SPEAKER_02: But in case that uh more teams uh get into that, or we need no, we start having more straight constraints on splitting them, we could use this for and splitting them.
• 24:56
  SPEAKER_03: Sorry, and especially for getting another abstraction layer of data scientists, they don't know what a service account is, they don't know what a coordinated service um or namespace is.
• 25:06
  SPEAKER_03: So it's like the more we can abstract things for them, the better.
• 25:11
  unknown: Yes.
• 25:11
  SPEAKER_00: It's almost like your job as the data platform folks are to build the guardrails so that it's very easy for other folks to self-service.
• 25:18
  SPEAKER_00: Again, ties back to the idea of not being a bottleneck, right?
• 25:20
  SPEAKER_00: Like being an enabler over a bottleneck.
• 25:23
  SPEAKER_00: Do you have data scientists often deploying DAGs to various airflow environments around Typeform?
• 25:30
  SPEAKER_02: Yes, more than once or twice per week a new DAC is created.
• 25:34
  SPEAKER_02: At the end, they it's what's with the queries and also with the dockers.
• 25:38
  SPEAKER_02: Now, as I said, uh we really make really easy to start running models on Dockers using the Kubernetes bot operator.
• 25:46
  SPEAKER_02: Yeah.
• 25:46
  SPEAKER_01: So and then you might have said this earlier, but I might have missed it.
• 25:51
  SPEAKER_01: Where are the DAGs stored?
• 25:52
  SPEAKER_01: Are they PVCs on the Kubernetes cluster or are you building images for each one?
• 25:57
  SPEAKER_03: No, the DAGs are stored on Git, and we have a sidecar container from that that runs as a sidecar for each airflow instance, and we get the DAX from Git.
• 26:07
  SPEAKER_03: Gotcha, git sync.
• 26:08
  SPEAKER_00: Pete, I'll kick it to you.
• 26:10
  SPEAKER_01: Sorry, I was like a little caught between.
• 26:12
  SPEAKER_00: No, you're all good.
• 26:13
  SPEAKER_00: You're all good.
• 26:14
  SPEAKER_00: So have you guys done any other work to accommodate or extend Airflow to accommodate more specialized use cases?
• 26:20
  SPEAKER_00: Maybe any work around the Airflow stack that you've done to make it better fit for specific use cases more so than others.
• 26:26
  SPEAKER_02: From one side, no, it's like all the monitoring, no, all the alerting.
• 26:31
  SPEAKER_02: We did an extra layer, no, uh, since the good part about using a DAC factory um in library is that you can put and orchestrate everything in just one place.
• 26:44
  SPEAKER_02: No, so all the integration with Datadoc, with uh Slack and mailing in case of alerts or to monitor performance of each task and DAC, it's free for everyone.
• 26:55
  SPEAKER_02: No, that's maybe an extra layer, no, because uh we have we see we saw this quarter, for example, some engineering teams starting requesting to deploy their DACs.
• 27:07
  SPEAKER_02: Well, they didn't have any DAC way, no, because at the end you have back-end people that are caring more about building the product, no, but not about pipelines.
• 27:17
  SPEAKER_02: This is more for data teams, but we saw some adoption from engineering asking for this because they needed some way.
• 27:24
  SPEAKER_02: They used to have Travis or Jenkins, well, more Jenkins jobs, so that was uh long in them.
• 27:30
  SPEAKER_02: But coming back to the question, the thing is that we have this data doc Slack emailing, and then for Kubernetes pod operator, we have all the secrets and connections being handled, and one of the good parts that we built also is then even that now is native, no Airflow 1.12.
• 27:52
  SPEAKER_02: No, they start integrating with the secrets manager from Amazon, but before it was uh more handmade, no.
• 28:01
  SPEAKER_02: So one of the things that we provide is a sync between the secrets manager and the connections of Airflow, or the secrets manager and the secret and the secrets provided in Kubernetes, no, the secrets available for the pods to access.
• 28:14
  SPEAKER_02: So this was also another extra layer that we did before.
• 28:18
  SPEAKER_02: Now we have to to review it with the new uh secrets manager uh native airflow supported way, but that was also a new good thing that we did.
• 28:29
  SPEAKER_01: All right, cool.
• 28:30
  SPEAKER_01: Shifting gears towards the future a little bit.
• 28:33
  SPEAKER_01: You mentioned that there's some cool functionality with 1.10.12 around integrating with secrets manager.
• 28:39
  SPEAKER_01: And I think earlier we said that 2.0 is right around the corner.
• 28:42
  SPEAKER_01: Is there anything in 2.0 that you're particularly excited for and features that you're itching to get your hands on?
• 28:48
  SPEAKER_02: To be fair, I'm really excited to see how the Airflow 2.0 goes with the UI.
• 28:54
  SPEAKER_02: No, I had the um the feeling that it was a bit old, no, even that is a tool that is being used a lot.
• 29:03
  SPEAKER_02: We are quite happy right now with all the capabilities that offers Airflow.
• 29:08
  SPEAKER_02: Maybe this is the second manager was solved in the last uh release, even that we want to try it, we didn't have it yet.
• 29:15
  SPEAKER_02: And I was, for example, two days ago I was checking the new way how is changing the Airflow 2.0 with the providers instead of the contributions.
• 29:25
  SPEAKER_02: And that looks quite promising, no, the way how you can integrate more models in an open source way.
• 29:33
  SPEAKER_02: So I wait that will show up a lot of things.
• 29:36
  SPEAKER_00: Yeah, we're excited about all of the above as well.
• 29:38
  SPEAKER_00: Obviously, the UI changes that will be coming right out of the gate, obviously eventually, hopefully, to be followed up by a more robust kind of full UI overhaul.
• 29:46
  SPEAKER_00: But right now the changes look very good.
• 29:49
  SPEAKER_00: We have an alpha image that you're able to test, it's out and you can run it locally.
• 29:53
  SPEAKER_00: And I was doing it this weekend.
• 29:54
  SPEAKER_00: And the UI improvement is like one of those things where it doesn't change anything functionally, but it just makes the experience feel so much better of developing locally, just clicking through the UI.
• 30:05
  SPEAKER_00: So we're very excited about that.
• 30:07
  SPEAKER_00: Yeah, but uh I'm also excited about the the provider decoupling.
• 30:11
  SPEAKER_00: That's something that I think is overlooked in the swath of uh features that are being dropped with 2.0, but the fact that the providers are now going to be independently versioned and released independently of the core airflow project will help us improve modularity and uh improve velocity on the actual provider development side of things itself.
• 30:30
  SPEAKER_00: So all very exciting stuff.
• 30:33
  SPEAKER_00: Yeah, and the secrets manager too, finally.
• 30:35
  SPEAKER_00: One a good friend of ours, Daniel Standish, did a lot of the baseline work to get that into Airflow 11010.
• 30:40
  SPEAKER_00: So I wanted to give him a quick shout out.
• 30:42
  SPEAKER_00: He worked with some folks on our team to extend it to other providers, but you know, uh a lot of hard work there from the community, and it's awesome to see it being used by so many folks in production.
• 30:51
  SPEAKER_00: So um very excited.
• 30:53
  SPEAKER_01: Just uh any shout-outs from the typeform team that you want to make.
• 30:56
  SPEAKER_02: Pete, you made yours, so you gotta give him a chance to I think that uh I would like to to congrats everyone that is behind the help and community refroid.
• 31:08
  SPEAKER_02: No, it's really easy to work.
• 31:10
  SPEAKER_02: For me, it's thanks for all the people that is behind and all the um open source.
• 31:15
  SPEAKER_02: No, all everyone that is contributing, they have my thumbs up.
• 31:21
  SPEAKER_02: Yeah, thumbs up for them.
• 31:24
  SPEAKER_02: It's nice.
• 31:25
  SPEAKER_00: All right, guys, thanks so much for coming on.