Latest / The Laravel Podcast / The Architecture Behind Laravel Cloud with Chris Fidao

Transcript

The Architecture Behind Laravel Cloud with Chris Fidao · The Laravel Podcast

• Matt Stauffer: Welcome back to Laravel podcast season seven. I'm your host, Matt Stauffer, CEO of Tighten. And in this season, I'll be joined every episode by a member of the Laravel team. Today I'm talking to my, I don't want to say my old friend, but it always makes it sound like you're old. My friend for many years, Chris Fidao, a team lead for infrastructure of Cloud at Laravel. Chris, can you say hi and share a little bit about what you do at Laravel? Chris Fidao: Yeah, hi. Like you said, I'm Chris. I am on the cloud team at Laravel. So that means we, I, and the team, of course, have built up the Cloud product. And right now, over time, I've transitioned. We were talking about this earlier. We transitioned from basically being an infrastructure engineer, doing Terraform, building infrastructure, all that fun stuff. And now I'm the team lead for the infrastructure Cloud team. I'm going say team a million times. Which means I'm effectively an engineering manager now. So my day-to-day now is managing people instead of actually doing the code. Although, you know, it's a little of everything. Matt Stauffer: Yeah. And I, I have a lot of questions for you about that, but before I get to those questions about what you do day to day, I want to know more about what the story was for you coming to Laravel. So people might not know, you know, what your history with the Laravel framework and everything like that looks like prior to actually having a job at Laravel. And also I'd love to hear a little bit about like what you did when you first joined here, and then we can kind of get into your job today. So first of all, what, what was your introduction to Laravel the framework and what was that kind of journey like? Chris Fidao: Yeah, my god. Okay, so one of my first jobs was at a marketing agency and we did content management systems, right? So that was, I mean, in our case, it was all PHP, right? So some WordPress, but actually mostly everything. It just skipped. We had an era of Drupal, Joomla, am I saying it right anymore? WordPress, a lot of Expression engine. That was very heavy. And... Matt Stauffer: Really? Didn't y'all do EE as well or did I make that up? Okay. Chris Fidao: I didn't get to others because then we did some custom development and guess what we do when you go to custom development in PHP, you use a framework. So CodingNighter, eventually Laravel. And by the time Laravel came around, it was Laravel 4, I think I jumped on it, or 3? Must have been 4 though, feel like. Something like that. Anyway, so around that era, I started learning Laravel and it was amazing, right? CodingNighter was great, but Laravel just had all the tools that you want in a way that made a little more sense than CodingNighter Matt Stauffer: Yeah, Yep. Chris Fidao: to me at the time. So then I was like all in on Laravel and was young, had no kids, right? So then I start blogging, write a book, making videos, you know, do all the things you do when you have time. Matt Stauffer: As one does when one does not have kids. Chris Fidao: And you know, that was all fun and great, which led me to talking to Ian Lansman, who at that time employed Taylor Otwell, right? Maker of Laravel. And I ended up working at Userscape with them. So then I was working with Taylor and Eric Barnes and kind of fun people, know, kind of like, you know, Laravel ecosystem people, right? And then I was working with them, which is great. So obviously, Laravel heavy there. Actually, the product we worked on was older, preceded frameworks even. like, Laravel got shoved in there eventually, but that kind of stuff. Okay, so that's like where I came with Laravel. And like, from there, it's just like... Matt Stauffer: Yeah, but real quick though, you're not just an average person blogging about Laravel at that time. You had a little bit of a bent, which actually kind of leads into you being on the infrastructure team. So if somebody is not familiar with your book, your blog, like what, what if you were the something guy in the Laravel world, talk about kind of what your focus was on. Chris Fidao: Yeah, okay, that's a good point because eventually I started the servers for hackers newsletter, right? And that is all about servers for programmers, basically. Hackers being programmers, not like, you know, illegal, whatever. Despite the occasional email I get of people requesting me, asking me how to hack something. I don't actually know. Matt Stauffer: script-canny-type stuff exactly. Chris Fidao: so yeah, that started as a newsletter. Got a bunch of people and then I turned it into a website and I made a bunch of video courses and that kind of stuff. The whole, theme was always like, how to do server stuff if you're like a PHP developer and don't necessarily, you you're not a sysadmin, you're just like a programmer. Just, quote unquote. And that was the bet there. So like over time I became known in the Laravel community at some phase of Laravel history as like the server guy. I don't know if that's true anymore. I don't publish a lot of public stuff anymore. So it's already like, know, younger people probably have no idea who I am. Matt Stauffer: Yep. But you definitely. Chris Fidao: But you know that was the bent so eventually Laravel cloud came along by then I'm not no longer a userscape. I was working at fly.IO Laravel got funding all that fun stuff They're starting to make cloud and Taylor you know message me and is like you be interested in and starting on Laravel Cloud and doing infrastructure and of course You know the mothership calls you you don't say no. Matt Stauffer: Yeah. Yeah. Okay. So at that point you are working at Fly and Fly is infrastructure as well, but I feel like you were more content there. Is that right? Chris Fidao: Exactly, yeah, that was an interesting pit stop in my career in that I wasn't directly doing server stuff anymore. I was kind of like evangelizing Laravel for Fly.IO, making it on the framework scene they called it. So they had different framework people, right? Rails, Laravel, whatever else, whatever JavaScript is. And then that job ended up being mostly content, but then also some programming, but not... Matt Stauffer: Okay. Chris Fidao: core infrastructure engineering. So I wasn't on the team like running servers, running firecracker virtual machines and all that fancy stuff they did. Matt Stauffer: Yeah, did you do infrastructure at Help Spot? Okay. Chris Fidao: Yeah, yeah, yeah, because we made the product, I'm going to call it product, HelpSpot Cloud, and that was basically hosting HelpSpot, because traditionally it was a self-hosted product for years and years and years, and then we started hosting it. Matt Stauffer: Yeah. Yeah. I had always assumed that you're the one who was the big part of that, but I never actually knew if that was technically your job, so... Chris Fidao: Right, because by then I had done the servers of hackers stuff also. Ian, who owns Userscape, knew I knew server stuff, and that's how that started. So we were not doing anything fancy there either. That was a few hundred, which sounds fancy, but isn't, just EC2 servers, just regular virtual machines in AWS. And each customer got their own virtual machine. So we had that kind of model. So the game there is just managing. Matt Stauffer: Okay. Chris Fidao: you know, this customer needs an update. So push the button to update and then make the process that does the update to the version of Help Spot there and that kind of fun stuff. Matt Stauffer: Got it. Okay. So you ended up at Laravel working as an infrastructure engineer originally on the team. And before we move into the kind of engineering role that you're in right now, what kind of stuff were you working on at the beginning there? Cause I, you we haven't actually gotten to talk a ton about what is actually running in the background behind Cloud. I've like, I've had somebody like drop like a one sentence version of it when I was introducing Cloud, but like what, what were you getting deep into and what was it like getting deep into something when you had kind of done more of like a content direction for the last few years at Fly? Chris Fidao: Yeah, that was fun. I think the short answer there is that I had a lot of energy to get back into real infrastructure stuff instead of just content. Because the content stuff is fun and something I always like to do, which is why I switched jobs to Fly. But it ended up being I actually need both in my life, some content and some not. I love development and that kind of stuff, which makes sense because you do the development and then you have stuff to write or write about, which now I'm too busy to actually even do that. But still happily Matt Stauffer: Yeah. Yeah. Chris Fidao: chugging along with being able to do some, well, it's changed now, but doing a lot of programming and that kind of thing. And then also learning new roles for me being an engineering manager is like a new thing. But anyway, to get back to your actual question, what was I doing in those early days is figuring out how to build Cloud. Because I came in specifically to help build the Cloud and that kind of stuff. So I was pretty early after funding. Even before... Matt Stauffer: Yeah. Chris Fidao: Andre started. I think by then he had confirmed he was joining Laravel, but hadn't even started yet. Tom, the CEO, was there, then me, and then Andre started. So I was there really early, right after funding and then figuring out how we're going to do stuff. So I got to help work with Joe, Joe Dixon, on how to figure out what we're going to do with Cloud. At first, we were looking at Vercell's model of using serverless, Lambda. And we ended up not doing that Matt Stauffer: Okay. Chris Fidao: for just a bunch of reasons. We can get into that if it's interesting. But we ditched that and then we're like, okay, so if we're going to use not serverless, then we're doing servers, and then what are we going do to manage servers? Kind of the next obvious thing is Kubernetes. Just because it has a huge ecosystem. Basically because it has a huge ecosystem. There's nothing it can't do effectively. And it helps with our model of how many... Matt Stauffer: Yeah. Chris Fidao: workloads can be put into a server reliably and all that kind of stuff. Kubernetes just has answers to that kind of stuff. So Kubernetes, it runs containers and it answers the question of, if I have a crap load of servers, how do I figure out what server to put a workload on and manage putting a workload there and run it and connect it to the internet and all that stuff. It just answers those questions for you. So that becomes the next logical thing to do if you're going to do a server full solution to run workloads. Matt Stauffer: Is Kubernetes like, I have toyed with it in the past enough to know that it's not my place. I'm like, there are people at Tighten who know it better than I do, but I don't. In Kubernetes, is it purely code configuration or is there a dashboard somewhere that you're interacting with? Chris Fidao: There's no dashboard unless you install one in which those exist. And even then the dashboards usually have some kind of functionality that you're looking for. we use one called Argo CD and it basically like GitOps' stuff inside of a Kubernetes cluster. And in fact, you can actually attach other clusters to it and manage those other Kubernetes clusters from this one thing. So. Matt Stauffer: Okay, that's what I thought. Yeah. Yeah. Chris Fidao: That gives you a UI that you can click around in and see what's running in any particular cluster and its status and look at its logs and all that kind of stuff. So essentially, no, it's mostly YAML. It's all YAML all the way down. And the YAML tells Kubernetes to do stuff. Matt Stauffer: Yeah. Okay. That and that's what I have experience with. But I was like, I haven't gone like to your level of Kubernetes. So I was just like, is there a world that I don't know about other than just the yaml and yaml and yaml? Okay, got it. Yeah. Chris Fidao: There is a fun thing where we do, and it's called an operator. And you can program against Kubernetes API. And essentially, the way to think about that is that you're not actually generating YAML at all. You're actually making API calls to the Kubernetes API. But you can think about it as code that generates YAML that tells Kubernetes what to do. The YAML, in the end, is getting converted to something that makes API calls anyway. And this way, you get to programmatically interact with Kubernetes stuff. So you can do fancier things than, as you can imagine with code, So like, YAML can tell Kubernetes to do something, like spin up this container, run it, blah, blah. Code can be more interactive and say, this is the status of the world. Do this task. Like an example of that is when you deploy an application on Cloud, you might want to run a task right before the code is released. So that might be like a migrate command, right? PHP, artisan, and migrate, something like that. Matt Stauffer: Then you should do this because of that. Chris Fidao: We can do that because we have this operator set up and it knows how to orchestrate everything to like, for example, have your newest deployment running, run PHP artists and migrate, but if it fails, roll back to your previous version of your code. Matt Stauffer: okay, yeah. Where building something like that in YAML would be basically impossible. Chris Fidao: Yeah, I was about to say, would it be impossible? Probably. I certainly couldn't make it a nice experience. Matt Stauffer: Yeah, if it was not impossible, would certain. Yeah, exactly. Okay. Well, that's really cool. You had mentioned a minute ago that we can go into that, you know, if we want. I would be loved to learn more about this kind of like serverless. It feels like serverless had its heyday and then it's on its way out of its heyday. And obviously you guys weren't chasing what's popular when you made the decision. You know, if serverless was the right decision for Cloud, you would have done it. But it is curious to me that part of the decline is people just seeing a lot of the shortcomings of serverless, you know. Chris Fidao: Mm-hmm. Matt Stauffer: So I'm curious, is there an easy way for you to share with normies what kind of problems you did run into as you guys were considering serverless? Chris Fidao: Yeah, no problems, because it didn't even get as far as deciding to use it and try it out. Well, we do know problems because we have Vapor as an example. Matt Stauffer: Vapor, yeah. Chris Fidao: And the biggest problem is visibility into what's going on into serverless is hard, just by nature of serverless. You can't SSH into anything and see what's up. But just by nature of how they built Lambda, also the 15-minute limit of running a Lambda is pretty killer, depending on your queue jobs and that kind of stuff. So AWS has their specific use cases if they want people to use Lambda for, and things that take longer than 15 minutes is not one of those use cases. Unless you do fancy things like step functions or whatever, can chunk jobs into different... whatever, who cares? Doesn't matter. Doesn't work for Laravel as easily. So that's one factor, just one. Other factor, it's expensive. It's the most expensive way to run compute in a public cloud, and that's AWS especially. Matt Stauffer: Got it. Chris Fidao: A fun thing I learned, we actually got access to, we got to talk to a bunch of people in AWS, thanks to being like an enterprise account and that kind of thing. So one of the fun things I learned is that one of the reasons Lambda is so expensive is because it has to burst to so much capacity necessarily. Like some customers have a lot of bursty traffic and just like all of a sudden, tens of thousands of micro virtual machines have to spin up. It's servers behind the scene, right? So AWS has to have... Matt Stauffer: huh. Chris Fidao: all of these servers just ready and available to run these little micro virtual machines on. Therefore, Lambda is expensive because they have to pay for all the servers to actually be running all the time. You know, like the power consumption of having all these servers, just whatever. So they have to allocate all these like huge servers to the Lambda service. Therefore, we pay a lot of money because, you know, for the benefit of having quote unquote unlimited, you know, bursting at any point. But anyway, it's expensive. So... Matt Stauffer: Sitting idling. Wow. Chris Fidao: That's another factor. It's expensive to us. We have to provide the service to other people. We have to mark it up so there's profit to be made. Therefore, it's not the best way to go because it's so expensive. And then we were thinking of Vercell, but Vercell is primarily front-end stuff, so a lot of caching can mean play. And therefore, they're serverless functions, whatever, get hit less than something in PHP where every request basically has some server component. So every request will have a server component. Hit lambda, run lambda, causing expense. Matt Stauffer: Yeah. Yeah. Yeah. Chris Fidao: Whereas something like for cell probably has a lot less of that because they can get cached so aggressively that a lot of stuff is like front end JavaScript stuff running that can be cached and returned and not necessarily hit a Lambda behind the scenes. I don't know that's the reality anymore with other features, but that's one of the reasons why we turned away from it. Matt Stauffer: Yeah, Ya know makes a lot of sense. And I know that this is kind of what you did in the past, so thank you for kind of being willing to kind of dive down this road a little bit. So at some point, you made a transition. Did you go straight from that level of kind of engineer to where you are today, or was there an intermediary phase? Chris Fidao: To where I am today being an engineering manager. I guess there was, so what I did is that I just pretended I was just kind of this position the whole time. So in the Aaron Francis point of view, if you can just do things, I was first. And so we brought on more people pretty quickly, because I can imagine you need a lot of people to do this type of thing. And the other people were and are great and have a lot of expertise, and a lot of cases, a lot more expertise than me. So it made sense for me to kind of like start organizing the team and kind of being that person, as well as having a lot of IC work, individual contributor work, like actually programming, whatever. So there was a transition period in that I kind of like play acted the part that I wanted to be of the engineering manager. And then eventually, you know, it's like 70, 30, 60, 40, you know, the split changes. Then eventually I'm basically like a 95 % engineering manager now. And the title became official also like a year ago. Matt Stauffer: So had you ever managed a team before? Chris Fidao: Hmm. Yes, but not as directly and officially as this. More like in unofficial capacities. So I had some experience there, but not in this like real capacity. Matt Stauffer: So like what was the learning process like? Like did you just throw your, was it, yeah, what is the learning process like? Chris Fidao: It's still going, it's still learning. Matt Stauffer: Like do you, did you say, well, I need to go in a very concerted and intentional learning process where I'm gonna go read books, watch videos. Or was it sort of more just like, yeah, just try it and figure out what to do and ask a friend for help? Yeah, right. Yeah. Chris Fidao: There's a book behind me that I have touched, but like, had to be an engineering manager, but I should have probably read it, but I did not. But the process was kind of obvious too, because, and this is like the benefit of just being kind of early in the process. I just kind of like saw all the steps. I saw new people come in, like product managers come in, different people on the team who are not on Cloud specifically, but you know, affect Cloud. So like,just other people coming to the team, right? So I had all this context, so it just makes sense for me to just kind of know what's going on, see what's happening, keep up with everything, and there for me, like relay it to the team of like how things need to get done. I don't even know sure what I'm trying to say. I'm kind of losing track of what I'm saying here. But like in terms of being an engineering manager, right? It's like a very different job from programming. It has a totally different skill set. But I think that was a part in my life where was kind of ready to do that and I kind of knew that going in. I kind of twisted my expectations. Twisted? That's the wrong word. Changed my expectations of what I was going to be doing along the way. And then I also had all the context of like what was going on, the history of everything. So I can relate that to people and try to like basically effectively communicate like why decisions were being made to, you know, other people on the team of, you know, I don't know. I guess I'll just throw an example. So one example is that like... Matt Stauffer: Yeah. Chris Fidao: We have to be very conscientious of cost because every decision, not every, but many decisions we make might include or might increase our cost per customer's workload. So, and that's different from a lot of people coming in from other places where maybe they have a product they're working on or maybe they're like a platform engineer who make an internal development tool and it's like a cost per developer isn't like a big deal, but like we have to charge for our product. Matt Stauffer: Yeah. Chris Fidao: Therefore, if we do some kind of infrastructure addition that has a definite cost per customer workload that runs, at scale, that makes a huge amount of cost, and then we either have to charge a customer for it or figure out a way to change that. So those kind of mindset shifts are something that I help people along with when we're, I mean, just as a manager, I guess. Matt Stauffer: Yeah. Do you have the ability to run a workload? What was it? Do you call it customer workload? Is that the phrase? Yeah. So if somebody does pull request and they change code configurations for that, do you have the ability to spin up a sample workload and say, it's going to be this more expensive? Or do you have to run that stuff intuitively through your brain saying, I think that's going to do this, and then we have to now go do some tests? Chris Fidao: That's the I use the phrase I'm using. Chris Fidao: Okay, so something that would be not intuitive that could increase would be something that uses more bandwidth than you might expect because AWS charges for bandwidth in all sorts of ways. So that's one way that it's not easy to know it's going to cost. But other things are easy. We run MySQL instances for people that also uses Kubernetes, so that's in a container. To monitor MySQL, you often use this thing called Metrix exporter. So it reads MySQL, does a bunch of queries, and then exports metrics to some other monitoring systems so that we can see how many queries per second are happening or whatever. Matt Stauffer: Hmm. Okay. Chris Fidao: One model of that is to attach a metric exporter, a thing monitoring your MySQL instance to every single MySQL instance. Therefore, you have this extra sliver of compute running for MySQL instance. Therefore, you're going to need more servers and aggregate because more costs, more compute is being used. So an alternative to that is to run very few of those exporters, but each one monitors multiple servers. Matt Stauffer: Got it. Yes. Yeah. Yeah. Matt Stauffer: share across. Okay, that makes sense. Yeah, you need to know things that I don't know. So I'm grateful you're the one doing that, not me. Chris Fidao: The team, the team. Now I'm managing a team. People are doing stuff. Not necessarily me. Sometimes me. Matt Stauffer: So you mentioned, yes, yes, y'all are doing things that I'm not. Yeah. And speaking of that, that was actually my next question. How much coding work are you doing at this point in your career? Chris Fidao: I am at a point where when I'm doing IC work, it's more of a failure than a success because it means that there's time pressure or something, some project where people are loaded up with work and like I might do that to try to offload some work for someone to like get something done. Or again, because I like kind of was there from the beginning, there's like some stuff that I know that's not intuitive to other people. So like I just kind of like end up helping out with that kind of thing. But from that point of view, it really should be almost none, except for things I want to do or side projects I'm investigating a strategy or might employ. But it still happens, because just the nature of work. There's a lot of work to do. Limited time, limited people, but unlimited scope of work to potentially do, right? So there's lots of stuff going on. And sometimes I end up doing actual code, which of course I enjoy. So not a big deal from that point of view, other than the fact that it takes time away from real Matt Stauffer: Yeah, yeah, Chris Fidao: managerial type work, know, improving processes or documenting stuff, that kind of thing. Matt Stauffer: With all love and respect to everybody you've worked for before because obviously we've already named and we love Ian and there's other folks you've worked before that you don't want to disrespect anyway and I'm not suggesting that but are there any ways that you as a manager are stepping in and saying you know what like this is something I've always wanted and it not because other people did a bad job or whatever but you're just like I as a manager really want to provide this or I've gotten this at previous places I want to make sure that I provide it or is it more just very instinctive is sort of like I think this is what makes sense here like does that make sense as a question? Chris Fidao: I think so. One thing I really liked from other jobs was a way to check in on Slack and be like, I did this. Everyone can do that. This was specifically at Fly. They had a bot where you would be at this bot and say, I did this. And you could do it multiple times a week, kind of whenever you did a thing. And then in aggregate, or really anyone could see it. They could just see what a person did, which that was really neat and helpful because as a manager you want to have evidence, evidence? Evidence, of what people have actually done, a history of the work people have done and then also a way to see when people are a great job and point to this so you can say this person should get a raise, get a better position or this person should, it hasn't really happened that much or anything, this person needs improvement. Right now we've hired lots of senior people and that kind of thing so everything's good from that point of view. Matt Stauffer: Yeah. Yeah. Chris Fidao: But that kind of thing is what I really wanted. And something like that did happen at Laravel. I advocated for that a bunch. In the end, we resurrected this beep app that Taylor made years ago. And that's like a, and it turned into like a Slack check-in app. And so that came to be, thanks a lot more to Joe Dixon than me, although I was advocating it for it behind the scenes. So that kind of thing certainly has been around. I don't know if I have any other great examples though. Matt Stauffer: Really? I love that. Matt Stauffer: That's awesome. Matt Stauffer: Okay. So, you know, my first question was kind of what do you do in a day to day from a code basis? But you mentioned kind of like if you're coding, it's somewhat of a failure unless it's a fun thing to decide. So what do you do day to day as a manager? How much your time is spent in meetings versus planning versus whatever else? Like, what is it that you do here? Yeah. Yeah. Chris Fidao: Yeah, All meetings. What is I doing here? I don't know anymore, Lots of meetings. For better or worse, there's, I mean, you could, yeah, there's a bunch of meetings and a lot of it is planned. So we work in six week cycles, which is, it's like that, my God, I want to call it Upwork. That's not right. What's the third, yeah, Shape Up, thank you. I think I would know, because we do it. Matt Stauffer: Shape Up. Chris Fidao: So we do the shape up thing every six weeks, six weeks work cycles, we have cool down cycle, that kind of thing. So a lot of management stuff is helping with the product managers and then Joe and Jeremy and myself who are kind of like leads of teams. Joe is like the overall tech lead. Jeremy who's the lead of like the web app side, myself the lead of the infrastructure side all in Cloud. Get together and do a lot of management of what projects are going to be coming up the next cycle, how the projects are going currently and all that kind of fun stuff. So there's a lot of those type meetings and there's meetings with partners. There's meetings with potential customers, usually kind of like leaning towards enterprise or large team side of things. There's one-on-ones with the team as the self, right? And that's the majority of them. Matt Stauffer: Okay. I've had a lot of different conversations with people about how they do one-on-ones and I've found that people can dogmatic is not the right word, but you've got a very strong opinions about what a good one-on-one looks like. And I'd never have thought that my one-on-ones are what they're supposed to be. And I like them. And as far as I know, the people who work for me like them. Do you have any aspects where you're like a good one-on-one must have this? Chris Fidao: Okay. Chris Fidao: No, I hate dogmatic answers, which is for the same reason I was never dogmatic about a single process running in a container, if you happen to know what that means or even care. I love to run multiple things in a container. I'm not dogmatic about meetings and that kind of stuff either. The one-on-ones end up being a combo of things. Sometimes they're work therapy. Sometimes they are like what's going on with your project. Matt Stauffer: Yes! Chris Fidao: Sometimes it's like we just completely talk about tech. It's not even the right phone call for that, but we end up doing it because why not? That's the subject that comes in the hands. It's really just being in touch with things like do they have issues they need resolving? Something not going the way they want to. It's a great place to get feedback on like some feedback I got is like we used to have an infrastructure meeting with just all the infrastructure people once a week and ended up just being like a round robin of this is what I'm working on. like people that were like that kind of sucks, let's just talk about the things we want to talk about and we did that and it's been great because then a lot of stuff flips to the top that you wouldn't have talked about otherwise but ended up being really important. So as long as there's a good way for you to communicate with your team where they feel like they can talk about harder things, easy things, whatever with you. That's kind of how I see a one-on-one being successful. Matt Stauffer: Yeah. Matt Stauffer: Good. I like that. That's how I do it. And so I'm encouraged to hear. Chris Fidao: So it can't be wrong. Matt Stauffer: Right. There's two of us. So outside of your day to day job and maybe it's something you're doing during job outside of your day to day assignments you mentioned kind of like things you're interested in learning inside projects like what outside of learning how to continue to learn to be a great manager has your attention right now. What are you nerding out about. What are you learning. Chris Fidao: It's more tech stuff about how to run stuff within our infrastructure. Things that are not Kubernetes sometimes are fun. That's like an official thing that would ever happen. But like, you know, just like Fly had this great system of running micro VMs with Firecracker and they have a lot of benefits. They also have a lot of harder things, but they have benefits in terms of like how fast they can spin up and that kind of thing. Think Lambda because Lambda is based on that, right? So you spin up a Lambda in microseconds or milliseconds. Matt Stauffer: Yeah. Yeah. Chris Fidao: Firecracker can do that if we use the system like that too. That's kind of a fun thing to think on, but not necessarily something we like try to implement or anything. But then there's a... Yeah, 100%. Yep. Matt Stauffer: Yeah, but like sometimes learning another system just changes how you think about your current system, right? It gives you new ideas and everything. Chris Fidao: There's more in the weed things. So remember these operators I was talking about in Kubernetes, like a lot of the ecosystem is like installing operators that do a task for you. So for example, we are going to manage infrastructure for... people, obviously, we are going to manage things like RDS databases for customers. And RDS databases is just a service in AWS. So in order to create a database in AWS and RDS, you're just making API calls. So instead of us making a container, running a container, managing my SQL, make sure it doesn't die, that kind of thing, which we're also doing as a SQL service. But some customers might want RDS just for reasons, all the features it has and that kind of thing. So we're going to provide that for customers. So how do we do that? Matt Stauffer: huh, huh. Yeah. Chris Fidao: There is an operator called Crossplane, and there's other ones that do this too, but there's an operator you can write in Kubernetes that called Crossplane, and they can actually spin up AWS resources for you. So like one thing is just investigating, we want to use that, and how would we do it, and that kind of thing, and implement it, and integrate in our systems, and make it so our web app can say this customer wants an RDS database, then talk to Crossplane or whatever, then make the RDS database, and then have a communication come back and tell the web app it's done, and all that kind of fun stuff. That's more of just in the weeds. Matt Stauffer: Is being responsible for code that spins up people's actual servers that run their actual websites is a lot more stressful than just being responsible for your own code. I feel like you're at multiple levels. Like, I'm like, if I push out a new feature and I make a mistake, like one client... May you know one user might be like hey, you know this PDF wasn't where I want it I'm like, I'm so sorry I'll fix that whereas if you guys push out some code you make mistakes somebody's entire server is messed up and they're like Hey, my whole app is broken. You like have multiple levels of responsibility. Is that stressful? What does it look like to write code where you have that level of responsibility? Chris Fidao: Yeah, that's fun for sure. Definitely adds a level of, I don't know if I want to call it, it is stressful. Stressful is when things break. It's not too stressful to like do that, but it does require a lot of thinking and planning, which is, you know, we've hired the right people for that. So it kind of just like happens. Like the real, some of it, so it's all process. It's banking process to make mistakes less likely to happen. Some of it's forced on you by like SOC 2 or other compliance things, which is. Matt Stauffer: Yeah. Chris Fidao: good because they have this forcing function to make you sure you have this process. So for an example there is that everything that might land in production has to have a review behind it. So that's usually like a GitHub review that updates a GitHub repository that then is read by something that adjusts itself because it sees the code and the repository has been updated. You know, GitOps, right? So Git changes, something reads that change, applies that change to your infrastructure. And then there's other fun things like... we can control changes down to the region and sometimes down to the Kubernetes cluster that runs in a region. So you might roll out a change in one region, make sure it's working fine, and then slowly roll it out to other regions. That kind of process too. In addition to having development, staging, production environments, and all kind of fun stuff. So all the touch points you have before. Then infrastructure can have a lot of it is code. Therefore, lot of tests can be written. So those test suites for these operators I keep talking about and all this ancillary kind of software you just kind of like in tooling you make along the way. Matt Stauffer: Yeah, I was having a conversation with Vishal from Laracon India on Twitter the other day and you know, he was saying, you know, we've written some very large enterprise applications for government, stuff like that. And it has led me to want to have a hundred percent test coverage and PHP stand level max and stuff like that, because I think that makes me feel more safe. And I was just like, that doesn't make me feel more safe. It just makes me feel like I'm wasting time, but I got to be transparent that like, while Tighten itself has worked on mega massive, you know, enterprise government things, that's not what I was doing. And now I don't write code anymore. So I'm just kind of like, I'm always kind of like, poking them like is there something that I need to learn here so like hearing you talk about kind of your workflows are there any things where you're like you know what has really saved our butt here is this thing that I wouldn't have done in a smaller setting because you wrote tests before right you had a development and staging environment before. Are there any ways of coding it doesn't have to be about PHP standard whatever else but is there any ways of coding or processes that you're like this is something I didn't need until I got to this level of intensity? Chris Fidao: Okay, the answer is not truly and that's a good thing because it means people listening don't have to worry about like it's just it's more it's not different like things happen at scale you have new issues at scale a big example of that is their database. So we've we've run bad migrations before and they're not bad migrations. They are migrations that took the database a lot It was a lot harder on the database than we thought because there's huge tables the tables are very busy Matt Stauffer: huh. Yeah. Yeah. Chris Fidao: a combination of foreign key constraints being used, which I just freaking hate, but like people love foreign key constraints, but they cause some issues. So some of those issues are when you run migrations, they don't run as smoothly because there's more deadlocks and transaction issues and that kind of stuff. So at scale you run into those kinds of issues, but I don't have a great answer to like, we do this process, but I never did this exact process before because you know, this scale requires it. There's no like magic or we can do this at scale, but nowhere else type things that, at least that we're using. We tend to keep things as simple as possible though. you know, there could be things that come up on the horizon as we get larger. But really it's like just a lot of testing and a lot of that testing is either automated or manual, you know, through our environments and that kind of thing. Having a set process, making sure the process is followed. And I mean, writing a lot of tests, honestly. Like the web application, the Laravel web application is like... Matt Stauffer: Yeah. Chris Fidao: very heavily tested, like extremely, to the point of being annoying. Which is good overall, the code quality is very good thanks to that. A combination of, you know, test coverage, syntax, or style guide, you know, all that kind of stuff. Typing, type tests, that kind of fun thing. Infrastructure side, there's like so many projects that it's not all like 100 % test coverage, because there's like one repository you get out for the web application. There's like Matt Stauffer: Yeah, totally. Chris Fidao: 50 repositories for infrastructure because we just have all these kind of like weird concerns where it doesn't make sense to throw it all into one one git repo. Matt Stauffer: Yeah, 100%. Chris Fidao: It's kind of just the nature of infrastructure. You're just kind of like always doing these smaller things that have a function and they're all glued together. Matt Stauffer: Infrastructure is not an app. It is a collection of pieces that each have independent responsibilities and may run in different technologies and different environments. Yeah, totally makes sense. Okay. Chris Fidao: Yep, yeah, we write a lot of go lang, for example. I've write very little PHP in the two years or so I've been at Laravel. Matt Stauffer: Interesting. I mean, that's not a surprise, but it is a surprise, you know, because you just imagine you work at Laravel, you write PHP, right? No, not necessarily. You mentioned a second ago that you don't believe are you you are critical of foreign key constraints. If you had your druthers, would they be gone entirely? Chris Fidao: So the trade-off of foreign key constraints is that you can apply them to your database and then you get neat things like being able to cascade delete and you get better data integrity because your database is going to yell at you if you try to like do something for data. Exactly, right. Matt Stauffer: Sort of reference that doesn't actually exist or whatever. Chris Fidao: The drawback is that when your database is kind of heavily used you get more deadlocks and that kind of thing which like I'm not a DBA so I don't I can't explain this in the level I want to. Which also means anyone who's seeing this or hearing this can be critical of my opinions. That's fine. Just know that people of Planet Scale agree with me. So, you know, they know what they're doing. So I don't know. Anyway. They're also biased because the technology they use does not like foreign key constraints under the hood. So. And everything with grains of salt. So the trade-off there is that more deadlocks, more issues with transactions, migrations become harder because at scale, at scale, quote, unquote, which sounds super fancy. There's a lot of stuff happening in the database. Matt Stauffer: Yeah, always. Chris Fidao: Just makes it harder because you run into issues that lock up a database table or something and things therefore take longer or fail altogether because of a transaction deadlock happened too long and therefore it failed back, failed and had a rollback or whatever. So what you have to do if you drop foreign key constraints is to code around things. So you have to write more logic to delete stuff because there's no cascade delete. And then you have to be more careful in your code to make sure the data integrity is present, right? I deleted this and this table, therefore this stuff and this table also needs to be deleted and they're And you don't want to leave dangling data behind. So you'd have to code that in instead of just get it from the database. Matt Stauffer: Yeah, yeah, we maintain a web application. I'm to be as generic as I can so nobody's going to know what I'm talking about. We maintain a web application that was written a decade ago, I think, by a developer who was learning as they went. And the number one problem we've run into in this database is it's just massively complicated application. I'm very impressed with this person being able to build that thing that had so many different views, so many different calls, so many different controllers. But the number one problem we run into is they're like, this page just errors out and we don't know why. And it's because there's data inconsistency where one thing got deleted or renamed or repoint or whatever. And there's zero foreign keys anywhere in the app. I'm 100 % with you in that something somewhere needs to maintain that data consistency. It's either got to be your database base through foreign keys, or it's gotta be your code, but either way, you can't allow yourself to get a situation where something's referencing an ID that doesn't exist somewhere else. Well, okay, yeah, it's gonna break. Foreign keys, I think, are nice as a default because you get that logic for free, right? If you just teach someone, like, if you build something that is related to another table, put a foreign key constraint onto the ID of the other table, and then you're good. But I love the idea that just because it's a good default doesn't mean you always need to use it because the number of times I've been bit by this exact situation that you're talking about where like, well, I'm trying to delete a thing or I'm trying to migrate a thing or whatever and just, and I'm also not a DBA, but I've been bit by foreign key constraints more often than I'd like to admit. So yeah, it depends, right? It's just another one of those. Chris Fidao: Right, yep. It's one of those things, and I think I agree with you. I think start with them and then back out of them as you need to, which isn't necessarily always easy to do, but in general, not too bad. But yeah, start with them. They're good default. They let you write less code in theory, and then you should get back out of them if you hit scaling issues. So, you know, fine. Matt Stauffer: Yeah. So I know we're kind of nearing the end of the time, that we have allotted as always, I can talk to you for hours, but I'm trying to kind of keep it in this one particular spot. So I was asking you, what you're kind of interested in. You named a few and I interrupted you because I was really fascinated by one, were there any mother, any more aspects of like what you're learning or what you're kind of playing around with that you wanted to share that I interrupted or do feel like you got to what's on your mind? Okay. Chris Fidao: No, I don't have anything else on top of mind that would be fun to talk about. I'm sure there's something, I just can't think of it. Matt Stauffer: Okay, yeah. And then is there anything else you hope we get a chance to talk about today? Chris Fidao: Man. I'm gonna have a tough time thinking of things. There's a lot going on and everything's very busy. It's actually been fun because I've been off social media a lot because I'm just so busy. I don't have time to think about it and I feel like my mental health is just way better because of it. So there's a there's a meta topic about being busy. Matt Stauffer: Okay, yeah, totally fine. Yes. Matt Stauffer: Yeah. Yeah, yeah. No, 100%. My wife, Imani, she's like, she has to do social media for work. And when she's choosing only to do it for work and not for personal, she's like, this is drastically better. I kind of wish I didn't have to do it for work. You and I have several friends who've been like, Yeah, I just quit social media. And then my life was better. Turns out it's perfectly fine. So yeah, love it. Well, Chris. Chris Fidao: Yeah. Matt Stauffer: You're amazing, as always. Thank you so much for hanging out. Thanks for sharing all these things. We learned the things I was not expecting. So thanks for kind of teaching us all this cool stuff. Chris Fidao: Cool. All right, yeah, happy to. Thanks for having me on. Matt Stauffer: And for the rest of you, we will see you all next time. Chris Fidao: See ya.