Latest / The Laravel Podcast / The Making of Laravel Cloud: Behind the Scenes with Joe Dixon

Transcript

The Making of Laravel Cloud: Behind the Scenes with Joe Dixon · The Laravel Podcast

• Matt Stauffer: All right, everybody. Welcome back to Laravel podcast season seven. I'm your host, Matt Stauffer, CEO of Tighten. And this season I'll be joined every episode by a member of the Laravel team. Today I'm talking to Joe Dixon, Engineering Team Lead at Laravel. Joe, can you say hi, share a little bit about what you do at Laravel day today? Joe Dixon: Hey, Matt, thanks for having me on. Yeah, so my name is Joe Dixon and I am the Engineering Team Lead for Laravel Cloud. And my days at Laravel are kind of forever changing, I guess I would say, like particularly in the last year or so, it's been a wild ride because we obviously kicked off, we broke ground on Cloud back in March of 2024, so not much over a year ago. And it was a very small team. So to begin with, was just kind of myself and Nuno and Chris Fidao who joined the team very early on and Mohammed at the time was working on Cloud. And so that was very kind of, I was very hands-on, I was very much in the code, I was very much kind of helping set some of the direction of which way we were gonna take the project. And then over the course of that year and a bit now, I was just counting up just before we came on the call, we're now up to kind of the team is around about 18, just about to be 18 people. 14 right now, but we're approaching 18. So we're growing from a very, very small team who are, you know, kind of trying to move really, really quickly to getting Laravel Cloud out and relaunched to the world. And now, yeah, my job role has changed significantly as I'm sure you could probably imagine in that time. Matt Stauffer: Yeah, I can imagine. Wow. Well, I want to ask you a million questions about that, but I'm always supposed to start kind of a little earlier back. So I'm to go little earlier back. Joe Dixon: Yeah, sorry, I jumped ahead a little bit there. Matt Stauffer: No, no, no, that's exactly what I wanted. But I want to dive there and we got to go back. So prior to working full time with Laravel, what were you doing? And then what did it look like for you to kind of go through that process of like going from not working in Laravel? Because everyone's kind of got a different story of what that went like. So, yeah, what were you doing before and kind of what got you into the space where you were being considered and then what was the story of coming to work at Laravel? Joe Dixon: Okay, I'm not sure how far back to go, but I'll go relatively far back. So I went to university like relatively late. I think I started when I was 21 because I at the time didn't really know what I wanted to do. And I had a lecturer there who I got on really, really well with and he was the guy that happened to teach PHP. Matt Stauffer: Okay. Joe Dixon: And so I graduated from there and wasn't sure like what my role was going to be after that and he got me in touch with a company who basically were doing what was called in the UK, it's called a KTP, which is a knowledge transfer partnership. And it's basically a collaboration between a graduate and a company and an academic institution. So in this case, the university that I went to, and I kind of went in there very fresh. And the opportunity for me was to basically rebuild all of their kind of web facing properties. Matt Stauffer: Okay. Joe Dixon: And the technology we decided to use that at the time was expression engine. I don't know if Matt, if you were in the expression engine world at any point. Yeah. Matt Stauffer: I'm very familiar with the E. Yeah, 100%. That's how I got into CI and coding in the first place, so. Joe Dixon: Gotcha. Yeah, so I was like attending lots of expression engine based conferences because the this KTP came with like a, what'd call it? a budget for training. Yeah. And so I was using it and trying to absorb as much information as I could. And I happened to be at one talk at an expression engine conference where there was a guy, his name, I think it's Joel Bradbury. I've given him a shout out once on, on Twitter before, but like, I owe a lot to this guy because he gave a talk. Uh, wasn't even expression engine focus from my memory it was, he was building a Frankenstein app between a WordPress backend or kind of a WordPress backend. And he was like building writings Laravel on top of it. And that was the first time I'd ever heard the phrase Laravel and he was kind of really singing its praises. Matt Stauffer: Wow. Joe Dixon: And so I went away from there and kind of tried it out. And then I was hooked from there ever since. Matt Stauffer: That's awesome. Joe Dixon: And yeah, I forget which year that was now, but that's probably, that was probably like version 4.2, maybe 4.1, 4.2. And I went back and managed to persuade that company to basically, the expression engine was completely the wrong choice for what we needed to build at the time. I, you know, with a bit more experience under my belt, still not a huge amount, I kind of went back and persuaded them to rewrite everything in Laravel. And we had a really good experience doing that. And I stayed there for another few years before having the opportunity to go and work on a startup. So it was just me and two other people. And the, I did that for six years and the entire tech stack that we used there was also Laravel through that whole time. Matt Stauffer: Very cool. Joe Dixon: So that's kind of the backstory of how I got into Laravel and kind of you know, how much I was using it through, through those years. And then much like I was listening to Jess's interview and much like Jess, she said she responded to a tweet from Taylor. I didn't quite have the elaborate pitch that Jess and Tim put together. Matt Stauffer: Did anybody have that? Mm-hmm. Joe Dixon: I don't think so. That was pretty creative. But what I did do, and I've never talked to Taylor about this to know if it had any impact, but I remember like having a very, conversation with my wife about whether I should do this or not. Like I wanted to have a way to kind of stand out when I sent the email across. And so I titled the subject of the email I made, Here Be Dragons, as a kind of in-joke to just kind of like make that email stand out. And I have no idea if it had any bearing on what was to come, but I was much like Jess and thought I've got. Matt Stauffer: Uh-huh. Yeah. Joe Dixon: I've got no chance of getting this role, but I kind of owe it to myself. I just want to throw my hat in the ring and see what happens. And then after some back and forth with Taylor, yeah, it happened. So that was coming up on three years now that I've been at Laravel. And honestly, it's still like a dream come true every day. I still have to pinch myself that this is actually, actually happening. Matt Stauffer: I love that. That's so great. And it's so funny because I talked to people like you and Jess and I was like, well, I knew you guys were gonna get that job. And you know, sometimes Taylor would talk to me and say like, what do think about this person? But most of the time he wouldn't. He would already be able to assess it out on his own. Like he's very capable of assessing people's abilities. But then I would like learn that when he all got the job, I'm like, of course that person got the job. But y'all are like, I wasn't sure if I was going to get it. I was like, you could have just asked me because I knew you were going to get the freaking job. So anyway, well deserved. So when you first got started at Laravel, what were you working on? Joe Dixon: I went straight in on Vapor. So was working a lot with Nuno on Vapor. It was the two of us working on that. And I did that for, I did that ever up until the start of the Cloud project, I guess. So that was getting on for two years at the time that I'd been working on Vapor, which I loved. Like Vapor was a it was a huge silver bullet for me in my previous role, because we didn't have budget for DevOps role or anything like that. So it was all basically me trying to kind of wrangle these servers together. And I know like, Vapor for us was just the perfect fit. It was just the right time for us to be moving to a technology like that. And it just solved so many problems. And I never had to kind of think about it weekends anymore. If the server was gonna fall over, I didn't have to worry about that. It was just bulletproof for us. Matt Stauffer: It's amazing that that story right there definitely shows a little bit of why you were so effective in working in Cloud because I'm like that's literally the pitch for Cloud. So I'm like, yeah, I get it. I get it. Joe Dixon: Yeah, yeah, it's definitely scratching my own itch again. Yeah. And then I also so I guess Matt Stauffer: Okay. Joe Dixon: round about a year in, I know Taylor had wanted to work on a WebSockets package for Laravel and I kind of, we used to do internal pitches for projects that we wanted to work on. So I had an idea that I wanted to tackle that one and put the pitch in and that became what was Laravel Reverb. So it's kind of, yeah, mostly responsible for that as well, I guess. Matt Stauffer: I was just going to ask where Reverb fit into all of that. If somebody is not familiar with Reverb, I try to make sure that we don't spend too much time introducing basics. But if somebody's never heard of Reverb before, can you give us just the quick pitch for what it is and how it's different and similar to anything that came before? Joe Dixon: Yeah, so I think... So basically what Reverb does is allows you to add real time connectivity to your Laravel application. So rather than having to wait for a round hop trip back and forth through the server by doing something like polling, if you want, know, if you have, let's think of a good example, like a news ticker on the screen and that news ticker needs to update in real time. There's broadly a few approaches that you can take. One of which you might reach for is something like short polling where you're continually polling the server for updates to that state that you then rerender on the page. And so Reverb is a WebSocket implementation. And what WebSockets do is allow you to create one connection to the server. And then you can basically pipe information back and forth through that pipe between the client and server in real time without having to go through an HTTP handshake every time. And so just makes that, those updates really snappy and kind of instantaneous. It's not... The right solution for everything, but there's, ya know, in all of the apps at Laravel, so Forge and Vapor and Envoyer and now Cloud. I don't know about Nightwatch yet, but I'm sure they're doing some real-time stuff in there. I just don't know if it uses Reverb. Matt Stauffer: Yeah, gotta be, yeah. Joe Dixon: But everything, so anytime you hit deployment on your application, every update that you see on the screen there is happening via a WebSocket connection, which is all powered by Reverb. Matt Stauffer: That's great. That's very cool. And in the earliest days there were, you like when I wrote my book, we were using third party services for the vast majority of WebSocket stuff. And then I remember Terry Lavardur put together a wrapper around, I think it was a wrapper on a node service that was like Laravel WebSocket server. And just that blew my mind. I was like, you mean we can host our own WebSocket servers like ourselves? Joe Dixon: Yeah. Matt Stauffer: And it wasn't written in PHP. It wasn't a Laravel product. It was just the fact that he was able to show me how to do it like myself and I was like, wow, this is really cool. And so when I heard about Reverb, I was like, you are freaking, that sounds like black magic to me. You know what mean? Like it doesn't seem like something we can actually do. So that was, I'm very grateful for Reverb, not just because of how easy it makes things, but just to show. I think Reverb was the first kind of like, oh, we can do things in PHP that I'm not used to thinking that are viable for us to do. And one of the questions actually came from Tighten was, Did you have a lot of experience working with tools like ReactPHP and the other kind of underlying architectures prior to Reverb? So you just kind of learned it on the job. Joe Dixon: Yeah, my experience with WebSockets is basically adding a Pusher API key to my config file and everything magically happens. Matt Stauffer: Yeah, okay, like the rest of us. Joe Dixon: So I literally, I channeled my Aaron Francis and I printed off the whole spec for the WebSocket spec and read that cover to cover, which was, you know, equal parts enlightening and boring at the same time. Matt Stauffer: Yeah. Joe Dixon: And then I hadn't played with React PHP at all. But there was another community package from the Beyond Code team that had already kind of tackled the WebSockets problem. So I was able to learn a lot from that. I was able to sift through the documentation for React PHP and came together pretty quickly. It is a big mindset shift writing asynchronous PHP. It's a very different beast to tackle. But yeah, it was a lot of fun. Matt Stauffer: Yeah, I'm glad you mentioned the Beyond code. I meant to mention that when I talk about Terry's, but Terry's was the first one I saw. But yeah, Beyond code also definitely did the one of the early ones. OK, so let's talk about well, is there anything you wanted to get into before I start diving in Cloud? Because I know that cloud is not like the only thing you've done in your career. You know, there are any topics on mind other than Cloud of your time together. You know, I actually thought about Vapor for just a second, because I think that because Cloud is so new and exciting. The people who use vapor freaking love Vapor. Like there is, it is not going away. Same with Forge. People use Forge, freaking love Forge. As you were working on the Vapor team, don't, were you there at the beginning or was it, did you step into an already functioning saleable product? Joe Dixon: Yeah, it's already functioning. So I think I joined in 2022. That's right. So coming up three years now. And I think Vapor was launched, Laracon US in 2019. So it's kind of three years old at the point I joined. It was mature. Matt Stauffer: Okay. So it was was mature. So what did you learn? What was new for you working with Vabor? Because just because it was mature does not mean it was a tech stack that everybody knows. Joe Dixon: I knew because I've been using Vapor in a previous role, I knew what Vapor was and how it worked and the problems that it could solve. I think that kind of gave me a leg up when I came in and I kind of knew what was happening within AWS behind the scenes. One thing that... I'm sure probably isn't news to anybody, but like it was so nice to come into Laravel and just read the code base. And like, can immediately start working on it. And there was just no, you know, there was no learning curve to it all. It was just as you would expect a Laravel application to be. And so getting up and running was pretty easy. In terms of what I learned, I guess there was some, I don't remember specifics, but I guess there would have been some, some kind of design patterns in there that I hadn't used before. Matt Stauffer: Okay. Joe Dixon: which was super interesting. And I think, you know, some of those patterns have probably made their way through into Cloud as well, because Nuno and I both worked on Vapor and we kind of laid the foundations for Cloud in that respect. And I guess there was kind of, because this... Matt Stauffer: Got it. Joe Dixon: loads of different configuration options within Vapor. And I was using in my previous role, like a very specific set of those configuration options. So I guess there was a whole part of Vapor that I didn't have so much knowledge of. So, because we were doing kind of support at the same time. So I was having to like take support questions in from people and then try and figure out what they were doing. Matt Stauffer: You're like, I don't know. Joe Dixon: Yeah, exactly. So there was an element of that and finding my feet in the beginning, but you know, the team here are fantastic and you know, having Nuno there to kind of help me and guide me through those early days was invaluable. Matt Stauffer: That's awesome. Every time I've had to deal with AWS APIs, it has made me want to jump off a cliff. Was it well established enough that the core work on AWS, was it established enough that you didn't have to kind of really be struggling? Or did you also have to print out the documentation for AWS? You know, in order to do the work you're doing there. Joe Dixon: I don't think there's enough printer paper in the world to print out the AWS docs. Matt Stauffer: Yep. Yep. Joe Dixon: It was pretty well established, there's always, they're always throwing curve balls at you. Like I remember, I remember the, obviously Vapor has RDS as a database backing or database offering that you can just provision RDS instances. And part of that, there's there's a, like a certificate handshake between the two and I remember the certificates that we had in the vapor code base were going to expire and all of a sudden like customers were getting emails to say your certificate isn't going to work your certificate isn't going to work. Matt Stauffer: Oh no. Joe Dixon: and so like to figure out how to do that and not not force downtime on people was like incredibly challenging Matt Stauffer: Oh man. Joe Dixon: and I remember that was a deep rabbit hole into the docs and the kind of blog posts and all of the information that AWS put out in all the various different places they like to put it was was pretty challenging. Matt Stauffer: I can imagine, because I've had to do really rudimentary stuff like that when we were like, oh gosh, the Valet certificates originally expired after two years and now people are two years into their Valet timeline. And I was like, now what? But that was like individual people's individual machines. And it's like a developer machine. So you have some downtime or you have to run a command. It's like no big deal. Y'all like, and that's such an interesting thing that y'all have is that you're building developer tooling, but it is developer tooling that developers are using to provide expected, you know, Joe Dixon: Yeah. Matt Stauffer: perfect uptime or whatever, you know what I mean? Like there's a level of expectation for client happiness that of course most of us have for our end customers, but when we're building tools for developers, it's usually more like, it's open source. People will be understanding and people will be flexible. So y'all are building tools for developers that are customers. And I just think like that's such like an interesting, like, nah, you gotta, you you can't just tell them, you all got to SSH in and run this command or whatever. Like, no, you gotta handle it. Joe Dixon: Yeah, no, it's good fun. It was like there was an element of solving the problem, how we were going to roll it out, plus how we're going to communicate that with customers and try and do that with enough lead time that customers would then be able to do it and not, you know, have it, you know, appear on them unexpectedly. So I think we did a pretty good job of that in the end, but that was, yeah, that was a good one. And I remember another very specific one was we just get the odd bug report where customers would deploy their application and their app would then serve some cached assets from, because it uses Cloud front behind the scenes to serve those static assets. And like, it was so intermittent. It was one of those really intermittent ones. It was just like impossible to track down. And then again, I think one day it was like, I'm just gonna solve this problem. And just went deep, deep, deep into the docs again and managed to find some very obscure setting that we were able to change. And that whole problem just went away, which is pretty nice. So yeah. Matt Stauffer: yeah, it is nice. Joe Dixon: People who tell you to read the docs, they're not lying. Like it is a superpower to do that. Matt Stauffer: Yeah, it's very interesting for me because in order to, you know, keep up my blog originally, I would read all the Laravel source code and docs. And then in order to write my book, I thought I'd read everything in Laravel. And then I wrote my book and I was like, I know like a third of this code base. And so I really like, I read every line of code, you know, across multiple versions, stuff like that. And I was like, oh wow, I thought it was a docs reader. I thought it was a source code reader until, you know, and I had read through the entire docs multiple times with the reading the source code was a thing. And now as a CEO, I don't have time to like really have my fingers in every single piece. So I'm still very good at what I know. But when there are new things, I get OK at those. I'm not like very good at those. And thankfully, there's tons of people at Tighten that are very good at those. But it is just crazy to me how I'm like I became an expert in Laravel and I've got hundreds and thousands of code bases, stuff like that. And yet if there's a new thing where I have not read through all the docs and I have not read through the source code. I can be just as brain dead about the thing as somebody else and granted I can benefit from it works like this other tool I've used or whatever, but like each new thing is its own beast. And again, AWS is just like AWS is next level beasting. So I, I do not envy you for now being on a, at least project two, maybe more where you're living in AWS day to day. So let's, let's do that transition to Cloud. Joe Dixon: But before we do that, Matt, you just mentioned your book and I had to tell you I fell hook line and sinker for your April Fool's joke. Matt Stauffer: Oh no! Joe Dixon: I was like, what's he doing? Matt Stauffer: No, for those who don't know, posted an April Fool's joke that, and it wasn't my idea actually, a guy at Tighten, John Vanacorcy, was like, I think in October or September, he was like, Matt, I know that you're not an April Fool's guy, but like, I just had this idea that you take your book and you announce an audiobook version and the audiobook samples you just reading, know, colon,comma, space, PHP, space, function. And I was like, I gotta do it. So we set a reminder in Tighten Slack. I put the thing out and what I realized is one of the reasons that I am not a April Fools person is cause I felt so awful about every person who told me they believed it. And several of them were like, I got really excited. And I'm like, I'm the worst person ever. So I'm sorry, but also I'm glad you're laughing about it. So good, good, Okay. Joe Dixon: That's good, I laughed at the end, Matt Stauffer: So let's talk about Cloud. What was the transition to, you know, what was the process like of you kind of transition from Vapor to cloud? And when you got started on Cloud, what was your, forgot the word that they use, but like they're like, here's your operating directions. Here's your marching orders. What were your marching orders on day one of being assigned to the Cloud team? And were you the lead at the beginning or were you just one of the devs at the beginning and you kind of moved into lead as the team got bigger? Joe Dixon: Yeah, we just put like a small little skunk works together to begin with, I guess, to just kind of try and prove some concepts. The transition from Vapor to cloud was... Matt Stauffer: Mm-hmm. Joe Dixon: challenging in some ways because it was like context shifting a little bit. I wasn't like actively doing development on Vapor, but it was more about making sure customers were supported. So bug fixes if they came up or even just being in customer support and asking questions and then trying to get ahead into like a brand new project was pretty challenging to begin with. But we navigated that pretty quickly. We had support people, new support people that came on and kind of took some of that responsibility. So that was great. Matt Stauffer: Yeah. Joe Dixon: And I think I mentioned it was at the very beginning, was Chris Fidao came on to join the team. And then it was me and it was Nuno. We were kind of the first three that started work on the project together. And for us, like... We split the project into three key components, which we thought would be three key components. And to be fair, that has pretty much remained true the whole way through. So one that Chris tackled predominantly was that how we would actually run customers compute, how we'd run their workloads and what that would be on. And we went through several iterations. I've got like telegram messages from me and Chris talking about it. And the very beginning, we thought we would be using Lambda to do it. So basically taking what we learned from, from Vapor and trying to do that. Matt Stauffer: Okay. Joe Dixon: scale. But for various reasons, we decided that probably wasn't the best approach to take. And so then we ended up going in the kind of direction of Kubernetes where we've stayed and continued. And so that was one side of things. There would obviously be the web app side of things that we would need to an app that would basically customers can log into a UI so they can click around and build their applications and configure their applications and then finally deploy their applications. It was quite hard to get meaningful movement on that at the beginning because there were so many unknowns about, you know, even what the data model would look like. Matt Stauffer: Yeah, yeah. Joe Dixon: So we tried to pick off some low hanging fruit around, we're going to need users and we're going to need teams of some description. I mean, even some of those things we got wrong from the very beginning. But that was something that that was a work stream that we got underway with. And then the final part was the build service. So I mentioned from the UI, customer would click the deploy button. Well, we need something if we're to run that in a kind of Dockerized environment that we need something that's going to build that application. And so we basically took one part of that each. So Chris took the, the compute side of things. Nuno took the, web app side of things. And for some reason I ended up working on the build service, which is something I had like no clue of how to do at all, but that's kind of the kind of project that I enjoy is like just throwing myself in and trying to learn how to do it. And I have it on, I don't really go near that part of the code base anymore, but I have it on good authority that a lot of what I did still exists. So I'm pretty happy about that. Matt Stauffer: That's beautiful. I love that. I go ahead. Joe Dixon: Yeah, no, sorry, you go. Matt Stauffer: You're the guest. Joe Dixon: Okay. So that got us a long way, honestly. We made some really good headway into that and then we started to kind of grow the team around it. So Mohammed then joined the kind of App team still to this day the team is broadly split into two with like one unit, but we in terms of disciplines we have what we call our app team, which are basically all full stack web developers, Laravel developers. And then we have our infrastructure team, which is kind of a new discipline for us, something we haven't really had until we worked on Cloud. And so we started to grow that team a little bit as well. So then we had Justin and Florian join the team. And we brought in David Hill. So he came in as kind of head of design and he was very active in Cloud to get us to basically to LaraCon US where we were going to announce Cloud to the world. And then we brought in Jason Beggs as well because we were very good team of backend engineers, but we were lacking some kind of, you know, some of that front end stuff. And so Jason came in and yeah, we can't, that was kind of apologies if I've missed anybody else from the, you know, that era of the project, but that was basically the team that got us to basically build out the proof of concept and have it ready to announce to the world at Laracon last year. Matt Stauffer: Okay. It's fun when we do have these different disciplines and yet both Justin and Florian are full stack Laravel developers. They just happen to be very ops focused full stack Laravel developers. And all of y'all backend people are actually very capable front end people as well. But you're specialized more in the backend. So it's just, I really appreciate it. Cause I think a lot of companies think the best thing is to have entirely independent teams that don't know anything about what each other are doing. And I just watched the conflict that happens. And so my advice usually is try to avoid splitting those teams. But what y'all have done is found a way where you're like, we all could each do all of this, but it is not the best orientation of the way we work. And I'm like, I didn't realize it until you said this, but I'm like, that's kind of what we end up doing at Tighten. It's like, usually, you know, we split the responsibilities and one person has a little bit more experience in one way or a little bit more, you know, interest in one particular thing or something like that. Even if of course they can do both. So it's fun hearing how a bunch of full stack capable Laravel developers still found those delineations and kind of structured themselves along those ways. Joe Dixon: Yeah, it's been a lot of fun, honestly. Yeah, just different challenges, different challenges to that I've been what I've been used to. But it's, ya know, and you mentioned about the infrastructure, guys, they are just all rock stars. They all are incredibly good at infrastructure, but they're all they could jump in the web app code base and actually just carry on working on. And in fact, from time to time, we see the odd pull requests that comes across because they don't want to wait for us to do it. So they just do it themselves. Matt Stauffer: Uh-huh. Yep, totally makes sense. So you were doing build at the beginning. And so, you know, are you building Docker images for each deploy? Because I don't fully understand how Cloud actually does all of that. And what is the tooling you're using to build all that? Joe Dixon: Yeah, so I guess I can walk you through a little bit of what happens under the hood when you click that deploy button. Matt Stauffer: I would love that. Joe Dixon: So from the web app, you, yeah, if you click the deploy button in the UI, we basically pop a job onto a queue, use SQS under the hood. So we pop a job onto the queue. And then we have a whole bunch of internal tooling built on top of basically built to orchestrate Kubernetes under the hood. So first thing that happens is that tooling will pick the job up off the queue and it will grab and clone the customer's repository using a piece of software called BuildKit. And it will basically do that. It will clone that down and then it will CD into the correct directories and then run the customer's build commands that you can configure within the UI. Matt Stauffer: Yeah. Joe Dixon: And then basically that will turn that into a Docker image, which we then store inside a Docker repository. And we put a job back on the queue to let the web app know that that project has now been built. And there's a theory why we did that because we think that we might want to separate those two stages out one day from build and deployment. But right now, as soon as we've received the notification that that build has been completed successfully, we drop another job on the queue to say, go ahead and deploy it. And then again, that internal tooling will pick up the job off the queue and it will grab that image from ECR and it will take all of the customer's settings in terms of how they want their application compute configured. And it will turn that into a custom resource for Kubernetes and apply that. Matt Stauffer: Okay, got it. Joe Dixon: to the Kubernetes API and spin up that workload in the configuration that they want. Matt Stauffer: Okay. That is magical. And I trash talk Kubernetes often Joe Dixon: We still do. Matt Stauffer: because 98 % and also, also like I'm sure you do because you're practically using it you're unhappy with it. And that is where my origin criticism came from. But I think it's an incredible tool. It's just 98 % of people using it don't need it. And so it's really funny being like, no, this is what Kubernetes is for. These are the people who actually should be using Kubernetes. So it's really fun kind of learning about that system and have you set up. That's really cool. So when did you make the transition away from being in the code day to day to being like, now we got to be managing people and expectations and teams and everything? Was it a hard moment or was it a gradual thing? Joe Dixon: Yeah, I guess much like Jess's interview, there was a time when I think it was more or less at the same time, Jess was made team leader for the Nightwatch project. James team lead on core services. That's what we call like the... the Envoyer, Forge, Vapor, that suite of products. And then I was made team leader of the Cloud project. And I guess that, I don't remember exactly, but I guess that was probably a couple of months in that that was sort of made official. And I still stayed in the code. I still am in the code, but just like it's becoming increasingly less that I'm in the code now. So up until Laracon, I was in a lot still because we just had a whole bunch that we wanted to do. We still weren't up to, we still running relatively lean. In all honesty, we were pretty lean until February when we launched. But that served us really well because we were able to move really, really quickly during that kind of just under a year period to get Cloud from kind of zero to one. So I guess relatively recently, I guess towards the back end of last year, we started to actively grow the team. And obviously that made things change a little bit because there was a lot of hiring to do. And I'm sure you know that hiring is pretty time consuming. And then there's onboarding people and trying to figure out what the shape of that team should look like. And we've changed kind of workflows a little bit in between as well. So there was kind of a lot of logistical dancing to do. And I guess through that process, I've just got further away from the code, but I still really enjoy it. I still keep up to date with how things are structured. And still a lot of the decisions that I guess Matt Stauffer: Yes. Joe Dixon: predominantly me and Nuno made at the very beginning are still intact today. There's obviously new patterns that need to be established from time to time, but yeah, it's still, like I said, for Vapor, Cloud is a very easy code base to get up and running on. So yeah, hopefully we've done a good job there. Matt Stauffer: I love it. I mean, from the outside, looks like a good job. You know. Joe Dixon: Thanks. Matt Stauffer: All right. So two questions I had for you before we move on to just some of the one-off questions from other folks at Tighten is, was there one thing that was unexpectedly hard in building Cloud? And was there one thing that was unexpectedly easy? Joe Dixon: Ooh, that's a great question. I wasn't expecting the second part of that. There was plenty of unexpectedly hard things. And maybe they were, maybe that's unfair, maybe they were expectedly hard. think there's broadly two things which we were, well, one of which we were very unsure how we were gonna solve until very, actually very late in the whole build process. That links into the other thing I wanted to talk about. So talk about the other thing first, because it will lead into the, the expectedly hard problem. But billing has been something which is, I think we always knew was going to be challenging, but because we just have so many different things that we need to collect in order to charge people, it's incredibly complicated. and I got a shout out Dries here because Dries Vince has worked a lot on billing. Matt Stauffer: Yes, that makes sense. Yeah. Joe Dixon: on his own. I've tried to step in at times to kind of help him out when things have been really busy, but for the most part, he's kind of taken that project and run with it himself. And we use, we quite heavily use the event sourcing pattern for billing because, and it served us really well already, but basically because we have billing events that we need to collect on a regular basis from lots of different places, we can basically take those and dump them in an event store. Matt Stauffer: Mm-hmm. Joe Dixon: and then we can use them to do with whatever we want. And so those events are like the raw data, which we then roll up over time and turn into, you know, costs over time. Matt Stauffer: Actual statements. Yeah, yeah. Joe Dixon: So in itself, that was challenging because it's like a completely different code base to the web app itself. It's like it's a whole app in its own right. That was pretty challenging. But the single biggest problem with billing is... it always comes so late in the day. And this is probably the one thing I didn't expect is you can't build billing until everything else is built that billing needs to collect things from. So it's like, always the last part of the chain, always the last link in the chain to get built. And I'm sure you can imagine in the build up to launch, there was a lot of lot of moving parts that needed to come together to get everything ready. And so billing, yeah, billing was the last piece of the puzzle to really get done. But I think we it's Matt Stauffer: Yeah. Joe Dixon: it's worked fantastically well for us. If there's ever been an occasion where we've collected data and then rolled it up in one way, but we actually decide we need to roll it up in a different way because we have this event store. We can basically just replay those events and do whatever we want with them. And that's happened a couple of times and been really, really useful. Matt Stauffer: That's very cool. Joe Dixon: So billing, I think we always knew it would be challenging, but it's probably more challenging than we thought it would be. And part of that is because of the other thing that I wanted to talk about, and that was bandwidth monitoring. Like it's an incredibly complicated challenge to understand what every single app within a Kubernetes cluster is using, where that bandwidth is going, whether if it's coming in or out or whatever, it's complicated problem. And we spoke to... Matt Stauffer: Hmm. Joe Dixon: tens of vendors to try and solve this problem for us because we really didn't want to have to build our own thing. And this is kind of why it went as far down the road as it did. And we just hadn't solved this problem. We're in December at this point, and we still know that this is something that we need to tackle, bearing in mind that we launched what end of February. We didn't have a huge amount of time to solve this problem, but... Matt Stauffer: Yeah. Joe Dixon: Another shout out for doing things like attending conferences is that we went to AWS re-invent in December in Vegas. And which by the way is an absolutely wild place. It's like 75,000 people over three days or whatever it is. I've never seen anything so large scale in my life. Matt Stauffer: Yeah, I can imagine. Joe Dixon: But I was there with Justin from the infrastructure team. And we just happened to stumble into a talk that was by somebody from Netflix. And it was basically talking about them solving not the exact same problem, but problem along the same lines. It basically sparked some ideas in Justin's head and he then went away. I think he was probably hacking on it before he even left the hotel for the conference, but then he was working on it on the plane on the way home. And he basically wrote a proof of concept that actually allowed us to collect these bandwidth metrics that we needed. Matt Stauffer: Wow. Joe Dixon: He was busy working on other stuff at the time, so he basically handed that over to Sylvester, who was another member of the team that joined not long after Laracon US, it's sort of September time. And he was able to kind of take that and run with it and fully solve the problem. And it's been, yeah, it's been incredible to see that come to life. Matt Stauffer: I mean, it's funny because you say bandwidth monitoring. I'm like, yeah, I would have absolutely no idea how to do that. But I also wouldn't have known to look forward to it being an issue. And for those who don't know, if you get your Cloud bill, it's not like a here's your monthly cost. It is here's your base monthly cost, which is a fixed number. And then here's 17 other things because you used an hour of this and you use 20 minutes of this and you use 24 days of this or whatever, which is totally reasonable. But I'm like, that is a lot to be maintaining for every single instance. And then of course, rolling up to every single customer and rolling up to all these other things. So yeah, that's a lot. When you're doing the event sourcing, are you getting individual, you know, whatever two cent, six cent charges from AWS on a daily basis? And that's what you're aggregating. Joe Dixon: It depends. It's a great, it's another, it depends type answer, but it depends on where we're pulling that data from. So for instance, for bandwidth, we'll pull that from our own internal tooling because we're collecting that completely separately and actually. Matt Stauffer: Okay. Joe Dixon: I don't think to this point we pull anything directly from AWS because even for things like customers compute running, that's abstracted a level away from AWS because it runs in Kubernetes. We basically tracking having to track separately the hours of usage of each piece of, you know, each part of their cluster that they're using. So no, that's, but with like, it depends as well, because some services will allow us to pull that on an hourly basis, but others are only rolling up their own usage. So for instance, we, partner with Cloudflare for our two buckets. And their data, think, is only available once a day. So that particular metric, we only go and grab once a day, whereas others we can do on a slightly more granular level. So yeah, it's like, it's a beast. Yeah. Matt Stauffer: Okay, it depends. Yeah, I can imagine. I mean, but it is, like you said, this is a great case for event sourcing because you don't have to worry too much about, you you just got to get the data in there and then you can project it out, you know, in a way that makes sense when you need it. So. Joe Dixon: Right. And I've never been like team event sourcing. I always struggled to find something that I thought would be a good use case for it, but this was just like, yeah, breath of fresh air. Matt Stauffer: Use cases, yeah. Too good. I love that. That's awesome. Well, I've done my thing where I'm running late and I have more questions. So I'm going to stop on my questions and I'm going get to everybody else's questions. So one of the questions we had was, what is it like to build a safety mechanism on this large of a project to protect for things like customer spikes? Obviously, you have some level of spikes and ups and downs that you're expecting. The idea that, let's say, one of your customers who has a certain level of predictability suddenly gets, you I don't know what the dig effect is these days, but you know, some, you know, Reddit or whatever, all of a sudden they're just kind of blowing up. What does predicting for something like that or preparing or protecting for something like that look like in an environment like Cloud? Joe Dixon: Yeah, it's a great question. And it's not just viral traffic. Like we've already experienced the DDoS attacks from, two customers app. like that's another challenge that we have to have. And one of the decisions we made relatively early on was that we would partner with CloudFlare to do the networking side of things for us. So for those who don't know all the traffic that comes into Cloud will be routed through CloudFlare's network. And so they have some really good tooling around DDoS protection. And we're actually doing more work with them now to allow customers to kind of opt in to some of that stuff. So they will automate some DDoS protection. But we can also leverage things like rate limiting rules to allow customers to, I'm under attack. I want to, or I think I might be under attack or there's, you know, I'm expecting a spike in traffic over Christmas or whatever, and I want to make whatever that might be, we're going to allow them from the UI to be able to opt in to setting certain rate limiting based rules to help with that. Matt Stauffer: That's cool. Joe Dixon: But from there on out, there's limited amounts that we can do. So we partnered with Cloudflare to kind of take away that burden for us. But we just have to deal with a lot of scale to each of those individual Kubernetes clusters. Matt Stauffer: Yeah. Yeah. Joe Dixon: And yeah, I think so far things have been going well from that respect. Matt Stauffer: All right, got it. OK, let's see. What else do have? We had, what does the communication look like for pieces to your Laravel applications interacting with pieces that are outside of Laravel? Guillermo said, have they opted for gRPC for those pieces? And I don't even know enough about Cloud's architecture to know kind of like what we're talking about there. So I don't know if this is something where you... This is an easy answer for you, but are there a lot of communication, cross kind of systems communications that are happening where PHP has some kind of a layer of interaction with another system? Joe Dixon: I don't know if the question is directed at customers applications interfacing with other stuff or... Matt Stauffer: I think he'd likely means yours, like the ways that Cloud is defining, communicating to things internally, not customers. Joe Dixon: Yeah, I mean, we've tried to talk about the kind of split in the team between the app team and the infrastructure team. We all work together, but we've tried to kind of decouple those things as much as we possibly can. So all of the communication that I've talked about is mostly... Matt Stauffer: Uh-huh. Joe Dixon: we put jobs on the queue and infrastructure will pick those jobs up off the queue. There are a couple of internal APIs that just standard like REST based APIs where infrastructure can query an endpoint. And this is kind of for disaster recovery types scenarios. So if there's a real problem with the cluster and it goes away for whatever reason, that whole world can be rebuilt relatively quickly to make sure that customers workloads aren't just gonna disappear. Matt Stauffer: Yeah. Okay. Joe Dixon: But yeah, I think the short answer to that question is we've tried to keep those communication mediums to a minimum and basically just try to decouple those systems as much as we can to just rely on a couple of places where we actually have to communicate between the two. Matt Stauffer: I think queues are an underappreciated means of communicating between systems. Are all the same systems interacting with a queue all Laravel? Are there any of them in other different frameworks or languages? Joe Dixon: No, the infrastructure side uses quite a bit of Go. So the whole web app is PHP and Laravel. The rest is, and the billing app is Laravel and PHP. And the rest is mostly Go based. Matt Stauffer: OK, cool. Yeah, so I think we infrequently think of queued jobs and also other kind of key value stores and just kind of like data storage mediums as being actually a really effective way for sending messaging because it's, know, you don't necessarily, it doesn't have to be a push. It can just be a, we put it there and you pick it up whenever you want and you know what to do with it. There was a question about the consumer facing aspect here with the key value store. I think he was asking kind of like what's the background behind the key value store and then the MySQL that you guys built? Because I know Postgres was kind of the main thing. I think Postgres is from an external vendor. I don't know anything actually. This is fun. I tell I've been saying this more often lately, but I'm like normally I pretend like I don't know the answers to the questions. I actually have no idea. What's the story behind the key value store in the MySQL database that you all are using? Joe Dixon: Yeah, so you were right. Postgres, serverless Postgres is backed by Neon. And actually KVStore is backed by Upstash. So those two products are backed by third party vendors, but they are vendors that we... Matt Stauffer: Okay, got it. Joe Dixon: We did a lot of kind of calls, partnership based calls with a lot of different companies. Matt Stauffer: Yes. Joe Dixon: And these were two that were kind of standouts for us as partners who we thought would do a really good job for us and our customers. And so far, think it's been that they've both been incredibly good. So I'm happy with the decisions that we made there. The MySQL offering is something that we are hand rolling. And there was a lot of back and forth for the longest time we were gonna launch with serverless Postgres only. But we felt pretty strongly that MySQL is something that a lot of Laravel developers use. And we felt that we needed to come out with a MySQL offering. And we felt that we wanted to build our own implementation of it for better or worse. But yeah, I think it's just been something that we felt we wanted to have control of and we wanted to build something that we can kind of build over time and make something that's a really kind of core foundation of Laravel Cloud. So yeah, that gets us how those came about. So two kind of third party offerings and the last one is our own custom solution. Matt Stauffer: That's very cool. I wish I could have a whole podcast just asking about that MySQL setup because I'm so curious. Let's see. I think that's probably the last one I'm going to allow through because I we need to start wrapping it up. So I do want to ask you that. Have you been involved at all in the custom MySQL build or has it since it's a little bit later in the game, has it been something where you're more distant? Joe Dixon: I've had actually very little to do with it. So Chris Fidao, who people may probably know him more commonly, kind of made the proof of concept of my sequel and towards, guess, probably post-Laracon, he kind of broke some ground on that and made some progress on my sequel. And then a lot of the Infra team have now been involved in kind of taking his proof of concept on. And then... Matt Stauffer: Okay. Joe Dixon: think it's predominantly been Nuno who's worked on the web app side, the interface between the web app and the infrastructure side of MySQL, and probably a couple of the other app team as well, but I think predominantly Nuno has made most headway on that. Matt Stauffer: I gotta say, I've known Nuno since he was, you know, basically a kid in the Laravel world, but over the last six months, I've seen him really step into his like kind of streamer persona. So it is, I just keep getting reminded, I'm like, yeah, he's got a day job too. You every time you mentioned him coding, I'm like, yeah, I guess he actually codes too. Cause I'm like, yeah, he's a dev rel guy, you know, sits on streams all day long. So. Joe Dixon: No, he's a machine. Matt Stauffer: Love it. I love it. As a smart guy. Okay, so we now have officially hit our 45 minutes. So I want to wrap up by asking, is there anything that you wish we talked about any aspects of your journey or who you are or your time at time at Laravel, excuse me, or anything else that you want to make sure we get a chance to talk? Joe Dixon: No, I think we've covered a lot of ground. I think I've really enjoyed it. I hopefully did a good job of... Matt Stauffer: Me too. Joe Dixon: It's so much that we could talk about, like you said. Hopefully we got the important points out. But I think, yeah, I think we've a lot of ground. Matt Stauffer: Yes, we covered a lot and left a lot uncovered, which is my favorite way. We talked about this before we started recording. It's my favorite way to end a podcast because I'm like, I want to be so curious about you and what you do that I'm like, but there's so many more things to cover because that's that means it's a good conversation and I am there. I am I'm proud of myself for noticing that timer, but I wish it wasn't there. So well, Joe, thank you so much for hanging out with us. We will make sure that some of Joe's best contributions, including... Gosh, that Laracon talk with the reverb was one of the coolest things I've ever seen in my entire life. So we will make sure, I didn't even talk about it, but we will make sure that some of Joe's kind of like coolest moments are posted in the show notes so you all can kind of check it out and follow him and everything like that. And obviously we're gonna have to have you back. I have so many more things I wanna talk to you about. So we can look forward to, yeah. Joe Dixon: I want to talk about the drone now. I'd forgotten about the drone. Matt Stauffer: Well, let's talk about the drone for a second. Okay, so I know that you've been told this. But that was the most unexpected moment at a conference I've ever had. People have brought in mystery speakers and whatever. So can you like, first of all, just kind of give the pitch of like, what were you doing for people who didn't see it? And then, yeah, what's interesting to you about that? was fun about that? Joe Dixon: It was more terrifying than fun, I think. Matt Stauffer: Fair. Joe Dixon: I don't know. Like the Laracon talks have just been incredibly good and the bar keeps getting higher and higher and higher. And I had been asked by Taylor to give a talk. So was like, I really want to do a good job. And I thought, I've been working on reverb. a reverb was fresh in my mind. It probably hadn't even been out that long at the time. So it was going to be a Reverb talk. That was always the premise. And I thought, what can I do that's like... Matt Stauffer: Yeah. Matt Stauffer: don't think so. Joe Dixon: a little bit different than building a chat app on the stage or something. Matt Stauffer: A little bit, yeah. Joe Dixon: And so I just had this crazy idea, what if I could just buy a cheap drone, like a drone that I could, like a programmable drone, basically, I thought, what if I could do that? And I found this like $100 drone that was like, I think it was meant for teaching school kids or whatever. It was very cheap. It was very lightweight. It was ugly. The worst part about it was it's so small, I think it got drowned a little bit on the stage. But anyway, Matt Stauffer: Okay. Joe Dixon: The whole premise of the talk was what if I can control a drone via web sockets powered by Laravel Reverb? And so I bought this drone and I sat here one weekend and I spent an hour just like knocking up a proof of concept and I got the drone to fly and I was like, right, that's it. I have to do this now. So I pitched it to Taylor and Taylor, classic Taylor was like, yeah, pretty cool. And so I was like, all right, that's enough. That's exciting enough that I'll do this. And so I did it and I like made the talk and I was practicing and stuff at home in my kitchen area and I would practice it and sometimes it would fly that the drone would fly really well and then other times it would just like fly and just float off. It's not anything to do with my controls but like the sensors in the drone just weren't working well and I figured out eventually like if I turned the kitchen lights on full it was absolutely fine because it could like detect the pattern on the floor and hover as it's supposed to. So that was like the first sign of Matt Stauffer: The air movements. Oh really? Huh. Joe Dixon: I have no idea what the stage is going to be like. Is this even going to work when I get up on the stage? Then I wasn't sure if I was allowed to even fly with the drone. So I had to get it to the States and I was a bit worried. Well, what happens if it gets taken from me at border control? And so I had to, it's a whole video recorded, but I was like, it would be really real shame if I had to like bail at the end of the talk and play a video. Matt Stauffer: Show the video version. Yeah. Joe Dixon: So I got it there and that was all fine. They let it through customs fine. And then it got to the day before the conference and it was the speakers tech checks. And I had basically wanted to put the drone on a box on the stage and fly it. So nobody knew what it was. Like, so the whole first of my part of my talk was very technical. Nobody knew what was gonna happen. I was gonna walk on the stage and just put this box on the stage. So I wanted to test it during the tech check. And I think I was the very last person to go. And there was all these like rock stars from the Laravel community, like on the stage. And this was going to be the first time that I tested this drone out on the stage. And like, that was almost more terrifying than the room full of 900 people. Matt Stauffer: Well actually... Joe Dixon: But I took it, the lights were really good on the stage. So I took off and I don't think anybody on the stage was expecting it to happen. And like, I got this round of applause from all of the speakers on the stage. And that was like, that was a moment where I thought, oh this, this talk might go down well. Matt Stauffer: Yeah. Joe Dixon: So yeah, all of the stars essentially aligned, but it could have been a very different story where it kind of fell flat on my face a little bit. Matt Stauffer: Yeah, well, I literally like was sitting in the back at the Tighten sponsor booth, texting my brothers being like, you will not believe what's happening at Laracon right now. And they're programmers, but like they don't really care about Laravel stuff. But I was like, they are going to care about this because this is amazing. I love it. I was that was such a clever and fun idea and also such a great demonstration of what WebSockets can do, because I mean, I gave a talk years ago about, hey, you can use your Laravel app to to control lights in the background, right? And that's a single command sent over an API. I even think I was using if this, then that to make it happen, because there was no way to do it directly. And just the fact that I could interact with physical devices at all blew my mind. But the idea of using that kind of real-time communication, it just feels so outside of what we normally do. And I'm just praying for more. I know it's just a fun demonstration. But I'm just looking forward to like the really cool thing that happens in the future that that looks back at that talk as the reference of somebody being inspired saying, well, when I saw him with the drone, I knew I could whatever. So I love it. Joe Dixon: That would be pretty wild, yeah. Yeah, that was a lot of fun. Very fond memories of that, once it was over. Matt Stauffer: Yeah, I mean, as an audience member, I have very fond memories of that. So I can't imagine as the creator what it was like. So, well, thanks for bringing that up. We will link that video in the show notes. And yeah, man, it's just been such a good time having you here. Thank you for being here today. Thank you for sharing so much with us. And I look forward to having you back again soon. Joe Dixon: Thank you for having me. Honestly, I think I said to you in the build up to this, that this is like a bucket list for me. So I'm very grateful to have been invited. So thank you very much. Matt Stauffer: I appreciate your humility because I'm like dude. Joe, you are a rock star I need you to you know but but whatever you know what if you if this is what you got to do to stay humble do what you got to do stay humble but in my mind you're a rock star. So well thanks Joe and for the rest of you thanks for hanging out with us and we'll see you next time Joe Dixon: Thank you. Appreciate it.