Latest / The Laravel Podcast / How We Built Nightwatch, with Jess Archer
Transcript
- Matt Stauffer: Welcome back to Laravel podcast, season seven. I'm your host, Matt Stauffer, CEO of Tighten. And in this season, I'll be joined every episode by a member of the Laravel team. And today I'm talking to my old dear friend, Jess Archer, the Engineering Team Lead of the APAC team at Laravel. should have written that down, but I think I got it right. So Jess, whether or not I got that right, can you say hi and talk a little bit about what you do at Laravel day to day? Jess Archer: Yes. Hello. That was perfect by the way. Matt Stauffer: Yes! Jess Archer: So my name is Jess Archer and I'm from Brisbane, Australia, Engineering Team Lead on the Nightwatch slash APAC team, which is the Australia Pacific. And yeah, what do I do at Laravel at the moment building Nightwatch? I've kind of forgotten the rest of the question. So if you want to go ahead and... Matt Stauffer: That's totally fine. That's it. You build Nightwatch. I'll walk through the rest of it. It's so funny because when I say your name, it's hard for me not to say it in your accent because I just like it's Jess Archer, you know, like that's the way it should be. But I, you know, it's not respectful to, to mock somebody's accent. So, but in my head it is not Jess Archer. I do not hear it in English. I hear it or an American accent. I hear it in Australian accent. So there you go. Jess Archer: I love that. Matt Stauffer: Okay, so the first thing I wanted to check in with which I do with everybody else is what is the story of you joining Laravel? You know where were you beforehand? What were you doing kind of and what interesting things happened between you know? I don't work at Laravel, and I didn't even know that was a possibility to oh my god. I'm at my first day at Laravel. Jess Archer: Sure. So, I mean, I'd seen kind of Taylor put out, you know, tweets every now and then looking for people. And it was always that thing of like, there's no way, like he'd never pick me kind of thing. So I never even entertain the idea. Matt Stauffer: Yeah, right. Jess Archer: But I'd been working with Laravel, you know, like using the framework for, I don't know, maybe eight years, nine years at that point. and I'd been, you know, speaking at Laracon's kind of getting involved with the community, doing podcasts, all that sort of stuff. And I'd met Taylor at a Laracon once and kind of awkwardly said hi and chatted for a little bit and I'd done some PLs to the framework as well. So one time he did put out, you know, a, a job ad, on Twitter and I thought, you know, why not? Let's just give it a go. Matt Stauffer: Yeah. Jess Archer: So I was working, at a company writing, medical software for like the health industry. And I was working with my very good friend, Tim McDonald. And so we both kind of were like, wouldn't that be cool? What if we, what if we did like a joint job application? Because we kind of, I think we knew, I don't know whether Taylor had said it or whether we just had a kind of feeling that our time zone would be problematic. And so we kind of thought if we came as like a bundled, offer that there wouldn't be some of the, like the main time zone concerns of that person just being isolated on their own. So it was like, okay, if we, if we both apply and we both get it, then we won't have the time zone problem and we'll get to keep working together. Cause we adore working together. So. We, we hatched this, this kind of crazy plan where we, we wrote this script, right? That we were going to send to Taylor, but it was going to be like, he was part of just a chat between Tim and I. So it was a, it was a, like a conversation where it was like, Hey Tim, did you see that Laravel was hiring? And then he'd go, yeah. And then we kind of like basically were each other's references. So he was like, you know, you'd be great at doing this because you've got all these skills. And I'd like, yeah. And you'd be awesome working, you know, on Forge because of all these skills. And we kind of addressed the time zone thing, like, you know, Matt Stauffer: Uh-huh. Jess Archer: We'll be working together. So we won't be alone. All those sorts of things. So was this long conversation. We did it at the, the middle of the night, I think for Taylor, so that he wouldn't kind of wake up as it was happening. He'd just wake up with this like message of backwards and forwards between Tim and I. And yeah, one day we just, we just sit, enter, enter, send it all. And, I think Taylor, I think it was just like a grin emoji and what are your email, what are your email addresses was the, was the response to that. And then, we kind of. Yeah. Backwards and forwards with Taylor from that. So that's, that's the story of how Tim and I both did a joint job application and started working at Laravel. Matt Stauffer: I love that. That's wonderful. And I told Mohammed this and it's also true for the two of you. I talked to Taylor when he was, you know, like looking through your application and I was like, please, please dear God hire these people because I would love to have them work for me. There's and with Mohammed, I just told him, I was like, hey, if you don't hire Mohammed, I'm gonna hire Mohammed. With you guys, I was like, I wanted to hire both of them, but I just can't figure out the time zone. But if you have both of them, it's this brilliant blah, blah, blah, blah. And I was just like, please dear God. And it's amazing to me because the fact that the two of you are two of the independently most brilliant people I've ever met in my entire life and to the most creative and innovative and amazing and kind and wonderful people in the Laravel community. But the two of you together, the package, it feels unfair to drop on somebody's lap like that level of talent and ability. So anyway, thank you both for having that idea because the benefit we have all gotten from it is just immeasurable. So, okay. Jess Archer: Thank you. It's, yeah, definitely. I still have days I wake up and I'm like, I work at Laravel. Like it's still kind of not, not real, which is really weird. Matt Stauffer: Is this real? Yeah. I love that. Well, you absolutely belong. And one thing I want to ask everybody is, on day one, I don't think you were working on Nightwatch on day one. So what did you actually do when you got started? Jess Archer: Uh, so Taylor and I kind of talked about a few ideas, um, before I even started of what sort of stuff I'd be working on. He kind of, he liked the idea that I'd be kind of doing like R and D skunk works kind of stuff, um, in the open source area. So the very first thing was building the Vite, um, plugin for Laravel. So that was my, my first kind of official project wasn't even like a PHP project. Matt Stauffer: I mean, you went deep on that, yeah. Jess Archer: Super deep. mean, there is PHP components in the framework. That also kind of wired all up and make it work. But, yeah, that was to, you know, replace web pack, with Vite that was just showing so much promise of being like so much faster, all these good things, and just trying to figure out like how to create that really nice developer experience that, Laravel mix had, but with, with Vite and we kind of decided that we didn't want to do like a wrapper like mix was because it kind of makes it harder to like use other Vite stuff. So we kind of thought, let's just build it as a plugin. Matt Stauffer: Uh-huh. Yes. Jess Archer: so that you can just use, can consume Vite directly and then have this other thing fit in. So yeah. Matt Stauffer: Very cool. Did you go, I, and one of the questions, so for those who don't know, I asked on Twitter, blue sky, does anybody have any questions for Jess? And one of the questions was what was the process of working? And the person who asked, they actually said, what was it like going from prompts to Nightwatch? So I don't actually know what the story was, whether prompts was officially a part of your assignments there, but regardless, I know that you were doing kind of like open sourcey stuff and then you moved to a product. Was that a big kind of cultural, what am I doing day-to-day shift for you or did it feel like a really natural and easy movement? Jess Archer: It was definitely a big kind of, a big shift. I'd been doing a lot of, yeah, like all open source stuff, a lot of stuff I was doing in collaboration with Tim. So Tim helped me with vote plugin. We built Laravel Pulse together. prompts was kind of my, my like baby that I built kind of on my own. That was one of the few ideas that was actually my idea. Most of the great ideas are Taylor saying, Hey, can you explore this idea? But, prompts was, was something that I just desperately wanted to solve. Matt Stauffer: So grateful for prompts, by the way. I freaking love that tool. Jess Archer: It's yeah, I, I'm still very proud of, of prompts. I love it very much. So yeah, then switching to kind of more producty stuff. I mean, it's where I came from, like all my other kind of jobs had been more in that space. So it wasn't unusual. But it was definitely like a big shift. Matt Stauffer: Yeah. Okay. Jess Archer: A lot of the way we work is still the same, right? Like we're still kind of thinking of developer experience or building tools for developers. So it's the same sorts of, the same sorts of goals. We probably go put maybe a little bit more polish and effort in. I mean, we always do our best, but we have, I guess, more time to go even further, with, with polish and all those sorts of things. I guess we kind of have to come out of the gate with something that's like. Matt Stauffer: Yeah, okay. Jess Archer: really strong, whereas sometimes with the open store stuff, you can come out with like something that's like really good and then just wait for like the community to kind of get involved and help steer it, all those sorts of things. Whereas I feel like with a product, you kind of need to, yeah, just come out that little bit stronger, the little bit more opinionated. yeah. Matt Stauffer: You also don't have the ability to expect any contribution because since it's a paid product, people aren't gonna get access to the code there. If anybody's gonna make any fixes afterwards, it's you. But even the best intentioned person has a little bit more entitlement to expectation of excellence when I'm like, hey, if I'm paying you however much a month, it's just gonna feel different than you gave me this free thing and now I can help you make it better. So I can see that there's just like a different cost to get to launch there. Jess Archer: Yeah, it's a different dynamic. Matt Stauffer: Yeah, so we've talked, we've said Nightwatch a couple times and I hope anybody listening to the podcast is familiar with Nightwatch, but let's just off the bat, let's just say what is Nightwatch and what's it kind of like the timeline of launching and kind of where is it in its journey towards being live right now? Jess Archer: Sure. So I like to think of Nightwatch as almost like three services rolled into one. So it's exception tracking, it's performance monitoring and it's logging. So all of those three things combined, they're all very interrelated with each other. So it kind of is really nice to have them all linked together. So effectively it's something that you install in your Laravel project and it will send all these kind of telemetry data to the Nightwatch service. And then you can view that information in your dashboard. So you can see any errors you're getting, any performance issues, slow routes, slow queries, slow jobs, all the things. And then also any logs that you've kind of emit in any of those processes. It's all linked together. So if you start zoomed out at, you know, there's, you can see all of the requests that happen. You can kind of see on a graph, there was like a patch of errors there. You can zoom in on that graph, go see which routes were causing it, click on the route. Then click on an individual request. So you can get down to like the individual execution level. And then yeah, if there was a log, it kind of shows up straight on that page. You don't have to go and like map it up with any other systems. It's all right there. So yeah, that's, that's Nightwatch. Matt Stauffer: Yeah. And we, yeah, we've done a podcast episode, at Laracon Australia talking about kind of the basics and kind of what I'm excited about some of the aspects of how you all are surfacing errors and stuff. So for those who aren't, who have not listened to that and were curious about it, there's that podcast episode, I'l link in the show notes. also did a demo, recently, and I think you spoke it was at Laracon India. You spoke somewhere recently introducing this. So you've done a lot of Nightwatch content. Jess Archer: Laracon Europe. Yep. Matt Stauffer: You're up, okay. So we will link all those in the show notes. And so because we've already kind of covered that, I actually kind of focused a little bit more today on the developers asking you about the experience of working on Nightwatch versus, what is Nightwatch specifically? But before we get there, I just had one thing I wanted to talk about. Because this season is about like, hey, get to know the Laravel team, you mentioned that you are the lead of the APAC or Nightwatch team. And I wanted to see if you could kind of just give us a quick introduction to like... What is that team and who is it outside of you and Tim and you know, it is the Nightwatch team now. But do you think that there's you you'll be assigned to projects together in the future Just kind of give us the intro to the APAC team in general. Jess Archer: Sure. So there's currently seven of us on the, the Nightwash team. So yeah, we're getting to a decent size. I mean, when, like before we kind of started all this, there was like 10 people in the entire company and now we've got seven on this one team. So, there is our product manager, Phil Horton. There's our designer, Jeremy, there is our infrastructure, James Carpenter. Who else we've got? Luta, who's doing development and Tim. And then also Sabrina has recently joined us as well as a developer. I hope I remember everybody. I hope that was 7. Matt Stauffer: Okay. Is Philip on the team too? Jess Archer: Yeah, he was the first one I mentioned. Matt Stauffer: I somehow I missed him. Sorry, Phil. Cool. Yeah. And I mean, I realized when I asked you, was like, man, you you asked me to list every single person at Tighten and I'll probably give you half people who don't don't even work here anymore. I'm just like, that's a stressful thing to put people on. So sorry for putting you on the spot. But yes, it's very cool to me because when I originally think about that team, of course, I think of, you know, you and Tim. And then I think about, yeah, you know, a couple of other people have joined. But what you just named is not just like a couple rogue developers like doing their own little thing. It is a fully functional team with all the different components of, you know, the backend and the front end, the design and the project management and product and everything like that. So that's very cool to me that you all have been able to build it in that way. How much have you been involved in hiring versus how much are you just sort of like kind of throwing people Taylor's way and then if he brings them on the team, you say yay. Jess Archer: I think like when we kind of, when it was just Tim and I, I wasn't involved with, the kind of third person, was, Lyota Hamasaki. So I got to interview him. But I wasn't kind of involved in like the job description, that kind of stage or anything. It was just kind of like, I think Andre had already interviewed him at that point and kind of said, Hey, go interview this guy. And we adored him. After that, I was a lot more involved. I think that was kind of when I was promoted to team lead and was kind of around that same sort of time. So since then I've been a lot more involved kind of doing the first interviews, helping out with like the position descriptions. Funnily enough, like there's a lot of people we've hired of people that we've worked with before that we kind of already knew. So yeah, I mean, it's always nice when you can you know, when you have people that you work so well together with that you can kind of reach out and go like, Hey, are interested? Um, so yeah. Matt Stauffer: Yeah. How do you enjoy taking kind of the leadership role? I mean, we know you know how to code. What's it like being a people manager leader, not just a code leader? Jess Archer: It's pretty interesting. It's something I always resisted. I remember. Yeah. For, for a very long time, it was always something that people had kind of wanted me to go towards management sorts of things for whatever reason. And I always resisted it because I just wanted to code. But when it happened at Laravel, I was like, if I was going to kind of go into more of those senior sorts of roles, Laravel would be the place where I would probably actually quite enjoy it. Matt Stauffer: Yeah. Jess Archer: And because my team is all incredibly talented, they're all very kind of self-guided and self-directed. There's no, like, I'm kind of, you know, the lead in name, but in reality, we're all just still working very collaboratively. I just have that as like, as like an official thing, but it's yeah, we're all, I look at us as all, leading the team. So I still get to do a lot of code. Matt Stauffer: Great. Jess Archer: But I get to do like one-on-ones with folks, which is really nice. I love having chats with my team, kind of one-on-one. Yeah, it's, it's been good. I think it's probably different than most other companies, the way they would do it potentially. But I like that at Laravel, we're allowed to kind of operate our teams how we want to. So each team can kind of work how they want to work. So yeah, we kind of just kept doing what we were doing really, and just having fun collaborating, even like the design of the software, like because we're building tools for developers, we get to just ask ourselves, what do we think would be cool? Most other places I've worked, it's been building software, like, you know, it's working in health. I have to ask other people, like what, what's good here? Like I need to talk to doctors to say like, what should this do? Whereas when you're building tools for developers and you are a developer, you go, I know what's cool. Matt Stauffer: That's very cool. And at very worst, you can ask your friends, right? And they're like, hey, what do you think about this tool? Jess Archer: Exactly. Matt Stauffer: Yeah, that's awesome. Okay, so moving on to the questions that we got from folks on the internet. This is a no particular order. I ordered them a little bit, but basically no particular order. Maybe prioritized by the ones I'm curious about as well. How many, if any, PRs to the framework were required in order to make Nightwatch functional? Jess Archer: To make it functional, can fairly confidently say zero. Matt Stauffer: Okay. But... Jess Archer: We deliberately wanted to support at least Laravel 10. So that kind of meant working with what Laravel 10 had. That's not to say that we won't potentially push some improvements all the way back to 10. There are some, I mean...Nightwatch is listening for events. it's as long as there's an event that we can listen to it and it has information on it. We can capture it. And Laravel has had great events for such a long time. we built Laravel Pulse, which was all based on those events. So we kind of knew the events we already wanted. So we didn't have to add too much. There have been improvements to some of the events, some new events that, aren't in old versions. We may go and backport those. But they're not kind of deal breaking things if they're not there. We have other ways to kind of achieve things. Matt Stauffer: Got it. Okay, very cool. All right, the next one is what was the hardest problem you had to solve and what was your process like in solving it? Jess Archer: I mean, the hardest problem has got to be dealing with that amount of data. Just, you know, gigabytes, terabytes, petabytes, billions and billions of rows in databases. It's yeah, hands down the absolute biggest challenge. And the process. So, I mean, a lot of experimenting, a lot of research. Tim and I went deep reading like academic papers for a while because we were trying to find like, like different algorithms for aggregating and pre-aggregating data and doing like rolling aggregation where you kind of incorporate new data into an aggregate without having to like store the whole aggregate, which is easy for things like averages, an easy for like min and max, but really hard for, like quantiles like P95s or those sorts of things. Matt Stauffer: Wow. Oh yeah. Jess Archer: So we kind of used a lot of the knowledge we got from Pulse, but then, exposed ourselves to whatever other tools are out there. So that's when we learned about, like OLAP databases, which is online analytical processing, as opposed to OLTP database, which is online transactional processing. Those were terms I'd never heard before. And I kind of wish I knew them earlier, cause then I would have known what to like to search, right? But OLTP databases, that's the name for the databases that we all use, you know, MySQL, Postgres, those transactional databases, but there's this entire other class of databases that's designed for doing analytical stuff. And it's not that great at doing the transactional things, because it's like a trade-off thing, right? You've got to, if you go make one thing work really good, then there's sacrifices in the other. So learning about these other types of databases, playing with them, evaluating a bunch of them, trying to put as much data through them as we could. Just, yeah, a ton of proof of concepts. Matt Stauffer: Yeah. Yeah. And you did give a whole talk about some of the kind of aspects of the solutions you came up with. So we will make sure that is also linked in the show notes. And that was Laracon US last year, right? Okay. Jess Archer: That was kind of like a, almost a behind the scenes of Nightwatch before we'd even announced Nightwatch. Because I was just so excited about this database Clickhouse. It's all I was thinking about. It was just running through my head. So when it was, you know, when I was thinking of a talk, it was like, this is the only thing I care about right now. I want to talk, I want to tell the whole world about this database. Matt Stauffer: Yes. Yes, I love it. We'll link that in the show notes for anybody who has not heard it. It's a wonderful kind of journey into that key component of what solving it looked like. But you did just name something that references one of the other questions that we got, which was, how do you simulate load to validate elements of the design and the architecture? Jess Archer: That's a great question. So data goes through various kind of systems to make its way into the database. So we had to, I guess, like test some of those components in isolation first. Testing the database, I wrote some, you know, some scripts that would basically generate a ton of realistic looking random data. It was kind of important that like the data wasn't completely random. Matt Stauffer: Hmm, okay. Jess Archer: If you're generating requests and like they're storing the durations, if you just choose a random number between, you know, one and a thousand for all of them, then the distribution is not actually realistic. The majority of requests will kind of be more towards one end. So creating things that are as realistic as possible because different, algorithms will handle that stuff differently, right? Like a P95 algorithm, some of them work better when the data is more towards one side versus being like super distributed. So yeah. Matt Stauffer: Oh interesting. Jess Archer: putting tons and tons of data in databases was probably the biggest, the biggest thing we wanted to test. We were fairly confident we could get the data in like that just kind of comes in in this constant stream. We played with a few different technologies for that. We're using a lot of AWS stuff. So we played with, I think, Kinesis streams, but then we ended up going with Kafka. And then yeah, we've done a whole lot of load testing on Kafka in isolation, again, using as realistic as possible payloads to try and figure out how much a particular instance size can, can handle for our type of data. So yeah, just testing all those things. And then probably the biggest thing was actually real world testing, which was installing Nightwatch on Laravel Forge and Laravel Forge. I think we have about three and a half billion records in a single month of just database queries that Forge does. So Forge is very database heavy. yeah, millions and millions of requests. So that. Matt Stauffer: Yeah, okay. Yeah. Jess Archer: put a lot of data through the system and we were able to see that for what we'd kind of chosen, like that was just basically like a blip in terms of like the Kafka resources we chose. We did find that Forge had more data than we kind of had tested, because we kind of, we had some assumptions about Forge that were not a hundred percent correct. We actually learned a lot about Forge once we had Nightwatch on there and it's a lot, a, does a lot more than we thought it did in terms of queries and cache events and all these sorts of things. Matt Stauffer: Amazing. Jess Archer: So we ran into performance issues and we had to solve those, come up with new aggregation strategies. So yeah, that was kind of the process. Matt Stauffer: Okay. Well, and you touched on yet another of the questions. This was asked by several people, which is how are you using Kafka? I'm embarrassed to say I'm only mildly familiar. I know Kafka is an Apache thing that has to do with streaming, managing streaming data, but that's as far as I know. Could you give us like a real quick intro of what its role is in a tool like Nightwatch? Jess Archer: Absolutely. So, I mean, I didn't know anything about Kafka either before building Nightwatch. so you're not alone there. Don't worry. Matt Stauffer: I feel better. Thank you. Jess Archer: So it's yeah, it's like a data streaming, service. You basically put messages onto it, which are put on by things called producers, and then you have other things consuming from it called consumers. So it's really just kind of like a message bus, put things on, pull them off. People use it for different things. You can use it for like event broadcasting where you can have multiple consumers Matt Stauffer: Okay. Jess Archer: listening for events. So if you had like a microservice architecture and a thing that, you know, an event happens here, all these consumers can read it. In our case, it's basically just one producer and one consumer. It's just going through there and then into the database. But it does, it helps us in a lot of ways. So one really handy way is if we need to make any changes to the database schema, we can just pause the ingest and the data will build up in Kafka. We can do whatever we want in the database. Matt Stauffer: Very cool. Yeah. Wow. Jess Archer: And then just hit resume and it'll just slurp up everything that was kind of waiting there. So it acts as like this little queue that lets us like basically pull the pipe apart, do some work and then connect it back together again. Matt Stauffer: That's incredible. Jess Archer: So that's really handy. So Kafka has a concept of topics. So each message goes on different topics. So for each different type of metric we collect, they all go into different topics. Those topics go into different tables in the database. yeah. And it also helps us insert the data because the database that we're using, it works best when you insert like bulk data. So you don't just insert each row as like an individual insert. It wants to get like chunks of data at a time. So having it all go onto this stream means that it can just read out like Matt Stauffer: Well, okay. Jess Archer: you know, five seconds worth of data at a time and insert it. And, know, that's potentially millions of rows in a single insert. It much prefers that than, than one by one. Matt Stauffer: Wow, which is kind of the opposite of how you traditionally think with our databases. You're like, no, what we don't want to do is queue up massive amounts of work and then it's going to gum up all the works. It's like, no, this is actually how it likes to. So this is a lot of different stuff from your average Laravel application. That's very cool. Jess Archer: Definitely, yeah. Matt Stauffer: OK. Let's see. Next question. OK. I'm curious whether you think that the premise of this question is actually valid. It says, how do you design for a virtually unbounded scale? And so you were kind of saying like, you well, we thought Laravel or Forge are going to be the certain size. And, know, obviously you have not taken on the entirety of the clientele that you're going to eventually have, but that is a, know, how much was what you're building, you know, like we need something that can scale perfectly and how much what you're building is we're just going to throw something really powerful at it. And if we see it starting to hit its limits, we're going to make it more powerful. Like how much of this the auto magic auto scaling versus the, you know, just the heavily overpowered, you know, service. Jess Archer: It's, it's an interesting question. I mean, the scaling aspect is the thing that's kind of the most scary thing, right? Like we don't know how many people are going to sign up. We think it's going to be very popular. We don't know how big people's apps are going to be, which is kind of the bigger question as well. If everyone that signs up has, you know, a fairly small app, that's one thing, but if everyone that signs up has these apps that are bigger than we kind of anticipated, then that's something else. So. Matt Stauffer: Yeah. Jess Archer: We wanted to make sure that, like one big consideration with the database is to make sure that other customers data doesn't impact the performance of like another customer, like the noisy neighbor type thing, right? So the database structure is designed so that when you're querying your data, the other data that's in there doesn't, it doesn't have to like scan through any of that data to get to your data. So there's no kind of penalty. Matt Stauffer: Got it. Okay. Jess Archer: for a big customer being on there, the small customers don't pay performance penalties because of someone else on there that's putting a lot of, a lot of load there. We kind of know for Kafka, how we, how we can scale that, how we can add more, replicas and instances and all those sorts of things. We can add extra databases. One of the kind of great things with the way that it's architected is we can, if we wanted to create a dedicated instance for one customer and the dashboard application can say, okay, for this customer, I'm going to talk to that database over there, but these customers, I'll talk to this one. So it lets us kind of scale horizontally and vertically, right? Like that's kind of the terminology we can scale in both directions as needed. And yeah, we kind of have a strategy for that and we have all the monitoring tools in place so we can see when we need to make those decisions and get early warning when those things are going to happen. And going into early access, just getting more and more customers on there just lets us test some of our assumptions because so much of what we tested was kind of like, know, in laboratory conditions, right? Whereas when you deal with real things, like even just the spikes, Forge is really interesting because you can look at its graphs and on the hour, every hour, huge spike, because that's when all of the backup jobs kick off. So there's this massive, massive spike there. Even every 15 minutes, there's little, little blips that happen exactly on the 15 minute mark, which is... Matt Stauffer: Yeah, interesting. Yeah. Jess Archer: becomes predictable because of that application, but other applications will have their own interesting usage patterns. yeah. Matt Stauffer: Yeah. Interesting. Yeah. I bet you there's going to be some 2 a.m. spikes and other things where people say, okay, everyone's asleep. Let's start doing our crunching. Huh. Jess Archer: Yeah. And with different time zones, right? Like business hours for one customer is different than another. Matt Stauffer: Yep. No, that's fascinating. Do you have, this wasn't one of the questions, but I'm curious. Do you have different entirely separate applications for ingest versus dashboard, or is it all just one mega Laravel app right now? Jess Archer: there's basically just two main, so there's three code repositories really that kind of handle the whole thing. So there's the package that you install in your application. So that's open source. That's actually, public right now. Anyone can go look at that. Then there is our ingest infrastructure and then there's the dashboard app. So the ingest infrastructure is very AWS services heavy. It's mostly infrastructure code. and then yeah, that's the main, the main app that's kind of querying the data. Matt Stauffer: Okay, that's fun because I'm almost always the person counseling people like you don't need multiple apps, you don't need to put different databases for this, you know, there's all this architecture that you don't actually need and you're talking about this and I'm like, you need it. And you know, like, it's not that nobody needs it. It's just that people hear stories like this when it's needed and they say, therefore I'm going to put it on my app that's, you know, crud for dogs or something like that. Like, no, that app just doesn't need all that complexity. This one does though. So that's fun. Jess Archer: I a hundred percent agree with that. And I think that I feel like we've been pretty conservative with that. Like having those, you know, those two, two separate things. I imagine a lot of people could solve the problem with probably 10 different services. but we deliberately wanted to keep it as simple as possible. We have, we do have two databases. It's the only time I've ever felt the need to have two databases, but we did need, we still need a transactional database. So we still use, Postgres for doing all of the things that it's amazing at, but it's just not amazing at those like, you know, aggregations over billions and billions of rows. It's fine having billions of rows when you want to look up one row. Postgres will find that in a heartbeat, but yeah, not aggregating. Matt Stauffer: Yeah. Yeah. Not so much the aggregation. Okay. Okay. The next question, I feel like we already covered, but I'm going to ask it just in case you tell me there's something we didn't, which was what techniques are used to avoid slowing down the client application with monitoring code? Oh actually, maybe this is not, I was thinking this is about slowing down Nightwatch application, but I guess this is talking about my application. So. Jess Archer: Exactly, yes. Matt Stauffer: What does it look like? What's the process of ensuring that when I install Nightwatch, I'm not suddenly slowing down my app? Jess Archer: Yes. So, I mean, the first thing to, to state is that any, any observability tool will add some performance overhead. It's physically impossible to observe an instrument something without affecting it in some small way. But we've found through load testing that, the, it's basically not noticeable. Even when we put it on Forge, we didn't notice any, any, you know, spikes or anything like that. So we have done a lot of things very intentionally and deliberately. In the collection code, we basically don't get to use any of Laravel's kind of nicer features. Like we don't get to use collection pipelines, any of that sort of stuff. Matt Stauffer: Collections and everything else, Because they add just a little bit, right? Yeah. Jess Archer: Cause those like just little tiny bits, they, you know, they add a, extra object that's in memory. And sometimes, you know, they, they iterate over things in, you know, less efficient ways, but they create much more readable code. And for most applications, that's a hundred percent fine. And it's, that's a better trade-off is to have more readable, more maintainable code. But because we wanted to make sure we kept the impact as minimal as possible, we kind of had to go a lot more low level. One of the big things we do is there's a local agent that runs. And so all of the metrics that are happening in all the different web processes that are happening, they can just send those metrics to that local agent. Matt Stauffer: Oh okay. Jess Archer: So instead of it making an outgoing request to our server at the end of like the actual request life cycle, it kind of makes just as that locally, which makes it much faster. Cause even if you send that outgoing data, like in the terminating phase, so the response has already been delivered to the user, the user won't notice if we were doing that, but that process is still kind of tied up doing that task before it can go and answer the next one. So we wanted to make sure that that just happens as quickly as possible, sends it all locally. And then that agent can then batch that data up. And then I think every 10 seconds or six megabytes is when we send it to our ingest infrastructure. Matt Stauffer: Hmm. What's the agent running like in and on? Jess Archer: So the agent is just PHP. We, we were initially thinking like, we're probably going to have to write this in, you know, Rust or Go or one of these sorts of things. Matt Stauffer: Yeah, that's what I figured. Jess Archer: But we were like, well, we'll just, you know, play with it in PHP. Tim has built the majority of the agent and did some early proof of concepts. And we were pleasantly surprised that it was like absolutely fine. Like it was used very little memory. Matt Stauffer: Yes, fantastic. Jess Archer: It was easy to control. Like, yeah, we can monitor the memory usage and see that it clears itself fine. So we just haven't bothered to rewrite it in something that, you know, might get you extra milliseconds here or there. I don't know, but yeah. Matt Stauffer: Right. Jess Archer: And it's not to say that maybe we won't at some point, but the PHP has been excellent. use, react PHP. So it's got like a, asynchronous kind of event life cycle process. So we use that so that we can receive cause the agent almost acts like a mini web server. It receives data from all sorts of different processes. So it needs to be able to answer all of those quickly, batch them all up and then make outgoing requests to our ingest while still receiving new data from the web processes. And yeah, we can do all that in PHP. So why not? Matt Stauffer: That's amazing. Is that going to put any limitations of where these applications can be hosted or can most modern setups handle something like Jess Archer: Well, because it's PHP and Laravel's PHP, it runs anywhere Laravel runs, which is one of the other benefits of it, right? Is that, I mean, if, if we were monitoring something that wasn't PHP and we were writing it in PHP, then it'd be like, yeah, you'll have to install PHP on the server to use this thing. But with this, it's like, you've clearly already got PHP, so you're good. Matt Stauffer: Yeah. That's awesome. that's so cool. Okay Can we get through I think we can get the rest of these in time. The rest of these actually there was one last one before we go into the process and you mentioned this a little bit But you talked a little bit about the seeders that you built to test the project in your talk and also earlier in the episode here We just wrote a a big blog post about seeders at Tighten and I'm a really, really, really big fan of helping clients move from the only way we can accurately test our app is by having a full database dump of the production data. And I'm like, nope, we can move from that and we can help you make good seeders. And a lot of people have this idea that seeders are just lorem ipsum with completely random numbers. Are there any tips, tricks, tools, patterns or anything like that that you found allowed you to build these seeders easily to have much more realistic data or is it each time is just sort of like I'm going to hack on it, add it until it gives the distribution that I want. there, is there anything that made setting that up more easy? Jess Archer: I wouldn't say easy. There's a, like, our seeder code is, is pretty gross. Really. It has like, it's, it's got just a lot of very, you know, not very declarative code that's like doing all these kinds of things, but there's a lot of probability stuff built into there that you would have with a lot of seeders, but to, when it's like cycling through what request am I going to make? So we register a bunch of routes that the seeder can theoretically hit. And then we have some probability weightings for each route so that it goes, you know, there's this, this, this percent chance that I'm going to hit this route. And then it chooses which user is going to make the request. So it's got to simulate a user making the request to that. And we have logic in there so that all of the users in the system have like working hours kind of specified. So when it's randomly choosing which user is going to make the request, it only chooses users who are like. Matt Stauffer: Oh my gosh, that's so cool. Jess Archer: basically where it's like business hours for that seeded user, which is kind of important because when in Nightwatch you can click on a specific user profile and see a graph of their requests. And to make that realistic, the graph's not going to be there consistently hitting it 24 seven. They hit it during business hours and then they don't make any requests overnight, right? So just having some of those kinds of scenarios in there, having requests that sometimes fail and sometimes don't. Matt Stauffer: Yeah, yeah. Jess Archer: So there's a lot of, again, random number stuff where it says, you know, there's a 5 % chance that this outgoing request will not be a 200 response. It'll get, you know, a 500 gateway timeout from the, the upstream thing that it's kind of simulating. And yeah, just connecting all that sort of stuff together and even simulating like the durations of everything. Cause we capture in the timeline, you can zoom into an individual request and see down to the microsecond, how long each database query and everything else took. So those are kind of randomly generated between like a certain amount of time. So it's not just like, yeah, just choose a random number. It's like, this thing is going to take about a second, but give or take, you know, like a hundred milliseconds so that it's random, but still consistent. So that when you actually click on like the slow queries page, it's not like they all have like some slowness. It's like, no, there's some queries that are always fast and others that are always slow and some that are sporadically fast. So we have Matt Stauffer: Yeah, that's so cool. Jess Archer: certain times of day that the seeder will go during this time of day, we're pretending that this external API is down. So any request that goes to that will get an error. So that when we look at the graphs, there's like a red patch. That's not just evenly distributed across the whole graph. It's actually concentrated in one area. Cause that creates something that you can kind of zoom in on and then creating like the scenario so that when you do drill into them, cause we use the seeder for like all of the demos we do that they all demonstrate a scenario that's kind of easy to quickly understand. Matt Stauffer: Yeah. Jess Archer: So the domain we chose was a flight booking app. Flights are great because they've got a number of different relationships. You've got like tickets, flights, airlines, all those sorts of things. There's enough complexity, but it's also still like a domain that's easy to get into your head of how it should work. And just creating all these fun scenarios, like, you know, this flight was overbooked. So there's like an overbooked exception that can happen. And only on this particular endpoint. Yeah. We had a, had a scenario where there's a cache event. Matt Stauffer: Yeah, and we instantly understand that. Mm-hmm. Yeah, that's amazing. Jess Archer: That the cache happens for like, gets a, it caches data for 10 minutes. And then it has to run a slow query after that every 10 minutes. So the seeder knows when it has cached the data. And so every request for the next 10 minutes, we'll have a cache hit. But then after 10 minutes, it'll be get a cache miss and then a really slow database query will happen. So the seeder has all that programmed into it. Matt Stauffer: Wow. Jess Archer: So can imagine the code is pretty crazy. We kind of tried to divide it into these like named scenarios so that they're, it's easy to understand. Like, I'm triggering the occasionally slow, slow route, but yeah. Matt Stauffer: Yep, yep. That is fascinating. I mean, I think I built some very complex seeders in my life, but this is next level and that's so fun to hear. You mentioned that you wanted to kind of show some real world, like, yeah, some of these are gonna fail. And when we talked last time, one of the things I pointed out was that I love how much you all have paid attention to... What do I care about when I'm opening up the SaaS? Because I think that's a thing that people often forget to be considering as they're building these SaaS is like, it's not just data. It's like, well, I'm coming to learn something and I don't always know when I'm coming to learn and you can make it easier for me to figure out here's what you should pay attention to right now. And you weren't really able to do that until you got good enough data on your own to say, oh you know what, you know, I hope I'm not, not allowed to say this, but we are, we got early access this morning and I have been clicking around seeing it on one of our live softwares of service and I'm like, oh my God, the fact that I can go to the users page and it says of the, you know, whatever the last 24 hours of requests this user made, how many of them were 200s, 300s, 400s, 500s? And I'm just like, this is incredible because now I can know that this user is probably very frustrated because they have run into this inordinate number of errors. And I instantly have it for I can dig in I reach it out to them if I need or if I look at my requests I can say my gosh this one API endpoint that we have has an unbelievable number of 402 errors why is that you know 402 status codes just in clicking through the thing just to make sure my team had set it up right I learned things about our user base about our code base or about our application that I never would have on my own so the way that you all were able to prioritize discover ability of the things that people care about is really incredible and I really do believe that the seeder setup has I mean obviously you built it so because you knew you needed to do this but I'm just like I hope this encourages people to say like building these real and complex and nuanced seeders gives you insight into reality in a way that you wouldn't have otherwise. Jess Archer: Definitely. And there were certainly scenarios that we kind of, like one of the problems with the seeder approach is that we had to come up with scenarios that we wanted the software to tell us about. So we had to think of like, think of what we wanted to get out of it and then create that scenario. When you've got real data, there's scenarios you don't yet know about. Matt Stauffer: Yeah, yeah. Jess Archer: So when you look at the data, then you actually go, I don't actually know what's happening here. So I, so you look at it through a different lens. Whereas when it's like, yeah, it told me the thing I told it to tell me. It's like, well, good job. But so we, learned like we did like the seeders are great, but we did learn a lot. Once we had it had like Laravel Forge on there. We got a ton more information from that because it genuinely showed us things that we were curious about that it didn't give us enough information. We were like, this is showing me something, but I want that little bit more. Matt Stauffer: Yeah. Jess Archer: But then we can incorporate that into our seeders. So having that kind of combination of seeder data with tons of scenarios, but then continually incorporating new things into it and still testing with live data and dogfooding it. I mean, we use Nightwatch to monitor Nightwatch constantly. So we're always in there and we have to kind of switch context of like, I'm using this like to solve a problem versus like I'm building the thing. And sometimes it's kind of frustrating because it's like, I'm using Nightwatch. Matt Stauffer: Yeah. Yeah. Jess Archer: Like just as a tool, like I'm just using it. And then I spot something and I'm like, do I go fix that? Or do I continue actually doing the thing I was originally there for? Yeah. Matt Stauffer: It's not what I'm here for, but I do want to fix it. Yep. I love that. Okay, we are at 44 minutes. I said we were gonna stop at 45, so I'm gonna ask you one last question. Sorry to anybody who else's questions weren't asked. But my friend Kenny Myers asked a three-parter. It was, how do you approach scoping and planning a project? As a part of that, how do you break down the work? And then a part of his question was, can I be your best friend? And I included this both because Kenny is very funny, but because also I just want you to see that it's not just me that thinks you're incredible and brilliant delightful, but there's other people in the world who have the opportunity to ask a question and they just need to kind of make sure they can throw that in. But realistically, how do you approach as a lead with the type of team that you have? And you and multiple other people at Laravel have mentioned, despite the size of the team, it still is very much like a we just do it our own way. There's not a ton of process and overhead and each team runs their own way, but still. You've got a team that's more than just you and Tim hacking things out, right? It's a big enough team to be really interested in the question of what does it look like for you all to say, okay, we have to build the entirety of Nightwatch. Where do you get started? How do you break it apart? Are you doing weekly cycles? Just kind of what's the process like for you? Jess Archer: Yeah. So, I mean, the process has changed a lot as things have matured. So in the very early stages, it was just, just hacking around proof of concepts, just following wherever kind of the thing that was exciting was taking you, like letting, letting, just following like the interesting path, right. And, and just learning, it was kind of that discovery stage. Once we kind of had done all that exploration and learnt what was possible, what was hard, all those sorts of things. We were able to kind of then start to figure out, okay, what is this thing? How are we going to build it? What, what should it be? We got more people involved. So getting a designer in really early was really helpful. So we got, Jeremy in and Jeremy's this amazing designer who really understands, like he's a product designer, not just a graphic designer. He's also like a front end developer as well. So he can implement his crazy designs, but he's kind of really great to work with because he can kind of. Matt Stauffer: Yeah. Yeah. Jess Archer: You kind of talked to him about like, this is what I'm trying to achieve. And he can kind of go through like the user experience of that. And he really understands the problem space. So yes. Kind of collaborating in that sort of way, but still very, very loose on organic. We weren't doing a whole lot in the early stages in terms of like actually writing out tasks and tickets and all these sorts of things. But as it got, as we got more people and as you know, was starting to get more serious, then we started using linear. I didn't really know how to use, you know, project management software. So I kind of just had a few different stabs at like different approaches of how to break stuff up and then basically put a bunch of things in there and kind of, then just forgot about it and wasn't updating statuses. But then we got a product manager who's also a developer. So it basically everyone on our team codes, no matter what the job is and he helped bring a lot more kind of order to the chaos. Matt Stauffer: Yeah, brilliant. Jess Archer: And yeah, really break it down into like separate parts of the application, figure out what the other teams are doing so that from like the management perspective, there's like the reporting sides of things is a little bit more consistent while still letting us kind of work in the way that we want to work. So yeah, we now break things down into, into cycles. So we do, I think we're doing three week cycles at the moment. We do like a little bit of kind of refining where we go through all of the things in the backlog and decide what we think we can fit into it. So it's, it has become a lot more official in those sorts of ways, but we still, you know, we'll go and pick up things that aren't in the current cycle and all those sorts of things. Like we want to be like agile in the truest sense, right? Of this thing over here is really calling my name right now and I can, I can justify it in some way or another. Matt Stauffer: Yeah. Yeah. Yeah. Is there a line somewhere in linear that just says at this point it can be launched? Like, do you have a definition of done somewhere? Jess Archer: Not, not strictly speaking. We kind of, set like various deadlines of like, we get to this point and it's just going to be as good as it's going to be. At that point, like we don't want to continually keep refining this thing forever and never launch. So we had to kind of draw a line in the sand where we go, okay, of all the features we are committing to, we now want to get those as polished so that they kind of meet up to the Laravel standard and anything else we'll just, you know put it behind a feature flag for now, and we can then finish it off properly and bring it in later. So we'd rather get something out there and yeah, have those few little bits and pieces turned off and add them in when they're ready. Matt Stauffer: Okay, as we mentioned before, we could talk for hours, I'm when I'm supposed to cut us off, but I have to ask you, is there anything else that you wanted to cover about any of these individual topics or what's on your mind or your heart about this or about anything else before we wrap for the day? Jess Archer: Not that I can think of. I forgot to answer Kenny's question about being best friends. Let's just start with friends and see where it goes. Matt Stauffer: Very well responded. Jess Archer: But yeah, nothing else I can specifically think of. I'm just super excited about this thing. I'm really proud of what the team has built. I'm really excited to get more and more people's hands on it. Yeah, it's, as I said, like using it ourselves is exciting. Like putting it on our own application has been exciting to see and learn things about our application that we didn't know about. So I'm really looking forward to everyone else kind of getting that same experience. Matt Stauffer: Well, this is not a brag, I promise, but until yesterday, I would have said I haven't looked forward to anything like I'm looking forward to Nightwatch, you know, like in quite a while. I, know, Cloud is very exciting, but I'm not in like the perfect target market for what's exciting about Cloud. But like Nightwatch, I've been like, since I heard about this, I cannot wait. And I've only gotten to click around it for maybe three minutes today. You know, I've been too busy on other things, but those three minutes were everything I hoped for, you know? And I'm like, I'm trying not to like overhype it. But I just like, I can't believe I learned so much about an app that has been running for years in a couple minutes in this tool. So I'm 100 % with you. I cannot wait to see everyone's response and everyone's experience of it. And personally, I can't wait to use it more. So as always, thank you for all the work that you've done here and everywhere else. And thanks for hanging out for a little bit today. Jess Archer: Thank you, Matt, I appreciate it and thank you for having me. Matt Stauffer: That's always a pleasure. And for the rest of you, thank you so much for hanging out with us and we'll see you next time.