Latest / The Laravel Podcast / Queues, with Mohamed Said
Transcript
- Matt Stauffer: Welcome back to the Laravel Podcast, season four. Today, we're talking to Mohamed Said, who is the employee number one of Laravel queue master, and just a general, great guy. Stay tuned. Matt Stauffer: All right, welcome back to the Laravel Podcast, season four, where every single episode is about a single topic. And today, once again, I'm taking one of my friends who I know, knows a lot of things and is willing to flex with me a little bit. We got a little bit more than one topic in this one. Matt Stauffer: Today, we're talking about queues and events. And actually when we were talking before, and I was like, "It's three topics." And he's like, "No, it's not, it's two, Matt." So queues and jobs, queues and pushing jobs in queues and then events on the other side. Matt Stauffer: I have joining me, the man, the mystery, the first employee of Laravel, Mohamed Said. So Mohamed, welcome back to the show. I know you're one of our wonderful repeats. You and I are, I think we've known each other for years at this point, so I can call us maybe not old friends, but mid-time friends. Matt Stauffer: Could you say hi to the people? And then as always, tell them what do you do? And how do you tell people what you do when you meet them wherever you go these days, whether grocery stores or whatever else? Mohamed Said: Hello, everyone. My name is Mohamed Said and I'm a web developer at Laravel. Mohamed Said: Well, if I meet someone at the grocery store, and I explain to him what I do for a living, I basically say I'm a programmer. This job is actually very popular here. So everyone knows what a programmer does. It's not hard to explain it. Mohamed Said: But if I meet someone else who is working in the industry, in the software industry, and I try to tell him what I do exactly, like I work for a company that builds a framework that people use to build web applications, that's when things get a little bit interesting. I have to explain, I have to explain to them that I use PHP and that the framework is built on PHP, and that it can do this and that. So that's when the conversation is interesting actually. Matt Stauffer: I like that. Where you live, what is the most common framework? If you meet another programmer, or somebody else who's in the tech industry, what's the most common framework that people have heard of that you can compare it to? Mohamed Said: Well, if he's a web developer, it's definitely Laravel. It's the most popular framework here. In the world, actually. Matt Stauffer: Really? Mohamed Said: Yeah. Matt Stauffer: Wow. So when I compare it to people, for some reason, Rails is always the one that people have heard of more than anything else. So I say it's like Rails, but I wondered if for you it was something different. So Laravel is actually, they know what Laravel is basically, then. Mohamed Said: Yeah, Laravel is the most popular framework here, because PHP is the most popular language here in Egypt. Like most companies, they build their stuff on top of PHP or using PHP, and they build the stuff actually on top of Laravel. Matt Stauffer: That's incredible. I know it's very popular if you look at the internet, but when I casually meet a programmer, they've probably heard of PHP, and there's somebody making fun of like old head people using PHP. But the easiest thing, if I meet somebody who's super enterprisey, I'll probably end up saying it's like .NET MVC, or something like that. And if I meet somebody who's a little bit younger, I might say it's like Rails. That's real interesting. Matt Stauffer: Okay, so I've got a lot to ask you about, not a lot of time. So I will stop dillying around and just ask you. So we've got two, not three primary topics in front of us right now. I think the one that is the most complicated and that you happen to be the man around town about is queues. So let's start there. Matt Stauffer: If you were to talk to somebody about queues and pushing jobs into queues, who's five-years-old, which as a father you might have some general ideas about, how would you describe that general concept to them? Mohamed Said: Well, actually, if I want to describe it to a five-year-old or someone who is starting with software in general, he's a beginner, he doesn't understand what queues are and he reads something like daemons and workers and stuff like that, it's always confusing. Mohamed Said: Because that's actually my personal problem. When you have to go to the grocery store or the hypermarket, you spend time putting stuff in the shopping cart, and then you go stand in a queue until you reach the cashier. And then the cashier starts loading the stuff from your shopping cart, and calculating the amount that you have to pay. And then you pay and then you put it back in the shopping cart, and that's basically 30 to 60 minutes wasted of your time. Mohamed Said: So what if instead of this, you put stuff in your shopping cart, and that's the actual thing that you care about. You went to the market to put stuff in the shopping cart for you to take home. And what if you just put the stuff in the shopping cart, and then hand it over to one of the staff and then you go home, or you go to the playground, or go eat ice cream? And then when you go home, you will find your things at the door and everything is done for you, so you don't have to wait for it. Mohamed Said: And that's basically what the queue component of Laravel does, it helps you offload some of the tasks that you are not interested in, or that you don't want to worry about right now. You just offload it to someone else, or some other process on the computer, some other program on the computer to take care of it, while you continue doing your thing. Mohamed Said: For a grocery store customer, your thing or the main job or the main part of what you're here for is to buy stuff. And if you spend 30 minutes buying stuff, and then another 30 minutes just standing in a queue, you might reconsider the next time you have to go to the grocery store. And you say now I'll just starve myself today, because I don't have to stand in line for 30 minutes. I think that's how I'm going to explain it. Matt Stauffer: All right, the episode is over. That's perfect. We're done. That was so good, man. I love that. Oh, that's good. Okay, so before we walk over to events, we'll do those in a little bit. Matt Stauffer: Let's take that a little bit further and imagine now a practical circumstance in which a programmer would choose to reach for a queue? Can you tell me a little bit about what it would look like for them, give me an example, when's the last time... anyway, when's the first time you think somebody would tend to reach for one? And what high level programmer wise does it look like to set up and use the queue? Mohamed Said: Well, first for a programmer to need to use queues, I explain it in this way, a user interacts with a program using user interface, a graphical user interface, like buttons and forms and such. And a program speaks to another program using a programmable interface, an API, like when you call a method, when you call a function. That's how a programmer speaks with the program. Mohamed Said: And what the queue is, is imagine if you have several programs running in the background, or running on your machine. These programs speak with each other. And you need to find a way or you need to tell the programs how to speak to each other, and how to interact or how to react in different situations. Like if one of the programs took too much time doing a process or another program failed to finish a process, and so on. Mohamed Said: And that's basically the kind of thing that you're going to need to use queues for. Matt Stauffer: So let's get a little bit more concrete now and say, if I'm a Laravel programmer, and I am building a Laravel app that allows users to have user profiles, so that they can shop or something like that, just think about what are some common circumstances in which I would choose not to just do something synchronously, but instead push a job up onto the queue? What's one thing you could think of that would lead me to choose to queue something? Mohamed Said: Well, of course, the obvious answer, and the example that people give every time they discuss the importance of using queues is that people won't have to wait until the action is done. So for example, when people sign up, they don't have to wait until your application speaks with the mail service provider to send them the email verification email. They just can sign up and go to a welcome screen until everything is done in the background. Mohamed Said: They find the email in their inbox later. They can check it right now, they can check it next day, they don't have to wait until this is done. So that's one thing. Mohamed Said: Another thing actually, and taking the grocery store example is, when you put stuff in your shopping cart, and one staff member has to take all the stuff from the shopping cart to put it on the table and count it at the counter, what if you have several staff members, and each of them offloading some of the stuff on the shopping cart? So instead of you wasting five or 10 minutes for a single person to do the job, if five persons started working on that job at the same time, they will finish faster. Mohamed Said: And that's another thing that you can use queues for. You can program several instance of your applications to run in the background and process or run a process at the same time, and they will finish it quickly. Mohamed Said: Another thing is, if you do things synchronously, you have to handle everything. It's either all of it succeeded or all of it failed. If things succeeded, it's done. If things failed and the user wants to retry it, they will have to start the process all over. Mohamed Said: But when you flow things to the queue, you can have, if there is a temporary downtime on one of your servers or so, only that server will be isolated in the failure. And then you can restart only the task that this server was trying to do, not the entire task that you sent to the queue. So that's basically the scenarios that you are going to need queues for. Matt Stauffer: Okay, perfect. So let's talk about one of those scenarios, the first one you said was the most common example people give. And I love that you're teaching us that there's times to reach for queues other than the most common, the user interface is going to be slow. I love that. Matt Stauffer: But before we go any further into those, I want to make sure that people have a general architectural understanding of what's happening when I push something on the queue. This is in part because in the past, I used queues using something called IronMQ. And the actual architecture of how we're interacting with it was we were actually pushing pieces of code onto the queue that then the queue itself executed, which is not how we do things in Laravel, right? Matt Stauffer: So if I were, for example, to want to send up an email, and instead of directly sending the email right now, I wanted that email to be queued to be sent later, what data am I actually sending up to that queue? And how is it storing it, and then how is it eventually getting done when it gets pulled off the queue? Mohamed Said: Well, the queue system in Laravel is two parts, part that runs on the server and the other parts that sends or the requests that sends the job to the queue. So on the server, you have several workers running. And a worker is basically a Laravel application. They call it a daemon, but I don't know why they have this term for a long running process or an ongoing process. It's a scary term. I remember reading it the first time, it was very confusing what a daemon is. Mohamed Said: But a daemon is like a process that lives on your server and it never ends. For example, an HTTP request starts a Laravel application instance. And then when the request is done, this instance is done, is terminated. But a daemon process is a Laravel application that doesn't... just staying there and wait or just looping over and over, waiting for jobs to process. Mohamed Said: The other part is sending jobs to this daemon. When you dispatch an email job, for example, you send the email address and maybe the name of the user, maybe the time they registered. You send this data to the queue. And it's going to get stored in a persistent storage, like a database or a Redis instance. Mohamed Said: And then the daemon or the ongoing process on your server or the Laravel worker is constantly checking this storage. Let's say for example, you're sending jobs to a Redis instance. So this worker is checking the Redis instance over and over, trying to find jobs to process. Mohamed Said: So once it finds the job of the email that you are trying to send, it's going to get the job and get the data stored in that job. The data would be the name of the job class in your application that you want the worker to execute, and the data that should be passed to the job instance. Mohamed Said: So for example, if the job instance is called send confirmation email, and you have to pass the name of the customer and the email address, Laravel or the worker is going to create an instance of this class, the send confirmation email, and pass those data to it, to create the instance. And once the instance is created, the worker is going to call the handle method inside that instance to execute to your job. So that's basically how the queue works. Matt Stauffer: Yeah, that's beautiful. I love that. And that simplifies it so much for me, because I think, like I said, so I had worked with queues before, where we had to write the code and push the code up. So queues were hard, because I had to write these standalone... Because we were using Code Igniter. And I'd write these in standalone PHP things that I was pushing up that had nothing to do with the rest of my Code Igniter that had just enough code to function. Matt Stauffer: And then I didn't really know how they were working or what environment they were in. And when I started with Laravel's queues, oh, no, it's pushing just enough data for Laravel to know how to operate on it. And then Laravel itself, a full instance of my exact same Laravel application running in the daemon, news up basically that job based on that. Matt Stauffer: You described that perfectly, there's a long running Laravel instance somewhere that says, "Oh, there's a new thing here. Cool, I'm going to open it up, I'm going to use it to new this thing up, and I'm going to act on it." So it's really just the same as our normal Laravel application, right? Matt Stauffer: The only thing that's happening there is that it's a convention that Laravel has, that if you are going to push something up, and then it's going to be pulled down by the daemon with that data, here's the structure or class we want for it. It's a job or something else that has should queue on it. Matt Stauffer: But in general, it's really just data being put into a database. It happens to be Redis or whatever. And that data in the database is instructions for the Laravel app that's running long term on the daemon to create something and act on it, right? Mohamed Said: Yes, basically, that's how we describe it. It's like instead of calling or instead of creating an instance of the send confirmation job class, and calling the handle method synchronously, you just send the name of the class and the properties to pass to it. You store it in the database on Redis. And then the worker is going to create the instance and call the method for you. Matt Stauffer: Yeah, I love it. Also, when you said that, I looked it up on Wikipedia, because I couldn't remember why it's called a daemon. So a note to anybody, it can be pronounced either daemon or daemon. Apparently, they actually mean the same thing. I was like, "Oh, it's not a daemon. It's a daemon." It's the same thing. Matt Stauffer: But it's because apparently, there's a thought experiment called Maxwell's demon back in 1867 about the second law of thermodynamics. But the thing that is relevant is that in the thought experiment, the demon spends a whole bunch of time basically in the background, constantly acting on something. Matt Stauffer: And so the people who originally created the concept of the daemon at MIT were like, "Oh, this thought experiment that is a mythical being constantly working in the background is very similar to what daemons do." So it turns out that's why, thank you for leading me down that path, Mohamed. Matt Stauffer: So let's talk a little bit more practically about how I would create my own. So you mentioned an email. I know that a lot of things in Laravel are really easy to queue themselves. Could you talk a little bit about the should queue and anything else that would be helpful for us to know, if we were to choose to create our own things and choose to make them queue in Laravel? Mohamed Said: Well, the basic form of a queued job is just a class that has a method called handle. And inside that handle message, you run any logic that you want the queue, or you want the worker to execute. Mohamed Said: But there are other things in Laravel that can be queued, like for example, a mailable, a mail object. You create a mail object, and you have the option to send that mail right away from your code, with using the mail facade. So mail send and then you pass the mailable object, and this is going to execute the mail right away. It's going to send it synchronously. Mohamed Said: Or you can say that I'd like to queue that mailable. So you add the interface should queue to that mailable class. And Laravel will understand that you want to queue that mail. Every time you want to send it, it's going to be sent to the queue, it's not going to be sent synchronously. Mohamed Said: Same goes to notifications. Whenever you send a notification, Laravel will check if you have the should queue interface implemented on the notification class. And if so, it's going to send it to the queue. Mohamed Said: Same to event listeners. If you have an event listener that has the should queue interface, Laravel is going to execute the code and that listener in the queue. It will not execute it synchronously. So these are the things that you can queue. Mohamed Said: Another thing that can be queued and actually, it's queued by default is broadcasted events, like when you're doing WebSockets. And you're just sending an event to the WebSocket server, Laravel will queue it by default. You don't have to implement anything or you do anything. It's going to be queued by default. Mohamed Said: Because if you're sending something to the WebSocket server, Laravel is smart enough to understand that you don't want to do the sending synchronously, it's going to be done in the background. Matt Stauffer: Yeah, that's good. So the only thing we didn't talk about, you mentioned the fact that if you wanted to create your own thing, you want to make your own job, you want to push on, then it has a handle method. If I were to create something of my own, there's been a couple different names for the thing that you push in the job manually. A job that you push manually and yourself, commands and jobs and all kind of stuff. Matt Stauffer: So if today, I wanted to create a fresh PHP class or object that is solely intended to be used to push something on the queue, what does that look like? What's the name of that? Mohamed Said: Well, it's a bit like the job or a job instance or a job class, is like a basic PHP class. There is nothing special about it. You send that an instance to the pass component in the Laravel framework, and the pass component is going to convert it to, or it's going to serialize it to a string form. Matt Stauffer: Like we mentioned. Mohamed Said: That string form can be stored in Redis or it can be stored on a database or sent to a queue driver. And sometimes, or before the pass component was introduced in Laravel, when you had to send a job to the queue, you had to use the queue facade, and then you called the push method. And you send or you dispatch or you put a class name. And you have to do the serialization of this class for Laravel to understand how to unserialize it, when the worker receives the job in the end. Mohamed Said: But you don't have to do this anymore. And for a long time, Laravel is doing this automatically behind the scenes, so people won't have to worry about it. Matt Stauffer: So it's really helpful for you to say that, and that's one of the reasons why I pushed at this a lot is because it has felt in the past that you need really smart classes to be queued, right? Like you have to make sure you create it the right way. Matt Stauffer: What you keep pushing at is really all it needs is this handle method and it needs this interface. And when you have those things, Laravel is taking all the rest of the responsibility off of your plate and basically, you just dispatch that and that's it. I love that. Matt Stauffer: All right. I know we also have the entire eventing system to talk about, but before we go there, what do you think the most common thing is that gets people stuck when they're first getting started working with queues? Mohamed Said: I think the most confusing part in the queue system is the number of configurations around jobs and workers and connection and so on. I think in my opinion, it's the most or the biggest component that has the largest number of configurations. Mohamed Said: And this gives the developer a lot of flexibility to tell the workers or tell the queue what to do exactly. But with great flexibility comes some confusion, because you have to configure a lot of things. Mohamed Said: Of course, Laravel makes decisions on behalf of people for the most common use cases, but you have to learn all the configuration keys, in order for you to use the, or to have the actual or the real power of the queue system. You have to understand all these configurations. That's the most confusing part. Mohamed Said: But besides that, one of the most important concepts, like we said that a worker is an ongoing, or a daemon, or how do you pronounce it? Matt Stauffer: I say a daemon. Apparently they're both right. Mohamed Said: You have a daemon running on the server and it's a single instance of a Laravel application that's running and it's processing all your jobs. So for example, if you send a job that sends an email, and another job that creates a record in the database, they are both totally two different job instance, two different classes, but they are going to be executed by the same instance of the Laravel application. Mohamed Said: Unlike an HTTP request, each HTTP request will start a separate instance of the Laravel framework. So having a single instance processing all jobs, one of the things that confuses people, that is state management when doing so. Mohamed Said: So for example, if you have a singleton defined in your Laravel application, a singleton is a class, the container creates once and uses it during the lifetime of the application. And since the application will not get terminated, that instance is going to be used for every job. Mohamed Said: So for example, if your job relies on the fact that each HTTP request will start the application from the start, and you will have a fresh instance of this singleton, when you send this job to the queue, it's not going to be a fresh instance, it's going to be the same instance that was created when the worker started running maybe 30 days ago. That's something that is a cause of so many bugs and so many problems and so many confusion, especially when people, for example, have static classes and static properties. Mohamed Said: They store values in these static properties. They expect on the next job the values will be empty. But that won't be the case, because it's the same application, the same memory, the same everything. Matt Stauffer: Wow. Now that you say it, it makes sense, but I've never been bitten by that. So that's really helpful that you said that. So if we're using singletons, we can't expect those singletons to refresh at the beginning of every job, because each job is not a new... And it's like working with something else like Ruby or something like that where we can do some lazy programming things in PHP, because we knew everything's going to refresh on every single request. And when we're working with queues, we don't have that luxury, that's really helpful. Mohamed Said: One of the things that Laravel is good at, besides the perfect documentation, and all the things that we know is that it makes a lot of decisions on behalf of the user, so they don't shoot themselves in the leg. Matt Stauffer: I like that. Mohamed Said: Something like this, most of the cleaning that happens around jobs is done in the background. So the problem of having to deal with the same state for a long time is not felt by as much people as it would have been without Laravel doing this. Mohamed Said: So it does a lot of cleaning in the background. It makes a lot of decisions on behalf of you. So you don't get into this situation a lot. But again, if you move to a more advanced kind of jobs, like jobs that do a lot of things, you will realize that you are using the same memory allocated, it's the same process. Mohamed Said: So if you store things in the memory and you don't clean after it, it's going to stay in the memory, and then you will have crashes, and then you will have problems and so on. So that's exactly one thing that people need to be aware of when dealing with queues. Matt Stauffer: That's great. Mohamed Said: Another thing, it's actually a little bit trivial. But we ourselves while developing Vapor, we had this problem, we had a job that was dispatched from inside a database transaction, and the job was dispatched before the transaction commits. So it was relying on a model, an eloquent model that was created inside the transaction, but the transaction didn't commit yet. Mohamed Said: So the model wasn't in the database yet. The job was dispatched, and because the worker process is a separate process, it picked the job before the transaction commits. So when it started processing that job and looked at the database, the database or the record wasn't in the database and it errored. Mohamed Said: And that's another problem that I've seen. I've seen it on the forums, on the Laracasts forums several times. And we actually faced this issue while debugging something on Vapor a couple of months ago. But a few weeks ago, we released another version of Laravel that has automatic configuration that you can use to say, "Hey, every time I dispatch a job from inside a database transaction, don't dispatch this job until the transaction commits." Matt Stauffer: Cool. Mohamed Said: So it's done automatically for you right now, you don't have to worry about it. You just have to tell the queue system or you have to tell the command pass, "Do not dispatch any jobs while there are open database transaction." And if that transaction was rolled back, discard the job. Do not send it, because it doesn't make sense to send it anymore because- Matt Stauffer: Now that the transaction is not happening. Mohamed Said: The transaction is not happening. So that's another thing- Matt Stauffer: I like that. Yet another way you're keeping us from shooting ourselves in the foot, right? Mohamed Said: Yes. Matt Stauffer: I love it. So you had mentioned a little bit ago that there's all these different configuration options. I know that there's two places at least where you're thinking about configuration options, and one of them has to do with the configuration in your system. Matt Stauffer: But one of them has to do with the configurations you pass into the daemon when you're spinning it up. I realize we haven't actually talked about what the process of spinning up a daemon looks like. Obviously, you could go to great detail teaching us how to do it on our own. But I would say if somebody is just listening for the first time, and just learning about queues right now, the answer for, how do I spin it up is used Forge. We're not going to go any further. Matt Stauffer: So if I were to recommend everybody, we're using Forge to do it, can you talk a little bit about what the setup is? And what configuration options Forge makes available to us and whether we should be thinking about those? Or should we just stick with the defaults when we're creating a new daemon? Mohamed Said: Well, when you're creating a worker, let's say you are using Laravel Forge to host your application, you're restarting a worker on Laravel Forge, Forge has this nice interface that it shows all the configurations that a worker accepts, like the timeout of the worker, or how many processes you need to start, how many workers. Mohamed Said: For example, how much time should the worker wait before pulling jobs from the queue each time and so on? It's like several configuration that you can see on a single screen. It has hints on what each configuration does. So you can easily decide what you want to do. Mohamed Said: But actually, the defaults when creating a worker are basically the ones needed by let's say 90% of the use cases. Except for one thing maybe is the number of retries. By default, a worker is configured to fail the job if an exception was thrown. Mohamed Said: So in our example, if you're sending an email in the queue and sending that email failed due to, for example, a temporary outage at the email sending service, that job is going to fail and it's going to be stored in the failed jobs table for you to decide what to do with it later. But for 90% of the cases, that's not what you want. You want the worker to retry the job for you automatically. Mohamed Said: Because it's one of the things that you use a queue for, the queue system for. So there are automatic retries when things fail. So when starting a worker, you probably will want to configure the worker to retry the job several times before it considers it a failure. Mohamed Said: Another thing you need to mind for is the amount of time for each job is allowed to run. So you don't want your workers, for example, to keep running a job for hours, get stuck for hours on a job that's going to fail due to a network that's not responding, or a problem on the server and so on. Mohamed Said: So you might want to think about this when you start a worker. How much time should the worker allow a single job to run before it terminates it right away? So these are the two things, most common that people will need to configure each time they start a worker. The timeout for each job, and the number of attempts a job may have before it's considered failure. Matt Stauffer: Those are really great points, especially because sometimes we're queuing things where we want there to be a longer timeout. Because we're queuing it, because it takes a long time. Let's say you're queuing something that you're dealing with this really finicky external API, and every API call takes a minute and a half. You got to make sure your timeout is longer than a minute and a half. But like you said, you don't want it to just sit in timeout mode forever. That's really helpful. Matt Stauffer: Because we haven't even touched events at all, I want to make sure that I gave you the chance to say everything you want to say about queues, because I know that queues, I think, to me, queues are more important. I'm going to be honest that while I want us to talk about events, I very, very seldom use events. I actually just wrote an event yesterday on a client project. So it's not that I never use them. But I use queues daily, and I don't use events daily. I do want to make sure you really have the time to talk about it. Matt Stauffer: What other things do you think it would be helpful for us to cover about queues before we roll over to events? Mohamed Said: Well, basically, I would like just to mention that I agree with you that events, it's not the favorite thing that I use when building applications. The problem with events is that you don't see what happens after you send the event. It's hard to figure out what happens. You have to check all the listeners that listen to this event, and check their code one by one until you figure out what happens when this event occurs. Mohamed Said: To me, I'd like to see or it's better for me to see the complete execution plan of the code that runs synchronously at least. And the parts that are going to be run in the background or via queue worker, I have them, I have a job, I dispatch job from the synchronous part. I know that if I want to see what happens after dispatching this job, or what happened inside a job, I'll go to the job class and check it out to find what's going on. Mohamed Said: But with using events, you have to see all the listeners and check all the listeners. I don't enjoy this way of building applications. I love to see everything in front of my eyes to see what happens when this piece of code is executed. Matt Stauffer: I think of queues and events as being similar because both of them involve not necessarily asynchronicity. But both of them involve the fact that you say something should happen and the thing happening is disconnected a little bit. And we'll talk about events in a second. Matt Stauffer: But just to your point there, with events, what's disconnected is that you trigger the event, but the processing of the event is not defined there. With queued things, you triggered the job putting on the queue and the execution definition is defined in the same spot. But it's not actually synchronously acting at the same time. So functionally, they both are separate a little bit. Matt Stauffer: But like you just mentioned, when you're queuing something, whether or not it's executing right now, the code is right next to where you just triggered it, which is helpful for us as programmers. Whereas with events, that's not the case. Matt Stauffer: So you know what, even though I said why don't we finish with queues, I think let's just get to two events real quick. And then I think we'll wrap up the whole concept. So can you give me, I think we'll keep events short, can you give me the five-year-old concept of what the eventing system is like in Laravel? Mohamed Said: There is this event that my son woke up, this is an event. There are several listeners listening to this event. So when my son wakes up, you have to prepare food for him, you have to change his diapers, you have to do a lot of stuff, you run crazily all over the place. So this is an event, my son woke up. Mohamed Said: A lot of listeners get executed when this event runs. And some people use this pattern to build their entire application. So for example, when a user gets registered, you do several things. You send a welcome email, you send a confirmation email, you set up an account for them, you do several things when a single user registers. Mohamed Said: Some people do all this stuff in the controller or in a repository. Other people choose to use events. So they file the event that a user was registered and then they create several listeners to this event. A listener that creates an account, another listener that sends a welcome email and so on. Mohamed Said: So that's basically what events do. I know it's not for a five-year-old, but that's it. Matt Stauffer: Well, your son waking up was for a five-year-old, so you're good, and you dropped right in to what you know is my second question always, which is, what's the little bit more technical version of it? So if I were to build an event, so that the event that I just built, what was it called that I built yesterday? Because I was going to use that as an example. Matt Stauffer: It was something about Stripe. Let's say it's a Stripe webhook. So every single time Stripe sends a webhook back to me, usually I'm not actually expecting it, because Stripe webhooks don't come at a time where a user is interacting with your app. Matt Stauffer: There's just Stripe decided to send you, with your app, Stripe said, "Hey, you know what, this user logged and updated their payment information. And now it's no longer failing." And so when they send a webhook back to me, I've got a webhook controller that translates that webhook into a user updated credit card event. Matt Stauffer: What does it look for me to create that event and create those listeners? You mentioned a little bit about it. I'm going to create a class that defines the event, and I'm going to create another class somewhere that defines a listener. Can you tell me a little bit about the methods they have and everything like that? Mohamed Said: Yeah, you have to create a class with the event or you can actually have a string event, an event in a string form. Matt Stauffer: I read about that. Mohamed Said: You can just dispatch an event that says string. And then you register listeners to this string event. A listener is basically another class or a closure, you can register a closure to be a listener to a specific event. And you can actually register a closure, register another, a listener class, to listen to all events on your application. Mohamed Said: So a listener can listen to multiple events, and it will get executed. The way things work, or the way Laravel works is that when you dispatch an event, Laravel is going to check the event dispatcher to see all the listeners registered to this event. Or if there are any listeners that are supposed to listen to all the events on your system. Mohamed Said: And it's going to collect these listeners, and it's going to execute them one by one. That's basically how it works. Matt Stauffer: Got it. So each of those, when they're executed, they are past this particular event. So if the event is just a string thing happened, then cool, they'll respond to it. If your event actually is a class, it might have some data attached to it. Matt Stauffer: For example, in my example, there might be the user ID, or anything else that came back through. So that event listener will be past the event, and if optionally any data that's attached to it, and then they'll act on it. Matt Stauffer: So in my circumstance, one thing I could do, let's say there's 20 different webhooks that I might actually be getting back from Stripe. I could handle all those in a controller, I could create a custom class for handling them. Matt Stauffer: Or I can dispatch a Stripe webhook received event that has all the data we got back. And then there can be listeners for individual different types, or whatever it is. So there's ways for you to basically build these architectures such that you're listening to these events, and it's not happening directly in the controller. Matt Stauffer: So one of the benefits of that is, let's say you've got all these different things happening, it's not all just, "Oh, we need to add one more thing happening every time the user registers. We're going to add a new synchronous action directly in the user registration controller." Matt Stauffer: But you also mentioned the downsides, which is, especially for new folks in the code base, it's really hard to understand what's actually happening and when and why, which especially for those of us who are used to just walking through it and seeing everything happening and looking at it, it feels much more disconnected. It feels a lot harder to debug these things. Matt Stauffer: You mentioned that you tend to not prefer them. Are there any circumstances in which you find yourself still saying, "Well, this is the one circumstance in which I do use events?" Or you just don't use them at all? Mohamed Said: Well, I try to avoid them, because it's easier to track down what's going on inside the code that runs synchronously when it's all in one place. But I imagine if there are things that are common, or things that should happen when several other things happen, for example, a listener that listens to when a customer is created or updated or deleted. Or a listener that listens to a subscription that's created or updated or deleted. Mohamed Said: For this kind, it's better to write it in one place, and then dispatch or run this code in several places, or on these several occasions. But since it's a piece of code that has to run on several occasions, it doesn't make sense to put it all in one place. Mohamed Said: So in this situation, it might be a good idea to put it in a listener and register the listener to this event. Matt Stauffer: That makes more sense. Mohamed Said: So you know that when this or this events are fired, your code or your listener will get executed. But some people out there, they build the entire application event page. They always dispatch events, and they do everything in listeners. Mohamed Said: I find this very hard to follow what the application does. If I have to work on a code base that's four or five years old, that it's all built in events and I sit and try to understand how things work, it's going to be very, very difficult. That's why I'm actually not a very big fan of using the event system... Mohamed Said: Inside the application, I understand the importance if you are having microservices, so several applications listening to events and they get executed. For example, you may have a PHP application and a Rails application or a Python application, and you fire an event in the PHP application. The Python application listens to this event and executes it. That's a legit way of using events. Mohamed Said: But if it's the same application, the same code base, you don't have to dispatch events and listen to all these events and several listeners. It's going to be hard to follow what's going on inside that single application. You understand what I am saying? Matt Stauffer: Yeah. I love that you just said microservices, because that's where my brain was going. I was thinking using events sparingly, sometimes it makes sense, you gave a really great example. But using events extensively seems like it's trying to solve the same thing that microservices are trying to solve, which is different teams, different groupings, different pieces of your architecture communicating to each other in what feels like a more architecturally pure way. Matt Stauffer: But majority of the time ends up... It's not a bad tool, but preemptive use of it, or using it when it's not necessary, makes things unnecessarily complex without actually getting the value you're getting. Matt Stauffer: I can imagine if someone had a really, really massive team, and one part of the team was responsible for just one thing, and another was responsible for just another thing, that events might be a really reasonable way to work on it. Team one would say, you're basically treating them like hooks in something like WordPress. "We're going to be building our thing, our user registration flow, and you're not allowed touch it. But we'll send out events for certain things that you all are able to make your own listeners to." Matt Stauffer: So if you've got two entirely disparate teams, one of which is responsible for this, one of them is responsible for that, events could be a really helpful way for those two teams to collaborate communication. But if you don't have two teams, forcing everything through an event system just makes it unnecessarily complex and distant. I think that's a really great note there. Mohamed Said: Yeah. And with the example that there are several teams, there is a difference between, when using several teams, you can actually dispatch jobs that other parts or other teams that execute it in other teams' code. For example, as you said, when the user is registered, you send, for example, create account job. And that job is written by another team that handles the account creation. You don't have to worry about it. Mohamed Said: But the difference in this situation, the difference between dispatching jobs and dispatching events is that a job is like an event. But it can only have one listener. An event on the other hand can have several listeners. Mohamed Said: So if you send the job of user registered, only one team can execute the job and create the account. But if the situation that you have several other teams, each want to do something different, and you dispatch a user created or user registered event, several teams can listen to this event and each team do its jobs. So that's the main difference between the two concepts. Matt Stauffer: That's a great differentiation. I think it's a good note that there's a lot of circumstances in which some of us might naturally want to write an event and Java is just fine there. And so you're helping us draw the line of when do you actually need the event? Matt Stauffer: I'm building something right now, I now remembered, where a user can come back from Stripe Connect, and not actually have finished their Stripe Connect onboarding flow. And so Stripe Connect puts the responsibility on us as the application authors to not just take that user's information back, but also treat it as an incomplete state. Matt Stauffer: And there's multiple different times which we need to be checking back with Stripe, including listening to webhook to figure out whether it's complete or not. And so in that, I have basically six different times in the application where I basically go to Stripe and say, "Hey, is it done yet? Hey, is it done yet?" Matt Stauffer: And so I could, in multiple different circumstances, say something happened that might merit us checking whether or not they're done yet. That could be an event that has a listener. But the thing is, that thing happening only ever has one action to take. Matt Stauffer: So in that circumstance, in that same line of code, dispatch new event, user may have done something that completed their thing at Stripe Connect, I also can just say, "Dispatch new." And then the job, dispatch the job, that is check if user has finished the application status. Matt Stauffer: And like you said, because there's only one listener there, writing it as a job, like you mentioned, I wrote the code in one place, which is that job, I can dispatch it in multiple places. But because there's only one listener in the job, I now can keep my code much clearer and more direct, because you never have to ask, "Oh, how many listeners are there in this event?" No, there's one. There's one job, the job is running, end of story. That's a great delineation. Matt Stauffer: If you don't need more than one thing responding to it, there's a good chance you don't need an event. Mohamed Said: There is a confusion between jobs and events and there is another confusion between jobs or the queues and the schedule component inside Laravel. So people sometimes ask, "Should I use the queue for this, or should I use a scheduled job?" The difference between them is a little bit confusing. Mohamed Said: The difference is the scheduled job, you put a specific schedule, that Laravel is going to check every minute and see if any of the jobs in the schedule need to run right now and they run it. They do it over and over. Mohamed Said: Like, for example, you say at the beginning of the day, send users a good morning email. I know nobody does that. But that's the example I have for you now. Matt Stauffer: That's a fun example, I like it. Mohamed Said: Yeah. So in the morning, you check each morning that your Laravel application is going to see that you have this scheduled at 9:00 a.m. to send to your customers a good morning email. But if you have to do it using a job, you can tell the worker to do something repeatedly. Mohamed Said: You just dispatch a job to the worker and you tell them say good morning, and it is going to say good morning. That's it. It's not going to do anything again until you tell it to say good morning another time. Mohamed Said: But as a scheduled job, it's going to repeat the instructions that you give it each time, the time for this instruction comes to run. So that's mainly the difference between the two. Matt Stauffer: I do like that you brought that up though, because one of the most common times I do find myself creating custom queued jobs is something that's running on a cron, that is massive. So every Sunday at three o'clock in the morning, for every single user in the entire application, do X, Y, Z. What happens when we have 50,000 users? Matt Stauffer: Now granted, there's some other solutions, but one of the first solutions you can read for is, every Sunday morning at 3:00 a.m. now go dispatch 50,000 jobs. So you don't have to worry about that one thing that's happening, once every Sunday timing out. You just loaded up your queue for the day and your queue workers got a fun Sunday ahead of them, without you having to worry about it timing out. Matt Stauffer: And what happens when you have 500,000? Well, give it a couple more queue workers. So queues and scheduled tasks can often have a really good relationship, but like you said, it's important to know the distinction between the two. Matt Stauffer: Okay, so we are getting close to the end of our time. I asked for you a couple things where people get stuck when they're working with queues. Are there any tips or high level things where you say, what are two things that you could say if only everybody in the Laravel world understood this about queues, I think life would be better? Is there anything else you have on your list like that? Mohamed Said: Well, I think there is another area of confusion that trips everyone is the concept of retrying jobs and job duplication. So for example, if you send a job to the queue, and that job is a long running job, you say it's a job that sends emails to several thousands of people, it's going to keep running for a long time. Mohamed Said: But there is a configuration in the worker that says by default, if I don't listen back, if a worker doesn't listen back from a job after 90 seconds from it starting, consider that this job failed. And then dispatch another instance of the job. Mohamed Said: So this 90 seconds is the default value. If you have jobs that are going to run for over 90 seconds, you have to change this configuration inside your queue, the PHP configuration file, or your job might get dispatched or might get run several times. This can lead to problems. Mohamed Said: Like for example, my example is saying good morning. Saying good morning once is nice, but saying good morning several times- Matt Stauffer: 17 times. Not so nice. Mohamed Said: Yeah, so that's a very common issue with people using queues when running jobs that run for a long time, is that they forget to update the retry after value in their queue configuration. So the job gets dispatched several times while it's still running on another worker. Matt Stauffer: Yeah. And that's very helpful. Because there's more than one configuration option that has to do with how long something can run. So often, you think to change one of them, but not the other. I've done that a lot, or I've helped clients get out of that a lot, where they say, "Wait a minute, I changed this one and yet somehow we're still getting this duplication," and it was exactly that. So that's a really great point. Matt Stauffer: Are there any other things you want to tip us off about, common pitfalls or super tips before we start closing up? Mohamed Said: Yeah, well, after every deployment, if you have workers running, you have to restart those workers, because as we said, those workers are long running Laravel instances. And if you deploy application, you make a change, these long running jobs will not get updated. You have to terminate these processes and start new ones, so the fresh instance of your application starts with the changes you made in the deployment. That's also another common issue. Matt Stauffer: Yeah, because if you don't do that, your application that's long running won't get the new code. It never got spun up with whatever your updated code is. Mohamed Said: Yes. So the HTTP part of your application will have the new changes, while the worker part is still running the old instance of your application with the old form or the old version. So we have to restart the workers for them to use the new version. Matt Stauffer: Am I right in remembering that you also need to restart it after maybe making any modifications to your environment variables? Mohamed Said: Yes. Because it's a single instance. Any changes- Matt Stauffer: It reads the file once. Mohamed Said: Yes, any changes that you made, this instance will not know about them until you restore the instance, you start the process. So a new environment is started, a new part of the memory on the server starts with new stuff and that's it. Matt Stauffer: Yeah, that's a good one. I've got some ideas here, including one that your name is attached to. But if someone were to want to learn a little bit more about queues or events, where would you recommend they go look? Mohamed Said: Well, the Laravel documentation. One of the things that everyone loves Laravel for is the great documentation. So everything can be found in the documentation. A lot of what I say today is already mentioned in the documentation. And it's like if people read the documentation top to bottom, every time they try to use a component of Laravel, they will avoid a lot of bugs and a lot of problems, because everything is in the documentation. Mohamed Said: And if it's not in the documentation, or if you find yourself trying to find another way of saying the same thing, because we all learn in different ways, we all understand things in different type of thing, if you find yourself want to learn about it in a different way than the recommendation, I wrote a book. And this book covers- Matt Stauffer: Yes, you did. Mohamed Said: Everything about the queues. It's an e-book in a PDF format. You can get this book, and it has everything I know about queues. If you don't want to read an entire book about queues, I also have a website called divinglaravel.com. And it's supposed to be covering everything about Laravel. But for the past couple of years, it's all about queues. Mohamed Said: Because that's my favorite component in Laravel. It's one thing that I spent, if I'm not working on Vapor and I'm not working on Spark, I'm putting all the focus on the queues, trying to enhance the queue system, trying to find ways to make it better, make it faster, make it easier, and so on. So that's my passion, the queues part. Matt Stauffer: I love it. So we will have links to your website and your book and all this kind of stuff all available in the show notes. And of course, if people want to hear more from you, where are you creating content? You mentioned Diving Laravel, where else are you? Mohamed Said: Well, I'm on Twitter. I'm also on YouTube. My handle on Twitter and YouTube are very simple. It's called themsaid. And it's basically a shortcut for the Mohamed Said. I did it when I was young. I thought it was cool to have my name as the Mohamed Said, the only. It sounds ridiculous now, but I'm stuck with it, I can't change it. Matt Stauffer: It works. It's easy to remember, it's easy to type. It's easy to read. So it's doing the job. I love it. I'm trying to think if there's anything else I wanted to make sure we covered. I think that was it. Matt Stauffer: So onto our personal moment. Last time we talked, we got to talk a lot about Regatta and biking and diving and stuff like this. But I don't think you were a father at that point. Matt Stauffer: I wanted to ask for you, first of all, I know you as a person, you are a kind and caring individual. I know that you're a great dad. But also even on your social media, you've been really open a lot about various aspects of being a father. Matt Stauffer: A lot of people who become parents for the first time, either get a lot of bad advice, or don't get any advice at all. So since you are fresh in here, I'm just going to ask you for the number one piece of advice that you wish that you had been given before having your first kid. Matt Stauffer: Because literally just in the last week, I feel like I've seen a half dozen people in the Laravel community mentioning they're about to have their first kid. I'm sorry if this feels more generic, but I'm really curious, what is the one piece of advice that you wish you had been given before you had your first child? Mohamed Said: I think the first piece of advice that I wish someone gave me is that not every child is the same. It's not- Matt Stauffer: Come on, that's the truth. Mohamed Said: It's not how you see it in the movies. Because in the movies, all kids are the same. There is a typical form in the movies about kids. And if you go outside and you meet people for one hour or two hours, people with kids, you also see a small part of what being a parent is like. You have to be prepared for the unknown. Mohamed Said: And another thing, most of the advice are bad advice, so it's better to not listen. Matt Stauffer: That's so true. Mohamed Said: Yeah, it's better to not listen to advice and actually listen to your kid. They have the answers, they just can't tell you the answers, you have to find it in them. Mohamed Said: It was very frustrating at the beginning that when we tried to apply any of the advices that we get, it doesn't work. But after a year and more, when we started listening to him, and when he actually started to... He doesn't speak yet, but he gives you cues about what he needs. When you listen and when you spend effort on listening on what they want and what they need, things get a lot easier. Mohamed Said: So don't listen to advice. Just listen to your kid. That's the advice I can give. Matt Stauffer: I love that. And it should be noted that I'm very wary of ever asking somebody to publicly give advice about kids, because of that exact same point. And I asked you because I knew... I didn't know whether you'd say this. But I know that you know that concept, right? Matt Stauffer: Like, just because something worked with my kid does not mean it's going to work with your kid. And so that means so many people who give advice are just giving advice about what worked for them, assuming that it's going to be the same for everybody. It's not. Oh, my God, that's so good. Most advice is bad, learn to listen to your kid. You heard it here, themsaid told us this. I love it. Matt Stauffer: All right. Is there anything else that we missed covering today, any other way that people can support you or any other aspects of queues? Anything else you want to cover before we cut out for today? Mohamed Said: I think we're good. But just again, I would like to say, mention it again, that people should really read the Laravel documentation, because it has everything. Unlike any other piece of software that's out there, the Laravel documentation has everything. If you read the Laravel documentation, you're going to build great applications. And you will sleep easier at night, knowing how each component works if you read the documentation, so read the documentation. Matt Stauffer: I agree. I love it. Mohamed, you are amazing. Thank you so much for working with me. For the rest of y'all who didn't know, I made him sit and wait in an empty Zoom Room for 20 minutes before this and he still showed up with a smile on his face, gracious and kind as always. So Mohamed, I can't tell you how much I appreciate you, the work you do for all of us and just you as a friend. You're a great guy and really appreciate your time today. Mohamed Said: Thank you for having me. I always enjoy talking to you, Matt. Matt Stauffer: Yeah, you as well. Thank you all and we'll see you all next time.