Latest / The Laravel Podcast / Spatie's Laravel-Backup, with Freek Van der Herten
Transcript
- Matt Stauffer: All right. Welcome back to Laravel Podcast season five. As always, I'm your host, Matt Stauffer. And this is the season where we're talking about a package for every single one. And you know, if we're talking about packages in the Laravel ecosystem, we're going to talk to Frank. So, joining me today is Frank Van der Herten, the co-founder and big guy around town, it's Spatie who you've heard of before. Of course, we do Laravel stuff. So, Frank, would you say hi to the people? And, can you tell us a little bit about Laravel-Backup that we're going to be talking about today? Freek Van der Herten: Yes. Hello. I'm glad to be on. Yeah, you've thrown a lot of titles at me. Normally, I introduce myself as just a regular developer- Matt Stauffer: Just a regular guy. Freek Van der Herten: ... At a Belgium company. I'm a Belgium modest guy, little bit. But, I'm happy about, yeah, talking about the packages on this podcast. Matt Stauffer: Yeah. We all know Spatie's made a lot of packages. And so what I did, was I lined them up... But sorry, I should have not thrown this to you yet. I lined them up in order to most downloads, basically. And so, it turns out, that Laravel-Backup, according to the... I don't know if I used to look at packages or what, but Laravel-Backup has the most downloads volume. Does that line up with your numbers? I know you guys keep track of this stuff. Freek Van der Herten: I'm not keeping my eye on it from day to day, but I think there are a couple that are downloaded even more. Matt Stauffer: That's what I thought. Freek Van der Herten: Yeah. Matt Stauffer: I don't even know why I got this one at the top list. Either way, it's definitely popular, or it wouldn't have made it on this list at all. So, let's talk about Laravel-Backup. Can you give us the elevator pitch of this package? What is the main thing that it's solving? Freek Van der Herten: Yeah, I think it's named pretty well. Matt Stauffer: Clear. Freek Van der Herten: Yeah. This package can make a backup of your application. So, if your server just burns down, or maybe you accidentally just wipe out the S3 or your Vapor application, then this package can, yeah, just save you from tiers and worse, I should say. Yeah. So- Matt Stauffer: So, when you say backup, we can talk about a couple different things though. What are we actually backing up with this package? Freek Van der Herten: Yeah. You can basically choose what should be backed up. So there are two main things it can back up. It can back up files from the file system, and you can decide what gets backed up, and you can back up your database, because yeah, that's important of your application as well. And how it does that basically, it just dumps the database onto the file system. So, it's just a regular file. It puts that together with the other files in a zip file and it copies that to a external storage that you can configure yourself as well. Matt Stauffer: Yeah. So, I'm sitting here and I'm going, "I'm a modern programmer. I use git. Why would I need to back up my files?" Can you give me an example of some files I might actually want to back up? Freek Van der Herten: Yeah, that's a very good question, because yeah, people have asked me that before. Source code doesn't need to be backed up, because it's inversion control, but on your server there are a lot of other things that aren't backed up. Take for instance, maybe some media files that your users have uploaded, or maybe the environment file where every key that your application uses is listed in. I think if you imagine if your server would be gone, it might take you a while to just get all those values back. So it makes sense to have a backup of that file as well. Freek Van der Herten: Yeah. And the database, of course, because that's something that in most cases isn't inversion control at all. So, you really still need backups. And I think even when you use something like Vapor, where probably you use something like S3, or something for your user-generated content, it can't be bad, if it's not too much data to have a backup of that as well. Because yeah, maybe by accident you're deleting your S3 storage, or maybe one of your users just deletes their account, you delete all the files, but afterwards the customer wants to come back, and then you really want to have a backup. So, there are lots of scenarios where you're going to need this. Matt Stauffer: Yeah. So, normally I would not ask this question until later, but I know that there's a few people listening that are going, "Backup my environment variables. What on earth are you talking about? Security concern." Can you dip tiny little bit into, tell me about the security of the zip file. Am I worried about the contents of the zip file? Let's say, users upload their personal pictures in there. Let's say, we've got environment keys that are super protected. What does security look like when it comes to these backups? And then, we can go into the normal. But I felt like, that there's going to be a lot of people who until we answer that question aren't going to be able to listen to a single other word. Freek Van der Herten: Yeah. So, it's really an important question. And there are two ways for secureness. One is that the package can also encrypt the backups itself. And that's, yeah, a feature that's baked in the package itself. But what you can also do is, where you are copying your backup to, you can add some extra security there as well. I know that at Amazon, at S3, you can configure your backup in such a way that nothing can come out of it, that is just dropping stuff in there, and nothing can be read there. Matt Stauffer: Yeah. Freek Van der Herten: You can configure it, that all the contents on there should be encrypted as well. And basically you can use the features of the backup destination, yeah, to make sure that your backups are secure as well. But everything stands or falls with, yeah, keeping your credentials to the backup destination secure as well. That's a very good idea to do. Now, I'm not a security expert, but what security experts always tell me is that it's not necessarily a bad thing to keep your backups of everything in one place so that you have one thing to secure very well, as opposed to have 10 or 100s of different backup destinations, just pick one and just, yeah, secure it very, very well. Matt Stauffer: Fantastic job with that. No, that's really helpful to hear. Yeah, and I mean, yeah, AWS has three. For anybody who's not familiar, that's Amazon's file storage system. The permissions can be really confusing there. However, the default permissions these days, which didn't used to be the case, are nobody can read anything at all, which just like Frank just mentioned is wonderful, because if nobody can read it, the only thing you could do is write stuff in it. And you're actually going to have to modify the settings at all to ever even get something out of it. So, that might make it a little bit difficult to get something out maybe your first time. But the benefit of that is by default, nothing can be touched by anyone, or at least nothing from a public. Matt Stauffer: I can't remember the specifics, but regardless, the native S3 permissions are super, super, super locked down. So, even with somebody who doesn't have a ton of knowledge, you're not going to get in the circumstance where you're like, "Oops, I accidentally backed it up to a public URL." Or something like that. Literally you can't do that, unless you're pressing the wrong buttons. Freek Van der Herten: Yeah. Matt Stauffer: So, makes a ton of sense. Freek Van der Herten: Yeah. Indeed. And, I would really recommend that if you use S3 to store your backups, create a dedicated bucket for that. Matt Stauffer: Yes. Freek Van der Herten: Don't mix it with any other data or something. Just do something separately, make it as secure as possible. That's a good thing to do. Now, we're talking about S3 the whole time, but the backup package can actually backup to any destination that you want. We leverage Laravel's file system for that, a cloud file system. I don't know how it's- Matt Stauffer: Yeah, whatever it's called. Yeah. Freek Van der Herten: ... Marketed these days. But, it just leverages that. So if you want, you can also just back up to, yeah, another disk for, yeah, whatever vendor that you might like. So, I think in terms of security, you have a lot of options just because you just control where those backups are being stored. Matt Stauffer: Totally. All right. So, let's go back to the original agenda. The next question, is what is the history of this package? Where did it actually come from? Freek Van der Herten: Yeah. This is actually one of the first big packages that we've made. Matt Stauffer: Oh, yeah. Freek Van der Herten: And, I've made no secret of this, those packages they have always been made for our own projects and our own things. I can only create open source if I know that we're going to use it. And, this was one of the first. So, it ties a little bit with the history of Spatie itself. So, before that we used Laravel, we created yeah other frameworks, like Zend framework, or maybe no framework at all. And, we made small-ish websites. And, we didn't need dedicated hosting for that. We started out as a small agency and we just used shared hosting. And, yeah, it sounds really crappy. And- Matt Stauffer: That was the world those days. Freek Van der Herten: ... Yeah. In those days, it felt pretty good, because you didn't have to do anything about server management in your- Matt Stauffer: There was no forge back then. Okay? Freek Van der Herten: ... There was no Forge in this. Matt Stauffer: Yeah. Freek Van der Herten: And, something that you got with shared hosting is that your backups were made for you by your provider. So, you shouldn't care about backups too much, because you can just pick up the phone and ask you hosting provider, "I need that file back." But of course, yeah, we started to use Laravel. We are very early adopters of Forge as well. I think if Taylor would look up our user or team ID, I think it would have one or maybe two digits. We were really fast with that. Matt Stauffer: That sounds right for me too. Yeah. Freek Van der Herten: Yeah, of course, when you provision your service, we need to have backups of that as well. And I think one of the earliest hosting providers where Forge integrated with was DigitalOcean, and they had weekly backups. But, I felt that weekly backups wasn't good enough. If one of our customers would've accidentally delete something and I would say, "Yeah, we can only go back for a week. That wouldn't be enough." So then, we started thinking, "How can we backup our application?" And there basically wasn't anything in this space. So, we just, yeah, made our own solution for that. And yeah, that's where the idea was born. And I think in the time... This was when we were still in the Laravel four days, some things like the cloud file system, and the notification system, it wasn't in Laravel. So, in the earlier versions of this package, we're I think at version eight right now, I think in version one or two, you'll find that there is an old notification system in there and an old cloud file system- Matt Stauffer: Wow. Freek Van der Herten: ... In there. And for me, yeah, as Laravel grew, it was such fun to just throw away my own code and just fill it up with Laravel's native features. Matt Stauffer: Yeah. Freek Van der Herten: I still remember just throwing all the notification stuff and replace it with the native notification function. And yeah, it felt so good to do that. So, it's also a little bit of my playground to learn new-ish Laravel features. Matt Stauffer: I love that. That's really cool. I mean, that's one of my favorite parts of having open source packages, although just because of who I am, mine are more often open source softwares and service than packages that I get to go play around and have fun with, so. And they're always the experimenting place, "Oh, we're going to try..." Whatever, new framework, new tool, new pattern, because it's in public, other people can benefit from it, that I use this thing in a regular basis or whatever. So, totally get you there. And I love that. So, what's the installation story. So, let's say I've got a server right now. Freek Van der Herten: Yeah. Matt Stauffer: And it's not backing up the database and users can upload their profile pictures. And I would really like for those profile pictures to be saved, I'd like the data be saved. And it's a Forge server, and let's say, I'm going to use S3. What are the steps between where I am right now, and actually getting this thing up and running? Freek Van der Herten: Well, you'll be happy to know that there aren't too many steps involved. So, what you should do is install the package inside of your own Laravel application. And you can just do that with Composer. And then, in the configuration file, you can just choose, yeah, which files do you want to back up. And, do you wish to back up the database, then you have to configure a destination, which can be one of the disks that you configured in the configuration file system .PHP. Matt Stauffer: Mm-hmm. Freek Van der Herten: I think then, you should schedule the backup job. Matt Stauffer: Mm-hmm. Freek Van der Herten: And that's where you can basically choose the frequency of the backups. So, if you just schedule the backups to run once a day, then you have daily backups. If you schedule it, yeah, once an hour, then you get hourly backups. It's as simple as that really. Matt Stauffer: Yeah. Freek Van der Herten: And then, there are a couple of optional things that you can do, which I highly recommend doing. But with these steps that I've said, you're already backing up, so you already have backups somewhere. Matt Stauffer: Yeah. Freek Van der Herten: But after a while, yeah, you're going to store a lot of backups, right? And you don't need every backup, maybe. So, there's also built-in functionality to clean up the older backups. And that comes out of the box. It's also highly configurable, but by default we use something that's called, I think the Grandfather-father-son scheme. Matt Stauffer: Okay. Freek Van der Herten: And, I think if you Google that on Wikipedia, you get the formal definition. But if I had translated it to you, then it's very simple. It's just keeping a number of backups for an amount of days. After that period, just keeping a single backup for every week. Matt Stauffer: Yeah. Freek Van der Herten: So, the first eight days you have a backup a day, for the next four weeks after those eight weeks, you have a backup a week, then per month, then per year. Matt Stauffer: Yeah. Freek Van der Herten: And you don't need to do anything for that. You can just fill out how many backups do you want to keep daily, how many monthly. So, that just takes care of that. And there's also another functionality that I highly recommend setting up. And that is monitoring the health of the backups. Matt Stauffer: Mm-hmm. Freek Van der Herten: Because if something goes wrong backing up your application and you don't know about it. Matt Stauffer: Yeah. Freek Van der Herten: Yeah. Then, yeah. Then you're in some misery. Matt Stauffer: You're in trouble. Freek Van der Herten: And then, tears and worse is going to happen. So, what's also built in the package is a way of monitoring your backup. When is the last time you took the backup? And if it's older than, I think it's by default one day, then we will send you a notification. Yeah, notifications are also built-in, so we can notify you via Slack, mail, whatever you want basically, when there's a new backup being made, or this is the common use case, when something goes wrong, when there's no backup found for a day, or something went wrong taking the backup, you can immediately send a notification for that as well. Matt Stauffer: Yeah. Freek Van der Herten: So, I highly recommend just turning on those features as well. Matt Stauffer: Mm-hmm. Freek Van der Herten: And I think, but this is a little bit of an advanced thing already, you can set up the monitoring part of the package in a separate Laravel application. So, that it's another application monitoring all of your other applications. Matt Stauffer: Yeah. Freek Van der Herten: So, if your original server goes down, then your backup server will still say, "Hey, there's something wrong with your package." Matt Stauffer: I didn't know about that. That sounds like a cron check basically, but it's purely just for this particular webhook. Freek Van der Herten: Yeah. Yeah. Indeed. Matt Stauffer: Okay. Freek Van der Herten: Yeah. Matt Stauffer: Very cool. Is it a webhook, just basically set it up, so that each of your individual consumer sends a webhook out to the monitoring one, and then if it doesn't get that webhook, it gives an alert, or is it a different system there? Freek Van der Herten: It's a little bit different. It's not necessarily via webhook. It's the monitoring server, it's really going to go to your application and check where is it backuping to, I'm going to check if it has already done that. Matt Stauffer: Got it. Freek Van der Herten: Yeah. Matt Stauffer: Cool. That's awesome. I didn't know that. I mean, I'm trying to pretend like I didn't know these other things, but that I actually didn't know, so. All right. Very cool. Freek Van der Herten: Yeah. Matt Stauffer: So, you walked me through the workflow, what it's going to be like to set it up for the first time. So, I think one thing that folks who aren't familiar with the file system might ask is, "How is it getting access to these files? Is it getting access through a disk I have to set up? Or, can I just get it any directory I want in the entire level app, or any directory in the entire server? What does it look like to configure where those files are that I'm actually going to be making a part of the backup?" Freek Van der Herten: I think it's in the configuration file. It's been a while since I've checked it myself, actually. I think, in the configuration file, you can just say, "Hey, these directories need to be backed up." And you got me there. I don't really know if you can just back up another file system. It's been a while. Matt Stauffer: Yeah. Yeah. Again, I don't actually know the answer to this one, so. Freek Van der Herten: You know what? I'm going to just live check and I'm pretty sure it'll be in the configuration file. And it seems like it's only the local files that this package can back up. So, you can say, "Hey, I want to include these files." Matt Stauffer: Mm-hmm. Freek Van der Herten: And then you can say, "I want to exclude these ones." Matt Stauffer: Okay. Freek Van der Herten: And by default, we are going to include the entire base part. So, that's your entire Laravel application, but what we're not going to back up, what we're going to exclude is the vendor directory and the node modules, because probably you don't want to back up that at all. And then, you can specify stuff like, do you want to follow symlinks? If you have storage directory that isn't really in your application, but symlinked into the application, then we can follow the symlink and back up those files as well. I've already mentioned that this package is already at version eight, so there it's already a lot of polishing- Matt Stauffer: Yeah, for sure. Freek Van der Herten: ... Going on here. So, it's really battle tested. And, the features that people have requested and that we need it, that makes sense, I think at this point, they're all in the package. Matt Stauffer: Love it. Freek Van der Herten: Yeah. Matt Stauffer: That's awesome. Freek Van der Herten: Yeah. Matt Stauffer: Well, and that's a perfect transition to our next question, which is, are there any lesser used features that are really cool or really cool things you've seen people to do with this that are outside of the norm? Freek Van der Herten: I don't think so, really. It's a little bit of a dry package- Matt Stauffer: It does what it does. Freek Van der Herten: ... It has this aim and you just do what's on the box with it. Matt Stauffer: Exactly, what's on the tin. Yeah. Freek Van der Herten: Maybe, something worthwhile to know is that the backing up part of Laravel-Backup is a package on its own. It's called the DB-Dumper. Matt Stauffer: Okay. Freek Van der Herten: And if you want to hand roll some of, yeah, dumping databases, then you can use that package to do that. And in fact, we have another package that leverages that DB-Dumper as well. Matt Stauffer: Okay. Freek Van der Herten: I don't want to straight too far, but I'll say the basics of it. So we have another package, which is called DB-Snapshots. Matt Stauffer: Okay. Freek Van der Herten: And it's basically used for testing, where if you're developing, you can just dump your testing or your local database, use that in tests, and have a couple of different database. So, if you want to test a certain scenario, you can just dump and load up a certain version of your database. Matt Stauffer: That's very cool. Freek Van der Herten: And yeah, that makes use of the DB-Dumper of Laravel-Backup itself. So, I guess that's a little bit of a strange usage of some functionality- Matt Stauffer: Yeah. That's cool though. Freek Van der Herten: ... In the Laravel-Backup package. Matt Stauffer: That's very cool. And we'll make sure we'll link up Laravel DB-Snapshots in the show notes as well, if anybody's interested in that, so. Freek Van der Herten: Cool. Matt Stauffer: All right. So, are there any other aspects of this package before we move on to the development roadmap? Is there any other aspects of using it, or the history of it, or anything else that you want to cover before we're done? Freek Van der Herten: Yeah, I think, we got it all. I'm going here through the documentation. I think, we touched upon everything here, so. Matt Stauffer: All right. Freek Van der Herten: That's good. Yeah. Matt Stauffer: All right. Well, our next one is, do you have a development roadmap you'd like to share? It sounds like you're good, right? You said you're pretty happy with feature complete-ish. Freek Van der Herten: I think it's feature complete, but what we do with this package, and we do that basically with every package, is that we make sure that it uses the latest Laravel and PHP versions. Matt Stauffer: Mm-hmm. Freek Van der Herten: So yeah, that's basically the only thing that's on the big... Yeah, or the small roadmap for this. Basically, no feature span, because yeah, we don't need any features. And there aren't many things proposed. Yeah, like I said, we're already at version eight here. This package is six or seven years old, at this point. Matt Stauffer: Yeah. Freek Van der Herten: So, it's a grand daddy of packages. Matt Stauffer: You know what you wanted to do and it does it, so. Freek Van der Herten: Yeah. Matt Stauffer: I love it. Freek Van der Herten: I don't know if we need to delve in too much, but there's basically also another backup package that we have, that isn't too well-known, because it's one of our paid ones, which we don't market too hard, because yeah, we're still thinking about, "Should we make this a little bit bigger or not?" We're still thinking a little bit about it. And it's called the backup server. Matt Stauffer: Okay. Freek Van der Herten: And what this thing will do is a little bit different to what Laravel-Backup does. So, what Laravel-Backup server does, you'll find it already on our website, even though we don't talk about it too much, but we use it ourselves, is that, you install the backup server on one server. And that server will basically SSH into all of your other servers and backup all the other applications to it. And, it has something really incredible, I think, inside of it. And that is, data to data deduplication. So, if we take a backup and after a while you take another backup, and we see the contents of the file is the same. Then we're only going to take space of one of those files. So, it works a little bit like if you're familiar with that, like Mac OS's time machine. Matt Stauffer: Mm-hmm. Yeah. It's just basically the Delta. Freek Van der Herten: It's not really the diff, but it uses something like heart links. Matt Stauffer: Yeah. Freek Van der Herten: If it sees it's the same file, then it'll still have... You have backup one backup two, and inside, if you open up the backups, it'll say like, "Hey, here's a comparison of chasing. Here's a comparison of chasing." But, those two files will actually point to the same disc space on your disc. Matt Stauffer: Right. Freek Van der Herten: And it'll take care that if you delete one of them, then the other one will still work. So, it's a little bit different than diffing. And yeah, the first time that I got at working, it was really cool. Matt Stauffer: That's very clever. Freek Van der Herten: But the reason why we're not marketing it yet is because it has no UI yet. So, it's just the core functionality. And I think if we want to market this, it needs to be the whole package, that everybody can use it, even if you're not a programmer elite. So, yeah, it's maybe a little bit far off, but that's a little bit on the backup road map of Spatie. Just growing that a little bit more. Matt Stauffer: Well, the next question was, would you like to request any help support? So, you just gave me the one, the help and support is, if that gets released, then consider buying that, both because it may be helpful, and also because that's a way to say thank you for the free work that Spatie does in terms of open source packages, so. Freek Van der Herten: That's always a nice thing to do, I think. Matt Stauffer: Love it. Freek Van der Herten: I think, if some people want to contribute with Laravel-Backup, with a free version. Yeah, whenever there's a new PHP, or Laravel version come out just to help with just getting the source code even better, make sure that, that's run on the right things. You know what? I'm going to check out the issue tab. And yeah, Matt, it's really quite something. There are zero open issues and 760 closed one. There are no pull requests and 517 boom request handled. So, we are on top of things with this package. Matt Stauffer: Love it. Well, congratulations, because I don't think any of my repos have zero issues. So, you're winning right now. I love it. Well, I think that's it for my questions for today. Is there anything else about this package you wanted to cover today before we're done? Freek Van der Herten: I would recommend people to really use something like this. If you're not backing up, just do it, because one day your server will go down. It's really a question of when, not if. Matt Stauffer: Not when, yeah. Freek Van der Herten: And then, you really want to have this. I'll tell a very short story about this. Matt Stauffer: Mm-hmm. Freek Van der Herten: And this is something that if you Google it, you'll come across quite some stories. So, one day this happened to our company as well that, we were using DigitalOcean and they lost a server. So, it was just gone from one minute to the other, it's gone. Matt Stauffer: Wow. Freek Van der Herten: It doesn't exist anymore. Bam. And I don't want to, yeah, badge on DigitalOcean too much. If you Google around, you'll find similar stories for every host. These things can happen. They don't happen a lot, but if they happen to you, you really want to have backups. Matt Stauffer: Yeah. Freek Van der Herten: And not just a local backup, just going to copy the directory, no, your whole server can be gone. Matt Stauffer: Vanish. Freek Van der Herten: Yeah. Matt Stauffer: Wow. Freek Van der Herten: And this could be some human accident, maybe there's fire breaking out in the data server and it destroys the original and the backups. It could happen. So, just take backups. Matt Stauffer: Yeah. It's not free, but it's really close to free, because you're basically just paying for S3 storage. And it's well-recycled S3 storage. So, it's not even that much, so. Good call. Freek Van der Herten: Yeah. Matt Stauffer: I like that. Freek Van der Herten: Indeed. Matt Stauffer: Well Frank, thank you as always for all the packages you create, for the love and attention you give to them, and thanks for hanging out with me today to explain this package. I really appreciate you, man. Freek Van der Herten: It's been a pleasure. Thanks for inviting me, Matt. Matt Stauffer: Of course. And the rest of y'all. We'll see y'all next time. Freek Van der Herten: Bye-Bye.