Latest / DevQuestions with Tim Corey / 229. Is HA/DR Worth It? The Costs/Benefits of Disaster Recovery & High Availability

Transcript

229. Is HA/DR Worth It? The Costs/Benefits of Disaster Recovery & High Availability · DevQuestions with Tim Corey

• 0:00
  SPEAKER_01: HADR or high availability and disaster recovery are two separate but related concepts in software development.
• 0:08
  SPEAKER_01: Both are important and both are wildly misunderstood.
• 0:12
  SPEAKER_01: In today's episode of Dev Questions, we're going to talk about what these two topics mean, when to use them, and how these will affect your budget.
• 0:22
  SPEAKER_01: Everyone needs to consider high availability and disaster recovery, and everyone should have a plan in writing for when things go wrong.
• 0:30
  SPEAKER_01: Let's talk about it.
• 0:39
  SPEAKER_00: Welcome to the Dev Questions Podcast with Tim Corey.
• 0:43
  SPEAKER_01: Software development is more than just writing code.
• 0:46
  SPEAKER_01: So let's talk about the rest of it.
• 0:48
  SPEAKER_01: Specifically, let's talk about high availability and disaster recovery.
• 0:52
  SPEAKER_01: We'll start with the definitions.
• 0:54
  SPEAKER_01: High availability refers to an application's ability to keep running without failure.
• 0:59
  SPEAKER_01: For example, you might hear about a company having five nines of availability.
• 1:04
  SPEAKER_01: That means that the system will be up for 99.999% of the year.
• 1:11
  SPEAKER_01: In terms of time, that means your app would be down for no more than about five minutes per year.
• 1:19
  SPEAKER_01: 99.99% or four nines would mean no more than about 53 minutes of downtime.
• 1:26
  SPEAKER_01: 99.9% or three nines would mean no more than about eight hours 45 minutes of downtime.
• 1:34
  SPEAKER_01: As you can see, the the smaller the number of nines, the longer you can be down.
• 1:39
  SPEAKER_01: And high availability refers to how off or how much your application is up.
• 1:45
  SPEAKER_01: Now, obviously, five nines is what you would expect to want.
• 1:50
  SPEAKER_01: We'll talk more about that in a minute.
• 1:52
  SPEAKER_01: Disaster recovery, so the DR, refers to how fast you can get your application back up after it goes down and how little data you will lose in that time.
• 2:04
  SPEAKER_01: So let's say you push out a bad update.
• 2:07
  SPEAKER_01: So it corrupts all the new data that enters into the database, as well as any data that was modified.
• 2:14
  SPEAKER_01: You identify that something's going on.
• 2:17
  SPEAKER_01: How fast can you get your application back up and running in a good state with a clean database?
• 2:24
  SPEAKER_01: And then how much data are you going to lose?
• 2:27
  SPEAKER_01: So that would be disaster recovery.
• 2:29
  SPEAKER_01: How fast can you get these things up and running?
• 2:32
  SPEAKER_01: So out of disaster recovery comes two terms you should probably be aware of: recovery point objective or RPO and recovery time objective.
• 2:43
  SPEAKER_01: I know I'm throwing terms at you.
• 2:44
  SPEAKER_01: We're going to talk about this in practical terms, but I want you to understand RPO and RTO, recovery point objective and recovery time objective.
• 2:53
  SPEAKER_01: Recovery point objective simply means how much data will you lose in terms of hours.
• 3:00
  SPEAKER_01: If your database gets corrupted, do you only do backups once per day?
• 3:05
  SPEAKER_01: Well, then you would lose up to 24 hours worth of data.
• 3:10
  SPEAKER_01: So if you had a roll back in your database and you're only backing up once per day, you would lose up to, now not, you know, average would be 12 hours, but you lose up to 24 hours worth of data.
• 3:22
  SPEAKER_01: So your RPO would be 24 hours.
• 3:26
  SPEAKER_01: Recovery point objective is up to 24 hours of lost data.
• 3:31
  SPEAKER_01: That's you say we're okay with that.
• 3:34
  SPEAKER_01: Now you might have just twinged when you heard lose a full day's worth of data.
• 3:38
  SPEAKER_01: Again, we're going to talk about well, what does it mean to not do that?
• 3:42
  SPEAKER_01: Okay.
• 3:42
  SPEAKER_01: Now, recovery time objective, RTO, refers to how fast you can get a system back up and running.
• 3:49
  SPEAKER_01: Meaning, if your app is corrupted or if you have a hardware failure on a server, how fast can you get things back up and running?
• 3:57
  SPEAKER_01: That's a lot of definitions.
• 3:59
  SPEAKER_01: But what does have what does it have to do with your application?
• 4:02
  SPEAKER_01: Well, you need to identify what's acceptable in terms of how available your application will be, how many nines of availability, as well as how much data you're willing to lose in an event of an emergency.
• 4:16
  SPEAKER_01: Now, the standard answer from your boss will probably be that they want five nines of availability.
• 4:21
  SPEAKER_01: Again, that five minutes of downtime a year, with no data loss ever.
• 4:26
  SPEAKER_01: This is why understanding HADR is important.
• 4:30
  SPEAKER_01: You can get close to both of those goals.
• 4:32
  SPEAKER_01: However, the cost will be enormous.
• 4:36
  SPEAKER_01: This is why getting a plan down in writing before disaster strikes is so important.
• 4:42
  SPEAKER_01: Having your boss sign off on the choices now will save you later on.
• 4:48
  SPEAKER_01: So let's talk through the things you need to consider when coming up with a high availability slash disaster recovery plan.
• 4:55
  SPEAKER_01: One is backups, of course.
• 4:58
  SPEAKER_01: So replication isn't good enough.
• 5:02
  SPEAKER_01: Okay, so you might say, well, we don't need to have backups because you have another server that's replicating everything.
• 5:08
  SPEAKER_01: That's not good enough because there are times when you have a corrupted system.
• 5:15
  SPEAKER_01: Well, that means that corruption will get sent up to the replica as well because replicating.
• 5:21
  SPEAKER_01: So now you have two corrupted systems.
• 5:23
  SPEAKER_01: So if you have no backups or you think that a replica is a backup, then you got a massive glaring hole in your potential for downtime and for massive losses of data.
• 5:37
  SPEAKER_01: So replication isn't enough.
• 5:40
  SPEAKER_01: Good monitoring with review is going to be important.
• 5:43
  SPEAKER_01: Because imagine for a minute that you have an application that's corrupting data.
• 5:48
  SPEAKER_01: It's not down, but it's corrupting data that goes in the database.
• 5:51
  SPEAKER_01: Well, how long does that go on before you catch it?
• 5:54
  SPEAKER_01: Because imagine for a minute that you have, and this is a this is not good enough, but imagine you have a backup of once per day of your database.
• 6:03
  SPEAKER_01: So your thought is that you'd lose up to one day's worth of data if you had corruption.
• 6:10
  SPEAKER_01: But if you weren't monitoring your data closely and you roll out a change, let's just say you rolled it on a Friday, because why wouldn't you roll out on a Friday?
• 6:18
  SPEAKER_01: And that change rolls out, but starts to corrupt data.
• 6:23
  SPEAKER_01: And you don't catch that on Friday or Saturday or Sunday or even Monday.
• 6:29
  SPEAKER_01: And sometime on Tuesday, you go, oh no, we're corrupting data.
• 6:34
  SPEAKER_01: Well, because you weren't monitoring and didn't have a logging system that's kind of tracking that type of thing.
• 6:40
  SPEAKER_01: Well, now you can't just roll back to Monday's data because that's corrupted.
• 6:44
  SPEAKER_01: You can't roll back to Sunday's data, that's corrupted.
• 6:46
  SPEAKER_01: And if you're not storing those sequential day backups back far enough, you might be going back weeks in your backups.
• 6:56
  SPEAKER_01: If you only store, let's say, three days of daily backups and then three weeks of weekly backups, well, you might be going back two or three weeks.
• 7:05
  SPEAKER_01: And that's not a that's not good enough.
• 7:07
  SPEAKER_01: So having a good logging and monitoring system is important, and also making sure that you're paying attention to it because you need to make sure that it's not just are my apps up, but are my apps up and running properly.
• 7:21
  SPEAKER_01: This is also why testing is imperative.
• 7:26
  SPEAKER_01: Knowing that your application is going to work in the production environment and going to work correctly is important.
• 7:33
  SPEAKER_01: Being able to test and make sure that yes, it's going to have correct data.
• 7:37
  SPEAKER_01: It's not going to corrupt our data, it's not going to do something bad that's going to cause an irreversible change we have to roll back from.
• 7:45
  SPEAKER_01: So that's all part of backups.
• 7:48
  SPEAKER_01: Now, number two, depending on third-party systems.
• 7:52
  SPEAKER_01: So you might say, we want to have five nines of availability.
• 7:55
  SPEAKER_01: Well, if they break, you break.
• 7:58
  SPEAKER_01: So if you're dependent on API that is a third-party API, well, now if they go down, then that hurts your availability unless you have some type of fallback for that API.
• 8:13
  SPEAKER_01: So thinking through your dependencies on third-party systems is important.
• 8:17
  SPEAKER_01: But it's not just APIs, it's also things like your internet service provider.
• 8:23
  SPEAKER_01: So, for example, maybe you say, you know what, we're not going to host in the cloud.
• 8:26
  SPEAKER_01: It's a whole lot cheaper to host locally.
• 8:28
  SPEAKER_01: We've got these servers in the server room.
• 8:31
  SPEAKER_01: We're good to go.
• 8:32
  SPEAKER_01: Well, okay, so now you're dependent on one internet service provider, one power provider, one location, one building, one maintenance system, one fire suppression system.
• 8:46
  SPEAKER_01: These are all one things.
• 8:48
  SPEAKER_01: And what if one of those goes down?
• 8:50
  SPEAKER_01: Well, then your app goes down.
• 8:52
  SPEAKER_01: If your building power goes out, sure you might have UPSs to have some packet power, but do you have a generator too?
• 8:59
  SPEAKER_01: If you don't, that's going down.
• 9:00
  SPEAKER_01: And how long does that generator run for?
• 9:02
  SPEAKER_01: And is it the switchover clean and have you tested it?
• 9:05
  SPEAKER_01: If your internet service provider goes out and they say, oh, we ran a back over backhoe over the fiber line, it's going to take us three days of repair.
• 9:15
  SPEAKER_01: Well, now you have three days of downtime, unless you have a backup.
• 9:19
  SPEAKER_01: And it's not running the same trench as your current ISP.
• 9:24
  SPEAKER_01: So thinking through dependencies on third-party systems, not just software, not just APIs, but everything around your entire application is important.
• 9:35
  SPEAKER_01: Number three, thinking about application deployment process.
• 9:40
  SPEAKER_01: So some deployment processes take down the server and then do some upgrades and bring the server back up, which means you might have downtime of anywhere from a couple minutes to 10, 15 minutes.
• 9:53
  SPEAKER_01: Well, remember, if you have more than five minutes of downtime per year, then you've missed your five nines of availability.
• 10:02
  SPEAKER_01: So knowing how to deploy in a way that's not going to cause downtime is pretty important.
• 10:08
  SPEAKER_01: And if you can't do that, well, then you're gonna have to start knocking off nines of your availability.
• 10:13
  SPEAKER_01: And that could be okay.
• 10:14
  SPEAKER_01: Just to be clear, it can be okay to say we're a three nines of availability shop.
• 10:19
  SPEAKER_01: That's what we do.
• 10:21
  SPEAKER_01: So you don't have to stick with five nines, and five nines is very hard to do.
• 10:26
  SPEAKER_01: But you don't have to stick there.
• 10:28
  SPEAKER_01: But you have to communicate to your boss that we're not gonna do five nines, we're gonna do three nines, we're gonna do two nines, we're gonna do this.
• 10:35
  SPEAKER_01: So we're gonna have a downtime window of every Wednesday at midnight to 4 a.m.
• 10:40
  SPEAKER_01: Maybe that works for you.
• 10:42
  SPEAKER_01: I've worked at companies that do that.
• 10:43
  SPEAKER_01: They say we have a downtime window of this time every week.
• 10:46
  SPEAKER_01: That means your availability is pretty low.
• 10:49
  SPEAKER_01: If you're looking at the 99 percentage, you're maybe at 98% even.
• 10:55
  SPEAKER_01: But that could be okay.
• 10:57
  SPEAKER_01: Okay, but you have to think through your deployment process if you want to have high availability.
• 11:03
  SPEAKER_01: Because your deployment process itself could be more than the downtime you're allowed for the year.
• 11:10
  SPEAKER_01: Also, testing before deployment with real-world data and real-world hardware is important.
• 11:18
  SPEAKER_01: This is why I love the features of Azure.
• 11:21
  SPEAKER_01: Because in Azure, for example, I can deploy my website to a pre-production server where it's the same as production.
• 11:29
  SPEAKER_01: It has the same resources, it's the same deployment process, everything.
• 11:34
  SPEAKER_01: And I can test it and make sure that everything's working fine.
• 11:37
  SPEAKER_01: And when I'm ready, I can do what's called a slot swap, where I say, okay, the pre-production becomes the production, and the production becomes the pre-production.
• 11:47
  SPEAKER_01: And it switches those two over, and I don't have downtime.
• 11:51
  SPEAKER_01: So that right there allows me to test my deployment and go, oh, I broke something.
• 11:56
  SPEAKER_01: Don't, don't push the button.
• 11:58
  SPEAKER_01: We gotta fix that because when you're testing with real-world hardware and with real world data, you find different things than if you were testing in your development environment, which is probably different.
• 12:10
  SPEAKER_01: That's also why I encourage you to have a development environment that is as close to production as possible.
• 12:17
  SPEAKER_01: Take your production databases and back them up and restore them into development.
• 12:22
  SPEAKER_01: Clean some data up if you have to for safety or obfuscation purposes, but use real data because that's a whole lot better than your sample database that's that's smaller, it's got different data, it doesn't have the edge cases that your real life production does.
• 12:39
  SPEAKER_01: So testing as close to production as possible will help you reduce the amount of downtime you have from bad deployments.
• 12:48
  SPEAKER_01: Also, knowing how to roll back is really important.
• 12:52
  SPEAKER_01: Because if things do go wrong, hopefully you've tested everything, you've gotten as close as possible to the real world, and you've made sure that you put a good product out there.
• 13:01
  SPEAKER_01: But stuff still happens.
• 13:03
  SPEAKER_01: And if that's the case, well, knowing how to get back out of that is also important.
• 13:08
  SPEAKER_01: Because if you do all the good stuff up front but don't know how to roll back, well, that's gonna take you hours, even an hour is too long.
• 13:18
  SPEAKER_01: So you need to know how to roll back quickly.
• 13:20
  SPEAKER_01: So you say, oh nope, swap it back, bring back the old production because something went wrong.
• 13:24
  SPEAKER_01: We've got now two minutes of downtime, not an hour or two hours.
• 13:31
  SPEAKER_01: Number four, look for any centralized system.
• 13:34
  SPEAKER_01: So again, your building, your ISP, your physical location, because things like hurricanes, earthquakes, tornadoes, floods, these things affect central locations.
• 13:47
  SPEAKER_01: There have been numerous cases where uh you know a hurricane came through Florida and it flooded offices that now they need to get their servers out of there.
• 14:00
  SPEAKER_01: A hurricane came in New York City once and caused flooding, and there were certain major websites that have their servers physically located in New York City.
• 14:11
  SPEAKER_01: And that can cause a major outage unless you have a failback location.
• 14:16
  SPEAKER_01: So thinking through your centralization and saying, hey, maybe we need to think through diversifying this.
• 14:24
  SPEAKER_01: Because natural disasters happen.
• 14:26
  SPEAKER_01: Again, earthquakes, tornadoes, hurricanes, floods, you know, major power outages, there's a whole lot of things, fires.
• 14:35
  SPEAKER_01: These things all happen.
• 14:36
  SPEAKER_01: And if you're centrally located, even in one of the Azure data centers or one of the uh Amazon data centers, these things do go down.
• 14:48
  SPEAKER_01: There are still problems, even with their massive redundancies.
• 14:52
  SPEAKER_01: So knowing that, you might need to think about what we can do with multiple locations?
• 14:58
  SPEAKER_01: And there's lots more to cover when you're thinking through high availability and disaster recovery.
• 15:04
  SPEAKER_01: Okay?
• 15:05
  SPEAKER_01: Every application you build can go down.
• 15:09
  SPEAKER_01: What you need to evaluate is how much you're willing to pay to minimize those chances.
• 15:15
  SPEAKER_01: Think of it like buying insurance.
• 15:17
  SPEAKER_01: The bigger the insurance bill, the more protected you are.
• 15:21
  SPEAKER_01: You're never going to be fully protected though, but you can improve your odds to avoid problems.
• 15:27
  SPEAKER_01: So let's talk through a simple example and you can go from there.
• 15:32
  SPEAKER_01: So let's say you work for a company that's a software as a service company.
• 15:36
  SPEAKER_01: I've worked for a couple of those.
• 15:38
  SPEAKER_01: The entire business depends on that web application running.
• 15:42
  SPEAKER_01: If the application goes down, you immediately start losing money.
• 15:46
  SPEAKER_01: And you want to document your high availability and disaster recovery plan.
• 15:51
  SPEAKER_01: Right now, your app is running on one Windows server with IAS at your office.
• 15:56
  SPEAKER_01: Again, I've worked at a company that's done this as a software as a service.
• 16:00
  SPEAKER_01: Your SQL database is on a second server.
• 16:03
  SPEAKER_01: Full backups of your website are done once per day, and you do full SQL backups once per day as well.
• 16:09
  SPEAKER_01: You also backup your log files from SQL every four hours.
• 16:13
  SPEAKER_01: So, what do you need to consider for your high availability or disaster recovery plan?
• 16:18
  SPEAKER_01: Well, what if your database server has a hardware issue?
• 16:22
  SPEAKER_01: Smoke comes out of it.
• 16:24
  SPEAKER_01: How fast can you get a new server up and running with SQL installed?
• 16:28
  SPEAKER_01: How long will it take you to restore a backup to that new server?
• 16:33
  SPEAKER_01: And how are you going to get that new server in place?
• 16:37
  SPEAKER_01: So get a new server ordered, shipped, and installed.
• 16:40
  SPEAKER_01: Again, been there, done this, will probably take three days.
• 16:44
  SPEAKER_01: Now maybe a little faster, but let's just say three days.
• 16:48
  SPEAKER_01: Restoring SQL, restoring that data, and making sure the connections, trainings, and everything else will be set up properly will probably take another four hours.
• 16:55
  SPEAKER_01: That means you're looking at about a 76-hour recovery time objective or RTO.
• 17:03
  SPEAKER_01: You'll probably lose up to four hours of data.
• 17:05
  SPEAKER_01: And you might say, wait, 76 hours, but four hours?
• 17:07
  SPEAKER_01: Yes, because you did four hours up to four hours worth of work that had to get destroyed because you don't have a backup of it, because you have to roll back to the backups.
• 17:17
  SPEAKER_01: And you didn't do any work since then, but there's four hours where work was done that you have lost.
• 17:23
  SPEAKER_01: So you lost that four hours.
• 17:25
  SPEAKER_01: So your recovery point objective is four hours.
• 17:30
  SPEAKER_01: So unfortunately, from that one instant, you'd be down to 99.13% of availability or two nines.
• 17:38
  SPEAKER_01: So what can you do to make it better?
• 17:40
  SPEAKER_01: Well, you could replicate your database to Azure SQL and use that as a backup.
• 17:45
  SPEAKER_01: Now again, replication is not the only thing to do for backups.
• 17:49
  SPEAKER_01: But that could be a backup site.
• 17:52
  SPEAKER_01: So that would solve the issue of the hardware failing.
• 17:55
  SPEAKER_01: So if the hardware fails, you could just switch over to using the Azure SQL version.
• 18:01
  SPEAKER_01: And that might be seconds if you have an automated system in place, or minutes if you had to do it manually.
• 18:08
  SPEAKER_01: That's a whole lot better than 76 hours.
• 18:10
  SPEAKER_01: So that would solve the issue of the hardware failing.
• 18:13
  SPEAKER_01: You could do the same thing for the web server so that you're protected against the other server going down.
• 18:18
  SPEAKER_01: Next, you could be considering how long it takes to switch over, including pointing your domain name to the new location if you move your website.
• 18:26
  SPEAKER_01: After that, you should evaluate how to deploy your application.
• 18:29
  SPEAKER_01: What happens if you push out a bad update that corrupts your site?
• 18:32
  SPEAKER_01: That would corrupt Azure as well.
• 18:34
  SPEAKER_01: So if you have to roll back your code, can you do that?
• 18:37
  SPEAKER_01: Have you tested that?
• 18:38
  SPEAKER_01: Will it take hours to figure out the process?
• 18:41
  SPEAKER_01: Or can you automate that and say, just roll back?
• 18:46
  SPEAKER_01: So understanding that process and practicing it will be a big part of that as well.
• 18:51
  SPEAKER_01: So there's a lot of work that goes in a set up a high availability disaster recovery plan, but it's extremely valuable.
• 18:58
  SPEAKER_01: And you know what?
• 18:59
  SPEAKER_01: Sometimes you're gonna say that's too expensive.
• 19:02
  SPEAKER_01: And that's okay.
• 19:03
  SPEAKER_01: You don't have to do everything, but you have to think through well, if we don't do this, here is our vulnerability.
• 19:08
  SPEAKER_01: And if you put that in writing, your boss signs off on it.
• 19:11
  SPEAKER_01: Well, now you're both in agreement.
• 19:13
  SPEAKER_01: If bad things happen, bad things happen.
• 19:16
  SPEAKER_01: But you wanna make sure that you're not the one blamed for just not thinking about it.
• 19:21
  SPEAKER_01: When you've been asking for a backup server and your boss is like, I want a backup server, that's too much money.
• 19:27
  SPEAKER_01: You're what you're not telling them is if we don't have this, this could happen.
• 19:31
  SPEAKER_01: Give them the reasons for why you need these things and make sure they understand.
• 19:36
  SPEAKER_01: So you also don't want your boss to assume that nothing bad will ever go wrong.
• 19:41
  SPEAKER_01: Putting a plan like this in writing with the costs attached to each level of added protection is extremely valuable.
• 19:49
  SPEAKER_01: It helps you communicate to even non-technical bosses the costs and the benefits of what you can do.
• 19:57
  SPEAKER_01: And if you had to choose not to take on a specific high availability disaster recovery measure, then they will see that it will reduce your costs, but it will also leave you vulnerable to a potential issue.
• 20:10
  SPEAKER_01: They get to decide what to risk and what not to risk.
• 20:15
  SPEAKER_01: This is allowing you to put into your boss's hands the tools they need to make a wise decision.
• 20:23
  SPEAKER_01: But it also takes a lot of the burden off your shoulders.
• 20:27
  SPEAKER_01: Because when you're giving them the information, not just saying, we need three more servers, but we need to have these three servers because it will prevent against these potential issues, then they have a better chance to make an informed decision.
• 20:43
  SPEAKER_01: Now, there's a lot to this topic, but that should get you started thinking about high availability, disaster recovery, and what you need to do to evaluate your system before there's an issue.
• 20:54
  SPEAKER_01: I'll probably cover more of this in a future video.
• 20:57
  SPEAKER_01: I'll probably do a video where we'll do some graphics and we'll we'll draw some stuff out and maybe even create a plan for how to talk to your boss.
• 21:04
  SPEAKER_01: Let me know your thoughts on that.
• 21:05
  SPEAKER_01: Um, if you're on YouTube, put them down in the comments.
• 21:08
  SPEAKER_01: If you're listening to the podcast, well, go to YouTube and leave a comment there.
• 21:11
  SPEAKER_01: But thanks for listening.
• 21:13
  SPEAKER_01: As always, I am Tim Corey.
• 21:15
  SPEAKER_00: Thank you for joining us for this episode of Dev Questions.
• 21:18
  SPEAKER_00: When you're ready to learn to think and code like a professional developer, head over to IamtimCorey.com and enroll in a course.