Latest / DevQuestions with Tim Corey / 186 What Do I Do When Production Goes Down?

Transcript

186 What Do I Do When Production Goes Down? · DevQuestions with Tim Corey

• 0:00
  SPEAKER_00: Welcome to the Dev Questions Podcast with Tim Corey.
• 0:03
  SPEAKER_00: Join us each episode as we tackle the questions you are asking about a career in software development, understanding the industry, and new technology.
• 0:12
  SPEAKER_00: If you're just starting out or you want to grow stronger as a developer, this is the place to get your questions answered.
• 0:19
  SPEAKER_00: Now, here's your host, expert developer and online educator, Tim Corey.
• 0:25
  SPEAKER_02: What do I do when production goes down?
• 0:29
  SPEAKER_02: What are the steps I need to take to make sure that I am being wise about how I act and get things up quickly while doing it right?
• 0:38
  SPEAKER_02: This is the question we're going to address on today's episode of Dev Questions, and it comes from a suggestion site.
• 0:44
  SPEAKER_02: And if you have a question you want to get answered, go to suggestions.imtimcorey.com and ask it there or upvote an existing question to hopefully see it answered sooner.
• 0:54
  SPEAKER_02: Okay, so the the first thing you do when production goes down is panic, right?
• 1:01
  SPEAKER_02: Like that's what your what goes through your mind is oh no.
• 1:05
  SPEAKER_02: So let's work beyond the panic and let's try and figure out how to address these issues in a way that is both quick but also the right way of doing things.
• 1:17
  SPEAKER_02: Because I have seen a number of times, and I've even done sometimes the panic mode where you you quickly make a change and makes things worse and people get mad at you, and now you're really panicking, and now you're making even more mistakes, and it just compounds the problem.
• 1:33
  SPEAKER_02: And that's not a great way of approaching a crisis situation like production being down or there being a bug in production that causes a massive outage.
• 1:44
  SPEAKER_02: So let's go through the steps that I go through when production goes down.
• 1:49
  SPEAKER_02: And it's good to think about these now, even practice them now, so that when a crisis does happen, you aren't scrambling to figure out what were those steps again?
• 2:02
  SPEAKER_02: So go through them now, practice them now, figure out what works best for you.
• 2:06
  SPEAKER_02: But here's the steps that I go through.
• 2:08
  SPEAKER_02: Number one, clearly identify the problems.
• 2:13
  SPEAKER_02: And this may seem obvious, like your production is down.
• 2:17
  SPEAKER_02: No, no, no, that's probably a symptom.
• 2:20
  SPEAKER_02: What's the root cause?
• 2:22
  SPEAKER_02: Is it that the website is down because the website is broken or because our host has a problem?
• 2:30
  SPEAKER_02: Or is it because one of our services is down?
• 2:33
  SPEAKER_02: Maybe your access to the database is no longer there and so your site crashes because of that.
• 2:40
  SPEAKER_02: What's the root cause, not just the symptom of the problem?
• 2:45
  SPEAKER_02: So identify the problem clearly.
• 2:48
  SPEAKER_02: That's step number one.
• 2:50
  SPEAKER_02: Find that root cause.
• 2:52
  SPEAKER_02: Then number two, identify the scope.
• 2:55
  SPEAKER_02: Does this affect everyone or just certain people?
• 2:59
  SPEAKER_02: Sometimes it might affect you, and you might go, oh no, everyone's having this problem, but you're maybe different for some reason.
• 3:08
  SPEAKER_02: For example, I've had problems where I've been fine and I've figured that everyone's fine, but people on iOS who are using the Safari browser, for example, that doesn't work.
• 3:20
  SPEAKER_02: And I have to figure out, oh, let me try my Safari browser and see if that works.
• 3:25
  SPEAKER_02: And it doesn't.
• 3:26
  SPEAKER_02: And I go, okay, now I've identified that it seems to be Safari as a problem.
• 3:31
  SPEAKER_02: Not saying it's Safari's problem, but that might be the scope.
• 3:34
  SPEAKER_02: So we try on Firefox and Chrome on iOS.
• 3:38
  SPEAKER_02: And then we try it on other devices, like maybe a Mac with Safari, as opposed to an iPhone.
• 3:45
  SPEAKER_02: And then we try it in different places to find out what's the true scope of this.
• 3:50
  SPEAKER_02: Identifying the scope will not only help you know how many people are affected, but it will also help you start to diagnose what the real what the real issues could be.
• 4:02
  SPEAKER_02: Because if you know that this works on Chrome and it works on Edge and it works on Firefox, but it doesn't work on Safari, well, you got a pretty good clue of where to start looking for something.
• 4:13
  SPEAKER_02: Or maybe you find out that it works fine in the US, but it doesn't work in Europe.
• 4:21
  SPEAKER_02: So what's the difference there?
• 4:23
  SPEAKER_02: Or maybe it works in your building, but not outside your building, or outside your building, but not inside your building.
• 4:29
  SPEAKER_02: There's lots of different things you can identify from that.
• 4:33
  SPEAKER_02: For example, I've seen cases where people say, oh no, the website's down.
• 4:37
  SPEAKER_02: And we look and sure enough, the website's down, but it turns out it's because it was our firewalls problem.
• 4:44
  SPEAKER_02: Well, that doesn't affect the rest of the world.
• 4:46
  SPEAKER_02: We don't have to panic quite as much because it's an internal issue, not a customer-facing issue.
• 4:52
  SPEAKER_02: It's we've got a badly configured network rather than a broken product.
• 4:59
  SPEAKER_02: So clearly identify the problem and identify the scope.
• 5:03
  SPEAKER_02: Those are the first two steps of this process.
• 5:06
  SPEAKER_02: Number three is communicate to the affected people and to leadership.
• 5:12
  SPEAKER_02: Let them know what's going on and that you're working on it.
• 5:15
  SPEAKER_02: Now you don't know what the the solution is yet.
• 5:19
  SPEAKER_02: You don't know how long it's going to take yet, but starting off that communication early is important.
• 5:26
  SPEAKER_02: Otherwise, people send lots of help tickets into the help desk or or they panic and they call you, or they're just frustrated because they think that no one knows this is happening.
• 5:38
  SPEAKER_02: So if you clearly communicate, then you get ahead of that and say, yes, we're working on it.
• 5:44
  SPEAKER_02: Yes, we understand there's a problem.
• 5:45
  SPEAKER_02: This is what we're seeing so far, and we're working on it, letting people know.
• 5:50
  SPEAKER_02: And with leadership, letting them know, hey, we're on this, and these are the steps we're taking.
• 5:56
  SPEAKER_02: That way they know what's going on and that you are engaged in the problem and don't feel like they have to then uh lead you and try and insert themselves and make things potentially worse.
• 6:08
  SPEAKER_02: So you can clearly communicate we're on track, we know what's going on, and we're working towards a fix, and these are the things we're doing.
• 6:16
  SPEAKER_02: That way they can have confidence in you and your ability to get the job done.
• 6:23
  SPEAKER_02: Now, next up, look for a real solution.
• 6:27
  SPEAKER_02: So when I say real solution, there are some times when maybe production is down and you go, okay, well, we're going to do this, and it's kind of a workaround patch that'll get us back up, we think.
• 6:39
  SPEAKER_02: Look for the real solution first.
• 6:41
  SPEAKER_02: Try and figure out is there a way to solve this long term, not just throw a quick patch in here, because you might be creating another problem and you might not actually help yourself in the long run.
• 6:58
  SPEAKER_02: Yes, it might get production up now, but it might cause a more allergist later.
• 7:03
  SPEAKER_02: And that's not really a great look either.
• 7:05
  SPEAKER_02: So if you can, identify the real solution first.
• 7:09
  SPEAKER_02: If you absolutely have to, then identify a workaround or a quicker pit uh fix that you then put in place and then figure out what the long term is going to be.
• 7:22
  SPEAKER_02: But try to find the real solution first.
• 7:25
  SPEAKER_02: Now, next step is to communicate again.
• 7:29
  SPEAKER_02: It's time to talk again to the affected people and the leadership.
• 7:33
  SPEAKER_02: And maybe don't do both.
• 7:34
  SPEAKER_02: It depends on who the affected people are, but at least on a status page somewhere, say, hey, this is we found, this is the thing that's broken.
• 7:43
  SPEAKER_02: Here's how we're working to solve this problem.
• 7:46
  SPEAKER_02: So people know what's going on.
• 7:48
  SPEAKER_02: And again, they feel like you're not just wasting time or it's just this long interrupted period of no news, you're giving them some type of information out of this.
• 7:59
  SPEAKER_02: Next up, and this is really important, test any change thoroughly.
• 8:06
  SPEAKER_02: You're often tempted to say, oh, this is the fix, throw that in production.
• 8:11
  SPEAKER_02: Bypass all of our checks, all of our testing process, just get this thing out there.
• 8:18
  SPEAKER_02: Don't do that.
• 8:19
  SPEAKER_02: And the reason why is because I have seen so many times where the solution looks good, but it causes a different problem, or it just doesn't work.
• 8:29
  SPEAKER_02: And you put it in production, and maybe production even comes back up and then it crashes again.
• 8:35
  SPEAKER_02: And that's the worst, is when you make things seemingly better, but then and people get kind of excited.
• 8:42
  SPEAKER_02: Oh, it's back up, I can start working again, and then it crashes again.
• 8:45
  SPEAKER_02: It makes it people feel worse than if you just didn't bring production up at all.
• 8:50
  SPEAKER_02: So test any change thoroughly.
• 8:52
  SPEAKER_02: This will also protect you against damaging data or causing other problems that have ripple effects that go beyond even the initial issue.
• 9:02
  SPEAKER_02: Don't make things worse with a bad fix.
• 9:06
  SPEAKER_02: Test it thoroughly.
• 9:08
  SPEAKER_02: Now, deploy that fix.
• 9:11
  SPEAKER_02: So once you have the fix tested and you are ready, deploy the fix, but monitor it closely.
• 9:18
  SPEAKER_02: Again, even if you test it, maybe you missed a long-term ramification of that change.
• 9:24
  SPEAKER_02: Maybe there's a reason why that wasn't in place in the first place.
• 9:28
  SPEAKER_02: So monitor to make sure that you're not generating bad data.
• 9:32
  SPEAKER_02: Because there's nothing worse, I don't think, than having a site that looks like it's working, but is generating bad data.
• 9:39
  SPEAKER_02: Maybe you know a form that people fill out goes to the wrong table or kind of is off by one with the with the linking.
• 9:48
  SPEAKER_02: And so now you have uh mismatched data, or you're updating the wrong data.
• 9:53
  SPEAKER_02: That's really bad.
• 9:54
  SPEAKER_02: So monitor that fix to make sure that not only is the is production back up and running, but it's up and running correctly.
• 10:02
  SPEAKER_02: So monitor that fix closely.
• 10:05
  SPEAKER_02: And at this point, communicate again.
• 10:08
  SPEAKER_02: You may be saying, Tim, you've got to communicate in here multiple times, and I do for a reason.
• 10:13
  SPEAKER_02: Communication is really important.
• 10:16
  SPEAKER_02: This is what's going to affect how people think about this downtime.
• 10:22
  SPEAKER_02: It's going to affect how leadership thinks about your department.
• 10:26
  SPEAKER_02: Are you a bunch of slackers who are just writing bad code to feel because you feel like it?
• 10:30
  SPEAKER_02: Which of course isn't true, but that can be the perception if you're not talking about what's going on, they start filling in the gaps in their own mind.
• 10:39
  SPEAKER_02: So communicate often, communicate clearly.
• 10:43
  SPEAKER_02: That way people know what's happening and know what you're doing to fix it.
• 10:48
  SPEAKER_02: Now, next up, document the problem and the solution.
• 10:54
  SPEAKER_02: Write down what the problem was and how you solved it.
• 10:58
  SPEAKER_02: That way you can again communicate that later.
• 11:01
  SPEAKER_02: This is what happened, this is what you did to solve it.
• 11:04
  SPEAKER_02: But then the next point is identify how you can prevent this from happening in the future.
• 11:10
  SPEAKER_02: So was it a bad process somewhere?
• 11:14
  SPEAKER_02: Was there a missing test that you could do in your CIC pipeline that could catch something like this?
• 11:20
  SPEAKER_02: Was it a misunderstanding of how production works?
• 11:24
  SPEAKER_02: What was it, and how can you prevent it from happening again?
• 11:29
  SPEAKER_02: So after that, so you've got the what the problem was and how it was how it was fixed, and then what you can do to prevent this from happening again.
• 11:39
  SPEAKER_02: And next, identify any part of this process.
• 11:43
  SPEAKER_02: The production is down, oh no, what do we do?
• 11:46
  SPEAKER_02: This process.
• 11:47
  SPEAKER_02: Identify any part of that process that you could streamline or improve.
• 11:52
  SPEAKER_02: For example, what if you know during this process you're trying to contact people but you didn't know who to contact?
• 11:58
  SPEAKER_02: Well, having a contact list of here's the key players that need to be contacted in case of emergency.
• 12:05
  SPEAKER_02: That could streamline your process and make it faster.
• 12:09
  SPEAKER_02: So when production's down, you have a faster process that shortens the downtime.
• 12:15
  SPEAKER_02: Or what if communication between your team members was important and you didn't feel like you did a good job?
• 12:22
  SPEAKER_02: Maybe establishing a process of opening up a Zoom call or a Teams call where you keep a call open and you can have that kind of group conversation while you're individually working on different parts or discovering different parts.
• 12:35
  SPEAKER_02: That might be something you could do to improve the process.
• 12:38
  SPEAKER_02: Or having better familiarity with the debugging tools.
• 12:42
  SPEAKER_02: So you figured out there's a problem, but you had a hard time identifying how to really key in on what the specific problem was.
• 12:51
  SPEAKER_02: And knowing how to better use those developer tools would have been helpful.
• 12:55
  SPEAKER_02: Identifying all those things will help you learn how to make this process shorter next time.
• 13:04
  SPEAKER_02: So the next step would then be to implement those changes.
• 13:08
  SPEAKER_02: All right.
• 13:09
  SPEAKER_02: So once you've implemented those changes, the last step here is to communicate again.
• 13:16
  SPEAKER_02: Communicate what went wrong, how you fixed it, how you're going to prevent this from happening in the future, and how you can make the process of fixing a downtime issue better for the future.
• 13:30
  SPEAKER_02: Communicate that clearly and also communicate that you actually did it.
• 13:34
  SPEAKER_02: So communicate that clearly to your leadership.
• 13:37
  SPEAKER_02: So that you're not just saying, hey, we had downtime, we fixed it.
• 13:41
  SPEAKER_02: But instead you're saying, hey, we had downtime, we fixed it, but we also fixed the the gap that caused the downtime, the problem that was the kind of the thing that let that that issue through into production.
• 13:55
  SPEAKER_02: And we also figured out how to be better at this process next time.
• 13:59
  SPEAKER_02: You're communicating competence.
• 14:01
  SPEAKER_02: You're communicating that not just we allow downtime to happen, but we take it seriously and we want to have less of it, and we're working towards that solution.
• 14:14
  SPEAKER_02: No one's perfect, but if you are showing that you're trying to improve for the future, that goes a long way when it comes to how leadership perceives you and your team.
• 14:25
  SPEAKER_02: Okay, so those are my thoughts on what you do in a crisis when production is down and you are kind of in that panic mode where don't panic, but we're in that mode of, oh no, we have to do something quickly.
• 14:40
  SPEAKER_02: Those are the steps to accomplish this task of getting back up and running well in a way that helps you for the future as well.
• 14:49
  SPEAKER_02: All right, so those are my thoughts.
• 14:50
  SPEAKER_02: Thanks for listening.
• 14:51
  SPEAKER_02: And as always, I am Tim Corey.
• 14:54
  SPEAKER_01: Thank you for joining us for this episode of Def Questions.
• 14:56
  SPEAKER_01: Tim is committed to making it easier for you to become a developer.
• 15:00
  SPEAKER_01: If you would like to help make more content like this possible, please like, subscribe, rate, and share Deaf Questions.
• 15:07
  SPEAKER_01: You can also send your questions to questions at IamTimcorey.com.
• 15:12
  SPEAKER_01: Until next time, remember, you are too smart and your time too valuable to waste it making all the mistakes Tim did.
• 15:19
  SPEAKER_01: When you're ready to learn to think and code like a professional developer, head over to imtimcore.com and enroll in a course.