Latest / DevQuestions with Tim Corey / 325. What Data Should We Log/Trace And What Should We Ignore

Transcript

325. What Data Should We Log/Trace And What Should We Ignore · DevQuestions with Tim Corey

• 0:05
  SPEAKER_00: Logging lets us know where the problems are happening.
• 0:09
  SPEAKER_00: It also lets us see how our app is being used.
• 0:12
  SPEAKER_00: But if you aren't careful, logging can actually make it harder to understand where the problems are occurring and how your application is being used.
• 0:19
  SPEAKER_00: So let's talk about how to properly implement logging in a way that will actually benefit our application in today's episode of Dev Questions.
• 0:35
  SPEAKER_00: Software development is more than just writing code.
• 0:38
  SPEAKER_00: So let's talk about the rest of it.
• 0:40
  SPEAKER_00: Specifically, let's talk about how to properly implement logging.
• 0:44
  SPEAKER_00: And let's start with an illustration.
• 0:45
  SPEAKER_00: Back when I was a very new IT director at a small college, I was working with my network administrator to implement server logging.
• 0:54
  SPEAKER_00: Up until that point, we didn't have any central logging.
• 0:57
  SPEAKER_00: Errors were discovered by the users or, you know, but we stumbled across them, or maybe if we looked through the event viewer.
• 1:06
  SPEAKER_00: And we had a full network with Active Directory and Exchange Server and SQL Server and File Servers and all the rest.
• 1:14
  SPEAKER_00: So we we found this software that would centrally manage all these logs.
• 1:20
  SPEAKER_00: It would track the different things going on in our servers.
• 1:23
  SPEAKER_00: It would make sure that it looked out for things like, hey, you got low disk space on this server, or hey, this server is hitting a very high CPU all day long, or we've got a bottleneck in memory here, or whatever the case may be.
• 1:36
  SPEAKER_00: We're throwing exceptions or whatever.
• 1:38
  SPEAKER_00: So this software was great, and it could do so much stuff.
• 1:42
  SPEAKER_00: We went to training, we we figured out how to make the whole thing work correctly, how to get the most out of that software, and then we implemented it in I think it was like 20 servers.
• 1:53
  SPEAKER_00: Um and so we we implemented all these different servers, we had it, we set it up and configured it, so we made sure we looked at all the different errors, all different issues, and made sure that we were tracking all the things we needed to in order to have a really good bird's eye view of what's going on with the entire network.
• 2:13
  SPEAKER_00: And we barely ever used it.
• 2:14
  SPEAKER_00: And here's why.
• 2:16
  SPEAKER_00: Because the first day, my network administrator was getting the emails for all of the errors, all the big problems he'd get the emails for.
• 2:26
  SPEAKER_00: And his inbox was flooded to the point that he created a rule that said, hey, if it's coming from the logging set server, put it in a different folder because I can't even see my normal emails.
• 2:38
  SPEAKER_00: Then what do you think happened?
• 2:39
  SPEAKER_00: At first, we started going through those emails, but then some of them were like, hey, that that you know, that server is getting low on disk space, but we knew that, and that's not some address, or hey, that server is kind of you know going at 80% plus of CPU usage, but that's a good thing.
• 2:56
  SPEAKER_00: We want it to be there.
• 2:57
  SPEAKER_00: And so we started seeing all these issues where for some servers that would have been an issue, but for others it's not.
• 3:03
  SPEAKER_00: And so we started ignoring issues.
• 3:05
  SPEAKER_00: But then we had a hard time finding the issues that were really issues.
• 3:12
  SPEAKER_00: So what we found out is just because an issue seems critical in a vacuum doesn't mean it actually is.
• 3:19
  SPEAKER_00: And if you're not gonna do something about it, then it's not actually an issue.
• 3:24
  SPEAKER_00: So either critical alerts need to be dealt with or they need to be downgraded.
• 3:28
  SPEAKER_00: Otherwise, you drown out the real critical alerts.
• 3:32
  SPEAKER_00: And that's really hard to do because when you're looking at things, you're saying that is an issue.
• 3:38
  SPEAKER_00: But if you're not gonna deal with it, is it?
• 3:41
  SPEAKER_00: I mean, it may be something that you want to address.
• 3:44
  SPEAKER_00: It may be on your, I would love to take care of that list, but unless it's an issue you're gonna say, yes, I'm going to address that now, it's not really something you should have in your critical alert list.
• 3:56
  SPEAKER_00: And so what happened was his inbox was constantly full of alerts.
• 4:01
  SPEAKER_00: And we just never seemed to do a good job of figuring out which ones are critical, which ones aren't.
• 4:08
  SPEAKER_00: And so, yes, we found a problem when a user reported something, we could go to the log server and go, oh, yep, that issue was there.
• 4:15
  SPEAKER_00: We got the alert for it 18 times, but it got lost in the middle of the thousands of other alerts that were going off.
• 4:23
  SPEAKER_00: And so just because those things seemed like they should be logged doesn't mean they should have been.
• 4:31
  SPEAKER_00: We should have been logging just what we're going to fix.
• 4:35
  SPEAKER_00: So let's talk about how to set up logging that works.
• 4:38
  SPEAKER_00: Now, we're talking more about logging inside of your application.
• 4:42
  SPEAKER_00: So this is not exactly about server logs, but they the things still apply.
• 4:47
  SPEAKER_00: So let's talk about how to set up the logging that actually works for you.
• 4:51
  SPEAKER_00: Number one, only mark issues as critical if you absolutely need to do something about them.
• 5:00
  SPEAKER_00: Too many people say, well, that's a big deal.
• 5:03
  SPEAKER_00: Is it though?
• 5:04
  SPEAKER_00: Again, if it's a big deal but you don't do anything about it, it's not a big deal.
• 5:11
  SPEAKER_00: It that's just information at that point.
• 5:14
  SPEAKER_00: If you're not going to act on it, there's no reason to make it a critical issue, make it an informational issue.
• 5:21
  SPEAKER_00: And yes, I know sometimes you may go, but that's that's that's important.
• 5:26
  SPEAKER_00: It's only important if you do something about it.
• 5:29
  SPEAKER_00: That's really what it comes down to.
• 5:31
  SPEAKER_00: It doesn't matter what you say, it's what you do.
• 5:35
  SPEAKER_00: This is a fact for all of life.
• 5:38
  SPEAKER_00: If it doesn't matter what you say, it matters what you do.
• 5:41
  SPEAKER_00: So if you say this is critical, but then you do nothing, it's not critical.
• 5:46
  SPEAKER_00: So be very careful to only bubble up things that are critical, things that you're actually going to do something about.
• 5:56
  SPEAKER_00: Number two, only log errors as errors if they're unexpected or need to be addressed, such as, you know, like, hey, this service has crashed.
• 6:10
  SPEAKER_00: Okay, that needs to be addressed.
• 6:11
  SPEAKER_00: And that's that's an error that needs to be addressed.
• 6:13
  SPEAKER_00: Um, if there's an exception that happens, but it's not something we're gonna address, then don't log it as an error.
• 6:21
  SPEAKER_00: Um, even though the error logs are the ones that take the exceptions.
• 6:24
  SPEAKER_00: And you might go, well, I have to put them there.
• 6:27
  SPEAKER_00: No, you don't.
• 6:28
  SPEAKER_00: So you don't need to log the network connection was went down for the user.
• 6:34
  SPEAKER_00: You don't need to log that unless you need to know, oh, the network is going down in this area.
• 6:39
  SPEAKER_00: For example, if you have microservices and two different microservices, maybe one's an API and the other one's talking to the API, maybe that network connection is all internal.
• 6:50
  SPEAKER_00: And if that goes down, yeah, you should know about that.
• 6:53
  SPEAKER_00: And that would be something critical.
• 6:55
  SPEAKER_00: But if the user is on their laptop and they go through a tunnel and they lose the connection to the cell tower and it comes back up afterwards, that's not an issue that you need to be logging.
• 7:06
  SPEAKER_00: So even though it might throw an exception on the user's machine, even though it might be triggered as an error, you need to go, that's not really an error.
• 7:16
  SPEAKER_00: And so this is where being granular about catching these exceptions and going, this one's important, this one's not, is important.
• 7:26
  SPEAKER_00: If you're doing a uh a try catch and that's catching EX, you know, the raw exception ex where you're not specifying which exceptions and you're just saying, hey, make every one of these an error, that's not very efficient.
• 7:41
  SPEAKER_00: It's gonna clog up your errors and your logs.
• 7:43
  SPEAKER_00: So maybe at first you would do that, but then figure out, oh, we're getting these network drop-offs.
• 7:49
  SPEAKER_00: And those network drop-offs aren't really a big deal.
• 7:51
  SPEAKER_00: Well, then in your catch, have another catch before your generic exception ex that's like network exception or whatever it is that's the actual network error, catch that one and maybe let the user know, hey, you don't have a network connection right now.
• 8:08
  SPEAKER_00: But don't log that.
• 8:10
  SPEAKER_00: There's no reason to log that if you're not going to do something with it.
• 8:15
  SPEAKER_00: Number three, only log warnings that are precursors to actual errors or critical issues.
• 8:23
  SPEAKER_00: If there's something that is an indicator that something is going to happen in the future and it will be a major issue in the future, that's a great warning.
• 8:34
  SPEAKER_00: So a simple example might be, hey, I noticed we're getting low on disk space.
• 8:40
  SPEAKER_00: Okay?
• 8:41
  SPEAKER_00: That's something that should be a warning because that's leading up to, hey, we're out of disk space.
• 8:46
  SPEAKER_00: So, yes, that could be a good warning because, again, you're going to do something about it.
• 8:52
  SPEAKER_00: Maybe you won't do something about it right away, but you'll do something about it in a week or within the next few days as you get time, because that warning let you know before it became a critical issue.
• 9:05
  SPEAKER_00: So warnings are good to have, but again, don't log things as warnings that aren't precursors to actual errors or critical issues.
• 9:15
  SPEAKER_00: If it's just a, hey, the user doesn't like us, or, you know, hey, the user put bad data in, these are not warnings.
• 9:22
  SPEAKER_00: This is just information that you've elevated wrongly to a warning level.
• 9:28
  SPEAKER_00: So again, it has to be based upon something that you're going to do something about.
• 9:34
  SPEAKER_00: If you've noticed a theme so far, this is what logs are for.
• 9:38
  SPEAKER_00: Logs are not for the pack rats, the digital pack rats who want to store all this information.
• 9:44
  SPEAKER_00: That's not what logs are for.
• 9:46
  SPEAKER_00: Logs are for doing something with.
• 9:49
  SPEAKER_00: And if you're not doing something with your logs, you're not using them correctly.
• 9:53
  SPEAKER_00: So, number four, understand your application's importance and adjust what you capture and where.
• 10:03
  SPEAKER_00: What do I mean by this?
• 10:05
  SPEAKER_00: Well, not all applications are created equal.
• 10:08
  SPEAKER_00: Some applications are mission critical applications.
• 10:11
  SPEAKER_00: I worked on a software as a service for a company that their whole line of business was a software as a service.
• 10:17
  SPEAKER_00: If that software as a service went down, then we didn't get paid.
• 10:21
  SPEAKER_00: All of our customers wouldn't pay us because we're not actually providing them what they need.
• 10:26
  SPEAKER_00: The whole business would shut down.
• 10:28
  SPEAKER_00: That's mission critical.
• 10:30
  SPEAKER_00: And that application was logged differently.
• 10:32
  SPEAKER_00: Now, I've used other or built other applications where it supported the development staff, where it made deployments easier, or it made cleanup easier, or it made simple tasks, documentation easier.
• 10:45
  SPEAKER_00: These are not important applications.
• 10:48
  SPEAKER_00: These are support applications that we could get by without or that are nice to have, but no one is coming in the weekend to fix.
• 10:59
  SPEAKER_00: So you need to adjust your application's logs accordingly.
• 11:04
  SPEAKER_00: So just because this application has a critical failure doesn't mean it's it's crazy important to do.
• 11:13
  SPEAKER_00: Again, you're not coming out on a Sunday to fix this application if it's not an important application.
• 11:21
  SPEAKER_00: So you need to understand your application's importance to the organization and adjust your logging accordingly.
• 11:30
  SPEAKER_00: Also, where you log.
• 11:32
  SPEAKER_00: Because there's going to be the central place that everybody should look at.
• 11:37
  SPEAKER_00: Again, read your logs.
• 11:39
  SPEAKER_00: Um, everyone should look at that tells you what's going on, what's critical, what's an error, what's a warning.
• 11:46
  SPEAKER_00: Those are things you should be looking at probably on a daily basis.
• 11:49
  SPEAKER_00: You should be aware of what's going on and be fixing those things because they're priorities.
• 11:55
  SPEAKER_00: But if you have minor applications, they probably shouldn't go in the same system.
• 12:01
  SPEAKER_00: Maybe you should be logged to a text file, or maybe you should log to a secondary server that you check once in a while when you have time.
• 12:08
  SPEAKER_00: When you're looking to make a change to a secondary system, you also look and see, hey, are there any critical or error warnings in there that I should address as well?
• 12:18
  SPEAKER_00: So where you log is also important because also logging is expensive.
• 12:24
  SPEAKER_00: And you don't need to be logging lots of messages from non-critical systems into your main logging system that's going to cost more money.
• 12:33
  SPEAKER_00: And again, it's going to clog up that main system with a whole bunch of critical error and warnings that aren't actually critical errors or warnings that need to be dealt with right away.
• 12:46
  SPEAKER_00: So make sure not to clog up your systems with too many false positives.
• 12:52
  SPEAKER_00: Try very hard to have zero false positives.
• 12:56
  SPEAKER_00: Number six, do not capture information you're not going to act on every day until unless you're investigating an issue.
• 13:05
  SPEAKER_00: Okay.
• 13:06
  SPEAKER_00: So when you're capturing information, don't gather information you're not going to act on.
• 13:12
  SPEAKER_00: So if you're saying, well, I need to know what the user does when they use the application, do you?
• 13:18
  SPEAKER_00: Or is that more like a testing thing?
• 13:20
  SPEAKER_00: Or is that more like a sometimes thing?
• 13:22
  SPEAKER_00: And that's where you have things like information logs where you can turn those on just for investigating an issue, or just for um, you know, trying to figure out how users are using your application.
• 13:35
  SPEAKER_00: Not all the time.
• 13:36
  SPEAKER_00: So don't log information, don't capture things that you aren't going to act on.
• 13:42
  SPEAKER_00: That also means what types of data you capture.
• 13:46
  SPEAKER_00: So this might be, you know what?
• 13:49
  SPEAKER_00: We need to know that we're out of disk space, but we don't need to know what you were doing when you found out you're out of disk space.
• 13:56
  SPEAKER_00: We don't need to know that information.
• 13:57
  SPEAKER_00: We just need to know that the system went, hey, I try to save a file and we're out of disk space.
• 14:03
  SPEAKER_00: That's all you need to know.
• 14:04
  SPEAKER_00: You don't need to know who was who was trying to do something.
• 14:08
  SPEAKER_00: That's irrelevant information.
• 14:09
  SPEAKER_00: Capture the information you're actually going to act on.
• 14:12
  SPEAKER_00: And again, you can use those lower log levels for when you're investigating issues or investigating how the system is used, et cetera.
• 14:21
  SPEAKER_00: You can turn those on.
• 14:22
  SPEAKER_00: Maybe you log them somewhere else, or maybe just make sure they're filtered out of the main system and only used for those investigations.
• 14:32
  SPEAKER_00: Number seven, do not save data beyond when you will reasonably use it.
• 14:40
  SPEAKER_00: I have found that in most cases, 30 days is about all you need.
• 14:44
  SPEAKER_00: Because if you get beyond that, you're not actually using the data.
• 14:48
  SPEAKER_00: First of all, if you have a critical issue that stays in the critical issues list for over 30 days, you're not using the critical data system correctly.
• 14:58
  SPEAKER_00: You need to have addressed it before now.
• 15:01
  SPEAKER_00: That needs to be turned into a ticket that needs to be starting to be worked on right away.
• 15:05
  SPEAKER_00: So if you have data for more than 30 days, why?
• 15:10
  SPEAKER_00: Why is it there?
• 15:11
  SPEAKER_00: Why is it not converted into an issue?
• 15:13
  SPEAKER_00: For the most part, delete it after 30 days.
• 15:16
  SPEAKER_00: That's all you need at most.
• 15:19
  SPEAKER_00: So you can have a log system that turns over quite frequently that even handles things like hey, we had Christmas break and we're off for a week.
• 15:27
  SPEAKER_00: No problem, because you have 30 days.
• 15:29
  SPEAKER_00: You still have 21 days of usable data.
• 15:33
  SPEAKER_00: So there's still a lot of data there for you, even if you have breaks, even if you have people go on vacation, even if you have both combined at the same time.
• 15:41
  SPEAKER_00: Like 30 days is a lot of data if you were acting on it.
• 15:47
  SPEAKER_00: The times when I've seen people say, I need more than 30 days, is when they are not acting on it and they are only looking at it every once in a while.
• 15:56
  SPEAKER_00: And they say, Well, I need to know historically how things are working.
• 15:59
  SPEAKER_00: You should have been fixing these problems, so historically shouldn't matter, because you should have fixed these problems by now.
• 16:06
  SPEAKER_00: So, therefore, if you have a critical error or critical issue that's happened over 30 days long, then you're doing something wrong.
• 16:15
  SPEAKER_00: Or it's it's in a ticket and you're trying to fix it.
• 16:18
  SPEAKER_00: But at that point, it's already on tickets.
• 16:20
  SPEAKER_00: So why do you need it in a logging system?
• 16:22
  SPEAKER_00: So try not to save data beyond when it's reasonably useful.
• 16:27
  SPEAKER_00: Otherwise, what happens is you become a digital pack rat and you got so much information there that you just never use it, but it just keeps getting more and more expensive to store.
• 16:36
  SPEAKER_00: And it can actually cause some other problems where you are storing things beyond when you should.
• 16:42
  SPEAKER_00: So don't save data beyond when it's reasonably useful.
• 16:47
  SPEAKER_00: Number eight, do not log sensitive data unless it's absolutely necessary.
• 16:55
  SPEAKER_00: This is one that often trips people up, and this is where a lot of security breaches can come in because you're logging the sensitive data and taking it out of a secure system and putting it into a less secure system.
• 17:06
  SPEAKER_00: So, first of all, your logging system should be pretty secure.
• 17:09
  SPEAKER_00: And then you should also try not to include anything that's of person identified information or other sensitive data in your logs.
• 17:20
  SPEAKER_00: Yes, there's exceptions.
• 17:21
  SPEAKER_00: Yes, there are times when you might have to, but in that case, you need to make sure that your logging system is locked down secure, and it's also in compliance with all the regulations around forgetting a user if they ask you to.
• 17:36
  SPEAKER_00: That means you need to go through your log system and clean out those logs, referring to that user or change that data.
• 17:42
  SPEAKER_00: So being compliant is harder if you're logging sensitive information.
• 17:48
  SPEAKER_00: So maybe try not to.
• 17:50
  SPEAKER_00: And again, these regulations are there for a reason, because people are putting sensitive data in systems that aren't as secure because they think, well, it's just a logging system.
• 18:01
  SPEAKER_00: Or it's not as protected because, well, everyone has to have access to it, right?
• 18:06
  SPEAKER_00: And then next thing you know, you're leaking sensitive data and endangering your users because you were sloppy.
• 18:13
  SPEAKER_00: Don't do that.
• 18:15
  SPEAKER_00: So number nine.
• 18:18
  SPEAKER_00: Number nine is don't log to the browser's console in production.
• 18:23
  SPEAKER_00: This is one where you know, if you're using something like uh React or Angular or Vue, uh, or even like Blaze WebAssembly, where you are front-end fully.
• 18:33
  SPEAKER_00: Try really hard not to put console log statements there.
• 18:38
  SPEAKER_00: Use an actual logging service.
• 18:40
  SPEAKER_00: Now, sometimes that logging service is not available, where maybe you have a network issue or other thing where you need to log or tell a user in some way, try and tell the user directly.
• 18:50
  SPEAKER_00: Don't try very hard not to put it into the console log.
• 18:55
  SPEAKER_00: We really don't want to have to ask users to open up the developer console, look at the actual console message, and figure out the message says.
• 19:02
  SPEAKER_00: That's not good.
• 19:03
  SPEAKER_00: So if we're not asking them to do that, then why are you putting those messages in there?
• 19:08
  SPEAKER_00: Because the other thing is those messages can leak information about what's going on.
• 19:12
  SPEAKER_00: Now, everything you do in the browser is going to be unsecured, or at least unsecured as for that person.
• 19:21
  SPEAKER_00: So if that person logged in and they have their credentials, well, sure, they have access to all of their things.
• 19:27
  SPEAKER_00: But that also means that any code that runs in the front end is going to be completely exposed to the user.
• 19:33
  SPEAKER_00: They can read all the code, they can read all the data.
• 19:35
  SPEAKER_00: So it's not quite the same as logging like a C sharp applications information and displaying that.
• 19:43
  SPEAKER_00: You know, if you have a Blazor uh web app where you're showing the server side logs, yeah, it's more that's more information leakage than a console log would be.
• 19:52
  SPEAKER_00: But still, don't put it in the console logs.
• 19:55
  SPEAKER_00: Not if you can help it.
• 19:56
  SPEAKER_00: There are exceptions.
• 19:58
  SPEAKER_00: There are exceptions.
• 19:59
  SPEAKER_00: You know, no pun intended.
• 20:02
  SPEAKER_00: But there are exceptions to that rule.
• 20:04
  SPEAKER_00: But for the most part, try not to put um errors in your console.
• 20:09
  SPEAKER_00: Try and expose it to the user or send it to a logging system if you're going to act on it.
• 20:17
  SPEAKER_00: If your logs have too much information, they become useless.
• 20:20
  SPEAKER_00: If your application treats too many things as important, nothing is important.
• 20:25
  SPEAKER_00: And if you aren't addressing the actual issues as they come up, you're going to hide the real issues.
• 20:31
  SPEAKER_00: You need to make sure you're only looking at logs you'll actually act upon.
• 20:35
  SPEAKER_00: That's how you protect your applications.
• 20:37
  SPEAKER_00: That's how you improve your user experience, and it's how you ensure you'll actually get value out of your logs.
• 20:44
  SPEAKER_00: Thanks for listening.
• 20:45
  SPEAKER_00: As always, I am Tim Corey.