Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Logging Is the Unsung Hero of Debugging
Transcript
- Lucas: You know that moment when a test passes locally, passes in CI, but fails in production — and you have no idea why? Luna: The classic 'works on my machine' — except the machine is a server you can't SSH into. Lucas: Exactly. And that's where logging becomes the unsung hero of debugging. Not just any logging — structured, semantic, well-leveled logging. Today I want to talk about why your log strategy might matter more than your debugger. Luna: Bold claim. But I've definitely spent hours stepping through code locally only to find the bug was a race condition that only happens under load. Lucas: Right. A debugger gives you a freeze-frame of one specific execution path. Logs give you a timeline of what actually happened across millions of requests. There's a reason the first thing any SRE does during an incident is check the logs. Luna: So what does 'structured' mean in practice? I've seen teams just dump print statements everywhere. Lucas: Structured logging means every log entry is a machine-readable record — usually JSON — with fields like timestamp, severity, request ID, user ID, and a message template. Instead of 'User 12345 logged in', you get a JSON object: event equals 'login_success', user_id equals 12345, latency in milliseconds equals 42. Luna: That makes it searchable. You can grep for all events from a specific user session, or filter by latency above a threshold. Lucas: Exactly. And when you centralise those logs — say, with the ELK stack or a cloud-native service — you get real-time dashboards. I worked with a team that reduced mean time to resolution from four hours to forty-five minutes just by adding structured logging and a proper query interface. Luna: Forty-five minutes? That's huge. What kind of bugs were they catching? Lucas: Mostly transient failures — network timeouts, throttling limits, cache misses. Those are nearly impossible to reproduce in a test environment because they depend on exact timing and load. But a log entry with a timestamp and a stack trace tells the whole story. Luna: So how do you decide what to log? I've seen teams log everything and then drown in noise. Lucas: That's the trap. You need clear log levels — DEBUG, INFO, WARN, ERROR — and a policy for each. In production, you typically only log INFO and above. DEBUG is for development. But you should be able to dynamically increase verbosity for a specific service or user session when you're investigating an issue. Luna: Dynamic log levels — that's a feature I don't see enough teams use. Most just set it at deploy time and never touch it. Lucas: And that's a missed opportunity. A good observability platform lets you change log levels at runtime without restarting the service. So when a customer reports a weird error, you can flip a switch and get detailed DEBUG logs for just that session. Luna: That feels safer than enabling debug globally. But it still requires infrastructure. What about teams that are just starting out? Lucas: Start simple. Use a library like structlog in Python or Serilog in.NET. Send logs to stdout, and let your container runtime or a sidecar collect them. You don't need a full ELK stack on day one — even a simple log aggregation tool like Papertrail or a managed CloudWatch can give you 80 percent of the value. Luna: I want to tie this back to testing. How does logging help your test suite? Lucas: Great question. One of the biggest sources of flaky tests is race conditions — tests that pass or fail depending on timing. If your test harness logs every step with timestamps, you can pinpoint exactly where the timing diverges. Also, logs from failed tests can be automatically attached to bug reports, giving developers a head start. Luna: So your test suite itself should generate structured logs. That way, when a test fails in CI, you don't just see 'assertion failed' — you see the sequence of events leading up to it. Lucas: Exactly. I've seen teams reduce flaky test debugging time by half just by adding structured logging to their test framework. And it works for both integration tests and end to end tests. Luna: What about the trade-off? More logging means more I/O, more storage costs, potentially slower tests. Lucas: That's real. But modern logging libraries use asynchronous, non-blocking writes. The performance impact is usually negligible for most applications. And storage is cheap — you can set retention policies, like keeping DEBUG logs for 7 days, INFO for 30, ERROR for a year. Luna: So it's about being intentional. Not logging everything, but logging the right things in the right format. Lucas: Right. And the single most impactful thing you can do is add a unique request ID to every log entry. That one change makes it possible to trace a user's entire journey through your system, across microservices and queues. Luna: Distributed tracing is the next level. But even without full tracing, a request ID can tie together logs from different services. That's huge for debugging microservices. Lucas: Absolutely. And once you have that, you can start building dashboards for common failure modes. For example, if you see a spike in 503 errors from the payment service, you can drill into the logs and see if the pattern is a database timeout or a downstream API failure. Luna: That kind of visibility changes how you approach testing. Instead of guessing what might go wrong, you're guided by actual production patterns. Lucas: Which brings us back to the theme of this show: testing isn't just about writing test cases. It's about building confidence in your system. And good logging is one of the highest-leverage ways to build that confidence. Luna: I think a lot of teams underestimate how much debugging time they waste because their logs are unstructured or incomplete. If this episode gave you something actionable, consider supporting the show — it helps us keep the content ad-free and focused on practical advice. You can find us at buy me a coffee dot com slash fexingo. Lucas: Thanks for that. And to close out: next time you're writing a log statement, ask yourself — if this is the only clue I have during an outage, will it tell me what I need to know? If the answer is no, rewrite it. Luna: Great rule of thumb. Thanks for listening — we'll see you next time on Software Testing with Fexingo.