Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / Why Testing Your Logs Prevents Production Nightmares
Transcript
- Lucas: Let me start with a story. In early 2025, a large e-commerce platform—one you've definitely bought from—deployed what looked like a routine back-end update. No UI changes, no new features. Just a fix for an internal inventory service. Luna: So the kind of deployment that usually gets a quiet green light from QA. Lucas: Exactly. The update passed all unit, integration, and end to end tests. But three hours after rollout, their logs started showing something strange: every few thousand orders, the quantity field for a random line item would silently become zero. Luna: Silently? As in the order still went through, but the customer got an empty box? Lucas: Worse. The system deducted inventory, charged the customer, but the picking list told the warehouse that item quantity was zero. So the warehouse shipped everything except that one item. The bug was only caught when a customer complained. And the team traced it back to a log message that looked slightly different than expected. Luna: A log message. That's fascinating. So the test suite missed it, but the logs caught the anomaly. Lucas: Right. And that's what we're talking about today—log-based testing. The idea is simple: use your application's production logs not just for debugging, but as a source of truth to generate test cases and detect regressions. Luna: That actually reminds me of something—before we go deeper, I want to mention that this kind of practical, real-world insight is exactly why we keep this show ad-free. We believe good testing knowledge should be freely accessible. And if you find value in episodes like this, you can support us at buy me a coffee dot com slash fexingo. It's a small way to keep the conversations going without any commercial breaks. Lucas: Absolutely. And your support really does make a difference. So back to logs—most teams think of logs as a post-mortem tool. You look at them when something breaks. But what if you treated logs as a continuous test input? Luna: You mean replaying real user sessions into a test environment? Lucas: Exactly. That's one approach. You capture logs from production—say from a load balancer or application server—parse them to extract request parameters, user actions, and response codes, then feed those into your test suite. It's like having thousands of real-world test cases generated for you every day. Luna: But production logs are messy. You've got noise—health checks, bots, internal calls. How do you filter that? Lucas: That's the key challenge. You need structured logging. If your logs are just plain text like 'User 12345 bought item 67890', parsing is fragile. But if you log in JSON—with fields like event_type, user_id, item_id, quantity, timestamp—then filtering becomes straightforward. You can exclude any event where event_type equals 'health_check' or user_id is in a bot list. Luna: So you're advocating for structured logging not just for observability, but for testing. Lucas: Precisely. And once you have structured logs, you can build a pipeline. Ingest logs from production, anonymize sensitive fields—like credit card numbers or emails—then store them in a format that your test framework can read. Then your regression suite runs not against hand-crafted test data, but against actual user journeys. Luna: And you catch silent failures like that inventory bug because the test expects a certain log pattern and flags deviations. Lucas: Exactly. In that e-commerce example, the fix had inadvertently changed the log format for the quantity field from integer to string. The application still worked—at the code level—but the downstream warehouse system expected an integer and silently defaulted to zero when it got a string. Luna: A classic type mismatch masked by silent coercion. Lucas: Totally. And a unit test wouldn't catch it because the unit test mocked the downstream system. Integration tests might have caught it if they used real warehouse endpoints, but those were stubbed too. Only the logs showed the discrepancy—but only if you were looking for it. Luna: So log-based testing would have caught it because the test would compare the log output from the new build against a baseline of logs from the previous build. Lucas: Exactly. You run your test suite with the same set of production log inputs on both builds. Then you diff the logs generated by each build. If the new build produces a log line that's different—say the quantity is a string instead of an integer—you flag it as a regression. Luna: That's elegant. But what about latency? Logging itself is asynchronous. If the test environment is slower than production, the log timestamps might not match exactly. Lucas: Good question. You don't compare timestamps; you compare logical sequences. You group logs by session or transaction ID. Then you check that for each session, the sequence of events is the same, and each event's payload has the same structure. That's more robust. Luna: Right. So it's about structural equivalence, not exact timing. Lucas: Exactly. There's a tool called LogDiff—open-source, built at a fintech company—that does exactly this. It takes a baseline directory of log files and a candidate directory, groups by session ID, and reports any differences in event types or field types. Luna: I've also seen teams use log-based testing for chaos engineering. They replay logs that contain edge cases—like a user with an empty shopping cart checking out—and see if the system handles it. Lucas: That's a great use case. In fact, that connects to something we talked about in episode 68—chaos engineering for APIs. Log replay is a way to inject real-world chaos without writing custom fault injection code. Luna: Let's talk about the downsides though. Log bloat. If you're capturing every production log, you're storing terabytes of data. How do teams manage that? Lucas: Sampling is the obvious answer. You don't need every session—you need a representative set. The e-commerce team I mentioned sampled one percent of all production sessions, which gave them about ten thousand sessions per day. That was enough to catch the inventory bug. Luna: And you have to be careful about PII. Logs often contain sensitive data. Lucas: Critical point. You should never log raw PII in production. But if you do, you need to scrub it before feeding into test. Tools like LogAnonymize can replace email addresses with fake ones, but keep the domain format—so 'alice@example.com' becomes 'user123@example.com'. That way you preserve structure without exposing real data. Luna: That's a good approach. So what's the first step for a team that wants to start log-based testing? Lucas: First, get your logs structured. If you're still using printf-style logging, switch to JSON logging. Most modern logging frameworks—like Python's structlog, Java's Logback with JSON encoder, or structured logging for Node.js—make this easy. Luna: Then you need a way to export logs from production to a test environment. Lucas: Right. You can use a log shipper like Fluentd or Logstash to forward logs to an S3 bucket or a local file. Then in your test pipeline, you download a slice of those logs, anonymize them, and feed them into your test runner. Luna: And the test runner itself—does it need to be modified? Lucas: Ideally, yes. You want a test harness that reads log entries and calls your API endpoints with the parameters found in each log. For a REST API, that's straightforward: extract the HTTP method, path, headers, and body from the log, then make the same request to your test instance. Luna: But what about state-dependent requests? Like a user adding an item to cart and then checking out. The second request depends on the first. Lucas: That's where session grouping matters. You replay all requests from a single session in order, maintaining state in the test environment. So you'd create a test user, log in, add item to cart, then checkout—all based on the logs. Luna: That's simulating a full user journey. It's like a recorded playback. Lucas: Exactly. And the beauty is that you can compare the logs produced by the test environment to the original production logs. If they match, your code is behaving the same way. If they don't, you investigate. Luna: We've been talking about backend logs. Does this work for frontend logs too? Lucas: Absolutely. Browser console logs, network request logs, even user interaction logs—if you instrument your frontend to log events, you can replay those in a headless browser. That's how some teams do visual regression testing without screenshot comparisons. Luna: They verify by checking that the same sequence of DOM mutations occurs? Lucas: Yes. Instead of comparing pixels, you compare the sequence of DOM changes. It's much faster and less flaky. Luna: That's clever. So log-based testing spans both frontend and backend. Lucas: And it's not just for catching regressions. It's also a way to discover new test cases. When you replay production logs, you inevitably hit edge cases your manual testers never thought of—like a user with 200 items in their cart, or a session that spans a daylight saving time change. Luna: Those are exactly the kind of edge cases that cause production incidents. Lucas: Right. And once you've replayed a log session that triggers a bug, you can save that session as a permanent regression test. So your test suite grows organically from real usage. Luna: I love that—your test suite becomes a living record of your production behavior. Lucas: Exactly. And it shifts the testing mindset from 'what could go wrong?' to 'what did go wrong?'—which is a much more grounded approach. Luna: One concern: log-based testing might only cover paths that users actually take. What about rare paths that haven't been logged yet? Lucas: Great point. That's why you shouldn't rely solely on log replay. It's a complement to other techniques—property-based testing, chaos engineering, exploratory testing. But it fills a huge gap: the gap between your test data and reality. Luna: So the takeaway is: start structuring your logs, build a replay pipeline, and let production teach your test suite. Lucas: That's a perfect summary. And if you're just getting started, pick one service, export a day's worth of logs, and try replaying them manually against a local instance. You'll be surprised at what you find. Luna: I'm going to try that this week. Thanks, Lucas. Lucas: Happy testing, everyone.