Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Smoke Tests Catch Deployment Killers in Five Minutes
Transcript
- Lucas: So we talk a lot on this show about deep testing — mutation testing, fuzzing, visual regression — the heavy artillery. But today I want to talk about something almost stupidly simple. Smoke testing. Luna: I feel like every engineer I know has a slightly different definition of smoke testing. Some call it a subset of sanity testing, some treat it as the first gate in the pipeline. Where do you land? Lucas: For me, a smoke test is the shortest automated check you run on a deploy candidate before you even bother with the full regression suite. It answers one question: 'Did the build succeed and is the service alive?' That's it. Not correctness, not edge cases — just 'is the thing running.' Luna: Right. So it's like the old hardware test — plug it in, see if smoke comes out. If not, you're good to proceed. Lucas: Exactly. And it's surprising how many teams skip this step. I was looking at a recent case from a fintech startup — they had about two hundred microservices and a CI pipeline that took roughly forty-five minutes to run the full regression. Before they added a smoke test stage, about twelve percent of their deploys would fail within the first ten minutes of production exposure. Luna: That's a lot of wasted regression cycles. Twelve percent means every eighth deploy was essentially dead on arrival. Lucas: Exactly. So they added a five-minute smoke suite — one HTTP health check per service, plus a quick database connectivity check. That's it. Their rollback rate dropped to about seven percent in the first quarter. Luna: That's a forty percent reduction just from a few health checks. But I imagine some teams worry about pipeline friction — adding even five minutes to every deploy sounds like a tax. Lucas: It's a tax, sure. But compare five minutes to a forty-five minute regression that ends up being irrelevant because the build was broken anyway. And a well-designed smoke suite doesn't need to be five minutes — some teams get it down to under a minute by parallelizing the health checks. Luna: So what does a good smoke test actually look like in practice? I've seen teams throw in a single curl against the root endpoint and call it done. Lucas: That's better than nothing, but I'd argue you want a slightly richer signal. A minimal smoke test for a web service should hit at least one endpoint that exercises the stack — so it touches routing, authentication middleware, and data access. If you're running Kubernetes, you also want to verify that the pod actually became ready, not just that the deployment was created. Luna: So the distinction from sanity testing — where do you draw that line? Lucas: Sanity testing is broader. It's a quick check that the core functionality works — maybe logging in, creating a user, fetching a dashboard. Smoke testing is narrower. It's literally verifying that the service boots. Think of smoke as the bouncer at the door — 'Show me you're alive.' Sanity is the first conversation — 'Show me you can do the basics.' Luna: That's a useful mental model. So smoke is purely about infrastructure and service availability, while sanity starts to touch business logic. Lucas: Right. And one of the best examples I've seen of smoke catching something big was at a company that runs Kubernetes on AWS. They had a misconfigured pod security policy that was silently blocking all outbound network calls. The full integration suite passed because it ran within the same cluster namespace and didn't actually test external dependencies. But the smoke test — which called an external API endpoint — immediately failed. That saved them about an hour of debugging. Luna: So the smoke test caught a config error that the integration tests missed entirely. That's a strong argument for making smoke tests cross environment boundaries. Lucas: Absolutely. And I'd argue the best smoke tests are the ones that fail loudly. If your smoke test passes but the system is actually broken, you've got a false sense of security. That's why I recommend teams periodically review their smoke suite and inject failures — turn off the database, block the network — and see if the smoke test actually catches it. Luna: Like a fire drill for your test suite. Lucas: Exactly. And this ties into something we've touched on before: test observability. If your smoke test fails, you need to know why immediately. The log output should be concise enough that a developer can read it in under ten seconds and know which component is down. Luna: One thing I've noticed is that teams often forget to version their smoke tests alongside the application code. They have a separate repo, or the smoke test points to a fixed endpoint that changes in the new build. That's a recipe for false positives. Lucas: That's a really good point. Smoke tests should live in the same repository as the service they're testing, and they should be run as part of the deploy pipeline, not as a separate scheduled job. Otherwise you end up testing yesterday's build against today's expectations. Luna: And what about the ratio? You mentioned one test per ten endpoints — is that a rule of thumb? Lucas: It's more of a heuristic. For a REST API with fifty endpoints, you probably want five to ten smoke tests — one that hits a public health endpoint, one that hits a private endpoint to verify auth is wired, maybe one that exercises a write operation to a test database. You don't need all fifty. The key is coverage across different failure domains: network, auth, data, and configuration. Luna: That makes sense. But I've seen teams go overboard — they start adding assertions, checking response bodies, validating JSON schemas. At that point it's not a smoke test anymore, it's a shallow functional test. Lucas: That's the biggest anti-pattern. Smoke tests should be stateless and idempotent. They should never depend on test data you set up earlier, and they should never assert on specific values — just on the fact that a response came back, and maybe that the status code is 200 or 201. If you start asserting on response content, you've stepped into sanity or functional territory. Luna: So keep it dumb. Keep it fast. And keep it in the loop. Lucas: Exactly. And honestly, that simplicity is what makes smoke tests so underrated. Every other test technique we cover on the show requires careful design, tooling, maintenance. Smoke tests are almost trivial to write, but they catch an entire class of failures that nothing else catches quickly. Luna: Speaking of catching things — before we go further, I want to tie this to something that keeps shows like this one alive. We're able to keep Software Testing with Fexingo ad-free because of listeners who chip in a few bucks a month. If today's conversation gave you something usable — maybe you're going to add a smoke test to your pipeline tomorrow — that's exactly the kind of value that makes listener support feel meaningful. Lucas: Yeah, it's a small thing that makes a big difference. You can find us at buy me a coffee dot com slash fexingo. No pressure, but if the show has saved you an hour of debugging, consider it. Luna: Alright, back to smoke tests. You mentioned that smoke tests should be stateless. But what about stateful services — databases, message queues — how do you smoke test those? Lucas: Great question. For a database, a smoke test might just open a connection and run 'SELECT 1'. That's enough to verify the database is reachable and the connection pool is configured correctly. For a message queue, you might try to produce and consume a single message from a test topic. The key is to verify connectivity, not to validate the message schema or processing logic. Luna: So you're testing the pipes, not the water. Lucas: Exactly. And that's a useful way to think about it. The smoke test is checking that the infrastructure plumbing is intact. If the pipes are clogged, you don't want to waste time testing the water quality. Luna: One more thing — how do smoke tests fit into the testing pyramid we've discussed before? Unit tests at the bottom, integration in the middle, end to end at the top. Lucas: I see smoke tests as a pre-pyramid gate. They sit before the pyramid entirely — they're the first thing that runs after the build artifact is deployed to a test environment. If smoke fails, you don't even enter the pyramid. You roll back and fix the deployment configuration. Luna: That's a clean model. So the smoke stage is like the welcome mat — if the mat is torn up, you don't invite guests inside. Lucas: Exactly. And the beauty is that smoke tests are cheap to maintain because they change infrequently. A new endpoint might not need a new smoke test if it doesn't introduce a new failure domain. That's rare in testing, where most suites require constant updates. Luna: So next time I see a team with a forty-minute CI pipeline and a high rollback rate, the first thing I'll recommend is a five-minute smoke stage. Lucas: It's the highest ROI testing investment you can make. And it's so simple that it almost feels like cheating. But the data backs it up — that fintech startup I mentioned, after two quarters, their deploy confidence went way up, and the on-call rotation got a lot quieter. Luna: That's the kind of metric that matters. Not just test coverage percentages, but actual human sleep quality. Lucas: Right. And that's ultimately what testing is about — reducing uncertainty so you can ship with confidence. Smoke tests are a tiny but critical piece of that puzzle.