Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How Test Data Versioning Prevents Flaky CI Builds
Transcript
- Lucas: Let me paint a picture. It's Tuesday afternoon. You push a tiny change — a single CSS class rename — and your CI pipeline kicks off. Four minutes later, three tests fail. You inspect the logs. The failures are in a checkout module you didn't touch. You re-run. They pass. Two hours later, a teammate pushes a completely unrelated PR and the same tests flicker again. Luna: The flaky test horror story. Every team I've worked with has at least one of those — a test everyone just re-runs until it turns green. Lucas: Exactly. And the standard response is to add retry logic or mark the test as flaky. But a team at Shopify decided to actually dig into the root cause. What they found surprised them: the culprit wasn't the test logic or the code under test. It was the test data. Luna: Wait — the test data itself was causing flakiness? Not race conditions or timing issues? Lucas: Right. Race conditions are the usual suspect. But this team traced their flaky failures back to the fact that their test data was essentially unversioned. They had a shared database seeding script — a single SQL file that everyone ran locally before a test run. The problem was that the script changed over time as developers added new fixtures or modified existing ones. So Monday's test run used a slightly different dataset than Wednesday's. Luna: Ah, so test A might assume a user record has a certain email, but someone else's commit added an extra field or changed a default, and suddenly the assertion fails. Lucas: Precisely. The Shopify team called it 'data drift' — the silent mutation of test fixtures across commits. Their fix was surprisingly straightforward: they started treating test data like source code. Every test's fixture set — the database rows, the JSON blobs, the file system snapshots — got versioned alongside the code, using Git LFS for large binary files and checksums to detect accidental modifications. Luna: Git LFS for test data? That's not common. Most teams just have a seeds.rb or a setup.sql checked directly into the repo. Lucas: Right, and that script is mutable. Every time you run it, it creates the same rows — unless someone changed the script. The Shopify approach was to snapshot the actual database state at a specific commit and store that snapshot as a read-only blob. When you check out an old commit, you get the exact data that existed when that commit was written. No drift. Luna: So it's like a time capsule for your test environment. But doesn't that blow up the repo size? Especially if you have a lot of test scenarios. Lucas: It can. They used Git LFS with a weekly pruning policy — keep only the last four weeks of full snapshots, plus every snapshot that's referenced by an open PR. They also compressed aggressively. The whole history of test data for their main service ended up around 12 gigabytes, which is manageable with LFS. And the payoff was huge: their CI retry rate dropped from about 8 percent to under 1 percent within two months. Luna: Eight percent is enormous. If you're running a hundred builds a day, that's eight wasted re-runs. Over a year, that's hundreds of developer-hours and a lot of compute. Lucas: And it's not just the time. Flaky tests train developers to distrust test results. When a test fails, the first reaction becomes 'let me re-run it' rather than 'let me investigate'. That erodes the entire value of a test suite. Luna: There's actually a name for that in the literature — the 'cry wolf' effect. If tests fail too often for non-real reasons, engineers start ignoring failures altogether. Lucas: Exactly. So by versioning test data, Shopify didn't just fix flaky tests — they restored trust in the pipeline. But there's another angle here that I find even more interesting. The problem of data drift isn't limited to manually maintained fixtures. There's a newer approach called 'production data replay' that a tool like Keploy pioneered. Luna: Keploy — I've seen their GitHub. It captures real production traffic and replays it against your tests, right? Lucas: Yes. It runs as a proxy and records every API request and response. Then it turns those recordings into test cases with the real data. The benefit is that your tests use actual customer scenarios. The risk is that the data is mutable — production data changes constantly. So if you record a test on Tuesday and replay it on Thursday, the underlying data might have shifted. Luna: Right — a recorded response assumes a certain database state. If that state changes between capture and replay, the test fails even though the code is fine. Lucas: That's exactly the kind of data drift we're talking about. So the fix is the same: snapshot the data at the time of recording. Keploy actually does this — it captures the entire application state, including the database, as a dependency graph. When you replay, it spins up a fresh environment with the exact data from the recording. Luna: That sounds heavy. Is it practical for large applications? Lucas: It depends on the data size. For a microservice with a small database, it works beautifully. For a monolithic app with terabytes of data, you'd need to subset — take only the records touched during the recorded session. There are tools for that too, like PostgreSQL's pg_partman or custom query filters. Luna: So the principle is the same whether you're writing fixtures by hand or capturing production traffic: the test and its data must be an atomic unit. Lucas: Exactly. And once you think about it that way, a lot of best practices fall into place. For example, never generate random data in a setup method. If you do, two test runs with the same code can produce different results because the random seed might differ. Luna: I've been bitten by that. A test that passes locally with seed 42 but fails in CI with seed 99. The fix is to pin the seed, but most people don't think to do it. Lucas: Right. The broader lesson is that test data should be deterministic and versioned. Every test run should be reproducible — you should be able to check out a commit from six months ago and run the tests with the exact same data and get the exact same results. Luna: That's a strong claim. Most teams I know can't do that today. Their database migrations make old data incompatible, or they cleared the test database since then. Lucas: And that's the gap. The Shopify case shows it's achievable. They used a tool called 'test_db_snapshot' — it's a Ruby gem they open-sourced. It hooks into ActiveRecord and takes a snapshot of the test database before each suite run, then restores it after. Combined with Git LFS, every commit gets a deterministic data set. Luna: If today's tech conversation gave you something usable — maybe the idea of versioning your test data, or the Shopify snapshot approach — and you'd like to help us keep this show ad-free and independent, listener support is what makes that possible. We're at buy me a coffee dot com slash fexingo. No pressure, just a way to keep the signal clear. Lucas: Yeah, it's a small thing that makes a big difference for us. And we're grateful for every bit of support. Luna: So back to the data versioning idea — are there any risks in snapshotting the entire database? I could see issues with large binary blobs or time-sensitive data like timestamps. Lucas: Great question. Timestamps are a classic problem — a snapshot taken at one time will have frozen timestamps, which can cause tests that assert on relative time to fail. The Shopify team handled that by stripping timestamps from the snapshot and regenerating them at test setup time based on a fixed reference point. For binary blobs, they used Git LFS with a separate remote storage backend, and they made sure to ignore temporary or cache tables in the snapshot. Luna: So you need a whitelist of tables to snapshot. Not everything should be frozen. Lucas: Exactly. Their snapshot config explicitly includes only the tables that contain reference data — users, products, orders — and excludes things like logs, sessions, and metrics. That keeps the snapshot small and relevant. Luna: I want to ask about the human side. When you first propose versioning test data, what's the typical pushback? I imagine developers say 'our data changes so fast, it's not practical'. Lucas: That's the number one objection. But the counter is: if your data changes that fast, how do you know your tests are testing the same thing today as yesterday? The second objection is storage cost. But with Git LFS and pruning, the cost is minimal compared to the cost of lost developer time from flaky tests. Luna: So the ROI is actually pretty clear. A few gigabytes of storage vs. dozens of developer-hours per week. Lucas: Right. And there's a third objection that's harder to counter: 'we've never done it this way'. But that's just inertia. The Shopify team published their results in a 2025 case study — they estimated a 40 percent reduction in CI retries, which translated to over a hundred thousand dollars in annual compute savings alone. Luna: A hundred thousand dollars just from not re-running tests. That's the kind of number that gets CFOs to pay attention. Lucas: Exactly. And the real win is developer morale. Nobody likes wrestling with a test that fails for no reason. When you eliminate that, you get faster feedback loops, more confidence in deployments, and ultimately better software. That's hard to price but easy to feel.