Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / When Traditional QA Fails for Machine Learning
Transcript
- Lucas: There was a study from Google published in 2024 that looked at why machine learning models fail in production. The headline number: seventy percent of failures trace back to data issues, not bugs in the model code itself. Luna: Seventy percent — that is a big number. So the model can be mathematically perfect but still break because the input data changed? Lucas: Exactly. And that's the fundamental shift that traditional QA doesn't account for. When you test a deterministic piece of software — say, a function that adds two numbers — the same input always gives the same output. But machine learning models are probabilistic. Their behaviour depends on the statistical properties of the data they see. Luna: Right, so our usual test against expected output approach doesn't cut it. What do teams actually do instead? Lucas: Let me walk through a concrete example. A fintech company deploy a credit scoring model. In their staging environment, the model achieves ninety-four percent accuracy. But within two weeks of going live, approved loan defaults start creeping up. The data science team has a panic. Luna: And what did the root cause turn out to be? Lucas: The model was trained on data from 2023, when interest rates were lower and average applicant credit scores were higher. In early 2024, the applicant pool shifted — more subprime borrowers. The feature distribution for 'credit score' and 'debt to income ratio' drifted. The model was extrapolating outside its training range and making bad predictions. Luna: So the code was fine, but the data profile changed. How does a QA team catch that before it causes losses? Lucas: You need a two-pronged approach: data validation at inference time, and continuous monitoring of model performance. For data validation, tools like Great Expectations let you define expectations on incoming features — things like 'credit score must be between 300 and 850', 'number of open accounts must be less than 20'. If an incoming batch violates those expectations, you raise an alert. Luna: But isn't that just input validation? We already do that for traditional APIs. Lucas: Partly, but it's more nuanced. With ML, you also care about the distribution of values, not just ranges. For example, even if every credit score is between 300 and 850, the mean could shift from 700 to 620. That's a data drift. Detection tools like Alibi Detect can compute statistical distance metrics — like the Kolmogorov-Smirnov test — between the training distribution and the live distribution. Luna: So you're running a statistical test on every prediction batch? Lucas: Yes, and it scales. For high-volume models, you can sample or aggregate over windows. The key is to have thresholds for how much drift is acceptable. Too tight, and you get false alarms. Too loose, and you miss real shifts. Luna: Our listeners might be thinking: 'My team doesn't have a data scientist or a dedicated ML pipeline. Where do we even start?' Lucas: That's a fair point. The first step is to inventory any models in production, even simple ones — linear regressions, decision trees. Then, set up basic data logging. You can't monitor what you don't log. Tools like BigQuery or even a simple parquet dump with timestamps give you the raw material. Luna: And from there, one of those open-source libraries like Great Expectations or Evidently can generate reports. I've seen them produce a human-readable dashboard comparing training vs. live feature distributions. Lucas: Exactly. And that dashboard is a QA artifact. It's the equivalent of a test report for ML. The quality assurance function expands from 'does the code work?' to 'does the data still look like we expect?' Luna: There's another layer too: concept drift. The relationship between features and the target can change even if the input distribution stays the same. For example, consumer behaviour after a recession changes how credit risk correlates with income. Lucas: Right, concept drift is trickier to detect because you need actual outcomes — which might be delayed. For a credit model, you might not know if a loan defaulted for six months. So you monitor proxy metrics: approval rate, average score, or even model confidence distribution. Luna: That's a great segue into another question: how do you test the model's accuracy metrics themselves? Relying on a single number like 'accuracy' is dangerous. Lucas: Especially with imbalanced datasets. Accuracy can be ninety nine percent if you only ever predict 'no default', but that model is useless. You need to test precision, recall, F1-score, and possibly area under the ROC curve. And those should be computed on slices: by demographic group, by time period, by data source. Luna: And then set up automatic alerting when any of those slice metrics drop below a threshold. That's essentially your regression test suite for ML. Lucas: Right. And speaking of test suites, there's a growing practice of 'model testing' frameworks, like Deepchecks or TorchMetrics, that let you define assertions: 'Recall for the senior citizen segment must be above 0.8', 'Maximum predicted probability must not exceed 0.99'. These assertions run as part of your CI/CD pipeline. Luna: So you can fail a build not because the code broke, but because the model performance degraded on a specific slice. That's powerful. Lucas: It fundamentally changes the role of QA. Instead of just testing the application layer, QA teams are increasingly responsible for the data pipeline and model monitoring. And that requires new skills — statistics, basic machine learning, data engineering. Luna: If today's tech conversation gave you something usable, maybe consider supporting the show. Listener support is what keeps Fexingo ad-free and lets us go deep into niche topics like this. You can find us at buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small gesture that goes a long way in keeping the episodes focused and independent. And to continue our discussion — I think the biggest takeaway is that testing machine learning is not a one-time activity. It's a ongoing monitoring and validation practice that sits alongside traditional QA. The good news is that many tools are now mature enough to add to your pipeline without needing a PhD. Luna: And the cost of not doing it is exactly the kind of failure we opened with: seventy percent of ML failures are data-related. That's a huge risk for any company using models. Lucas: A risk that smart testing can mitigate. Start with data validation, add drift monitoring, and treat your model metrics like test assertions. Your future self — and your customers — will thank you. Luna: Great advice. That's all for this episode of Software Testing with Fexingo.