Latest / Software Testing with Fexingo: QA, Automation, and Reliable Software Engineering / How to Test AI Chatbots Without Losing Your Mind
Transcript
- Lucas: There's a moment every tester knows: you send a prompt to a new AI chatbot, and it gives you a confident, well-worded answer that is completely, dangerously wrong. Luna: And it says it with such certainty, that's the scary part. Lucas: Right, it's not like a broken login screen. It's a fluent lie. And that's the core problem with testing AI chatbots in 2026 — you can't write an assertion that says 'assert response equals expected.' Because there is no single expected response. The model is stochastic. It's a probability machine. Luna: So how do you QA something that's designed to be non-deterministic? I've seen teams try to just run a few manual prompts before launch and call it a day. Lucas: That's exactly what we're going to dig into today. I've been looking at how a fintech company — let's call them 'PayBridge' — rolled out a customer support chatbot powered by a large language model, and they almost shipped a version that would have told customers their payments were still processing when they were actually declined. Luna: Ouch. That's the kind of bug that lands you on the front page, the wrong way. Lucas: The root cause wasn't the model. It was the test strategy. They had a set of happy-path prompts: 'what's my balance,' 'how do I reset my password.' And the bot aced those. But they didn't test the edges — the angry customer, the confused customer, the customer with a typo-riddled question about a failed transfer. Luna: So what did they do? They must have had some kind of QA process, right? Lucas: They did, but it was classic: a spreadsheet of prompts, manual runs, and a 'looks fine' sign-off from the product manager. And that's what we need to upgrade. Because in the world of AI, a test that passes today can fail tomorrow with the same prompt, if the model's been updated or the context window shifts. Luna: So it's not just about testing once — it's about ongoing monitoring. Lucas: Exactly. And the first step is to stop thinking like a unit tester and start thinking like an evaluator. You're not checking for a specific output, you're scoring the output on a rubric. Is it factually correct? Is it within policy? Does it escalate when it should? That changes everything. Luna: Can you give me a concrete example? How do you score 'factually correct' when the bot is generating fresh text each time? Lucas: You use a golden set. That's a curated collection of real or near-real user prompts, each paired with an expected behavior or a set of acceptable answers. For PayBridge, they built a golden set of about 500 prompts, covering everything from 'I lost my card' to 'why did you charge me twice.' And each one has a list of criteria: does the bot acknowledge the issue, does it offer a next step, does it avoid saying 'we've refunded you' unless it actually has. Luna: So you're testing intent recognition as much as the response. Because if the bot misunderstands the intent, everything after that is garbage. Lucas: Bingo. They had a prompt that said, 'I forgot my password and I'm locked out.' The bot, in one test run, responded with 'I'm sorry to hear you're locked out. Let me help you with your account.' And that's fine — that's on script. But when they ran the same prompt after a model update, the bot said, 'No problem, I can help you reset your password. You'll need your account number.' And that's actually wrong — because the user is locked out, so they can't access their account number. The bot should have verified identity another way. Luna: So the golden set caught that? Because the expected behavior was 'help them reset without requiring account number.' Lucas: Exactly. The golden set didn't have a single 'right' answer, but it had a list of acceptable behaviors. And they scored the response against those. And here's the trick: they automated the scoring using another LLM as a judge. Luna: Ah, the old 'let an AI grade an AI' approach. That's both clever and a tiny bit terrifying. Lucas: It's surprisingly reliable if you do it right. You give the judge LLM the prompt, the candidate response, and a rubric, and ask it to output a score from 1 to 5 for each criterion. For instance, 'Does the response align with our policy of never confirming a refund before it's processed?' They found that a well-prompted judge model agrees with human evaluators about 90 percent of the time. It's not perfect, but it's consistent enough to run in your CI pipeline. Luna: That's a game-changer. But I'm guessing the judge model can also hallucinate. What if it gives a passing score to a truly bad response? Lucas: That's why you keep a human in the loop for the edge cases. You don't automate everything. You use the LLM judge to filter out the obvious passes and failures, and then you manually review the borderline ones. PayBridge set up a system where any response that scored a 3 or lower on any criterion gets flagged for human review. That's your safety net. Luna: That makes sense. You're not trusting the judge blindly, you're using it to triage. Lucas: Right. And that human review doesn't have to be a dedicated QA team. It can be a product manager or a support lead who knows the domain. The key is to have a defined process for what happens when something gets flagged. In PayBridge's case, they had a weekly session where they reviewed all the flagged responses and decided if it was a one-off or a systematic issue that needed a prompt tweak or a model retrain. Luna: That's the real work — not just testing, but feeding that back into the system. So, what metrics should a team track? I mean, you can't just look at 'pass rate' because that's too binary. Lucas: You're right. There are four key metrics I've seen work well. First, intent accuracy — did the bot correctly understand what the user wanted? Second, response accuracy — was the information correct? Third, policy compliance — did the bot stay within your guardrails? And fourth, escalation rate — how often did it decide it couldn't handle it and hand off to a human? Each one needs a target. PayBridge aimed for 95 percent intent accuracy on their golden set, and they tracked it over time. Luna: And I bet they saw it drop after that model update you mentioned. That's when the real value of continuous testing kicks in. Lucas: Exactly. They had a regression suite that ran every night against their golden set. When the model update came through, the score dropped by 12 points on intent accuracy. They caught it within 24 hours, before it ever touched production. That's the difference between a controlled rollout and a customer service disaster. Luna: That's a perfect example of why this matters. But let's talk about the challenges. What are the common pitfalls teams hit when they try to set this up? Lucas: The biggest one is the golden set itself. If your golden set is too small, it's not representative. If it's too big, it becomes unmanageable. And if it's full of fake, sanitized prompts, it won't reflect what real users actually type. PayBridge's golden set started as a spreadsheet, but they quickly realized they needed to pull real anonymized queries from their support ticketing system. That's the gold standard. Luna: So using real user input is key. But how do you keep that golden set fresh? Language changes, new products come out. Lucas: You treat it like any test artifact — it's versioned, it's reviewed, it's updated. They had a monthly review where they added new prompts based on recent support tickets, and removed ones that were no longer relevant. And they tracked the drift in scores over time. If intent accuracy starts trending down even without a model change, that's a signal that user behavior has shifted. Luna: That's a smart way to detect drift. So we've talked about the golden set, the LLM judge, and the metrics. Is there anything else a QA team should have in their toolkit? Lucas: One more thing: adversarial testing. You need to deliberately try to break the bot — inject prompts with harmful requests, try to get it to reveal other users' data, or see if it can be tricked into giving financial advice. That's a different skill set, but it's critical. PayBridge had a set of adversarial prompts that they ran separately, and they had a zero-tolerance policy for any response that leaked personal info. Luna: So it's like a combination of QA and security testing. That sounds like a lot to set up, but I guess it's the price of shipping an AI feature responsibly. Lucas: And speaking of the price of things, I want to take a quick moment here. You know, we keep these episodes free and ad-free because we think this kind of practical, no-hype information should be accessible. If you've found this conversation useful, and you want to support what we do, there's a link: buy me a coffee dot com slash fexingo. It's a way to keep the lights on without interrupting your listening with ads. Luna: Yeah, and it's completely optional, but every little bit helps us keep doing these deep dives. Now, back to the golden set — you mentioned versions. How do you handle versioning when the model itself changes? Lucas: That's the key. Every time you update the model — even a minor tweak — you run the full suite and compare the scores to the previous version. You're looking for regressions. And you store those results with the model version and the deployment date, so you have a history. That way, if something bad happens in production, you can trace it back to a specific model deployment. Luna: That's like having a time machine for your AI bugs. It's a lot of work, but it's the kind of discipline that separates a mature AI product from a toy. Lucas: Exactly. And the payoff is huge. PayBridge's bot now handles about 70 percent of customer support queries without human intervention, and their CSAT scores are actually higher than with human-only support. But that wouldn't have happened if they hadn't invested in this testing framework. Luna: So the takeaway is: don't fear the non-determinism, embrace the evaluation. Build that rubric, build that golden set, and keep a human on standby. Lucas: That's the fifty-thousand-foot view. And I'd add one more thing: start small. You don't need a thousand prompts on day one. Start with fifty, build the pipeline, and grow it. The important thing is to have a process in place before your AI feature goes live. Luna: Well said. And I think that's a wrap for today. Thanks for joining us, and we'll see you next time.