Latest / Hiring & Firing with Fexingo: Recruitment, Performance, and Workforce Decisions / How Probation Data Revealed a Flawed Interview Process
Transcript
- Lucas: So Luna, about a year ago I talked to the head of talent at a mid-size software company called NexGen. They had a problem: their interview process felt rigorous, multiple rounds, behavioral questions, technical tests, the works. But something like 30 percent of new hires weren't making it past the 90-day mark. Luna: Thirty percent failing probation? That's a massive leak. How many people was that, roughly? Lucas: About 45 people over two years. And the cost of replacing each one—recruiting, onboarding, training, lost productivity—was north of 50 grand each. So over a couple years, we're talking well over a million dollars in waste. Luna: And they didn't realize until they looked at probation data? I mean, most companies track turnover but not specifically probation failures. Lucas: Exactly. They had annual turnover numbers, but nobody had sliced it by tenure. When they did, they saw the spike: almost a third of departures were within the first 90 days. So they dug into exit interviews from those employees. And a pattern emerged. Luna: What kind of pattern? Lucas: The leavers consistently said the job was not what they expected. Not in a vague 'culture fit' way, but specific: the day-to-day work was more unstructured and project-based than the interview had suggested. They'd been asked behavioral questions like 'tell me about a time you handled ambiguity' and given textbook answers, but the actual work required a different kind of problem-solving. Luna: So the behavioral interview was actually filtering for people who were good at talking about ambiguity, not thriving in it. Lucas: Right. The head of talent described it as an 'interview prep skill' rather than a job performance skill. So they decided to replace the final round behavioral interview with a case study exercise. They designed a 90-minute simulated project: the candidate gets a realistic dataset, a vague client request, and has to produce a recommendation. No right answer, but they can see how the person structures their thinking. Luna: Did they measure the impact? I'm curious if probation failure actually dropped. Lucas: It did. After a year of using the case study, the probation failure rate fell from 30 percent to 12 percent. And they estimated that over two years, the change saved them about $1.2 million in replacement costs. Plus, the managers reported that new hires were more effective earlier. Luna: That's a huge ROI from just tweaking one interview round. It's almost like probation became a feedback mechanism for the hiring process itself. Lucas: That's exactly what it is. Probation data is an underutilized signal. Most companies treat it as a performance check on the employee, but it's also a performance check on your hiring criteria. If a lot of people pass your interview but fail probation, maybe your interview isn't measuring the right things. Luna: And yet so many organizations never look at that. They just assume the interview is fine and blame the candidate. Lucas: Right. The NexGen case shows how easy it is to miss. They had all the data but never connected the dots until someone decided to slice turnover by tenure. That one pivot—tracking probation failures as a distinct metric—led them to a single interview change that saved over a million dollars. Luna: It's also interesting that they didn't scrap the entire process. They just identified the weakest link and redesigned one component. Lucas: That's the smart approach. You don't need to overhaul everything. Just find the point where your predictions break down. For NexGen, it was the final behavioral round. For another company, it might be the technical test or the culture interview. But you only know if you actually measure probation outcomes. Luna: So what are the practical steps for a company that wants to do this? First, they need to track probation failures separately, right? Lucas: Yes. And not just the fact of failure, but the reasons. Exit interviews specifically for people leaving within 90 days should ask: 'What was different between what you expected and reality?' Then categorize those gaps. If a pattern emerges—like 'the job was more unstructured' or 'I didn't have enough guidance'—you know where your interview is misleading candidates. Luna: And you'd need a decent number of data points. NexGen had 45 departures over two years, which is a small sample but enough to see a trend. Lucas: Exactly. Even 15 to 20 cases can give you a strong signal if the pattern is consistent. And the best part is, this data is already being generated—you just have to collect it and look at it with the right lens. Luna: You know, this kind of conversation is exactly why I appreciate that we do this show without ads. We can dive into the nuance without selling anything. Lucas: Absolutely. If these conversations have moved your work forward in some small way, listener support is what keeps it ad-free. The link is buy me a coffee dot com slash fexingo. Simple as that. Luna: Yeah, it's a small way to keep the show independent and focused on what matters. Now, back to NexGen—one thing I wondered is whether they also changed how they onboard during probation. Lucas: Good question. They didn't change onboarding significantly, because the problem was expectation mismatch, not poor integration. The case study actually helped because it gave candidates a realistic preview. And the probation period itself became a structured check-in: managers and new hires had milestone meetings at 30, 60, and 90 days to assess alignment. Luna: So the probation wasn't just a passive waiting period—they actively used it to confirm the hire was working. Lucas: Right. And that reduced the cost of bad hires further because they could address issues earlier. If someone wasn't fitting by day 60, they could either adjust the role or part ways before the 90-day mark, minimizing sunk costs. Luna: I think the biggest takeaway is that your interview process isn't done when the candidate accepts the offer. It's validated when they succeed in the role. Probation is the real test of your hiring system. Lucas: Exactly. And the NexGen story shows that even a 30 percent probation failure rate can be fixed with one targeted change—if you're willing to look at the data.