Latest / Hiring & Firing with Fexingo: Recruitment, Performance, and Workforce Decisions / How a Work Simulation Cut Bad Hires by 50 Percent
Transcript
- Lucas: Luna, I want to talk about a hiring experiment that one company ran that basically made the case for ditching traditional interviews altogether. Luna: That sounds like a strong claim. Which company and what did they do? Lucas: It's a mid-sized B2B software company called CloudSmart — about 400 employees, based in Austin. They had a bad hire problem. Nearly one in three hires in their customer-facing roles — support, sales, account management — were gone within six months, either fired or quit. So the head of talent, a woman named Sarah Chen, decided to try something different. Luna: What did she do? New interview questions? New scoring rubric? Lucas: She scrapped the behavioral interview entirely for a set of roles and replaced it with a four-hour work simulation. Candidates didn't answer 'Tell me about a time you dealt with a difficult customer.' Instead, they sat down and actually dealt with simulated difficult customers. Luna: So it's like a job audition. I've heard of companies doing that for design or engineering roles — like a take-home project. But for customer-facing roles? That's less common. Lucas: Exactly. And the results were pretty dramatic. Over the next year, the bad hire rate for roles that used the simulation dropped from about 30 percent to 14 percent. That's a cut of more than half. Luna: Fourteen percent is still not zero, but that's a huge improvement. How exactly did the simulation work? Lucas: It was designed to mirror a real day in the role. For a customer support position, candidates received a set of five support tickets — some straightforward, some ambiguous, one that required escalating to engineering. They had to triage them, write responses, and log their reasoning. For a product manager role, they got a product backlog with competing priorities and had to justify their choices in a written brief. The whole thing was proctored virtually over four hours. Luna: Four hours is a big ask for a candidate. Didn't that turn people off? Lucas: That was the biggest internal pushback. Sarah's team tracked drop-off rates. Of the candidates who made it to that stage, about 15 percent declined to do the simulation. But the ones who completed it were almost 40 percent more likely to accept an offer later. So it served as a self-selection filter too. Luna: So the people who chose to do it were already more motivated. That's interesting. But what about bias? One critique of take-home assignments is that they disadvantage people with less free time. Lucas: CloudSmart tried to address that. They gave candidates a full week to schedule the simulation, and they offered it at different times — including evenings and weekends. They also paid candidates a $200 stipend for completing it. Not huge, but enough to offset the time cost. And the results on bias were actually positive. Candidates with no direct industry experience — say, someone coming from retail or hospitality — performed just as well on the simulation as candidates with years of B2B software experience. Luna: That's a big deal. Because traditional interviews often reward industry jargon and confident storytelling, which experienced candidates have practiced. The simulation tested actual problem-solving. Lucas: Right. And that's where the predictive power came from. CloudSmart ran a correlation study comparing simulation scores to six-month performance ratings. The correlation was 0.52, which is strong for hiring. Their behavioral interviews had a correlation of just 0.18. Luna: Point-five-two versus point-one-eight. That's not even close. So why don't more companies do this? It sounds like a no-brainer — at least for roles where you can simulate the work. Lucas: The main barriers are cost and scalability. Designing a good simulation takes time. For each role, you need subject matter experts to create realistic tasks, then you have to test and calibrate the scoring rubric. CloudSmart spent about 80 hours building the first simulation for one role. That's expensive if you're hiring for fifty different job families. Luna: But they only used it for a few roles initially, right? So the ROI depends on volume. Lucas: Exactly. For a role where you hire twenty or more people a year, the math works. The cost of bad hires — recruiting, training, lost productivity — can be three to five times the annual salary. Cutting that in half more than pays for the simulation design. For roles you hire once a year, it's harder to justify. Luna: Could a smaller company, say fifty people, pull this off? Or is this only for the CloudSmarts of the world? Lucas: There are off-the-shelf simulation platforms now. Companies like HireVue and Codility started with coding assessments, but they've expanded into other domains. You can buy a simulation kit for a customer service role for a few thousand dollars. The customization is limited, but it's better than a generic interview. Luna: So the barrier is dropping. I want to circle back to something you mentioned earlier — the candidates from non-traditional backgrounds. Did CloudSmart actually hire more of them as a result? Lucas: Yes. After the simulation was introduced, the percentage of hires without a four-year degree rose from 12 percent to 22 percent. And those hires had the same six-month retention rate as degree holders. The simulation essentially leveled the playing field. Luna: That's a powerful outcome. We talk a lot about hiring for potential or unconventional backgrounds, but the simulation gives you actual evidence, not just a hunch. Lucas: Right. And one more thing — CloudSmart also found that the simulation reduced gender bias. In their traditional interviews, female candidates were rated lower on average, but on the simulation, scores were nearly identical between men and women. The structured, task-based format removed some of the subjectivity. Luna: That's consistent with research on structured interviews. The more structured the assessment, the less room for bias. A simulation is about as structured as it gets. Lucas: If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. We keep the show free because of people who find value in it. Luna: Yeah, it's a small way to say this matters. And we really appreciate it. Lucas: Back to the simulation — one thing I didn't mention is how they scored it. They had a rubric with specific criteria for each task, and two independent raters scored each candidate. Inter-rater reliability was 0.85, which is high. Luna: So it wasn't just one hiring manager's gut feeling. It was calibrated. That makes me wonder: could this replace all interviews? Or is it best as a supplement? Lucas: Most companies that use simulations still do a shorter interview — maybe thirty minutes, focused on fit and clarifying questions. But the simulation carries about 70 percent of the decision weight. CloudSmart actually made offers based on simulation scores alone for some entry-level roles, and only did a brief check-in call before extending an offer. Luna: That's a pretty big shift. How did candidates react? Some might feel a four-hour test is impersonal. Lucas: Sarah's team surveyed candidates after the process. Satisfaction was higher than with the old interview process — 4.2 out of 5 versus 3.6. Candidates said they appreciated seeing what the job actually involved. One candidate wrote, 'This was the most honest job preview I've ever had.' Luna: That makes sense. A simulation is a reality check both ways. The candidate gets to see if they actually like the work, and the company gets to see if they can do it. Lucas: Exactly. And that self-selection effect is real. Some candidates realized during the simulation that the role wasn't for them and withdrew. That saves everyone time. CloudSmart estimated it saved about ten hours of recruiter time per hire. Luna: I'm curious about the long-term impact. Did the lower bad hire rate hold up after the first year? Or did it regress? Lucas: They tracked it for eighteen months. The rate stayed around 13 to 15 percent. And turnover in the first year dropped from 35 percent to 22 percent. The simulation wasn't just predicting better performance — it was predicting who would stay. Luna: That's the holy grail, isn't it? Hiring people who both perform and stay. Most tools only predict one or the other. Lucas: It's not a silver bullet. The simulation didn't work as well for senior leadership roles — the tasks were harder to simulate authentically. For a VP of Sales, you can't easily replicate a quarter of deal negotiations in four hours. But for individual contributor and first-level manager roles, it was strong. Luna: So the sweet spot seems to be roles where the work is relatively concrete and observable. Customer support, junior PM, data analyst, maybe some marketing roles. Lucas: Right. And the key lesson from CloudSmart is not that every company should build a four-hour simulation. It's that the closer your hiring process mimics the actual work, the better your predictions get. Even a one-hour mini-simulation — like asking a support candidate to respond to one email in real time — would probably outperform a traditional interview. Luna: That's a low-cost takeaway. You don't need a full simulation platform. You can start small. Ask a candidate to do one task, watch how they approach it, and score it with a simple rubric. Lucas: Exactly. And that's the kind of change any hiring manager can make starting tomorrow.