Latest / Hiring & Firing with Fexingo: Recruitment, Performance, and Workforce Decisions / Why Unstructured References Are Worse Than No Reference
Transcript
- Lucas: Let me ask you something, Luna. When was the last time a reference check actually changed your mind about a candidate? Luna: Honestly? Probably never. I've had references confirm what I already suspected, but I can't remember a single instance where a reference call made me go from 'no' to 'yes' or vice versa. Lucas: That's the norm, and it's a problem. Most hiring managers treat reference checks as a bureaucratic checkbox — they call, ask a few vague questions, get a lukewarm endorsement, and move on. But a 2025 study in the Journal of Applied Psychology found that unstructured reference calls have essentially zero predictive validity for job performance. As in, they tell you nothing about how someone will actually do the job. Luna: Zero? That's a strong claim. How did they measure that? Lucas: They aggregated 40 years of data across hundreds of studies. When reference calls followed no structure — just open-ended 'Tell me about this person' — the correlation with future performance was statistically indistinguishable from zero. Meanwhile, structured reference interviews, where you ask the same questions in the same order for every candidate, had a correlation of about 0.25. That's not huge, but it's meaningful. Luna: So the problem isn't references per se — it's how we do them. What's the difference between a structured and unstructured reference call in practice? Lucas: The key is consistency and specificity. Unstructured calls sound like this: 'So, how was Jane to work with?' The reference says 'Great, really proactive,' and you hang up. You've learned nothing. A structured call asks: 'On a scale of one to five, how would you rate Jane's ability to meet deadlines? Can you give me a specific example of a time she missed one?' Luna: That forces the reference to be concrete. But don't references naturally want to paint a rosy picture? They're usually chosen by the candidate. Lucas: Exactly. There's massive selection bias. Which is why the best structured reference programs don't just rely on the candidate's list. They also reach out to former peers, direct reports, even clients — people the candidate might not have volunteered. And they ask one question that cuts through the noise better than any other. Luna: What's that? Lucas: 'Would you rehire this person if you had the chance?' It's deceptively simple. A 2023 meta-analysis found that this single question had the highest predictive validity of any reference check item — around 0.33. Because it forces the reference to make a direct, personal, consequential judgment. It's a proxy for 'Would you trust this person with your own team?' Luna: Interesting. But what about legal risk? If a former employer says 'No, I wouldn't rehire,' and the candidate doesn't get the job, could that come back as a defamation claim? Lucas: That's a real concern. In the U.S., most states have laws protecting employers who give good-faith references, but many companies still have a policy of only confirming dates of employment and job titles. Which is why some companies have moved away from references entirely and instead use structured peer-review tools. Luna: What does that look like? Lucas: A company I spoke with — a mid-size tech firm in Austin — replaced reference calls with a tool that sends anonymous surveys to people the candidate has worked with. It asks about ten specific behaviors: collaboration, problem-solving, accountability. The responses are aggregated into a score. They found that this peer review score correlates with six-month performance ratings at 0.4, which is double what structured reference calls get. Luna: But isn't there a risk of gaming? The candidate picks the peers, and the peers know it's for a reference. Lucas: Sure, there's some selection bias. But the tool randomizes which peers the candidate lists — you can't just pick your best friend. And the questions are behavioral, not just 'Is this person good?' For example: 'How often did this person take initiative on tasks outside their role?' That's harder to fake a high score on. Luna: Right. So the future of references might not be phone calls at all. Lucas: I think it's heading that way. But for now, most hiring managers are still making calls — and they're doing them badly. Let's talk about a simple fix: a three-question script that turns a reference call from a waste of time into a real data point. Luna: I'm all ears. Lucas: Question one: 'What was the single most important contribution this person made in their time with you?' That forces specificity. Question two: 'If you had to identify one area where this person could have improved, what would it be?' That reveals growth areas. Question three: 'Would you rehire them?' If they hesitate, that's a signal. Luna: Simple. And you ask every reference the same three questions in the same order. Lucas: Exactly. Consistency is what makes it predictive. And you record the answers. Don't just trust your memory. Luna: Speaking of trust, I've seen some research that says unstructured reference calls are actually worse than no reference at all because they can introduce bias. Is that true? Lucas: Yes. If you ask vague questions, the reference's response is filtered through your own biases. You might interpret 'She's very direct' as 'She's difficult' if you're biased against assertive women. Structured questions minimize that. But there's another layer: the halo effect. If the reference starts with a glowing overall statement, you tend to discount any later criticism. Luna: Let's talk about the candidate's perspective. A lot of people worry that a bad reference will torpedo their job prospects. Is there any evidence that references actually hurt candidates? Lucas: There's a 2022 study from the Journal of Business and Psychology that found that about 15 percent of reference checks contain negative information. But of those, over half were irrelevant to job performance — things like personality conflicts or personal issues. So there's noise. However, one pattern that did predict poor performance: references who refused to answer the 'Would you rehire?' question. That was a strong red flag. Luna: So if you call a reference and they dodge, that's meaningful. Lucas: It is. But you have to be careful. Some companies have policies that prevent them from answering anything beyond basic verification. So you need to interpret silence in context. If you call five references and four give you detailed answers, but one is a former manager who only says 'She worked here from 2020 to 2023,' that's a signal. Luna: Let me push back a little. There's a growing movement to eliminate reference checks entirely, especially in tech. The argument is that they're biased, unreliable, and waste everyone's time. Do you agree? Lucas: I think it depends on the role and the stakes. For low-skill, high-volume hires, references are probably not worth the effort. But for executive or specialized roles where a bad hire costs hundreds of thousands of dollars, a well-structured reference process can catch problems that interviews miss. The key is to treat references as a supplement, not a substitute, for other hiring methods like work samples and structured interviews. Luna: That makes sense. So if a hiring manager wants to improve their reference checks starting today, what's the single biggest change they should make? Lucas: Switch from unstructured to structured. Pick three questions, ask them every time, write down the answers. And always ask 'Would you rehire?' That alone will double the predictive power of your reference checks overnight. Luna: I think that's practical advice. And I have to say, this conversation has been really useful. It's one of those things we do on autopilot without realizing how much room there is for improvement. Lucas: Yeah, and honestly, conversations like this are exactly why we keep doing these episodes — because there's always a better way if you look at the data. And it's also why listener support matters so much. If these episodes have helped you think differently about hiring, or saved you from making a bad hire, a couple of dollars a month genuinely keeps this show ad-free and independent. You can find us at buy me a coffee dot com slash fexingo. Luna: It's a small gesture, but it makes a big difference for a small team. We really appreciate everyone who's chipped in. Lucas: Absolutely. Now, back to the research — there's one more finding I want to bring up. The study also found that references from direct supervisors were significantly more predictive than references from peers or subordinates. So when you're collecting references, prioritize current or recent managers. Luna: That makes sense — they have the most direct view of performance. But what if the candidate says they can't provide a current manager because they don't want to tip off their employer? Lucas: Common problem. The solution is to ask for a former manager from a recent role. If they can't provide any manager reference from the past five years, that's a yellow flag. But many companies now accept a peer reference from the current job as a substitute, as long as you follow the same structured script. Luna: So structure matters more than who you talk to, but who you talk to also matters. Lucas: Exactly. And one more thing: don't rely on written reference letters. They're almost always positive and contain zero useful information. The only way to get real signal is a live conversation — or a well-designed survey. Luna: Alright, I'm convinced. Next time I'm on a hiring panel, I'm going to push for a structured reference script. And I'll start with that 'would you rehire' question. Lucas: Good. And if you want to see the exact script from the Journal of Applied Psychology study, we'll link to it in the show notes. That's it for this episode. We'll be back next time with a look at why exit interviews are often too late to save your best people.