Latest / Future of Work Tech with Fexingo: Remote Tools, AI Productivity, and Workplace Software / How AI Is Automating Your Internal Employee Survey Analysis
Transcript
- Lucas: So every year, the average large company runs somewhere between three and five company-wide employee engagement surveys. The response rate usually hovers around 65 to 70 percent. For a firm with 50,000 employees, that's roughly 35,000 completed surveys per cycle. Luna: And buried inside each one is a free-text comment box. Some people write a sentence. A lot of people write paragraphs. Lucas: Right. And until very recently, analyzing those open-ended responses meant one of two things: either the HR team skimmed a random sample, or they contracted an external firm to manually code themes. Both are slow, expensive, and wildly inconsistent. Luna: So this is where AI has quietly become a standard tool. Not the flashy generative stuff—more like natural language processing that categorizes sentiment and topics at scale. Lucas: Exactly. I want to talk about a specific case. Early last year, a large US retailer—about 50,000 employees, mostly hourly store staff—ran their annual engagement survey. They got 34,000 responses. The open-ended comments alone ran to about 1.2 million words. Luna: That's roughly the length of 'War and Peace' plus 'Anna Karenina' combined. Lucas: Yes. And they had a two-person HR analytics team. Manual coding would have taken them six weeks—if they did nothing else. Instead, they used a platform built on top of Microsoft's Azure Cognitive Services, which applied a custom-trained NLP model to tag each comment by topic—pay, scheduling, management, safety—and by sentiment. Luna: And how long did the AI take? Lucas: Ninety minutes. End to end. And it didn't just tag the comments. It surfaced something the HR team had missed entirely in the quantitative data. There was a cluster of comments from stores in the Southeast region complaining about a recent change in break policy. The standard Likert-scale questions hadn't flagged it because most employees rated 'overall satisfaction' as average. But the open-ended comments were where the frustration lived. Luna: That is exactly the promise. But there's a catch, right? The model has to know what it's looking for. If you train it only on previous themes, you might miss a completely new issue. Lucas: That's a real limitation. The retailer's model was trained on three years of historical survey data, so it was good at recognizing recurring patterns. But what about the employee who writes something truly novel? Or uses sarcasm? Or references a specific local event the model has no context for? Luna: Or writes in a mix of English and Spanish. That's common in retail and hospitality. Lucas: Right. Most off-the-shelf NLP models handle English well, but code-switching degrades accuracy significantly. Some vendors claim 85 percent accuracy on sentiment for English-only text, but that drops below 70 percent with mixed-language responses. Luna: And then there's the trust question. Employees are told their feedback is confidential. But if an algorithm is reading every word, does that feel different than a human reading a random sample? Lucas: It does. Qualtrics, which dominates this market, has published research showing that 38 percent of employees would be less likely to write candid comments if they knew AI was analyzing them. That's a significant chilling effect. Luna: So companies are caught between efficiency and trust. You want the granular insight, but you risk losing the raw honesty that makes open-ended comments valuable in the first place. Lucas: And this is where the smarter vendors are trying to differentiate. Some allow companies to set thresholds—for example, the AI only flags comments that contain certain keywords or sentiment extremes, and a human reads those. The rest are analyzed only at an aggregate level, never attributed back to an individual. Luna: But the aggregation itself can still feel surveilled. If you're the only person in your department who mentions a specific issue, the AI effectively identifies you. Lucas: Exactly. That's the next frontier. I know of at least one startup that's building differential privacy into survey analysis—adding what they call 'statistical noise' so that no single comment can be reverse-engineered to an individual, even if it's the only one on a topic. But we're not there yet in commercial products. Luna: It reminds me of how analytics tools in other domains have evolved. You start with raw efficiency, then you realize you need ethical guardrails, and eventually the market sorts out who builds trust best. Lucas: Speaking of trust, I should mention that this show stays ad-free thanks entirely to listener support. If you find conversations like this useful, you can help keep it that way at buy me a coffee dot com slash fexingo. No pressure—just if the work matters to you. Luna: And it genuinely does help. We don't run ads, we don't take sponsorship, so it's a direct line between listeners and what we cover. Lucas: Alright. Back to surveys. Let me throw out a number that surprised me: Gartner estimates that by the end of this year, 40 percent of large enterprises will be using some form of AI to analyze employee feedback data. That's up from about 12 percent in 2023. Luna: So it's moving from early adopter to mainstream. And with that adoption, the data quality question gets more important. Garbage in, garbage out still applies. Lucas: Absolutely. One HR analytics director I spoke to said the biggest mistake companies make is assuming the AI can handle 'human messiness'—typos, slang, incomplete sentences, emotional rants. Those need to be in the training data, or the model will misclassify them. Luna: So what's the best practice? If a company wants to implement this, where should they start? Lucas: Start small. Pick one survey, one subset of questions. Hand-code a few hundred responses with a team of three people, then use that labeled data to train a model. Compare the model's output to the human coding. Iterate. Don't roll it out company-wide until you can validate accuracy on your own population. Luna: That sounds obvious, but I bet most companies skip the validation step because they're eager to save time. Lucas: They do. And then they get a dashboard that says 'Top concern: compensation' when the actual top concern was 'my manager doesn't listen.' The model just saw the word 'pay' in a context where pay wasn't the real issue. Luna: Context is everything. Do you think there's a risk that companies will start designing survey questions to be ai friendly, dumbing down the nuance to get cleaner outputs? Lucas: I think that's already happening. Some platforms coach HR teams to write questions that produce more 'classifiable' responses. Instead of 'What else would you like to share?' they suggest 'Which of the following topics is most important to you?' Multiple choice eliminates the interpretive problem, but it also eliminates the surprise insight. Luna: So we're trading depth for ease. And in a world where AI can handle depth, that seems backward. Lucas: It is. The whole argument for AI is that it can handle the unstructured stuff. If you structure everything, you might as well just use a multiple-choice survey and skip the AI entirely. Luna: What about the timing? These surveys are usually annual or semi-annual. Could AI enable continuous listening—like analyzing every Slack message or email for sentiment? Lucas: A few companies are experimenting with that, but it's legally and ethically fraught. If you analyze internal communications without explicit consent, you risk violating privacy laws—not to mention destroying trust. The EU's GDPR and some US state laws restrict automated processing of employee communications. So for now, the sweet spot is structured surveys with open-ended components. Luna: But even there, the tech improves fast. I saw a demo from a startup called Echo Analytics that claims to detect emotional tone—not just positive/negative, but frustration, excitement, confusion—with 92 percent accuracy on English text. Lucas: That's impressive if it holds up in real-world conditions. But I'm skeptical of any vendor claiming above 90 percent on nuanced emotional categories. The research literature suggests that even trained human coders agree on emotional labels only about 80 percent of the time. Luna: So a good benchmark might be: does the AI match human agreement levels? If you have two HR people coding the same comment and they agree 80 percent of the time, and the AI agrees with them 80 percent of the time, that's probably acceptable. Luna: So the human-in-the-loop model isn't going away. It's just shifting from doing all the work to doing the exception handling. Lucas: That is exactly the shift. And it's a good one, because it lets HR teams focus on the things that actually require human judgment—investigating a pattern of complaints about a specific manager, or designing a new policy based on widespread feedback. Luna: I want to come back to the trust issue. If employees know the AI is reading their comments, but they also know a human will read anything flagged as high-risk, does that actually improve candor or reduce it? Lucas: The data is mixed. Some studies show that transparency about AI analysis increases candor because employees feel their feedback is actually being taken seriously at scale. Others show the opposite. One thing that consistently helps is giving employees control—letting them opt out of AI analysis, or allowing them to mark a comment as 'sensitive' so it goes only to a human. Luna: That seems like a smart design choice. It respects agency while still capturing the bulk of feedback efficiently. Luna: So where do you see this heading in the next couple of years? Beyond surveys, I mean. Lucas: I think we'll see AI analyzing not just what employees say, but what they don't say. For example, if someone consistently skips the open-ended box, is that a signal? Or if they write a lot in one survey and nothing in the next—that change might indicate disengagement. Luna: Behavioral metadata. That's getting into predictive people analytics. Lucas: It is. And that's powerful but dangerous. The line between helpful insight and creepy surveillance is thin. Companies that cross it will face backlash. My guess is the market will gravitate toward tools that are transparent, opt-in, and focused on aggregate trends rather than individual predictions. Luna: So the winners will be the ones that balance analytical power with ethical design. Lucas: Exactly. The technology is ready. The question is whether organizations are ready to use it wisely.