Latest / The Agentic Wave / Justin Robbins | Uncovering the unknown in CX metrics
Transcript
- Elizabeth: Say that again. This is this should scare everybody. My eye just started twitching. You don't know what's happening behind door number one, but you're just gonna trust the warm and fuzzies. We've got to rein this in as organizations before it is the very end of us. We have got to get eyes on the AI. The last thing you want to be if you are leading this operation is ignorable. No, we wanna set ourselves on fire. not be ignored. Maybe more CEOs need to get back on the phones. Let's throw up a toll-free number at the bottom of the screen right now. We've got people standing by. At the intersection of AI and CX, it's the agentic wave, a ship so huge it deserves its own podcast. I'm Latney Conant, Chief Market Officer at Parlowa. Get ready to hide 20 as we discuss the latest customer experience trends, advances in technology, and most importantly, how to confidently lead your teams to ride this wave. Today I'm joined by Justin Robbins, CEO and founder of Metric Sherpa. I am so excited, Justin, to have you on the show. Welcome to the Agentic Wave. Latney, thanks for having me. so happy to have you. Tell us about Metric Sherpa. Metric Sherpa is really curious about one big question, Latney. It's about the way that work works. Maybe, maybe more importantly, why does work seem to work against us in so many places? And so we study how organizations design and deliver their employee and customer experiences. And equip leaders with insight and most importantly the tools they need to make work better for all of us. Who doesn't want that? Well, I am happy to be partnered with you. We wanted to study shocker metrics, a pretty big part of of work, being able to measure our work and know if we're on the right track. And so, you know, maybe share some of the things that we our hypothesis and why we even wanted to do this together. Latin, if we think about why businesses even have metrics in the first place, they should do one of two things. One, help us predict whether we're moving closer to the outcomes we hope for. And then B, help us reflect on the progress that we made and how far we've come. And we had this little suspicion that as organizations face a time of unprecedented change, right? Let's look at what. AI has been doing to all of our lives, and just the way technology is evolving and around us and how customers expect their services to be delivered is changing. We have the suspicion that in the midst of all of that, let's call it what it is, chaos, that maybe organizations hadn't taken a step back to breathe and ask the question of are we looking in the right places and measuring the right things? And as we make decisions, what is our reflex? What is what is that? Teaching us and telling us about our business. And does the confidence that we have in how we lead, is it warranted, or are there some things that might be sneaking up on us and hopefully not catching us by surprise? So you introduce in the research a term called observability, which is a term that's kind of floating around out there. but I think you did a really nice job of really codifying what that means. So maybe share with the audience. How you think about observability and and what what the research showed, the reaction to it was. Yeah, what was interesting about this topic and this question is we certainly asked people point blank, hey, are you familiar with this term? And and it was mixed candidly, right? For for a small percentage it was. But then we started to talk about the attributes of observability. And I think before. We we certainly had a a definition that we put out there. But I think what was important for us to understand, not just even testing that definition, but what matters to organizations in terms of how they can see into their customer experiences and understand what's happening before things get out of control. What what I know to be true is for many organizations is they have very limited and and often fractured visibility into how they're human and AI interactions are playing out. And what they do have is often after the fact. So what we tried to to understand from leaders is what were different things that mattered to them in terms of insight and capability into where and how their interactions were playing out. Yeah. And so, I mean, what what are some of the key tenets of it? I mean, I when I when I think about it, people have always had a QA solution, right? And we've had surveys. I think the difference with observability, and correct me if you don't agree, is one, we're talking about every single call, human or AI. The other thing that we're talking about is risk, because with AI, there is a new risk that gets introduced that is pretty critical to be able to see in real time all the time. And then to me, the true crux of it is. Getting to the bottom of was the experience good or bad? And that sounds pretty basic, but it's sort of a challenge today. So I agree with you. It is this is not just QA. It's not customer listening. It's not even data analytics. I I almost think of it, let's go with something we all understand right now. I I would imagine most, if not all of us, probably have smoke and maybe even carbon monoxide detectors inside of our house. And the way I I think about like the reason we do that. Is because we need to have a way to detect what's happening either before we realize. I mean, if it's a fire, it's maybe a little too late without that. But in the case of carbon monoxide, right, we don't we don't even see it and it's potentially out there. And so we need something that has this capability and always onness that we know gives us the the confidence and assurance that if things go wrong, we're gonna find out in in enough time to take action. That's what I think about observability. So many of the things that you mentioned was too little, too late, if it was actioned at all. And this is really about living, breathing integration into all aspects of how our experiences are designed and deployed. And then I think what we wanted to do together was go a little bit deeper because I think about a mark being a marketer and When I first started, the primary measurement of success was something called a marketing qualified lead. And it was trackable. Everyone understood what it was, but everyone also knew it was kind of bullshit. That happens in every organization and every department. And I think there's some things in service that we can look at and say, okay, everyone understands containment. Great. It's important. We need to track it. Everyone understands average handle time. Great, it's important. We need to track it. Is it really telling us if the experience was good or bad? No. And so what metrics do we need so we can actually do that? That that's what I wanted to get to the bottom of with you. And and I think we did a good job getting there. The other thing is, does that even lead to money? At some stage in the game, we want to be making money. Where's the friction that we need to eliminate so people buy again? you know, where's the upsell opportunities potentially? and how do we get service leaders on the revenue team? I want the service team on the revenue team. You know what I mean? And honestly, Ladney, from the leaders that I've talked to, they want to be on the revenue team as well. Come on in, the water's fine, guys. I'm not speaking for all of them, but I think most of them that have any kind of Keen sense of like how this all plays out, recognize that's a critical part to what we do. Letney, you know, the interesting, we'll say affirming, but also a little bit just disheartening part of what what I saw as we went into the the research is there continues to be this fear of change and new and different approach in a lot of what's being measured. And when we looked at what are organizations doing, it was a a lot of the traditional, I could have asked you this question 30 years ago and most of the answers would have changed. Now, don't get me wrong, there were a few things that we saw that I got pretty excited about when I saw that organizations were shifting from from one metrics to another, and we can talk about that. But there's there's maybe this broader sentiment of well, I don't know what I don't know. And it feels almost just daunting to ask this question as a practitioner. Like, do I have to create the new metrics? Can I just apply my human metrics to AI? Which for spoiler alert, that's part of what we saw some organizations doing, saying, Hey, can I just take this thing and move it over here and make it work? But in a lot of ways, we didn't get good answers in terms of new things that that businesses were measuring. What we actually realized is there were a lot of things that They they didn't have any insight to, which on the back end, and and this is probably what I've gotten most excited about the work with your team is what does come next? What are the things that we look to and ask about and push on to have a better sense of what's happening out in inside of our organizations? Well, let's do this because I feel like we're, you know, we're teasing people along here. Top things that we found. What are some of the most interesting things for you? One of the most interesting things for me is that ninety-five percent of the people who run customer experience can't trust their numbers. Latney, we we really are like marketing. We had an overwhelming majority of people. It was ninety-five percent said that they had they had high confidence and and belief in what they were doing. But we also saw ninety percent recognize that and we by the way know that we can't see some of this. It's like, so hang on, you don't know what's happening behind door number one, but you're just gonna trust the warmen fuzzies. Cause if if out of anything that's the fensable evidence of what's happening behind door number one, like. It's kind of goodwill and hope is is all that I'm writing on here. And I don't wanna I don't wanna play it too harshly because again, there's there's lots of really, really good but but to me Well and also there have been structural reasons for that, right? Totally. It's kinda like back to the MQL. Like I I would say, Well, I understand why we're tracking this. It's trackable, it's understandable, it's trendable, it's better than having zero. But again, every marketer would say it could be better. And I I feel like that's very much where what this data showed was, hey, I have a system for this. I'm doing what the industry standard is, but I know there's probably a better way out there. Yeah. So I think that the almost 90% saying they had real blind spots was sort of a, hey, I would love a better way. That is at the core of this Latin at no level. resistance or kind of people crossing their arms and stomping their feet of saying, like, I love the way it's working for me. In fact, it was quite the opposite. And yet, so many leaders find themselves in this place where you have to make decisions. And I start to wonder when we look at, you know, consumer index reports not getting any better, when we look at employee engagement reports not getting any better, I've kind of walked away from this. research believing that a lot of organizations, like where they are landing is by by no no no kind of factor of their metrics. I think it's you could probably stop measuring a lot of what you're doing today and you're going to get there anyway because it's not really actionable. It's not really moving the needle for you. Well, this is what got me pretty fired up because as someone who's been so passionate about customer experience my entire career, it's A passion for wanting to work on it. That's all it is. Customer experiences, getting the insight, getting the data, making changes. It's a constant project. And I love a good project. And what that means is you have to be able to gather these insights and then get them to leadership, quantify them and make a change and do it again and again and again. That's the system for why you even do it. And we found that only one in four companies ever take. recommendations from their service team, which is horrible. So less than half of the CX leaders said that they were sharing actionable insights outside of their department. And then of the 50% that are sharing, it was only 50% were doing something about it. And it's it's like even I I think of it from two camps. One, shame on you if you're receiving insight from your organization and you're not actioning it. And and actioning it can be as simple as having a conversation about why why it's relevant right now, right? Give context to the leader on on how they're finding it valuable. But that other 50%, Latin, that to me is that's just opportunity. There's so much stuff that's happening inside of our customer experiences that if we have a good way to make it more visible and and distribute it across the organization, so much good could come from that. You know. The challenge is how you roll up those insights, right? And if you roll up those insights in a way that feels fuzzy or anecdotal or one-off or maybe not tied to business outcomes, they can be kind of, you know, farted off, right? And so I think part of what we wanna do together with this work is provide a great way to I call it uncover the dark journey. Like, let's let's have no more post-it notes, no more rooms full of people guessing about what the journey is. Like, let's really show it. Yeah. Well, and Letney, what I think also really matters here, and we did get into this in the research a bit, was just alignment between what operators say is important and how they think about measuring and defining success. And then executives in the same light, how they see success. Define inside of these these interactions. And while across the board, there wasn't just kind of wide gaps, there were certainly things where, as a service leader, if you can better translate it in terms that your executive thinks and cares about, that alone will help build trust and equity when you're you're giving these suggestions because you said it, it it's not connecting to the business of the business. And when that's the case, it's, you know, as as some of the executives I talked to in the study said, it's like kind of ignorable. And the last thing you want to be if you were leading this operation is ignorable. No, we want to set ourselves on fire and not be ignored. That's what I want to do. You know, if you think about it, everything that happens upstream in the experience rolls to service. So that's where the most rich insights exist. Kind of to that end, you were shocked at how many people were using MPS. There are a couple shocking things for me. One, I guess I'll I I've I've probably gone on record before, but if not, here it is. This is the moment. Latney, I have never been a fan of MPS, and I try to not bring my own bias into these studies, but the people spoke for me. Okay, okay. NPS, and this is interesting. So I've actually I I looked at different research studies that I've done over the years. And and mind you, these have been at different points in my career and and kind of different groups from what we've talked to. But NPS outside of customer satisfaction kind of had been the darling for a really long time. And the operators that we spoke to in this survey, well, we kind of put things in front of them. only fifteen percent of the operators talked about it in terms of a metric that they were looking to. What was really kind of the winner there was customer effort. We saw over sixty two percent of the people in the study who said that understanding and addressing the effort that it takes for these cer these experiences to be delivered was a bigger factor for them. So I think that is a sign of a more modern approach, actually. And it's recog well, what it's recognizing in in it's like tangible. One of my biggest ni about NPS, man, you got me on my soapbox. What am I? Yeah, let's soapbox. What what are break it down for us? So why are you not an MPS fan? Okay. A most people don't really understand the scale in terms of how it's it's applied and how businesses think about it. You know, what's a neutral, what's a distractor, what's a promoter. So so part one is just a misunderstanding. By the people taking it, but also how organizations apply it. Most often how I see it shared is in some sort of board or executive meeting, they talk about the number, maybe they celebrate and that's it. Totally lose sight of all the feedback, what's actionable, but maybe the biggest gap is just because you said you would doesn't mean you do. And there is this level of just because you said you will promote doesn't mean you do. Correct. And and w what I also know to be true about people is is most of us have this bias towards affirmation. We're not really good at like saying bad, mean, nasty things. What I watch play out is you have people who would promote, but then they never actually would. And and even if they did, it becomes untrackable by the brand. So just because you said you would do this, that doesn't really help me. And I mean to the to the other side of that. All I'll say is that the nature of this particular business was very sensitive. but I got a call from a CX leader a few years ago where her organization was using NPS as their de facto thing, but their customers, because of the sensitivity around this product, would never be a promoter. So their scores tanked, even though people were very happy. I've NPS'd a lot. I've done a lot of MPSing. And I found myself explaining the scoring. over and over and over again. Cause internally they were like, wait, ha that's a good score. You know, everyone thinks it should be like grades. Like you you should be at like ninety percent, right? And then you have to explain over and over again, no, no one's at we're not gonna have a ninety percent MPS. That's not how it works. And like explaining situations like that. And then it it I just got to the point where I like was less about the score and more about the verbatim. These are the top trends. Like, can we just focus on fixing things? That's that's the that's the magic of it all. Once you kind of remove the the promoter scale, what you're really talking about is voice of customer, customer insight, listening, server. It's like that's great. Exactly. And there, you know, there's other ways to do that versus just, you know, a survey. What else did we find? I I think that we found a ticking time bomb. related to AI. What's interesting about what we do, Latin, is like we live in this industry, but we are also consumers. And it's kind of like having your favorite restaurant spoiled when you read the when you read the report of of the kitchen inspection, right? So one of the things we discovered that among the organizations that are currently running AI inside of their customer service interactions, 78 and a half percent said that they have no measure For hallucination or factual accuracy. Say that again. This is this should scare everybody. My eye just started twitching. When we looked at the organizations who have AI happening right now, engaging with customers, seventy eight and a half percent have no way to measure hallucinations or factual accuracy. It's a black box. Their AI is a black box. I mean, don't get me wrong. I had some like members of my teams over the years where there also was occasionally hallucinations and and like things made up, but we had a way to understand what was happening. Yeah, but drafting a blog is different than interacting with customers and and especially as you get to more complicated use cases where you could be processing a payment. And and the the thing that I push on When people say it's like, hey, well, that can happen with people too. It's like, but this is a different scale and at a completely different pace of how it can change and evolve and get out into the wild if we don't have a way to have kind of some level of just reining it in. And that to me, Latin, was the thing that was like, we've got to rein this in as organizations before. It is the very end of us. Well, yeah. I mean, the the thing about AI is it's not like the system doesn't go down, right? Like there's there it's still operating. and so if you don't have observability, you real you really are staring at a black box. And especially with the way a lot of folks price, a lot of folks price based on containment or resolution. I mean the resolution could be up and to the right and looking fabulous. But that doesn't mean there's not risk, potentially a ton of risk. Honestly, when we then were looking at executives, you know, this idea of undetected errors reaching customers was top of mind for them. So there really is this tension between organizations saying we we can't see, and this is a pr a big worry for us, which, you know, of course is Part of why we're even having this conversation, right? Because you we need organizations who are bringing the technical capability to bear to go after this. Because there are some problems that that, especially as a researcher, I'll be like, you made that problem up to sell something. This is this is this is not that. Reading through this, I had a moment of this is a legitimate need. This is a genuine problem that is. It's it's necessary to address if organizations want to realize the potential future that I think most, if not all of us, want to see realized. That was eye-opening for me too, because my passion for observability is much more around that lighting of the journey. And, you know, again, the CMO thinks, my God, think of all the gold in those calls. And if I could call those insights. I'd never have to buy market research again, right? And so I get really excited about that. I get really excited about, you know, real-time CSAT and and friction analysis and real-time customer effort and all the confidence that it will give operators in in CS. But the true finding from this is the ticking time bomb, I believe, which is we have got to get eyes on the AI. Like, you got to get eyes on the AI. There's two things that get me excited and I r you know, remain very positive about all of this. One, that predictive, proactive future that you talked about, where I, if I'm a a CX leader, that I'm equipping my marketers or my sellers or my product team with intelligence and and we're able to just better serve and enable that like that's yes. This this is why everybody does this. And also to get there, there's just some basic stuff that let's like let's not get in our own way. And so that to me is this of like once you do this, solving like that that then very quickly becomes a solvable problem. So then you can chase after the other cool stuff. And I know you want to chase after the other cool stuff. So let's get these solvable problems solved. Right. Yeah, exactly. Exactly. So for those listening. What would you say they should do? For me, it always starts just get clarity on your current state. I would do the check on what's what's happening inside of these interactions. Run some tests to find out if you don't have visibility in in accuracy and and that kind of stuff today. Like you have to start there, get get some clarity into what's happening. the other thing that I'd say kind of quickly with this though is. Deciding who who's going to own this. one of the things we saw in the research was this lack of ownership across you know these different things. And you may you may feel the charge and get all excited about what needs to be done, but maybe you need to bring in a a a champion who can help you move this through the organization. Maybe you need to invite in a peer, maybe you need to ask somebody who's on your team. But I think identifying who's going to help you solve this problem is also kind of a key first step. Like get get clear about the problem is that that's always the first step, right? Name that you they're like, there's an issue here, and then figure out who's going to be part of the team that can help you go after digging into this deeper. So I'm gonna ask you a question I've asked you before and see if your answer has changed. Do you think there needs to be a chief customer officer? So and maybe it's a a new version of it. Cause maybe it maybe there's a reason why it didn't work before. Yeah. Latney, I think what every organization needs is someone who owns the customer experience. And where I've gotten up on that, maybe gotten caught up on the chief customer officer term of the past is a lot of organizations have done a not great job in terms of enabling and empowering that person to have the right level of. ownership and authority over the parts of the experience that they should be able to to influence and lead and control. And I think that that to me is the most again, looking at the the data here, that lack of ownership kind of pulls out of this mentality of customer experience is everybody's job. Well if it's everybody's job, then it becomes nobody's job. And and I I do think, you know, whether whether it's, you know, the the chief experience officer, whether it's chief customer officer Ownership over CX needs to happen inside of the executive level by a named accountable person. Absolutely. Yeah, I agree too. But I also think it needs the right end all be all. And and the reason I say that is I've had CX twice. Once I built, once I inherited. And I'll be honest. You know, I'm a pretty strong personality. You've you've probably already experienced that. So you know, I I'm someone who typically can get things done and moved and work cross functionally and all that, but I don't think I did the best job at it. When I look back, it was because I felt like we needed a score. And I think if I had been told or my mind had been, I'm my the result of this has to be customer lifetime value, I probably would have acted differently and been way more aggressive about the program. I think there's maybe three things that I I take away from that. Again, hearing out of your experience and and the others that I I've known and seen like it. One I actually want to touch back on something you said near the beginning of our conversation. That this is it's always changing, it's evolving. It it reminds me of the the the book Finite and Infinite Games. Customer experience is an infinite game. And when you do that and the scores resetting and the stakes are changing, like that makes it hard to be rooted in a measure or feel like you were successful and you won because you you don't like that that's part of it. A second thing on the customer lifetime value bit is it's a dance. It's not a perfect number and it's complicated. And I think that's one that sometimes it intimidates and it scares leaders or recognizing that it's not absolute. It's directional. And and I think part of that's just not enough leaders have other CX professionals either in their organization or in their life that are helping steer and say, Hey, you're not alone. I went through the same struggle. but I also Do you believe kind of as the third of this, it it also comes down to your peer group as well. Because even though you may be the leader responsible for customer experience, it still can and will not be done alone. You need alignment with product with support because all of those, right? It's it's it's one interconnected body. And and that I think is the other part of it, is often the people who step into CX leadership roles. They're really great humans, but maybe they don't have the drive and tenacity that other executives have. And I think sometimes that pairing can be difficult when you won't stand up for your program. You may own it and you may believe it, but it's like if you won't go to fight for it, then it can also be really difficult. Yeah. We have this beautiful, wonderful community. And we talk about like, God, it is such a fight. Like, you know, to get others to see what we know needs to be done, which again, I come back to this survey and I just want to give people the ammunition to not just take to the fight, but to go and win it, you know? Yeah. It's it's I go back to this, I say this word a lot, Latney, to me it's defensible. And every leader that is watching this, they they've all been doing really, really good work. And maybe it's been shot down, not because it's not good work, but because the evidence hasn't been there in such a way that the organization can believe and get behind it. And that doesn't mean they don't believe in you as the leader, but we need to make it defensible. And that's the other thing I get excited about when I look at me too, the findings and the opportunity here is it starts to give more substance and and and truth to the work that's happening every single day inside of these organizations. I'm I'm I could not be more excited and it was a pleasure to to work with you on this this research. My hope is that we do a lot, lot more. I'm I'm very passionate about it. What what's next for people? Like how can they learn more? I know you're gonna be talking about it a bunch. W where can they find you? Yeah, where can you find me? I mean, where we're all at once. Where in the world is Justin Robbins? Latney, the the all the easiest way is always is LinkedIn if you're on LinkedIn. If not, metricsherpa.com. It's easy enough to find me there. What's next? I mean, you said it well. How do we make sure that people action it? That that is kind of it. Like we did this the study, we have this report, great, but now the real work begins. Yeah. And that's what I think about in terms of what's next. You know, we've got some cool things that of course we're doing to resource leaders. but I think continuing to lean in, to continue to ask good questions and continue to build the business case for how and why customer experience is the the greatest generator of value your organization will ever have. That's what's next. Thank you, Justin of Metric Sherpa. Great conversation. This research is white hot. Everyone, you're gonna wanna check it out. Head to Parloa.com to get your copy.